A speech synthesis method, apparatus and device for speech synthesis

Abstract

Embodiments of the present application provide a speech synthesis method and device and a device for speech synthesis, applied to a terminal device. The method comprises: training a multi-person acoustic model based on multi-person speech training data, the multi-person acoustic model comprising an encoder, a prosody prediction network, a duration prediction network, and a decoder; the acoustic features output by the decoder comprising fundamental frequency features and mel spectrum features; adaptively training the multi-person acoustic model based on single-person speech training data of a target speaker to obtain a single-person acoustic model of the target speaker; performing parameter quantization processing on the single-person acoustic model to obtain a target single-person acoustic model; and synthesizing audio data of acoustic features of the target speaker using the target single-person acoustic model and text to be synthesized. Embodiments of the present application ensure model effectiveness, and make the target single-person acoustic model obtained by training applicable to offline devices with limited computing power and storage space.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, and in particular to a speech synthesis method and device and a device for speech synthesis. BACKGROUND

[0002] Speech synthesis, also known as text-to-speech (TTS), is a technology that can convert any input text into corresponding speech. Traditional offline speech synthesis methods mainly include waveform concatenation speech synthesis method and parameter analysis speech synthesis method.

[0003] The waveform concatenation speech synthesis method is based on the results of text analysis, and concatenates unit samples in a pre-recorded and annotated speech library to obtain a synthesized speech waveform. Because each unit in the synthesized sentence comes from a natural recording of a speaker, the waveform concatenation speech synthesis method better preserves the original sound quality of the speaker's recording. However, this method is affected by the size of the speech library and the unit selection algorithm, and has high requirements for hardware storage capacity, and cannot be applied to low-end offline devices.

[0004] In view of the shortcomings of the waveform concatenation speech synthesis method, a parameter analysis speech synthesis method is proposed. The parameter analysis speech synthesis method uses a vocoder to parameterize the speech in the corpus in the training stage, and statistically models the acoustic features. In the generation stage, the acoustic model obtained by training is used to predict the acoustic features from the text analysis results, and then the acoustic features are input into the vocoder to reconstruct the speech waveform corresponding to the text. Compared with the waveform concatenation speech synthesis method, the parameter analysis speech synthesis method has the advantages of strong model flexibility and small system size, and can be applied to low-end offline devices. However, the statistical modeling in the parameter analysis speech synthesis method can cause the problem of "over-smoothing" of acoustic feature parameters, and the vocoder can damage the sound quality, resulting in poor prosody and sound quality of the synthesized speech. SUMMARY

[0005] The embodiments of the present application provide a speech synthesis method, device and device for speech synthesis, which can ensure the model effect and make the target single acoustic model obtained by training applicable to offline devices with limited computing power and storage space.

[0006] To solve the above problems, the embodiments of the present application disclose a speech synthesis method, which comprises:

[0007] training a multi-person acoustic model based on multi-person speech training data, the multi-person acoustic model comprising an encoder, a prosody prediction network, a duration prediction network, and a decoder; the acoustic features output by the decoder include fundamental frequency features and mel spectrum features;

[0008] Adaptively training the multi-person acoustic model based on single-person speech training data of a target speaker to obtain a single-person acoustic model of the target speaker;

[0009] Parameter fixing the single-person acoustic model to obtain a target single-person acoustic model;

[0010] Synthesizing audio data of acoustic features of the target speaker using the target single-person acoustic model and to-be-synthesized text.

[0011] Optionally, the multi-person speech training data includes training data of each preset speaker, the training data of each preset speaker including speech data and corresponding text of the preset speaker, and acoustic features, text features, and duration information extracted from the speech data and the corresponding text; and the training of the multi-person acoustic model based on the multi-person speech training data includes:

[0012] Initializing model parameters of the multi-person acoustic model;

[0013] Inputting the text features into an encoder of the multi-person acoustic model to obtain a text encoding sequence corresponding to the text features;

[0014] Inputting the text encoding sequence and the duration information into a duration prediction network of the multi-person acoustic model to predict durations of characters / phonemes in the text encoding sequence, and outputting a duration sequence corresponding to the text encoding sequence;

[0015] Inputting the text encoding sequence into a prosody prediction network of the multi-person acoustic model to predict prosody features of characters / phonemes in the text encoding sequence, and outputting a prosody feature sequence corresponding to the text encoding sequence;

[0016] Inputting the duration sequence and the prosody feature sequence into a decoder of the multi-person acoustic model to output acoustic features corresponding to the text encoding sequence;

[0017] Iteratively updating the model parameters of the multi-person acoustic model according to differences between the predicted acoustic features and the extracted acoustic features.

[0018] Optionally, the iteratively updating the model parameters of the multi-person acoustic model according to the differences between the predicted acoustic features and the extracted acoustic features includes:

[0019] Inputting the predicted acoustic features and the extracted acoustic features into a discriminator of a generative adversarial network;

[0020] Calculating a model error of the multi-person acoustic model according to a discrimination result output by the discriminator;

[0021] updating parameters of the multi-person acoustic model and parameters of the discriminator iteratively through the model error.

[0022] Optionally, the updating the model parameters of the multi-person acoustic model according to the difference between the predicted acoustic features and the extracted acoustic features iteratively comprises:

[0023] updating parameters of the encoder, and / or the prosody prediction network, and / or the duration prediction network, and / or the decoder according to the difference between the predicted acoustic features and the extracted acoustic features iteratively.

[0024] Optionally, the outputting the acoustic features corresponding to the text encoding sequence comprises:

[0025] The decoder outputs the frame-level acoustic features based on a gated recurrent unit or a causal convolution.

[0026] Optionally, the synthesizing the audio data of the acoustic features of the target speaker by using the target single-person acoustic model and the text to be synthesized comprises:

[0027] inputting the text to be synthesized into the target single-person acoustic model, and outputting the acoustic features of the target speaker by the target single-person acoustic model;

[0028] inputting the acoustic features of the target speaker into a pre-trained vocoder to generate the audio data.

[0029] Optionally, the multi-person acoustic model and the single-person acoustic model comprise compressed neural network parameters.

[0030] In another aspect, an embodiment of the present application discloses a speech synthesis device, which comprises:

[0031] a first training module configured to train a multi-person acoustic model based on multi-person speech training data, the multi-person acoustic model comprising an encoder, a prosody prediction network, a duration prediction network, and a decoder, wherein the acoustic features output by the decoder comprise fundamental frequency features and mel spectrum features;

[0032] a second training module configured to perform adaptive training on the multi-person acoustic model based on single-person speech training data of a target speaker, so as to obtain a single-person acoustic model of the target speaker;

[0033] a fixed-point processing module configured to perform parameter fixed-point processing on the single-person acoustic model, so as to obtain a target single-person acoustic model;

[0034] a speech synthesis module configured to synthesize audio data of acoustic features of the target speaker by using the target single-person acoustic model and text to be synthesized.

[0035] Optionally, the multi-person speech training data comprises training data of each preset speaker, the training data of each preset speaker comprising speech data and corresponding text of the preset speaker, and acoustic features, text features, and duration information extracted from the speech data and the corresponding text; the first training module comprises:

[0036] an initialization submodule configured to initialize model parameters of the multi-person acoustic model;

[0037] an encoding submodule configured to input the text features into an encoder of the multi-person acoustic model to obtain text encoding sequences corresponding to the text features;

[0038] a duration prediction submodule configured to input the text encoding sequences and the duration information into a duration prediction network of the multi-person acoustic model to predict durations of characters / phonemes in the text encoding sequences, and output duration sequences corresponding to the text encoding sequences;

[0039] a prosody prediction submodule configured to input the text encoding sequences into a prosody prediction network of the multi-person acoustic model to predict prosody features of characters / phonemes in the text encoding sequences, and output prosody feature sequences corresponding to the text encoding sequences;

[0040] an acoustic feature prediction submodule configured to input the duration sequences and the prosody feature sequences into a decoder of the multi-person acoustic model to output acoustic features corresponding to the text encoding sequences;

[0041] a parameter updating submodule configured to iteratively update the model parameters of the multi-person acoustic model according to a difference between the predicted acoustic features and the extracted acoustic features.

[0042] Optionally, the parameter updating submodule comprises:

[0043] a data input unit configured to input the predicted acoustic features and the extracted acoustic features into a discriminator of a generative adversarial network;

[0044] an error calculation unit configured to calculate a model error of the multi-person acoustic model according to a discrimination result output by the discriminator;

[0045] a parameter updating unit configured to iteratively update parameters of the multi-person acoustic model and parameters of the discriminator through the model error.

[0046] Optionally, the parameter updating submodule is specifically configured to iteratively update parameters of the encoder, and / or the prosody prediction network, and / or the duration prediction network, and / or the decoder according to the difference between the predicted acoustic features and the extracted acoustic features.

[0047] Optionally, the acoustic feature prediction submodule is specifically configured to output frame-level acoustic features based on a gated recurrent unit or a causal convolution through the decoder.

[0048] Optionally, the speech synthesis module comprises:

[0049] a feature prediction submodule configured to input a text to be synthesized into the target single-person acoustic model, and output acoustic features of the target speaker through the target single-person acoustic model;

[0050] an audio conversion submodule configured to input the acoustic features of the target speaker into a pre-trained vocoder to generate audio data.

[0051] In still another aspect, an embodiment of the present application discloses a device for speech synthesis, the device comprising a memory, and one or more programs stored in the memory and configured to be executed by one or more processors, the one or more programs containing instructions for performing any of the above-mentioned speech synthesis methods.

[0052] In still another aspect, an embodiment of the present application discloses a machine-readable medium having instructions stored thereon, which, when executed by one or more processors, cause a device to perform any of the above-mentioned speech synthesis methods.

[0053] Embodiments of the present application include the following advantages:

[0054] The embodiment of the present application trains a multi-person acoustic model based on multi-person voice training data, and performs adaptive training on the trained multi-person acoustic model based on single-person voice training data of a target speaker to obtain a single-person acoustic model of the target speaker. The single-person acoustic model of the target speaker can be trained through a small amount of single-person voice data of the target speaker, which can improve the efficiency of training the single-person acoustic model and reduce the cost of collecting single-person voice data. In addition, the acoustic model (multi-person acoustic model / single-person acoustic model) in the embodiment of the present application includes an encoder, a prosody prediction network, a duration prediction network, and a decoder, the prosody prediction network, the duration prediction network, and the timbre prediction network (reflected in the decoder) are taken as three independent modules, and end-to-end joint training is performed, and the three modules are independently parameterized, which helps to improve the accuracy of each module of the acoustic model. In addition, the three modules interact with each other to make the multi-person acoustic model / single-person acoustic model achieve the best effect. Furthermore, after the multi-person acoustic model is adaptively trained based on the single-person voice training data of the target speaker to obtain the single-person acoustic model of the target speaker, the single-person acoustic model is subjected to parameter quantization processing to obtain a target single-person acoustic model, which can greatly reduce the time complexity and space complexity of the model and the storage space required by the model without reducing the performance of the model, and on the basis of ensuring the effect of the model, the finally obtained target single-person acoustic model can be applied to offline devices with limited computing power and storage space. Further, the acoustic features output by the decoder include fundamental frequency features and mel spectrum features, and the prediction of the fundamental frequency features can help the vocoder to solve the mute problem and improve the sound quality. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0056] Figure 1 is a step flowchart of a speech synthesis method embodiment of the present application;

[0057] Figure 2 is a structural schematic diagram of an acoustic model of the present application;

[0058] Figure 3 is a flowchart of training a multi-person acoustic model of the present application;

[0059] Figure 4 is a structural block diagram of a speech synthesis device embodiment of the present application;

[0060] Figure 5 is a block diagram of an apparatus 800 for speech synthesis according to an embodiment of the present application;

[0061] Figure 6 is a structural diagram of a server in some embodiments of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0063] Method embodiments

[0064] Referring to Figure 1 , a step flowchart of an embodiment of a speech synthesis method of the present application is shown, which can specifically include the following steps:

[0065] Step 101, training a multi-person acoustic model based on multi-person speech training data, the multi-person acoustic model including an encoder, a prosody prediction network, a duration prediction network, and a decoder; the acoustic features output by the decoder include fundamental frequency features and mel spectrum features;

[0066] Step 102, performing adaptive training on the multi-person acoustic model based on single-person speech training data of a target speaker to obtain a single-person acoustic model of the target speaker;

[0067] Step 103, performing parameter fixed-point processing on the single-person acoustic model to obtain a target single-person acoustic model;

[0068] Step 104, synthesizing audio data of acoustic features of the target speaker using the target single-person acoustic model and to-be-synthesized text.

[0069] The embodiment of the present application adopts an end-to-end speech synthesis scheme, realizes direct input of text and then output of acoustic features, and further conversion into audio through a neural network learning method. Through the end-to-end speech synthesis scheme, the originally complex language analysis part is greatly simplified, and the speech synthesis effect is further optimized, and the sound is closer to real people.

[0070] Further, the embodiment of the present application first trains a multi-person acoustic model based on multi-person speech training data, and then performs adaptive training on the trained multi-person acoustic model based on single-person speech training data of a target speaker to obtain a single-person acoustic model of the target speaker.

[0071] The multi-person speech training data refers to training set data of multiple speakers, which can be a plurality of preset speakers or some speakers, such as Zhang San, Li Si, and Wang Wu, and each speaker has a corresponding speaker identifier in the multi-person acoustic model. For example, the multi-person acoustic model is trained according to multi-person speech training data including speaker identifiers ID1, ID2, ID3, ID4, and ID5. After the multi-person acoustic model is trained, the speaker identifier ID1 and a certain text are input, and the acoustic feature corresponding to the speaker identifier ID1 can be output through the multi-person acoustic model. Similarly, the speaker identifier ID2 and a certain text are input, and the acoustic feature corresponding to the speaker identifier ID2 can be output through the multi-person acoustic model.

[0072] After the multi-person acoustic model is obtained through large-scale multi-person speech training, the multi-person acoustic model can be used as an initial single-person acoustic model of the target speaker, and the initial single-person acoustic model (the trained multi-person acoustic model) is further trained using small-scale single-person speech training data from the target speaker. The purpose of further training the model is to fine-tune the transfer parameters of the trained multi-person acoustic model, so that the output parameters of the model are closer to the acoustic parameters of the real speech of the target speaker. The data scale of the corpus data from the target speaker used in the further training is relatively small. Therefore, the single-person acoustic model of the target speaker can be trained through a small amount of single-person speech data of the target speaker, which can improve the efficiency of training the single-person acoustic model and reduce the cost of collecting single-person speech data.

[0073] In an optional embodiment of the present application, the multi-person acoustic model and the single-person acoustic model can include compressed neural network parameters. In order to make the trained single-person acoustic model applicable to low-end offline devices, so that the offline devices can run the single-person acoustic model for speech synthesis in a network-free environment, the neural network parameters of the multi-person acoustic model and the single-person acoustic model are reduced in the embodiment of the present application to simplify the model structure and reduce the model size.

[0074] It should be noted that the present embodiment does not limit the way of compressing the neural network parameters. In one example, the parameter amount of the neural network parameters can be reduced from 256 dimensions to 128 dimensions. It should be noted that the present embodiment does not limit the way of compressing the neural network parameters. In specific implementation, the parameter amount of the neural network parameters of the acoustic model can be determined according to the computing power of the offline device and the speech synthesis effect.

[0075] Further, in order to guarantee the quality of the acoustic model prediction result under the condition of compressing the model parameters, the embodiment of the present application independently takes out the prediction process of the prosody feature and the prediction process of the duration information as an independent module in the acoustic model during the process of training the multi-person acoustic model and the single-person acoustic model, and performs end-to-end joint training with the acoustic model.

[0076] In the embodiment of the present application, the multi-person acoustic model and the single-person acoustic model adopt the same model structure. Referring to Figure 2 , a structural schematic diagram of an acoustic model in the embodiment of the present application is shown. As Figure 2 shown, the multi-person acoustic model and the single-person acoustic model can include an encoder 201, a prosody prediction network 202, a duration prediction network 203, and a decoder 204. The encoder 201 is used to encode the input text feature to obtain a text encoding sequence convenient for model processing. The prosody prediction network 202 is used to predict the prosody feature corresponding to each character or phoneme. The prosody prediction network 202 is an independent module and is obtained by joint training during the training of the end-to-end acoustic model. The prosody feature belongs to a phonological structure of a language and is closely related to other linguistic structures such as syntax and discourse structure, information structure, etc. The prosody feature can be divided into three main aspects: intonation, time domain distribution, and stress. The duration prediction network 203 is used to predict the duration information corresponding to each character or phoneme. The duration prediction network 203 is an independent module and is obtained by joint training during the training of the end-to-end acoustic model. The decoder 204 is used to predict the acoustic feature conforming to the preset speaker timbre based on the input text feature and the duration sequence and the prosody feature sequence corresponding to the text feature. Further, the acoustic feature output by the decoder contains the fundamental frequency feature and the mel spectrum feature conforming to the preset speaker timbre.

[0077] In the embodiment of the present application, without changing the structure of the decoder, one-dimensional fundamental frequency features are output by the decoder. The decoder can not only predict the mel spectrum feature of each frame, but also predict the fundamental frequency feature of each frame. The fundamental frequency feature is an acoustic feature reflecting the size, thickness, tightness of the vocal cords, the effect of the air pressure difference between the glottis, and the tone change. The prediction of the fundamental frequency feature can help the vocoder to solve the mute problem and improve the sound quality.

[0078] The embodiment of the present application constructs the acoustic model of joint modeling of prosody, duration, and timbre, which not only can greatly reduce the model size, but also can improve the prediction effect of the model.

[0079] In an optional embodiment of the present application, the multi-person speech training data comprises training data of each preset speaker, the training data of each preset speaker comprising speech data and corresponding text of the preset speaker, and acoustic features, text features, and duration information extracted from the speech data and the corresponding text; and the training of the multi-person acoustic model based on the multi-person speech training data can comprise:

[0080] Step S11, initializing model parameters of the multi-person acoustic model;

[0081] Step S12, inputting the text features into an encoder of the multi-person acoustic model to obtain a text encoding sequence corresponding to the text features;

[0082] Step S13, inputting the text encoding sequence and the duration information into a duration prediction network of the multi-person acoustic model to predict durations of characters / phonemes in the text encoding sequence, and outputting a duration sequence corresponding to the text encoding sequence;

[0083] Step S14, inputting the text encoding sequence into a prosody prediction network of the multi-person acoustic model to predict prosody features of characters / phonemes in the text encoding sequence, and outputting a prosody feature sequence corresponding to the text encoding sequence;

[0084] Step S15, inputting the duration sequence and the prosody feature sequence into a decoder of the multi-person acoustic model to output acoustic features corresponding to the text encoding sequence;

[0085] Step S16, iteratively updating the model parameters of the multi-person acoustic model according to a difference between the predicted acoustic features and the extracted acoustic features.

[0086] In the embodiment of the present application, the multi-person speech training data refers to multi-speaker training set data. The multi-person speech training data can comprise preset training data of each preset speaker, and the training data of each preset speaker can comprise speech data and corresponding text of the preset speaker, and acoustic features, text features, and duration information extracted from the speech data and the corresponding text.

[0087] Referring to Figure 3 , a flowchart of training a multi-person acoustic model is shown.

[0088] The training of the multi-person acoustic model in the embodiment of the present application can comprise two stages. The first stage is data preparation, including text preprocessing, acoustic feature extraction, and duration information extraction; and the second stage is training of the multi-person acoustic model using data obtained in the first stage (including text features obtained after text preprocessing, acoustic features obtained through acoustic feature extraction, and duration information obtained through duration information extraction).

[0089] In order to ensure the learning accuracy of the model, the extraction of the text feature and the acoustic feature can be realized through a series of means, and various ways are adopted in the implementation, for example, a manually labeled way or a computer-aided manually labeled way. The embodiments of the present application do not limit this.

[0090] In the embodiments of the present application, the text preprocessing can include filtering out useless characters and format standardization processing on the text corresponding to the voice data of the preset speaker, so that a text input sequence (which can be a character sequence here) that is convenient for subsequent acoustic model processing can be obtained. Then, the text input sequence is converted into a vector to obtain a text feature, and then the text feature can be input into an encoder. It should be noted that the representation form of the text input sequence can also be in the form of a phoneme (phone) sequence, and the phoneme is the smallest unit of speech divided according to the natural properties of speech. The processing of the acoustic model on the phoneme sequence is the same as the processing of the character sequence.

[0091] The embodiments of the present application can model based on characters or phonemes, convert the text input sequence into a vector to obtain a text feature sequence that is convenient for model processing. Specifically, each character or phoneme in the text input sequence can be converted into a corresponding fixed-dimensional vector, and after each input character or phoneme is converted into a vector, a representation sequence of the text feature can be obtained, and then the encoder can be input for processing.

[0092] In addition, the voice data of the preset speaker and the corresponding text are automatically segmented to obtain the duration information of the voice data and the corresponding text.

[0093] Specifically, first, the model parameters of the multi-person acoustic model are initialized. The model parameters of the multi-person acoustic model can be compressed model parameters. Then, the text features are input into an encoder of the multi-person acoustic model to obtain a text encoding sequence corresponding to the text features. The network structure of the encoder is not limited in the embodiments of the present application. In one example, the encoder can include n encoding layers, and n is greater than or equal to 1. Next, the text encoding sequence and the duration information are input into a duration prediction network of the multi-person acoustic model to predict the duration of each character / phoneme in the text encoding sequence, and output a duration sequence corresponding to the text encoding sequence. The duration prediction network is an independent module, which is trained end-to-end with the acoustic model. In the training process, the duration prediction network is used to predict the duration information corresponding to each character / phoneme, and then obtain the duration sequence corresponding to the input text encoding sequence. Meanwhile, the text encoding sequence is input into a prosody prediction network of the multi-person acoustic model to predict the prosodic features of each character in the text encoding sequence, and output a prosodic feature sequence corresponding to the text encoding sequence. The prosody prediction network is an independent module, which is trained end-to-end with the acoustic model. In the training process, the prosody prediction network is used to predict the prosodic features corresponding to each character / phoneme, and then obtain the prosodic feature sequence corresponding to the input text encoding sequence. Finally, the duration sequence and the prosodic feature sequence are input into a decoder of the multi-person acoustic model to output acoustic features corresponding to the text encoding sequence, and update the model parameters of the acoustic model according to the difference between the predicted acoustic features and the extracted acoustic features. The above process is repeated, and after several rounds of iterative training, when the difference between the calculated predicted acoustic features and the extracted acoustic features is less than a preset value, the training is completed, and a trained multi-person acoustic model is obtained.

[0094] In the embodiments of the present application, the acoustic features can be acoustic features extracted based on speech, such as linear spectrum, MFCC (Mel-scale Frequency Cepstral Coefficients), Fbank, etc.

[0095] Further, the iterative updating of the model parameters of the multi-person acoustic model according to the difference between the predicted acoustic features and the extracted acoustic features can include: iteratively updating the parameters of the encoder, and / or the prosody prediction network, and / or the duration prediction network, and / or the decoder according to the difference between the predicted acoustic features and the extracted acoustic features.

[0096] In a specific implementation, the process of voice synthesis includes at least three important information, i.e., timbre information, prosody information and duration information, wherein the timbre represents the voice characteristic of a speaker in the voice; the timbre of the same speaker is the same, and the timbres of different speakers are different. The prosody represents the pronunciation rhythm of the speaker in the voice; and the duration represents the pronunciation length of each phoneme.

[0097] In the embodiment of the present application, the prosody prediction network, the duration prediction network and the timbre prediction network (implemented in the decoder) are taken as three independent modules for end-to-end joint training, and the independent parameter adjustment of the three modules helps to improve the accuracy of each module of the acoustic model. In addition, the three modules reach the best effect of the multi-person acoustic model / single-person acoustic model through mutual influence.

[0098] It should be noted that the process of training the single-person acoustic model is the same as the process of training the multi-person acoustic model shown in Figure 3 , and details are not repeated here.

[0099] In an optional embodiment of the present application, the iterative updating of the model parameters of the multi-person acoustic model according to the difference between the predicted acoustic features and the extracted acoustic features comprises:

[0100] Step S21, inputting the predicted acoustic features and the extracted acoustic features into a discriminator of a generative adversarial network;

[0101] Step S22, calculating a model error of the multi-person acoustic model according to a discrimination result output by the discriminator;

[0102] Step S23, iteratively updating the parameters of the multi-person acoustic model and the parameters of the discriminator through the model error.

[0103] The multi-person acoustic model trained based on large-scale multi-person voice training data often has an over-smoothing phenomenon, which is reflected in that the predicted spectrum tends to be the mean value, and the spectrum is relatively fuzzy, resulting in that the synthesized voice sounds relatively dull. To solve this problem, in the process of training the end-to-end acoustic model (multi-person acoustic model / single-person acoustic model), the embodiment of the present application introduces a generative adversarial network (GAN, Generative Adversarial Networks) to increase the variance of the spectrum, so that the distribution is closer to the real spectrum, and the sound quality of the synthesized voice is improved.

[0104] The generative adversarial network includes a generator G and a discriminator D. During the model training process, D receives true data and false data generated by G, and the task of D is to determine whether the data generated by G belongs to true data or false data. According to the determination result of D, the parameters of G and D can be adjusted at the same time. If D judges correctly, the parameters of G need to be adjusted to make the generated false data more realistic; if D judges incorrectly, the parameters of D need to be adjusted to avoid similar errors in the next judgment. The training will continue until G and D enter a balanced and harmonious state.

[0105] In the embodiment of the present application, the decoder in the acoustic model (multi-person acoustic model / single-person acoustic model) is used as the generator of the generative adversarial network. After the decoder outputs the predicted acoustic features, the predicted acoustic features and the pre-extracted acoustic features are input into the discriminator of the generative adversarial network. According to the determination result output by the discriminator, the model error of the acoustic model is calculated, which includes the error generated by the multi-person acoustic model and the error generated by the discriminator. The error generated by the multi-person acoustic model can be used to represent the accuracy of the predicted acoustic features; the error generated by the discriminator can be used to represent the accuracy of the determination result of the discriminator. In each round of training, the parameters of the multi-person acoustic model are adjusted according to the error generated by the multi-person acoustic model using the method of gradient back propagation, including adjusting the parameters of the encoder, and / or the prosody prediction network, and / or the duration prediction network, and / or the decoder, and adjusting the parameters of the discriminator according to the error generated by the discriminator, so as to supervise the acoustic feature distribution generated by the acoustic model to be closer to the real feature. When the model error is less than a preset value, the training is completed, and the trained acoustic model (multi-person acoustic model / single-person acoustic model) is obtained.

[0106] In an optional embodiment of the present application, the acoustic features corresponding to the text encoding sequence can be output by the decoder based on a gated recurrent unit or a causal convolution to output frame-level acoustic features.

[0107] In the embodiment of the present application, the processing processes of the encoder, the prosody prediction network, and the duration prediction network are at the phoneme level. The processing speed at the phoneme level is faster, and the model parameters of the acoustic model in the embodiment of the present application are compressed, so that the three modules have the characteristics of fast processing speed on the premise of ensuring the model effect. The time consumption of the acoustic model is mainly in the process of predicting acoustic features by the decoder.

[0108] In order to improve the real-time performance of the speech synthesis process and reduce the delay, the decoder of the embodiment of the present application adopts a streaming processing mode. Specifically, the decoder can output frame-level acoustic features based on GRU (Gated Recurrent Unit) or causal convolutions, to realize a low-delay streaming processing mode, improve the real-time performance of speech synthesis, and meet the requirements of high real-time performance of the speech synthesis scene.

[0109] GRU is a commonly used gated recurrent neural network, which is obtained by simplifying LSTM (Long Short-Term Memory). GRU is a model that maintains the effect of LSTM, has a simpler structure, fewer parameters, and better convergence, so the training speed of GRU is faster than that of LSTM.

[0110] When processing sequence problems, because time needs to be considered, that is, time t can only consider the input at time t and before, so a normal CNN (Convolutional Neural Networks) cannot be used, and causal convolution is generated accordingly. The role of causal convolution is to use CNN to simulate LSTM.

[0111] It should be noted that the above decoder outputs frame-level acoustic features based on GRU or causal convolutions to realize a low-delay streaming processing mode, which is only an application example, and the streaming processing mode of the decoder is not limited in the embodiment of the present application.

[0112] After the multi-person acoustic model is adaptively trained based on the single-person speech training data of the target speaker to obtain the single-person acoustic model of the target speaker, the single-person acoustic model can be subjected to parameter quantization processing to obtain a target single-person acoustic model.

[0113] In the parameter quantization processing, the parameters of the acoustic model are converted from float type to int8 type. In this way, the model changes from originally occupying 4 bytes per float data to occupying 1 byte per int8 data, which greatly reduces the time complexity and space complexity of the model and the required storage space of the model without reducing the performance of the model, so that the finally obtained target single-person acoustic model is more suitable for offline devices with limited storage space.

[0114] It should be noted that the embodiments of the present application do not limit the electronic device for training the multi-person acoustic model and the single-person acoustic model. The electronic device can be a physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0115] After the parameter quantization processing of the single-person acoustic model of the target speaker is performed to obtain the target single-person acoustic model, the target single-person acoustic model can be deployed to an offline device for use.

[0116] The offline device can include but is not limited to earphones, voice recorders, home intelligent terminals (including air conditioners, refrigerators, electric rice cookers, water heaters, etc.), business intelligent terminals (including video phones, conference table intelligent terminals, etc.), wearable devices (including smart watches, smart glasses, etc.), financial intelligent terminals, and smart phones, tablet computers, personal digital assistants (PDA), vehicle-mounted devices, computers, etc.

[0117] In an optional embodiment of the present application, the audio data of the acoustic characteristics of the target speaker synthesized by using the target single-person acoustic model and the text to be synthesized can include:

[0118] In step S31, the text to be synthesized is input into the target single-person acoustic model, and the acoustic characteristics of the target speaker are output by the target single-person acoustic model.

[0119] In step S32, the acoustic characteristics of the target speaker are input into a pre-trained vocoder to generate audio data.

[0120] After the target single-person acoustic model is deployed to the offline device, voice synthesis can be realized on the offline device. The text to be synthesized is input into the target single-person acoustic model, and the acoustic characteristics of the target speaker are output by the target single-person acoustic model. Then, the acoustic characteristics of the target speaker are input into a pre-trained vocoder to generate audio data.

[0121] The text to be synthesized can be any text. The text to be synthesized can be directly input text information, or text information obtained by scanning and recognizing through a camera. The text information obtained by scanning and recognizing through the camera can include text information obtained after scanning and text recognition processing of an image.

[0122] In an example, after obtaining the text information by scanning and recognizing through the camera of the offline device, the text information can be input into the target single-person acoustic model deployed in the offline device as the to-be-synthesized text, the acoustic features of the target speaker can be output by the target single-person acoustic model, and the acoustic features of the target speaker can be input into the pre-trained vocoder to generate audio data.

[0123] Therefore, by the embodiments of the present application, end-to-end speech synthesis can be implemented on a low-end offline device. Based on any input to-be-synthesized text, audio data of the to-be-synthesized text and conforming to the acoustic features of the target speaker can be directly output, and the synthesized audio data.

[0124] In summary, the embodiments of the present application train a multi-person acoustic model based on multi-person speech training data, and perform adaptive training on the trained multi-person acoustic model based on single-person speech training data of the target speaker to obtain a single-person acoustic model of the target speaker. The single-person acoustic model of the target speaker can be trained based on a small amount of single-person speech data of the target speaker, which can improve the efficiency of training the single-person acoustic model and reduce the cost of collecting single-person speech data. In addition, the acoustic model (multi-person acoustic model / single-person acoustic model) in the embodiments of the present application includes an encoder, a prosody prediction network, a duration prediction network, and a decoder, and the prosody prediction network, the duration prediction network, and the timbre prediction network (reflected in the decoder) are three independent modules, which are jointly trained in an end-to-end manner. The three modules are independently parameterized, which helps to improve the accuracy of each module of the acoustic model. In addition, the three modules interact with each other to achieve the best effect of the multi-person acoustic model / single-person acoustic model. Furthermore, after the multi-person acoustic model is adaptively trained based on the single-person speech training data of the target speaker to obtain the single-person acoustic model of the target speaker, the single-person acoustic model is subjected to parameter quantization processing to obtain a target single-person acoustic model. Without reducing the performance of the model, the time complexity and the space complexity of the model can be greatly reduced, and the storage space required by the model can be reduced. On the basis of ensuring the effect of the model, the finally obtained target single-person acoustic model can be applicable to offline devices with limited computing power and storage space. Further, the acoustic features output by the decoder include fundamental frequency features and mel spectrum features. The prediction of the fundamental frequency features can help the vocoder to solve the problem of hoarse voice and improve the sound quality.

[0125] It should be noted that, for the method embodiments, the series of acts / combinations thereof are described for simplicity, but those skilled in the art should understand that the embodiments of the present application are not limited by the order of the acts described, since, according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should understand that the embodiments described in the specification are all preferred embodiments, and the acts involved are not necessarily essential to the embodiments of the present application.

[0126] Apparatus embodiments

[0127] With reference to Figure 4 , a structural block diagram of an embodiment of a speech synthesis apparatus of the present application is shown, which can include:

[0128] A first training module 401 is configured to train a multi-person acoustic model based on multi-person speech training data, the multi-person acoustic model including an encoder, a prosody prediction network, a duration prediction network, and a decoder; the acoustic features output by the decoder include fundamental frequency features and mel-spectrum features;

[0129] A second training module 402 is configured to perform adaptive training on the multi-person acoustic model based on single-person speech training data of a target speaker, to obtain a single-person acoustic model of the target speaker;

[0130] A fixed-point processing module 403 is configured to perform parameter fixed-point processing on the single-person acoustic model, to obtain a target single-person acoustic model;

[0131] A speech synthesis module 404 is configured to synthesize audio data of acoustic features of the target speaker using the target single-person acoustic model and to-be-synthesized text.

[0132] Optionally, the multi-person speech training data includes training data of each preset speaker, the training data of each preset speaker including speech data and corresponding text of the preset speaker, and acoustic features, text features, and duration information extracted according to the speech data and the corresponding text; the first training module includes:

[0133] An initialization sub-module is configured to initialize model parameters of the multi-person acoustic model;

[0134] An encoding sub-module is configured to input the text features into the encoder of the multi-person acoustic model, to obtain text encoding sequences corresponding to the text features;

[0135] A duration prediction sub-module is configured to input the text encoding sequences and the duration information into the duration prediction network of the multi-person acoustic model, to predict durations of characters / phonemes in the text encoding sequences, and to output duration sequences corresponding to the text encoding sequences;

[0136] a prosody prediction sub-module, configured to input the text coding sequence into a prosody prediction network of the multi-person acoustic model, predict prosody features of each character / phoneme in the text coding sequence, and output a prosody feature sequence corresponding to the text coding sequence;

[0137] an acoustic feature prediction sub-module, configured to input the duration sequence and the prosody feature sequence into a decoder of the multi-person acoustic model, and output acoustic features corresponding to the text coding sequence;

[0138] a parameter updating sub-module, configured to iteratively update model parameters of the multi-person acoustic model according to a difference between the predicted acoustic features and the extracted acoustic features.

[0139] Optionally, the parameter updating sub-module comprises:

[0140] a data input unit, configured to input the predicted acoustic features and the extracted acoustic features into a discriminator of a generative adversarial network;

[0141] an error calculation unit, configured to calculate a model error of the multi-person acoustic model according to a discrimination result output by the discriminator;

[0142] a parameter updating unit, configured to iteratively update parameters of the multi-person acoustic model and parameters of the discriminator through the model error.

[0143] Optionally, the parameter updating sub-module is specifically configured to iteratively update parameters of the encoder, and / or the prosody prediction network, and / or the duration prediction network, and / or the decoder according to the difference between the predicted acoustic features and the extracted acoustic features.

[0144] Optionally, the acoustic feature prediction sub-module is specifically configured to output frame-level acoustic features based on a gated recurrent unit or a causal convolution through the decoder.

[0145] Optionally, the speech synthesis module comprises:

[0146] a feature prediction sub-module, configured to input to-be-synthesized text into the target single-person acoustic model, and output acoustic features of the target speaker through the target single-person acoustic model;

[0147] an audio conversion sub-module, configured to input the acoustic features of the target speaker into a pre-trained vocoder, and generate audio data.

[0148] Optionally, the multi-person acoustic model and the single-person acoustic model comprise compressed neural network parameters.

[0149] The embodiment of the present application trains a multi-person acoustic model based on multi-person voice training data, and performs adaptive training on the trained multi-person acoustic model based on single-person voice training data of a target speaker to obtain a single-person acoustic model of the target speaker. The single-person acoustic model of the target speaker can be trained through a small amount of single-person voice data of the target speaker, which can improve the efficiency of training the single-person acoustic model and reduce the cost of collecting single-person voice data. In addition, the acoustic model (multi-person acoustic model / single-person acoustic model) in the embodiment of the present application includes an encoder, a prosody prediction network, a duration prediction network, and a decoder, and the prosody prediction network, the duration prediction network, and the timbre prediction network (reflected in the decoder) are taken as three independent modules for end-to-end joint training. The three modules are independently parameterized, which helps to improve the accuracy of each module of the acoustic model. In addition, the three modules interact with each other to achieve the best effect of the multi-person acoustic model / single-person acoustic model. Furthermore, after the multi-person acoustic model is adaptively trained based on the single-person voice training data of the target speaker to obtain the single-person acoustic model of the target speaker, the single-person acoustic model is subjected to parameter quantization processing to obtain a target single-person acoustic model. On the basis of not reducing the performance of the model, the time complexity and the space complexity of the model can be greatly reduced, and the storage space required by the model can be reduced. On the basis of ensuring the effect of the model, the finally obtained target single-person acoustic model can be applied to offline devices with limited computing power and storage space. Further, the acoustic features output by the decoder include fundamental frequency features and mel spectrum features. The prediction of the fundamental frequency features can help the vocoder to solve the mute problem and improve the sound quality.

[0150] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts are described in the part of the method embodiment.

[0151] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts between the embodiments can be referred to each other.

[0152] As for the device in the above-mentioned embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0153] The embodiment of the present application provides a device for speech synthesis, the device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs contain instructions for: training a multi-person acoustic model based on multi-person speech training data, wherein the multi-person acoustic model comprises an encoder, a prosody prediction network, a duration prediction network and a decoder; the acoustic features output by the decoder comprise fundamental frequency features and mel spectrum features; performing adaptive training on the multi-person acoustic model based on single-person speech training data of a target speaker, to obtain a single-person acoustic model of the target speaker; performing parameter quantization processing on the single-person acoustic model, to obtain a target single-person acoustic model; and synthesizing audio data of acoustic features of the target speaker by using the target single-person acoustic model and to-be-synthesized text.

[0154] Optionally, the multi-person speech training data comprises training data of each preset speaker, wherein the training data of each preset speaker comprises speech data and corresponding text of the preset speaker, and acoustic features, text features and duration information extracted according to the speech data and the corresponding text; the training of the multi-person acoustic model based on the multi-person speech training data comprises:

[0155] initializing model parameters of the multi-person acoustic model;

[0156] inputting the text features into the encoder of the multi-person acoustic model, to obtain a text encoding sequence corresponding to the text features;

[0157] inputting the text encoding sequence and the duration information into the duration prediction network of the multi-person acoustic model, to predict durations of characters / phonemes in the text encoding sequence, and output a duration sequence corresponding to the text encoding sequence;

[0158] inputting the text encoding sequence into the prosody prediction network of the multi-person acoustic model, to predict prosody features of characters / phonemes in the text encoding sequence, and output a prosody feature sequence corresponding to the text encoding sequence;

[0159] inputting the duration sequence and the prosody feature sequence into the decoder of the multi-person acoustic model, to output acoustic features corresponding to the text encoding sequence;

[0160] iteratively updating the model parameters of the multi-person acoustic model according to differences between the predicted acoustic features and the extracted acoustic features.

[0161] Optionally, the iteratively updating the model parameters of the multi-person acoustic model according to the differences between the predicted acoustic features and the extracted acoustic features comprises:

[0162] input the predicted acoustic feature and the extracted acoustic feature into a discriminator of a generative adversarial network;

[0163] calculate a model error of the multi-person acoustic model according to a discrimination result output by the discriminator;

[0164] update parameters of the multi-person acoustic model and parameters of the discriminator iteratively through the model error.

[0165] Optionally, the updating the model parameters of the multi-person acoustic model iteratively according to the difference between the predicted acoustic feature and the extracted acoustic feature comprises:

[0166] updating parameters of the encoder, and / or the prosody prediction network, and / or the duration prediction network, and / or the decoder iteratively according to the difference between the predicted acoustic feature and the extracted acoustic feature.

[0167] Optionally, the outputting the acoustic feature corresponding to the text encoding sequence comprises:

[0168] the decoder outputs the acoustic feature at a frame level based on a gated recurrent unit or a causal convolution.

[0169] Optionally, the synthesizing the audio data of the acoustic feature of the target speaker by using the target single-person acoustic model and the text to be synthesized comprises:

[0170] inputting the text to be synthesized into the target single-person acoustic model, and outputting the acoustic feature of the target speaker by the target single-person acoustic model;

[0171] inputting the acoustic feature of the target speaker into a pre-trained vocoder to generate the audio data.

[0172] Optionally, the multi-person acoustic model and the single-person acoustic model comprise compressed neural network parameters.

[0173] Figure 5 is a block diagram of an apparatus 800 for speech synthesis according to an example embodiment. The apparatus 800 can be a mobile phone, computer, digital broadcast terminal, messaging device, gaming console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0174] Referring to Figure 5 , the apparatus 800 can include one or more of the following components: a processing component 802, a memory component 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0175] The processing component 802 generally controls the overall operations of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions

[0176] The memory 804 is configured to store various types of data to support operations of the device 800. Examples of these data include instructions to perform any of the applications or methods on the device 800, contact data, phonebook data, messages, pictures, videos, and so on. The memory 804 can be implemented by any type of volatile or nonvolatile memory devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disks, or optical disks.

[0177] The power component 806 supplies electrical power for the various components of the device 800. The power component 806 can include a power supply management system, one or more power sources, and other components associated with generating, managing and distributing electrical power for the device 800.

[0178] The multimedia component 808 includes a screen providing an output interface between the device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping and gestures on the touch panel. The touch sensors can not only sense a boundary of a touching or swiping action, but also detect duration and pressure associated with the touching or swiping action. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. The front and / or rear camera can receive external multimedia data when the device 800 is in an operating mode, such as a shooting mode or a video mode. Each of the front and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0179] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive an external audio signal when the device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0180] The I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which can include a keypad, a click wheel, buttons, and so on. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0181] The sensor component 814 includes one or more sensors for providing status assessments of various aspects of the device 800. For example, the sensor component 814 can detect an open / closed position of the device 800, relative positioning of components, such as a display and a keypad of the device 800, a change in position of a speech synthesis device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and a temperature change of the device 800, among a plethora of other examples. The sensor component 814 can include proximity sensor configured to detect presence of an object in a proximity without any physical contact. The sensor component 814 can also include a light sensor (e.g., a CMOS or CCD image sensor) configured to function in an imaging application. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0182] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a corresponding communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an example embodiment, the communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technology.

[0183] In exemplary embodiments, the apparatus 800 can be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic devices, to perform the above-described methods.

[0184] In exemplary embodiments, a non-transitory computer readable storage medium including instructions, such as the memory 804 including instructions, is also provided, which can be executed by the processor 820 of the apparatus 800 to complete the above-described methods. For example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0185] Figure 6 is a schematic diagram of a server in some embodiments of the present application. The server 1900 can vary greatly due to different configurations or performances, and can include one or more central processing units (CPUs) 1922 (e.g., one or more processors) and a memory 1932, one or more storage media 1930 (e.g., one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be temporary storage or persistent storage. The programs stored in the storage medium 1930 can include one or more modules (not shown in the figure), each of which can include a series of instruction operations in the server. Further, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.

[0186] The server 1900 can also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input / output interfaces 1958, one or more keyboards 1956, and / or one or more operating systems 1941, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0187] A non-transitory computer readable storage medium, when the instructions in the storage medium are executed by a processor of an apparatus (a server or a terminal), enable the apparatus to perform Figure 1 the voice synthesis method shown.

[0188] A kind of non-transitory computer readable storage medium, when the instruction in the storage medium is executed by the processor of device (server or terminal), it enables device to perform a kind of speech synthesis method, the method includes: training multi-person acoustic model based on multi-person speech training data, the multi-person acoustic model includes encoder, rhythm prediction network, time length prediction network and decoder;The acoustic feature that the decoder outputs includes fundamental frequency feature and mel spectrum feature;The multi-person acoustic model is adaptively trained based on the single-person speech training data of target speaker, obtains the single-person acoustic model of the target speaker;Parameter fixed-point processing is carried out to the single-person acoustic model, obtains target single-person acoustic model;The acoustic feature of audio data of the target speaker is synthesized using the target single-person acoustic model and the text to be synthesized.

[0189] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application be limited only by the scope of the claims, including equivalents thereof, and that the specification and drawings be for purposes of illustration only. The true scope and spirit of the application will be apparent to those skilled in the art from the following claims.

[0190] It is to be understood that the application is not limited to the precise construction herein disclosed and shown in the attached drawings, and that various changes in form and detail can be made therein without departing from the spirit and scope thereof. The scope of the application should only be limited by the appended claims.

[0191] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application, any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the protection scope of the present application.

[0192] The above provides a kind of speech synthesis method, a kind of speech synthesis device and a device for speech synthesis provided by the present application, the principle and implementation mode of the present application are described in this paper, the above embodiment is only used to help understand the method and core idea of the present application;Meanwhile, for the general technical personnel in the art, according to the idea of the present application, there will be changes in specific implementation mode and application range, the above description of the specification should not be understood as the limitation of the present application.

Claims

1. A speech synthesis method, characterized in that, The method includes: A multi-person acoustic model is trained based on multi-person speech training data. The multi-person acoustic model includes an encoder, a prosody prediction network, a duration prediction network, and a decoder. The acoustic features output by the decoder include fundamental frequency features and Mel spectrum features. The prosody prediction network, duration prediction network, and timbre prediction network are independent of each other and are jointly trained end-to-end. The multi-person acoustic model is adaptively trained based on the single-person speech training data of the target speaker to obtain the single-person acoustic model of the target speaker. The single-person acoustic model is subjected to parameter localization processing to obtain the target single-person acoustic model; Audio data of the acoustic features of the target speaker are synthesized using the target single-person acoustic model and the text to be synthesized. The training of the multi-person acoustic model based on multi-person speech training data includes: The text encoding sequence is input into the prosodic prediction network of the multi-person acoustic model to predict the prosodic features of each character / phoneme in the text encoding sequence, and outputs the prosodic feature sequence corresponding to the text encoding sequence. The duration sequence and the prosodic feature sequence are input into the decoder of the multi-person acoustic model, and the predicted acoustic features corresponding to the text encoding sequence are output. The model parameters of the multi-person acoustic model are iteratively updated based on the difference between the predicted acoustic features and the extracted acoustic features.

2. The method according to claim 1, characterized in that, The multi-person speech training data includes training data for each preset speaker, wherein the training data for each preset speaker includes the speaker's speech data and corresponding text, as well as acoustic features, text features, and duration information extracted based on the speech data and corresponding text; the training of the multi-person acoustic model based on the multi-person speech training data further includes: Initialize the model parameters of the multi-person acoustic model; The text features are input into the encoder of the multi-person acoustic model to obtain the text encoding sequence corresponding to the text features; The text encoding sequence and the duration information are input into the duration prediction network of the multi-person acoustic model to predict the duration of each character / phoneme in the text encoding sequence and output the duration sequence corresponding to the text encoding sequence.

3. The method according to claim 2, characterized in that, The iterative updating of the model parameters of the multi-person acoustic model based on the difference between the predicted acoustic features and the extracted acoustic features includes: The predicted acoustic features and the extracted acoustic features are input into the discriminator of the generative adversarial network; Based on the discrimination result output by the discriminator, the model error of the multi-person acoustic model is calculated; The parameters of the multi-person acoustic model and the parameters of the discriminator are updated iteratively through the model error.

4. The method according to claim 2, characterized in that, The iterative updating of the model parameters of the multi-person acoustic model based on the difference between the predicted acoustic features and the extracted acoustic features includes: The parameters of the encoder, and / or prosodic prediction network, and / or duration prediction network, and / or decoder are iteratively updated based on the difference between the predicted acoustic features and the extracted acoustic features.

5. The method according to claim 2, characterized in that, The acoustic features corresponding to the output text encoding sequence include: The decoder is based on the acoustic features at the output frame level of gated recurrent units or causal convolution.

6. The method according to claim 1, characterized in that, The audio data synthesized using the target single-person acoustic model and the text to be synthesized to generate the acoustic features of the target speaker includes: The text to be synthesized is input into the target single-person acoustic model, and the acoustic features of the target speaker are output through the target single-person acoustic model. The acoustic features of the target speaker are input into a pre-trained vocoder to generate audio data.

7. The method according to claim 1, characterized in that, The multi-person acoustic model and the single-person acoustic model include compressed neural network parameters.

8. A speech processing apparatus, characterized in that, The device includes: The first training module is used to train a multi-person acoustic model based on multi-person speech training data. The multi-person acoustic model includes an encoder, a prosody prediction network, a duration prediction network, and a decoder. The acoustic features output by the decoder include fundamental frequency features and Mel spectrum features. The prosody prediction network, duration prediction network, and timbre prediction network are independent of each other and are jointly trained end-to-end. The second training module is used to adaptively train the multi-person acoustic model based on the single-person speech training data of the target speaker to obtain the single-person acoustic model of the target speaker. A fixed-point processing module is used to perform parameter fixed-point processing on the single-person acoustic model to obtain the target single-person acoustic model; The speech synthesis module is used to synthesize audio data of the acoustic features of the target speaker using the target single-person acoustic model and the text to be synthesized; The first training module includes: a prosody prediction submodule, used to input the text encoding sequence into the prosody prediction network of the multi-person acoustic model, predict the prosody features of each character / phoneme in the text encoding sequence, and output the prosody feature sequence corresponding to the text encoding sequence; The acoustic feature prediction submodule is used to input the duration sequence and the prosodic feature sequence into the decoder of the multi-person acoustic model and output the predicted acoustic features corresponding to the text encoding sequence. The parameter update submodule is used to iteratively update the model parameters of the multi-person acoustic model based on the difference between the predicted acoustic features and the extracted acoustic features.

9. The apparatus according to claim 8, characterized in that, The multi-person speech training data includes training data for each preset speaker. The training data for each preset speaker includes the speech data and corresponding text of the preset speaker, as well as acoustic features, text features, and duration information extracted based on the speech data and corresponding text. The first training module further includes: The initialization submodule is used to initialize the model parameters of the multi-person acoustic model; The encoding submodule is used to input the text features into the encoder of the multi-person acoustic model to obtain the text encoding sequence corresponding to the text features; The duration prediction submodule is used to input the text encoding sequence and the duration information into the duration prediction network of the multi-person acoustic model, predict the duration of each character / phoneme in the text encoding sequence, and output the duration sequence corresponding to the text encoding sequence.

10. The apparatus according to claim 9, characterized in that, The parameter update submodule includes: The data input unit is used to input the predicted acoustic features and the extracted acoustic features into the discriminator of the generative adversarial network; An error calculation unit is used to calculate the model error of the multi-person acoustic model based on the discrimination result output by the discriminator. The parameter update unit is used to iteratively update the parameters of the multi-person acoustic model and the parameters of the discriminator through the model error.

11. The apparatus according to claim 9, characterized in that, The parameter update submodule is specifically used to iteratively update the parameters of the encoder, and / or prosody prediction network, and / or duration prediction network, and / or decoder based on the difference between the predicted acoustic features and the extracted acoustic features.

12. The apparatus according to claim 9, characterized in that, The acoustic feature prediction submodule is specifically used to output frame-level acoustic features through the decoder based on gated recurrent units or causal convolution.

13. The apparatus according to claim 8, characterized in that, The speech synthesis module includes: The feature prediction submodule is used to input the text to be synthesized into the target single-person acoustic model and output the acoustic features of the target speaker through the target single-person acoustic model; The audio conversion submodule is used to input the acoustic features of the target speaker into a pre-trained vocoder to generate audio data.

14. An apparatus for speech synthesis, characterized in that, The apparatus includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs containing instructions for performing the speech synthesis method according to any one of claims 1 to 7.

15. A machine-readable medium having instructions stored thereon that, when executed by one or more processors of a device, cause the device to perform the speech synthesis method as described in any one of claims 1 to 7.

Images