Speech synthesis method and model training method and apparatus

Abstract

The present disclosure provides a speech synthesis method and a model training method and device, relates to the technical field of computers, and particularly relates to the technical field of speech synthesis and speech transcription. The specific technical scheme comprises the following steps: obtaining a text to be processed, a speaking style identifier, and a speaker identifier; performing feature extraction based on the text to obtain text features; performing feature extraction based on the text and the speaking style identifier to obtain style features; performing feature extraction based on the speaker identifier to obtain speaker features; and obtaining synthesized audio based on the text features, the style features, and the speaker features. The technical scheme of the present disclosure can meet the demand for speech synthesis of multiple speakers and multiple styles.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the fields of speech synthesis and speech transcription technology, specifically to a speech synthesis method, model training method, and apparatus. Background Technology

[0002] Text-to-Speech (TTS) technology can meet the need to convert text into human-like speech, thus completing the human-computer interaction loop. This technology is widely applicable to business scenarios such as intelligent customer service, audio reading, news broadcasting, and human-computer interaction, improving the human-computer interaction experience and increasing the efficiency of building voice-based applications. However, current speech synthesis solutions cannot achieve multi-speaker, multi-style speech synthesis. Summary of the Invention

[0003] This disclosure provides a speech synthesis method, a model training method, and an apparatus.

[0004] According to a first aspect of this disclosure, a speech synthesis method is provided, comprising:

[0005] Retrieve the text to be processed, speech style identifiers, and speaker identifiers;

[0006] Text features are obtained by extracting features from the text.

[0007] Style features are obtained by extracting features based on text and speech style identifiers;

[0008] Speaker features are obtained by extracting features based on speaker identifiers;

[0009] Synthesized audio is obtained based on text features, style features, and speaker features.

[0010] According to a second aspect of this disclosure, a model training method is provided, comprising:

[0011] Obtain sample audio, and obtain the corresponding sample text, sample speech style identifier, and sample speaker identifier;

[0012] Based on sample audio, sample text, sample speech style identifiers, and sample speaker identifiers, the speech synthesis model is iteratively trained until the speech synthesis model converges; each training iteration includes:

[0013] Feature extraction is performed based on the sample text to obtain sample text features; feature extraction is performed based on the sample text and sample speech style identifiers to obtain sample style features; feature extraction is performed based on the sample speaker identifiers to obtain first sample speaker features; synthesized audio is obtained based on sample text features, sample style features, and first sample speaker features; the error loss between the synthesized audio and the sample audio is determined, and the parameters of each module in the speech synthesis model are updated according to the error loss.

[0014] According to a third aspect of this disclosure, a speech synthesis apparatus is provided, comprising:

[0015] The first data acquisition unit is used to acquire the text to be processed, speech style identifiers, and speaker identifiers;

[0016] The text feature extraction unit is used to extract features from text to obtain text features;

[0017] The style feature extraction unit is used to extract style features based on text and speech style identifiers.

[0018] The speaker feature extraction unit is used to extract features based on the speaker identifier to obtain speaker features;

[0019] The speech synthesis unit is used to obtain synthesized audio based on text features, style features, and speaker features.

[0020] According to a fourth aspect of this disclosure, a model training apparatus is provided, comprising:

[0021] The second data acquisition unit is used to acquire sample audio, and to acquire the sample text, sample speech style identifier and sample speaker identifier corresponding to the sample audio;

[0022] The training unit is used to iteratively train the speech synthesis model based on sample audio, sample text, sample speech style identifiers, and sample speaker identifiers until the speech synthesis model converges. Each training iteration includes:

[0023] Feature extraction is performed based on the sample text to obtain sample text features; feature extraction is performed based on the sample text and sample speech style identifiers to obtain sample style features; feature extraction is performed based on the sample speaker identifiers to obtain first sample speaker features; synthesized audio is obtained based on sample text features, sample style features, and first sample speaker features; the error loss between the synthesized audio and the sample audio is determined, and the parameters of each module in the speech synthesis model are updated according to the error loss.

[0024] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a speech synthesis method or model training method provided in any embodiment of this disclosure.

[0025] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute a speech synthesis method or model training method provided in any embodiment of this disclosure.

[0026] According to a seventh aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the speech synthesis method or model training method provided in any embodiment of this disclosure.

[0027] The technical solution disclosed herein can achieve at least the following beneficial effects:

[0028] In speech synthesis solutions, based on speech style identifiers and speaker identifiers, the style features of the desired style and the speaker features of the desired speaker timbre can be extracted in a targeted manner. Combined with text features, synthesized audio that simultaneously conforms to the text content, the desired style, and the desired speaker timbre can be obtained. This can improve the flexibility of speech synthesis and meet diverse speech synthesis needs, as well as the needs of speech synthesis with multiple speakers and styles.

[0029] In the model training scheme, the feature extraction of sample text can be guided by the sample speech style identifier and sample speaker identifier corresponding to the sample audio, and then the speech synthesis model can be trained. This can improve the training accuracy and make the audio synthesized by the trained speech synthesis model more realistic, thereby improving the user experience. Based on the sample speech style identifier and sample speaker identifier, the style features of the required style and the speaker features of the required speaker timbre can be extracted in a targeted manner. Combined with the sample text features and sample audio, the speech synthesis model can be trained, which can improve the flexibility of the trained speech synthesis model and meet the diverse speech synthesis needs and the needs of speech synthesis with multiple speakers and multiple styles.

[0030] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0031] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0032] Figure 1This is a schematic flowchart of a speech synthesis method provided in an embodiment of this disclosure;

[0033] Figure 2 This is a structural framework of a speech synthesis model and a schematic diagram of data transmission between the speech synthesis model and the speaker recognition model provided in an embodiment of this disclosure;

[0034] Figure 3 This is a partial flowchart of another speech synthesis method provided in this disclosure embodiment;

[0035] Figure 4 This is a schematic diagram of the structural framework of the first prosody predictor in this embodiment of the present disclosure;

[0036] Figure 5 This is a schematic diagram of the structural framework of the second prosody predictor in an embodiment of this disclosure;

[0037] Figure 6 This is a flowchart illustrating a model training method provided in an embodiment of this disclosure.

[0038] Figure 7 This is a partial flowchart of another model training method provided in this embodiment of the disclosure;

[0039] Figure 8 This is a schematic diagram of the structural framework of a speech synthesis device provided in an embodiment of this disclosure;

[0040] Figure 9 This is a schematic diagram of the structural framework of a model training device provided in an embodiment of this disclosure;

[0041] Figure 10 This is a schematic diagram of the structural framework of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0042] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0043] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0044] In the description of the embodiments of this disclosure, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0045] It should be further understood that the term "and / or" as used in the embodiments of this disclosure includes all or any unit and all combination of one or more associated listed items.

[0046] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used in the embodiments of this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0047] The inventors of this disclosure discovered in their research that traditional speech synthesis systems typically include two modules: a front-end and a back-end. The front-end module primarily analyzes the input text and extracts the linguistic information needed by the back-end module; the back-end module generates speech waveforms based on the front-end analysis results using certain methods. The speech synthesis methods of the back-end module generally fall into three categories:

[0048] Statistical parameter modeling-based speech synthesis (parameter synthesis for short): During the training phase, context-dependent modeling of language acoustic features and duration information is performed. During the synthesis phase, acoustic feature parameters are predicted through duration and acoustic models. The acoustic feature parameters are then post-processed, and finally, the speech waveform is recovered through a vocoder.

[0049] Speech synthesis based on unit selection and waveform splicing (referred to as splicing synthesis): The training phase is basically the same as parameter synthesis. In the synthesis phase, the unit selection is guided by the calculation cost of the model. The optimal unit sequence is selected by dynamic programming algorithm, and then the selected units are subjected to energy normalization and waveform splicing.

[0050] End-to-end speech synthesis based on deep learning (referred to as end-to-end synthesis): Taking the Tacoron model (an end-to-end speech synthesis model) as an example, this synthesis method directly inputs text or speech, models it directly through text or speech, and skips the vocoder stage.

[0051] The above synthesis methods can achieve multi-speaker, multi-style speech synthesis to a certain extent, but they have high requirements for training data. During the training data collection phase, audio recordings need to be made for multiple speakers, and multiple styles of audio recordings are required for the same speaker. This places high demands on recording costs (time and money) and the recording environment. In real-world scenarios, it is difficult to record enough audio resources to meet these requirements. When audio resources are limited, none of the above solutions can achieve multi-speaker, multi-style speech synthesis.

[0052] The technical solutions of this disclosure and how they solve the above-mentioned technical problems will be described in detail below with specific embodiments.

[0053] According to embodiments of this disclosure, this disclosure provides a speech synthesis method, such as... Figure 1 As shown, the method includes:

[0054] S101, Obtain the text to be processed, speech style identifier, and speaker identifier;

[0055] S102, extract features based on text to obtain text features;

[0056] S103, feature extraction is performed based on text and speech style identifiers to obtain style features;

[0057] S104, Feature extraction is performed based on the speaker identifier to obtain speaker features;

[0058] S105, synthesized audio is obtained based on text features, style features and speaker features.

[0059] The speech synthesis method disclosed herein, based on speech style identifiers and speaker identifiers, can selectively extract style features of the desired style and speaker features of the desired speaker timbre. Combined with text features, it can obtain synthesized audio that simultaneously conforms to the text content, the desired style, and the desired speaker timbre, thereby improving the flexibility of speech synthesis to meet diverse speech synthesis needs and the needs of speech synthesis with multiple speakers and styles.

[0060] Optionally, in step S101, obtaining the text to be processed may include: obtaining the initial text to be processed; segmenting the initial text according to at least one of two granularities, words and phonemes, to obtain the segmented text as the text to be processed.

[0061] This method can achieve segmentation at a specified granularity or multiple granularities, meeting the user's requirements for granularity. When performing segmentation at multiple granularities, a fine granularity (such as the granularity of phonemes) is conducive to extracting more refined granularity features, and a coarse granularity (such as the granularity of words) can make the content more intuitive and the style extraction easier. In addition, based on the segmentation scheme of at least one granularity, it is also possible to improve the accuracy of alignment between the subsequently extracted style features, prosody features, etc. and the pronunciation units (phonemes or words), which helps to synthesize more realistic audio.

[0062] When segmenting the initial text according to the above at least one granularity, a pre-trained segmentation model can be used for segmentation, and subsequent feature extraction can be performed on the segmented segments. In one example, assume the initial text is "百度熊". When segmenting this text according to the two granularities of words and phonemes, the segmented text is "bai3du4百度|,xiong2|熊". Based on the segmentation method of two granularities, the limitations of the single-granularity segmentation method can be compensated.

[0063] Among them, the segmentation model can be any one of models such as GMM-HMM (Gaussian Mixed Model-Hidden Markov Model), LSTM-CTC (Long Short-Term Memory-Connectionist Temporal Classifier), RNN-T (Recurrent Neural Network Transducer), Chain, etc. The segmentation model can be pre-trained based on open-source data such as aishell (open-source data developed by Hill Shell Company), librispeech (a large corpus containing approximately 1000 hours of English speech), etc.

[0064] The text content, speech style identifier, and speaker identifier can be obtained according to actual needs. The speech style identifier can be any specified style ID (Identity Document), and the speaker identifier can be any specified speaker ID.

[0065] Optionally, in steps S102 to S105, both the extraction of features and the synthesis of audio can be achieved through a speech synthesis model. In one example, such as Figure 2As shown, the speech synthesis model includes: a text feature extraction module 201, a style feature extraction module 202, a prosody prediction module 203, a speaker feature extraction module 204, and a spectrogram decoding module 205. Specifically, the text feature extraction module 201 extracts features from the text to obtain text features; the style feature extraction module 202 extracts features from the text and speech style identifiers to obtain style features; the speaker feature extraction module 204 extracts features from the speaker identifiers to obtain speaker features; and the prosody prediction module 203 and the spectrogram decoding module 205 generate synthesized audio based on the text features, style features, and the speaker features. Their specific implementation will be described later.

[0066] It should be noted that, Figure 2 The branch containing the prosodic features of the first sample and the calculation of each error loss are only used when training the speech synthesis model. The specific training scheme will be introduced later.

[0067] Optionally, the text feature extraction module 201 may include a text encoder, which may include at least one encoding structure. Each encoding structure may include a CNN (Convolutional Neural Network), a Multi-Head-Attention network, and a Linear unit connected in sequence. The specific number of encoding structures in the text encoder can be set according to actual needs, for example, it can be set to 3. In one example, the structure of the text encoder can be represented as {{CNN+LSTM+Multi-Head-Attention+Linear}*3. The text encoder can also be other structures, which are not limited in this disclosure.

[0068] A text encoder can map input text containing words and phonemes into a high-dimensional space, representing the pronunciation information represented by phonemes, the content of words, and the semantic information related to the context. This achieves a combination of coarse and fine granularity, which is more conducive to alignment with style features, such as where to pause on a word or where to stress a pronunciation.

[0069] Optionally, the style feature extraction module 202 may include a style classifier (or style encoder) formed by a multi-layer neural network. The multi-layer neural network may include TDNN (Time-Delay Neural Network), LSTM, ResNet (Residual Network), etc. The style classifier can generate style embedding vectors as style features S11 based on the input text and speech style identifiers. The style embedding vector can be composed of multiple feature vectors. The style classifier can be obtained through random initialization or through overall training of the speech synthesis model.

[0070] Optionally, the speaker feature extraction module 204 may include a speaker encoder, which can select the corresponding feature vector as speaker features based on the speaker identifier. The speaker encoder can be obtained through random initialization or through overall training of the speech synthesis model.

[0071] Optional, such as Figure 3 As shown, in step S105, synthesized audio is obtained based on text features, style features, and speaker features, including the following steps S301-S302:

[0072] S301, perform at least one prosodic prediction based on text features and style features to obtain prosodic features;

[0073] S302 decodes and synthesizes based on text features, prosodic features, and speaker features to obtain synthesized audio.

[0074] In this approach, at least one prosodic prediction based on text features and style features can fully extract prosodic information and obtain relatively accurate prosodic features. Then, by combining text features and speaker features, synthesized audio that conforms to a specified style and a specified speaker can be obtained, which can meet different speech synthesis needs.

[0075] In one optional implementation, in step S301, prosodic prediction is performed at least once based on text features and style features to obtain prosodic features, including: performing a first prosodic prediction based on text features and style features to obtain initial prosodic features; and performing a second prosodic prediction based on the initial prosodic features for different frame ranges to obtain prosodic features.

[0076] Performing a second prosodic prediction based on the first prosodic prediction can improve the accuracy of prosodic prediction and obtain more accurate local features. The second prosodic prediction is based on different frame ranges, which can further obtain wide features containing contextual information, and can extract more accurate and richer feature information for subsequent processing.

[0077] Optionally, first prosodic prediction can be performed based on text and style features, including: adding positional encoding to the text and style features, and then performing first prosodic prediction based on the text and style features with added positional encoding. Prosodicity is time-related and constitutes a sequence; prosodic features differ at different times. Adding positional encoding to the text and style features can enhance the richness of prosodic features.

[0078] Reference Figure 2 For example, the prosody prediction module 203 in the speech synthesis model may include a first prosody predictor and a second prosody predictor.

[0079] Reference Figure 4 For example, the first prosody predictor may include a 1D convolutional layer (Conv1d), layer normalization units (Layer Norm), activation layers (ReLU), dropout layers, and a fully connected layer (Dense). There can be multiple sets of 1D convolutional layers, layer normalization units, activation layers, and dropout layers; their specific number can be set according to actual needs. Figure 4 The example shows 3 groups. The first prosody predictor can perform first prosody prediction based on the input text features and style features to obtain the initial prosody feature. Before inputting the text features and style features into the first prosody predictor, positional encoding can be added to the text features and style features.

[0080] Reference Figure 5 For example, the second prosody predictor (also known as a prosody encoder) can include a 1D convolutional structure with multiple filter stacks (Conv1d bank + stacking), a max pooling layer, a 1D convolutional layer, and a BLSTM (Bidirectional Long Short-Term Memory) network. The multiple filters have different frame number ranges, and the specific number of filters can be set according to actual needs, for example, 8. Based on multiple filters with different frame number ranges, the second prosody predictor can extract features from the initial input prosody features within different frame number ranges. These features are then concatenated and combined to obtain the prosody representation as the prosody feature, thus achieving the second prosody prediction.

[0081] In another optional implementation, in step S301, prosodic prediction is performed at least once based on text features and style features to obtain prosodic features, including: performing prosodic prediction based on text features and style features to obtain prosodic features. In one example, the prosodic prediction module 203 in the speech synthesis model may include a first prosodic predictor, which is used to perform prosodic prediction based on text features and style features. Subsequent prosodic prediction for different frame ranges is not required. Before inputting the text features and style features into the first prosodic predictor, positional encoding can be added to the text features and style features.

[0082] In another alternative implementation, prosodic prediction is performed at least once based on text features and style features to obtain prosodic features, including: prosodic prediction across different frame ranges based on text features and style features to obtain prosodic features. In one example, the prosodic prediction module 203 in the speech synthesis model may include a second prosodic predictor used to perform prosodic prediction across different frame ranges based on text features and style features, without prior prediction of preliminary prosodic features.

[0083] Optionally, the initial prosodic features and prosodic features in the embodiments of this disclosure may include at least one type of information such as main frequency (pitch), loudness, and duration.

[0084] In an optional implementation, in step S302, decoding and synthesis are performed based on text features, prosodic features, and speaker features to obtain synthesized audio, including:

[0085] The text features and prosodic features are processed based on the attention mechanism to obtain the processed text features and prosodic features; the processed text features, prosodic features and speaker features are decoded and synthesized to obtain the spectrogram features; the spectrogram features are then converted into synthesized audio.

[0086] Reference Figure 2 For example, the spectrogram decoding module 205 in the speech synthesis model may include an attention submodule, a decoder, and a vocoder. The attention submodule can process text features and prosodic features based on an attention mechanism to obtain processed text features and processed prosodic features; the decoder can decode and synthesize based on the processed text features, processed prosodic features, and speaker features to obtain spectrogram features; the vocoder can convert spectrogram features into synthesized audio.

[0087] Feature processing methods based on attention mechanisms can further extract features with important information from text features and prosodic features, thereby improving the accuracy of text features and prosodic features.

[0088] In another optional implementation, in step S302, decoding and synthesis are performed based on text features, prosodic features and speaker features to obtain synthesized audio, including: decoding and synthesis based on text features, prosodic features and speaker features to obtain spectrogram features; and converting the spectrogram features into synthesized audio.

[0089] In one example, the spectrogram decoding module 205 in the speech synthesis model may include a decoder and a vocoder. The decoder can decode and synthesize spectrogram features based on processed text features, processed prosodic features, and speaker features; the vocoder can convert the spectrogram features into synthesized audio.

[0090] The decoder in this embodiment may include PreNet (Progressive Recurrent Network), Recurrent Neural Network (RNN), CBHG (Sequence Feature Extraction Unit), and BGRU (Bidirectional Gated Recurrent Unit). PreNet can be composed of a CNN (Convolutional Neural Network) and linear units; the number of these units can be set according to actual needs, for example, it can be set to 3. The sequence feature extraction unit can be composed of a convolutional neural network, a max-pooling layer, and a residual network. In one example, the decoder structure can be represented as {PreNet{{CNN+Linear}*3}+RNN+CBHG{CNN+Maxpooling+ResNet}+BGRU}.

[0091] According to embodiments of this disclosure, this disclosure also provides a model training method that can be used to train a speech synthesis model, and the trained speech synthesis model can execute the speech synthesis requests provided in this disclosure. Figure 6 As shown, the model training method includes:

[0092] S601, Obtain sample audio, and obtain the sample text, sample speech style identifier, and sample speaker identifier corresponding to the sample audio.

[0093] S602, based on sample audio, sample text, sample speech style identifiers and sample speaker identifiers, iteratively trains the speech synthesis model until the speech synthesis model converges.

[0094] Each training iteration in the above-mentioned training includes: extracting features based on sample text to obtain sample text features; extracting features based on sample text and sample speech style identifiers to obtain sample style features; extracting features based on sample speaker identifiers to obtain first sample speaker features; obtaining synthesized audio based on sample text features, sample style features, and first sample speaker features; determining the error loss between synthesized audio and sample audio, and updating the parameters of each module in the speech synthesis model according to the error loss.

[0095] The model training method disclosed herein can guide the feature extraction of sample text based on the sample speech style identifier and sample speaker identifier corresponding to the sample audio, and then train the speech synthesis model, which can improve the training accuracy and make the audio synthesized by the trained speech synthesis model more realistic, thereby improving the user experience.

[0096] Based on sample speech style identifiers and sample speaker identifiers, the style features of the desired style and the speaker features of the desired speaker timbre can be extracted in a targeted manner. Combined with sample text features and sample audio, the speech synthesis model can be trained to improve the flexibility of the trained speech synthesis model, so as to meet the diverse speech synthesis needs and the needs of speech synthesis with multiple speakers and styles.

[0097] When using a speech synthesis model, no reference audio is required; the corresponding audio can be obtained directly from the text, which can effectively improve the efficiency of speech synthesis and reduce the dependence on audio data.

[0098] In this embodiment, sample audio can be pre-recorded and collected. For the same speaker, only one style of audio needs to be recorded and collected. The number of speakers and the duration of each speaker's audio can be determined according to actual needs. In one example, the number of speakers can be more than 100, and the recording time for the same speaker can be more than 4 hours. For multiple speakers, the recorded audio styles can be different, specifically including reading aloud, broadcasting, novels, customer service, poetry recitation, e-sports games, and other different styles, so that the trained speech synthesis model can be applied to different scenarios.

[0099] When obtaining sample text corresponding to sample audio, the sample audio can be annotated to obtain the corresponding text content. Before annotating the sample audio, it can be preprocessed, such as removing ambient noise, busy tones, ringtones, and other noise factors. After obtaining the text content, at least one of the following operations can be performed: text cleaning, symbol normalization (e.g., normalization of %, kg, 2021, 150 tons, etc.), and word-to-sound conversion.

[0100] When obtaining the sample speech style identifier and sample speaker identifier corresponding to the sample audio, the sample audio can be annotated by referring to the method of obtaining the sample text, and then the corresponding speech style identifier and speaker identifier can be obtained.

[0101] For the specific structure of the speech synthesis model, please refer to the previous related content and Figure 2 In step S602, the extraction of sample text features can be performed by the text feature extraction module 201 in the speech synthesis model, the extraction of sample style features can be performed by the style feature extraction module 202 in the speech synthesis model, and the extraction of the first sample speaker features can be performed by the speaker feature extraction module 204 in the speech synthesis model. The specific structures, types, and principles of the text feature extraction module 201, style feature extraction module 202, and speaker feature extraction module 204 can be found in the relevant content of the speech synthesis method and will not be elaborated here.

[0102] Optional, such as Figure 7 As shown, in step S602, the synthesized audio is obtained based on the sample text features, the sample style features, and the first sample speaker features, which may include the following steps S701-S702:

[0103] S701, based on at least one of the target sample features and the first sample prosodic features of the sample audio, at least one prosodic prediction is performed through the prosodic prediction module in the speech synthesis model to obtain the second sample prosodic features; the target sample features include sample text features and sample style features.

[0104] S702 uses the spectrogram decoding module in the speech synthesis model to decode and synthesize the features of the sample text, the prosodic features of the second sample, and the speaker features of the first sample to obtain synthesized audio.

[0105] In this approach, at least one prosodic prediction based on text features and style features can fully extract prosodic information and obtain relatively accurate prosodic features. The first sample prosodic features are obtained from the sample audio and have high accuracy. Prediction based on the first sample prosodic features helps to obtain more accurate prosodic features. Then, by combining text features and speaker features, synthesized audio that conforms to the specified style and speaker can be obtained, which can meet different speech synthesis needs.

[0106] Optionally, the prosodic features of the first sample can be pre-extracted as follows: the sample audio is segmented according to at least one of the two granularities of words and phonemes, and the style features of the segmented audio segments are extracted to obtain the sample style features.

[0107] The segmentation of the sample audio can be achieved using the segmentation model described above. A segmented audio segment can be a frame, and the duration of a frame can be 25 milliseconds. The MFCC (Mel-Frequency Cepstral Coefficients) features of each frame are extracted, and the extracted MFCC features are input into the segmentation model. Based on the output of the segmentation model, the recognition results at least one of the two granularities, namely word and phoneme, are obtained.

[0108] When extracting style features from segmented audio clips, the dominant frequency of the audio clip can be obtained using Discrete Fourier Transform (e.g., harmonic summation), the loudness can be calculated using short-time energy, the duration of the audio clip can be obtained using the segmentation model, and then averaged across the audio clips. Finally, the results of word and / or phoneme segments are concatenated to obtain the sample style features at the word and / or phoneme level. All of the above calculations are performed at the frame level.

[0109] In an optional implementation, when the prosody prediction module 203 includes a first prosody predictor and a second prosody predictor, in step S701, based on at least one of the target sample features and the first sample prosody features of the sample audio, at least one prosody prediction is performed by the prosody prediction module in the speech synthesis model to obtain the second sample prosody features, including:

[0110] Based on sample text features and sample style features, a first prosodic prediction is performed by the first prosodic predictor in the prosodic prediction module to obtain the predicted third sample prosodic features; based on at least one of the third sample prosodic features and the first sample prosodic features, a second prosodic prediction is performed by the second prosodic predictor in the prosodic prediction module to obtain the predicted second sample prosodic features in different frame ranges.

[0111] Performing a second prosodic prediction based on the first prosodic prediction can improve the accuracy of prosodic prediction and obtain more accurate local features. The second prosodic prediction is based on different frame ranges, which can further obtain wide features containing contextual information, and can extract more accurate and richer feature information for subsequent processing, thereby improving training accuracy.

[0112] Optionally, based on sample text features and sample style features, a first prosodic prediction is performed using the first prosodic predictor in the prosodic prediction module, including:

[0113] Positional encoding is added to the sample text features and sample style features. These positionally encoded sample text features and sample style features are then input into a first prosodic predictor for first-order prosodic prediction. Prosodicity is time-dependent and forms a sequence; prosodic features differ at different times. Adding positional encoding to the text and style features enhances the richness of the prosodic features, thereby improving training accuracy.

[0114] In another optional implementation, when the prosody prediction module 203 includes a first prosody predictor, in step S701, based on at least one of the target sample features and the first sample prosody features of the sample audio, at least one prosody prediction is performed by the prosody prediction module in the speech synthesis model to obtain the second sample prosody features. This includes: based on sample text features and sample style features, prosody prediction is performed by the first prosody predictor to obtain the second sample prosody features, without performing prosody prediction for different frame ranges. Optionally, positional encoding can be added to the sample text features and sample style features before inputting them into the first prosody predictor.

[0115] In another alternative implementation, when the prosody prediction module 203 includes a second prosody predictor, in step S701, based on at least one of the target sample features and the first sample prosody features of the sample audio, at least one prosody prediction is performed by the prosody prediction module in the speech synthesis model to obtain the second sample prosody features. This includes: based on the sample text features and the sample style features, prosody prediction is performed by the second prosody predictor for different frame ranges to obtain the second sample prosody features, without the need for preliminary prosody feature prediction.

[0116] The specific structures and principles of the first and second prosodic predictors can be found in the relevant content and figures in the speech synthesis method embodiments, and will not be repeated here.

[0117] Optionally, the prosodic features of the first sample, the prosodic features of the second sample, and the prosodic features of the third sample in the embodiments of this disclosure may each include at least one type of information such as the dominant frequency (pitch), loudness, and duration.

[0118] In an optional implementation, in step S702, when the spectrogram decoding module 205 includes an attention submodule, a decoder, and a vocoder, the spectrogram decoding module in the speech synthesis model decodes and synthesizes the sample text features, the second sample prosodic features, and the first sample speaker features to obtain synthesized audio, including:

[0119] The attention submodule in the spectrogram decoding module processes the sample text features and the second sample prosodic features to obtain the processed sample text features and the processed second sample prosodic features. The decoder in the spectrogram decoding module decodes and synthesizes the processed sample text features, the processed second sample prosodic features, and the first sample speaker features to obtain the sample spectrogram features. The vocoder in the spectrogram decoding module converts the sample spectrogram features into synthesized audio.

[0120] Feature processing methods based on attention mechanisms can further extract features with important information from text features and prosodic features, thereby improving the accuracy of text features and prosodic features and thus improving training accuracy.

[0121] In another optional implementation, in step S702, when the spectrogram decoding module 205 includes a decoder and a vocoder, the spectrogram decoding module in the speech synthesis model decodes and synthesizes the sample text features, the second sample prosodic features, and the first sample speaker features to obtain synthesized audio, including: decoding and synthesizing the sample text features, the second sample prosodic features, and the sample speaker features through the decoder to obtain spectrogram features; and converting the spectrogram features into synthesized audio through the vocoder.

[0122] The specific optional structure of the decoder can be found in the relevant content of the speech synthesis method embodiments, and will not be repeated here.

[0123] In one example of step S602, the MSE (mean squared error) loss (also known as L2 loss) between the synthesized audio and the sample audio is determined, and the data is updated in reverse according to this MSE loss and the stochastic gradient descent criterion. Figure 2 The parameters of all modules in the speech synthesis model shown.

[0124] In an optional implementation, where the prosody prediction module 203 includes a first prosody predictor and a second prosody predictor, each training iteration in step S602 may further include:

[0125] Determine the error loss between the prosodic features of the third sample and the prosodic features of the first sample, and update the parameters of the text feature extraction module 201, style feature extraction module 202 and the first prosodic predictor in the speech synthesis model based on the error loss.

[0126] In one example, the L1 loss between the third sample prosodic features and the first sample prosodic features is determined, and the parameters of the text feature extraction module 201, style feature extraction module 202, and first prosodic predictor in the speech synthesis model are updated in reverse based on the L1 loss.

[0127] In another alternative implementation, if the prosody prediction module 203 includes a first prosody predictor or a second prosody predictor, then in step S602, each training iteration in the iterative training may further include:

[0128] Determine the error loss (e.g., L1 loss) between the second sample prosodic features and the first sample prosodic features, and update the parameters of the text feature extraction module 201, style feature extraction module 202, and first prosodic predictor in the speech synthesis model based on the error loss.

[0129] The implementation of updating the parameters of the text feature extraction module 201, style feature extraction module 202, and first prosodic predictor based on the error loss between the third sample prosodic feature (or second sample prosodic feature) and the first sample prosodic feature can improve the accuracy of the third sample prosodic feature (or second sample prosodic feature) output by the text feature extraction module 201, style feature extraction module 202, and first prosodic predictor (or second prosodic predictor), making the third sample prosodic feature (or second sample prosodic feature) closer to the first sample prosodic feature extracted from real sample audio, thereby further improving the training accuracy.

[0130] In an optional implementation, each training iteration in step S602 may further include:

[0131] The speaker in the synthesized audio is identified by a pre-trained speaker recognition model to obtain the second sample speaker features; the error loss between the second sample speaker features and the first sample speaker features is determined, and the parameters of the speaker feature extraction module 204 in the speech synthesis model are updated according to the error loss.

[0132] Speaker recognition models can include CNNs, BLSTMs, attention modules, and fully connected layers (FC). Multiple CNNs and BLSTMs can be used, with the specific number determined by requirements; for example, two CNNs and three BLSTMs could be used. In one example, the speaker recognition model structure can be represented as (CNN*2 + BLSTM*2 + Attention + FC). In another example, the attention module can be replaced by a TDNN, with multiple TDNNs also used, for example, four. In this case, the speaker recognition model structure can be represented as (CNN*2 + BLSTM*2 + TDNN*4 + FC).

[0133] In one example, the L1 loss between the second sample speaker features and the first sample speaker features can be determined, and the parameters of the speaker feature extraction module 204 in the speech synthesis model can be updated in reverse based on the L1 loss.

[0134] During style transformation, the features of the first sample speaker may be unstable and drift. The implementation of updating the parameters of the speaker feature extraction module 204 based on the error loss between the features of the second sample speaker and the features of the first sample speaker can improve the stability of the first sample speaker features output by the speaker feature extraction module 204 and further improve the training accuracy.

[0135] The above steps S601 to S602, as well as the corresponding loss calculation and parameter update steps, can be executed multiple times to achieve iterative training until the speech synthesis model converges, forming a speech synthesis model with high-precision synthesis capabilities and capable of synthesizing multiple speakers and styles of speech.

[0136] It should be noted that the text to be processed, sample text, sample audio, and other information in this embodiment are from a public dataset.

[0137] According to embodiments of this disclosure, this disclosure also provides a speech synthesis apparatus, such as... Figure 8 As shown, the device includes: a first data acquisition unit 801, a text feature extraction unit 802, a style feature extraction unit 803, a speaker feature extraction unit 804, and a speech synthesis unit 805.

[0138] The first data acquisition unit 801 is used to acquire the text to be processed, the speech style identifier, and the speaker identifier; the text feature extraction unit 802 is used to extract features based on the text to obtain text features; the style feature extraction unit 803 is used to extract features based on the text and the speech style identifier to obtain style features; the speaker feature extraction unit 804 is used to extract features based on the speaker identifier to obtain speaker features; and the speech synthesis unit 805 is used to obtain synthesized audio based on the text features, style features, and speaker features.

[0139] Optionally, the first data acquisition unit 801 is specifically used for: acquiring the initial text to be processed; segmenting the initial text according to at least one of the two granularities of words and phonemes to obtain the segmented text, which is used as the text to be processed.

[0140] Optionally, the speech synthesis unit 805 includes a prosody prediction subunit and a speech synthesis subunit.

[0141] The prosody prediction subunit is used to perform at least one prosody prediction based on text features and style features to obtain prosody features; the speech synthesis subunit is used to decode and synthesize based on text features, prosody features and speaker features to obtain synthesized audio.

[0142] Optionally, the prosody prediction subunit is specifically used for: performing a first prosody prediction based on text features and style features to obtain initial prosody features; and performing a second prosody prediction based on the initial prosody features for different frame ranges to obtain prosody features.

[0143] Optionally, the prosody prediction subunit is specifically used to: add positional encoding to text features and style features, and perform a first prosody prediction based on the text features with added positional encoding and the style features with added positional encoding.

[0144] Optionally, the speech synthesis subunit is specifically used for: processing text features and prosodic features based on an attention mechanism to obtain processed text features and processed prosodic features; decoding and synthesizing based on the processed text features, processed prosodic features and speaker features to obtain spectrogram features; and converting the spectrogram features into synthesized audio.

[0145] The functions of the units and subunits in the speech synthesis apparatus provided in this disclosure can be referred to the corresponding descriptions in the above-described speech synthesis method embodiments, and will not be repeated here.

[0146] According to embodiments of this disclosure, this disclosure also provides a model training apparatus, such as... Figure 9 As shown, the training device includes a second data acquisition unit 901 and a training unit 902.

[0147] The second data acquisition unit 901 is used to acquire sample audio and the corresponding sample text, sample speech style identifier, and sample speaker identifier; the training unit 902 is used to iteratively train the speech synthesis model based on the sample audio, sample text, sample speech style identifier, and sample speaker identifier until the speech synthesis model converges.

[0148] Each training iteration in the above iterative training includes:

[0149] Feature extraction is performed based on the sample text to obtain sample text features; feature extraction is performed based on the sample text and sample speech style identifiers to obtain sample style features; feature extraction is performed based on the sample speaker identifiers to obtain first sample speaker features; synthesized audio is obtained based on sample text features, sample style features, and first sample speaker features; the error loss between the synthesized audio and the sample audio is determined, and the parameters of each module in the speech synthesis model are updated according to the error loss.

[0150] Optionally, when obtaining synthesized audio based on sample text features, sample style features, and first sample speaker features, training unit 902 is specifically used to: perform at least one prosodic prediction using the prosodic prediction module in the speech synthesis model based on at least one of the target sample features and the first sample prosodic features of the sample audio to obtain the second sample prosodic features; the target sample features include sample text features and sample style features; and decode and synthesize the sample text features, the second sample prosodic features, and the first sample speaker features using the spectrogram decoding module in the speech synthesis model to obtain the synthesized audio.

[0151] Optionally, when performing at least one prosodic prediction, the training unit 902 is specifically used to: perform a first prosodic prediction based on sample text features and sample style features using a first prosodic predictor in the prosodic prediction module to obtain the predicted third sample prosodic features; and perform a second prosodic prediction for different frame ranges using a second prosodic predictor in the prosodic prediction module based on at least one of the third sample prosodic features and the first sample prosodic features to obtain the predicted second sample prosodic features.

[0152] Optionally, the training unit 902 is also used to: determine the error loss between the third sample prosodic features and the first sample prosodic features, and update the parameters of the text feature extraction module, the style feature extraction module, and the first prosodic predictor in the speech synthesis model based on the error loss.

[0153] Optionally, when performing the first prosody prediction, the training unit 902 is specifically used to: add positional encoding to the sample text features and sample style features, input the sample text features with added positional encoding and the sample style features with added positional encoding into the first prosody predictor, and perform the first prosody prediction through the first prosody predictor.

[0154] Optionally, when obtaining the synthesized audio, the training unit 902 is specifically used to: process the sample text features and the second sample prosodic features through the attention submodule in the spectrogram decoding module to obtain the processed sample text features and the processed second sample prosodic features; decode and synthesize the processed sample text features, the processed second sample prosodic features, and the first sample speaker features through the decoder in the spectrogram decoding module to obtain the sample spectrogram features; and convert the sample spectrogram features into synthesized audio through the vocoder in the spectrogram decoding module.

[0155] Optionally, the training unit 902 is further configured to: identify the speaker in the synthesized audio using a pre-trained speaker recognition model to obtain the second sample speaker features; determine the error loss between the second sample speaker features and the first sample speaker features; and update the parameters of the speaker feature extraction module in the speech synthesis model based on the error loss.

[0156] The functions of the units in the model training apparatus provided in this disclosure can be referred to the corresponding descriptions in the embodiments of the above model training method, and will not be repeated here.

[0157] According to embodiments of this disclosure, this disclosure also provides an electronic device, a non-transitory computer-readable storage medium, and a computer program product.

[0158] The electronic device provided in this disclosure includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the speech synthesis method or model training method provided in any embodiment of this disclosure.

[0159] The non-transitory computer-readable storage medium provided in this disclosure stores computer instructions for causing a computer to execute the speech synthesis method or model training method provided in any embodiment of this disclosure.

[0160] The computer program product provided in this disclosure includes a computer program that, when executed by a processor, implements the speech synthesis method or model training method provided in any embodiment of this disclosure.

[0161] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0162] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0163] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0164] The computing unit 1001 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above. For example, in some embodiments, the above methods can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 1001 can be configured to perform the above methods by any other suitable means (e.g., by means of firmware).

[0165] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0166] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0167] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0168] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0169] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0170] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0171] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0172] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A speech synthesis method, comprising: obtaining a text to be processed, a speaking style identifier, and a speaker identifier; performing feature extraction based on the text to obtain text features; performing feature extraction based on the text and the speaking style identifier to obtain style features; performing feature extraction based on the speaker identifier to obtain speaker features; obtaining synthesized audio based on the text features, the style features, and the speaker features; wherein the obtaining of the text to be processed comprises: obtaining an initial text to be processed; segmenting the initial text according to at least one of word granularity and phoneme granularity to obtain a segmented text as the text to be processed; wherein the obtaining of synthesized audio based on the text features, the style features, and the speaker features comprises: adding position encoding to the text features and the style features; inputting the text features with added position encoding and the style features with added position encoding into a first prosody predictor to perform first prosody prediction to obtain initial prosody features; wherein the first prosody predictor comprises a one-dimensional convolution layer, a layer regularization unit, an activation layer, a random dropout layer, and a fully connected layer; inputting the initial prosody features into a second prosody predictor to perform second prosody prediction to obtain the prosody features; wherein the second prosody prediction comprises feature extraction of the initial prosody features by the second prosody predictor in different frame number ranges, and feature splicing and combination of the feature extraction results to obtain the prosody features; wherein the second prosody predictor comprises a one-dimensional convolution structure stacked by a plurality of filters, a max-pooling layer, a one-dimensional convolution layer, and a bidirectional long short-term memory network; wherein the plurality of filters are used to extract features of the initial prosody features in different frame number ranges; performing decoding and synthesis based on the text features, the prosody features, and the speaker features to obtain synthesized audio.

2. The speech synthesis method of claim 1, wherein, The performing of decoding and synthesis based on the text features, the prosody features, and the speaker features to obtain synthesized audio comprises: processing the text features and the prosody features based on an attention mechanism to obtain processed text features and processed prosody features; performing decoding and synthesis based on the processed text features, the processed prosody features, and the speaker features to obtain spectrogram features; converting the spectrogram features into synthesized audio.

3. A model training method, comprising: obtaining a sample audio, and obtaining a sample text corresponding to the sample audio, a sample speaking style identifier, and a sample speaker identifier; iteratively training a speech synthesis model based on the sample audio, the sample text, the sample speaking style identifier, and the sample speaker identifier until the speech synthesis model converges; each training in the iterative training comprises: perform feature extraction based on the sample text to obtain a sample text feature, perform feature extraction based on the sample text and the sample speaking style label to obtain a sample style feature, and perform feature extraction based on the sample speaker label to obtain a first sample speaker feature; and obtain a synthesized audio based on the sample text feature, the sample style feature, and the first sample speaker feature; determine an error loss between the synthesized audio and the sample audio, and update parameters of modules in the speech synthesis model according to the error loss; The obtaining of the synthesized audio based on the sample text feature, the sample style feature, and the first sample speaker feature includes: adding position encoding to the sample text feature and the sample style feature; inputting the sample text feature and the sample style feature with the added position encoding into a first prosody predictor to perform first prosody prediction, to obtain a predicted third sample prosody feature; the first prosody predictor includes a one-dimensional convolution layer, a layer regularization unit, an activation layer, a random dropout layer, and a full connection layer; inputting at least one of the third sample prosody feature and a first sample prosody feature of the sample audio into a second prosody predictor to perform second prosody prediction, to obtain a second sample prosody feature; the second prosody prediction includes extracting, by the second prosody predictor, at least one of the third sample prosody feature and the first sample prosody feature of the sample audio in different frame number ranges, and obtaining the second sample prosody feature by combining the extracted results; the second prosody predictor includes a one-dimensional convolution structure stacked by multiple filters, a maximum pooling layer, a one-dimensional convolution layer, and a bidirectional long short-term memory network; the multiple filters are used to extract at least one of the third sample prosody feature and the first sample prosody feature of the sample audio in different frame number ranges; decoding and synthesizing, by a spectrum decoding module in the speech synthesis model, the sample text feature, the second sample prosody feature, and the first sample speaker feature, to obtain the synthesized audio. 4.The model training method of claim 3, wherein, Each iteration in the iterative training further includes: determining an error loss between the third sample prosody feature and the first sample prosody feature, and updating parameters of a text feature extraction module, a style feature extraction module, and the first prosody predictor in the speech synthesis model according to the error loss.

5. The model training method according to any one of claims 3-4, wherein the decoding and synthesizing, by a spectrum decoding module in the speech synthesis model, the sample text feature, the second sample prosody feature, and the first sample speaker feature, to obtain the synthesized audio includes: processing, by an attention sub-module in the spectrum decoding module, the sample text feature and the second sample prosody feature to obtain processed sample text feature and processed second sample prosody feature; decoding and synthesizing the processed sample text feature, the processed second sample prosody feature and the first sample speaker feature through a decoder in the acoustic spectrum decoding module to obtain a sample acoustic spectrum feature; converting the sample acoustic spectrum feature into synthesized audio through a vocoder in the acoustic spectrum decoding module.

6. The model training method of any one of claims 3-4, wherein, Each training in the iterative training further comprises: identifying a speaker in the synthesized audio through the pre-trained speaker recognition model to obtain a second sample speaker feature; determining an error loss between the second sample speaker feature and the first sample speaker feature, and updating parameters of a speaker feature extraction module in the speech synthesis model according to the error loss.

7. A speech synthesis device, comprising: a first data acquisition unit configured to acquire a text to be processed, a speaking style identifier and a speaker identifier; a text feature extraction unit configured to perform feature extraction based on the text to obtain a text feature; a style feature extraction unit configured to perform feature extraction based on the text and the speaking style identifier to obtain a style feature; a speaker feature extraction unit configured to perform feature extraction based on the speaker identifier to obtain a speaker feature; a speech synthesis unit configured to obtain synthesized audio based on the text feature, the style feature and the speaker feature; wherein the first data acquisition unit is specifically configured to: acquire an initial text to be processed; perform segmentation on the initial text according to at least one of word and phoneme granularities to obtain segmented text as the text to be processed; wherein the speech synthesis unit comprises: a prosody prediction subunit configured to add position encoding in the text feature and the style feature; input the text feature with added position encoding and the style feature with added position encoding into a first prosody predictor to perform first prosody prediction to obtain an initial prosody feature; wherein the first prosody predictor comprises a one-dimensional convolution layer, a layer regularization unit, an activation layer, a random dropout layer and a full connection layer; input the initial prosody feature into a second prosody predictor to perform second prosody prediction to obtain the prosody feature; wherein the second prosody prediction comprises feature extraction of the initial prosody feature by the second prosody predictor in different frame number ranges, and feature splicing and combination of the feature extraction results to obtain the prosody feature; wherein the second prosody predictor comprises a one-dimensional convolution structure stacked by a plurality of filters, a maximum pooling layer, a one-dimensional convolution layer and a bidirectional long short-term memory network; wherein the plurality of filters are used to extract features of the initial prosody feature in different frame number ranges; a speech synthesis subunit configured to perform decoding and synthesis based on the text feature, the prosody feature and the speaker feature to obtain synthesized audio.

8. The speech synthesis apparatus according to claim 7, wherein The speech synthesis subunit is specifically configured to: process the text feature and the prosody feature based on an attention mechanism to obtain a processed text feature and a processed prosody feature; decode and synthesize based on the processed text feature, the processed prosody feature and the speaker feature to obtain an acoustic spectrum feature; convert the acoustic spectrum feature into synthesized audio.

9. A model training apparatus, comprising: a second data acquisition unit configured to acquire a sample audio and acquire a sample text, a sample speaking style identifier and a sample speaker identifier corresponding to the sample audio; a training unit configured to iteratively train a speech synthesis model based on the sample audio, the sample text, the sample speaking style identifier and the sample speaker identifier until the speech synthesis model converges; each iteration of the iterative training comprises: performing feature extraction based on the sample text to obtain a sample text feature, performing feature extraction based on the sample text and the sample speaking style identifier to obtain a sample style feature, performing feature extraction based on the sample speaker identifier to obtain a first sample speaker feature, obtaining a synthesized audio based on the sample text feature, the sample style feature and the first sample speaker feature, determining an error loss between the synthesized audio and the sample audio, and updating parameters of modules in the speech synthesis model according to the error loss; the training unit is specifically configured to: add position encoding to the sample text feature and the sample style feature; input the sample text feature with added position encoding and the sample style feature with added position encoding into a first prosody predictor to perform first prosody prediction and obtain a predicted third sample prosody feature; wherein the first prosody predictor comprises a one-dimensional convolution layer, a layer regularization unit, an activation layer, a random dropout layer and a fully connected layer; input at least one of the third sample prosody feature and a first sample prosody feature of the sample audio into a second prosody predictor to perform second prosody prediction and obtain a second sample prosody feature; wherein the second prosody prediction comprises extracting at least one of the third sample prosody feature and the first sample prosody feature of the sample audio by the second prosody predictor in different frame number ranges, and obtaining the second sample prosody feature by combining the extracted results; wherein the second prosody predictor comprises a one-dimensional convolution structure stacked by a plurality of filters, a maximum pooling layer, a one-dimensional convolution layer and a bidirectional long short-term memory network; wherein the plurality of filters are used to extract at least one of the third sample prosody feature and the first sample prosody feature of the sample audio in different frame number ranges; decode and synthesize the sample text feature, the second sample prosody feature and the first sample speaker feature by a spectrum decoding module in the speech synthesis model to obtain a synthesized audio. 10.The model training apparatus according to claim 9, wherein the training unit is further configured to: determine an error loss between the third sample prosody feature and the first sample prosody feature, and update parameters of a text feature extraction module, a style feature extraction module and the first prosody predictor in the speech synthesis model according to the error loss. 11.The model training apparatus according to any one of claims 9-10, wherein the training unit is specifically configured to: process the sample text feature and the second sample prosody feature by an attention sub-module in the spectrum decoding module to obtain a processed sample text feature and a processed second sample prosody feature; decoding and synthesizing the processed sample text feature, the processed second sample prosody feature and the first sample speaker feature through a decoder in the acoustic spectrum decoding module to obtain a sample acoustic spectrum feature; converting the sample acoustic spectrum feature into synthesized audio through a vocoder in the acoustic spectrum decoding module. 12.The model training apparatus according to any one of claims 9-10, wherein The training unit is further configured to: identify a speaker in the synthesized audio through the pre-trained speaker recognition model to obtain a second sample speaker feature; determine an error loss between the second sample speaker feature and the first sample speaker feature, and update parameters of a speaker feature extraction module in the speech synthesis model according to the error loss.

13. An electronic device comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the speech synthesis method of any one of claims 1-2, or the model training method of any one of claims 3-6.

14. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the speech synthesis method of any one of claims 1-2, or the model training method of any one of claims 3-6.

15. A computer program product comprising a computer program which, when executed by a processor, implements the speech synthesis method of any one of claims 1-2, or the model training method of any one of claims 3-6.

Images