Speech synthesis model training method, speech synthesis method and related device

Abstract

The invention discloses a speech synthesis model training method, a speech synthesis method and a related device, and the method comprises the steps: inputting the prosodic features of phonemes in a phoneme sequence sample into a speech synthesis model for feature operation, and obtaining the reference prosodic features of the phoneme sequence sample; inputting the text features of the phoneme sequence samples into a speech synthesis model for feature extraction to obtain predicted rhythm features of the phoneme sequence samples; a loss value is determined based on the reference rhythm feature and the predicted rhythm feature, and a speech synthesis model is trained based on the loss value. Therefore, the speech synthesis model is trained in combination with the prosodic features of the phonemes in the phoneme sequence sample and the text features of the phoneme sequence sample, and then the speech synthesis model is used for synthesizing the speech, so that the naturalness and anthropomorphic degree of speech synthesis are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method for a speech synthesis model, a speech synthesis method and related devices. Background Art

[0002] Text-to-speech (TTS) technology can make machines speak. The non-autoregressive speech synthesis TTS model obtains synthesized speech by learning speech features. When synthesizing speech, the rhythmic features of speech, such as intonation, sound intensity, and pitch, are very important. Based on this, when synthesizing speech, how to improve the naturalness and anthropomorphism of speech synthesis based on the rhythmic features of speech has become an urgent problem to be solved. Summary of the invention

[0003] The embodiments of the present application provide a speech synthesis model training method, a speech synthesis method and related devices to improve the naturalness and anthropomorphism of speech synthesis.

[0004] In a first aspect, an embodiment of the present application provides a method for training a speech synthesis model, the method comprising:

[0005] Inputting the prosodic features of the phonemes in the phoneme sequence sample into the speech synthesis model for feature calculation to obtain reference prosodic features of the phoneme sequence sample;

[0006] Inputting the text features of the phoneme sequence sample into the speech synthesis model to extract features, thereby obtaining predicted prosodic features of the phoneme sequence sample;

[0007] A loss value is determined based on the reference prosodic feature and the predicted prosodic feature, and the speech synthesis model is trained based on the loss value.

[0008] In an embodiment of the present application, when training a speech synthesis model, the speech synthesis model is trained in combination with the prosodic features of the phonemes in the phoneme sequence samples and the text features of the phoneme sequence samples. Not only the predicted prosodic features obtained by feature extraction based on the text features of the phoneme sequence samples are used, but also the reference prosodic features obtained by feature operation on the prosodic features of the phonemes in the phoneme sequence samples are used. This allows the trained speech synthesis model to focus not only on the text features of the phoneme sequence samples and the prosodic features of the phonemes, but also on the reference prosodic features of the phoneme sequence samples. When speech synthesis is performed using a speech synthesis model trained based on this method, the resulting synthesized speech is more natural and has a better sense of rhythm, thereby improving the naturalness and degree of anthropomorphism of speech synthesis.

[0009] In a second aspect, an embodiment of the present application provides a speech synthesis method, the method comprising:

[0010] Extracting prosodic features from text features of a to-be-synthesized phoneme sequence through a speech synthesis model to obtain target prosodic features of the to-be-synthesized phoneme sequence; the speech synthesis model is trained according to the training method for the speech synthesis model described in the first aspect above;

[0011] Based on the text features of the to-be-synthesized phoneme sequence and the target prosodic features, speech synthesis is performed on the to-be-synthesized phoneme sequence.

[0012] In an embodiment of the present application, a speech synthesis model trained by the speech synthesis model training method described in the first aspect above is used for speech synthesis. Because the speech synthesis model not only uses the predicted prosodic features obtained by feature extraction based on the text features of the phoneme sequence samples during training, but also uses the reference prosodic features obtained by feature calculation on the prosodic features of the phonemes in the phoneme sequence samples. Therefore, the synthesized speech obtained after speech synthesis based on the speech synthesis model is more natural and has a better sense of rhythm, thereby improving the naturalness and degree of anthropomorphism of the speech synthesis.

[0013] In a third aspect, an embodiment of the present application provides a training device for a speech synthesis model, the device comprising:

[0014] A calculation unit, used for inputting the prosodic features of the phonemes in the phoneme sequence sample into the speech synthesis model to perform feature calculation to obtain reference prosodic features of the phoneme sequence sample;

[0015] A prediction unit, used for inputting the text features of the phoneme sequence sample into the speech synthesis model for feature extraction to obtain the predicted prosodic features of the phoneme sequence sample;

[0016] A training unit is used to determine a loss value based on the reference prosodic feature and the predicted prosodic feature, and train the speech synthesis model based on the loss value.

[0017] In a fourth aspect, an embodiment of the present application provides a speech synthesis device, the device comprising:

[0018] A feature extraction unit, used for extracting prosodic features from text features of the to-be-synthesized phoneme sequence to obtain target prosodic features of the to-be-synthesized phoneme sequence;

[0019] The speech synthesis unit is used to perform speech synthesis on the to-be-synthesized phoneme sequence based on the text features of the to-be-synthesized phoneme sequence and the target prosodic features.

[0020] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first and second aspects is implemented.

[0021] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the method described in the first aspect and the second aspect.

[0022] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed on a computer, enables the methods described in the first and second aspects above to be implemented.

[0023] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0025] Figure 1 A flowchart of a method for training a speech synthesis model provided in an embodiment of the present application;

[0026] Figure 2 A structural diagram of a feature processing module provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of the structure of a rhythm feature prediction module provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of the structure of a feature prediction module provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the structure of a speech synthesis model provided in an embodiment of the present application;

[0030] Figure 6 A flowchart of a speech synthesis method provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of a speech synthesis model provided in an embodiment of the present application;

[0032] Figure 8 A structural diagram of a training device for a speech synthesis model provided in an embodiment of the present application;

[0033] Fig. 9 A structural diagram of a speech synthesis device provided in an embodiment of the present application;

[0034] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0036] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0037] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0038] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0039] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0041] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0042] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, professional terms related to the embodiments of the present application are first introduced here.

[0043] FastSpeech2: A TTS model. This model directly uses real speech data as the training target to avoid information loss, while introducing more accurate duration information and other variable information in speech, such as pitch, volume, energy, etc., to improve the quality of synthesized speech. Based on FastSpeech1, it abandons the Teacher-Student knowledge distillation framework and reduces the complexity of training.

[0044] Transformer: A time series model based on the self-attention mechanism. It can effectively encode time series information in the encoder part. Its processing ability of time series information is much better than LSTM (Long Short-Term Memory) and its speed is fast. It is widely used in natural language processing, computer vision, machine translation, speech recognition and other fields.

[0045] Mel-spectrogram and MFCC (Mel-frequency cepstral coefficients): Mel-frequency cepstral coefficients are widely used features in voiceprint recognition, speech recognition, and speech synthesis. MFCC is a Hz spectrum feature calculated using the nonlinear correspondence between Mel frequency and Hz frequency, and is mainly used for voice data feature extraction.

[0046] Text-to-speech (TTS) technology can make machines speak. Non-autoregressive speech synthesis TTS models can obtain synthesized speech by learning speech features, which has attracted more and more attention from industry and academia. The most popular model is the FastSpeech2 model.

[0047] When synthesizing speech, the FastSpeech2 model needs to input the phoneme sequence of the speech, then output the corresponding Mel-spectrogram, and finally generate speech through the vocoder. When obtaining the Mel-spectrogram, the FastSpeech2 model uses the rhythmic features of each phoneme in the phonemes, such as Pitch (pitch), Energy (volume), and sound length. However, the rhythmic features of phonemes are too fine-grained and cannot accurately represent the rhythm of human speech. As a result, the synthesized speech obtained after processing the Mel-spectrogram output by the speech synthesis model sounds stiff, unlike human voice, and has a poor sense of rhythm.

[0048] Therefore, when synthesizing speech, how to improve the naturalness and anthropomorphism of speech synthesis based on the rhythmic features in the speech has become an urgent problem to be solved.

[0049] In view of this, a training method for a speech synthesis model, a speech synthesis method and a related device are proposed in the embodiments of the present application. When training the speech synthesis model, the prosodic features of the phonemes in the phoneme sequence sample are input into the speech synthesis model for feature calculation to obtain the reference prosodic features of the phoneme sequence sample; the text features of the phoneme sequence sample are input into the speech synthesis model for feature extraction to obtain the predicted prosodic features of the phoneme sequence sample; the loss value is determined based on the reference prosodic features and the predicted prosodic features, and the speech synthesis model is trained based on the loss value; then the trained speech synthesis model is used to perform speech synthesis on the phoneme sequence to be synthesized. The trained speech synthesis model not only pays attention to the text features of the phoneme sequence and the prosodic features of the phonemes, but also pays attention to the reference prosodic features of the phoneme sequence, so that the synthesized speech obtained by performing speech synthesis based on the trained speech synthesis model is more natural and has a better sense of rhythm, thereby improving the naturalness and anthropomorphism of the speech synthesis.

[0050] See also Figure 1 , a flowchart of a method for training a speech synthesis model is provided for an embodiment of the present application. The method is executed by an electronic device, and the electronic device may be a terminal or a server. Among them, the terminal may be a vehicle-mounted device, a mobile phone, a tablet, a laptop computer, a desktop computer, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a wearable device, and the like, and the server may be a physical server or a server cluster composed of multiple servers. Figure 1 The training method of the speech synthesis model can be specifically executed by a processor of an electronic device.

[0051] like Figure 1 As shown, the method comprises the following steps:

[0052] S101: Inputting the prosodic features of the phonemes in the phoneme sequence sample into a speech synthesis model for feature calculation to obtain reference prosodic features of the phoneme sequence sample.

[0053] In an embodiment of the present application, the phoneme sequence Phonemes is at least one phoneme arranged in the order of text corresponding to the speech to be synthesized. For example, the text corresponding to the speech to be synthesized is "Jeju Island, the largest island in South Korea", and the phoneme sequence corresponding to the text is "han2 guo27 zui4 da4 de57 dao6 yu67ji3zhou1 dao3$". The numbers 1 to 6 after the pinyin represent the tones, 1 represents the first tone, 2 represents the second tone, 3 represents the third tone, 4 represents the fourth tone, 5 represents the light tone, and 6 represents the changed tone. The numbers 7 to 9 represent the length of the rhythmic pause, 7 represents a shorter pause, 8 represents a centered pause, and 9 represents a longer pause.

[0054] It can be understood that a phoneme sequence sample is also a phoneme sequence. The phoneme sequence sample may correspond to one speech to be synthesized or multiple speech to be synthesized, and this application is not limited to this embodiment.

[0055] In the embodiment of the present application, each phoneme in the phoneme sequence sample has a prosodic feature, the prosodic feature includes N prosodic sub-features, and the reference prosodic feature includes N reference prosodic sub-features. Therefore, the prosodic features of the phonemes in the phoneme sequence sample are input into the speech synthesis model for feature calculation to obtain the reference prosodic features of the phoneme sequence sample, which can be specifically implemented as follows:

[0056] The N prosodic sub-features of the phonemes in the phoneme sequence sample are input into the speech synthesis model for feature calculation to obtain N reference prosodic sub-features of the phoneme sequence sample.

[0057] Furthermore, the prosody features in the embodiment of the present application include sub-features of M dimensions; the M dimensions include pitch and volume; the N prosody sub-features include N pitch sub-features and N volume sub-features; and the N reference prosody sub-features include N reference pitch sub-features and N reference volume sub-features.

[0058] Taking the pitch dimension as an example, the text corresponding to the phoneme sequence sample is "Who am I?", and the phoneme sequence sample corresponding to the text is "wo3shi47shui2a5$". The prosodic features of the pitch dimension (Phoneme-Level Pitch) of the phonemes in the phoneme sequence sample are {8, 12, 128}; among them, 8 is the amount of data processed at one time (batch size) by the speech synthesis model, that is, the phoneme sequence sample processed by the speech synthesis model at one time corresponds to 8 speech sounds to be synthesized, 12 is the length of the phoneme sequence sample "wo 3shi 47sh ui 2a 5$", that is, there are 12 phonemes in the phoneme sequence sample, and 128 are the N pitch sub-features corresponding to each phoneme, that is, 128 pitch sub-features corresponding to each phoneme, which represent the pitch feature information of the phoneme.

[0059] Similarly, taking the volume dimension as an example, the text corresponding to the phoneme sequence sample is "Who am I?", and the phoneme sequence sample corresponding to the text is "wo3shi47shui2a5$", and the prosodic features of the volume dimension (Phoneme-LevelEnergy) of the phonemes in the phoneme sequence sample are {8, 12, 128}; among them, 8 is the amount of data processed at one time (batchsize) by the speech synthesis model, that is, the phoneme sequence sample processed by the speech synthesis model at one time corresponds to 8 speech sounds to be synthesized, 12 is the length of the phoneme sequence sample "wo 3shi 47sh ui 2a 5$", that is, there are 12 phonemes in the phoneme sequence sample, and 128 are the N volume sub-features corresponding to each phoneme, that is, 128 volume sub-features corresponding to each phoneme, indicating the volume feature information of the phoneme.

[0060] In a possible implementation manner, in the embodiment of the present application, the N prosody sub-features include N pitch sub-features and N volume sub-features. Therefore, the N prosody sub-features of the phonemes in the phoneme sequence sample are input into the speech synthesis model for feature calculation to obtain N reference prosody sub-features of the phoneme sequence sample, which can be further executed as follows:

[0061] Through the pitch feature processing module in the speech synthesis model, feature operations are performed on N pitch sub-features of the phonemes in the phoneme sequence sample to obtain N reference pitch sub-features of the phoneme sequence sample;

[0062] The volume feature processing module in the speech synthesis model performs feature operations on N volume sub-features of the phonemes in the phoneme sequence sample to obtain N reference volume sub-features of the phoneme sequence sample.

[0063] like Figure 2As shown, it is a structural schematic diagram of a feature processing module provided in an embodiment of the present application. The feature processing module (word-level pooling) includes a pitch feature processing module and a volume feature processing module. The N pitch sub-features (Phoneme-Level Pitch) of the phonemes in the phoneme sequence sample are input into the pitch feature processing module for feature calculation to obtain N reference pitch sub-features (target word-level Pitch) of the phoneme sequence sample; the N volume sub-features (Phoneme-Level Energy) of the phonemes in the phoneme sequence sample are input into the volume feature processing module for feature calculation to obtain N reference volume sub-features (targetword-level Energy) of the phoneme sequence sample.

[0064] In a possible implementation, N prosodic sub-features of phonemes in a phoneme sequence sample are input into a speech synthesis model for feature calculation to obtain N reference prosodic sub-features of the phoneme sequence sample, which can be performed as follows:

[0065] Based on the phonemes in the phoneme sequence sample and the preset correspondence between the phonemes and the words, determine a plurality of words composed of the phonemes in the phoneme sequence sample; wherein each word corresponds to a plurality of phonemes;

[0066] For each word in the plurality of words, determining N reference prosodic sub-features of the word based on the N prosodic sub-features of the plurality of phonemes constituting the word;

[0067] The reference prosodic sub-features of multiple words are combined into N reference prosodic sub-features of the phoneme sequence sample.

[0068] Specific, combined Figure 2 The speech synthesis model shown can determine N reference pitch sub-features and N reference volume sub-features of the phoneme sequence sample for the pitch dimension and the volume dimension respectively:

[0069] With respect to the pitch dimension, N pitch sub-features of the phonemes in the phoneme sequence sample are input into the speech synthesis model for feature calculation to obtain N reference pitch sub-features of the phoneme sequence sample, which can be specifically implemented as follows: based on the phonemes in the phoneme sequence sample and the preset correspondence between phonemes and words, multiple words composed of the phonemes in the phoneme sequence sample are determined; wherein each word corresponds to multiple phonemes; for each of the multiple words, based on the N pitch sub-features of the multiple phonemes constituting the word, N reference pitch sub-features of the word are determined; and the reference pitch sub-features of the multiple words are combined into N reference pitch sub-features of the phoneme sequence sample.

[0070] With respect to the volume dimension, N volume sub-features of the phonemes in the phoneme sequence sample are input into the speech synthesis model for feature operation to obtain N reference volume sub-features of the phoneme sequence sample, which can be specifically performed as follows: based on the phonemes in the phoneme sequence sample and the preset correspondence between the phonemes and the words, multiple words composed of the phonemes in the phoneme sequence sample are determined; wherein each word corresponds to multiple phonemes; for each of the multiple words, based on the N volume sub-features of the multiple phonemes that constitute the word, N reference volume sub-features of the word are determined; and the reference pitch sub-features of the multiple words are combined into N reference volume sub-features of the phoneme sequence sample.

[0071] Wherein, for each of the multiple words, based on the N pitch sub-features of the multiple phonemes constituting the word, determining the N reference pitch sub-features of the word is performed as follows: averaging the feature values ​​belonging to the same pitch sub-feature among the N pitch sub-features of the multiple phonemes constituting the word to obtain the N reference pitch sub-features corresponding to the word;

[0072] For each of the multiple words, based on the N volume sub-features of the multiple phonemes constituting the word, N reference volume sub-features of the word are determined, and the execution is as follows: averaging the feature values ​​belonging to the same volume sub-feature among the N volume sub-features of the multiple phonemes constituting the word to obtain the N reference volume sub-features corresponding to the word.

[0073] The preset correspondence between phonemes and words is as follows: the pinyin corresponding to the same word in the phoneme sequence sample and the adjacent rhythms are all used as the phonemes corresponding to the word and are represented by the same numbers.

[0074] For example, the phonemes "wo 3shi47shui 2a 5$" corresponding to the phoneme sequence sample "wo3shi47 shui2a5$" are taken as the phonemes corresponding to "I", and so on. The corresponding relationship between phonemes and words Phoneme2Word is: [0,0,0,1,1,1,2,2,2,3,3,3]; among which, the length of Phoneme2Word is the same as the length of the phoneme sequence sample, and the same number represents the same word.

[0075] It can be determined that the phonemes "w", "o", and "3" correspond to the first word, the phonemes "shi", "4", and "7" correspond to the second word, and so on. The phoneme sequence "wo3shi47 shui2a5$" corresponds to four words. Then, for each word, the N reference prosodic sub-features corresponding to the word are calculated respectively.

[0076] Taking the pitch dimension as an example, for the first word, first add the feature values ​​of the first pitch sub-feature of the three phonemes "w", "o", and "3" that make up the first word, and then divide it by the number of phonemes that make up the first word, 3, to get the result a, and get the first reference pitch sub-feature of the first word; then add the feature values ​​of the second pitch sub-feature of the three phonemes "w", "o", and "3" that make up the first word, and then divide it by the number of phonemes that make up the first word, 3, to get the result b, and get the second reference pitch sub-feature of the first word. And so on, all the N pitch sub-features of the three phonemes that make up the first word are calculated, and the N reference pitch sub-features corresponding to the first word are obtained. Then, the N reference pitch sub-features of the other three words are obtained in the same way, and finally the reference pitch sub-features of multiple words corresponding to the phoneme sequence sample "wo3shi47 shui2a5$" are obtained, that is, the N reference pitch sub-features of the phoneme sequence sample "wo3shi47 shui2a5$" are obtained.

[0077] The number of features of the prosodic features of the phonemes in the phoneme sequence sample and the reference prosodic features of the phoneme sequence sample is the same. Taking the pitch dimension as an example, if the prosodic features of the pitch dimension of the phonemes in the phoneme sequence sample are {8, 12, 128}, then the reference prosodic features of the pitch dimension of the phoneme sequence sample are also {8, 12, 128}. The reference prosodic features of multiple phonemes constituting a word are the same.

[0078] It can be understood that the method for obtaining the N reference pitch sub-features of the phoneme sequence sample in the embodiment of the present application is the same as the method for obtaining the N reference volume sub-features of the phoneme sequence sample, but when obtaining the N reference pitch sub-features of the phoneme sequence sample, the N pitch sub-features of the phonemes in the phoneme sequence sample are input, and when obtaining the N reference volume sub-features of the phoneme sequence sample, the N volume sub-features of the phonemes in the phoneme sequence sample are input. Figure 2 The pitch feature processing module and the volume feature processing module shown have the same structure, but the input feature information and the output feature information are different.

[0079] In another possible implementation, the N reference pitch sub-features and the N reference volume sub-features of the phoneme sequence sample may be further processed through a specific network structure, such as a transformer, to enhance the ability to extract feature information.

[0080] It can be understood that the above is only an example and not a limitation. The amount of data processed by the speech synthesis model at one time and the number of prosodic sub-features corresponding to each phoneme can be specified according to actual needs, such as 64, 128, 256, etc., and the embodiments of this application are not limited.

[0081] S102: Inputting text features of the phoneme sequence sample into a speech synthesis model for feature extraction to obtain predicted prosodic features of the phoneme sequence sample.

[0082] In a possible implementation, the text features of the phoneme sequence sample are input into the speech synthesis model for feature extraction to obtain the predicted prosodic features of the phoneme sequence sample, which can be performed as follows:

[0083] Local feature extraction and global feature extraction are performed on the text features of the phoneme sequence samples to obtain the predicted prosodic features of the phoneme sequence samples.

[0084] Furthermore, local feature extraction and global feature extraction are performed on the text features of the phoneme sequence samples to obtain the predicted prosodic features of the phoneme sequence samples. The specific implementation is as follows:

[0085] Local prosodic features are extracted from text features of the phoneme sequence samples to obtain multiple local prosodic features of the phoneme sequence samples; global prosodic features are extracted from multiple local prosodic features to obtain predicted prosodic features of the phoneme sequence samples.

[0086] The specific implementation is as follows: first, the text features of the phoneme sequence samples are passed through multiple one-dimensional convolutional networks to extract local prosodic features from the text features of the phoneme sequence samples, so as to obtain multiple local prosodic features of the phoneme sequence samples; the multiple local prosodic features are used to indicate the correlation between the prosodic features of the phonemes in the phoneme sequence samples; then, global prosodic features are extracted from the multiple local prosodic features through a time recurrent neural network (LSTM, Long Short-Term Memory) to obtain the global prosodic features of the phoneme sequence samples; the global prosodic features are used to indicate the correlation between the prosodic features of each word in the phoneme sequence samples; and the global prosodic features are used as the predicted prosodic features of the phoneme sequence samples.

[0087] See also Figure 3, which is a structural schematic diagram of a rhythm feature prediction module provided in an embodiment of the present application. The rhythm feature prediction module includes a local rhythm feature extraction module and a global rhythm feature extraction module. Among them, the local rhythm feature extraction module is mainly composed of H second feature extraction units, and the H second feature extraction units include Q first feature extraction units and H one-dimensional convolutional layers. The Q first feature extraction units include a one-dimensional convolutional layer (Conv1D), an activation layer (ReLU), regularization (Dropout) and normalization (BatchNorm). Among them, the one-dimensional convolutional layer (Conv1D) is used to extract local features, that is, the correlation between the rhythm features of adjacent phonemes; normalization (BatchNorm) is used to normalize the data at the Batch level to accelerate convergence; the activation layer (ReLU) is used to solve the problem of gradient disappearance; regularization (Dropout) is used to prevent the model from overfitting.

[0088] like Figure 3 As shown, after the text features of the phoneme sequence samples are input into Q first feature extraction units for local prosodic feature extraction, multiple local prosodic feature information is obtained, and then the multiple local prosodic feature information is input into H second feature extraction units, and local prosodic feature extraction is performed again through H one-dimensional convolutional layers (Conv1D) to obtain multiple local prosodic features of the phoneme sequence samples; finally, the multiple local prosodic features output by the H second feature extraction units are input into the time recurrent neural network for global prosodic feature extraction to obtain the global prosodic features of the phoneme sequence samples, and the global prosodic features of the phoneme sequence samples are used as the predicted prosodic features of the phoneme sequence samples.

[0089] Among them, in order to extract local features within different ranges to the greatest extent and make the extracted local feature information richer, the convolution kernel size (kernel size) of the one-dimensional convolution layer in each of the Q first feature extraction units is different, and the convolution kernel size (kernel size) of the one-dimensional convolution layer in each of the H second feature extraction units is also different.

[0090] The values ​​of Q and H are set according to the number of phoneme sequence samples during training, and this application does not impose any restrictions on this.

[0091] In a possible implementation, the predicted prosodic features in the embodiment of the present application include features of M dimensions; the M dimensions include pitch and volume; local feature extraction and global feature extraction are performed on the text features of the phoneme sequence sample to obtain the predicted prosodic features of the phoneme sequence sample, which can be specifically performed as follows:

[0092] Through the pitch feature prediction module of the speech synthesis model, local pitch features are extracted from the text features of the phoneme sequence samples to obtain multiple local pitch features of the phoneme sequence samples; global pitch features are extracted from the multiple local pitch features to obtain predicted pitch features of the phoneme sequence samples;

[0093] Through the volume feature prediction module of the speech synthesis model, local volume features are extracted from the text features of the phoneme sequence samples to obtain multiple local volume features of the phoneme sequence samples; global volume features are extracted from the multiple local volume features to obtain predicted volume features of the phoneme sequence samples.

[0094] like Figure 4 As shown, it is a structural schematic diagram of a feature prediction module provided in an embodiment of the present application. The prediction module (word-level Predictor) includes a pitch feature prediction module and a volume feature prediction module. The text features of the phoneme sequence sample are input into the pitch feature prediction module for pitch feature extraction to obtain the predicted pitch features of the phoneme sequence sample (word-Pitch Predictor); the text features of the phoneme sequence sample are input into the volume feature prediction module for volume feature extraction to obtain the predicted volume features of the phoneme sequence sample (word-EnergyPredictor).

[0095] in, Figure 4 The network structures of the pitch feature prediction module and the volume feature prediction module shown are Figure 3 The structure of the rhythm feature prediction module is shown in FIG. However, the feature information input to the pitch feature prediction module and the feature information output from the volume feature prediction module are different.

[0096] Combination Figure 3 and Figure 4 As shown in the structure diagram, the text features of the phoneme sequence samples are input into the pitch feature prediction module for pitch feature extraction to obtain the predicted pitch features of the phoneme sequence samples. The specific implementation is as follows:

[0097] The text features of the phoneme sequence samples are input into the pitch feature prediction module, and the local pitch features of the text features of the phoneme sequence samples are first extracted through multiple one-dimensional convolutional networks to obtain multiple local pitch features of the phoneme sequence samples; the multiple local pitch features are used to indicate the correlation between the pitch features of the phonemes in the phoneme sequence samples; then, the global pitch features are extracted from the multiple local pitch features through a time recurrent neural network (LSTM, Long Short-Term Memory) to obtain the global pitch features of the phoneme sequence samples; the global pitch features are used to indicate the correlation between the pitch features of each word in the phoneme sequence samples; and the global pitch features are used as the predicted pitch features of the phoneme sequence samples.

[0098] Similarly, combined Figure 3 and Figure 4 As shown in the structure diagram, the text features of the phoneme sequence samples are input into the volume feature prediction module to extract the volume features, and the predicted volume features of the phoneme sequence samples are obtained. The specific implementation is as follows:

[0099] The text features of the phoneme sequence samples are input into the volume feature prediction module. First, the local volume features of the text features of the phoneme sequence samples are extracted through multiple one-dimensional convolutional networks to obtain multiple local volume features of the phoneme sequence samples; the multiple local volume features are used to indicate the correlation between the volume features of the phonemes in the phoneme sequence samples; then, the global volume features of the multiple local volume features are extracted through a time recurrent neural network (LSTM, Long Short-Term Memory) to obtain the global volume features of the phoneme sequence samples; the global volume features are used to indicate the correlation between the volume features of each word in the phoneme sequence samples; and the global volume features are used as the predicted volume features of the phoneme sequence samples.

[0100] In another embodiment, the one-dimensional convolutional layer (Conv1D) in the pitch feature prediction module and the volume feature prediction module can be replaced with a network structure that shines in the image field, such as Vgg16, Vgg32, and ResNet. The overall network structure of the pitch feature prediction module and the volume feature prediction module can also be replaced with a network structure based on Transformer.

[0101] It can be understood that the above is only an example and not a limitation. In the one-dimensional convolution layer (Conv1D) and the time recurrent neural network, the convolution kernel of each layer can be specified according to actual needs, and the embodiments of the present application are not limited.

[0102] S103: Determine a loss value based on the reference prosodic features and the predicted prosodic features, and train a speech synthesis model based on the loss value.

[0103] In a possible implementation, the reference prosodic feature in the embodiment of the present application includes features of M dimensions; the M dimensions include pitch and volume, and a loss value is determined based on the reference prosodic feature and the predicted prosodic feature, and a speech synthesis model is trained based on the loss value, including:

[0104] determining a first loss value based on a reference pitch feature and a predicted pitch feature, and determining a second loss value based on a reference volume feature and a predicted volume feature;

[0105] Based on the first loss value, a pitch feature prediction module of the speech synthesis model is trained, and based on the second loss value, a volume feature prediction module of the speech synthesis model is trained.

[0106] In specific implementation, when calculating the first loss value, the reference pitch feature is first added to the text feature of the phoneme sequence sample, and then the first loss value is determined using the result of the addition and the predicted pitch feature. Similarly, the reference pitch feature is first added to the text feature of the phoneme sequence sample, and then the second loss value is determined using the result of the addition and the predicted volume feature.

[0107] It should be noted that in the embodiment of the present application, the network structures of the pitch feature prediction module and the volume feature prediction module are similar, but since the pitch feature prediction module is trained based on a first loss value determined by a reference pitch feature and a predicted pitch feature, and the volume feature prediction module is trained based on a second loss value determined by a reference volume feature and a predicted volume feature, the parameter information of the pitch feature prediction module and the volume feature prediction module are different, resulting in different output results.

[0108] In another possible implementation, in the embodiment of the present application, the phoneme sequence sample is input into the speech synthesis model, and the Mel spectrum of the phoneme sequence sample will be output. Therefore, when training the speech synthesis model, in addition to determining the first loss value by referring to the pitch feature and the predicted pitch feature, and determining the second loss value by referring to the volume feature and the predicted volume feature, in the embodiment of the present application, the third loss value of the speech synthesis model can be determined based on the Mel spectrum output when training the speech synthesis model and the actual Mel spectrum of the phoneme sequence sample, and the speech synthesis model can be trained based on the third loss value of the speech synthesis model.

[0109] Optionally, the speech synthesis model in the embodiment of the present application may be a FastSpeech2 model.

[0110] See also Figure 5 , which is a structural diagram of a speech synthesis model provided by an embodiment of the present application. Among them, the phoneme embedding module of the speech synthesis model is used to obtain the text features of the phoneme sequence; the position coding module connected to the phoneme embedding module is used to obtain the position information between each phoneme in the phoneme sequence; the phoneme coding module is used to learn the text features between higher-level phonemes such as syntax; the output of the phoneme coding module, the predicted pitch features of the phoneme sequence output by the pitch feature prediction module, and the predicted volume features of the phoneme sequence output by the volume feature prediction module are input into the variance adapter module, and the variance adapter module includes multiple predictors for predicting duration, pitch, and energy; the position coding module connected to the variance adapter module is used to obtain the position information between speech frames; the Mel decoder is used to obtain the Mel spectrum of the phoneme sequence, and finally synthesize speech according to the Mel spectrum.

[0111] In training Figure 5 In the speech synthesis model shown, the N pitch sub-features of the phonemes in the phoneme sequence sample are first input into the pitch feature processing module to obtain N reference pitch sub-features of the phoneme sequence sample, and then the N reference pitch sub-features of the phoneme sequence sample are added to the text features output by the phoneme encoding module to obtain the addition result, and at the same time, the text features output by the phoneme encoding module are input into the pitch feature prediction module to obtain the predicted pitch features of the phoneme sequence sample, and then based on the addition result and the predicted pitch features, the first loss value of the pitch feature prediction module is determined.

[0112] Similarly, the N volume sub-features of the phonemes in the phoneme sequence sample are input into the volume feature processing module to obtain N reference volume sub-features of the phoneme sequence sample, and then the N reference volume sub-features of the phoneme sequence sample are added to the text features output by the phoneme encoding module to obtain the addition result, and at the same time, the text features output by the phoneme encoding module are input into the volume feature prediction module to obtain the predicted volume features of the phoneme sequence sample, and then based on the addition result and the predicted volume features, the second loss value of the volume feature prediction module is determined.

[0113] Then, the output of the phoneme encoding module, the predicted pitch feature, and the predicted volume feature are input into the variance adapter module, and the Mel spectrum of the phoneme sequence sample is obtained through the variance adapter module, the position encoding module, and the Mel decoder. Finally, the pitch feature prediction module is trained according to the first loss value; the volume feature prediction module is trained according to the second loss value. Furthermore, the speech synthesis model can also be trained according to the third loss value determined by the Mel spectrum output when training the speech synthesis model and the actual Mel spectrum of the phoneme sequence sample.

[0114] Thus, when the first loss value and the second loss value are minimum, and / or the third loss value is minimum, and / or the number of training times reaches the set number threshold, a trained speech synthesis model is obtained.

[0115] In the embodiment of the present application, when training a speech synthesis model, the speech synthesis model is trained in combination with the prosodic features of the phonemes in the phoneme sequence samples and the text features of the phoneme sequence samples. Not only the predicted prosodic features obtained by feature extraction based on the text features of the phoneme sequence samples are used, but also the reference prosodic features obtained by feature operation on the prosodic features of the phonemes in the phoneme sequence samples are used. This allows the trained speech synthesis model to focus not only on the text features and prosodic features of the phonemes in the phoneme sequence samples, but also on the reference prosodic features and predicted prosodic features of each word in the phoneme sequence samples. When speech synthesis is performed using a speech synthesis model trained based on this method, the resulting synthesized speech is more natural and has a better sense of rhythm, thereby improving the naturalness and anthropomorphism of speech synthesis.

[0116] After the speech synthesis model is trained based on the above method, speech synthesis can be performed based on the speech synthesis model. Therefore, a speech synthesis method is proposed in the embodiment of the present application. Figure 6 , a flowchart of a speech synthesis method is provided for an embodiment of the present application. The method is executed by an electronic device, and the electronic device may be a terminal or a server. The terminal may be a vehicle-mounted device, a mobile phone, a tablet, a laptop computer, a desktop computer, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a wearable device, and the like, and the server may be a physical server or a server cluster composed of multiple servers. Figure 6 The speech synthesis method shown can be specifically executed by a processor of an electronic device.

[0117] like Figure 6 As shown, the method comprises the following steps:

[0118] S601: extracting prosodic features from text features of the to-be-synthesized phoneme sequence using a speech synthesis model to obtain target prosodic features of the to-be-synthesized phoneme sequence.

[0119] S602: Perform speech synthesis on the phoneme sequence to be synthesized based on the text features and target prosodic features of the phoneme sequence to be synthesized.

[0120] It should be emphasized that Figure 6 The speech synthesis model used in the speech synthesis method shown is trained according to the training method of the speech synthesis model described above.

[0121] In a possible implementation, prosodic feature extraction is performed on the text features of the to-be-synthesized phoneme sequence through the speech synthesis model to obtain the target prosodic features of the to-be-synthesized phoneme sequence, which can be performed as follows:

[0122] The speech synthesis model is used to perform local feature extraction and global feature extraction on the text features of the phoneme sequence to be synthesized, and the target prosodic features of the phoneme sequence to be synthesized are obtained.

[0123] In a possible implementation, the target prosodic features include features of M dimensions; the M dimensions include pitch and volume; local feature extraction and global feature extraction are performed on the text features of the to-be-synthesized phoneme sequence by the speech synthesis model to obtain the target prosodic features of the to-be-synthesized phoneme sequence, which is specifically performed as follows:

[0124] Through the pitch feature prediction module of the speech synthesis model, local pitch features are extracted from the text features of the to-be-synthesized phoneme sequence to obtain multiple local pitch features of the to-be-synthesized phoneme sequence; global pitch features are extracted from the multiple local pitch features to obtain target pitch features of the to-be-synthesized phoneme sequence;

[0125] Through the volume feature prediction module of the speech synthesis model, local volume features are extracted from the text features of the phoneme sequence to be synthesized to obtain multiple local volume features of the phoneme sequence to be synthesized; global volume features are extracted from the multiple local volume features to obtain the target volume features of the phoneme sequence to be synthesized.

[0126] In a possible implementation, after the target pitch feature and the target volume feature of the phoneme sequence to be synthesized are obtained, speech synthesis is performed on the phoneme sequence to be synthesized based on the text feature and the target prosodic feature of the phoneme sequence to be synthesized.

[0127] The specific implementation is to input the text features output by the phoneme encoding module, the target pitch features and the target volume features of the phoneme sequence to be synthesized into the variance adapter module, and obtain the Mel spectrum of the phoneme sequence to be synthesized through the variance adapter module, the position encoding module and the Mel decoder. Finally, the speech is synthesized based on the Mel spectrum of the phoneme sequence to be synthesized.

[0128] Among them, the text features between phonemes output by the phoneme encoding module can learn the degree of correlation between each phoneme.

[0129] like Figure 7 As shown, the phoneme sequence to be synthesized is input into the speech synthesis model, and the text features of the phoneme sequence to be synthesized are obtained through the phoneme embedding module, the position encoding module connected to the phoneme embedding module, and the phoneme encoding module. Then, the text features of the phoneme sequence to be synthesized are input into the pitch feature prediction module and the volume feature prediction module to obtain the target pitch features and target volume features of the phoneme sequence to be synthesized. Then, the text features of the phoneme sequence to be synthesized output by the phoneme encoding module, the target pitch features of the phoneme sequence to be synthesized output by the pitch feature prediction module, and the target volume features of the phoneme sequence to be synthesized output by the volume feature prediction module are input into the variance adapter module together. After the variance adapter module, the position encoding module and the Mel decoder, the Mel spectrum of the phoneme sequence to be synthesized is obtained. Finally, speech is synthesized based on the Mel spectrum of the phoneme sequence to be synthesized.

[0130] Therefore, by performing speech synthesis using the speech synthesis model trained by the speech synthesis model training method provided in the embodiment of the present application, the synthesized speech obtained is more natural and has a better sense of rhythm, thereby improving the naturalness and degree of anthropomorphism of the speech synthesis.

[0131] Based on the same inventive concept as the training method of the speech synthesis model, the embodiment of the present application provides a training device for a speech synthesis model. The device is used to implement any of the training methods for the speech synthesis model described above. Figure 8 , the training device 800 of the speech synthesis model includes:

[0132] The calculation unit 801 is used to input the prosodic features of the phonemes in the phoneme sequence sample into the speech synthesis model to perform feature calculation to obtain the reference prosodic features of the phoneme sequence sample;

[0133] A prediction unit 802, configured to input the text features of the phoneme sequence sample into the speech synthesis model for feature extraction to obtain predicted prosodic features of the phoneme sequence sample;

[0134] The training unit 803 is used to determine a loss value based on the reference prosodic feature and the predicted prosodic feature, and train the speech synthesis model based on the loss value.

[0135] In a possible implementation manner, the prosodic feature includes N prosodic sub-features, and the reference prosodic feature includes N reference prosodic sub-features; then the calculation unit 801 is specifically configured to:

[0136] The N prosodic sub-features of the phonemes in the phoneme sequence sample are input into the speech synthesis model for feature calculation to obtain N reference prosodic sub-features of the phoneme sequence sample.

[0137] In a possible implementation, the prosody feature includes sub-features of M dimensions; the M dimensions include pitch and volume; the N prosody sub-features include N pitch sub-features and N volume sub-features; the N reference prosody sub-features include N reference pitch sub-features and N reference volume sub-features; then the calculation unit 801 is specifically used to:

[0138] Performing feature operations on the N pitch sub-features of the phonemes in the phoneme sequence sample through the pitch feature processing module in the speech synthesis model to obtain N reference pitch sub-features of the phoneme sequence sample;

[0139] The volume feature processing module in the speech synthesis model performs feature calculation on the N volume sub-features of the phonemes in the phoneme sequence sample to obtain N reference volume sub-features of the phoneme sequence sample.

[0140] In a possible implementation manner, the calculation unit 801 is specifically configured to:

[0141] Based on the phonemes in the phoneme sequence sample and the preset correspondence between the phonemes and the words, determine a plurality of words composed of the phonemes in the phoneme sequence sample; wherein each word corresponds to a plurality of phonemes;

[0142] For each word in the plurality of words, determining N reference prosodic sub-features of the word based on N prosodic sub-features of a plurality of phonemes constituting the word;

[0143] The reference prosodic sub-features of the plurality of words are combined into N reference prosodic sub-features of the phoneme sequence sample.

[0144] In a possible implementation manner, the prediction unit 802 is specifically configured to:

[0145] Local feature extraction and global feature extraction are performed on the text features of the phoneme sequence samples to obtain predicted prosodic features of the phoneme sequence samples.

[0146] In a possible implementation, the predicted prosody feature includes features of M dimensions; the M dimensions include pitch and volume; then the prediction unit 802 is specifically configured to:

[0147] By using the pitch feature prediction module of the speech synthesis model, local pitch feature extraction is performed on the text features of the phoneme sequence sample to obtain multiple local pitch features of the phoneme sequence sample; global pitch feature extraction is performed on the multiple local pitch features to obtain predicted pitch features of the phoneme sequence sample;

[0148] Through the volume feature prediction module of the speech synthesis model, local volume feature extraction is performed on the text features of the phoneme sequence samples to obtain multiple local volume features of the phoneme sequence samples; global volume feature extraction is performed on the multiple local volume features to obtain predicted volume features of the phoneme sequence samples.

[0149] In a possible implementation manner, the reference prosodic feature includes features of M dimensions; the M dimensions include pitch and volume, and the training unit 803 is specifically configured to:

[0150] determining a first loss value based on the reference pitch feature and the predicted pitch feature, and determining a second loss value based on the reference volume feature and the predicted volume feature;

[0151] Based on the first loss value, the pitch feature prediction module of the speech synthesis model is trained, and based on the second loss value, the volume feature prediction module of the speech synthesis model is trained.

[0152] Based on the same inventive concept as the speech synthesis method, the present application embodiment provides a speech synthesis device. The device is used to implement any of the above-mentioned speech synthesis methods. Fig. 9 , the speech synthesis device 900 includes:

[0153] A feature extraction unit 901 is used to extract prosodic features from text features of a to-be-synthesized phoneme sequence to obtain target prosodic features of the to-be-synthesized phoneme sequence;

[0154] The speech synthesis unit 902 is used to perform speech synthesis on the to-be-synthesized phoneme sequence based on the text features of the to-be-synthesized phoneme sequence and the target prosodic features.

[0155] In a possible implementation, the feature extraction unit 901 is specifically configured to:

[0156] The speech synthesis model is used to perform local feature extraction and global feature extraction on the text features of the phoneme sequence to be synthesized, and the target prosodic features of the phoneme sequence to be synthesized are obtained.

[0157] In a possible implementation, the target prosodic feature includes features of M dimensions; the M dimensions include pitch and volume; then the feature extraction unit 901 is specifically configured to:

[0158] Through the pitch feature prediction module of the speech synthesis model, local pitch feature extraction is performed on the text features of the to-be-synthesized phoneme sequence to obtain a plurality of local pitch features of the to-be-synthesized phoneme sequence; global pitch feature extraction is performed on the plurality of local pitch features to obtain a target pitch feature of the to-be-synthesized phoneme sequence;

[0159] Through the volume feature prediction module of the speech synthesis model, local volume feature extraction is performed on the text features of the phoneme sequence to be synthesized to obtain multiple local volume features of the phoneme sequence to be synthesized; global volume feature extraction is performed on the multiple local volume features to obtain the target volume feature of the phoneme sequence to be synthesized.

[0160] As a possible product form of the above device, see Fig.10 The embodiment of the present application further provides an electronic device 1000, including:

[0161] At least one processor 1001; and a communication interface 1003 communicatively connected to the at least one processor 1001; the at least one processor 1001 executes instructions stored in the memory 1002, so that the electronic device 1000 executes the above-mentioned speech synthesis model training method and speech synthesis method through the communication interface 1003.

[0162] Optionally, the memory 1002 is located outside the electronic device 1000 .

[0163] Optionally, the electronic device 1000 includes the memory 1002, the memory 1002 is connected to the at least one processor 1001, and the memory 1002 stores instructions that can be executed by the at least one processor 1001. Fig.10 The dashed lines indicate that the memory 1002 is optional for the electronic device 1000 .

[0164] The processor 1001 and the memory 1002 may be coupled via an interface circuit or may be integrated together, which is not limited here.

[0165] The specific connection medium between the processor 1001, the memory 1002 and the communication interface 1003 is not limited in the embodiment of the present application. Fig.10 In the embodiment, the processor 1001, the memory 1002 and the communication interface 1003 are connected via a bus 1004. Fig.10 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0166] It should be understood that the processor mentioned in the embodiments of the present application can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor implemented by reading software code stored in a memory.

[0167] Exemplarily, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0168] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DirectRambus RAM, DR RAM).

[0169] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0170] It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0171] As another possible product form, an embodiment of the present application also provides a computer-readable storage medium, which is used to store instructions. When the instructions are executed, the computer executes the training method of the speech synthesis model and the speech synthesis method described above.

[0172] In an exemplary embodiment, a computer program product is also provided, including a computer program, which, when executed by the processor 1001, implements the above-mentioned speech synthesis model training method and speech synthesis method.

[0173] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0174] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0175] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0177] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for training a speech synthesis model, characterized in that: The method comprises: Inputting the prosodic features of the phonemes in the phoneme sequence sample into the speech synthesis model for feature calculation to obtain reference prosodic features of the phoneme sequence sample; Inputting the text features of the phoneme sequence sample into the speech synthesis model to extract features, thereby obtaining predicted prosodic features of the phoneme sequence sample; A loss value is determined based on the reference prosodic feature and the predicted prosodic feature, and the speech synthesis model is trained based on the loss value.

2. The method according to claim 1, characterized in that The prosodic feature includes N prosodic sub-features, and the reference prosodic feature includes N reference prosodic sub-features; the prosodic feature of the phonemes in the phoneme sequence sample is input into the speech synthesis model for feature calculation to obtain the reference prosodic feature of the phoneme sequence sample, including: The N prosodic sub-features of the phonemes in the phoneme sequence sample are input into the speech synthesis model for feature calculation to obtain N reference prosodic sub-features of the phoneme sequence sample.

3. The method according to claim 2, characterized in that The prosody feature includes sub-features of M dimensions; the M dimensions include pitch and volume; the N prosody sub-features include N pitch sub-features and N volume sub-features; the N reference prosody sub-features include N reference pitch sub-features and N reference volume sub-features; the N prosody sub-features of the phonemes in the phoneme sequence sample are input into the speech synthesis model for feature calculation to obtain the N reference prosody sub-features of the phoneme sequence sample, including: Performing feature operations on the N pitch sub-features of the phonemes in the phoneme sequence sample through the pitch feature processing module in the speech synthesis model to obtain N reference pitch sub-features of the phoneme sequence sample; The volume feature processing module in the speech synthesis model performs feature calculation on the N volume sub-features of the phonemes in the phoneme sequence sample to obtain N reference volume sub-features of the phoneme sequence sample.

4. The method according to claim 2, characterized in that The step of inputting the N prosodic sub-features of the phonemes in the phoneme sequence sample into the speech synthesis model for feature calculation to obtain the N reference prosodic sub-features of the phoneme sequence sample includes: Based on the phonemes in the phoneme sequence sample and the preset correspondence between the phonemes and the words, determine a plurality of words composed of the phonemes in the phoneme sequence sample; wherein each word corresponds to a plurality of phonemes; For each word in the plurality of words, determining N reference prosodic sub-features of the word based on N prosodic sub-features of a plurality of phonemes constituting the word; The reference prosodic sub-features of the plurality of words are combined into N reference prosodic sub-features of the phoneme sequence sample.

5. The method according to claim 1, characterized in that The step of inputting the text features of the phoneme sequence sample into the speech synthesis model for feature extraction to obtain the predicted prosodic features of the phoneme sequence sample includes: Local feature extraction and global feature extraction are performed on the text features of the phoneme sequence samples to obtain predicted prosodic features of the phoneme sequence samples.

6. The method according to claim 5, characterized in that The predicted prosodic features include features of M dimensions; the M dimensions include pitch and volume; then the local feature extraction and global feature extraction are performed on the text features of the phoneme sequence sample to obtain the predicted prosodic features of the phoneme sequence sample, including: By using the pitch feature prediction module of the speech synthesis model, local pitch feature extraction is performed on the text features of the phoneme sequence sample to obtain multiple local pitch features of the phoneme sequence sample; global pitch feature extraction is performed on the multiple local pitch features to obtain predicted pitch features of the phoneme sequence sample; Through the volume feature prediction module of the speech synthesis model, local volume feature extraction is performed on the text features of the phoneme sequence samples to obtain multiple local volume features of the phoneme sequence samples; global volume feature extraction is performed on the multiple local volume features to obtain predicted volume features of the phoneme sequence samples.

7. The method according to claim 6, characterized in that The reference prosodic feature includes features of M dimensions; the M dimensions include pitch and volume, and the loss value is determined based on the reference prosodic feature and the predicted prosodic feature, and the speech synthesis model is trained based on the loss value, including: determining a first loss value based on the reference pitch feature and the predicted pitch feature, and determining a second loss value based on the reference volume feature and the predicted volume feature; Based on the first loss value, the pitch feature prediction module of the speech synthesis model is trained, and based on the second loss value, the volume feature prediction module of the speech synthesis model is trained.

8. A speech synthesis method, characterized in that: The method comprises: Extracting prosodic features from text features of a to-be-synthesized phoneme sequence by a speech synthesis model, obtaining target prosodic features of the to-be-synthesized phoneme sequence; the speech synthesis model is trained according to the training method for a speech synthesis model described in any one of claims 1 to 7; Based on the text features of the to-be-synthesized phoneme sequence and the target prosodic features, speech synthesis is performed on the to-be-synthesized phoneme sequence.

9. A training device for a speech synthesis model, characterized in that: The device comprises: A calculation unit, used for inputting the prosodic features of the phonemes in the phoneme sequence sample into the speech synthesis model to perform feature calculation to obtain reference prosodic features of the phoneme sequence sample; A prediction unit, used for inputting the text features of the phoneme sequence sample into the speech synthesis model for feature extraction to obtain the predicted prosodic features of the phoneme sequence sample; A training unit is used to determine a loss value based on the reference prosodic feature and the predicted prosodic feature, and train the speech synthesis model based on the loss value.

10. A speech synthesis device, characterized in that: The device comprises: A feature extraction unit, used for extracting prosodic features from text features of the to-be-synthesized phoneme sequence to obtain target prosodic features of the to-be-synthesized phoneme sequence; The speech synthesis unit is used to perform speech synthesis on the to-be-synthesized phoneme sequence based on the text features of the to-be-synthesized phoneme sequence and the target prosodic features.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer-readable storage medium, characterized in that: It includes program codes. When the program product is run on an electronic device, the program codes are used to enable the electronic device to execute the steps of the method as claimed in any one of claims 1 to 8.

13. A computer program product, characterized in that The invention comprises computer instructions, which, when executed on a computer, enable the method described in any one of claims 1 to 8 to be implemented.

Images