Training method of speech synthesis model, speech synthesis method and related device

Abstract

The application provides a speech synthesis model training method, a speech synthesis method and related devices, including: obtaining a sample phoneme sequence and an acoustic feature label of training sample data; performing encoding processing on the sample phoneme sequence through an encoder to obtain a first sample phoneme encoding feature; inputting the acoustic feature label into a reference encoder to extract features, obtaining a word-level speech feature label, and performing feature extraction on the speech feature label to obtain a speech feature label vector; performing feature fusion on the first sample phoneme encoding feature and the speech feature label vector to obtain a first sample prosody fusion feature; performing decoding processing on the first sample prosody fusion feature through a decoder to obtain a sample acoustic feature; calculating a first loss value between the sample acoustic feature and the acoustic feature label by using a first loss function, and training the speech synthesis model based on the first loss value. The application can generate more human-like and more emotional speech synthesis results.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to a training method for a speech synthesis model, a speech synthesis method, and related apparatus. Background Technology

[0002] Speech synthesis refers to the process of converting text into spoken audio. Currently, speech synthesis models are commonly used to perform the above-mentioned operations of synthesizing spoken audio. However, the commonly used speech synthesis models (such as FastSpeech2) mainly model and learn prosody at the frame level or phoneme level, which results in synthesized speech that sounds like the mechanical speech of a robot, which is rather stiff and lacks emotion.

[0003] Therefore, how to generate more human-like and emotional synthesized speech has become an urgent technical problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of this application provide a training method for a speech synthesis model, a speech synthesis method, and related apparatus, which can generate more human-like and more emotional speech synthesis results.

[0005] In a first aspect, embodiments of this application provide a training method for a speech synthesis model. The method includes: acquiring sample phoneme sequences and acoustic feature labels from training sample data, wherein the training sample data is used to train the speech synthesis model, and the speech synthesis model includes an encoder, a reference encoder, and a decoder; encoding the sample phoneme sequences using the encoder to obtain first sample phoneme encoded features; inputting the acoustic feature labels into the reference encoder for feature extraction to obtain word-level speech feature labels, and extracting features from the speech feature labels to obtain a speech feature label vector; fusing the first sample phoneme encoded features and the speech feature label vector to obtain first sample prosodic fusion features; decoding the first sample prosodic fusion features using the decoder to obtain sample acoustic features; calculating a first loss value between the sample acoustic features and the acoustic feature labels using a first loss function, and training the speech synthesis model based on the first loss value.

[0006] Secondly, embodiments of this application provide a speech synthesis method, comprising: acquiring a phoneme sequence and word feature vectors of a text to be synthesized, wherein the word feature vectors are used to reflect the semantic information and emotional information carried by the text to be synthesized; inputting the phoneme sequence and word feature vectors into a speech synthesis model for speech synthesis processing to obtain a speech synthesis result corresponding to the text to be synthesized, wherein the speech synthesis model is obtained based on the training method described in the first aspect above.

[0007] Thirdly, embodiments of this application provide a training apparatus for a speech synthesis model, comprising: a first acquisition module, configured to acquire sample phoneme sequences and acoustic feature labels of training sample data, wherein the training sample data is used to train the speech synthesis model, the speech synthesis model including an encoder, a reference encoder, and a decoder; an encoding processing module, configured to encode the sample phoneme sequences using the encoder to obtain first sample phoneme encoded features; a feature extraction module, configured to input the acoustic feature labels into the reference encoder for feature extraction to obtain word-level speech feature labels, and to extract features from the speech feature labels to obtain a speech feature label vector; a feature fusion module, configured to fuse the first sample phoneme encoded features and the speech feature label vector to obtain first sample prosodic fusion features; a decoding processing module, configured to decode the first sample prosodic fusion features using the decoder to obtain sample acoustic features; and a parameter adjustment module, configured to calculate a first loss value between the sample acoustic features and the acoustic feature labels using a first loss function, and to train the speech synthesis model based on the first loss value.

[0008] Fourthly, embodiments of this application provide a speech synthesis apparatus, comprising: a second acquisition module, configured to acquire a phoneme sequence and word feature vectors of a text to be synthesized, wherein the word feature vectors are used to reflect the semantic information and emotional information carried by the text to be synthesized; and a synthesis module, configured to input the phoneme sequence and word feature vectors into a speech synthesis model for speech synthesis processing to obtain a speech synthesis result corresponding to the text to be synthesized, wherein the speech synthesis model is obtained based on the training method described in the first aspect above.

[0009] Fifthly, embodiments of this application provide a computationally readable storage medium storing a computer program, which, when loaded by a processor, is used to execute the training method for the speech synthesis model described in the first aspect, or the speech synthesis method described in the second aspect.

[0010] In a sixth aspect, embodiments of this application provide an electronic device, including: a processor; and a memory for storing processor-executable instructions, wherein the processor is used to execute the training method of the speech synthesis model described in the first aspect, or the speech synthesis method described in the second aspect.

[0011] This application provides a training method for a speech synthesis model, a speech synthesis method, and related apparatus. By acquiring acoustic feature labels corresponding to training sample data and inputting these acoustic feature labels into a reference encoder in the speech synthesis model for feature extraction, word-level speech feature labels are obtained. These word-level speech feature labels are then applied to the process of generating speech synthesis results, making the generated speech synthesis results more in line with the characteristics of human speech, more human-like, and more emotional. This avoids the problem of synthesized speech sounding mechanical and lacking emotion caused by the speech synthesis model learning frame-level or phoneme-level speech features. Attached Figure Description

[0012] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0013] Figure 1 This is a schematic diagram of an implementation environment for speech synthesis provided in an exemplary embodiment of this application;

[0014] Figure 2 This is a schematic diagram of the structure of a speech synthesis model provided in an exemplary embodiment of this application;

[0015] Figure 3 This is a flowchart illustrating a training method for a speech synthesis model provided in an exemplary embodiment of this application;

[0016] Figure 4 This is a flowchart illustrating a training method for a speech synthesis model provided in another exemplary embodiment of this application;

[0017] Figure 5 This is a schematic diagram of the structure of a reference encoder provided in an exemplary embodiment of this application;

[0018] Figure 6 This is a flowchart of the training process of a speech feature predictor provided in an exemplary embodiment of this application;

[0019] Figure 7 This is a schematic diagram of the structure of a speech synthesis model provided in another exemplary embodiment of this application;

[0020] Figure 8 This is a schematic flowchart of a speech synthesis method provided in an exemplary embodiment of this application;

[0021] Figure 9 This is a flowchart illustrating a speech synthesis method provided in another exemplary embodiment of this application;

[0022] Figure 10 This is a schematic diagram of the structure of a training device for a speech synthesis model provided in an exemplary embodiment of this application;

[0023] Figure 11 This is a schematic diagram of the structure of a speech synthesis device provided in an exemplary embodiment of this application;

[0024] Figure 12 This is a block diagram of an electronic device for speech synthesis or speech synthesis model training provided in an exemplary embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Non-autoregressive text-to-speech (TTS) models have attracted increasing attention from industry and academia. A commonly used non-autoregressive model is FastSpeech2. FastSpeech2 is a TTS model jointly proposed by Microsoft Research Asia and Zhejiang University. Building upon FastSpeech1, FastSpeech2 abandons the Teacher-Student knowledge distillation framework, reducing training complexity. It directly uses real speech data as the training target to avoid information loss, while introducing more accurate duration information and other variable information in speech, such as pitch and energy.

[0027] Speech synthesis models (such as FastSpeech2) mainly model and learn prosody at the frame level or phoneme level, while semantic and emotional information of text needs to be modeled and learned on the entire text sentence, which leads to the synthesized speech being flat and lacking in emotion.

[0028] To address the aforementioned problems, this application provides a method for training a speech synthesis model. Various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings. The speech synthesis model training method can be executed by an electronic device, specifically by the processor of the electronic device. The electronic device may include a terminal or a server. The terminal refers to mobile phones, laptops, tablets, smart wearable devices, and in-vehicle terminals, etc. The server may include an independent physical server, a server cluster consisting of multiple servers, or a cloud server capable of cloud computing.

[0029] See Figure 1 This is a schematic diagram of an implementation environment for speech synthesis provided in an exemplary embodiment of this application. Figure 1 As shown, the implementation environment includes a phoneme sequence 110 of the text to be synthesized, a word feature vector 120 of the text to be synthesized, a speech synthesis model 130, and a speech synthesis result 140.

[0030] Specifically, the phoneme sequence 110 and word feature vector 120 of the text to be synthesized are input into the speech synthesis model 130 to obtain a more human-like and more emotional speech synthesis result 140 (also called "synthesized speech").

[0031] For example, methods for generating more human-like speech synthesis results can be applied to customer service scenarios. A customer service scenario may include a server and a client connected to the server. Specifically, the server may include a speech synthesis model. The client displays the user interface for customer inquiries and receives customer-inputted questions, which are then sent to the server. Upon receiving a customer's question, the client sends it to the server, which then analyzes the question and inputs the phoneme sequence and word feature vectors of the corresponding response text (i.e., the text to be synthesized) into the speech synthesis model. This produces a speech synthesis result that more closely resembles the emotional expression of the customer service representative, which is then sent to the client. The client plays the synthesized speech to answer the customer's question.

[0032] It should be noted that the application scenarios can also include scenarios such as reading e-books aloud, playing prompts in amusement parks, and providing navigation prompts, etc., but this application embodiment does not specifically limit these scenarios.

[0033] Figure 2 This is a schematic diagram of the structure of a speech synthesis model provided in an exemplary embodiment of this application. For example... Figure 2 As shown, the speech synthesis model includes the following components.

[0034] The phoneme embedding layer 210 is used to embed the phoneme sequence 201 of the text to be synthesized, and obtain the vector representation of the phoneme sequence 201 (i.e., the phoneme feature vector), which can be a vector matrix.

[0035] The encoder 215 is used to encode the phoneme feature vectors to obtain second phoneme encoded features, in order to learn text information such as syntax in higher-level phoneme sequences.

[0036] Positional encoding includes a first positional encoding 220 and a second positional encoding 255, wherein the first positional encoding 220 and the second positional encoding 255 are used to obtain a vector representation of the position of each phoneme in the text to be synthesized (e.g., the first positional encoding feature) based on the phoneme sequence 201 or the phoneme feature vector.

[0037] The speech synthesis model may also include a third concatenation layer (not shown), which is used to concatenate the output of the encoder and the output of the first position encoding to obtain the first phoneme encoding feature.

[0038] The duration predictor 225 is used to predict the duration of the first phoneme encoded feature of the input and obtain the duration prediction result.

[0039] The pitch predictor 230 is used to predict the pitch of the word feature vector 202 of the input text to be synthesized, and obtain the pitch prediction result.

[0040] The Energy Predictor 235 is used to predict the volume of the input pitch prediction result and obtain the volume prediction result.

[0041] The pitch embedding layer 240 is used to embed the pitch prediction results to obtain the embedding vector, i.e., the pitch feature vector.

[0042] The volume embedding layer 245 is used to embed the volume prediction results to obtain the embedding vector, which is the volume feature vector.

[0043] The length regulator 250 is used to map the length of the feature fusion result of the first phoneme coding feature, the pitch feature vector and the volume feature vector (i.e., the first prosodic fusion feature) to the same length as the acoustic feature (e.g., the Mel spectrum).

[0044] Decoder 260 can employ a Mel-spectrogram Decoder to output predicted acoustic features, such as a Mel-spectrogram.

[0045] It should be understood that the embodiments of this application have changed the inputs of the pitch predictor and volume predictor in the speech synthesis model, replacing the feature fusion results output by the encoder and position encoder in the prior art with word feature vectors.

[0046] Based on the above speech synthesis model, see Figure 3, which is a schematic flowchart of a method for training a speech synthesis model provided by an exemplary embodiment of the present application. Figure 3 The method is executed by an electronic device such as a server. As Figure 3 shown, the method for training the speech synthesis model includes the following:

[0047] S310: Obtain the sample phoneme sequence and acoustic feature label of the training sample data.

[0048] In one embodiment, the training sample data is used to train the speech synthesis model, and the speech synthesis model includes an encoder, a reference encoder, and a decoder.

[0049] Specifically, a phoneme is the smallest speech unit divided according to the natural attributes of speech. Analyzing according to the pronunciation actions in a syllable, one action constitutes one phoneme. For example, "yu" has two phonemes, and "yan" has three phonemes.

[0050] The sample phoneme sequence of the training sample data can be obtained by arranging the phonemes corresponding to the characters in the sample text corresponding to the training sample data in the order of pronunciation time, pitch, and pause duration. For example, if the sample text is "Jeju Island, the largest island in South Korea", by analyzing the pinyin, pitch, and pause duration between adjacent pinyins of each character in the sample text, the sample phoneme sequence corresponding to the training sample data can be obtained as {han2 guo2 7zui4 da4de5 dao6 yu6 ji3zhou1 dao3}, where "han2" represents the phoneme corresponding to the character "韩" in the sample text, "guo2" represents the phoneme corresponding to the character "国" in the sample text, and so on. And the sample phoneme sequence is composed of the pinyin of the character and the identifier representing the pitch of the pinyin. The identifier 1 after the pinyin indicates the first tone, the identifier 2 indicates the second tone, the identifier 3 indicates the third tone, the identifier 4 indicates the fourth tone, the identifier 5 indicates the light tone, the identifier 6 indicates the variant tone, the identifier 7 between phonemes indicates a short pause duration, the identifier 8 between phonemes indicates a moderate pause duration, and the identifier 9 between phonemes indicates a long pause duration.

[0051] The training sample data can be sample speech audio, and the acoustic feature label can be the true Mel spectrogram (Target Mel-Spectrogram) corresponding to the sample speech audio.

[0052] S320: Perform encoding processing on the sample phoneme sequence through the encoder to obtain the first sample phoneme encoding feature.

[0053] Specifically, the sample phoneme sequence is input into the phoneme embedding layer for feature extraction, resulting in a sample phoneme feature vector. This feature vector reflects information such as the pitch and rhythm of each phoneme in the sample phoneme sequence. The sample phoneme feature vector is then input into an encoder for encoding, yielding a second sample phoneme encoded feature. Additionally, the sample phoneme feature vector is input into a first positional encoder for encoding, yielding a first sample positional encoded feature. This first sample positional encoded feature represents information such as the pronunciation order of each phoneme.

[0054] Furthermore, the phoneme encoding features of the second sample and the position encoding features of the first sample are fused to obtain the phoneme encoding features of the first sample. The feature fusion method can be feature addition, or it can utilize the third concatenation layer of the speech synthesis model, etc. This application embodiment does not specifically limit the feature fusion method.

[0055] In one embodiment, the second sample phoneme encoding features and the first sample position encoding features can be input into the third concatenation layer for feature concatenation to obtain the first sample phoneme encoding features of the phoneme sequence.

[0056] S330: Input the acoustic feature labels into the reference encoder for feature extraction to obtain word-level speech feature labels, and extract features from the speech feature labels to obtain speech feature label vectors.

[0057] Specifically, during the training phase of the speech synthesis model, the model may also include a reference encoder. The reference encoder can include convolutional modules, feature processing modules, average pooling modules, and fully connected layers. The reference encoder can be used to learn word-level speech feature labels from acoustic feature tags.

[0058] In one embodiment, the speech feature labels may include prosodic feature labels such as pitch labels and volume labels. The process of inputting the acoustic feature labels into the reference encoder for feature extraction to obtain word-level speech feature labels may include: inputting the acoustic feature labels into the reference encoder for word-level feature extraction to obtain word-level pitch labels and word-level volume labels, wherein the pitch label can be understood as the actual pitch prediction result (TargetPitch), and the volume label can be understood as the actual volume prediction result (Target Energy).

[0059] Furthermore, feature extraction is performed on the speech feature labels to obtain speech feature label vectors. This may include: inputting pitch labels into a pitch embedding layer for embedding processing to obtain a pitch feature label vector, and inputting volume labels into a volume embedding layer for embedding processing to obtain a volume feature label vector.

[0060] It should be noted that for a detailed description of this step, please refer to [link / reference needed]. Figure 5 and 6 The embodiments are described herein and will not be repeated here.

[0061] S340: The phoneme encoding features and speech feature label vectors of the first sample are fused to obtain the prosodic fusion features of the first sample.

[0062] Specifically, the speech synthesis model may further include a first concatenation layer and a second concatenation layer. The first concatenation layer concatenates the phoneme encoding features and pitch feature label vectors of the first sample to obtain a first sample concatenated feature vector. The second concatenation layer concatenates the first sample concatenated feature vector and the sample volume feature label vector to obtain the first sample prosodic fusion feature. Alternatively, the first sample phoneme encoding features, volume feature label vector, and pitch feature label vector can be simultaneously added together to obtain the first sample prosodic fusion feature, where feature addition can be understood as a method of feature fusion. It should be noted that the specific method of feature fusion can be flexibly set according to the actual situation, and the embodiments of this application do not impose specific limitations on it.

[0063] S350: The first sample's prosodic fusion features are decoded by the decoder to obtain the sample's acoustic features.

[0064] Specifically, before inputting the prosodic fusion features of the first sample into the decoder, the method may further include: inputting the phoneme encoding features of the first sample into the duration predictor for duration prediction processing, and obtaining the duration prediction result of the sample.

[0065] In one embodiment, the sample duration prediction result and the first sample prosodic fusion feature are input into a length adjuster for length adjustment to obtain a second sample prosodic fusion feature, wherein the second sample prosodic fusion feature has the same dimension as the acoustic label. Then, the second sample prosodic fusion feature is concatenated and fused with the second sample positional encoding feature output by the second positional encoding to obtain a third sample prosodic fusion feature, wherein the concatenation and fusion method can be feature addition. This third sample prosodic fusion feature is input into a decoder for decoding processing to obtain sample acoustic features, wherein the decoder can be a Mel decoder, and the sample acoustic features can be Mel-spectrum.

[0066] S360: Calculate the first loss value between the acoustic features of the sample and the acoustic feature labels using the first loss function, and train the speech synthesis model based on the first loss value.

[0067] Specifically, a first loss function is used to calculate the first loss value between the acoustic features of the training sample data output by the speech synthesis model and the acoustic feature labels. This speech synthesis model can be understood as an incompletely trained model. Then, based on this first loss value, the parameters of the speech synthesis model are adjusted until the desired speech synthesis model is obtained. The first loss function can be an L1 loss function, an L2 loss function, a cross-entropy loss function, or a mean squared error loss function, etc., and this embodiment does not specifically limit its application. It should be noted that the process of adjusting the parameters of the speech synthesis model using the first loss value can be understood as a backpropagation process, for example, using a back-propagation algorithm to adjust the parameters of the speech synthesis model.

[0068] Preferably, in this embodiment, the L1 loss function or the L2 loss function is used as the first loss function. It should be noted that the speech synthesis model can be a word-level prosodic network structure.

[0069] Therefore, this embodiment of the application obtains the acoustic feature labels corresponding to the training sample data and inputs the acoustic feature labels into the reference encoder in the speech synthesis model for feature extraction to obtain word-level speech feature labels. Then, the word-level speech feature labels are applied to the process of generating speech synthesis results, making the generated speech synthesis results more in line with the characteristics of human speech, more human-like and more emotional. This avoids the problem of the synthesized speech being mechanical and emotionally bland due to the speech synthesis model learning frame-level or phoneme-level speech features.

[0070] In one embodiment, the reference encoder includes a convolutional module, a feature processing module, an average pooling module, and a fully connected layer. Speech feature labels include pitch labels and volume labels. See specifically... Figure 5 The reference encoder 500 may include a convolutional stack 510, a flattern 520, a token-wise pooling 530, and a dense fully connected layer 540. The convolutional stack 510 may include a first convolutional layer 511, a rectified linear unit (ReLU) 512, and a dropout 513. The number of convolutional stacks may be one or more; however, this embodiment does not specifically limit the number of convolutional stacks.

[0071] The first convolutional layer 511 can be Conv1d or Conv2d, and the specific implementation of the first convolutional layer in this application is not limited. Since the input of the reference encoder is an acoustic feature label with a data dimension of 2, the acoustic feature label can be regarded as an image and processed using the first convolutional layer (e.g., Conv2d).

[0072] Based on this, the following is a summary: Figure 4 The feasible implementation methods of step S330 above are described in detail. Specifically, inputting the acoustic feature labels into the reference encoder for feature extraction to obtain word-level speech feature labels may include:

[0073] S410: Input the acoustic feature labels into the convolution module for convolution processing to obtain the acoustic label feature vector, and input the acoustic label feature vector into the feature processing module for dimensionality processing to obtain the dimensionality-processed acoustic label feature vector.

[0074] Specifically, the convolution module can be used to learn the feature information of acoustic feature labels. Then, the feature processing module can perform dimensionality operations such as unrolling on the acoustic label vector output by the convolution module, thereby obtaining the dimensionality-processed acoustic label feature vector, for example, unrolling to a dimension of...<batch_size,mel_length*dim> Where batch_size represents the number of samples passed to the electronic device at one time for training the speech synthesis model, mel_length represents the mel spectrum length, and dim represents the dimension.

[0075] S420: Input the dimension-processed acoustic label feature vector into the average pooling module for pooling processing to obtain the pooled acoustic label feature vector.

[0076] In one embodiment, the pooled acoustic label feature vector is used to reflect the word-level feature vectors of the training sample data.

[0077] Specifically, the average pooling module transforms frame-level feature vectors into word-level (token-level) feature vectors. For example, consider the phoneme sequence "xin1 yuan3 8di4 7zi4pian1". The text length (text_length) of this phoneme sequence (excluding punctuation) is 8. Assuming the Mel spectrum length corresponding to this phoneme sequence is 20 (dimension <20,dim>), the duration prediction result is [2,4,2,3,1,2,4,2]. Therefore, the total duration is sum(duration) = 2 + 4 + 2 + 3 + 1 + 2 + 4 + 2 = 20. This indicates that the pronunciation length of "xin1" is 2, the pronunciation length of "yuan3" is 4, and so on. Based on the pronunciation duration of each phoneme and the Mel feature, the dimension of the Mel spectrum for each phoneme's corresponding region can be obtained, where "xin1" is <2,dim> and "yuan3" is <4,dim>. Furthermore, the pronunciation duration of each character or word in the phoneme sequence can be averaged to obtain a word-level feature vector.

[0078] It should be noted that during the training of the speech synthesis model, each phoneme in the duration label has its corresponding duration information. Adding the duration information of all phonemes yields a value consistent with the length of the acoustic feature. For example, the expanded dimension of the acoustic label feature vector after dimensionality processing is...<batch_size,mel_length*dim> In this case, the output dimension after average pooling is<batch_size,text_length*dim> The mapping between text_length and mel_length is determined by the duration tag.

[0079] S430: Input the pooled acoustic label feature vector into the fully connected layer for feature processing, and perform dimensional splitting on the feature-processed acoustic label feature vector to obtain pitch and volume labels.

[0080] Specifically, the fully connected layer is used to extract features and map dimensions of the pooled acoustic label features to obtain the acoustic label feature vector after feature processing. The feature dimension after the dimension mapping by the fully connected layer is 2, for example, {batch_size,text_legnth,2}.

[0081] In one embodiment, since the pitch and volume labels are both 1-dimensional (e.g., {batch_size, text_length, 1}), while the output of the fully connected layer is 2-dimensional (e.g., {batch_size, text_length, 2}), the speech synthesis model can also include a splitting module (e.g., Ground-Truth Prosody). The splitting module is used to split the 2-dimensional vector output from the fully connected layer in the reference encoder into two 1-dimensional vectors of equal dimension, where each vector has a different feature value: one vector represents the pitch label, and the other represents the volume label.

[0082] Therefore, it can be seen that by setting a reference encoder, the embodiments of this application learn word-level feature vectors in acoustic feature labels, which ensures that the speech synthesis model generates more expressive and emotional speech synthesis results, and avoids the problem of the synthesized speech having a strong mechanical feel due to the speech synthesis model learning frame-level or phoneme-level speech features.

[0083] In one embodiment, as described above, the speech synthesis model may further include a speech feature predictor. Specifically, the speech feature predictor may include a pitch predictor and a volume predictor. During the training of the speech synthesis model, the pitch predictor and volume predictor may be trained simultaneously. The following will illustrate... Figure 6 This section details the training process for the pitch predictor and volume predictor. For example... Figure 6 As shown, training the pitch predictor and volume predictor may include the following steps:

[0084] S610: Obtain the feature vector of the sample words in the training sample data.

[0085] In one embodiment, the sample word feature vector is used to reflect the semantic information of the training sample data and the emotional information carried by the training sample data.

[0086] Specifically, obtaining the sample word feature vector of the training sample data may include: inputting the sample phoneme sequence of the training sample data into a pre-trained language representation model to obtain the sample sentence feature vector rich in semantic and emotional information; and performing feature extraction on the sample sentence feature vector at the word level to obtain the sample word feature vector.

[0087] The feature vector of the sample word can be the feature vector corresponding to the word in the training sample data, or it can be the feature vector corresponding to the character in the training sample data. This application does not specifically limit this.

[0088] In one embodiment, the language representation model can be a Bidirectional Encoder Representations from Transformer (BERT) model. The Transformer is a temporal model based on a self-attention mechanism. Its encoder part can effectively encode temporal information, demonstrating significantly better temporal information processing capabilities than Long Short-Term Memory (LSTM) models, and it is also faster. It is widely used in natural language processing, computer vision, machine translation, speech recognition, and other fields.

[0089] S620: Input the sample word feature vector into the pitch predictor to perform pitch prediction, obtain the sample pitch prediction result, and input the sample pitch prediction result into the volume predictor to perform volume prediction, obtain the sample volume prediction result.

[0090] S630: Calculate the second loss value between the tone label and the sample tone prediction result using the second loss function, and train the tone predictor based on the second loss value.

[0091] Specifically, the second loss function can be an L1 loss function, an L2 loss function, a cross-entropy loss function, or a mean squared error loss function, etc. Preferably, in this embodiment, the L1 loss function is used as the second loss function.

[0092] For example, the L1 loss function is used to calculate a second loss value between the pitch label and the sample pitch prediction result, and then the pitch predictor parameters are adjusted based on this second loss value.

[0093] S640: Calculate the third loss value between the volume label and the sample volume prediction result using the third loss function, and train the volume predictor based on the third loss value.

[0094] Specifically, the third loss function can also be the L1 loss function, L2 loss function, cross-entropy loss function, or mean squared error loss function, etc. It should be noted that the second and third loss functions can be the same or different; this application embodiment does not impose specific limitations on this. Preferably, this application embodiment uses the L1 loss function as the third loss function.

[0095] For example, the L1 loss function is used to calculate a third loss value between the volume label and the sample volume prediction result, and then the parameters of the volume predictor are adjusted based on this third loss value.

[0096] It should be noted that the input to the duration predictor is the phoneme encoding feature of the first sample. The predicted duration output of the duration predictor also needs to be used together with the duration label (target duration) in the calculation of the loss function (e.g., L1 Loss function) to make the duration prediction result closer to the duration label.

[0097] It should be noted that during the training of the speech synthesis model, as the number of iterations increases, the distance between the sample pitch prediction results obtained from the pitch predictor and the sample volume prediction results obtained from the volume predictor and the pitch label and volume label will become smaller and smaller. Therefore, when applying the converged speech synthesis model for prediction, a reference encoder is not required, only the pitch predictor and volume predictor are needed.

[0098] Therefore, the embodiments of this application train at least one predictor (e.g., a pitch predictor) in the speech synthesis model using at least one loss function, thereby making the output of at least one predictor more similar to the real result, thus enhancing the emotional information of the speech synthesis result.

[0099] Based on the foregoing Figures 1-6 The embodiments described herein provide another speech synthesis model, see [link to embodiment description]. Figure 7 This is a schematic diagram of the structure of a speech synthesis model provided in another exemplary embodiment of this application. The following is in conjunction with... Figure 7 The aforementioned speech synthesis model specifically introduces a training method and steps for a speech synthesis model:

[0100] The sample phoneme sequence 701 is input into the phoneme embedding layer 710 to obtain the sample phoneme feature vector. Then, the sample phoneme feature vector is input into the encoder 715 to obtain the second sample phoneme encoding feature, which is then concatenated and fused with the first sample position encoding feature obtained by inputting the sample phoneme feature vector into the first position encoding 720 to obtain the first sample phoneme encoding feature. The acoustic feature label 775 is input into the reference encoder 780 to obtain the word-level speech feature label, which is then input into the splitting module 785 to obtain the volume label 790 and the pitch label 792. The volume label 790 is input into the volume embedding layer (not shown) to obtain the volume feature label vector 745, and the pitch label 792 is input into the pitch embedding layer (not shown) to obtain the pitch feature label vector 740.

[0101] Further, the first sample phoneme encoding features, pitch feature label vector 740, and volume feature label vector 745 are concatenated and fused to obtain the first sample prosodic fusion feature. Then, the first sample prosodic fusion feature and duration label are input into the length adjuster 750 for length adjustment to obtain the second sample prosodic fusion feature. This second sample prosodic fusion feature is then fused with the second sample position encoding feature obtained from the second position encoding 755 to obtain the third sample prosodic fusion feature. Finally, this third sample prosodic fusion feature is input into the decoder 760 for decoding to obtain the sample acoustic feature 765. A first loss function 770 (e.g., L1 loss function) is used to calculate the first loss value between the sample acoustic feature 765 and the acoustic feature label 775, and the speech synthesis model is trained based on this first loss value.

[0102] The sample word feature vector 702 is input into the pitch predictor 730 to obtain the sample pitch prediction result, and the sample pitch prediction result is input into the volume predictor 735 to obtain the sample volume prediction result. A second loss function 796 is used to calculate a second loss value between the pitch label 792 and the sample pitch prediction result, and the parameters of the pitch predictor 730 are adjusted based on the second loss value. Similarly, a third loss function 794 is used to calculate a third loss value between the volume label 790 and the sample volume prediction result, and the parameters of the volume predictor 735 are adjusted based on the third loss value.

[0103] In addition, it also includes inputting the phoneme encoding features of the first sample into the duration predictor 725 to perform duration prediction, obtaining the sample duration prediction result, and calculating the loss value with the duration label, so as to adjust the parameters of the duration predictor 725 based on the loss value.

[0104] Therefore, this embodiment of the application obtains acoustic feature labels corresponding to training sample data and inputs these acoustic feature labels into the reference encoder in the speech synthesis model for feature extraction to obtain word-level speech feature labels. These word-level speech feature labels are then applied to the process of generating the speech synthesis result, making the generated speech synthesis result more consistent with human speech characteristics. Simultaneously, this embodiment of the application utilizes at least one loss function to train at least one predictor (e.g., a pitch predictor) in the speech synthesis model, making the output of at least one predictor closer to the real result, thereby ensuring the generation of more human-like speech synthesis results and enhancing the user experience.

[0105] Based on the aforementioned speech synthesis models and their training methods, this application also provides a speech synthesis method. See [link to relevant documentation]. Figure 8 This is a flowchart illustrating a speech synthesis method provided in an exemplary embodiment of this application. Figure 8The method is executed by a speech synthesis model running on an electronic device. For example... Figure 8 As shown, the speech synthesis method includes the following steps:

[0106] S810: Obtain the phoneme sequence and word feature vectors of the text to be synthesized.

[0107] S820: Input the phoneme sequence and word feature vector into the speech synthesis model for speech synthesis processing to obtain the speech synthesis result corresponding to the text to be synthesized.

[0108] The speech synthesis model is obtained based on the training method described in the above embodiments.

[0109] Specifically, the output layer of the speech synthesis model can have a speech reconstruction function. By reconstructing the acoustic features corresponding to the speech synthesis result, a sound wave is obtained. Then, using speech generation technology, a speech synthesis result corresponding to the text to be synthesized is generated based on the sound wave. It should be noted that the method of generating the speech synthesis result can be flexibly set according to actual needs, and this application embodiment does not impose specific limitations on it.

[0110] It should be noted that, in terms of data, the main difference between the training and application phases is that the training phase uses training sample data, while the application phase uses phoneme sequences of the text to be synthesized. Furthermore, in terms of the structure of the speech synthesis model, the training phase includes an additional reference encoder, which is used to learn word-level speech feature labels (e.g., pitch and volume labels) from the acoustic feature tags.

[0111] Therefore, compared with the prior art which only uses the phoneme sequence of the text to be synthesized as the input of the speech synthesis model, the embodiments of this application use the phoneme sequence of the text to be synthesized, as well as word feature vectors rich in semantic and emotional information, as the input of the speech synthesis model. This allows the speech synthesis model to combine word-level predicted speech features (i.e., prosodic feature vectors such as pitch feature vectors and volume feature vectors) obtained based on word feature vectors during the prediction process, taking into account information such as emotion and speech, thereby making the generated speech synthesis result more human-like and more emotional.

[0112] In one embodiment, inputting a phoneme sequence and word feature vectors into a speech synthesis model for speech synthesis processing to obtain the speech synthesis result corresponding to the text to be synthesized may include the following steps S910-S930:

[0113] S910: The phoneme sequence is encoded by the encoder to obtain the first phoneme encoding feature, and the word feature vector is input into the speech feature predictor for prediction to obtain the speech prediction result. The speech prediction result is then used to extract features to obtain the predicted speech features.

[0114] In one embodiment, step S910 includes the following feasible implementation: inputting the phoneme sequence into the phoneme embedding layer for feature extraction to obtain the phoneme feature vector of the text to be synthesized; inputting the phoneme feature vector into the encoder for encoding processing to obtain the second phoneme encoding feature; and inputting the phoneme feature vector into the first position encoder to obtain the first position encoding feature; and fusing the second phoneme encoding feature and the first position encoding feature to obtain the first phoneme encoding feature.

[0115] In one embodiment of this application, the speech feature predictor includes a duration predictor. The speech synthesis method further includes: inputting a first phoneme encoding feature into the duration predictor for duration prediction processing to obtain a duration prediction result; inputting a first prosodic fusion feature into a decoder for decoding processing to obtain acoustic features; and including: adjusting the length of the first prosodic fusion feature according to the duration prediction result to obtain a second prosodic fusion feature; and inputting the second prosodic fusion feature into the decoder for decoding processing to obtain acoustic features.

[0116] Specifically, the acoustic features can be Mel spectra. Since the length of a phoneme sequence is usually shorter than the length of a Mel spectrum, the speech synthesis model has a length adjuster to accommodate the feature length of the decoder output. This length adjuster is used to pad the length of the first prosodic fusion feature of the phoneme sequence to match the length of the Mel spectrum.

[0117] The duration prediction result and the first prosodic fusion feature are input into the length adjuster. The length of the first prosodic fusion feature is adjusted according to the duration prediction result to obtain the second prosodic fusion feature. It should be noted that the duration prediction result does not need to go through the feature extraction layer to obtain the duration feature vector. The role of the duration predictor is to predict the duration information corresponding to each phoneme. Then, through the length adjuster, the first prosodic fusion feature is mapped to the same length as the Mel spectrum.

[0118] Therefore, it can be seen that the embodiments of this application, by setting a length adjuster, map the length of the first prosodic fusion feature to the same length as the Mel spectrum, thereby ensuring the normal output of the Mel spectrum feature during the process of applying the speech synthesis model to output synthesized speech.

[0119] S920: The first phoneme coding feature and the predicted speech feature are fused to obtain the first prosodic fusion feature.

[0120] S930: Input the first prosodic fusion feature into the decoder for decoding to obtain acoustic features, and perform speech synthesis based on the acoustic features to obtain the speech synthesis result.

[0121] Therefore, compared with the prior art, the embodiments of this application change the input of the tone predictor from the first phoneme encoding feature (the input of the prior art) to a word feature vector rich in semantic and emotional information, thereby obtaining word-level tone feature vectors and volume feature vectors. In other words, the embodiments of this application combine word-level prosodic features and consider emotional information, thereby reducing the mechanical feel of the speech synthesis results.

[0122] In one embodiment of this application, the speech feature predictor includes a pitch predictor and a volume predictor, and the speech prediction result includes a pitch prediction result and a volume prediction result. The process of inputting word feature vectors into the speech feature predictor for prediction processing to obtain the speech prediction result includes: inputting word feature vectors into the pitch predictor for pitch feature prediction processing to obtain a pitch prediction result; and inputting the pitch prediction result into the volume predictor for volume feature prediction processing to obtain a volume prediction result.

[0123] Based on the aforementioned training method embodiments for speech synthesis models, this application provides a speech synthesis model, see [link to relevant documentation]. Figure 10 This is a schematic diagram of the structure of a training device 1000 for a speech synthesis model provided in an exemplary embodiment of this application. Figure 10 As shown, the device includes: a first acquisition module 1010, an encoding processing module 1020, a feature extraction module 1030, a feature fusion module 1040, a decoding processing module 1050, and a parameter adjustment module 1060.

[0124] The first acquisition module 1010 is used to acquire the sample phoneme sequence and acoustic feature labels of the training sample data, wherein the training sample data is used to train the speech synthesis model, which includes an encoder, a reference encoder, and a decoder; the encoding processing module 1020 is used to encode the sample phoneme sequence through the encoder to obtain the first sample phoneme encoding features; the feature extraction module 1030 is used to input the acoustic feature labels into the reference encoder for feature extraction to obtain word-level speech feature labels, and to extract features from the speech feature labels to obtain a speech feature label vector; the feature fusion module 1040 is used to fuse the first sample phoneme encoding features and the speech feature label vector to obtain the first sample prosodic fusion features; the decoding processing module 1050 is used to decode the first sample prosodic fusion features through the decoder to obtain the sample acoustic features; the parameter adjustment module 1060 is used to calculate the first loss value between the sample acoustic features and the acoustic feature labels using a first loss function, and to train the speech synthesis model based on the first loss value.

[0125] This application provides a training device for a speech synthesis model. By acquiring acoustic feature labels corresponding to training sample data and inputting these acoustic feature labels into a reference encoder in the speech synthesis model for feature extraction, word-level speech feature labels are obtained. These word-level speech feature labels are then applied to the process of generating speech synthesis results, making the generated speech synthesis results more in line with the characteristics of human speech, more human-like, and more emotional. This avoids the problem of synthesized speech sounding mechanical and lacking emotion caused by the speech synthesis model learning frame-level or phoneme-level speech features.

[0126] According to one embodiment of this application, the reference encoder includes a convolutional module, a feature processing module, an average pooling module, and a fully connected layer; the speech feature labels include pitch labels and volume labels.

[0127] When the feature extraction module 1030 inputs the acoustic feature labels into the reference encoder for feature extraction and obtains word-level speech feature labels, it performs the following steps:

[0128] The acoustic feature labels are input into the convolution module for convolution processing to obtain acoustic label feature vectors. The acoustic label feature vectors are then input into the feature processing module for dimensionality processing to obtain dimensionality-processed acoustic label feature vectors. The dimensionality-processed acoustic label feature vectors are then input into the average pooling module for pooling processing to obtain pooled acoustic label feature vectors. The pooled acoustic label feature vectors are used to reflect the word-level feature vectors of the training sample data. The pooled acoustic label feature vectors are then input into the fully connected layer for feature processing, and the dimensionality of the feature-processed acoustic label feature vectors is split to obtain pitch and volume labels.

[0129] According to one embodiment of this application, the speech synthesis model further includes a speech feature predictor, which includes a pitch predictor and a volume predictor. The first acquisition module 1010 is further configured to perform the following steps:

[0130] The process involves obtaining sample word feature vectors from the training sample data, where the sample word feature vectors reflect the semantic information and sentiment information carried by the training sample data; inputting the sample word feature vectors into a pitch predictor to predict pitch, obtaining the sample pitch prediction results; inputting the sample pitch prediction results into a volume predictor to predict volume, obtaining the sample volume prediction results; calculating a second loss value between the pitch label and the sample pitch prediction results using a second loss function, and training the pitch predictor based on the second loss value; and calculating a third loss value between the volume label and the sample volume prediction results using a third loss function, and training the volume predictor based on the third loss value.

[0131] It should be understood that the specific working process and functions of the first acquisition module 1010, encoding processing module 1020, feature extraction module 1030, feature fusion module 1040, decoding processing module 1050, and parameter adjustment module 1060 in the above embodiments can be referred to the above. Figures 3 to 6 The description of the training method for the speech synthesis model provided in the embodiments will not be repeated here to avoid repetition.

[0132] Figure 11 This is a schematic diagram of the structure of a speech synthesis device 1100 provided in an exemplary embodiment of this application. Figure 11 As shown, the device includes a second acquisition module 1110 and a synthesis module 1120.

[0133] The second acquisition module 1110 is used to acquire the phoneme sequence and word feature vector of the text to be synthesized, wherein the word feature vector is used to reflect the semantic information and emotional information carried by the text to be synthesized; the synthesis module 1120 is used to input the phoneme sequence and word feature vector into the speech synthesis model for speech synthesis processing to obtain the speech synthesis result corresponding to the text to be synthesized.

[0134] According to one embodiment of this application, the speech synthesis model includes an encoder, a speech feature predictor, and a decoder. When the synthesis module 1120 inputs a phoneme sequence and word feature vectors into the speech synthesis model for speech synthesis processing to obtain the speech synthesis result corresponding to the text to be synthesized, it performs the following steps:

[0135] The phoneme sequence is encoded by an encoder to obtain the first phoneme encoding feature. The word feature vector is then input into a speech feature predictor for prediction to obtain the speech prediction result. Feature extraction is performed on the speech prediction result to obtain the predicted speech feature. The first phoneme encoding feature and the predicted speech feature are fused to obtain the first prosodic fusion feature. The first prosodic fusion feature is then input into a decoder for decoding to obtain the acoustic feature. Speech synthesis is then performed based on the acoustic feature to obtain the speech synthesis result.

[0136] According to one embodiment of this application, the speech feature predictor includes a pitch predictor and a volume predictor, and the speech prediction result includes a pitch prediction result and a volume prediction result. When the synthesis module 1120 inputs the word feature vector into the speech feature predictor for prediction processing and obtains the speech prediction result, it performs the following steps:

[0137] The word feature vector is input into the pitch predictor for pitch feature prediction processing to obtain the pitch prediction result; the pitch prediction result is input into the volume predictor for volume feature prediction processing to obtain the volume prediction result.

[0138] According to an embodiment of this application, the speech synthesis model further includes a first positional encoding and a phoneme embedding layer. When the synthesis module 1120 encodes the phoneme sequence through the encoder to obtain the first phoneme encoding features, it performs the following steps:

[0139] The phoneme sequence is input into the phoneme embedding layer for feature extraction to obtain the phoneme feature vector of the text to be synthesized; the phoneme feature vector is input into the encoder for encoding to obtain the second phoneme encoding feature, and the phoneme feature vector is input into the first position encoder to obtain the first position encoding feature; the second phoneme encoding feature and the first position encoding feature are fused to obtain the first phoneme encoding feature.

[0140] According to one embodiment of this application, the speech feature predictor includes a duration predictor, and the second acquisition module 1110 is further configured to perform the following steps:

[0141] The first phoneme encoding feature is input into the duration predictor for duration prediction processing to obtain the duration prediction result;

[0142] When the synthesis module 1120 inputs the first prosodic fusion feature into the decoder for decoding processing to obtain the acoustic feature, it performs the following steps:

[0143] The length of the first prosodic fusion feature is adjusted based on the duration prediction result to obtain the second prosodic fusion feature, wherein the length of the second prosodic fusion feature is consistent with that of the acoustic feature; the second prosodic fusion feature is input into the decoder for decoding processing to obtain the acoustic feature.

[0144] It should be understood that the specific working process and functions of the second acquisition module 1110 and the synthesis module 1120 in the above embodiments can be referred to the above. Figure 8 and 9 The description of the speech synthesis method provided in the embodiments.

[0145] Figure 12 This is a block diagram of an electronic device for speech synthesis or speech synthesis model training, provided in an exemplary embodiment of this application. (Refer to...) Figure 12 The electronic device 1200 includes a processing component 1210, which further includes one or more processors, and memory resources represented by memory 1220 for storing instructions executable by the processing component 1210, such as application programs. The application programs stored in memory 1220 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1210 is configured to execute instructions to perform the aforementioned speech synthesis method or speech synthesis model training method.

[0146] Electronic device 1200 may also include a power supply component configured to perform power management of electronic device 1200, a wired or wireless network interface configured to connect electronic device 1200 to a network, and an input / output (I / O) interface. Electronic device 1200 can be operated based on an operating system stored in memory 1220, such as Windows Server. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0147] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by the processor of the aforementioned electronic device 1200, enables the electronic device 1200 to execute a training method for a speech synthesis model, comprising:

[0148] The process involves acquiring sample phoneme sequences and acoustic feature labels from training sample data. This training sample data is used to train a speech synthesis model, which includes an encoder, a reference encoder, and a decoder. The encoder encodes the sample phoneme sequences to obtain first sample phoneme encoded features. The acoustic feature labels are input into the reference encoder for feature extraction, resulting in word-level speech feature labels. These labels are then further processed to obtain speech feature label vectors. The first sample phoneme encoded features and the speech feature label vectors are fused to obtain first sample prosodic fusion features. The decoder then decodes these features to obtain sample acoustic features. Finally, a first loss function is used to calculate a first loss value between the sample acoustic features and the acoustic feature labels, and the speech synthesis model is trained based on this first loss value.

[0149] Alternatively, when the instructions in the storage medium are executed by the processor of the aforementioned electronic device 1200, the electronic device 1200 is enabled to perform a speech synthesis method, including:

[0150] Obtain the phoneme sequence and word feature vector of the text to be synthesized, wherein the word feature vector is used to reflect the semantic information and emotional information carried by the text to be synthesized; input the phoneme sequence and word feature vector into the speech synthesis model for speech synthesis processing to obtain the speech synthesis result corresponding to the text to be synthesized, wherein the speech synthesis model is obtained based on the training method described in the above embodiment.

[0151] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0152] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0153] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0157] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program verification codes, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] It should be noted that in the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0159] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications or equivalent substitutions made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for a speech synthesis model, characterized in that, include: The training sample data includes sample phoneme sequences and acoustic feature labels, wherein the training sample data is used to train a speech synthesis model, the speech synthesis model includes an encoder, a reference encoder and a decoder, and the reference encoder includes an average pooling module. The encoder is used to encode the sample phoneme sequence to obtain the first sample phoneme encoding feature; The acoustic feature labels are input into the reference encoder for feature extraction to obtain word-level speech feature labels, and the speech feature labels are further processed to obtain speech feature label vectors. The average pooling module of the reference encoder is used to convert the frame-level feature vectors into word-level feature vectors. The phoneme encoding features of the first sample and the speech feature label vector are fused to obtain the prosodic fusion features of the first sample. The first sample prosodic fusion features are decoded by the decoder to obtain the sample acoustic features; The first loss value between the sample acoustic features and the acoustic feature labels is calculated using a first loss function, and the speech synthesis model is trained based on the first loss value.

2. The training method according to claim 1, characterized in that, The reference encoder further includes a convolutional module, a feature processing module, and a fully connected layer. The speech feature labels include pitch labels and volume labels. The step of inputting the acoustic feature labels into the reference encoder for feature extraction to obtain word-level speech feature labels includes: The acoustic feature labels are input into the convolution module for convolution processing to obtain acoustic label feature vectors, and the acoustic label feature vectors are input into the feature processing module for dimensionality processing to obtain dimensionality-processed acoustic label feature vectors. The acoustic label feature vector after dimensional processing is input into the average pooling module for pooling processing to obtain the pooled acoustic label feature vector, wherein the pooled acoustic label feature vector is used to reflect the word-level feature vector of the training sample data. The pooled acoustic label feature vector is input into the fully connected layer for feature processing, and the feature vector after feature processing is split in dimension to obtain the pitch label and the volume label.

3. The training method according to claim 2, characterized in that, The speech synthesis model further includes a speech feature predictor, which comprises a pitch predictor and a volume predictor. The training method further includes: Obtain the sample word feature vector of the training sample data, wherein the sample word feature vector is used to reflect the semantic information of the training sample data and the sentiment information carried by the training sample data; The sample word feature vector is input into the pitch predictor to perform pitch prediction, and the sample pitch prediction result is obtained. The sample pitch prediction result is input into the volume predictor to perform volume prediction, and the sample volume prediction result is obtained. A second loss value is calculated between the tone label and the sample tone prediction result using a second loss function, and the tone predictor is trained based on the second loss value; The third loss function is used to calculate the third loss value between the volume label and the sample volume prediction result, and the volume predictor is trained based on the third loss value.

4. A speech synthesis method, characterized in that, include: Obtain the phoneme sequence and word feature vector of the text to be synthesized, wherein the word feature vector is used to reflect the semantic information and the emotional information carried by the text to be synthesized; The phoneme sequence and the word feature vector are input into the speech synthesis model for speech synthesis processing to obtain the speech synthesis result corresponding to the text to be synthesized, wherein the speech synthesis model is obtained based on the training method of any one of claims 1 to 3.

5. The speech synthesis method according to claim 4, characterized in that, The speech synthesis model includes an encoder, a speech feature predictor, and a decoder. The step of inputting the phoneme sequence and the word feature vector into a speech synthesis model for speech synthesis processing to obtain the speech synthesis result corresponding to the text to be synthesized includes: The phoneme sequence is encoded by the encoder to obtain the first phoneme encoding feature, and the word feature vector is input into the speech feature predictor for prediction to obtain the speech prediction result. The speech prediction result is then used to extract features to obtain the predicted speech features. The first phoneme coding feature and the predicted speech feature are fused to obtain the first prosody fusion feature; The first prosodic fusion feature is input into the decoder for decoding to obtain acoustic features, and speech synthesis is performed based on the acoustic features to obtain the speech synthesis result.

6. The speech synthesis method according to claim 5, characterized in that, The speech feature predictor includes a pitch predictor and a volume predictor, and the speech prediction result includes a pitch prediction result and a volume prediction result. The step of inputting the word feature vector into the speech feature predictor for prediction processing to obtain the speech prediction result includes: The word feature vector is input into the pitch predictor for pitch feature prediction processing to obtain the pitch prediction result. The pitch prediction result is input into the volume predictor for volume feature prediction processing to obtain the volume prediction result.

7. The speech synthesis method according to claim 5, characterized in that, The speech synthesis model also includes a first positional encoding and a phoneme embedding layer. The step of encoding the phoneme sequence using the encoder to obtain the first phoneme encoding feature includes: The phoneme sequence is input into the phoneme embedding layer for feature extraction to obtain the phoneme feature vector of the text to be synthesized; The phoneme feature vector is input into the encoder for encoding to obtain the second phoneme encoding feature, and the phoneme feature vector is input into the first position encoder to obtain the first position encoding feature; The second phoneme coding feature and the first position coding feature are fused to obtain the first phoneme coding feature.

8. The speech synthesis method according to any one of claims 5 to 7, characterized in that, The speech feature predictor includes a duration predictor. The speech synthesis method further includes: The first phoneme encoding feature is input into the duration predictor for duration prediction processing to obtain the duration prediction result; The step of inputting the first prosodic fusion feature into the decoder for decoding to obtain acoustic features includes: Based on the duration prediction result, the length of the first prosodic fusion feature is adjusted to obtain the second prosodic fusion feature, wherein the length of the second prosodic fusion feature is consistent with that of the acoustic feature. The second prosodic fusion feature is input into the decoder for decoding processing to obtain the acoustic feature.

9. A training device for a speech synthesis model, characterized in that, include: The first acquisition module is used to acquire the sample phoneme sequence and acoustic feature labels of the training sample data, wherein the training sample data is used to train the speech synthesis model, the speech synthesis model includes an encoder, a reference encoder and a decoder, and the reference encoder includes an average pooling module. The encoding processing module is used to encode the sample phoneme sequence through the encoder to obtain the first sample phoneme encoding features; The feature extraction module is used to input the acoustic feature labels into the reference encoder for feature extraction to obtain word-level speech feature labels, and to extract features from the speech feature labels to obtain speech feature label vectors. The average pooling module of the reference encoder is used to convert the frame-level feature vectors into word-level feature vectors. The feature fusion module is used to fuse the phoneme encoding features of the first sample and the speech feature label vector to obtain the prosodic fusion features of the first sample. The decoding processing module is used to decode the prosodic fusion features of the first sample through the decoder to obtain the acoustic features of the sample; The parameter adjustment module is used to calculate a first loss value between the sample acoustic features and the acoustic feature labels using a first loss function, and to train the speech synthesis model based on the first loss value.

10. A speech synthesis device, characterized in that, include: The second acquisition module is used to acquire the phoneme sequence and word feature vector of the text to be synthesized, wherein the word feature vector is used to reflect the semantic information of the text to be synthesized and the emotional information carried by the text to be synthesized; The synthesis module is used to input the phoneme sequence and the word feature vector into the speech synthesis model for speech synthesis processing to obtain the speech synthesis result corresponding to the text to be synthesized, wherein the speech synthesis model is obtained based on the training method of any one of claims 1 to 3.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when loaded by a processor, is used to execute the training method of the speech synthesis model according to any one of claims 1 to 3, or to execute the speech synthesis method according to any one of claims 4 to 8.

12. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions. The processor is used to execute the training method of the speech synthesis model according to any one of claims 1 to 3, or to execute the speech synthesis method according to any one of claims 4 to 8.

Images