Speech synthesis method and device, electronic equipment and readable storage medium

Abstract

The application provides a speech synthesis method and device, electronic equipment and readable storage medium. The method comprises: obtaining the emotional intensity of each audio segment of the text to be synthesized, each audio segment corresponding to one syllable of the text to be synthesized, and the emotional intensity being phoneme-level emotional intensity; obtaining the mel-frequency spectrum corresponding to the text to be synthesized according to the emotional intensity and an acoustic model; and generating speech based on the mel-frequency spectrum. In the embodiment of the application, the emotional intensity of each audio segment of the text to be synthesized is obtained, and the acoustic model obtains the mel-frequency spectrum corresponding to the text to be synthesized based on the emotional intensity; and speech is generated based on the mel-frequency spectrum. The audio segment is the audio segment corresponding to each syllable, the obtained emotional intensity is phoneme-level emotional intensity, fine-grained emotion-controllable speech synthesis is realized, and the synthesized speech has more accurate emotion.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech synthesis method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Speech synthesis, also known as text-to-speech (TTS) technology, is a technique that converts text into speech. Emotion is the intrinsic emotional expression of a speaker and is an indispensable part of human-computer interaction. The expression of emotion includes both the category and intensity of emotion. Emotional speech synthesis aims to synthesize speech with different categories and intensities of emotion. Current mainstream emotional speech synthesis typically relies on two approaches: explicitly learning emotional information through labels and implicitly learning emotional information through reference audio. Current research on controllable emotion mostly focuses on sentence-level emotional intensity control, failing to finely control sentence details, resulting in inaccurate emotional expression in the synthesized speech. Summary of the Invention

[0003] The purpose of this application is to provide a speech synthesis method, apparatus, electronic device, and readable storage medium that solves the problem of inaccurate emotion in speech synthesized by existing speech synthesis methods.

[0004] To achieve the above objectives, embodiments of this application provide a speech synthesis method, the method comprising:

[0005] The emotional intensity of each audio segment of the text to be synthesized is obtained, where each audio segment corresponds to a syllable of the text to be synthesized, and the emotional intensity is at the phoneme level.

[0006] Based on the emotional intensity and the acoustic model, the Mel spectrum corresponding to the text to be synthesized is obtained;

[0007] Speech is generated based on the Mel spectrum.

[0008] Optionally, obtaining the emotional intensity of each audio segment of the text to be synthesized includes:

[0009] Extract the duration information corresponding to each syllable in the text to be synthesized;

[0010] Based on the duration information, audio segments are extracted from the text to be synthesized to obtain audio segments corresponding to syllables;

[0011] The emotional intensity at the syllable level for each audio segment is obtained using a sorting function;

[0012] Align the syllable-level emotional intensity with the syllable-level text to be synthesized to obtain the text to be synthesized with syllable markers;

[0013] Based on the syllable markers of the text to be synthesized, an upsampling process from syllable level to phoneme level is performed to obtain the phoneme-level emotional intensity of the audio segment.

[0014] Optionally, obtaining the syllable-level emotional intensity of each audio segment using a sorting function includes:

[0015] The openSMILE feature extractor was used to extract the openSMILE features of the audio segment;

[0016] Based on the openSMILE features, the syllable-level emotional intensity of each audio segment is calculated using a sorting function.

[0017] Optionally, the method further includes:

[0018] A ranking function is trained using training data, which includes emotional audio data and neutral audio data; the training process of the ranking function includes:

[0019] Two audio data points are randomly selected from the training data;

[0020] The OpenSMILE features of the audio data are extracted using the OpenSMILE feature extractor.

[0021] The openSMILE features are weighted and summed using the weight values ​​of the ranking function to obtain the sentence-level sentiment intensity;

[0022] The ranking function is iteratively optimized based on the sentence-level sentiment intensity.

[0023] Optionally, the acoustic model includes: a text encoder, an emotion encoder, a prosody adapter, and a decoder;

[0024] The step of obtaining the Mel spectrum corresponding to the text to be synthesized based on the emotional intensity and the acoustic model includes:

[0025] The phoneme-level text corresponding to the text to be synthesized is input into the text encoder to obtain the encoded first text representation vector;

[0026] The first emotion representation vector is obtained through the emotion encoder and based on the emotion intensity;

[0027] Add the first text representation vector and the first sentiment representation vector to obtain the first text sentiment representation vector;

[0028] Input the first text sentiment representation vector into the prosody adapter and obtain the output of the prosody adapter;

[0029] The output of the prosody adapter is input into the decoder to obtain the predicted Mel spectrum.

[0030] Optionally, the phoneme-level text corresponding to the text to be synthesized is input into the text encoder to obtain the encoded first text representation vector, including:

[0031] The phoneme-level text corresponding to the text to be synthesized is input into the text encoder to extract the language information of the phoneme-level text; the text encoder is used to: encode and output the first text representation vector according to the language information;

[0032] The language information includes: the number of audio segments, the number of phonemes, and the feature dimensions.

[0033] Optionally, obtaining a first emotion representation vector through the emotion encoder and based on the emotion intensity includes:

[0034] The neutral label and the sentiment label are input into the sentiment encoder to obtain a neutral vector and a sentiment vector; the sentiment encoder is used to: use the sentiment intensity to perform a weighted summation on the neutral vector and the sentiment vector to obtain and output a first sentiment representation vector.

[0035] Optionally, the first text sentiment representation vector is input into the prosodic adapter to obtain the output of the prosodic adapter, including:

[0036] The first text sentiment representation vector is input into the prosodic adapter, which is used to: obtain the duration of each phoneme and expand the text sentiment representation vector using the duration to obtain a frame-level first text sentiment representation vector.

[0037] The frame-level first text sentiment representation vector is input into the voiced / unvoiced sound predictor to generate voiced / unvoiced sound labels; the prosodic adapter is further used to add the voiced / unvoiced sound labels and the frame-level first text sentiment representation vector as the output of the prosodic adapter.

[0038] Optionally, the acoustic model includes: a text encoder, an emotion encoder, a prosody adapter, and a decoder;

[0039] The method further includes: training the acoustic model using training data;

[0040] The training process of the acoustic model includes:

[0041] The phoneme-level text corresponding to the training data is input into the text encoder to obtain the encoded second text representation vector.

[0042] A second emotion representation vector is obtained through the emotion encoder and based on the emotion intensity of the training data;

[0043] The second text representation vector and the second sentiment representation vector are added together to obtain the second text sentiment representation vector;

[0044] Input the second text sentiment representation vector into the prosodic adapter to obtain the output of the prosodic adapter;

[0045] The output of the prosody adapter is input into the decoder to obtain the predicted Mel spectrum;

[0046] Calculate the mean square error between the predicted Mel spectrum and the actual Mel spectrum.

[0047] The acoustic model is iteratively optimized based on the mean square error.

[0048] Optionally, the phoneme-level text corresponding to the training data is input into the text encoder to obtain the encoded second text representation vector, including:

[0049] The phoneme-level text corresponding to the training data is input into the text encoder to extract the language information of the phoneme-level text; the text encoder is used to: encode and output the second text representation vector based on the language information;

[0050] The language information includes: the number of audio segments, the number of phonemes, and the feature dimensions.

[0051] Optionally, obtaining a second sentiment representation vector through the sentiment encoder and based on the sentiment intensity of the training data includes:

[0052] Neutral labels and sentiment labels are input into the sentiment encoder to obtain neutral vectors and sentiment vectors;

[0053] The emotion encoder is used to: use the emotion intensity of the training data to perform a weighted summation of the neutral vector and the emotion vector to obtain and output a second emotion representation vector.

[0054] Optionally, the second text sentiment representation vector is input into the prosodic adapter to obtain the output of the prosodic adapter, including:

[0055] The second text sentiment representation vector is input into the prosodic adapter, which is used to: obtain the duration of each phoneme and expand the text sentiment representation vector using the duration to obtain a frame-level second text sentiment representation vector.

[0056] The frame-level second text sentiment representation vector is input into the voiced / unvoiced sound predictor to generate voiced / unvoiced sound labels; the prosodic adapter is further used to add the voiced / unvoiced sound labels and the frame-level second text sentiment representation vector as the output of the prosodic adapter.

[0057] Optionally, the training process of the acoustic model further includes:

[0058] The second text representation vector is input into the sentiment classifier to obtain the sentiment category of the training data;

[0059] Calculate the cross-entropy loss based on the sentiment category and the true sentiment label;

[0060] The gradient backpropagation is performed using the cross-entropy loss to remove the implicit sentiment information in the second text representation vector.

[0061] Optionally, the training process of the acoustic model further includes:

[0062] The second text sentiment representation vector is input into the tone predictor to obtain the tone of each phoneme;

[0063] Calculate the cross-entropy loss based on the tone and the actual tone label.

[0064] Optionally, after obtaining the predicted Mel spectrum, the training process of the acoustic model further includes:

[0065] The generator is frozen and the predicted Mel spectrum and the real Mel spectrum are input into the discriminator to determine the probability of the predicted Mel spectrum being true or false and the first label of the real Mel spectrum being true or false.

[0066] Calculate the mean square error between the first label and the true / false labels;

[0067] The discriminator is optimized based on the mean square error between the first label and the real / fake labels.

[0068] Optionally, the training process of the acoustic model further includes:

[0069] The generator is unfrozen and the predicted Mel spectrum and the real Mel spectrum are input into the discriminator to determine the probability of the predicted Mel spectrum being true or false and the second label of the real Mel spectrum being true or false.

[0070] Calculate the mean square error between the second label and the true / false labels;

[0071] The generator is optimized based on the mean squared error between the second label and the true / false labels.

[0072] To achieve the above objectives, embodiments of this application provide a speech synthesis apparatus, comprising:

[0073] The first acquisition module is used to acquire the emotional intensity of each audio segment of the text to be synthesized, wherein each audio segment corresponds to a syllable of the text to be synthesized, and the emotional intensity is a phoneme-level emotional intensity.

[0074] The second acquisition module is used to obtain the Mel spectrum corresponding to the text to be synthesized based on the emotional intensity and the acoustic model.

[0075] A speech generation module is used to generate speech based on the Mel spectrum.

[0076] To achieve the above objectives, embodiments of this application provide an electronic device, including a processor and a transceiver;

[0077] The processor is configured to: obtain the emotional intensity of each audio segment of the text to be synthesized, wherein each audio segment corresponds to a syllable of the text to be synthesized, and the emotional intensity is a phoneme-level emotional intensity;

[0078] Based on the emotional intensity and the acoustic model, the Mel spectrum corresponding to the text to be synthesized is obtained;

[0079] Speech is generated based on the Mel spectrum.

[0080] To achieve the above objectives, embodiments of this application provide a readable storage medium storing a program or instructions thereon, which, when executed by a processor, implement the steps of the above-described speech synthesis method.

[0081] The beneficial effects of the above technical solution in this application are as follows:

[0082] In embodiments of this application, the emotional intensity of each audio segment of the text to be synthesized is obtained, and the acoustic model obtains the Mel spectrum corresponding to the text to be synthesized based on the emotional intensity; speech is generated based on the Mel spectrum. The audio frequency bands are the audio segments corresponding to each syllable, and the obtained emotional intensity is at the phoneme level, achieving fine-grained, emotion-controllable speech synthesis, resulting in more accurate emotional expression in the synthesized speech. Attached Figure Description

[0083] Figure 1 This is a schematic flowchart of the speech synthesis method according to an embodiment of this application;

[0084] Figure 2 This is a schematic diagram illustrating the reasoning process of the acoustic model in an embodiment of this application;

[0085] Figure 3 This is a schematic diagram of the reasoning process of the prosody adapter in an embodiment of this application;

[0086] Figure 4This is a schematic diagram illustrating the training process of the sorting function in an embodiment of this application;

[0087] Figure 5 This is a schematic diagram illustrating the process of generating emotion intensity according to an embodiment of this application;

[0088] Figure 6 This is one of the schematic diagrams illustrating the training process of the acoustic model in an embodiment of this application;

[0089] Figure 7 This is a schematic diagram illustrating the training process of the prosody adapter according to an embodiment of this application;

[0090] Figure 8 This is a second schematic diagram illustrating the training process of the acoustic model in an embodiment of this application.

[0091] Figure 9 This is a schematic diagram of the speech synthesis device according to an embodiment of this application;

[0092] Figure 10 This is one of the structural schematic diagrams of the electronic device according to an embodiment of this application;

[0093] Figure 11 This is a second schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0094] To make the technical problems, technical solutions and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0095] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0096] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0097] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0098] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0099] like Figure 1 As shown, this application provides a speech synthesis method, including:

[0100] Step 101: Obtain the emotional intensity of each audio segment of the text to be synthesized. Each audio segment corresponds to a syllable of the text to be synthesized, and the emotional intensity is a phoneme-level emotional intensity.

[0101] In this embodiment, the text to be synthesized is text that needs to be converted into speech. The text to be synthesized is divided into multiple audio segments according to syllables, with each syllable corresponding to one audio segment. The phoneme-level emotional intensity of each audio segment is calculated. Here, a phoneme is the smallest unit of speech determined based on the natural properties of speech, i.e., the emotional intensity of the smallest unit of speech obtained in this application.

[0102] Step 102: Based on the emotional intensity and the acoustic model, obtain the Mel spectrum corresponding to the text to be synthesized;

[0103] Step 103: Generate speech based on the Mel spectrum.

[0104] In this embodiment, the acoustic model is a trained model for speech synthesis. The acoustic model can obtain the Mel spectrum of the text to be synthesized based on the emotion intensity, thereby enabling speech synthesis based on the Mel spectrum.

[0105] In embodiments of this application, the emotional intensity of each audio segment of the text to be synthesized is obtained, and the acoustic model obtains the Mel spectrum corresponding to the text to be synthesized based on the emotional intensity; speech is generated based on the Mel spectrum. The audio frequency bands are the audio segments corresponding to each syllable, and the obtained emotional intensity is at the phoneme level, achieving fine-grained, emotion-controllable speech synthesis, resulting in more accurate emotional expression in the synthesized speech.

[0106] As an optional embodiment, obtaining the emotional intensity of each audio segment of the text to be synthesized includes:

[0107] Step A: Extract the duration information corresponding to each syllable in the text to be synthesized;

[0108] Step B: Extract audio from the text to be synthesized based on the duration information to obtain audio segments corresponding to syllables;

[0109] In this embodiment, an alignment model based on a Hidden Markov Model (HMM) can be used to obtain the duration information corresponding to each syllable in the text to be synthesized. The duration information is then used to truncate the audio to obtain the audio corresponding to each syllable. Specifically, an HMM-based model can be trained using audio-syllable-level text pairs from the training data to align syllables and audio segments. The training data includes audio and its corresponding text, forming an audio-syllable-level text pair.

[0110] Step C: Use a sorting function to obtain the syllable-level emotional intensity of each audio segment.

[0111] Optionally, obtaining the syllable-level emotional intensity of each audio segment using a sorting function includes:

[0112] The openSMILE feature extractor is used to extract openSMILE features from the audio segments; based on the openSMILE features, a ranking function is used to calculate the syllable-level emotional intensity of each audio segment.

[0113] In this embodiment, the openSMILE feature extractor is used to extract the openSMILE features of each audio segment, which can be 384-dimensional features. A trained sorting function is used, with the openSMILE features of the audio segment as input, to output the emotional intensity value of each audio segment. This process outputs the emotional intensity at the syllable level, that is, the emotional intensity of the audio segment corresponding to each syllable.

[0114] Optionally, the emotional intensity value output by the sorting function is a continuous value between [0, 1].

[0115] Step D: Align the syllable-level emotional intensity with the syllable-level text to be synthesized to obtain the text to be synthesized with syllable markers;

[0116] Step E: Based on the syllable markers of the text to be synthesized, perform an upsampling process from syllable level to phoneme level to obtain the phoneme-level emotional intensity of the audio segment.

[0117] In this embodiment, the syllable-level emotional intensity is first aligned with the syllable-level text to make the syllable-level emotional intensity correspond to the text. Then, based on the syllable markers in the text, the upsampling process from syllable-level to phoneme-level is completed, and finally, the phoneme-level emotional intensity is obtained.

[0118] Optionally, the method further includes:

[0119] A ranking function is trained using training data, which includes emotional audio data and neutral audio data; the training process of the ranking function includes:

[0120] Two audio data points are randomly selected from the training data;

[0121] The OpenSMILE features of the audio data are extracted using the OpenSMILE feature extractor.

[0122] The openSMILE features are weighted and summed using the weight values ​​of the ranking function to obtain the sentence-level sentiment intensity;

[0123] The ranking function is iteratively optimized based on the sentence-level sentiment intensity.

[0124] In this embodiment, a ranking function based on relative attributes is trained for each emotion. Specifically, OpenSMILE features of neutral and emotional audio are first extracted from all training data. The predictor inputs two random features each time and sums them with weights to obtain two ranking scores, which are the emotion intensity values. After obtaining the emotion intensity scores, it is determined whether to continue the above training steps based on the scores, until the output ranking function meets certain conditions. For example, the predictor uses Newton's method for optimization to ensure that scores for the same emotion category are the same, and that the emotion intensity score of emotional audio is greater than that of neutral audio. After training and obtaining the ranking function, it can be used to obtain the emotion intensity score of the input text to be synthesized during actual inference.

[0125] As an optional embodiment, the acoustic model includes: a text encoder, an emotion encoder, a prosodic adapter, and a decoder; the acoustic model is as follows: Figure 2 As shown.

[0126] The step of obtaining the Mel spectrum corresponding to the text to be synthesized based on the emotional intensity and the acoustic model includes:

[0127] Step F: Input the phoneme-level text corresponding to the text to be synthesized into the text encoder to obtain the encoded first text representation vector.

[0128] Optionally, the phoneme-level text corresponding to the text to be synthesized is input into the text encoder to obtain the encoded first text representation vector, including:

[0129] The phoneme-level text corresponding to the text to be synthesized is input into the text encoder to extract the language information of the phoneme-level text; the text encoder is used to: encode and output the first text representation vector according to the language information; wherein, the language information includes: the number of audio segments, the number of phonemes, and the feature dimension.

[0130] like Figure 2As shown, the phoneme-level text corresponding to the text to be synthesized is input into the text encoder to obtain the first text representation vector. The text encoder extracts the linguistic information from the phoneme-level text and encodes the first text representation vector, whose dimension can be denoted as B*T1*D1, where B is the number of audio segments, T1 is the number of phonemes, and D1 is its feature dimension. Optionally, the feature dimension can be 256 or 384, and T1 is approximately 100.

[0131] Step G: Obtain the first emotion representation vector through the emotion encoder and based on the emotion intensity.

[0132] Optionally, obtaining a first emotion representation vector through the emotion encoder and based on the emotion intensity includes:

[0133] The neutral label and the sentiment label are input into the sentiment encoder to obtain a neutral vector and a sentiment vector; the sentiment encoder is used to: use the sentiment intensity to perform a weighted summation on the neutral vector and the sentiment vector to obtain and output a first sentiment representation vector.

[0134] In this embodiment, a neutral label and a specified sentiment label are input into the sentiment encoder to obtain a neutral embedding (i.e., a neutral vector) and a sentiment embedding (i.e., a sentiment vector). The sentiment label can be the sentiment type label corresponding to the audio. Sentiment embeddings and neutral embeddings are obtained by embedding the sentiment label and neutral label of the audio respectively, and the dimension of each embedding can be B*T1*D1. The sentiment encoder then performs a weighted summation of the sentiment vector and the neutral vector using the sentiment intensity obtained in the preceding steps and "1-sentiment intensity" as weights to obtain the final first sentiment representation vector.

[0135] Step H: Add the first text representation vector and the first sentiment representation vector to obtain the first text sentiment representation vector.

[0136] This step adds the first text representation vector output by the text encoder to the first sentiment representation vector output by the sentiment encoder to obtain the first text sentiment representation vector with dimension B*T1*D1.

[0137] Step 1: Input the first text sentiment representation vector into the prosodic adapter and obtain the output of the prosodic adapter.

[0138] Optionally, the first text sentiment representation vector is input into the prosodic adapter to obtain the output of the prosodic adapter, including:

[0139] The first text sentiment representation vector is input into the prosodic adapter, which is used to: obtain the duration of each phoneme and expand the text sentiment representation vector using the duration to obtain a frame-level first text sentiment representation vector.

[0140] The frame-level first text sentiment representation vector is input into the voiced / unvoiced sound predictor to generate voiced / unvoiced sound labels; the prosodic adapter is further used to add the voiced / unvoiced sound labels and the frame-level first text sentiment representation vector as the output of the prosodic adapter.

[0141] This step inputs the first text sentiment representation vector into the prosodic adapter. The prosodic adapter is as follows: Figure 3 As shown, the duration of each phoneme is predicted by the duration predictor using the first text sentiment representation vector as input; the duration warper expands the first text sentiment representation vector at the phoneme level using the predicted duration to obtain the first text sentiment representation vector at the frame level; the first text sentiment representation vector at the frame level is input into the voiced / unvoiced sound predictor to generate voiced / unvoiced sound tags, which are added to the first text sentiment representation vector at the frame level and output as the prosodic adapter.

[0142] Step J: Input the output of the prosody adapter into the decoder to obtain the predicted Mel spectrum. The decoder receives the output of the prosody adapter as input, predicts the Mel spectrum, and can then synthesize speech based on the Mel spectrum.

[0143] It should be noted that the acoustic model is a pre-trained model for synthesizing speech. The training process of the acoustic model is described below.

[0144] The acoustic model includes: a text encoder, an emotion encoder, a prosodic adapter, and a decoder;

[0145] Optionally, the method further includes: training the acoustic model using training data;

[0146] The training process of the acoustic model includes:

[0147] Step a: Input the phoneme-level text corresponding to the training data into the text encoder to obtain the encoded second text representation vector.

[0148] Optionally, the phoneme-level text corresponding to the training data is input into the text encoder to obtain the encoded second text representation vector, including:

[0149] The phoneme-level text corresponding to the training data is input into the text encoder to extract the language information of the phoneme-level text; the text encoder is used to: encode and output the second text representation vector according to the language information; wherein, the language information includes: the number of audio segments, the number of phonemes, and the feature dimension.

[0150] Step b: Obtain the second emotion representation vector through the emotion encoder and based on the emotion intensity of the training data.

[0151] Optionally, obtaining a second sentiment representation vector through the sentiment encoder and based on the sentiment intensity of the training data includes:

[0152] The emotion encoder is input with neutral labels and emotion labels to obtain neutral vectors and emotion vectors; the emotion encoder is used to: use the emotion intensity of the training data to perform a weighted summation of the neutral vectors and the emotion vectors to obtain and output a second emotion representation vector.

[0153] Step c: Add the second text representation vector and the second sentiment representation vector to obtain the second text sentiment representation vector;

[0154] Step d: Input the second text sentiment representation vector into the prosodic adapter and obtain the output of the prosodic adapter;

[0155] Optionally, the second text sentiment representation vector is input into the prosodic adapter to obtain the output of the prosodic adapter, including:

[0156] The second text sentiment representation vector is input into the prosodic adapter, which is used to: obtain the duration of each phoneme and expand the text sentiment representation vector using the duration to obtain a frame-level second text sentiment representation vector.

[0157] The frame-level second text sentiment representation vector is input into the voiced / unvoiced sound predictor to generate voiced / unvoiced sound labels; the prosodic adapter is further used to add the voiced / unvoiced sound labels and the frame-level second text sentiment representation vector as the output of the prosodic adapter.

[0158] Step e: Input the output of the prosody adapter into the decoder to obtain the predicted Mel spectrum;

[0159] Step f: Calculate the mean square error between the predicted Mel spectrum and the true Mel spectrum.

[0160] Step g: Iteratively optimize the acoustic model based on the mean square error.

[0161] In this embodiment, during the training process of the acoustic model, after obtaining the predicted Mel spectrum, the mean square error (MSE) between the predicted Mel spectrum and the actual Mel spectrum of the training data is calculated. Iterative optimization is then performed based on the MSE until the MSE satisfies the iteration condition. Using the MSE of the Mel spectrum to constrain the decoder during model training ensures the quality of the synthesized spectrum.

[0162] Optionally, the training process of the acoustic model further includes:

[0163] The second text representation vector is input into the sentiment classifier to obtain the sentiment category of the training data; cross-entropy loss is calculated based on the sentiment category and the true sentiment label; gradient backpropagation is performed using the cross-entropy loss to remove the implicit sentiment information in the second text representation vector.

[0164] In this embodiment, the training process of the acoustic model also includes optimizing the output data of the text encoder. After inputting the text feature vector output by the text encoder into the sentiment classifier to obtain the sentiment category, the cross-entropy loss between the sentiment category and the true sentiment type of the training data is calculated and gradient inversion is performed to remove the implicit sentiment information in the text representation vector.

[0165] Optionally, the training process of the acoustic model further includes:

[0166] The second text sentiment representation vector is input into the tone predictor to obtain the tone of each phoneme; cross-entropy loss is calculated based on the tone and the real tone label.

[0167] In this embodiment, the training process of the acoustic model further includes optimizing the text sentiment representation vector obtained by adding the text feature vector and the sentiment feature vector. The text sentiment representation vector is input into the tone predictor to predict the tone of each phoneme in the training data. The cross-entropy loss between the predicted tone and the true tone of the training data is calculated. The accuracy of the text sentiment representation vector is judged based on the cross-entropy loss, thereby enhancing the tone information in the hidden layer representation and preventing tone distortion caused by over-interpretation of emotions.

[0168] In this embodiment, the total loss function for the training process of the acoustic model includes: the mean squared error of the Mel spectrum, the cross-entropy loss of the gradient inversion emotion classification, the cross-entropy loss of the tone prediction, the weighted sum of the mean squared errors of the prosodic adapter fundamental frequency and duration.

[0169] Optionally, after obtaining the predicted Mel spectrum, the training process of the acoustic model further includes:

[0170] The generator is frozen and the predicted Mel spectrum and the real Mel spectrum are input into the discriminator to determine the probability of the predicted Mel spectrum being true or false and the first label of the real Mel spectrum being true or false.

[0171] Calculate the mean square error between the first label and the true / false labels;

[0172] The discriminator is optimized based on the mean square error between the first label and the real / fake labels.

[0173] Optionally, the training process of the acoustic model further includes:

[0174] The generator is unfrozen and the predicted Mel spectrum and the real Mel spectrum are input into the discriminator to determine the probability of the predicted Mel spectrum being true or false and the second label of the real Mel spectrum being true or false.

[0175] Calculate the mean square error between the second label and the true / false labels;

[0176] The generator is optimized based on the mean squared error between the second label and the true / false labels.

[0177] In this embodiment, a discriminator is added to determine the probability of the predicted Mel spectrum being true or false. First, the generator is frozen, and the real and predicted Mel spectra are used as input to the discriminator. The discriminator determines whether each spectrum is true or false, obtaining the probability of true or false. This probability is then compared with the true / false labels of the training data to calculate the mean squared error, optimizing the discriminator and ensuring its ability to distinguish between true and false signals. Next, the discriminator's parameters are frozen, and the generator is unfrozen. Again, the real and predicted Mel spectra are used as input to the discriminator, determining whether each spectrum is true or false, obtaining the probability of true or false signals, and comparing this probability with the true / false labels of the training data to calculate the mean squared error loss, optimizing the generator and improving its generation quality.

[0178] In this embodiment, the total loss function for the training process of the acoustic model is a weighted sum of the discriminator mean square error loss, which is added to the mean square error of the Mel spectrum mean square error, the gradient inversion emotion classification cross-entropy loss, the tone prediction cross-entropy loss, the prosody adapter fundamental frequency and duration mean square error.

[0179] After obtaining the trained acoustic model, the synthesized speech can be obtained by using the model and predicting the Mel spectrum based on the emotional intensity of the audio segments of the text to be synthesized.

[0180] This application embodiment specifically includes eight main execution modules: a sentiment intensity ranking function, a text encoder for the sentiment ranking model, a sentiment classifier, a sentiment encoder, a tone predictor, a prosody adapter, a decoder, and a discriminator. The sentiment intensity ranking function is modeled at the sentence level during the training phase, at the syllable level during the inference phase, and then upsampled to the phoneme level. The text encoder and sentiment encoder are modeled at the phoneme level, while the decoder and discriminator are modeled at the frame level.

[0181] The text encoder and decoder can be configured with network structures such as Transformer and Conformer. The emotion classifier and tone predictor can be configured with network structures such as feedforward neural networks, one-dimensional convolution, and bidirectional long short-term memory (BLSTM). The fundamental frequency predictor, duration predictor, and voiced / unvoiced predictor in the prosody adapter can be configured with network structures such as one-dimensional convolution and BLSTM. The multi-scale temporal discriminator and molecular band frequency domain discriminator can be configured with network structures such as feedforward neural networks and two-dimensional convolution.

[0182] The model training process before executing the actual inference process in this application includes four stages, which are described below through specific embodiments.

[0183] Phase 1, such as Figure 4 As shown, in the first stage: a) First, all audio files, including emotional and neutral audio, are used to form a training set. Two audio files from the training set are randomly selected to form a training audio pair (i, j), which is then input into the OpenSMILE feature extractor. Then, the OpenSMILE feature extraction algorithm is used to extract OpenSMILE features from the randomly selected training audio pairs; b) The OpenSMILE features of each audio file are weighted and summed with the 384 weight values ​​in the ranking function to obtain the sentence-level sentiment intensity value for each audio file. The OpenSMILE features used here are 384-dimensional sentence-level features. Newton's method is used for each training step of the ranking function, and the formula for optimizing the weight values ​​of the ranking function is as follows:

[0184]

[0185]

[0186]

[0187] ξ ij ≥0; γ ij ≥0

[0188] Where f is the openSMILE feature extraction process, w is the weight value in the ranking function, and C is used to balance the marginal ξ. ij and slack variable γ ij The size of the dataset is given by E, where E represents the sentiment dataset and N represents the neutral dataset; i and j represent the two input training audio files.

[0189] The second stage, such as Figure 5As shown, in the second stage: a) First, using the audio-syllable-level text pairs of all training data, a model based on HMM is trained to complete the alignment, obtaining the duration of each syllable (i.e., the duration information of each syllable). Then, based on the alignment results, the original training audio is segmented into sentences, resulting in audio segments of one sentence per syllable, thus obtaining syllable-level audio segments; b) Similar to the first stage, the openSMILE feature extractor is used to extract 384-dimensional features for each audio segment; c) Using the ranking function trained in the first stage, the openSMILE features of the audio segments are used as input to output the syllable-level sentiment intensity value of each audio segment; d) The syllable-level sentiment intensity is first aligned with the syllable-level text, and then, based on the syllable markers in the text, the upsampling process from syllable-level to phoneme-level is completed, finally obtaining the phoneme-level sentiment intensity.

[0190] For the third and fourth stages, such as Figure 6 , Figure 7 as well as Figure 8 As shown.

[0191] It should be noted that in this application example:

[0192] 1. The text encoder and decoder are both based on the Comformer architecture;

[0193] 2. Feedforward neural networks are chosen for both the sentiment classifier and the tone predictor;

[0194] 3. The fundamental frequency predictor, duration predictor, and voiced / unvoiced tone predictor are selected using one-dimensional convolutional structures;

[0195] 4. Two-dimensional convolutional structures are selected for the multi-scale time-domain discriminator and the molecular band frequency-domain discriminator.

[0196] Furthermore, the text in the model is at the phoneme level, with each phoneme corresponding to an emotional intensity value generated in the second stage and a tone value obtained by mapping characters to pinyin; neutral and emotional labels are manually labeled labels that must be prepared in the training dataset; the true Mel spectrum is obtained from real audio through framing, windowing, short-time discrete Fourier transform, and scaling; the true duration is obtained through the HMM-based alignment model trained in the second stage; the extraction of the true fundamental frequency at the sentence level is first performed by extracting the frame-level true fundamental frequency from the Mel spectrum through the DIO fundamental frequency extractor, then performing sentence-level normalization, and then averaging the true duration to the phoneme level; the true voiced / unvoiced sound label is calculated by judging whether the frame-level true fundamental frequency is 0, with a label of 0 when the fundamental frequency is 0 and a label of 1 when the fundamental frequency is not 0.

[0197] Specifically, the third stage, such as Figure 6 and Figure 7 As shown, the third phase of training includes:

[0198] Step 61: The text features are fed into a text encoder to extract language information and encode a three-dimensional tensor, denoted as B*T1*D1, where B is the number of audio samples, T1 is the number of phonemes, and D1 is its feature dimension. Typically, the feature dimension is 256 or 384, and T1 is usually around one hundred.

[0199] Step 62: The text representation vector is input into the sentiment classifier to obtain a prediction score of dimension B*T1*C1, where C1 is the number of sentiment categories, which is set to 7 here (including six sentiment categories and neutral). The predicted score and the true sentiment label are used to calculate the cross-entropy loss. During gradient backpropagation, the sentiment classifier calculates the gradient value normally to ensure that the sentiment classifier plays its role in predicting the sentiment category from the text representation vector. When the gradient propagates to the text encoder, the gradient value is multiplied by -1, causing the text encoder to optimize in the direction where it cannot encode sentiment category information. This achieves adversarial training between the text encoder and the sentiment classifier in terms of sentiment categories.

[0200] Steps 63 and 64: Obtain the sentiment embedding and neutral embedding from the sentiment labels and neutral sentiment labels of the audio samples respectively using the embedding method. The dimension of the embedding is B*T1*D1.

[0201] Step 65: Weight the emotional embedding and the neutral embedding with emotional intensity and "1-emotional intensity" as weights to obtain the final emotional representation vector.

[0202] Step 66: Add the text representation vector output by the text encoder to the sentiment representation vector output by the sentiment encoder to obtain a text sentiment representation vector of dimension B*T1*D1. This text sentiment representation vector is first input into the tone predictor to obtain a prediction score of dimension B*T1*C2, where C2 is the number of tone categories, which is set to 5 here (the four tones of pinyin plus the neutral tone). The prediction score and the true tone label are used to calculate the cross-entropy loss to ensure that the text and sentiment representation vectors contain the correct tone information.

[0203] Step 67: Input the text sentiment feature vector into the prosody adapter.

[0204] Step 68: The decoder uses the output of the prosody adapter as input to predict the Mel spectrum for each frame, obtaining a Mel spectrum with dimension B*T2*80. The mean square error of this spectrum is calculated with the true Mel spectrum to ensure the quality of the generated Mel spectrum.

[0205] The specific process of inputting the text sentiment feature vector into the prosody adapter in step 67 is as follows: Figure 7 As shown, it includes:

[0206] Step 71: First, the fundamental frequency prediction value with dimension B*T1 is obtained through the fundamental frequency predictor. The mean square error of this prediction value and the phoneme-level true fundamental frequency normalized at the sentence level is calculated. This ensures that the text sentiment vector contains fundamental frequency information. The true fundamental frequency is not added to the text sentiment representation vector here because the true fundamental frequency will conflict with the sentiment information and reduce the naturalness of the generated speech. However, if the supervision of the fundamental frequency prediction loss is not increased, the quality of the generated Mel spectrum will drop sharply.

[0207] Step 72: The duration predictor takes the text sentiment vector as input, predicts the duration of each phoneme in frames, and outputs phoneme-level duration information with a dimension of B*T1. The mean squared error loss is calculated between the prediction result and the actual duration.

[0208] Step 73: The duration warper expands the text sentiment vector according to the actual duration value to obtain a frame-level text sentiment representation with dimensions B*T2*D1, where T2 is the number of frames, which generally ranges from tens to more than a thousand.

[0209] Step 74: The frame-level text sentiment feature vector is processed by the voiced / unvoiced predictor to obtain voiced / unvoiced prediction values ​​with a dimension of B*T2. The cross-entropy loss is calculated between the predicted values ​​and the real voiced / unvoiced labels. The real voiced / unvoiced label values ​​are then added to the frame-level text sentiment feature vector to obtain the output of the prosodic adapter.

[0210] In the fourth stage of training, the text encoder, emotion encoder, emotion classifier, tone predictor, prosody adapter, and decoder are trained in the same way as in the third stage, except that an adversarial network is added to the training. For example... Figure 8 As shown, steps 81 to 88 are related to the third stage (i.e. Figure 6 The same content as shown will not be repeated here. The following section explains the differences between this stage and the third stage.

[0211] Specifically, such as Figure 8 As shown, the actual Mel spectrum and the Mel spectrum predicted by the decoder are input together into the molecular band frequency domain discriminator and the multi-scale time domain discriminator.

[0212] Step 89: The molecular band frequency domain discriminator divides the 80-dimensional Mel spectrum into multiple sub-graphs according to the frequency domain. These sub-graphs are input into an equal number of identically structured frequency domain discriminators to determine whether the data is true or false. Each frequency domain discriminator outputs two values ​​for each sub-graph, representing the probability of being true and the probability of being false. The molecular band frequency domain discriminator outputs the results of all frequency domain discriminators, with dimensions B*N1*C3*C4. Here, B is the number of audio samples, N1 is the number of frequency domain discriminators (typically 4 to 6), C3 is 2 (representing the predicted and true Mel spectra), and C4 is 2 (representing the probabilities of being true and false).

[0213] Step 810: The multi-scale temporal discriminator randomly extracts multiple 80-dimensional Mel spectrum segments of varying lengths from the input Mel spectrum. These segments are then fed into an equal number of identically structured temporal discriminators to determine whether the spectrum is true or false. Each discriminator outputs two values ​​for each spectrum segment, representing the probability of being true and the probability of being false. The multi-scale temporal discriminator outputs the results of all temporal discriminators, with a dimension of B*N²*C³*C⁴, where N² is the number of temporal discriminators, typically ranging from one to five. The extracted spectrum lengths range from tens to hundreds of frames.

[0214] After obtaining all discriminator outputs, the generator is first frozen. The discriminator's output is compared with the actual true and false labels to calculate the mean squared loss. This loss is used only to optimize the discriminator and does not affect the generator, ensuring that the discriminator can distinguish between true and false labels. Then, the discriminator is frozen, the generator is unfrozen, and steps 89 and 810 are repeated to obtain new discrimination results. The mean squared error is then calculated with the actual labels. This mean squared error loss is used only to optimize the generator and does not affect the discriminator, thus improving the generator's generation quality.

[0215] Once training is complete, the trained ranking function and acoustic model can be used to perform the inference process. The acoustic model is, for example... Figure 2 and Figure 3 As shown, it includes:

[0216] Step 21: Input the phoneme-level text corresponding to the text to be synthesized into the text encoder to obtain the text representation vector.

[0217] Steps 22 and 23: Input neutral labels and specified sentiment labels into the sentiment encoder to obtain neutral and sentiment embeddings.

[0218] Step 24: Use the specified phoneme-level emotional intensity to perform a weighted sum of the neutral and emotional embeddings to obtain the emotional representation vector.

[0219] Step 25: Add the text representation vector and the sentiment representation vector to obtain the text sentiment representation vector, and input it into the prosody adapter.

[0220] Step 26: The decoder receives the prosody adapter output as input and predicts the Mel spectrum.

[0221] The processing steps after the text sentiment representation vector is input into the prosodic adapter are as follows: Figure 3 As shown, it includes:

[0222] Step 31: Using the text sentiment representation vector as input, predict the duration of each phoneme through the market forecast period.

[0223] Step 32: Expand the phoneme-level text sentiment representation vector using the predicted duration to obtain the frame-level text sentiment representation.

[0224] Step 33: Input the frame-level text sentiment representation into the voiced / unvoiced predictor to generate voiced / unvoiced labels, add them to the frame-level text sentiment representation, and use the result as the output of the prosodic adapter. The decoder receives the prosodic adapter output as input and predicts the Mel spectrum.

[0225] The embodiments of this application implement model training and inference based on structures such as emotion intensity ranking function, text encoder, emotion encoder, decoder, and adversarial device. The training phase includes a two-stage training scheme for fine-grained emotion-controlled speech synthesis, which solves the instability and unnaturalness problems in emotion-controlled speech synthesis by introducing a discriminator, tone predictor, and emotion predictor.

[0226] This application obtains more granular emotional intensity through an emotional intensity ranking function, followed by alignment and upsampling processes. It controls the intensity of different emotions at the phoneme level, allowing for more flexible control. Furthermore, the emotional intensity used is a continuous value between [0, 1], making it more convenient for users. Moreover, the method of obtaining emotional representation by weighted summation of neutral and emotional embeddings ensures a natural transition from neutral to high-intensity emotions. Using the category and intensity of emotion as prior information for the acoustic model, compared to implicitly learning fine-grained prosodic features, this explicitly constructed emotional space offers stronger interpretability and more precise control over the different intensities of various emotions.

[0227] Furthermore, this application addresses the issue of poor stability caused by emotional interpretation. It tackles the problem from two aspects: pitch shifting in synthesized audio and intensity mismatch during training and inference. By adding a cross-entropy loss function to the acoustic model and introducing an adversarial network, the generated speech is explicitly constrained, thereby improving the stability of emotional speech synthesis. In order to solve the coupling problem between emotion and text, a domain adversarial training strategy is introduced to remove emotional information in the text and achieve decoupling between emotion and text.

[0228] This application proposes a fine-grained, emotion-controlled speech synthesis method based on generative adversarial networks (GANs). It uses a sentiment intensity ranking function to obtain phoneme-level sentiment intensity, which is then used as prior information for the acoustic model. A weighted approach is then used to achieve continuous sentiment intensity control at the phoneme level during speech synthesis. Tone shifting in synthesized audio is addressed by adding a tone loss function for supervision. Fundamental frequency supervision loss and GANs are employed to improve the quality of the synthesized audio. Furthermore, to resolve the coupling between sentiment and text, a domain adversarial training strategy is introduced to decouple sentiment and text.

[0229] like Figure 9 As shown in the figure, this application embodiment also provides a speech synthesis device 900, including:

[0230] The first acquisition module 910 is used to acquire the emotional intensity of each audio segment of the text to be synthesized, wherein each audio segment corresponds to a syllable of the text to be synthesized, and the emotional intensity is a phoneme-level emotional intensity.

[0231] The second acquisition module 920 is used to obtain the Mel spectrum corresponding to the text to be synthesized based on the emotional intensity and the acoustic model.

[0232] The speech generation module 930 is used to generate speech based on the Mel spectrum.

[0233] Optionally, the first acquisition module includes:

[0234] The first extraction unit is used to extract the duration information corresponding to each syllable in the text to be synthesized;

[0235] The first acquisition unit is used to extract audio from the text to be synthesized based on the duration information to obtain an audio segment corresponding to the syllable.

[0236] The second acquisition unit is used to acquire the syllable-level emotional intensity of each audio segment using a sorting function;

[0237] The third acquisition unit is used to align the syllable-level emotional intensity with the syllable-level text to be synthesized to obtain the text to be synthesized with syllable markers.

[0238] The fourth acquisition unit is used to perform a syllable-to-phoneme upsampling process based on the syllable markers of the text to be synthesized, and obtain the phoneme-level emotional intensity of the audio segment.

[0239] Optionally, the second acquisition unit is specifically used for:

[0240] The openSMILE feature extractor was used to extract the openSMILE features of the audio segment;

[0241] Based on the openSMILE features, the syllable-level emotional intensity of each audio segment is calculated using a sorting function.

[0242] Optionally, the device further includes:

[0243] The first training module is used to train a ranking function using training data, which includes emotional audio data and neutral audio data; wherein the training process of the ranking function includes:

[0244] Two audio data points are randomly selected from the training data;

[0245] The OpenSMILE features of the audio data are extracted using the OpenSMILE feature extractor.

[0246] The openSMILE features are weighted and summed using the weight values ​​of the ranking function to obtain the sentence-level sentiment intensity;

[0247] The ranking function is iteratively optimized based on the sentence-level sentiment intensity.

[0248] Optionally, the acoustic model includes: a text encoder, an emotion encoder, a prosody adapter, and a decoder;

[0249] The second acquisition module includes:

[0250] The fifth acquisition unit is used to input the phoneme-level text corresponding to the text to be synthesized into the text encoder to obtain the encoded first text representation vector;

[0251] The sixth acquisition unit is used to obtain a first emotion representation vector through the emotion encoder and based on the emotion intensity;

[0252] The seventh acquisition unit is used to add the first text representation vector and the first sentiment representation vector to obtain the first text sentiment representation vector;

[0253] The eighth acquisition unit is used to input the first text sentiment representation vector into the prosodic adapter and obtain the output result of the prosodic adapter;

[0254] The ninth acquisition unit is used to input the output of the prosody adapter into the decoder to obtain the predicted Mel spectrum.

[0255] Optionally, the fifth acquisition unit is specifically used for:

[0256] The phoneme-level text corresponding to the text to be synthesized is input into the text encoder to extract the language information of the phoneme-level text; the text encoder is used to: encode and output the first text representation vector according to the language information;

[0257] The language information includes: the number of audio segments, the number of phonemes, and the feature dimensions.

[0258] Optionally, the sixth acquisition unit is specifically used for:

[0259] The neutral label and the sentiment label are input into the sentiment encoder to obtain a neutral vector and a sentiment vector; the sentiment encoder is used to: use the sentiment intensity to perform a weighted summation on the neutral vector and the sentiment vector to obtain and output a first sentiment representation vector.

[0260] Optionally, the eighth acquisition unit is specifically used for:

[0261] The first text sentiment representation vector is input into the prosodic adapter, which is used to: obtain the duration of each phoneme and expand the text sentiment representation vector using the duration to obtain a frame-level first text sentiment representation vector.

[0262] The frame-level first text sentiment representation vector is input into the voiced / unvoiced sound predictor to generate voiced / unvoiced sound labels; the prosodic adapter is further used to add the voiced / unvoiced sound labels and the frame-level first text sentiment representation vector as the output of the prosodic adapter.

[0263] Optionally, the acoustic model includes: a text encoder, an emotion encoder, a prosody adapter, and a decoder;

[0264] The device further includes: a second training module, used to train the acoustic model using training data;

[0265] The training process of the acoustic model includes:

[0266] The phoneme-level text corresponding to the training data is input into the text encoder to obtain the encoded second text representation vector.

[0267] A second emotion representation vector is obtained through the emotion encoder and based on the emotion intensity of the training data;

[0268] The second text representation vector and the second sentiment representation vector are added together to obtain the second text sentiment representation vector;

[0269] Input the second text sentiment representation vector into the prosodic adapter to obtain the output of the prosodic adapter;

[0270] The output of the prosody adapter is input into the decoder to obtain the predicted Mel spectrum;

[0271] Calculate the mean square error between the predicted Mel spectrum and the actual Mel spectrum.

[0272] The acoustic model is iteratively optimized based on the mean square error.

[0273] Optionally, the phoneme-level text corresponding to the training data is input into the text encoder to obtain the encoded second text representation vector, including:

[0274] The phoneme-level text corresponding to the training data is input into the text encoder to extract the language information of the phoneme-level text; the text encoder is used to: encode and output the second text representation vector based on the language information;

[0275] The language information includes: the number of audio segments, the number of phonemes, and the feature dimensions.

[0276] Optionally, obtaining a second sentiment representation vector through the sentiment encoder and based on the sentiment intensity of the training data includes:

[0277] Neutral labels and sentiment labels are input into the sentiment encoder to obtain neutral vectors and sentiment vectors;

[0278] The emotion encoder is used to: use the emotion intensity of the training data to perform a weighted summation of the neutral vector and the emotion vector to obtain and output a second emotion representation vector.

[0279] Optionally, the second text sentiment representation vector is input into the prosodic adapter to obtain the output of the prosodic adapter, including:

[0280] The second text sentiment representation vector is input into the prosodic adapter, which is used to: obtain the duration of each phoneme and expand the text sentiment representation vector using the duration to obtain a frame-level second text sentiment representation vector.

[0281] The frame-level second text sentiment representation vector is input into the voiced / unvoiced sound predictor to generate voiced / unvoiced sound labels; the prosodic adapter is further used to add the voiced / unvoiced sound labels and the frame-level second text sentiment representation vector as the output of the prosodic adapter.

[0282] Optionally, the training process of the acoustic model further includes:

[0283] The second text representation vector is input into the sentiment classifier to obtain the sentiment category of the training data;

[0284] Calculate the cross-entropy loss based on the sentiment category and the true sentiment label;

[0285] The gradient backpropagation is performed using the cross-entropy loss to remove the implicit sentiment information in the second text representation vector.

[0286] Optionally, the training process of the acoustic model further includes:

[0287] The second text sentiment representation vector is input into the tone predictor to obtain the tone of each phoneme;

[0288] The cross-entropy loss is calculated based on the tone and the actual tone label.

[0289] Optionally, after obtaining the predicted Mel spectrum, the training process of the acoustic model further includes:

[0290] The generator is frozen and the predicted Mel spectrum and the real Mel spectrum are input into the discriminator to determine the probability of the predicted Mel spectrum being true or false and the first label of the real Mel spectrum being true or false.

[0291] Calculate the mean square error between the first label and the true / false labels;

[0292] The discriminator is optimized based on the mean square error between the first label and the real / fake labels.

[0293] Optionally, the training process of the acoustic model further includes:

[0294] The generator is unfrozen and the predicted Mel spectrum and the real Mel spectrum are input into the discriminator to determine the probability of the predicted Mel spectrum being true or false and the second label of the real Mel spectrum being true or false.

[0295] Calculate the mean square error between the second label and the true / false labels;

[0296] The generator is optimized based on the mean squared error between the second label and the true / false labels.

[0297] It should be noted that the apparatus provided in this application embodiment can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0298] like Figure 10 As shown in the figure, an electronic device 1000 according to an embodiment of this application includes a processor 1010 and a transceiver 1020, wherein,

[0299] The processor 1010 is configured to: acquire the emotional intensity of each audio segment of the text to be synthesized, wherein each audio segment corresponds to a syllable of the text to be synthesized, and the emotional intensity is a phoneme-level emotional intensity;

[0300] Based on the emotional intensity and the acoustic model, the Mel spectrum corresponding to the text to be synthesized is obtained;

[0301] Speech is generated based on the Mel spectrum.

[0302] Optionally, the processor acquires the emotional intensity of each audio segment of the text to be synthesized, including:

[0303] Extract the duration information corresponding to each syllable in the text to be synthesized;

[0304] Based on the duration information, audio segments are extracted from the text to be synthesized to obtain audio segments corresponding to syllables;

[0305] The emotional intensity at the syllable level for each audio segment is obtained using a sorting function;

[0306] Align the syllable-level emotional intensity with the syllable-level text to be synthesized to obtain the text to be synthesized with syllable markers;

[0307] Based on the syllable markers of the text to be synthesized, an upsampling process from syllable level to phoneme level is performed to obtain the phoneme-level emotional intensity of the audio segment.

[0308] Optionally, the processor uses a sorting function to obtain the syllable-level emotional intensity of each audio segment, including:

[0309] The openSMILE feature extractor was used to extract the openSMILE features of the audio segment;

[0310] Based on the openSMILE features, the syllable-level emotional intensity of each audio segment is calculated using a sorting function.

[0311] Optionally, the processor is further configured to:

[0312] A ranking function is trained using training data, which includes emotional audio data and neutral audio data; the training process of the ranking function includes:

[0313] Two audio data points are randomly selected from the training data;

[0314] The OpenSMILE features of the audio data are extracted using the OpenSMILE feature extractor.

[0315] The openSMILE features are weighted and summed using the weight values ​​of the ranking function to obtain the sentence-level sentiment intensity;

[0316] The ranking function is iteratively optimized based on the sentence-level sentiment intensity.

[0317] Optionally, the acoustic model includes: a text encoder, an emotion encoder, a prosody adapter, and a decoder;

[0318] The processor obtains the Mel spectrum corresponding to the text to be synthesized based on the emotional intensity and the acoustic model, including:

[0319] The phoneme-level text corresponding to the text to be synthesized is input into the text encoder to obtain the encoded first text representation vector;

[0320] The first emotion representation vector is obtained through the emotion encoder and based on the emotion intensity;

[0321] Add the first text representation vector and the first sentiment representation vector to obtain the first text sentiment representation vector;

[0322] Input the first text sentiment representation vector into the prosody adapter and obtain the output of the prosody adapter;

[0323] The output of the prosody adapter is input into the decoder to obtain the predicted Mel spectrum.

[0324] Optionally, the processor inputs the phoneme-level text corresponding to the text to be synthesized into the text encoder to obtain the encoded first text representation vector, including:

[0325] The phoneme-level text corresponding to the text to be synthesized is input into the text encoder to extract the language information of the phoneme-level text; the text encoder is used to: encode and output the first text representation vector according to the language information;

[0326] The language information includes: the number of audio segments, the number of phonemes, and the feature dimensions.

[0327] Optionally, the processor obtains a first emotion representation vector through the emotion encoder and based on the emotion intensity, including:

[0328] The neutral label and the sentiment label are input into the sentiment encoder to obtain a neutral vector and a sentiment vector; the sentiment encoder is used to: use the sentiment intensity to perform a weighted summation on the neutral vector and the sentiment vector to obtain and output a first sentiment representation vector.

[0329] Optionally, the processor inputs the first text sentiment representation vector into the prosodic adapter to obtain the output of the prosodic adapter, including:

[0330] The first text sentiment representation vector is input into the prosodic adapter, which is used to: obtain the duration of each phoneme and expand the text sentiment representation vector using the duration to obtain a frame-level first text sentiment representation vector.

[0331] The frame-level first text sentiment representation vector is input into the voiced / unvoiced sound predictor to generate voiced / unvoiced sound labels; the prosodic adapter is further used to add the voiced / unvoiced sound labels and the frame-level first text sentiment representation vector as the output of the prosodic adapter.

[0332] Optionally, the acoustic model includes: a text encoder, an emotion encoder, a prosody adapter, and a decoder;

[0333] The processor is further configured to: train the acoustic model using training data;

[0334] The training process of the acoustic model includes:

[0335] The phoneme-level text corresponding to the training data is input into the text encoder to obtain the encoded second text representation vector.

[0336] A second emotion representation vector is obtained through the emotion encoder and based on the emotion intensity of the training data;

[0337] The second text representation vector and the second sentiment representation vector are added together to obtain the second text sentiment representation vector;

[0338] Input the second text sentiment representation vector into the prosodic adapter to obtain the output of the prosodic adapter;

[0339] The output of the prosody adapter is input into the decoder to obtain the predicted Mel spectrum;

[0340] Calculate the mean square error between the predicted Mel spectrum and the actual Mel spectrum.

[0341] The acoustic model is iteratively optimized based on the mean square error.

[0342] Optionally, the processor inputs the phoneme-level text corresponding to the training data into the text encoder to obtain an encoded second text representation vector, including:

[0343] The phoneme-level text corresponding to the training data is input into the text encoder to extract the language information of the phoneme-level text; the text encoder is used to: encode and output the second text representation vector based on the language information;

[0344] The language information includes: the number of audio segments, the number of phonemes, and the feature dimensions.

[0345] Optionally, the processor obtains a second sentiment representation vector through the sentiment encoder and based on the sentiment intensity of the training data, including:

[0346] Neutral labels and sentiment labels are input into the sentiment encoder to obtain neutral vectors and sentiment vectors;

[0347] The emotion encoder is used to: use the emotion intensity of the training data to perform a weighted summation of the neutral vector and the emotion vector to obtain and output a second emotion representation vector.

[0348] Optionally, the processor inputs the second text sentiment representation vector into the prosodic adapter to obtain the output of the prosodic adapter, including:

[0349] The second text sentiment representation vector is input into the prosodic adapter, which is used to: obtain the duration of each phoneme and expand the text sentiment representation vector using the duration to obtain a frame-level second text sentiment representation vector.

[0350] The frame-level second text sentiment representation vector is input into the voiced / unvoiced sound predictor to generate voiced / unvoiced sound labels; the prosodic adapter is further used to add the voiced / unvoiced sound labels and the frame-level second text sentiment representation vector as the output of the prosodic adapter.

[0351] Optionally, the training process of the acoustic model further includes:

[0352] The second text representation vector is input into the sentiment classifier to obtain the sentiment category of the training data;

[0353] Calculate the cross-entropy loss based on the sentiment category and the true sentiment label;

[0354] The gradient backpropagation is performed using the cross-entropy loss to remove the implicit sentiment information in the second text representation vector.

[0355] Optionally, the training process of the acoustic model further includes:

[0356] The second text sentiment representation vector is input into the tone predictor to obtain the tone of each phoneme;

[0357] Calculate the cross-entropy loss based on the tone and the actual tone label.

[0358] Optionally, after obtaining the predicted Mel spectrum, the training process of the acoustic model further includes:

[0359] The generator is frozen and the predicted Mel spectrum and the real Mel spectrum are input into the discriminator to determine the probability of the predicted Mel spectrum being true or false and the first label of the real Mel spectrum being true or false.

[0360] Calculate the mean square error between the first label and the true / false labels;

[0361] The discriminator is optimized based on the mean square error between the first label and the real / fake labels.

[0362] Optionally, the training process of the acoustic model further includes:

[0363] The generator is unfrozen and the predicted Mel spectrum and the real Mel spectrum are input into the discriminator to determine the probability of the predicted Mel spectrum being true or false and the second label of the real Mel spectrum being true or false.

[0364] Calculate the mean square error between the second label and the true / false labels;

[0365] The generator is optimized based on the mean squared error between the second label and the true / false labels.

[0366] It should be noted that the electronic device provided in this application embodiment can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0367] Another embodiment of the electronic device in this application, such as Figure 11 As shown, it includes a transceiver 1110, a processor 1100, a memory 1120, and a program or instructions stored in the memory 1120 and executable on the processor 1100; when the processor 1100 executes the program or instructions, it implements the above-mentioned speech synthesis method.

[0368] The transceiver 1110 is used to receive and send data under the control of the processor 1100.

[0369] Among them, Figure 11 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 1100) and memory (memory 1120). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 1110 may be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 1100 is responsible for managing the bus architecture and general processing, and the memory 1120 may store data used by the processor 1100 during operation.

[0370] An embodiment of this application provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps in the speech synthesis method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0371] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0372] It should be further noted that the terminal devices described in this specification include, but are not limited to, smartphones, tablets, etc., and many of the described functional components are referred to as modules in order to more specifically emphasize the independence of their implementation.

[0373] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0374] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable type of data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0375] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0376] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of this application. Therefore, this application should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this application complete and convey the scope of this application to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of the range and any subranges in between.

[0377] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A speech synthesis method, characterized in that, include: The emotional intensity of each audio segment of the text to be synthesized is obtained, where each audio segment corresponds to a syllable of the text to be synthesized, and the emotional intensity is at the phoneme level. Based on the emotional intensity and the acoustic model, the Mel spectrum corresponding to the text to be synthesized is obtained; Speech is generated based on the Mel spectrum; The process of obtaining the emotional intensity of each audio segment of the text to be synthesized includes: Extract the duration information corresponding to each syllable in the text to be synthesized; Based on the duration information, audio segments are extracted from the text to be synthesized to obtain audio segments corresponding to syllables; The emotional intensity at the syllable level for each audio segment is obtained using a sorting function; Align the syllable-level emotional intensity with the syllable-level text to be synthesized to obtain the text to be synthesized with syllable markers; Based on the syllable markers of the text to be synthesized, an upsampling process from syllable level to phoneme level is performed to obtain the phoneme-level emotional intensity of the audio segment; The acoustic model includes: a text encoder, an emotion encoder, a prosodic adapter, and a decoder; The step of obtaining the Mel spectrum corresponding to the text to be synthesized based on the emotional intensity and the acoustic model includes: The phoneme-level text corresponding to the text to be synthesized is input into the text encoder to obtain the encoded first text representation vector; The first emotion representation vector is obtained through the emotion encoder and based on the emotion intensity; Add the first text representation vector and the first sentiment representation vector to obtain the first text sentiment representation vector; Input the first text sentiment representation vector into the prosody adapter and obtain the output of the prosody adapter; The output of the prosody adapter is input into the decoder to obtain the predicted Mel spectrum.

2. The method according to claim 1, characterized in that, The step of obtaining the syllable-level emotional intensity of each audio segment using a sorting function includes: The openSMILE feature extractor was used to extract the openSMILE features of the audio segment; Based on the openSMILE features, the syllable-level emotional intensity of each audio segment is calculated using a sorting function.

3. The method according to claim 1, characterized in that, The method further includes: A ranking function is trained using training data, which includes emotional audio data and neutral audio data; the training process of the ranking function includes: Two audio data points are randomly selected from the training data; The OpenSMILE features of the audio data are extracted using the OpenSMILE feature extractor. The openSMILE features are weighted and summed using the weight values ​​of the ranking function to obtain the sentence-level sentiment intensity; The ranking function is iteratively optimized based on the sentence-level sentiment intensity.

4. The method according to claim 1, characterized in that, The phoneme-level text corresponding to the text to be synthesized is input into the text encoder to obtain the encoded first text representation vector, including: The phoneme-level text corresponding to the text to be synthesized is input into the text encoder to extract the language information of the phoneme-level text; the text encoder is used to: encode and output the first text representation vector according to the language information; The language information includes: the number of audio segments, the number of phonemes, and the feature dimensions.

5. The method according to claim 1, characterized in that, The first emotion representation vector is obtained through the emotion encoder and based on the emotion intensity, including: The neutral label and the sentiment label are input into the sentiment encoder to obtain a neutral vector and a sentiment vector; the sentiment encoder is used to: use the sentiment intensity to perform a weighted summation on the neutral vector and the sentiment vector to obtain and output a first sentiment representation vector.

6. The method according to claim 1, characterized in that, Input the first text sentiment representation vector into the prosodic adapter to obtain the output of the prosodic adapter, including: The first text sentiment representation vector is input into the prosodic adapter, which is used to: obtain the duration of each phoneme and expand the text sentiment representation vector using the duration to obtain a frame-level first text sentiment representation vector. The frame-level first text sentiment representation vector is input into the voiced / unvoiced sound predictor to generate voiced / unvoiced sound labels; the prosodic adapter is further used to add the voiced / unvoiced sound labels and the frame-level first text sentiment representation vector as the output of the prosodic adapter.

7. The method according to claim 1, characterized in that, The acoustic model includes: a text encoder, an emotion encoder, a prosodic adapter, and a decoder; The method further includes: training the acoustic model using training data; The training process of the acoustic model includes: The phoneme-level text corresponding to the training data is input into the text encoder to obtain the encoded second text representation vector. A second emotion representation vector is obtained through the emotion encoder and based on the emotion intensity of the training data; The second text representation vector and the second sentiment representation vector are added together to obtain the second text sentiment representation vector; Input the second text sentiment representation vector into the prosodic adapter to obtain the output of the prosodic adapter; The output of the prosody adapter is input into the decoder to obtain the predicted Mel spectrum; Calculate the mean square error between the predicted Mel spectrum and the actual Mel spectrum. The acoustic model is iteratively optimized based on the mean square error.

8. The method according to claim 7, characterized in that, The phoneme-level text corresponding to the training data is input into the text encoder to obtain the encoded second text representation vector, including: The phoneme-level text corresponding to the training data is input into the text encoder to extract the language information of the phoneme-level text; the text encoder is used to: encode and output the second text representation vector based on the language information; The language information includes: the number of audio segments, the number of phonemes, and the feature dimensions.

9. The method according to claim 7, characterized in that, The second sentiment representation vector is obtained through the sentiment encoder and based on the sentiment intensity of the training data, including: Neutral labels and sentiment labels are input into the sentiment encoder to obtain neutral vectors and sentiment vectors; The emotion encoder is used to: use the emotion intensity of the training data to perform a weighted summation of the neutral vector and the emotion vector to obtain and output a second emotion representation vector.

10. The method according to claim 7, characterized in that, Input the second text sentiment representation vector into the prosodic adapter to obtain the output of the prosodic adapter, including: The second text sentiment representation vector is input into the prosodic adapter, which is used to: obtain the duration of each phoneme and expand the text sentiment representation vector using the duration to obtain a frame-level second text sentiment representation vector. The frame-level second text sentiment representation vector is input into the voiced / unvoiced sound predictor to generate voiced / unvoiced sound labels; the prosodic adapter is further used to add the voiced / unvoiced sound labels and the frame-level second text sentiment representation vector as the output of the prosodic adapter.

11. The method according to claim 7, characterized in that, The training process of the acoustic model also includes: The second text representation vector is input into the sentiment classifier to obtain the sentiment category of the training data; Calculate the cross-entropy loss based on the sentiment category and the true sentiment label; The gradient backpropagation is performed using the cross-entropy loss to remove the implicit sentiment information in the second text representation vector.

12. The method according to claim 7, characterized in that, The training process of the acoustic model also includes: The second text sentiment representation vector is input into the tone predictor to obtain the tone of each phoneme; The cross-entropy loss is calculated based on the tone and the actual tone label.

13. The method according to claim 7, characterized in that, After obtaining the predicted Mel spectrum, the training process of the acoustic model further includes: The generator is frozen and the predicted Mel spectrum and the real Mel spectrum are input into the discriminator to determine the probability of the predicted Mel spectrum being true or false and the first label of the real Mel spectrum being true or false. Calculate the mean square error between the first label and the true / false labels; The discriminator is optimized based on the mean square error between the first label and the real / fake labels.

14. The method according to claim 13, characterized in that, The training process of the acoustic model also includes: The generator is unfrozen and the predicted Mel spectrum and the real Mel spectrum are input into the discriminator to determine the probability of the predicted Mel spectrum being true or false and the second label of the real Mel spectrum being true or false. Calculate the mean square error between the second label and the true / false labels; The generator is optimized based on the mean squared error between the second label and the true / false labels.

15. A speech synthesis device, characterized in that, include: The first acquisition module is used to acquire the emotional intensity of each audio segment of the text to be synthesized, wherein each audio segment corresponds to a syllable of the text to be synthesized, and the emotional intensity is a phoneme-level emotional intensity. The second acquisition module is used to obtain the Mel spectrum corresponding to the text to be synthesized based on the emotional intensity and the acoustic model. The speech generation module is used to generate speech based on the Mel spectrum; The first acquisition module includes: The first extraction unit is used to extract the duration information corresponding to each syllable in the text to be synthesized; The first acquisition unit is used to extract audio from the text to be synthesized based on the duration information to obtain an audio segment corresponding to the syllable. The second acquisition unit is used to acquire the syllable-level emotional intensity of each audio segment using a sorting function; The third acquisition unit is used to align the syllable-level emotional intensity with the syllable-level text to be synthesized to obtain the text to be synthesized with syllable markers. The fourth acquisition unit is used to perform a syllable-to-phoneme upsampling process based on the syllable markers of the text to be synthesized, so as to obtain the phoneme-level emotional intensity of the audio segment. The acoustic model includes: a text encoder, an emotion encoder, a prosodic adapter, and a decoder; The second acquisition module includes: The fifth acquisition unit is used to input the phoneme-level text corresponding to the text to be synthesized into the text encoder to obtain the encoded first text representation vector; The sixth acquisition unit is used to obtain a first emotion representation vector through the emotion encoder and based on the emotion intensity; The seventh acquisition unit is used to add the first text representation vector and the first sentiment representation vector to obtain the first text sentiment representation vector; The eighth acquisition unit is used to input the first text sentiment representation vector into the prosodic adapter and obtain the output result of the prosodic adapter; The ninth acquisition unit is used to input the output of the prosody adapter into the decoder to obtain the predicted Mel spectrum.

16. An electronic device, characterized in that, Includes processor and transceiver; The processor is configured to: obtain the emotional intensity of each audio segment of the text to be synthesized, wherein each audio segment corresponds to a syllable of the text to be synthesized, and the emotional intensity is a phoneme-level emotional intensity; Based on the emotional intensity and the acoustic model, the Mel spectrum corresponding to the text to be synthesized is obtained; Speech is generated based on the Mel spectrum; The processor acquires the emotional intensity of each audio segment of the text to be synthesized, including: Extract the duration information corresponding to each syllable in the text to be synthesized; Based on the duration information, audio segments are extracted from the text to be synthesized to obtain audio segments corresponding to syllables; The emotional intensity at the syllable level for each audio segment is obtained using a sorting function; Align the syllable-level emotional intensity with the syllable-level text to be synthesized to obtain the text to be synthesized with syllable markers; Based on the syllable markers of the text to be synthesized, an upsampling process from syllable level to phoneme level is performed to obtain the phoneme-level emotional intensity of the audio segment; The acoustic model includes: a text encoder, an emotion encoder, a prosodic adapter, and a decoder; The processor obtains the Mel spectrum corresponding to the text to be synthesized based on the emotional intensity and the acoustic model, including: The phoneme-level text corresponding to the text to be synthesized is input into the text encoder to obtain the encoded first text representation vector; The first emotion representation vector is obtained through the emotion encoder and based on the emotion intensity; Add the first text representation vector and the first sentiment representation vector to obtain the first text sentiment representation vector; Input the first text sentiment representation vector into the prosody adapter and obtain the output of the prosody adapter; The output of the prosody adapter is input into the decoder to obtain the predicted Mel spectrum.

17. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the speech synthesis method as described in any one of claims 1-14.

Images