A method and apparatus for synthesizing audio recordings with different emotions

Abstract

This invention provides a method and apparatus for synthesizing audio with different emotions, including a training phase and an inference phase. The training phase includes the following steps: S11, collecting training corpora, including audio from different speakers and corresponding text, as well as emotion tags, and extracting the spectral features of the corresponding speech; S12, training an emotion speech feature extraction model based on the speech spectral features and corresponding emotion tags; S13, extracting the emotion feature vector of the training corpora and the text encoding vector of the corresponding training corpora; S14, combining the text encoding vector with the emotion feature vector of the speech, and training a speech synthesis model through the acoustic features of the corresponding speech; S15, training an emotion feature prediction model using the emotion feature vector of the speech and the text encoding vector as input; S16, training a vocoder through the acoustic features of the speech and the corresponding speech. This invention solves the problem of bland tone and unclear emotion in traditional speech synthesis.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and more specifically to a method and apparatus for synthesizing audio with different emotions. Background Technology

[0002] With the continuous development of speech technology, people have higher and higher requirements for the quality of speech synthesis. The mechanical pronunciation of traditional speech synthesis is no longer satisfactory, and people hope to add more emotional expressiveness to synthesized speech.

[0003] Current speech technologies typically synthesize emotions directly or convert emotion tags into vector codes. These methods usually rely on pre-defined results without considering different contexts, and the encoding is too simplistic, with fixed tags that fail to reflect subtle changes. Therefore, dynamic encoding is more effective at reflecting actual emotional states and adjusting emotional intensity. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for synthesizing audio with different emotions, in order to solve the problems existing in the prior art. This invention addresses the issues of lack of emotion and difficulty in distinguishing emotions in current sound synthesis methods.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method and apparatus for synthesizing audio with different emotions, comprising a training phase and an inference phase, wherein the training phase includes the following steps:

[0007] S11. Collect training data, including audio and corresponding text of different speakers, as well as sentiment tags and extract the spectral features of the corresponding speech.

[0008] S12. Based on the speech spectrum features and corresponding emotion tags, train the emotion speech feature extraction model;

[0009] S13. Extract the sentiment feature vector and the corresponding text encoding vector of the training corpus;

[0010] S14. Combine the text encoding vector with the emotional feature vector of the speech, and train the speech synthesis model using the corresponding acoustic features of the speech.

[0011] S15. Using the emotional feature vector of speech and the text encoding vector as input, train the emotional feature prediction model;

[0012] S16. Train the vocoder using the acoustic features of the speech and the corresponding speech.

[0013] The reasoning phase includes the following steps:

[0014] S21. Process the text to obtain a text encoding vector, and generate sentiment feature information based on the text encoding vector using a sentiment feature prediction model.

[0015] S22. Combine the text encoding vector with emotional feature information to generate speech acoustic features through a speech synthesis model.

[0016] S23. Based on the acoustic characteristics of speech, synthesize audio using a vocoder.

[0017] Furthermore, in order to extract the spectral features of the corresponding speech, S11 also includes:

[0018] The training corpus includes, but is not limited to, publicly available speech synthesis training datasets or self-recorded speech synthesis training datasets, and the extracted speech spectral features include, but are not limited to, linear spectral features and Mel spectral features.

[0019] Furthermore, in order to train the emotional speech feature extraction model, S12 also includes:

[0020] The emotion recognition network is formed by using the spectral features and labels of the trained speech as input. The network structure of the emotion recognition model includes, but is not limited to, CNN, GRU and other network structures.

[0021] Furthermore, in order to collect training data and extract the spectral features of the corresponding speech, S11 also includes:

[0022] The extracted speech spectral features include, but are not limited to, linear spectral features and Mel spectral features.

[0023] Furthermore, in order to train the emotional speech feature extraction model, S12 also includes

[0024] The emotion recognition network is formed by using the spectral features and labels of the trained speech as input. The network structure of the emotion recognition model includes, but is not limited to, CNN, GRU and other network structures.

[0025] Furthermore, in order to extract the sentiment feature vector and text feature vector of the training corpus, S13 also includes:

[0026] The training corpus is used to extract sentiment feature vectors from the training corpus through a trained sentiment feature extraction network. The text is normalized and the text encoding vectors corresponding to the training corpus are extracted through a text encoding network based on the text information. Then, multi-scale sentiment features of the training corpus are extracted through a convolutional neural network to obtain the feature matrix.

[0027] Furthermore, in order to train the speech synthesis model, S14 also includes:

[0028] During the training of the speech synthesis model, the generated speech feature vectors are used to extract multi-scale emotional features through the emotion extraction model trained in S12, and compared with the multi-scale emotional features extracted in S13. The network is adjusted by feeding back the error through the loss function.

[0029] First, the text encoding vector and the sentiment feature vector extracted from S13 are added together for encoding. Then, the sentiment feature vector is expanded by the alignment result between the spectral features and the text before being input into the decoding layer of the speech synthesis model.

[0030] Alignment methods between spectral features and text can include, but are not limited to, forced alignment and monotonic alignment search.

[0031] Furthermore, in order to train the text prediction sentiment feature model, S15 also includes:

[0032] The text encoding vector is used as input, and a deep learning network outputs a sentiment feature prediction vector. This vector is compared with the sentiment feature vectors extracted from the training corpus in S13, and the network is adjusted by feeding back the error through a loss function. The network structure of the sentiment feature prediction model includes, but is not limited to, RNN, Transformer, and other network structures.

[0033] Furthermore, to train the vocoder, S16 also includes:

[0034] The speech features and speech signals generated by the speech synthesis model trained via S14 are used as input, and a vocoder is trained using an adversarial neural network. The vocoder used includes, but is not limited to, WaveNET, WavRNN, and MelGAN.

[0035] Furthermore, in order to perform emotional speech synthesis, S21-23 also includes:

[0036] The parameters of the speech synthesis model in the inference stage are obtained from the training stage, and the network structure is consistent. The text processing method in the inference stage is consistent with that in the training stage. The sentiment feature vector generated in the inference stage is expanded according to the length of the text and combined with the text. The inference stage does not use the sentiment feature extraction model. The sentiment feature vector is obtained from the text prediction sentiment feature model. The sentiment intensity can be amplified or weakened by multiplying the sentiment feature vector by the feature coefficient.

[0037] This invention provides a device for synthesizing audio recordings of different emotions, comprising:

[0038] The speaker audio feature extraction unit is used to train the speech feature extraction model and extract speech feature vectors.

[0039] The emotional speech feature extraction unit is used to extract the emotional features of the target.

[0040] The sentiment feature prediction unit is used to predict sentiment feature vectors from the input text information.

[0041] The speech synthesis unit is used to synthesize speech information from input text information and emotional feature information;

[0042] The vocoder unit converts the generated speech features into audio signals.

[0043] The beneficial effects of the method and apparatus for synthesizing different emotional audio provided in this application include, but are not limited to:

[0044] This invention discloses a method and apparatus for synthesizing emotional audio, capable of real-time emotional speech synthesis of a speaker's statements. This invention solves the problem that traditional synthesis methods cannot express the speaker's emotions. This invention is applied to the field of emotional speech synthesis, but is not limited to this field. By predicting emotional feature vectors and incorporating them into the synthesized statements, the synthesized audio can be made richer in emotion and more expressive. This invention can be applied to the field of emotional speech synthesis, but is not limited to this field. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating a method for synthesizing audio with different emotions according to an embodiment of the present invention.

[0046] Figure 2 This is a flowchart illustrating the training process described in an embodiment of the present invention.

[0047] Figure 3 This is a flowchart illustrating the inference process described in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] Conversely, this application covers any substitutions, modifications, equivalent methods, and schemes made within the spirit and scope of this application as defined in the claims. Furthermore, to provide the public with a better understanding of this application, certain specific details are described in detail below. However, this application can be fully understood by those skilled in the art even without these detailed descriptions.

[0050] The following will combine Figure 1-3 This application provides a detailed description of a method and apparatus for synthesizing audio recordings with different emotions, as illustrated in the embodiments of this application. It is worth noting that the following embodiments are merely illustrative of this application and do not constitute a limitation thereof.

[0051] Example 1:

[0052] A method for synthesizing audio with different emotions includes the following steps during the training phase:

[0053] S11. Collect training data, including audio and corresponding text of different speakers, as well as sentiment tags and extract the spectral features of the corresponding speech.

[0054] Optionally, the training corpus may include, but is not limited to, publicly available speech synthesis training datasets or self-recorded speech synthesis training datasets, and the extracted speech spectral features may include, but are not limited to, linear spectral features and Mel spectral features.

[0055] For example, 80-dimensional Mel-spectral features can be extracted from the acquired audio through a window with a window length of 0.05s and a sliding distance of 0.015s, or 513-dimensional linear spectral features can be extracted through Fourier transform.

[0056] S12. Based on the speech spectrum features and corresponding emotion tags, train the emotion speech feature extraction model;

[0057] Specifically, the spectral features and labels of the trained speech are used as input to the emotion feature extraction network for speech emotion recognition. The network structure of the emotion feature recognition model includes, but is not limited to, CNN, GRU, and other network structures.

[0058] For example, linear spectral features are extracted into feature vectors using a 2D convolutional network and a GRU, then passed through a fully connected layer and classified using softmax.

[0059] S13. Extract the sentiment feature vector and the corresponding text encoding vector of the training corpus;

[0060] Specifically, the sentiment feature vectors of the training corpus are extracted based on the trained sentiment feature extraction network. The text is normalized and the text encoding vectors of the corresponding texts in the training corpus are extracted through the text encoding network based on the text information. Then, the multi-scale sentiment features of the training corpus are extracted through the convolutional neural network to obtain the feature matrix.

[0061] For example, sentiment feature vectors are extracted using the bottleneck of a trained classification model. By standardizing Chinese text, illegal syllables are filtered out, and legal input is segmented and labeled with parts of speech. The extracted comprehensive linguistic features are then input into a prosodic prediction model to obtain pause level labels. Chinese characters are converted into corresponding pinyin phonemes, and the text information is encoded to obtain a text encoding vector.

[0062] S14. Combine the emotional feature vector of the text with the corresponding acoustic features of the speech to train the speech synthesis model.

[0063] Specifically, during the training of the speech synthesis model, the generated speech feature vectors are compared with the multi-scale emotional features extracted by the emotion extraction model trained in S12 and the multi-scale emotional features extracted in S13. The network is adjusted by feeding back the error through the loss function.

[0064] First, the text encoding vector and the sentiment feature vector extracted from S13 are added together for encoding. Then, the sentiment feature vector is expanded by the alignment result between the spectral features and the text before being input into the decoding layer of the speech synthesis model.

[0065] Alignment methods between spectral features and text can include, but are not limited to, forced alignment and monotonic alignment search.

[0066] S15. The emotional feature vector of the speech and the text encoding vector are used as inputs to train the text emotional feature model;

[0067] The text encoding vector is used as input, and a deep learning network is used to output a sentiment feature prediction vector. This vector is then compared with the sentiment feature vector of the training corpus extracted in S13. The network is adjusted by feeding back the error through a loss function.

[0068] Optionally, the network structure of the text feature prediction model includes, but is not limited to, network structures such as Transformer and RNN.

[0069] S16. Train the vocoder using the acoustic features of the speech and the corresponding speech.

[0070] Specifically, the speech features and speech information generated by the speech synthesis model trained by S14 are used as input, and the vocoder is trained by an adversarial neural network.

[0071] Optionally, the vocoder used may include, but is not limited to, WavNET, WavRNN, and MelGAN.

[0072] The reasoning phase includes the following steps:

[0073] S21. Process the text to obtain a text encoding vector, and generate sentiment feature information based on the text encoding vector using a sentiment feature prediction model.

[0074] S22. Combine the text encoding vector with emotional feature information to generate speech acoustic features through a speech synthesis model.

[0075] S23. Based on the acoustic characteristics of speech, synthesize audio using a vocoder.

[0076] Understandably, the parameters of the speech synthesis model in the inference stage are obtained from the training stage, and the network structure remains consistent. The text processing method in the inference stage is consistent with that in the training stage. The sentiment feature vector generated in the inference stage is expanded according to the length of the text and combined with the text. The inference stage does not use a sentiment feature extraction model; instead, the sentiment feature vector is obtained from a text-predicted sentiment feature model. By multiplying the sentiment feature vector by a feature coefficient, the feature coefficient can generally amplify or weaken the sentiment intensity.

[0077] For example, multiplying the emotional feature vector by 2 can double the emotional intensity, while multiplying it by 0.5 can reduce the emotional intensity by half. The coefficient is generally set between 0 and 2.

[0078] Example 2:

[0079] In this embodiment, a device for synthesizing audio recordings of different emotions includes:

[0080] The speaker audio feature extraction unit is used to train the speech feature extraction model and extract speech feature vectors.

[0081] The emotional speech feature extraction unit is used to extract the emotional features of the target.

[0082] The sentiment feature prediction unit is used to predict sentiment feature vectors from the input text information.

[0083] The speech synthesis unit is used to synthesize speech information from input text information and emotional feature information;

[0084] The vocoder unit converts the generated speech features into audio signals;

[0085] The device for synthesizing audio with different emotions provided in Embodiment 2 of the present invention generates emotional feature vectors from text and combines them with text feature vectors for speech synthesis, which can make the synthesized audio more emotional and expressive.

[0086] It should be noted that the modules (or units) in this embodiment are logical in nature. In actual implementation, multiple modules (or units) can be merged into one module (or unit), and one module (or unit) can also be split into multiple modules (or units).

[0087] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for synthesizing audio recordings with different emotions, characterized in that, It includes a training phase and an inference phase. The training phase includes the following steps: S11. Collect training data, including audio and corresponding text of different speakers, as well as sentiment tags and extract the spectral features of the corresponding speech. S12. Based on the speech spectrum features and corresponding emotion tags, train the emotion speech feature extraction model; S13. Extract the sentiment feature vector and the corresponding text encoding vector of the training corpus; S14. Combine the text encoding vector with the emotional feature vector of the speech, and train the speech synthesis model through the acoustic features of the corresponding speech. S15. Using the emotional feature vector of speech and the text encoding vector as input, train the emotional feature prediction model; S16. Train the vocoder using the acoustic features of the speech and the corresponding speech. The reasoning phase includes the following steps: S21. Process the text to obtain a text encoding vector, and generate sentiment feature information based on the text vector encoding using a sentiment feature prediction model; S22. Combine the text encoding vector with emotional feature information to generate speech acoustic features through a speech synthesis model; S23. Based on the acoustic characteristics of speech, synthesize audio using a vocoder.

2. The method for synthesizing audio recordings with different emotions according to claim 1, characterized in that, Step S11 further includes: The training corpus includes publicly available speech synthesis training datasets or self-recorded speech synthesis training datasets, and the extracted speech spectral features include linear spectral features or Mel spectral features.

3. The method for synthesizing audio recordings with different emotions according to claim 1, characterized in that, Step S12 further includes: The emotion recognition network is used to perform emotion recognition based on the spectral features and labels of the trained speech as input. The network structure of the emotion recognition model includes CNN or GRU network structure.

4. The method for synthesizing audio recordings with different emotions according to claim 1, characterized in that, Step S13 further includes: The training corpus is used to extract sentiment feature vectors from the training corpus through a trained sentiment feature extraction network. The text is normalized and the text encoding vectors corresponding to the training corpus are extracted through a text encoding network based on the text information. Then, multi-scale sentiment features of the training corpus are extracted through a convolutional neural network to obtain the feature matrix.

5. The method for synthesizing audio recordings with different emotions according to claim 1, characterized in that, Step S14 further includes: During the training of the speech synthesis model, the generated speech feature vectors are used to extract multi-scale emotion features through the emotion extraction model trained in S12, and compared with the multi-scale emotion features of the training corpus extracted in S13. The network is adjusted by feeding back the error through the loss function. First, the text encoding vector and the sentiment feature vector extracted from S13 are added together for encoding. Then, the sentiment feature vector is expanded by the alignment result between the spectral features and the text before being input into the decoding layer of the speech synthesis model. Alignment methods between spectral features and text include forced alignment or monotonic alignment search.

6. The method for synthesizing audio recordings with different emotions according to claim 1, characterized in that, Step S15 further includes: The text encoding vector is used as input, and a deep learning network is used to output a sentiment feature prediction vector. This vector is compared with the sentiment feature vector of the training corpus extracted by S13. The network is adjusted by feeding back the error through a loss function. The sentiment feature prediction model network structure includes RNN or Transformer network structure.

7. The method for synthesizing audio recordings with different emotions according to claim 1, characterized in that, Step S16 further includes: The speech features and speech information generated by the speech synthesis model trained by S14 are used as input, and the vocoder is trained by an adversarial neural network; the vocoder used includes WaveNET, WavRNN or MelGAN.

8. The method for synthesizing audio recordings with different emotions according to claim 1, characterized in that, Steps S21-23 further include: The parameters of the speech synthesis model in the inference stage are obtained from the training stage, and the network structure is consistent. The text processing method in the inference stage is consistent with that in the training stage. The sentiment feature vector generated in the inference stage is expanded according to the length of the text and combined with the text. The inference stage does not use the sentiment feature extraction model. The sentiment feature vector is obtained from the text prediction sentiment feature model. The sentiment intensity can be amplified or weakened by multiplying the sentiment feature vector by the feature coefficient.

9. A device for synthesizing audio recordings of different emotions, characterized in that: The method for synthesizing different emotional audio as described in claim 1 is performed using the apparatus for synthesizing different emotional audio.

Images