Voice synthesis emotion regulation method and device, and computer device

Abstract

The present application relates to a kind of voice synthesis emotion regulation method, device and computer equipment, the emotional characteristics of input text are extracted by text processing module, and the prosody parameter of voice rhythm that is adapted with text emotional characteristics is generated;The emotional characteristics and the prosody parameter of voice reading rhythm are input to the pre-training emotional timbre synthesis model and synthesized mixed timbre parameter set;Based on the emotional characteristics, the prosody parameter of voice and the mixed timbre parameter, parameter integration is carried out by dynamic interaction mechanism, and comprehensive control parameter set is generated;The comprehensive control parameter set is input to voice synthesis engine, real-time synthesis and output with corresponding emotional color voice signal, to realize the purpose of voice synthesis with accurate emotional expression.

Claims

1. A speech synthesis emotion modulation method, characterized in that, Includes the following steps: The text processing module extracts the emotional features of the input text and generates speech prosody parameters that match the emotional features of the text. The steps of inputting the emotional features and the speech prosody parameters into a pre-trained emotional timbre synthesis model to synthesize a mixed timbre parameter set include: The emotional features and speech prosody parameters are input into a pre-trained emotional timbre synthesis model to generate initial values ​​of core acoustic parameters for the mixed timbre. The core acoustic parameters include fundamental frequency range, timbre brightness coefficient, harmonic distortion, and formant frequency. The initial values ​​of the core acoustic parameters are dynamically corrected by the parameter optimization module in the emotion-rhythm-timbre association model, and the final set of mixed timbre parameters is output. The step of integrating parameters based on the emotional features, the speech prosody parameters, and the mixed timbre parameters to generate a comprehensive control parameter set through a dynamic interaction mechanism includes: The emotional features, speech prosody parameters and mixed timbre parameters are standardized by pre-stored association weight matrix in the dynamic interaction mechanism, and each parameter is converted into a normalized feature vector. The association weight matrix includes the association weight values ​​between each dimension of emotional features, each dimension of speech prosody parameters and each dimension of mixed timbre parameters. The standardized feature vectors are weighted and fused based on the weight values ​​in the correlation weight matrix to generate three-dimensional feature vectors. The three-dimensional feature vector is verified by the parameter coordination rules in the dynamic interaction mechanism. The verified three-dimensional feature vector is then transformed to generate a comprehensive set of control parameters, including emotion control coefficient, prosodic coordination coefficient and timbre adaptation coefficient. The integrated control parameter set is input into the speech synthesis engine, which synthesizes and outputs speech signals with corresponding emotional coloring in real time. 2.The method of claim 1, wherein the step of extracting the emotional features of the input text by the text processing module and generating the prosody parameters adapted to the emotional features of the text is characterized by, include: The sentiment analysis unit in the text processing module extracts multi-dimensional sentiment features from the input text. These multi-dimensional sentiment features include: sentiment keywords, sentiment modifiers, basic sentiment types, sentiment intensity, and sentiment expression priority. Based on the aforementioned multi-dimensional emotional features, corresponding speech prosody parameters are generated through a prosody generation unit.

3. The method for controlling emotion in speech synthesis according to claim 1, wherein the step of training the emotional timbre synthesis model is characterized in that, include: A training dataset was collected, which included multi-domain text, manually annotated text sentiment features, and corresponding speech reading samples. The text sentiment features included sentiment category, sentiment intensity reference value, and sentiment detail description. An end-to-end neural network model based on Transformer is constructed, which includes an input layer, a modulation layer, a decoder layer and a parameter optimization layer. The modulation layer contains multiple stacked Transformer encoder blocks for encoding input features, and the decoder layer contains multiple stacked Transformer decoder blocks for outputting an initial timbre parameter vector. Using the training dataset, the model was iteratively trained using the Adam optimizer and the MSE loss function; After the model is trained, the model parameters and the pre-constructed correlation weight matrix of emotion dimension-prosodic features-timbre acoustic parameters are saved to form an emotion timbre synthesis model that can be directly called.

4. The speech synthesis emotion modulation method according to claim 3, characterized in that, The steps for constructing the correlation weight matrix of the emotional dimension, prosodic features, and timbre acoustic parameters include: The three-dimensional feature dimensions of the matrix are determined, including the emotional dimension, the prosodic feature dimension, and the timbre acoustic parameter dimension; Multiple sets of sample data were collected. Each set of data included the sentiment dimension labels of the text, prosodic feature parameters and corresponding timbre acoustic parameters, and was divided into training set and validation set according to a preset ratio. The feature association analysis algorithm is used to calculate the correlation degree between the emotion dimension and prosodic features, the prosodic features and timbre acoustic parameters, and the emotion dimension and timbre acoustic parameters, and the initial weight values ​​between each dimension are determined based on the correlation degree. Construct a three-dimensional matrix structure with emotional dimension as rows, prosodic features as columns, and timbre and acoustic parameters as layers, and fill in the initial weight values ​​to form an initial correlation weight matrix; The initial association weight matrix is ​​validated and adjusted using the validation set to obtain the final association weight matrix.

5. The speech synthesis emotion modulation method according to claim 1, characterized in that, the step of inputting the comprehensive control parameter set into the speech synthesis engine, synthesizing and outputting a speech signal with corresponding emotional color in real time, is as follows: include: The speech synthesis engine parses the comprehensive control parameter set, extracts the emotion control coefficient, prosodic coordination coefficient and timbre adaptation coefficient, and maps them into acoustic feature parameters that can be recognized within the engine. Based on acoustic feature parameters, the engine's built-in emotion rendering module is invoked. This module adjusts the emotion tendency weight of the acoustic feature parameters according to the emotion control coefficient, so that the fluctuations of the fundamental frequency curve and the brightness of the spectral envelope match the intensity of the emotional color. The prosody generation unit of the engine optimizes the duration distribution and pause position in the acoustic feature parameters by combining the prosody coordination coefficient, and calibrates the spectral envelope and formant characteristics in the acoustic feature parameters by using the timbre adaptation coefficient. The speech synthesis engine converts the processed acoustic feature parameters into the original speech signal, and then performs noise reduction, smoothing and volume equalization processing through the signal optimization module to generate a speech signal with corresponding emotional color.

6. A speech synthesis emotion modulation system, characterized in that, include: The data acquisition and control module is used to extract the emotional features of the input text through the text processing module and generate speech prosody parameters that match the emotional features of the text. The timbre parameter synthesis module, used to input the emotional features and the speech prosody parameters into a pre-trained emotional timbre synthesis model to synthesize a mixed timbre parameter set, includes the following steps: The emotional features and speech prosody parameters are input into a pre-trained emotional timbre synthesis model to generate initial values ​​of core acoustic parameters for the mixed timbre. The core acoustic parameters include fundamental frequency range, timbre brightness coefficient, harmonic distortion, and formant frequency. The initial values ​​of the core acoustic parameters are dynamically corrected by the parameter optimization module in the emotion-rhythm-timbre association model, and the final set of mixed timbre parameters is output. The modulation parameter output module, for using the steps of the emotional features, the speech prosody parameters, and the mixed timbre parameter set, includes: The emotional features, speech prosody parameters and mixed timbre parameters are standardized by pre-stored association weight matrix in the dynamic interaction mechanism, and each parameter is converted into a normalized feature vector. The association weight matrix includes the association weight values ​​between each dimension of emotional features, each dimension of speech prosody parameters and each dimension of mixed timbre parameters. The standardized feature vectors are weighted and fused based on the weight values ​​in the correlation weight matrix to generate three-dimensional feature vectors. The three-dimensional feature vector is verified by the parameter coordination rules in the dynamic interaction mechanism. The verified three-dimensional feature vector is then transformed to generate a comprehensive set of control parameters, including emotion control coefficient, prosodic coordination coefficient and timbre adaptation coefficient. The speech signal generation module is used to input the comprehensive control parameter set into the speech synthesis engine, synthesize and output speech signals with corresponding emotional coloring in real time.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the speech synthesis emotion modulation method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the speech synthesis emotion modulation method according to any one of claims 1 to 5.

Images