Multi-modal fusion driven emotion perception enhanced TTS speech synthesis method

Abstract

The invention provides a TTS speech synthesis method for emotion perception enhancement under multi-modal fusion driving, and the method comprises the following steps: S1, carrying out the collection and preprocessing of multi-modal data, the multi-modal data comprising text data, speech data and facial expression data; s2, extracting and analyzing emotion features; s3, emotion perception speech synthesis model training; s4, speech synthesis and post-processing; s5, performing model evaluation and optimization; by collecting and analyzing multi-mode data such as texts, voices and facial expressions, emotion features can be captured more comprehensively and accurately, complementary information among different modes is fully mined through application of a multi-mode fusion network and a collaborative attention mechanism, the emotion expression of the synthesized voices is closer to real emotions, and the user experience is improved. And the precision of emotion perception is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a TTS speech synthesis method with enhanced emotion perception driven by multimodal fusion. Background Art

[0002] In the current field of speech synthesis, traditional TTS technology mainly generates speech based on text input and can only simulate simple changes in intonation. The generated speech often lacks real emotions and rich expressiveness, sounds mechanical and stiff, and is difficult to meet users' growing emotional interaction needs.

[0003] Although some methods have attempted to incorporate emotional information into speech synthesis, most of them rely solely on single-modal data, such as adjusting speech synthesis parameters based solely on sentiment analysis results based on text. However, human emotional expression is multimodal, and single-modal data is difficult to fully capture and accurately reflect emotional characteristics, resulting in obvious deficiencies in the accuracy and delicacy of emotional expression in synthesized speech. In addition, in different application scenarios, speech synthesis needs to adapt to diverse needs. For example, in intelligent customer service scenarios, different service attitudes must be accurately conveyed. In the field of virtual anchors, rich and diverse character emotions need to be presented. Existing speech synthesis technology performs poorly in generalization ability and cannot flexibly respond to complex and changing application scenarios, limiting its application effect and user experience in actual scenarios.

[0004] After searching, the applicant retrieved the closest comparative documents as follows: Multimodal speech synthesis method, device, equipment and storage medium with application number 2021107384248, which discloses a multimodal speech synthesis method, device, equipment and storage medium, the method including: obtaining and preprocessing original voice data and original EEG data to obtain voice data and EEG data; inputting the voice data and EEG data into a preset autoencoder for variational autoencoding to obtain implicit representation; performing probability calculation on the implicit representation to obtain a prior distribution; according to the preset Bayesian formula, multimodal deep representation learning is performed on the prior distribution to obtain the posterior distribution, and according to the preset expectation-maximum algorithm, the posterior distribution is subjected to mixed Gaussian processing; according to the processed prior distribution and posterior distribution, the implicit representation is parameter reconstructed to generate emotional speech; the present invention processes EEG data and voice data to extract emotional information to synthesize emotional speech, thereby improving the naturalness of the synthesized speech and the friendliness of human-computer interaction.

[0005] The above-mentioned method only extracts and synthesizes emotional information from EEG data and speech data. However, the above-mentioned modal data still cannot fully capture and accurately reflect emotional characteristics, resulting in obvious deficiencies in the accuracy and delicacy of emotional expression in the synthesized speech.

[0006] In summary, it is necessary to provide a new technical solution to solve the above technical problems. Summary of the Invention

[0007] This application provides a TTS speech synthesis method with enhanced emotion perception driven by multimodal fusion, comprising the following steps:

[0008] S1: collecting and preprocessing multimodal data, where the multimodal data includes text data, voice data, and facial expression data;

[0009] S2: emotional feature extraction and analysis;

[0010] S3: emotion-aware speech synthesis model training;

[0011] S4: Speech synthesis and post-processing;

[0012] S5: Perform model evaluation and optimization.

[0013] As a preferred solution, in S1, regarding the collection and preprocessing of text data, text information is collected from multiple text data sources, the collected text information is cleaned, and then vectorized using a specific model after cleaning.

[0014] As a preferred solution, after cleaning, vectorization is performed using a specific model: encoding is performed using a pre-trained language model based on the Transformer architecture. The specific formula is:

[0015] T=[t1,t2,…t n ], T represents the text sequence, and the text vector is obtained after encoding by the pre-trained model Each vector The calculation is obtained through the transformation of the model's multi-layer self-attention mechanism and feedforward neural network layer.

[0016] As a preferred solution, after the text information is vectorized, an adversarial training mechanism is introduced to define a discriminator D to distinguish the real text vector from the perturbed text vector; a generator G (i.e., text encoder) is defined to generate text vectors that can deceive the discriminator D, and the loss function L of the discriminator D is minimized. D And maximize the loss function L of the generator G G To conduct adversarial training, the formula is as follows:

[0017]

[0018] Among them, E represents the mathematical expectation, which is the average value of a random variable under its probability distribution; P data (V T ) is the distribution of true text vectors, P gen (VT ) is the text vector distribution generated by the generator.

[0019] As a preferred solution, in S1, regarding the collection and preprocessing of voice data, professional recording equipment is used or voice samples are obtained from a voice database, the collected voice signal is pre-emphasized to enhance the high frequency part, then frame division, MFCC coefficient calculation and noise reduction are performed.

[0020] As a preferred solution, the formula for pre-emphasizing the collected voice signal to enhance the high frequency part is:

[0021] y(n) = x(n) - αx(n-1), where x(n) is the original voice signal, y(n) is the pre-emphasized signal, and α is the pre-emphasis coefficient, taking a value between 0.9 and 1.0.

[0022] As a preferred solution, the specific steps of frame division operation are:

[0023] Let the frame length be L and the frame shift be S, then the i-th frame voice signal s i is i = [x(iS), x(iS-1), …, x(iS+L-1)], and for each frame of voice, the mel frequency cepstral coefficient is calculated.

[0024] Step 1: Get the frequency spectrum X(k) of the voice signal by fast Fourier transform;

[0025] Step 2: Calculate the energy output E m of the mel filter bank, the formula is where H m (k) is the frequency response of the m-th mel filter, and N is the number of FFT points.

[0026] Step 3: Get the mel frequency cepstral coefficient by discrete cosine transform, let c n be the n-th MFCC coefficient, then where M is the number of mel filters.

[0027] As a preferred solution, the specific steps of noise reduction are:

[0028] A speech enhancement model is constructed using a convolutional neural network, the noisy speech signal spectrum is taken as input, features are extracted through multiple convolutional layers and pooling layers of the convolutional neural network, and the enhanced speech spectrum is obtained through the full connection layer and the activation function.

[0029] Let the noisy speech spectrum be I, the enhanced speech spectrum be O, and the parameters of the CNN model be θ CNN , then O = f CNN (I, θ CNN ), where fCNN It is the forward propagation function of the CNN model.

[0030] As a preferred solution, in S1, regarding the collection and preprocessing of facial expression data, a facial expression image sequence is obtained using a camera or a video database, and face detection and key point positioning are performed on the facial expression image sequence. For each detected face, the coordinate information of its key feature points is extracted. Let the face image be F, and the feature point coordinate sequence obtained after key point positioning is P=[(x1, y1), (x2, y2), ..., (x k ,y k )], use a three-dimensional convolutional neural network to process the image sequence, extract facial expression features, and obtain the facial expression feature vector V F .

[0031] As a preferred solution, the three-dimensional convolutional neural network is used to process the image sequence as follows: the image sequence is arranged in time order into a three-dimensional tensor T F =[F1,F2,…,F T ], where F t is the face image of the t-th frame, T is the number of frames in the image sequence, and the three-dimensional convolutional neural network extracts spatiotemporal features through convolution operations in the three dimensions of time, width, and height. Let the facial expression feature vector obtained after processing by the three-dimensional convolutional neural network be V F , then V F =f 3D-CNN (T F ,θ 3D-CNN ), where f 3D-CNN is the forward propagation function of the three-dimensional convolutional neural network model, θ 3D-CNN is the set of parameters of the model.

[0032] As an optimal solution, a multi-scale feature fusion mechanism is introduced. In the 3D convolutional neural network model, convolution kernels and pooling layers of different sizes are set to extract facial features of different scales. These multi-scale features are fused through the feature fusion layer. The feature maps of different scales are denoted as F1, F2, ..., F s , the fused feature map is F fusion , then F fusion =g(F1,F2,…,F s ), where g is the feature fusion function, which uses a simple splicing operation or a fusion method based on the attention mechanism. The formula is as follows:

[0033]

[0034] where w i is the weight calculated by the attention mechanism, satisfying

[0035] As a preferred solution, S2 includes the following steps:

[0036] S21: Using the text vector preprocessed by S1, build a deep network model based on recurrent neural network and attention mechanism to obtain the sentiment feature vector V of the text T_emo :

[0037] S22: Using the speech data preprocessed by S1, a hybrid model consisting of a convolutional neural network and a long short-term memory network is constructed to obtain the speech emotion feature vector V S_emo ;

[0038] S23: The obtained text sentiment feature vector V is obtained based on the method of multimodal fusion network T_emo , speech emotion feature vector V S_emo , facial expression feature vector V F Fusion is performed to obtain a joint feature vector V joint ;

[0039] S24: The joint eigenvector V is transformed by nonlinear transformation joint Further processing and fusion are performed to obtain the fused emotional feature vector V fusion .

[0040] As a preferred solution, the step S21 includes the following steps:

[0041] S211: The text vectors are sequentially input into the deep network model. The deep network model models the sequence information of the text. The hidden layer state calculation formula is: h t =tanh(W hh h t-1 +W xh x t +b h ), where h t is the hidden layer state at time t, h t-1 is the hidden layer state at the previous moment, x t is the input text vector at time t, W hh is the weight matrix, b h is the bias term;

[0042] S212: Introduce the attention mechanism to determine the importance of each word for emotional expression by calculating the correlation score between the hidden layer state at each time step and a query vector q. The correlation score calculation formula is: t =v T tanh(W he h t +b e ), where e tis the correlation score at time t, v, W he , b e is a learnable parameter;

[0043] S213: Perform softmax normalization on the relevance score to obtain the attention weight α t :

[0044]

[0045] S214: Perform weighted summation on the hidden layer states according to the attention weights to obtain the sentiment feature vector V of the text T_emo :

[0046]

[0047] As a preferred solution, the S22 includes the following steps:

[0048] S221: The speech features are extracted through a convolutional neural network layer. The convolutional neural network layer consists of multiple convolution kernels. Each convolution kernel slides in the time and frequency dimensions to extract features of different local areas. The output calculation formula of a convolution layer is:

[0049]

[0050] Among them, F CNN is the output feature map of the convolutional neural network layer, w ij is the convolution kernel weight, x i,j is the local area of ​​the input speech feature, b is the bias term, C is the number of convolution kernels, K is the size of the local area, and f is the activation function (such as ReLU);

[0051] S222: The output of the convolutional neural network layer is flattened and then input into the long short-term memory layer, which is used to capture the time series information of speech features;

[0052] S223: The output after processing by the long short-term memory layer passes through a fully connected layer and a softmax function to predict the emotion category and intensity of the speech, and obtain the speech emotion feature vector V S_emo .

[0053] As a preferred solution, the step S23 includes the following steps:

[0054] V joint Input into a multilayer perceptron, which contains multiple hidden layers, and transforms the joint feature vector V through nonlinear transformation. joint For further processing and fusion, the output of a hidden layer is calculated as: h l+1 =σ(W1h l +bl ), where: h l+1 is the output of the l+1th layer, h l is the input of the lth layer, W1 is the weight matrix, b l is the bias term, σ is the activation function, and after multi-layer perceptron processing, the fused emotional feature vector V is obtained. fusion .

[0055] As a preferred solution, S3 includes the following steps:

[0056] S31: Model architecture construction and initialization;

[0057] S32: Model training and optimization loop.

[0058] As a preferred solution, the S31 includes a multimodal feature fusion layer, which combines the text encoding features with the V fusion During fusion, an adaptive modal fusion network is used. The adaptive modal fusion network is built based on a dynamic graph convolutional network. When updating node features at each layer of the dynamic graph convolutional network, the calculation formula is as follows:

[0059]

[0060] in, is the feature representation of node v at layer l, N(v) is the set of neighbor nodes of node v, and W l (e uv ) is based on the edge e uv The weight matrix of the dynamic adjustment of the weight, b l is the bias term and σ is the activation function.

[0061] As a preferred solution, in S32, a reward function is defined, which comprehensively considers the factors of the emotional expression accuracy, speech quality and semantic consistency of the generated speech with the text; the emotional expression accuracy is measured by the degree of matching between the classification result of the generated speech by the pre-trained emotion classifier and the target emotion; the speech quality is evaluated by some speech quality evaluation indicators; the semantic consistency is judged by calculating the similarity between the semantic vector of the generated speech and the semantic vector of the text.

[0062] As a preferred solution, the text-emotion features are compared with the generated speech features. At the same time, the text-emotion features and speech features under different emotion categories are compared respectively. For the text-emotion features x under the same emotion category, t and speech features x s , calculate their cosine similarity sim(x t ,x s ), and optimized by contrastive loss function:

[0063] The calculation formula of cosine similarity is as follows:

[0064]

[0065] Among them, τ is the temperature parameter, i represents the sample pairs under the same emotion category, and j represents all samples;

[0066] The formula for optimization through contrast loss function is as follows:

[0067] The loss function of the discriminator is:

[0068]

[0069] The loss function of the generator is:

[0070]

[0071] Among them, x real is a real speech sample, x gen is the generated speech sample, P real (x) and P gen (x) are the distributions of real speech and generated speech respectively, and D is the discriminator.

[0072] As a preferred solution, S4 includes the following steps:

[0073] S41: preprocessing the text to be synthesized;

[0074] S42: Input the preprocessed text into the Transformer-based deep semantic coding model to obtain the deep semantic coding representation H of the text text ;

[0075] S43: The fused emotional feature vector V fusion Perform decoupling and refinement analysis to obtain the refined sentiment component vector;

[0076] S44: Refined sentiment component vector and text deep semantic encoding representation H text Fusion is performed to obtain the fused encoding representation H text-emo ;

[0077] S45: Use the decoder based on long short-term memory network and attention mechanism to represent H according to the fused code text-emo Generate spectral features in a targeted manner;

[0078] S46: Enhance the generated spectral features using multimodal information.

[0079] As a preferred solution, in the S42, the deep semantic encoding model captures the semantic information of the text from different angles through the multi-head attention mechanism, wherein the attention calculation of each head is as follows:

[0080]

[0081] wherein Q, K, and V are query, key, and value matrices, which are obtained by linear transformation of the word vector and position encoding of the text, d k is the dimension of the key vector, and the attention mechanisms of different heads focus on different local and global information of the text. After processing through multiple Transformer layers, the deep semantic encoding representation H text of the text is obtained.

[0082] As a preferred solution, in the S44, a fusion method based on a gating mechanism is used to calculate the influence weight of each sentiment component on the encoding of the text. Let the sentiment component vector be The weight calculation is as follows:

[0083]

[0084] wherein W i and b i are learnable parameters, and σ is a sigmoid activation function. The weighted sentiment component vector is added to H text to obtain the fused encoding representation H text-emo .

[0085] As a preferred solution, in the S45, at each decoding time step, the decoder focuses on different parts of the fused encoding representation H text-emo according to the current hidden state and attention mechanism. The attention weight calculation is as follows:

[0086]

[0087] e t =v T tanh(W a [h t-1 ,H test-emo ]+b a )

[0088] wherein a t is the attention weight, h t-1 is the hidden state of the decoder at the previous time, W a and b a are learnable parameters, and v is the attention vector. Through the attention mechanism, the decoder can generate spectral features according to the text and sentiment information.

[0089] As a preferred solution, other modal information related to text and emotion is integrated in S46; and a method based on a graph convolutional network is used to update the spectral features.

[0090] As a preferred solution, S5 includes the following steps:

[0091] S51: Construction of multi-dimensional evaluation indicators;

[0092] S52: Optimization strategy based on reinforcement learning;

[0093] S53: Model parameter fine-tuning and updating.

[0094] As a preferred solution, S51 includes the following steps:

[0095] S511: Build a large-scale emotional speech dataset covering multiple emotion categories and emotional expressions of varying intensity. This dataset contains natural language text and high-quality speech samples corresponding to the natural language text, with each sample professionally annotated and verified for emotion.

[0096] S512: Use deep learning technology to train a highly accurate sentiment classifier. This uses a Transformer-based sentiment classification model. At the model's input layer, speech samples are converted into mel-spectrograms and feature extracted using a convolutional neural network. The extracted features are then fed into the Transformer encoder for sentiment classification.

[0097] S512: For the generated speech to be evaluated, input it into the trained emotion classifier to obtain the predicted emotion category and intensity. At the same time, compare it with the emotion label of the corresponding reference speech and calculate the root mean square error of the emotion category accuracy and intensity difference between the two. The formula is as follows:

[0098]

[0099] Where N is the number of samples, y i and are the emotion category labels of the reference speech and the generated speech, respectively. I is the indicator function, which is 1 when the two are equal and 0 otherwise. i and are the emotion intensity values ​​of the reference speech and the generated speech respectively.

[0100] A deep learning-based speech quality assessment model, a perceptual speech quality assessment tool, and a short-term objective intelligibility assessment tool are introduced. The generated speech to be evaluated is input into the deep learning-based speech quality assessment model, the perceptual speech quality assessment tool, and the short-term objective intelligibility assessment tool, respectively, to obtain speech quality assessment scores in multiple dimensions. These scores are then combined into a comprehensive speech quality assessment indicator through weighted averaging:

[0101] Q total =α×Q GAN +β×PESQ+γ×STOI

[0102] Among them, α, β, γ are weight coefficients set according to the importance of different indicators; Q GAN represents the speech quality assessment score obtained by the deep learning-based quality assessment model; PESQ represents the speech quality assessment score obtained by the Perceptual Speech Quality Assessment Tool; STOI represents the speech quality assessment score obtained by the Short-Time Objective Intelligibility Assessment Tool;

[0103] Using semantic parsing technology in natural language processing, the generated speech is converted into text through an automatic speech recognition system, and a semantic parse tree is constructed. At the same time, a semantic parse tree is also constructed for the original input text. The semantic difference is measured by calculating the edit distance between the two trees. The smaller the edit distance, the higher the semantic consistency. The edit distance is calculated using a dynamic programming algorithm. For two semantic parse trees T1 and T2, d(i,j) is defined as the edit distance between the first i nodes of T1 and the first j nodes of T2. The recursive formula is as follows:

[0104] d(i,j)=min(d(i-1,j)+1,d(i,j-1)+1,d(i-1,j-1)+c(i,j))

[0105] Where c(i,j) is 0 when the i-th node of T1 is the same as the j-th node of T2, and 1 otherwise;

[0106] A semantic matching model based on deep learning is used for auxiliary evaluation. The semantic matching model based on deep learning is built based on a bidirectional long short-term memory network and an attention mechanism. After encoding the original text and the text converted from speech separately, the semantic similarity score between the two is calculated through the attention mechanism. In the bidirectional long short-term memory network, for the input text sequence x = [x1, x2, ..., x n ], its hidden layer state is calculated as follows:

[0107]

[0108] Then the semantic similarity score S is calculated through the attention mechanism, and combined with the edit distance indicator to obtain the final semantic consistency evaluation result.

[0109] As a preferred solution, in the S52, the reward function R is constructed by integrating the above multi-dimensional evaluation indicators, and its formula is as follows:

[0110] R=α×(1-RMSE emo )+β×Q total +γ×(1-ED semantic )+δ×S semantic

[0111] Where, α, β, γ, δ are weight coefficients set according to the importance of different indicators, RMSE emo is the root mean square error of the intensity of emotion, Q total is the comprehensive speech quality evaluation indicator, ED semantic is the edit distance of the semantic parsing tree, S semantic is the similarity score based on the deep learning semantic matching model.

[0112] The proximal policy optimization algorithm is adopted, in the training process, the model generates actions according to the current state, and then obtains reward feedback according to the reward function; through continuous iteration training, the model learns the optimal action strategy to maximize the long-term cumulative reward, so as to realize the optimization of the speech synthesis model, in the PPO algorithm, the parameters of the policy network are updated by calculating the advantage function and the policy gradient, the formula is as follows:

[0113] A t =R t -V(s t )

[0114] g=Ε t (▽ θ logπ θ (a t |s t )A t )

[0115] Where, A t is the advantage function, R t is the cumulative reward, V(s t ) is the value function estimate, π θ (a t |s t ) is the policy network, θ is the parameter of the policy network, g is the policy gradient; by continuously updating the parameters of the policy network, the model can better balance the emotional expression, speech quality and semantic consistency when generating speech.

[0116] As a preferred solution, in S53, the automatic differentiation technology is used to calculate the gradient of the model parameters with respect to the evaluation index; in the deep learning framework, the gradient value of each parameter is calculated by the back propagation algorithm; then, a gradient-based parameter importance evaluation method is used, such as calculating the norm or absolute value of the parameter gradient, to sort the parameters; at the same time, the parameters are sparsely processed in combination with the L1 regularization method, and parameters with smaller absolute values ​​are regarded as unimportant parameters and ignored, and the focus is on parameters that are sensitive to changes in the evaluation index.

[0117] According to the selected key parameters, the adaptive learning rate adjustment method is used to update the parameters. The Adagrad algorithm is used, and the learning rate update formula is as follows:

[0118]

[0119] Among them, r t is the learning rate of step t, r0 is the initial learning rate, g i is the parameter gradient of the i-th step, and ε is a small positive number to prevent the denominator from being 0. During the update process, for important parameters, the learning rate should be increased to speed up the convergence. For parameters that are prone to overfitting, the learning rate should be reduced for fine-tuning. At the same time, the parameter momentum term, such as in the Adadelta algorithm, calculates the moving average gradient and moving average square gradient of the parameters to give the parameter update a certain inertia, avoid frequent oscillations near the local optimal solution, and improve the optimization efficiency of the model.

[0120] This application has the following advantages:

[0121] 1. By collecting and analyzing multimodal data such as text, speech, and facial expressions, this invention can capture emotional characteristics more comprehensively and accurately. The application of multimodal fusion networks and collaborative attention mechanisms fully taps into the complementary information between different modalities, making the emotional expression of synthesized speech closer to real emotions and greatly improving the accuracy of emotional perception.

[0122] 2. During model training, we comprehensively consider factors such as emotional expression accuracy, voice quality, and semantic consistency, using multiple technologies such as reinforcement learning, cross-modal contrastive learning, and generative adversarial networks. These technologies work together to effectively improve the clarity and naturalness of synthesized speech, bringing it closer to natural human speech and significantly enhancing speech quality.

[0123] 3. By utilizing large-scale general-purpose datasets for transfer learning, the model can better adapt to speech synthesis tasks in different fields and scenarios. Whether it is intelligent customer service, virtual anchors, or audiobook production, the speech synthesis model of the present invention can work stably and efficiently, greatly broadening the application scope of the technology.

[0124] 4. During the speech synthesis process, the synthesized text is subjected to in-depth semantic analysis, including lexical, syntactic, and semantic role labeling. At the same time, during the model evaluation phase, the semantic consistency between the synthesized speech and the original text is accurately measured through the semantic parse tree and deep learning semantic matching model, ensuring that the synthesized speech accurately conveys the meaning of the text and avoids semantic deviation. BRIEF DESCRIPTION OF THE DRAWINGS

[0125] Figure 1 This is a flowchart of the application; DETAILED DESCRIPTION

[0126] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. It should be noted that the specific embodiments described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0127] Example 1:

[0128] like Figure 1 As shown, this embodiment provides a TTS speech synthesis method with enhanced emotion perception driven by multimodal fusion:

[0129] S1: collecting and preprocessing multimodal data, where the multimodal data includes text data, voice data, and facial expression data;

[0130] Regarding the collection and preprocessing of text data, we first obtain text information from multiple different sources, including but not limited to online resources, document materials, databases, multimedia resources, etc. We then use web crawler technology combined with natural language processing tools (such as NLTK and Spacy) to clean the collected text information, removing HTML tags and non-text characters to improve data quality. After cleaning, we use a specific model to vectorize the cleaned text into a digital vector form that can be processed by computers, so that it can be processed by subsequent machine learning or deep learning models.

[0131] Regarding the collection and preprocessing of voice data, professional recording equipment is used or voice samples are obtained from a voice database. The collected voice signals are pre-emphasized to enhance the high-frequency part, and then framed, MFCC coefficients are calculated, and noise reduction is performed.

[0132] Regarding the collection and preprocessing of facial expression data, a camera or video database is used to obtain facial expression image sequences, and face detection and key point positioning are performed on the facial expression image sequences. After detection and positioning, a three-dimensional convolutional neural network (3D-CNN) is used to extract features, and a fusion mechanism is introduced. The above-mentioned face detection and key point positioning can be performed using deep learning-based face detection algorithms, such as YOLO, MTCNN, etc., and key point positioning models, such as Dlib, HRNet, etc.

[0133] S2: emotional feature extraction and analysis;

[0134] First, the text vectors are processed based on the recurrent neural network (RNN) and attention mechanism, and adversarial training is used to improve accuracy; a hybrid model (CNN-LSTM hybrid model) consisting of a convolutional neural network (CNN) and a long short-term memory (LSTM) network is constructed to process speech features, and transfer learning is used to enhance generalization ability; then, the emotional feature vectors of different modalities are fused, and a collaborative attention mechanism is introduced for optimization.

[0135] S3: emotion-aware speech synthesis model training;

[0136] First, build the model architecture and use the adaptive modal fusion network (AMFN) to fuse multimodal features. Then, use the reinforcement learning algorithm to optimize, define the reward function, and introduce contrastive learning and generative adversarial network (GAN) structures.

[0137] S4: Speech synthesis and post-processing;

[0138] First, the text to be synthesized is preprocessed, then the preprocessed text is input into the semantic encoding model, and then the refined sentiment feature vectors are decoupled and fused. Then, a specific decoder is used to generate spectral features, which are enhanced with multimodal information and optimized with a generative adversarial network (GAN).

[0139] S5: Perform model evaluation and optimization;

[0140] First, build a dataset and classifier to evaluate emotional expression, voice quality, and semantic consistency; then build a reward function and optimize the model using a proximal policy optimization algorithm; finally, calculate the parameter gradient, screen key parameters, and update them.

[0141] Example 2:

[0142] This embodiment specifically describes S1:

[0143] 1. About Text Data

[0144] After cleaning, vectorization is performed using a specific model. Specifically, a pre-trained language model based on the Transformer architecture (such as BERT, RoBERTa, etc.) is used for encoding. The specific formula is:

[0145] T=[t1,t2,…t n ], T represents the text sequence, and the text vector is obtained after encoding by the pre-trained model Each vector The calculation is obtained through the transformation of the model's multi-layer self-attention mechanism and feedforward neural network layer; for example, in BERT, Here f BERT represents the forward propagation function of the BERT model, θBERT is the set of parameters of the model.

[0146] Preferably, in order to further improve the quality of text vector representation, an adversarial training mechanism can be introduced, specifically: define a discriminator D to distinguish between real text vectors and perturbed text vectors; define a generator G (i.e., text encoder) to generate text vectors that can deceive the discriminator D; and minimize the loss function L of the discriminator D. D And maximize the generator loss function L G To conduct adversarial training, the formula is as follows:

[0147]

[0148] Among them, E represents the mathematical expectation, which is the average value of a random variable under its probability distribution; P data (V T ) is the distribution of true text vectors, P gen (V T ) is the text vector distribution generated by the generator.

[0149] Through the above-mentioned adversarial training, the text vector representation can be made more robust and expressive.

[0150] 2. About Voice Data

[0151] 1. The formula for pre-emphasis processing on the collected speech signal to enhance the high-frequency part is:

[0152] y(n)=x(n)-αx(n-1), where x(n) is the original speech signal, y(n) is the pre-emphasized signal, and α is the pre-emphasis coefficient, which ranges from 0.9 to 1.0.

[0153] 2. The specific steps of the framing operation are:

[0154] Assume that the frame length is L and the frame shift is S, then the speech signal of the i-th frame s i For s i =[x(iS),x(iS-1),…,x(iS+L-1)], for each frame of speech, calculate the Mel-frequency cepstral coefficient (MFCC);

[0155] Step 1: Obtain the spectrum X(k) of the speech signal through fast Fourier transform (FFT);

[0156] Step 2: Calculate the energy output E of the Mel filter bank m , the formula is Among them H m (k) is the frequency response of the mth Mel filter, and N is the number of FFT points;

[0157] Step 3: Obtain Mel-frequency cepstral coefficients (MFCC) through discrete cosine transform (DCT), let c n is the nth MFCC coefficient, then Where M is the number of Mel filters.

[0158] 3. Specific steps for noise reduction:

[0159] Noise reduction can reduce noise interference in speech data;

[0160] A convolutional neural network (CNN) is used to build a speech enhancement model. The spectrogram of a noisy speech signal is used as input. Features are extracted through the CNN's multi-layer convolutional and pooling layers. The enhanced speech spectrogram is obtained through the fully connected layer and activation function.

[0161] Let the noisy speech spectrogram be I, the enhanced speech spectrogram be O, and the parameter of the CNN model be θ CNN , then O=f CNN (I,θ CNN ), where f CNN It is the forward propagation function of the CNN model.

[0162] 3. About Facial Expression Data

[0163] The specific steps for face detection and key point location on facial expression image sequences are as follows:

[0164] For each detected face, the coordinate information of its key feature points is extracted. Let the face image be F. After the key points are located, the feature point coordinate sequence is P = [(x1, y1), (x2, y2), ..., (x k ,y k )]; In order to extract facial expression features, the image sequence is processed using a three-dimensional convolutional neural network (3D-CNN);

[0165] Arrange the image sequence in time order into a three-dimensional tensor T F =[F1,F2,…,F T ], where F t is the face image of the tth frame, T is the number of frames in the image sequence, 3D-CNN extracts spatiotemporal features through convolution operations in three dimensions: time, width, and height. Let the facial expression feature vector obtained after 3D-CNN processing be V F , then V F =f 3D-CNN (T F ,θ 3D-CNN ), where f 3D-CNN is the forward propagation function of the 3D-CNN model, θ 3D-CNN is the set of parameters of the model.

[0166] Preferably, in order to improve the accuracy and stability of facial expression features, a multi-scale feature fusion mechanism is introduced. Specifically, in the 3D-CNN model, convolution kernels and pooling layers of different sizes are set to extract facial features of different scales, and then these multi-scale features are fused through the feature fusion layer; let the feature maps of different scales be F1, F2, ..., F s , the fused feature map is F fusion , then F fusion =g(F1,F2,…,F s ), where g is the feature fusion function, which can adopt a simple splicing operation or a fusion method based on the attention mechanism. The formula is as follows:

[0167]

[0168] Among them, w i is the weight calculated by the attention mechanism, satisfying

[0169] The introduction of the multi-scale feature fusion mechanism can enable the model to better capture subtle changes and global features of facial expressions.

[0170] Example 3:

[0171] This embodiment specifically describes S2:

[0172] The S2 comprises the following steps:

[0173] S21: Using the text vector preprocessed by S1, build a deep network model based on recurrent neural network (RNN) and attention mechanism to obtain the sentiment feature vector V of the text T_emo : More specifically:

[0174] S211: The text vector is input into RNN in sequence. RNN will model the sequence information of the text. The hidden layer state calculation formula is: h t =tanh(W hh h t-1 +W xh x t +b h ), where h t is the hidden layer state at time t, h t-1 is the hidden layer state at the previous moment, x t is the input text vector at time t, W hh is the weight matrix, b h is the bias term;

[0175] S212: Introduce attention mechanism, determine the importance of each word for sentiment expression by calculating the relevance score of each time step hidden layer state and a query vector q, the relevance score calculation formula is: e t = v T tanh(W he h t + b e ), where e t is the relevance score at time t, v, W he , b e are learnable parameters;

[0176] S213: Softmax normalization of the relevance score to get attention weight a t :

[0177]

[0178] S214: Weighted sum of the hidden layer state according to the attention weight to get the sentiment feature vector V T_emo of the text:

[0179]

[0180] S22: Use the preprocessed speech data S1, such as MFCC, to build a hybrid model (CNN-LSTM hybrid model) containing a convolutional neural network (CNN) and a long short-term memory (LSTM) network, to get the speech emotion feature vector V S_emo ; More specifically:

[0181] S221: Local feature extraction of speech features through CNN layer, the CNN layer is composed of multiple convolution kernels, each convolution kernel slides in time and frequency dimensions to extract features of different local regions, the output calculation formula of a convolution layer is:

[0182]

[0183] Where, F CNN is the output feature map of the CNN layer, w ij is the convolution kernel weight, x i,j is the local region of the input speech feature, b is the bias term, C is the number of convolution kernels, K is the size of the local region, and f is the activation function (such as ReLU);

[0184] S222: The output of the CNN is flattened and input to the LSTM layer, the LSTM layer is used to capture the time sequence information of the speech features, and its calculation process is similar to the RNN in the previous text sentiment analysis, but contains input gate, forget gate and output gate structures to better process long sequence information, which will not be described in detail herein.

[0185] S223: The output after LSTM layer processing passes through a fully connected layer and softmax function to predict the emotion category and intensity of the speech, and obtain the speech emotion feature vector V S_emo .

[0186] S23: The obtained text sentiment feature vector V is obtained based on the method of multimodal fusion network T_emo , speech emotion feature vector V S_emo , facial expression feature vector V F Fusion is performed to obtain a joint feature vector V joint ; Specifically: First, the text sentiment feature vector V T_emo , speech emotion feature vector V S_emo , facial expression feature vector V F Spliced ​​together to form a joint feature vector V joint , then V joint Input into a multi-layer perceptron (MLP), which contains multiple hidden layers and transforms the joint feature vector V through nonlinear transformation. joint For further processing and fusion; the output calculation formula of a hidden layer is: l+1 =σ(W1h l +b l )

[0187] Among them, h l+1 is the output of the l+1th layer, h l is the input of the lth layer, W1 is the weight matrix, b l is the bias term, and σ is the activation function (such as ReLU or tanh).

[0188] After MLP processing, the fused emotional feature vector V is obtained fusion .

[0189] Preferably, in S21, in order to improve the accuracy of text sentiment analysis, an adversarial training method is adopted; in addition to the conventional sentiment classifier, a discriminator is introduced. The sentiment classifier attempts to accurately classify the text sentiment, while the discriminator distinguishes between the real text sentiment features and the features generated by the classifier; through the adversarial training of the two, the performance of the sentiment classifier is continuously optimized.

[0190] Preferably, in order to better utilize the complementary information between different modalities, a collaborative attention mechanism between modalities is introduced in S23, the attention weights between different modal features are calculated, and the modal features are dynamically weighted and fused according to the weights to further optimize the fusion effect, so that the fused emotional feature vector can more accurately reflect the emotional content in the multimodal information.

[0191] Example 4:

[0192] This embodiment specifically describes S3:

[0193] The S3 comprises the following steps:

[0194] S31, model architecture construction and initialization;

[0195] This step includes a multimodal feature fusion layer: after combining the text encoding features with V fusion During fusion, an adaptive modal fusion network (AMFN) is used. AMFN is built on the dynamic graph convolutional network (DGCN) to achieve more accurate feature fusion. When updating node features at each layer of DGCN, the calculation formula is as follows:

[0196]

[0197] in, is the feature representation of node v at layer l, N(v) is the set of neighbor nodes of node v, and W l (e uv ) is based on the edge e uv The weight matrix of the dynamic adjustment of the weight, b l is the bias term and σ is the activation function.

[0198] This approach enables the model to automatically learn the optimal fusion method for different modalities in different emotional contexts.

[0199] S32: Model training and optimization loop;

[0200] A reinforcement learning algorithm is used to further optimize the model. Specifically, a reward function is defined that comprehensively considers the emotional expression accuracy, speech quality (such as clarity and naturalness) of the generated speech, and semantic consistency with the text; the emotional expression accuracy is measured by the degree of match between the classification results of the generated speech by the pre-trained emotion classifier and the target emotion; the speech quality is evaluated by some speech quality evaluation indicators (such as PESQ, STOI, etc.); the semantic consistency is judged by calculating the similarity between the semantic vector of the generated speech and the semantic vector of the text.

[0201] In the training process, a cross-modal contrastive learning mechanism is introduced: the text-emotion features are compared with the generated speech features, and the text-emotion features and speech features under different emotion categories are also compared. t and speech features x s , calculate their cosine similarity sim(x t ,x s ):

[0202]

[0203] where τ is the temperature parameter, i represents the sample pair under the same emotion category, and j represents all samples.

[0204] The loss function is optimized by comparison: a generative adversarial network (GAN) structure is introduced for adversarial training. In the training process, the generator tries to generate more realistic speech to deceive the discriminator, and the discriminator tries to improve the discrimination ability. Through this adversarial training, the quality and realism of the generated speech are further improved. The loss function of the discriminator is:

[0205]

[0206] The loss function of the generator is:

[0207]

[0208] where x real is a real speech sample, x gen is a generated speech sample, P real (x) and P gen (x) are the distributions of real speech and generated speech respectively, and D is the discriminator.

[0209] Example five:

[0210] This embodiment specifically describes S4:

[0211] The S4 includes the following steps:

[0212] S41: Preprocessing the text to be synthesized; including lexical analysis, syntax analysis and semantic role labeling operations, using natural language processing tools (such as StanfordCoreNLP, etc.) to identify the parts of speech such as nouns, verbs and adjectives in the text, and parse the sentence structure to determine the grammatical relationship of subject-predicate-object, etc. At the same time, through semantic role labeling, the semantic role of each word in the sentence is clarified, such as agent, patient, etc. For the text "Xiaoming happily ran to the playground", it can accurately identify that "Xiaoming" is the agent, "playground" is the target location, "run" is the action, and "happy" is the emotional description word.

[0213] S42: Input the preprocessed text into the deep semantic encoding model based on Transformer to obtain the deep semantic encoding representation H text of the text; specifically, the deep semantic encoding model captures the semantic information of the text from different angles through the multi-head attention mechanism, where the attention calculation of each head is as follows:

[0214]

[0215] where Q, K, V are query, key and value matrices, which are obtained by linear transformation of word vectors and position encoding of the text, d k is the dimension of key vectors, different heads of attention mechanism focus on different local and global information of the text, and after processing by multiple layers of Transformer layers, the deep semantic encoding representation H text of the text is obtained.

[0216] S43: decoupling and refining analysis is performed on the fused sentiment feature vector V fusion to obtain a refined sentiment component vector; specifically, principal component analysis (PCA) and other technologies are used to decompose V fusion into multiple main sentiment component vectors, such as happiness, sadness, anger, and other basic sentiment components, and each component vector represents the intensity information of one sentiment dimension.

[0217] S44: the above refined sentiment component vector is fused with the deep semantic encoding representation H text of the text to obtain a fused encoding representation H text-emo ; specifically, a fusion method based on a gating mechanism is adopted to calculate the influence weight of each sentiment component on the text encoding. Let the sentiment component vector be The weight is calculated as follows:

[0218]

[0219] where W i and b i are learnable parameters, and σ is a sigmoid activation function. The weighted sentiment component vector is added to H text to obtain the fused encoding representation H text-emo .

[0220] S45: a decoder based on LSTM and attention mechanism is used to generate mel spectrum features of speech according to the fused encoding representation H text-emo ; specifically:

[0221] At each decoding time step, the decoder focuses on different parts of the fused encoding representation H text-emo according to the current hidden state and attention mechanism, and the attention weight is calculated as follows:

[0222]

[0223] e t = v T tanh(W a [h t-1 , H test-emo ] + b a )

[0224] Among them, a t is the attention weight, h t-1 is the hidden state of the decoder at the previous moment, W a and b a is a learnable parameter and v is the attention vector. Through the attention mechanism, the decoder can generate spectral features in a targeted manner based on text and sentiment information.

[0225] S46: Enhance the generated spectral features using multimodal information; specifically:

[0226] Integrate other modal information related to text and emotion, such as the emotional features corresponding to facial expression image sequences and the dynamic features of body language. Using a graph convolutional network (GCN)-based approach, the spectral features are treated as nodes in the graph, and the other modal emotional features are used as node attributes. GCN is used to propagate and aggregate information on the graph to update the spectral features.

[0227] Preferably, a generative adversarial network (GAN) is used to optimize the spectral features. The generator in the GAN attempts to generate more realistic and emotionally consistent spectral features, while the discriminator distinguishes between real and generated spectral features. Through adversarial training between the two, the quality of the generated spectrum is continuously improved. The loss function of the discriminator is:

[0228]

[0229] The loss function of the generator is:

[0230]

[0231] Among them, x real is a real speech sample, x gen is the generated speech sample, P real (x) and P gen (x) are the distributions of real speech and generated speech respectively, and D is the discriminator.

[0232] Example 6:

[0233] This embodiment specifically describes S5:

[0234] S5 includes the following steps:

[0235] S51. Construction of multi-dimensional evaluation indicators; specifically:

[0236] S511: Construct a large-scale emotional speech dataset that covers multiple emotion categories (such as joy, sadness, anger, fear, calmness, etc.) and emotional expressions of different intensity levels. The dataset contains natural language text and high-quality speech samples corresponding to the natural language text. Each sample has been professionally annotated and verified for emotion. For text with joyful emotions, such as "I won the lottery today, I'm so happy!", there are speech samples recorded by professional actors and the emotion intensity is annotated between 0.8 and 1.

[0237] S512: Use deep learning technology to train a high-accuracy sentiment classifier, adopt a sentiment classification model based on the Transformer architecture, convert the speech sample into a mel-spectrogram at the input layer of the model, and extract features through a convolutional neural network (CNN), and then input the extracted features into a Transformer encoder for sentiment classification; preferably, during the training process, use a cross-entropy loss function and a large amount of emotional speech data for optimization, so that the model can accurately identify the emotion category and intensity in the speech.

[0238] S513: For the generated speech to be evaluated, input it into the trained emotion classifier to obtain the predicted emotion category and intensity. At the same time, compare it with the emotion label of the corresponding reference speech (i.e., the standard speech sample in the dataset) and calculate the root mean square error (RMSE) of the emotion category accuracy and intensity difference between the two. The formula is as follows:

[0239]

[0240] Where N is the number of samples, y i and are the emotion category labels of the reference speech and the generated speech, respectively. I is the indicator function, which is 1 when the two are equal and 0 otherwise. i and They are the emotional intensity values ​​of the reference speech and the generated speech, respectively. These two indicators are used to comprehensively evaluate the performance of the model in terms of emotional expression accuracy.

[0241] A deep learning-based speech quality assessment model is introduced. This model is based on the architecture of generative adversarial networks (GANs). The generator attempts to generate speech samples with the same quality characteristics as high-quality speech, and the discriminator is used to distinguish between real high-quality speech and generated speech. During the training process, the generator's ability to generate high-quality speech is improved by minimizing the discriminator's loss function.

[0242] The perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOl) evaluation tools are introduced. PESQ evaluates the distortion, noise and other factors of speech by simulating the auditory perception process of human ears, and its calculation process involves filtering, weighting and other operations on speech signals to better match the auditory characteristics of human ears. STOI focuses on evaluating the intelligibility of speech by calculating the correlation coefficient of the original speech and the processed speech in the short-time Fourier transform (STFT) domain to measure the intelligibility.

[0243] The generated speech to be evaluated is input into the above-mentioned quality evaluation model based on deep learning, PESQ and STOI evaluation tools, and multiple dimensions of speech quality evaluation scores are obtained. Then, these scores are combined into a comprehensive speech quality evaluation index through weighted average:

[0244] Q total = α × Q GAN + β × PESQ + γ × STOI

[0245] Where α, β, γ are weight coefficients set according to the importance of different indicators, determined through experiments and data analysis, which can comprehensively and accurately evaluate the quality of generated speech; Q GAN represents the speech quality evaluation score obtained by the quality evaluation model based on deep learning; P ESQ represents the speech quality evaluation score obtained by the perceptual speech quality evaluation tool; STOI represents the speech quality evaluation score obtained by the short-time objective intelligibility evaluation tool.

[0246] Using semantic parsing technology in natural language processing, the generated speech is converted into text through an automatic speech recognition (ASR) system, and a semantic parsing tree is constructed. The original input text is also constructed into a semantic parsing tree, and the semantic difference between the two trees is measured by calculating the edit distance. The smaller the edit distance, the higher the semantic consistency. The edit distance calculation can use dynamic programming algorithm. For two semantic parsing trees T1 and T2, define d(i,j) as the edit distance between the first i nodes of T1 and the first j nodes of T2, and its recursive formula is as follows:

[0247] d(i,j) = min(d(i-1,j) + 1, d(i,j-1) + 1, d(i-1,j-1) + c(i,j))

[0248] Where c(i,j) is 0 when the i-th node of T1 and the j-th node of T2 are the same, otherwise it is 1.

[0249] A semantic matching model based on deep learning is used for auxiliary evaluation. The model is built on a bidirectional long short-term memory network (BiLSTM) and an attention mechanism. After encoding the original text and the text after speech conversion respectively, the semantic similarity score between the two is calculated through the attention mechanism. In Bi-LSTM, for the input text sequence x = [x1, x2, ..., x n ], its hidden layer state is calculated as follows:

[0250]

[0251] Then the semantic similarity score S is calculated through the attention mechanism and combined with the edit distance indicator to obtain the final semantic consistency evaluation result.

[0252] S52. Optimization strategy based on reinforcement learning; specifically:

[0253] Combining the above multi-dimensional evaluation indicators to construct the reward function R, its formula is as follows:

[0254] R = α × (1-RMSE emo )+β×Q total +γ×(1-ED semantic )+δ×S semantic

[0255] Among them, α, β, γ, and δ are weight coefficients set according to the importance of different indicators, and RMSE emo is the root mean square error of sentiment intensity, Q total It is a comprehensive voice quality evaluation indicator, ED semantic is the edit distance of the semantic parse tree, S semantic It is the similarity score of the semantic matching model based on deep learning. By adjusting the weight coefficient, the performance of a certain aspect can be optimized according to actual needs.

[0256] Using the Proximal Policy Optimization (PPO) algorithm, during training, the model generates actions (such as adjusting the weights of certain neural network layers in the speech synthesis model, changing the way emotional features are integrated, etc.) based on the current state (including the parameters of the speech synthesis model, input text, and emotional features), and then obtains reward feedback based on the reward function. Through continuous iterative training, the model learns the optimal action strategy to maximize the long-term cumulative reward, thereby optimizing the speech synthesis model. In the PPO algorithm, the parameters of the policy network are updated by calculating the advantage function and policy gradient. The formula is as follows:

[0257] A t =R t -V(s t )

[0258] g=Εt (▽ θ logπ θ (a t |s t )A t )

[0259] Among them, A t is the advantage function, R t is the cumulative reward, V(s t ) is the estimated value of the value function, π θ (a t |s t ) is the policy network, θ is the parameter of the policy network, and g is the policy gradient; by continuously updating the parameters of the policy network, the model can better balance the emotional expression voice quality and semantic consistency when generating speech.

[0260] S53. Fine-tune and update model parameters; specifically:

[0261] Automatic differentiation technology is used to calculate the gradient of model parameters with respect to evaluation indicators. In a deep learning framework (such as PyTorch or TensorFlow), the gradient value of each parameter is calculated through the backpropagation algorithm. Then, gradient-based parameter importance evaluation methods are used, such as calculating the norm or absolute value of the parameter gradient to sort the parameters. At the same time, the parameters are sparsely processed in combination with the L1 regularization method, and parameters with smaller absolute values ​​are regarded as unimportant parameters and ignored, focusing on parameters that are sensitive to changes in evaluation indicators.

[0262] According to the selected key parameters, the adaptive learning rate adjustment method is used to update the parameters. The Adagrad algorithm is used, and the learning rate update formula is as follows:

[0263]

[0264] Among them, r t is the learning rate of step t, r0 is the initial learning rate, g i is the parameter gradient of the i-th step, and ε is a small positive number to prevent the denominator from being 0. During the update process, for important parameters, the learning rate should be increased to speed up the convergence. For parameters that are prone to overfitting, the learning rate should be reduced for fine-tuning. At the same time, the parameter momentum term, such as in the Adadelta algorithm, calculates the moving average gradient and moving average square gradient of the parameters to give the parameter update a certain inertia, avoid frequent oscillations near the local optimal solution, and improve the optimization efficiency of the model.

[0265] In summary, due to the adoption of the above technical solution, this application has the following advantages:

[0266] 1. By collecting and analyzing multimodal data such as text, speech, and facial expressions, this invention can capture emotional characteristics more comprehensively and accurately. The application of multimodal fusion networks and collaborative attention mechanisms fully taps into the complementary information between different modalities, making the emotional expression of synthesized speech closer to real emotions and greatly improving the accuracy of emotional perception.

[0267] 2. During model training, we comprehensively consider factors such as emotional expression accuracy, voice quality, and semantic consistency, using multiple technologies such as reinforcement learning, cross-modal contrastive learning, and generative adversarial networks. These technologies work together to effectively improve the clarity and naturalness of synthesized speech, bringing it closer to natural human speech and significantly enhancing speech quality.

[0268] 3. By utilizing large-scale general-purpose datasets for transfer learning, the model can better adapt to speech synthesis tasks in different fields and scenarios. Whether it is intelligent customer service, virtual anchors, or audiobook production, the speech synthesis model of the present invention can work stably and efficiently, greatly broadening the application scope of the technology.

[0269] 4. During the speech synthesis process, the synthesized text is subjected to in-depth semantic analysis, including lexical, syntactic, and semantic role labeling. At the same time, during the model evaluation phase, the semantic consistency between the synthesized speech and the original text is accurately measured through the semantic parse tree and deep learning semantic matching model, ensuring that the synthesized speech accurately conveys the meaning of the text and avoids semantic deviation.

[0270] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0271] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented with the help of software plus the necessary general hardware platform, and of course can also be implemented through hardware, but in many cases the former is a better implementation method; based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM / RAM, disk, CD-ROM), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0272] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are protected by this application.

Claims

1. A TTS speech synthesis method with enhanced emotion perception driven by multimodal fusion, characterized in that: The steps include: S1: Collect and preprocess multimodal data, which includes text data, voice data, and facial expression data; preprocess the facial expression data to obtain the facial expression feature vector V F ; S2: emotional feature extraction and analysis; S21: Using the text vector obtained from the text data in S1, build a deep network model based on recurrent neural network and attention mechanism to obtain the sentiment feature vector V of the text T_emo : S22: Using the speech data preprocessed by S1, a hybrid model consisting of a convolutional neural network and a long short-term memory network is constructed to obtain the speech emotion feature vector V S_emo ; S23: The obtained text sentiment feature vector V is obtained based on the method of multimodal fusion network T_emo , speech emotion feature vector V S_emo , facial expression feature vector V F Fusion is performed to obtain a joint feature vector V joint ; S24: The joint eigenvector V is transformed by nonlinear transformation joint Further processing and fusion are performed to obtain the fused emotional feature vector V fusion ; S3: Based on the sentiment feature vector V fusion Conduct emotion-aware speech synthesis model training; S4: Speech synthesis and post-processing; S5: Perform model evaluation and optimization.

2. The method for TTS speech synthesis with enhanced emotion perception driven by multimodal fusion according to claim 1, characterized in that: Regarding the collection and preprocessing of facial expression data, a facial expression image sequence is obtained using a camera or video database, and face detection and key point positioning are performed on the facial expression image sequence. For each detected face, the coordinate information of its key feature points is extracted. Let the face image be F. After key point positioning, the feature point coordinate sequence obtained is P = [(x1, y1), (x2, y2), ..., (x k ,y k )], use a three-dimensional convolutional neural network to process the image sequence, extract facial expression features, and obtain the facial expression feature vector V F .

3. The TTS speech synthesis method with enhanced emotion perception driven by multimodal fusion according to claim 1, characterized in that: The S21 includes the following steps: S211: The text vectors are sequentially input into the deep network model. The deep network model models the sequence information of the text. The hidden layer state calculation formula is: h t =tanh(W hh h t-1 +W xh x t +b h ), where h t is the hidden layer state at time t, h t-1 is the hidden layer state at the previous moment, x t is the input text vector at time t, W hh is the weight matrix, b h is the bias term; S212: Introduce the attention mechanism to determine the importance of each word for emotional expression by calculating the correlation score between the hidden layer state at each time step and a query vector q. The correlation score calculation formula is: t =v T tanh(W he h t +b e ), where e t is the correlation score at time t, v, W he , b e is a learnable parameter; S213: Perform softmax normalization on the relevance score to obtain the attention weight α t : S214: Perform weighted summation on the hidden layer states according to the attention weights to obtain the sentiment feature vector V of the text T_emo :

4. The method for TTS speech synthesis with enhanced emotion perception driven by multimodal fusion according to claim 1, characterized in that: The S22 includes the following steps: S221: The speech features are extracted through a convolutional neural network layer. The convolutional neural network layer consists of multiple convolution kernels. Each convolution kernel slides in the time and frequency dimensions to extract features of different local areas. The output calculation formula of a convolution layer is: Among them, F CNN is the output feature map of the convolutional neural network layer, w ij is the convolution kernel weight, x i,j is the local area of ​​the input speech feature, b is the bias term, C is the number of convolution kernels, K is the size of the local area, and f is the activation function; S222: The output of the convolutional neural network layer is flattened and then input into the long short-term memory layer, which is used to capture the time series information of speech features; S223: The output after processing by the long short-term memory layer passes through a fully connected layer and a softmax function to predict the emotion category and intensity of the speech, and obtain the speech emotion feature vector V S_emo .

5. The TTS speech synthesis method with enhanced emotion perception driven by multimodal fusion according to claim 1, characterized in that: The S23 includes the following steps: V joint Input into a multilayer perceptron, which contains multiple hidden layers, and transforms the joint feature vector V through nonlinear transformation. joint For further processing and fusion, the output of a hidden layer is calculated as: h l+1 =σ(W1h l +b l ), where: h l+1 is the output of the l+1th layer, h l is the input of the lth layer, W1 is the weight matrix, b l is the bias term, σ is the activation function, and after multi-layer perceptron processing, the fused emotional feature vector V is obtained. fusion .

6. The method for TTS speech synthesis with enhanced emotion perception driven by multimodal fusion according to claim 1, characterized in that: The S3 comprises the following steps: S31: Model architecture construction and initialization; Contains a multimodal feature fusion layer that combines text encoding features with V fusion During fusion, an adaptive modal fusion network is used. The adaptive modal fusion network is built based on a dynamic graph convolutional network. When updating node features at each layer of the dynamic graph convolutional network, the calculation formula is as follows: in, is the feature representation of node v at layer l, N(v) is the set of neighbor nodes of node v, and W l (e uv ) is based on the edge e uv The weight matrix of the dynamic adjustment of the weight, b l is the bias term, σ is the activation function; S32: Model training and optimization loop; Define a reward function that comprehensively considers the emotional expression accuracy, speech quality, and semantic consistency of the generated speech with the text; emotional expression accuracy is measured by the degree of match between the classification results of the generated speech by the pre-trained sentiment classifier and the target emotion; speech quality is evaluated by speech quality evaluation indicators; semantic consistency is judged by calculating the similarity between the semantic vector of the generated speech and the semantic vector of the text. The text-emotional features are compared with the generated speech features. At the same time, the text-emotional features and speech features under different emotion categories are compared respectively. For the text-emotional features x under the same emotion category, t and speech features x s , calculate their cosine similarity sim(x t ,x s ), and optimized by contrastive loss function: The calculation formula of cosine similarity is as follows: Among them, τ is the temperature parameter, i represents the sample pairs under the same emotion category, and j represents all samples; The formula for optimization through contrast loss function is as follows: The loss function of the discriminator is: The loss function of the generator is: Among them, x real is a real speech sample, x gen is the generated speech sample, P real (x) and P gen (x) are the distributions of real speech and generated speech respectively, and D is the discriminator.

7. The method for TTS speech synthesis with enhanced emotion perception driven by multimodal fusion according to claim 1, characterized in that: S4 includes the following steps: S41: preprocessing the text to be synthesized; S42: The preprocessed text is input into the deep semantic coding model based on Transformer. The deep semantic coding model captures the semantic information of the text from different angles through a multi-head attention mechanism to obtain the deep semantic coding representation H of the text. text ; S43: The fused emotional feature vector V fusion Perform decoupling and refinement analysis to obtain the refined sentiment component vector; S44: Refined sentiment component vector and text deep semantic encoding representation H text Fusion is performed to obtain the fused encoding representation H text-emo ; S45: Use the decoder based on long short-term memory network and attention mechanism to represent H according to the fused code text-emo Generate spectral features in a targeted manner; S46: Enhance the generated spectral features using multimodal information.

8. The method for TTS speech synthesis with enhanced emotion perception driven by multimodal fusion according to claim 7, characterized in that: In the S44, a fusion method based on a gating mechanism is used to calculate the influence weight of each sentiment component on the text encoding. The sentiment component vector is set to The weights are calculated as follows: Among them, W i and b i is a learnable parameter, σ is the sigmoid activation function, and the weighted emotional component vector is combined with H text Add together to get the fused code representation H text-emo .

9. The method for TTS speech synthesis with enhanced emotion perception driven by multimodal fusion according to claim 1, characterized in that: The S5 comprises the following steps: S51: Construction of multi-dimensional evaluation indicators; Build a large-scale emotional speech dataset that covers multiple emotion categories and emotional expressions of different intensity levels. The dataset contains natural language text and high-quality speech samples corresponding to the natural language text. S52: Optimization strategy based on reinforcement learning; S53: Model parameter fine-tuning and updating.

10. The method for TTS speech synthesis with enhanced emotion perception driven by multimodal fusion according to claim 9, characterized in that: S51 includes the following steps: S511: Build a large-scale emotional speech dataset that covers multiple emotion categories and emotional expressions of varying intensity. The dataset contains natural language text and high-quality speech samples corresponding to the natural language text. S512: Use deep learning technology to train a highly accurate sentiment classifier. This uses a Transformer-based sentiment classification model. At the model's input layer, speech samples are converted into mel-spectrograms and feature extracted using a convolutional neural network. The extracted features are then fed into the Transformer encoder for sentiment classification. S513: For the generated speech to be evaluated, input it into the trained emotion classifier to obtain the predicted emotion category and intensity. At the same time, compare it with the emotion label of the corresponding reference speech and calculate the root mean square error of the emotion category accuracy and intensity difference between the two. The formula is as follows: Where N is the number of samples, y i and are the emotion category labels of the reference speech and the generated speech, respectively. I is the indicator function, which is 1 when the two are equal and 0 otherwise. i and are the emotion intensity values ​​of the reference speech and the generated speech respectively; A deep learning-based speech quality assessment model, a perceptual speech quality assessment tool, and a short-term objective intelligibility assessment tool are introduced. The generated speech to be evaluated is input into the deep learning-based speech quality assessment model, the perceptual speech quality assessment tool, and the short-term objective intelligibility assessment tool, respectively, to obtain speech quality assessment scores in multiple dimensions. These scores are then combined into a comprehensive speech quality assessment indicator through weighted averaging: Q total =α×Q GAN +β×PESQ+γ×STOI Among them, α, β, and γ are weight coefficients set according to the importance of different indicators; Q GAN represents the speech quality assessment score obtained by the deep learning-based quality assessment model; PESQ represents the speech quality assessment score obtained by the Perceptual Speech Quality Assessment Tool; STOI represents the speech quality assessment score obtained by the Short-Time Objective Intelligibility Assessment Tool; Using semantic parsing technology in natural language processing, the generated speech is converted into text through an automatic speech recognition system, and a semantic parse tree is constructed. At the same time, a semantic parse tree is also constructed for the original input text. The semantic difference is measured by calculating the edit distance between the two trees. The smaller the edit distance, the higher the semantic consistency. The edit distance is calculated using a dynamic programming algorithm. For two semantic parse trees T1 and T2, d(i,j) is defined as the edit distance between the first i nodes of T1 and the first j nodes of T2. The recursive formula is as follows: d(i,j)=min(d(i-1,j)+1,d(i,j-1)+1,d(i-1,j-1)+c(i,j)) Where c(i,j) is 0 when the i-th node of T1 is the same as the j-th node of T2, and 1 otherwise; A semantic matching model based on deep learning is used for auxiliary evaluation. The semantic matching model based on deep learning is built based on a bidirectional long short-term memory network and an attention mechanism. After encoding the original text and the text converted from speech separately, the semantic similarity score between the two is calculated through the attention mechanism. In the bidirectional long short-term memory network, for the input text sequence x = [x1, x2, ..., x n ], its hidden layer state is calculated as follows: Then the semantic similarity score S is calculated through the attention mechanism and combined with the edit distance indicator to obtain the final semantic consistency evaluation result.

Images