Multimodal speech emotion recognition method and model fusing speech and text information

Abstract

The invention provides a multi-mode speech emotion recognition method and model fusing speech and text information. The method comprises the following steps: distinguishing a training set and a test set from a data set related to speech emotion recognition according to a five-fold cross validation mode, and performing data enhancement on the training set and the test set; designing a TMAK text encoder comprising an SFKAN module, and improving the adaptability and robustness of the encoder in response to the change of the utterance length by using relative position encoding and a residual unit; designing an SMTK voice encoder comprising a time sequence fusion convolution module, a voice multi-head attention module and an SFKAN module to optimize local feature extraction and global information integration; the obtained information is fused through a bidirectional dense block module, and a TS-MEFM network model combining voice and text multi-mode emotion fusion is formed; according to the method, the final emotion is obtained by fusing the text information and the voice information, and the correlation between the information is fully utilized, so that the task performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer speech recognition, and in particular to a multimodal speech emotion recognition method and model integrating speech and text information. Background Art

[0002] Speech emotion recognition is an important topic in the field of natural language processing and affective computing. Its main goal is to identify the speaker's emotional state, such as happiness, anger, sadness, etc., by analyzing speech signals. With the rapid development of human-computer interaction technology, systems that can accurately recognize and understand human emotions are becoming more and more important, especially in applications such as intelligent voice assistants, voice customer service, and voice healthcare. Speech emotion recognition not only helps improve human-computer interaction, but also can delve into the complexity of emotional expression and subtle changes in speech. Progress in this field will promote the development of affective computing technology and help us understand and simulate human emotions more comprehensively.

[0003] Traditional speech emotion recognition methods usually rely on unimodal speech signals, which are easily affected by environmental noise, the quality of recording equipment, and the speaker's status, such as health status, resulting in reduced accuracy of emotion recognition. Emotions are multidimensional and difficult to fully capture through speech signals alone. Different speakers may show significant individual differences when expressing the same emotion, which further increases the difficulty of emotion recognition using unimodal methods. However, when processing unimodal data in a high-noise environment, it is still not enough to guarantee high accuracy of emotion recognition, especially when the noise level exceeds a certain threshold, the model performance drops significantly.

[0004] In order to overcome the limitations of single-modal methods, researchers began to explore multimodal emotion recognition methods. By improving the restricted Boltzmann machine (RBM), the emotions of three modalities, namely acoustic signals, text information and expression information, are distinguished, and multimodal information fusion is used to make decisions. Multimodal methods provide more comprehensive and accurate emotion recognition by combining multiple signal sources such as speech, body language, intonation, vocabulary and physiological signals. This not only improves the accuracy and robustness of emotion recognition, but also enhances the recognition effect by leveraging the complementarity between different modalities. In addition, LSTM networks and multi-resolution convolutional networks are used to improve recognition performance by fusing sound features and word features, but this method still has problems in terms of the mutual promotion of the two.

[0005] In recent years, the application of Transformer model in speech emotion recognition has made it feasible to use multimodal and different time series information fusion to identify emotions. Transformer model has been widely used in natural language processing and computer vision due to its advantages in processing long sequence data. However, the multi-layer perception mechanism of Transformer model has the problem of too many parameters and poor accuracy, which needs further research and optimization. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a multimodal speech emotion recognition method and model that integrates speech and text information. By utilizing the transcription of speech and its corresponding text, the subtle interactions between the two modalities are captured, thereby reducing the impact of unimodal bias. In addition, by designing the TMAK text encoder (Text MultimodalAttention SFKAN Encoder) and the SMTK speech encoder (Speech Multimodal Temporal SFKAN Encoder) to integrate local feature extraction and global information integration, the comprehensive capture of fine-grained emotional features is achieved, thereby improving the performance of speech emotion recognition, reducing the amount of parameters, and making it more effective and reliable in practical applications.

[0007] The technical solution adopted by the present invention to solve the technical problem is:

[0008] A multimodal speech emotion recognition model integrating speech and text information, the model comprising:

[0009] TMAK text encoder: used to cope with the change of speech length, including SFKAN module, residual unit, text attention module, relative position encoding and Dropout; wherein the SFKAN module is formed by reflective switch activation and custom linear layer;

[0010] SMTK speech encoder: used to optimize local feature extraction and global information integration, including a temporal fusion convolution module, a speech multi-head attention module and a SFKAN module; wherein the temporal fusion convolution module is combined with a Bi-LSTM, the speech multi-head attention module limits all attention calculations to a small range of adjacent tokens by utilizing an application window, and the SFKAN module is formed by a reflective switch activation and a custom linear layer;

[0011] Bidirectional dense block module: used to fuse the TMAK text encoder and the SMTK speech encoder to form a TS-MEFM network model that integrates speech and text information.

[0012] The present application also provides a multimodal speech emotion recognition method integrating speech and text information, using the above model, the steps include:

[0013] S1. The speech emotion recognition related dataset is divided into training set and test set by five-fold cross validation;

[0014] S2, perform data augmentation on the training set and test set;

[0015] S3. Design a TMAK text encoder including the SFKAN module (Supper Fast KAN), using relative position encoding and residual units to improve the encoder's adaptability and robustness in dealing with changes in utterance length;

[0016] S4. Design an SMTK speech encoder including a temporal fusion convolution module, a speech multi-head attention module, and a SFKAN module to optimize local feature extraction and global information integration, thus achieving comprehensive capture of fine-grained emotional features.

[0017] S5, fusing the information obtained in step S3 and step S4 through a bidirectional dense block module, and finally forming a new TS-MEFM network model combining speech and text multimodal emotion fusion;

[0018] S6, using the training set to train the network model finally formed in step S5;

[0019] S7, using the test set to adjust the network model trained in step S6;

[0020] S8. Input the audio data to be recognized into the network model adjusted in step S7, and perform emotion recognition on the audio data to be recognized.

[0021] Through the above method, a new multimodal emotion fusion model combining speech and text is constructed. The final emotion is obtained by fusing text information with speech information, making full use of the correlation between information to improve the performance of the task. The design of TMAK text encoder improves the adaptability and robustness of the encoder in dealing with changes in speech length. The fusion of SMTK speech encoder and time series fusion convolution module optimizes local feature extraction and global information integration, and realizes the comprehensive capture of fine-grained emotion features, thereby improving the model's ability to understand the entire sequence and improving the accuracy of speech emotion recognition.

[0022] Furthermore, step S2 specifically includes:

[0023] S21, converting the original text dataset into French text using the NLLB model, and then back-translating into English text as enhanced text;

[0024] S22, putting the enhanced text information of step S21 into the RoBERTa-BASE model to extract the features of the original text and the enhanced text;

[0025] S23, generating a corresponding enhanced voice file using the enhanced text features obtained in step S22 through the Coqui-TTS model;

[0026] S24. The enhanced voice file and the original voice file obtained in step S23 are used to obtain their respective voice features using the HuBERT-Base model.

[0027] The above method adopts data enhancement design, combined with multi-model collaboration, aiming to improve the accuracy and robustness of emotion recognition. The present invention realizes the effective extraction of features, creating favorable conditions for subsequent feeding into network learning processing. Combining the feature extraction capabilities of the RoBERTa-BASE model and the HuBERT-Base model, deep features in text and speech can be extracted. The enhanced speech generated by the Coqui-TTS model can further enrich the speech data set, thereby improving the overall performance of the emotion recognition model. This step not only generates enhanced text through multilingual translation, which helps to enrich and balance the acoustic feature data, but also provides richer information and higher performance levels for subsequent emotion recognition models, thereby improving the accuracy and robustness of the emotion recognition system.

[0028] Furthermore, step S3 specifically includes:

[0029] S31, splicing the original text features and enhanced text features obtained in step S22;

[0030] S32, using the concatenated text features obtained in step S31 to capture the information of the relative positions between words through a text attention module, thereby enhancing the model's understanding of local and global dependencies in the sequence;

[0031] S33, applying Dropout to the information captured in step S32 for regularization;

[0032] S34, passing the data after regularization processing in step S33 to the SFKAN module to perform linear transformation and nonlinear activation;

[0033] S35, adding the data processed in step S34 back into the input through residual connection;

[0034] S36: Apply Dropout to regularize the data obtained in step S35 to prevent overfitting of the model and improve generalization ability.

[0035] This step can effectively enhance the model's understanding of local and global dependencies in the sequence, thereby improving the adaptability and robustness of the encoder when processing input sequences of different lengths. Specifically, by capturing the relative position information between words, relative position encoding solves the problem of insufficient generalization ability of traditional absolute position encoding when faced with input sequences of different lengths. In this way, the model can achieve flexible transformation under different input lengths, avoiding dependence on absolute position, thereby improving the adaptability of the encoder when dealing with changes in speech length. In the training and regularization process of deep models, the combination of residual units and Dropout can stabilize the gradient and alleviate the gradient vanishing problem. The residual unit ensures the effective transmission of information by adding the result back to the input;

[0036] Dropout randomly discards some neurons to prevent overfitting of the model and improve generalization ability. This design not only helps to effectively train deep models, but also improves the performance of the model in practical applications.

[0037] Furthermore, the calculation process in the SFKAN module is specifically as follows:

[0038] Throughout the module, the input data x is normalized through the LayerNorm layer, using standardized data to speed up training and improve stability.

[0039]

[0040] in, represents the predicted value of the i-th item; x i Represents the data entered in the i-th item; μ is the mean, σ is the standard deviation, and ∈ is a small constant to prevent division by zero.

[0041] Next, the normalized data is input into the Reflectional Switch Activation layer, where the difference with the predefined grid is calculated and normalized, and then the hyperbolic tangent function and square operation are applied to generate new activation values.

[0042]

[0043] activation = 1-(tanh(diff)) 2

[0044] Among them, g represents the predefined grid, d represents the normalization coefficient, x is the input data, and activation represents the new activation value;

[0045] The activation values ​​are flattened into a two-dimensional matrix with dimensions [batch_size, input_dim*num_grids]. Finally, the flattened activation values ​​are linearly transformed through the SplineLinear layer to generate the final output.

[0046] flattened_activation=activation.view(batch_size,-1)

[0047] output=W·flattened_activation

[0048] Where W is the weight matrix. The input data is processed sequentially through the above layers to achieve complex nonlinear transformations.

[0049] Furthermore, step S4 specifically includes:

[0050] S41, concatenating the original speech features and the enhanced speech features obtained in step S24;

[0051] S42, the concatenated speech features obtained in step S41 are passed through the designed speech multi-head attention module, and all attention calculations are limited to a small range of adjacent tokens by using an application window to reduce the computational burden;

[0052] S43, passing the data processed in step S42 to the designed time series fusion convolution module, starting with Bi-LSTM;

[0053] S44, regularizing the data in step S43 using Dropout to prevent overfitting;

[0054] S45, performing point convolution and GLU activation on the data of step S44;

[0055] S46. Batch normalize the data in step S45 to help train the deep model.

[0056] S47. Pass the data obtained in step S46 to the SFKAN module, perform linear transformation and nonlinear activation to obtain corresponding speech features for later use.

[0057] Through this step, the performance of speech feature processing and emotion recognition can be effectively improved. Specifically, splicing the original speech features and enhanced speech features can enrich the feature input and ensure that the model can capture more speech information. In step S42, through the designed speech multi-head attention module, the application window is used to limit all attention calculations to a small range of adjacent tokens, which can significantly reduce the computational burden and improve computational efficiency, while ensuring that the attention mechanism can effectively capture local dependencies. Next, the data is passed to the time series fusion convolution module, which can capture the temporal dependencies in the sequence and enhance the temporal representation ability of the features. The batch normalization step helps to train deep models, accelerate convergence through standardized processing, and stabilize the training process. Finally, the obtained data is passed to the SFKAN module, and linear transformation and nonlinear activation are performed to further extract and optimize speech features, providing high-quality feature input for subsequent emotion recognition tasks. Through the design and implementation of this series of steps, the performance of the model in processing complex speech data can be significantly improved, the adaptability and generalization ability of the model can be enhanced, and the accuracy and robustness of emotion recognition can be ultimately improved.

[0058] Furthermore, the calculation process in the speech multi-head attention module is specifically as follows:

[0059] First, assume that H w is the application window of feature information statistics, and t is the time step. Select a window size H from the key sequence K w The marking interval is centered at the current marking position t, that is, from arrive These tokens are scaled dot products with the token at position t in the query sequence Q. These dot product results are then normalized using the Softmax function to obtain the attention weight α(t).

[0060]

[0061] α(t)=Softmax(scores(t))

[0062] in, Represents the scaling factor, which is used to adjust the dot product value to make the calculated value more stable. In the attention mechanism, the dot product between Q and K is calculated to get the relevance score. If no scaling is performed, as d k As the value increases, the dot product will become very large, which will cause the gradient of the Softmax function to become very small, affecting the training effect of the model. To avoid this problem, use Scale the dot product result.

[0063] Select the same window size H as above from the value sequence V wThese tags are multiplied by the corresponding attention weights, and then the products are added to get the output of the t-th position.

[0064]

[0065] Repeat the previous operation until the value of t changes from 1 to T, where T represents the entire sequence length.

[0066] Furthermore, in step S5, the acquired information is fused through a bidirectional dense block module, and finally a new TS-MEFM network model combining speech and text multimodal emotion fusion is formed. Specifically, it includes:

[0067] S51, aligning and fusing the text and speech features obtained in step S36 and step S47;

[0068] S52, performing linear transformation on the data obtained in step S51 through a linear layer, laying a foundation for subsequent time series processing;

[0069] S53, passing the data transformed in step S52 to a bidirectional long short-term memory network to capture the forward and backward temporal dependencies in the input sequence;

[0070] S54, using Dropout to randomly discard some neurons from the information captured in step S53 to prevent overfitting of the model and improve the generalization ability of the model;

[0071] S55, passing the data obtained in step S54 through the ReLU activation function, so that the model can capture the complex nonlinear relationship in the data, which helps to improve the model's expression ability and understanding ability of emotional information;

[0072] S56: Further process and map the features of the data processed in step S55 after nonlinear transformation, and convert them into categories in the sentiment classification task.

[0073] Through this step, the text and speech features can be effectively aligned and fused, thereby improving the accuracy and robustness of multimodal sentiment analysis. The model combines the advantages of Bi-LSTM and dense layers, further processes and optimizes the feature representation after feature fusion, and significantly improves the model's ability to understand sentiment and semantic information. It can effectively process and fuse complex features in time series data and enhance the ability to extract and analyze sentiment information. By capturing local and global temporal dependencies, the BLDM model enhances the transfer and utilization of features, thereby providing richer and more accurate sentiment feature representations in multimodal sentiment analysis tasks. This design method can not only fully utilize the advantages of multimodal data and capture complex temporal and nonlinear relationships, but also prevent overfitting through regularization measures, thereby improving the overall performance and generalization ability of the model.

[0074] The beneficial effects of the present invention include:

[0075] Design of TMAK text encoder: The TMAK text encoder is designed by using the relative sinusoidal position encoding, text attention and SFKAN modules in Transformer. The TMAK text encoder uses relative position encoding and residual units to improve the adaptability and robustness of the encoder in dealing with changes in speech length.

[0076] The design of the SMTK speech encoder: By integrating the temporal fusion convolution module, the speech multi-head attention module and the SFKAN module, the local feature extraction and global information integration are optimized, and the comprehensive capture of fine-grained emotional features is achieved.

[0077] TS-MEFM network model design: Based on the deep fusion of speech and text, a novel multimodal emotion fusion neural network model is proposed, which can capture fine-grained emotion information from acoustic and text modalities. The model captures the subtle interaction between the two modalities by utilizing the transcription of speech and its corresponding text, thereby reducing the impact of unimodal bias.

[0078] SFKAN module: The SFKAN module is formed by reflecting the switch activation and the custom linear layer. This module is used to replace the original MLP to reduce the number of model parameters and improve the accuracy of the model.

[0079] Speech Multi-Head Attention Module: By utilizing application windows to restrict all attention computations to a small range of adjacent tokens, the computational burden is greatly reduced.

[0080] Temporal fusion convolution module: By combining with Bi-LSTM, the model is allowed to consider past and future sequence information, enhancing its ability to capture local features and global interactions.

[0081] Training and adjustment: The network model is trained with a training set and adjusted with a test set to ensure that the model's performance on different data sets is stable and reliable.

[0082] Emotion recognition performance: Through the above methods, the final network model is able to perform emotion recognition on the audio data to be recognized and provide more accurate emotion classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is a training flow chart of the network model of the present invention;

[0084] Figure 2 This is a structural diagram of the network model of the present invention;

[0085] Figure 3This is the structure diagram of the speech multi-head attention module;

[0086] Figure 4 This is the structure diagram of the temporal fusion convolution module;

[0087] Figure 5 It is a confusion matrix diagram of each module of the present invention. DETAILED DESCRIPTION

[0088] In order to make the technical problems, technical solutions and beneficial effects to be solved by this application more clearly understood, the application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0089] At present, speech emotion recognition, as an important part of affective computing, has broad application prospects. Traditional unimodal methods have limited accuracy due to the influence of environmental noise and individual differences. Multimodal methods provide a more comprehensive and robust means of emotion recognition by combining multiple signal sources. With the introduction of emerging models such as Transformer, the accuracy and performance of multimodal emotion recognition are expected to be further improved. Future research can continue to explore the optimization methods of multimodal information fusion and how to effectively use these technologies in practical applications, so as to promote the development and application of speech emotion recognition.

[0090] The present invention designs a multimodal speech emotion recognition model that integrates speech and text information. The model uses relative position coding and residual units by designing a TMAK text encoder to improve the adaptability and robustness of the encoder in dealing with changes in speech length. The SMTK speech encoder (Speech Multimodal Temporal SFKAN Encoder) and the temporal fusion convolution module (Temporal Fusion Convolution Module, TFCM) are integrated to optimize local feature extraction and global information integration, and achieve comprehensive capture of fine-grained emotional features. The SFKAN module (Supper Fast KAN) is designed to reduce the number of model parameters. The performance of the model on the data set is ensured to be stable and reliable through training and adjustment. Finally, the invention has made significant progress in speech emotion recognition, improved the accuracy, and made it more suitable for practical application scenarios.

[0091] Example 1

[0092] The experiment was implemented in NVIDIA RTX A4000, with a total of 120 training epochs and a learning rate of 1e -5, the batch size is 2. To ensure the reasonable setting of the sequence dimension, this embodiment sets the dimension of the IEMOCAP and MELD datasets to 1024. Through experiments, the model takes 8 minutes for each training and inference step on a single card A4000 GPU. The TS-MEFM network model has a total of about 221.3M parameters.

[0093] like Figure 1 As shown, a multimodal speech emotion recognition method integrating speech and text information comprises the following steps:

[0094] For the relevant data sets of speech emotion recognition, the training set and the test set are distinguished by the five-fold cross-validation method, and the training set and the test set are enhanced.

[0095] This embodiment uses the open source IEMOCAP dataset and MELD dataset. IEMOCAP is a widely used dataset in the field of speech emotion recognition, collected by the Sail Lab at the University of Southern California. The dataset contains 12 hours of multimodal data such as audio, video, facial motion capture, and text transcription, covering five sessions, each of which includes a male and a female speaker. A total of 5531 expressions are labeled as angry, neutral, happy, and sad. In the experiment, the present invention adopts a five-fold cross-validation strategy to train the model and evaluate its performance. In addition, the IEMOCAP database provides detailed motion capture information and supports interactive settings for real emotions. This large-scale, multimodal dataset has become an important resource for studying and modeling multimodal and expressive human communication.

[0096] MELD is another important dataset used in emotion recognition (SER). This dataset consists of 13,708 utterances containing seven emotions (happy, angry, neutral, sad, fearful, surprised, and disgusted). The dataset is divided into training, validation, and test sets, which contain 9,989, 1,109, and 2,610 samples respectively.

[0097] At the same time, three evaluation indicators are used to evaluate the performance of different learning algorithms, including weighted accuracy (WeightedAccuracy, WA), unweighted accuracy (UnweightedAccuracy, UA), and weighted average F1 (Weighted F1 Score, WF1). The calculation method is as follows:

[0098]

[0099] Among them, S n is the number of samples of the nth class, Acc(n) and F1(n) are the classification accuracy and F1 score of the nth class respectively.

[0100] Next, enhance the voice data and text data and extract features. The specific operations are as follows:

[0101] In terms of text, the NLLB 600M Distilled model is first used to convert English text into French text, and then back-translated into English text. During the back-translation, num_return_sequences is set to 5, and the best text is obtained as the enhanced text. Then RoBERTa-BASE is used as the encoder of the text context to obtain the original text features and the enhanced text features, and then the pre-processed text features are passed to the TMAK text encoder for subsequent operations.

[0102] In terms of speech, the Coqui-TTS model is first used to generate the corresponding speech signal from the enhanced text as the enhanced speech information. Then the pre-trained HuBERT-Base model is used, which is pre-trained on the 960-hour LibriSpeech dataset. Through this model, the duration of each frame of each speech is set to 25 milliseconds, and a jump length of 20 milliseconds is used to generate overlapping frames, with an overlap of 5 milliseconds between adjacent frames. A 1024-dimensional frame granularity feature is extracted for each speech sample and then passed to the SMTK speech encoder.

[0103] like Figure 2 As shown, the TS-MEFM network model of the present application uses speech and text features to perform modeling in a fusion manner. The extracted text feature information is passed to the TMAK text encoder, which applies the relative sinusoidal position encoding scheme in the Transformer to capture the relative position information between words.

[0104] like Figure 3 and Figure 4 As shown, the speech features are used to establish a global and local relationship through the speech multi-head attention module and the temporal fusion convolution module proposed in this application, so as to improve the ability to capture local data and obtain context information. Then, the captured information is passed to the SFKAN module (Supper Fast KAN), which integrates and processes the context information obtained for each word to reduce the number of model parameters.

[0105] In the speech multi-head attention module, assuming H w is the application window of feature information statistics, and t is the time step. Select a window size H from the key sequence K w The marking interval is centered at the current marking position t, that is, from arrive These tokens are scaled dot products with the token at position t in the query sequence Q. These dot product results are then normalized using the Softmax function to obtain the attention weight α(t).

[0106]

[0107] α(t)=Softmax(scores(t))

[0108] in, Represents the scaling factor, which is used to adjust the dot product value to make the calculated value more stable. In the attention mechanism, the dot product between Q and K is calculated to get the relevance score. If no scaling is performed, as d k As the value increases, the dot product will become very large, which will cause the gradient of the Softmax function to become very small, affecting the training effect of the model. To avoid this problem, use Scale the dot product result.

[0109] Next, select the same window size H as above from the value sequence V w These tags are multiplied by the corresponding attention weights, and then the products are added to get the output of the t-th position.

[0110]

[0111] Repeat the above until the value of t changes from 1 to T, where T represents the entire sequence length.

[0112] In the SFKAN module, the input data x is first normalized through the LayerNorm layer, and standardized data is used to accelerate training and improve stability.

[0113]

[0114] in, represents the predicted value of the i-th item; x i Represents the data entered in the i-th item; μ is the mean, σ is the standard deviation, and ∈ is a small constant to prevent division by zero.

[0115] Next, the normalized data is input into the Reflectional Switch Activation layer, where the difference with the predefined grid is calculated and normalized, and then the hyperbolic tangent function and square operation are applied to generate new activation values.

[0116]

[0117] activation = 1-(tanh(diff)) 2

[0118] Among them, g represents the predefined grid, d represents the normalization coefficient, x is the input data, and activation represents the new activation value.

[0119] The activation values ​​are flattened into a two-dimensional matrix with dimensions [batch_size, input_dim*num_grids]. Finally, the flattened activation values ​​are linearly transformed through the SplineLinear layer to generate the final output.

[0120] flattened_activation=activation.view(batch_size,-1)

[0121] output=W·flattened_activation

[0122] Where W is the weight matrix. The input data is processed sequentially through the above layers to achieve complex nonlinear transformations.

[0123] The processed speech and text features are integrated using the bidirectional dense block module to form the final TS-MEFM network model, which provides a basis for practical application scenarios while maintaining a good balance and stability, providing support for subsequent speech processing and recognition tasks.

[0124] Example 2

[0125] In order to evaluate the effectiveness of the present invention, the recognition accuracy of this embodiment is compared with the most advanced speech emotion recognition method. This experiment tests these models on the IEMOCAP dataset and the MELD dataset. The test results of the IEMOCAP dataset are as follows:

[0126]

[0127] The results of the MELD dataset are shown in the following table:

[0128]

[0129] Experimental results show that the speech emotion recognition method of the present invention has a higher accuracy rate.

[0130] In order to further verify the effectiveness of each module, the present invention conducted experiments on different data sets. The specific test results are shown in the following table:

[0131]

[0132] In the above table, (T) represents TMAK, (S) represents SMTK, (K) represents SFKAN, and (C) represents TFCM.

[0133] As can be seen from the table, the recognition performance of using text modality TMAK and speech modality SMTK alone is relatively low. When TFCM is removed, the performance decreases, indicating that TFCM plays an important role in improving the accuracy of speech emotion recognition. When combining speech and text modalities, without using the SFKAN module, the traditional method can achieve 83.33% on the IEMOCAP dataset and 67.24% on the MELD dataset, which is significantly higher than a single modality. This shows that multimodal fusion has significant advantages in emotion recognition. After introducing the SFKAN module, the model's WA on the IEMOAP dataset increased by 0.88%, UA increased by 1.56%, and WF1 on the MELD data increased by 1.9%, while the number of parameters was reduced by about 13.82%. It is verified that the SFKAN module further improves the accuracy and robustness of the model while reducing the number of parameters.

[0134] In order to avoid the problem that the recognition rate cannot fully and objectively evaluate the model due to the imbalance in the number of emotion samples in the data set, the present invention calculates the normalized confusion matrix for the results of the TS-MEFM network model on the IEMOCAP data set, such as Figure 5 shown.

[0135] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A multimodal speech emotion recognition model integrating speech and text information, characterized in that: The model includes: TMAK text encoder: used to cope with the change of speech length, including SFKAN module, residual unit, text attention module, relative position encoding and Dropout; wherein the SFKAN module is formed by reflective switch activation and custom linear layer; SMTK speech encoder: used to optimize local feature extraction and global information integration, including a temporal fusion convolution module, a speech multi-head attention module and a SFKAN module; wherein the temporal fusion convolution module is combined with a Bi-LSTM, the speech multi-head attention module limits all attention calculations to a small range of adjacent tokens by utilizing an application window, and the SFKAN module is formed by a reflective switch activation and a custom linear layer; Bidirectional dense block module: used to fuse the TMAK text encoder and the SMTK speech encoder to form a TS-MEFM network model that integrates speech and text information.

2. A multimodal speech emotion recognition method integrating speech and text information, based on the model as claimed in claim 1, characterized in that the steps include: S1. The speech emotion recognition related dataset is divided into training set and test set by five-fold cross validation; S2, perform data augmentation on the training set and test set; S3. Design a TMAK text encoder containing the SFKAN module, and use relative position encoding and residual units to optimize the TMAK text encoder's response to changes in utterance length. S4. Design an SMTK speech encoder including a temporal fusion convolution module, a speech multi-head attention module, and a SFKAN module to optimize local feature extraction and global information integration; S5, fusing the information obtained in step S3 and step S4 through a bidirectional dense block module to form a TS-MEFM network model that fuses speech and text information; S6, using the training set to train the network model finally formed in step S5; S7, using the test set to adjust the network model trained in step S6; S8. Input the audio data to be recognized into the network model adjusted in step S7, and perform emotion recognition on the audio data to be recognized.

3. The multimodal speech emotion recognition method integrating speech and text information according to claim 2 is characterized in that: The data enhancement described in step S2 is as follows: S21, converting the original text dataset into French text using the NLLB model, and then back-translating into English text as enhanced text; S22, putting the enhanced text information of step S21 into the RoBERTa-BASE model to extract the features of the original text and the enhanced text; S23, generating a corresponding enhanced voice file using the enhanced text features obtained in step S22 through the Coqui-TTS model; S24. The enhanced voice file and the original voice file obtained in step S23 are used to obtain their respective voice features using the HuBERT-Base model.

4. The multimodal speech emotion recognition method integrating speech and text information according to claim 3 is characterized in that: Step S3 specifically includes: S31, splicing the original text features and enhanced text features obtained in step S22; S32, using the concatenated text features obtained in step S31 to capture the information of the relative positions between words through a text attention module; S33, applying Dropout to the information captured in step S32 for regularization; S34, passing the data after regularization processing in step S33 to the SFKAN module to perform linear transformation and nonlinear activation; S35, adding the data processed in step S34 back into the input through residual connection; S36: Apply Dropout to regularize the data obtained in step S35.

5. The multimodal speech emotion recognition method integrating speech and text information according to claim 4 is characterized in that: Step S4 specifically includes: S41, concatenating the original speech features and the enhanced speech features obtained in step S24; S42, passing the concatenated speech features obtained in step S41 through the speech multi-head attention module, using the application window to limit all attention calculations to a small range of adjacent tokens; S43, passing the data processed in step S42 to the time series fusion convolution module, starting with Bi-LSTM; S44, regularizing the data in step S43 using Dropout; S45, performing point convolution and GLU activation on the data of step S44; S46, batch normalizing the data of step S45; S47. Pass the data obtained in step S46 to the SFKAN module, perform linear transformation and nonlinear activation to obtain corresponding speech features for later use.

6. The multimodal speech emotion recognition method integrating speech and text information according to claim 5 is characterized in that: Step S5 specifically includes: S51, aligning and fusing the text and speech features obtained in step S36 and step S47; S52, performing linear transformation on the data obtained in step S51 through a linear layer for subsequent time series processing; S53, passing the data transformed in step S52 to a bidirectional long short-term memory network to capture the forward and backward temporal dependencies in the input sequence; S54, using Dropout to randomly discard some neurons from the information captured in step S53; S55, passing the data obtained in step S54 through a ReLU activation function so that the model captures the complex nonlinear relationship in the data; S56: Process and map the features of the data processed in step S55 after nonlinear transformation, and convert them into categories in the sentiment classification task.

Images