Speech emotion recognition method based on multi-modal feature fusion

Abstract

The invention discloses a speech emotion recognition method based on multi-modal feature fusion, and the method comprises the steps: designing a multi-modal speech emotion recognition model, enabling an audio to serve as a main mode, employing the function of wav2vec2.0, and combining a text mode with a visual mode to enhance the performance of emotion recognition based on the audio; the self-attention mechanism, the GRU and the LVC are combined to extract global and local information from a visual mode, the global information is extracted only from text features, visual clues and text clues related to emotion can be comprehensively understood, the extracted text and visual features are integrated into a wav2vec2.0 model, multi-mode representation of multi-level audio information is formed, and the multi-mode representation of the multi-level audio information is realized. And meanwhile, the discrimination capability of the model is enhanced by utilizing multi-modal homogeneous feature difference learning, and the proposed model is evaluated on two data sets, so that the model is proved to have excellent robustness and generalization compared with the existing most advanced model.

Description

Technical Field

[0001] The present invention relates to the technical field of speech emotion recognition, and in particular to a speech emotion recognition method based on multimodal feature fusion. Background Art

[0002] With the advancement of human-computer interaction technology, there is an increasing demand for machines to accurately recognize and understand human emotions. Speech emotion recognition (SER) plays a vital role in laying the foundation for emotional interaction in emotional intelligence systems, helping to improve communication and emotion perception between machines and humans. It aims to analyze human speech signals to identify and understand the speaker's emotional state. Speech emotion recognition technology has been applied in various fields, including customer service and market research, learning and education, mental health, and social media analysis.

[0003] In real-life scenarios, everyone expresses emotions not only through speech, but also through other means such as text and vision. Therefore, relying solely on speech for accurate emotion recognition is far from enough. In order to accurately identify the emotions expressed by the speaker, information from multiple modalities must be effectively combined. In the field of multimodal emotion recognition (MER), the use of different modalities is often complementary, providing complementary clues to mitigate semantic and emotional ambiguity. This inherent complementarity between modalities enhances the overall understanding of potential emotional expressions and promotes a more thorough understanding of the conveyed emotions.

[0004] In audio waveforms, higher pitches and tones are usually associated with positive emotions such as happiness and excitement, while lower pitches and tones are usually associated with negative emotions such as sadness and frustration. In fact, due to the presence of both higher and lower pitches in an audio clip, it can be challenging to analyze the audio pattern alone to determine the exact emotional state. Therefore, it is difficult to accurately identify the exact emotional state by relying solely on audio patterns.

[0005] Moreover, multimodal speech emotion recognition (MSEA) faces several unsolved problems that hinder its development. First, obtaining large-scale, high-quality multimodal datasets is a challenging task. Compared with unimodal data, multimodal data is more expensive to obtain and label. The scarcity of data will directly limit the training and evaluation of the model, resulting in limited generalization ability and performance of the model. Second, multimodal data often exhibit asynchrony, and information from different modalities is not completely synchronized in time. Finally, features extracted from different modalities may look similar but can convey different emotional meanings. For example, some features in speech and text may show similarity in form but express different emotional states. This variability further complicates modeling and highlights the importance of accurately distinguishing and understanding the emotional states of cross-modal homogeneous features. Summary of the invention

[0006] In view of the above-mentioned defects, the technical problem to be solved by the present invention is to provide a speech emotion recognition method based on multimodal feature fusion, comprising:

[0007] Auxiliary modality encoder, primary modality encoder, and multimodal homogeneous feature difference learning;

[0008] It also includes the following processes:

[0009] S1: Extract separate video and text features using auxiliary modality encoder; extract fixed T frames from each segment and use efficientNet pre-trained on VGGface and AFEW datasets as feature extractor to obtain visual features;

[0010] S2: After obtaining the video features, they are input into the A-GRU-LVC module consisting of a gated recurrent unit, self-attention, and LVC block to capture global and local information; for text, the roberta-base pre-trained model is used as the feature extractor, and then the gated recurrent unit and self-attention mechanism are used to obtain global features;

[0011] S3: Use wav2vec2.0 as an encoder to learn the contextual information of the audio sequence, embed the gated cross-modal attention block into wav2vec2.0, and finally, the shared encoder is responsible for integrating the features of the auxiliary modality and the main modality, utilizing multimodal homogeneous feature difference learning, focusing on distinguishing representations of the same modality but with different emotions.

[0012] In the above-mentioned technical solution of a speech emotion recognition method based on multimodal feature fusion, preferably, given a multimodal signal The unimodal original sequence extracted from video segment j is represented as Here, modality is denoted by {a, t, v}, referring to audio, text, and visual modalities. Multimodal emotion recognition aims to predict the emotion category of each utterance in the signal; it focuses on classifying the emotion conveyed in each person's speech and assigning it to a specific emotion category or class. c is the number of sentiment categories.

[0013] In the above-mentioned technical solution of the speech emotion recognition method based on multimodal feature fusion, preferably, the visual representation in the auxiliary modal encoder in step S1 specifically has the following process:

[0014] For the visual modality, the efficientNet pre-trained model is used as a feature extractor to obtain visual features. It can be expressed as:

[0015]

[0016] where Φ visual Represents efficientNet pre-trained model function;

[0017] The visual features obtained in the previous step are then input into the proposed A-GRU-LVC module, which is designed to extract global and local features. In the first block, the gated recurrent unit (GRU) and self-attention mechanism are used to capture the global long-range dependencies within the visual features. This block focuses on extracting global information from the visual input, enabling the model to understand the overall context and relationship between different visual elements.

[0018]

[0019] Among them, F SA and F GRU Represent the gated recurrent unit and self-attention mechanism learning functions respectively;

[0020] At the same time, in order to preserve the local corner area and extract local information, a learnable visual center (LVC) is implemented on the visual features to aggregate the features of the local area and ensure that important local information is preserved; LVC is an encoder that contains an intrinsic dictionary, which contains two main components: an intrinsic codebook and a set of scaling factors related to the learnable visual center:

[0021] B={b1,b2,...,b K}

[0022] S={s1,s2,...,s K}

[0023] Where K represents the number of embedding dimensions in the input features. In the case where the feature map is flattened during visual feature extraction, K can be viewed as the product of the spatial dimensions of the feature map. The visual features are encoded using a combination of convolutional layers, which, in contrast to the method, utilizes one-dimensional convolutions instead of two-dimensional convolutions. These encoded features are then processed by a CBR block, which contains a one-dimensional convolution with a kernel size of 3, a batch normalization layer, and a ReLU activation function to reprocess the visual features for further processing in the codebook.

[0024] To map the visual features into the codebook, a set of scaling factors s is used k , these scaling factors will (represents the i-th visual feature th Pixels ) and b k (denotes the kth th codeword) is mapped to its corresponding item, the kth thThe information of the entire image corresponding to the codeword can be calculated as follows:

[0025]

[0026] represents the information about the position of each pixel relative to the corresponding codeword, where K represents the total number of visual centers. Subsequently, the function Fusion of all e k , which includes 1D-BN (one-dimensional batch normalization) and ReLU layers and average pooling layers, the complete information of the entire image about K codewords is calculated as follows:

[0027]

[0028] After obtaining the output of the codebook, e is passed through a fully connected layer and a one-dimensional convolutional layer with a kernel size of 1 to predict features that specifically highlight the key classes; next, in the visual input feature A channel-by-channel multiplication is performed between and the scaling factor δ:

[0029] δ=F sigmoid (Conv1d(e))

[0030]

[0031] F sigmoid is the sigmoid activation function, It is a channel-by-channel multiplication. Then, the input visual feature And the channel summation between the local area features Z, is the channel sum and can be expressed as:

[0032]

[0033] Finally, by concatenating the output of the self-attention module along the last dimension and the output of the LVC block Get the output of the A-GRU-LVC block:

[0034]

[0035] In the above-mentioned technical solution of the speech emotion recognition method based on multimodal feature fusion, preferably, the context text encoding in the auxiliary modal encoder in step S1 specifically includes:

[0036] Tokenize the input text and convert the text into a series of tokens. After segmentation, roberta adds some special tags to the input text data, such as the start tag <cls>and separator <sep>, the start tag is used to indicate the beginning of the text, the separation tag is used to separate different sentences, then, in order to preserve the position information of the words, position embedding is added to each tag, which enables the model to understand the relative position of the words in the sentence in the input text, and finally, the processed data is input into the transformer encoder to capture the relationship between words and context information:

[0037] It can be expressed as:

[0038]

[0039] where Φ text Represents the roberta pre-trained model learning function;

[0040] After obtaining the text context embedding, it is input into the GRU and self-attention mechanism to obtain the global features of the text information. Due to the strong temporal continuity in the text information, the LVC mechanism is not used to capture local features:

[0041]

[0042] Among them, F SA and F GRU Represent the GRU and self-attention mechanism learning functions respectively.

[0043] In the above-mentioned technical solution of a speech emotion recognition method based on multimodal feature fusion, preferably, the main modal encoder specifically includes: a method for distinguishing by replacing the self-attention mechanism in the wav2vec2.0 transformer layer with a gated cross-modal attention mechanism, so that the text and visual The signal is encoded and fused with multiple layers of audio information. The low-level audio features are encoded through a shallow transformer layer, and then the text and visual features are combined through a modified transformer layer to form a comprehensive multimodal representation. Merging text and vision into wav2vec2.0 can detect relevant information from a large amount of pre-trained audio knowledge, thereby enhancing the emotional information in the multimodal fusion representation. The low-level acoustic features extracted by the shallow transformer block The calculation is as follows:

[0044]

[0045] F ST is the learning function of the shallow transformer layer;

[0046] The audio features Text Features and visual features They are input into two cross-modal attention mechanisms to obtain inter-modal correlation information. Specifically, when performing attention calculation, the audio features are used as queries, and the text features or visual features are used as keys and values. The representation of audio features can be enhanced by introducing text or visual information into audio features:

[0047]

[0048]

[0049] Then, the enhanced features and is fed into the subsequent gating and filtering mechanism;

[0050]

[0051]

[0052] The ratio of each channel can be dynamically defined by a learnable parameter that can filter out erroneous information generated during cross-modal interaction.

[0053] In the above-mentioned technical solution of a speech emotion recognition method based on multimodal feature fusion, preferably, the multimodal homogeneous feature difference learning specifically includes:

[0054] The unfused audio features Text Features and visual features Feeding into a shared encoder to obtain homogeneous features minimizes the feature gap between different modalities and helps multimodal alignment:

[0055]

[0056] SD is a shared encoder learning function consisting of a simple linear layer;

[0057] Homogeneous representations from the same emotion but different modalities should be more similar than homogeneous representations from the same modality but different emotions. Therefore, in the work of

[15] , multimodal homogeneous feature difference learning is performed to enhance the interaction between the same emotion but different modalities and amplify the difference between the same modality but different emotions. This loss function is defined as the marginal loss:

[0058]

[0059] Among them, M = {(i, j, k) | m[i] ≠ m[j], m[i] = m[k], c[i] = c[j], c[i] ≠ c[k]} is a triple, m[i] is the modality type of sample i, c[i] is the label category of sample i, cos represents the cosine similarity between two feature vectors, and by applying the distance margin α, it is ensured that the distance between positive samples is smaller than the distance between negative samples. Here, positive samples refer to the same emotion but different sentiments, and negative samples refer to the same modality but different sentiments;

[0060] As in many multimodal emotion recognition studies, cross entropy is often used as a loss function to optimize model parameters and improve classification accuracy during training:

[0061]

[0062] Among them, y j The true label of the sample, The predicted label of the sample, N D The total number of samples in the dataset D; Finally, the two constraints are combined to form the entire multimodal emotion recognition loss:

[0063]

[0064] Here, λ is the balancing factor.

[0065] It can be seen from the above technical solution that the present invention provides a speech emotion recognition method based on multimodal feature fusion. Compared with the prior art, the present invention has the following beneficial effects:

[0066] The present invention designs a multimodal speech emotion recognition model, in which audio is the main modality, utilizes the function of wav2vec2.0, and combines text and visual modes to enhance the performance of audio-based emotion recognition, combines self-attention mechanism, GRU and LVC to extract global and local information from the visual modality, and extracts global information only from text features, which can more comprehensively understand the visual clues and text clues related to emotions, integrates the extracted text and visual features into the wav2vec2.0 model, forms a multimodal representation of multi-level audio information, and utilizes multimodal homogeneous feature difference learning to enhance the model's discrimination ability, and evaluates the proposed model on two data sets to prove that it has excellent robustness and generalization compared with the existing state-of-the-art models. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces and describes the drawings required for use in the embodiments of the present invention or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 This is a schematic diagram of the WavFusion model structure;

[0069] Figure 2 It is a schematic diagram of the structure of the A-GRU-LVC module;

[0070] Figure 3 Schematic diagram of the architecture of the gated cross-modal attention mechanism;

[0071] Figure 4 Schematic diagram of the confusion matrix on the IEMOCAP dataset;

[0072] Figure 5 Schematic diagram of the confusion matrix on the MELD dataset. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In order to explain and illustrate the technical solutions and implementation methods of the present invention more clearly, several preferred specific embodiments for implementing the technical solutions of the present invention are introduced below.

[0074] Specific embodiment 1.

[0075] Dataset:

[0076] The model is trained and evaluated on the IEMOCAP and MELD datasets, which are mature datasets commonly used in multimodal emotion recognition research. The datasets provide diverse multimodal data to comprehensively train and evaluate the model.

[0077] The IEMOCAP dataset is a widely used benchmark in emotion recognition research, consisting of 12 hours of improvisational and scripted audio-visual data of 10 UC drama actors (five men and five women). The dataset is divided into five binary sessions, each of which is annotated with emotional information in four ways: video, audio, transcription, and motion capture of facial movements. Here, the models evaluated using audio, transcription, and video data focus on six emotions: happy, neutral, angry, excited, sad, and frustrated. The dataset contains a total of 7380 data samples, and the statistics of the dataset are listed in Table 1. For evaluation, a five-fold cross-validation method is adopted, and the dataset is divided into five equal parts (80% training and 20% testing), and each session is used as a test set once.

[0078] The MELD dataset is widely used for multimodal emotion recognition and is derived from more than 1,400 dialogues and 13,000 utterances extracted from the TV series Friends. Each sentence in the dataset is annotated with one of seven emotion labels: neutral, surprise, fear, sadness, happiness, disgust, and anger. The dataset contains multimodal scenes, making it suitable for studying multimodal emotion recognition tasks. In the experiments, the predefined training / validation splits provided by the MELD dataset are utilized. This ensures consistency with existing methods and allows fair comparison with other models. Table 2 provides detailed statistics about the MELD dataset, including the number of utterances and emotion labels.

[0079] Table 1, Statistics of IEMOCAP dataset:

[0080]

[0081]

[0082] Table 2, statistics of the MELD dataset:

[0083] Emotional categories train verify test Total surprise 1205 150 281 1636 fear 268 40 50 358 sad 683 112 208 1003 hapiness 1744 163 402 2309 disgust 271 22 68 361 anger 1109 153 345 1607 neutral 4709 469 1256 6434

[0084] Experimental setup:

[0085] To investigate the benefits of multimodal fusion, a unimodal baseline was established that exploits audio, text, and visual modalities individually. The text baseline uses RoBERTa-Base as the contextual text encoder, followed by a GRU block, a self-attention mechanism, a single linear layer, and a softmax activation function for classification. Similarly, the unimodal speech baseline follows a similar setup to the text baseline, with the only difference being the encoder, where the RoBERTa-Base encoder is replaced with a pre-trained wav2vec2.0 model designed for speech processing. For the visual modality, the effectiveNet pre-trained model on the VGGface and AFEW datasets is used as the encoder to obtain visual features. These visual features are then fed into an A-GRU-LVC block, a linear layer, and a softmax activation for classification.

[0086] The experiments were conducted on a single NVIDIA GeForce RTX 3060 graphics card using the PyTorch framework. During training, epochs were set to 50 and batchsize was set to 2. To calculate the loss during the training phase, cross entropy loss and marginal loss were used. The Adam optimizer was used to update the model parameters with a learning rate of 1e-5. To control the strength of the marginal loss, a hyperparameter λ denoted as was introduced. The hyperparameter λ allows adjusting the impact of the marginal loss on the entire training process.

[0087] On the IEMOCAP and MELD datasets, accuracy (ACC) and weighted F1 (WF1) are used as evaluation metrics. These metrics provide an in-depth understanding of the overall accuracy and weighted performance of different emotion categories:

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] N i Represents category i th The number of utterances in i Indicates i th The number of correctly identified sentences in a category, k represents the number of categories, M i Indicates that it is recognized as i th The number of all emotions in the class.

[0094] Baseline:

[0095] In the study, the results are comprehensively compared with several state-of-the-art multimodal fusion frameworks, which can be roughly divided into different groups based on their fusion methods and strategies. The categories of the considered baselines are mainly as follows:

[0096] Early multimodal fusion methods: These methods focus on fusing modalities at the early stage of the model. Examples include low-order multimodal fusion (LMF), tensor fusion network (TFN), and multimodal factorization model (MFM).

[0097] Modal Interaction Modeling Methods: These methods combine modal interactions to fuse multimodal information, including Interactive Canonical Correlation Network (ICCN), Multimodal Fusion Network with Multi-head Attention Mechanism (M2FNet), Hierarchical Adaptive Attention Network for Multimodal Emotion Recognition in Dialogue (HAAN-ERC), and Exploring Intra-modal and Inter-modal Emotional Behavior in Dialogue (DialogueTRM).

[0098] Multimodal conversational context methods: These methods consider the conversational context to solve the problem of emotion recognition in conversations, including Multimodal Fusion via Deep Graph Convolutional Networks (MMGCN), Multimodal Dynamic Fusion Networks (MMDFN), and Multimodal Emotion Recognition with Contextualized Graph Neural Networks (COGMEN).

[0099] Combining ERC and MSA tasks: Combine the emotion recognition in conversation (ERC) task and the multimodal sentiment analysis (MSA) task for multimodal emotion recognition, including the implementation of unified multimodal sentiment analysis and emotion recognition (UniMSE).

[0100] Comparative analysis:

[0101] To validate the robustness of the proposed network, a thorough comparison of the proposed WavFusion model with several state-of-the-art techniques in the baselines is performed. The results are shown in Tables 3 and 4, which show the average performance on the IEMOCAP and MELD datasets, respectively. The evaluation metrics used are accuracy (ACC) and weighted F1 score (WF1). The experimental results show that the proposed method achieves new state-of-the-art results on the IEMOCAP and MELD datasets. On the IEMOCAP dataset, the method outperforms the previous state-of-the-art method by 0.84% ​​in ACC and 0.74% in WF1. Similarly, on the MELD dataset, the method outperforms the previous state-of-the-art method by 0.43% in ACC and 0.44% in WF1. These results clearly demonstrate the superior performance of the WavFusion model in the multimodal emotion recognition task. Overall, the experiments demonstrate the effectiveness and robustness of the proposed method, which has been shown to be highly competitive and achieves new state-of-the-art results in multimodal emotion recognition.

[0102] Table 3, performance of different methods on the IEMOCAP dataset:

[0103]

[0104]

[0105] Table 4, performance of different methods on the MELD dataset:

[0106] Method ACC(%) WF1(%) TFN 60.70 57.74 LMF 61.15 58.30 TFN 60.70 57.74 MFM 60.80 57.80 DialogueTRM 65.66 63.55 MMGCN - 58.65 MM-DFN 62.49 59.46 UniMSE 65.09 65.51 HAAN-ERC 66.50 65.66 66.93 66.10

[0107] Ablation experiment:

[0108] To gain a deeper understanding of the contributions of different modules in the WavFusion model, an ablation study is performed on the IEMOCAP dataset to compare the accuracy (ACC) and weighted F1 score (WF1) on the test dataset.

[0109] In the ablation study, the proposed network was trained in several scenarios: audio (A), text (T), vision (V), audio and text (A+T), audio and vision (A+V), and the whole modality (A+T+V). Specifically, when a single modality was used, gated cross-modal attention and multimodal homogeneous feature differential learning were excluded. When both modalities were used, the gating mechanism in gated cross-modal attention and multimodal homogeneous feature differential learning was removed. The results in Table 5 show that the highest accuracy and weighted average F1 score can be achieved when all three modalities are used. This is better than other scenarios where only one or two modalities are used. In the unimodal setting, audio features performed the best, followed by text and then vision, which had the lowest performance. Combining audio with text produced better results compared to combining audio with vision, indicating a higher correlation between audio and text modalities. However, combining audio with video resulted in a lower WF1 value compared to audio unimodal, indicating that the fusion of audio and video information may introduce some redundancy or noise.

[0110] Table 5, ablation study for each modality:

[0111] Modal ACC WF1 A 66.06 65.59 T 58.74 58.63 V 29.88 26.31 A+T 67.75 67.45 A+V 66.33 64.14 A+V+T 70.53 70.60

[0112] On the visual side, a Local Visual Center (LVC) block is introduced to capture local information about visual features. To evaluate the importance of the LVC block, an experiment is conducted in which the LVC block is omitted from the model, so that local information about visual features cannot be captured. The corresponding results are shown in Table 6, and it is observed that the model with the LVC block outperforms the model without the LVC block. The experiments show that the inclusion of the LVC block improves the accuracy (ACC) by 0.63% and the weighted F1 score (WF1) by 0.76%. The LVC block is beneficial in capturing relevant contextual details and spatial dependencies in visual features. In turn, it contributes to the performance of the model in recognizing emotions expressed visually.

[0113] Table 6, ablation experiment of LVC block:

[0114] Model ACC WF1 Without LVCblock 69.90 69.84 With LVCblock 70.53 70.60

[0115] In these experiments, weights were assigned to the balancing factor (λ) of the margin loss, and its effects were observed under different weight values. The corresponding results are shown in Table 4. The results show that the best performance is achieved on the IEMOCAP dataset when λ = 1. Under this setting, the model's accuracy (ACC) is significantly improved by 2.64%, and the weighted F1 score (WF1) is significantly improved. Compared with not taking margin loss into account (λ = 0), it is reduced by 2.94%. This demonstrates the effectiveness of multimodal homogeneous feature difference learning in enhancing the model's ability to distinguish emotions from different modalities. However, it is also noted that when the balancing factor is too large (λ = 10), the performance degrades. This suggests that over-emphasizing margin loss may have a negative impact on the original classification task.

[0116] Table 7. Ablation experiment of the balance factor λ in marginloss

[0117] λ ACC WF1 0 67.89 67.66 0.01 68.63 68.39 0.1 69.11 68.96 1 70.53 70.60 10 64.43 64.19

[0118] Visualization:

[0119] On the IEMOCAP and MELD datasets, two visualization techniques are employed to gain insight into the performance of the models: confusion matrix and t-SNE.

[0120] Confusion Matrix:

[0121] To gain a deeper understanding of the performance of the proposed WavFusion model, confusion matrices are used to visualize the results within and between different emotion categories on the IEMOCAP and MELD datasets. The confusion matrix provides a comprehensive view of the model predictions and the true labels. Figure 4 and Figure 5 In the confusion matrix, each column and row represent the true and predicted emotion labels, respectively. By inspecting the entries of the matrix, we can observe how the model performs for each emotion category. The diagonal elements represent the number of correct predictions for each emotion, while the off-diagonal elements represent the misclassifications between different emotions. By visualizing the distribution of correct and incorrect predictions, we can gain insight into the strengths and weaknesses of the model in identifying different emotions.

[0122] exist Figure 5 In the IEMOCAP dataset, the abbreviations used are as follows: "hap" for happy, "exc" for excited, "neu" for neutral, "fru" for frustrated, "sad" for sad, and "ang" for angry. On the IEMOCAP dataset, the samples in each class are relatively evenly distributed, and the models of show similar performance in all classes. It is worth noting that the model of achieves an encouraging accuracy of 81% in identifying the "sad" class. However, the accuracy of the model of is slightly lower in the "happy" class, which is 58.19%, compared with other classes. Some samples belonging to the "happy" class are misclassified as "excited", and this misclassification may be attributed to the similarity of the underlying expressions between these two emotion classes. In addition, in some studies on the IEMOCAP dataset, the "happy" and "excited" classes are often merged into one class during the classification process. Another noteworthy observation is that several emotions, including "happy", "excited", "frustrated", and "sad", seem to be often confused with the "neutral" class, which highlights the difficulty in distinguishing neutral emotions from other emotion classes. These findings are consistent with previous studies on the IEMOCAP dataset, which showed that neutral emotions tend to be located in the center of the activation space, making them difficult to distinguish from other emotion categories.

[0123] In the case of the MELD dataset, it was observed that there was a significant variation in the sample size of each emotion category, which greatly affected the performance of the model in different categories. As the number of samples in each category increased, the performance of the model in these categories also improved accordingly. It is worth noting that the proposed WavFusion model showed strong performance in most of the categories, especially in identifying the “Neutral” category with the largest sample size, achieving an impressive classification accuracy of 82.56%. However, the model achieved lower accuracy for the “Fear” and “Disgust” categories, which were 22% and 13.24% respectively. This performance is attributed to the relatively small sample size of these two categories. The limited training data for these emotions may have biased the model, resulting in suboptimal predictions for the test samples. In addition, samples were observed to be misclassified as “Neutral” in each category, such as Figure 5 As shown in the fourth column, we believe that this misclassification is partly caused by a serious imbalance in the sample size. The number of training samples for the "neutral" class is disproportionately large compared to other classes, which may affect the model's tendency to predict more instances as "neutral".

[0124] In summary, the analysis of IEMOCAP and MELD datasets revealed the importance of sample size distribution and the potential challenges posed by data imbalance. Future research should explore strategies to address these issues, such as class balancing, to improve the performance of the model and promote more robust multimodal speech emotion recognition.

[0125] Finally, it should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the restrictive conditions that can be implemented in this application, so they have no technical substantive significance. Any structural modification, change in proportional relationship or adjustment of size, without affecting the effects and purposes that can be achieved by this application, should still fall within the scope of the technical content disclosed in this application. The present invention is not limited to the above-mentioned best implementation mode. Anyone should be aware that the structural changes made under the enlightenment of the present invention, all of which have the same or similar technical solutions as the present invention, fall within the protection scope of the present invention.< / sep> < / cls>

Claims

1. A speech emotion recognition method based on multimodal feature fusion, characterized in that: Including auxiliary modality encoder, main modality encoder and multimodal homogeneous feature difference learning; It also includes the following processes: S1: Extract separate video and text features using auxiliary modality encoder; extract fixed T frames from each segment and use efficientNet pre-trained on VGGface and AFEW datasets as feature extractor to obtain visual features; S2: After obtaining the video features, they are input into the A-GRU-LVC module consisting of a gated recurrent unit, self-attention, and LVC block to capture global and local information; for text, the roberta-base pre-trained model is used as the feature extractor, and then the gated recurrent unit and self-attention mechanism are used to obtain global features; S3: Use wav2vec2.0 as an encoder to learn the contextual information of the audio sequence, embed the gated cross-modal attention block into wav2vec2.0, and finally, the shared encoder is responsible for integrating the features of the auxiliary modality and the main modality, utilizing multimodal homogeneous feature difference learning, focusing on distinguishing representations of the same modality but with different emotions.

2. According to claim 1, a method for speech emotion recognition based on multimodal feature fusion is characterized in that: Given a multimodal signal The unimodal original sequence extracted from video segment j is represented as Here, modality is denoted by {a, t, v}, referring to audio, text, and visual modalities. Multimodal emotion recognition aims to predict the emotion category of each utterance in the signal; it focuses on classifying the emotion conveyed in each person's speech and assigning it to a specific emotion category or class. c is the number of sentiment categories.

3. The method for speech emotion recognition based on multimodal feature fusion according to claim 1, characterized in that: The visual representation in the auxiliary modal encoder in step S1 specifically has the following process: For the visual modality, the efficientNet pre-trained model is used as a feature extractor to obtain visual features. It can be expressed as: where Φ visual Represents efficientNet pre-trained model function; The visual features obtained in the previous step are then input into the proposed A-GRU-LVC module, which is designed to extract global and local features. In the first block, the gated recurrent unit (GRU) and self-attention mechanism are used to capture the global long-range dependencies within the visual features. This block focuses on extracting global information from the visual input, enabling the model to understand the overall context and relationship between different visual elements. Among them, F SA and F GRU Represent the gated recurrent unit and self-attention mechanism learning functions respectively; At the same time, in order to preserve the local corner area and extract local information, a learnable visual center (LVC) is implemented on the visual features to aggregate the features of the local area and ensure that important local information is preserved; LVC is an encoder that contains an intrinsic dictionary, which contains two main components: an intrinsic codebook and a set of scaling factors related to the learnable visual center: B={b1,b2,...,b K } S={s1,s2,...,s K } Where K represents the number of embedding dimensions in the input features. In the case where the feature map is flattened during visual feature extraction, K can be viewed as the product of the spatial dimensions of the feature map. The visual features are encoded using a combination of convolutional layers, which, in contrast to the method, utilizes one-dimensional convolutions instead of two-dimensional convolutions. These encoded features are then processed by a CBR block, which contains a one-dimensional convolution with a kernel size of 3, a batch normalization layer, and a ReLU activation function to reprocess the visual features for further processing in the codebook. To map the visual features into the codebook, a set of scaling factors s is used k , these scaling factors will (represents the i-th visual feature th Pixels ) and b k (denotes the kth th codeword) is mapped to its corresponding item, the kth th The information of the entire image corresponding to the codeword can be calculated as follows: represents the information about the position of each pixel relative to the corresponding codeword, where K represents the total number of visual centers. Subsequently, the function Fusion of all e k , which includes 1D-BN (one-dimensional batch normalization) and ReLU layers and average pooling layers, the complete information of the entire image about K codewords is calculated as follows: After obtaining the output of the codebook, e is passed through a fully connected layer and a one-dimensional convolutional layer with a kernel size of 1 to predict features that specifically highlight the key classes; next, in the visual input feature A channel-by-channel multiplication is performed between and the scaling factor δ: δ=F sigmoid (Conv1d(e)) F sigmoid is the sigmoid activation function, It is a channel-by-channel multiplication. Then, the input visual feature And the channel summation between the local area features Z, is the channel sum and can be expressed as: Finally, by concatenating the output of the self-attention module along the last dimension and the output of the LVC block Get the output of the A-GRU-LVC block:

4. The method for speech emotion recognition based on multimodal feature fusion according to claim 1, characterized in that: The context text encoding in the auxiliary modal encoder in step S1 specifically includes: Tokenize the input text and convert the text into a series of tokens. After segmentation, roberta adds some special tags to the input text data, such as the start tag <cls>and separator <sep> , the start tag is used to indicate the beginning of the text, the separation tag is used to separate different sentences, then, in order to preserve the position information of the words, position embedding is added to each tag, which enables the model to understand the relative position of the words in the sentence in the input text, and finally, the processed data is input into the transformer encoder to capture the relationship between words and context information:< / sep> < / cls> It can be expressed as: where Φ text Represents the roberta pre-trained model learning function; After obtaining the text context embedding, it is input into the GRU and self-attention mechanism to obtain the global features of the text information. Due to the strong temporal continuity in the text information, the LVC mechanism is not used to capture local features: Among them, F SA and F GRU Represent the GRU and self-attention mechanism learning functions respectively.

5. The method for speech emotion recognition based on multimodal feature fusion according to claim 1, characterized in that: The main modality encoder specifically includes: a method for distinguishing by replacing the self-attention mechanism in the wav2vec2.0 transformer layer with a gated cross-modal attention mechanism, so that the text and visual The signal is encoded and fused with multiple layers of audio information. The low-level audio features are encoded through a shallow transformer layer, and then the text and visual features are combined through a modified transformer layer to form a comprehensive multimodal representation. Merging text and vision into wav2vec2.0 can detect relevant information from a large amount of pre-trained audio knowledge, thereby enhancing the emotional information in the multimodal fusion representation. The low-level acoustic features extracted by the shallow transformer block The calculation is as follows: F ST is the learning function of the shallow transformer layer; The audio features Text Features and visual features They are input into two cross-modal attention mechanisms to obtain inter-modal correlation information. Specifically, when performing attention calculation, the audio features are used as queries, and the text features or visual features are used as keys and values. The representation of audio features can be enhanced by introducing text or visual information into audio features: Then, the enhanced features and is fed into the subsequent gating and filtering mechanism; The ratio of each channel can be dynamically defined by a learnable parameter that can filter out erroneous information generated during cross-modal interaction.

6. The method for speech emotion recognition based on multimodal feature fusion according to claim 1, characterized in that: The multimodal homogeneous feature difference learning specifically includes: The unfused audio features Text Features and visual features Feeding into a shared encoder to obtain homogeneous features minimizes the feature gap between different modalities and helps multimodal alignment: SD is a shared encoder learning function consisting of a simple linear layer; Homogeneous representations from the same emotion but different modalities should be more similar than homogeneous representations from the same modality but different emotions. Therefore, in the work of [15], multimodal homogeneous feature difference learning is performed to enhance the interaction between the same emotion but different modalities and amplify the difference between the same modality but different emotions. This loss function is defined as the marginal loss: Among them, M = {(i, j, k) | m[i] ≠ m[j], m[i] = m[k], c[i] = c[j], c[i] ≠ c[k]} is a triple, m[i] is the modality type of sample i, c[i] is the label category of sample i, cos represents the cosine similarity between two feature vectors, and by applying the distance margin α, it is ensured that the distance between positive samples is smaller than the distance between negative samples. Here, positive samples refer to the same emotion but different sentiments, and negative samples refer to the same modality but different sentiments; As in many multimodal emotion recognition studies, cross entropy is often used as a loss function to optimize model parameters and improve classification accuracy during training: Among them, y j The true label of the sample, The predicted label of the sample, N D The total number of samples in the dataset D; Finally, the two constraints are combined to form the entire multimodal emotion recognition loss: Here, λ is the balancing factor.

Images