Audio-visual emotion recognition method and device

Abstract

The invention provides an audiovisual emotion recognition method and device, and the method comprises the steps: firstly extracting the emotion representation of a video mode and an audio mode, carrying out the high-order interaction and self-adaptive fusion in the modes and between the modes through a mutual attention module, fully capturing the deep dependence relation in different modes and between the modes, and carrying out the recognition of the mood of the video mode and the mood of the audio mode. And meanwhile, key information in each mode is reserved, and an emotion label is output through a classifier, so that the expression of emotion recognition is improved. According to the method, experimental verification is carried out on two public data sets, namely IEMOCAP and RAVDESS. The audio-visual emotion recognition method and the audio-visual emotion recognition device show more excellent performance on an emotion recognition task, and effectively improve the accuracy and the robustness of multi-modal emotion analysis.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition, and in particular to a method and device for audiovisual emotion recognition. Background Art

[0002] In the field of multimodal emotion recognition (MER), current research has largely focused on the fusion and interaction of bimodal or trimodal features. Furthermore, most of this research focuses on the textual modality of speech or contextual dependencies within continuous conversations. However, while text can convey emotional information, it suffers from polysemy (a single word can have multiple meanings), while visual cues (such as facial expressions) and sounds typically carry richer and more direct emotional information. In real-world scenarios, emotions are often expressed in facial expressions and sounds in a natural and instantaneous manner, often accompanied by dynamic changes in facial expressions and vocal features during speech, rather than static expressions. Therefore, our research focuses on audio-visual emotion recognition (AVER), which is independent of speaker and context relationships, focusing on dynamic visual cues and sounds.

[0003] In recent years, the field of emotion recognition (ER) has made significant progress with the development of deep learning and multimodal fusion technologies. While traditional methods primarily rely on manual feature extraction and classification, modern approaches increasingly employ end-to-end deep learning models to automatically learn emotion-related features from raw data. Furthermore, the introduction of technologies such as attention mechanisms, self-supervised learning, and cross-modal information exchange have further enhanced the performance of audiovisual emotion recognition.

[0004] Existing mainstream AVER research methods mostly use transformers or attention mechanisms to explore audio-visual fusion, but most of them fail to fully capture the fine-grained correlations and information complementarity within and between modalities, especially at the level of modal feature dimensions (such as the correspondence between facial expression changes and sound features).

[0005] Existing emotion recognition methods often focus on a single modality or rely on simple weighted fusion strategies, which makes it difficult to fully explore the complex relationships and potential information within and between different modalities. Summary of the Invention

[0006] The purpose of the present invention is to provide a method and device for audio-visual emotion recognition, which calculates the correlation between the feature levels of modalities, thereby helping the model to better integrate information from different modalities.

[0007] To achieve the above object, the present invention provides a method for audiovisual emotion recognition, comprising the following steps:

[0008] Extracting a first visual feature F′ and a first audio feature S′ from the original video clip;

[0009] After obtaining the first visual feature F′ and the first audio feature S′, convolution is performed on them respectively to align them in the time dimension and feature dimension and locate them in the same space, and the aligned second visual feature F is obtained. v and the second audio feature F a ;

[0010] Then the aligned second visual feature F v and the second audio feature F a The data is sent to the mutual attention module for interaction to obtain the emotional representation that integrates the audiovisual emotional information;

[0011] The emotional representation that integrates audiovisual emotional information is weighted into the input of the mutual attention module and pooled in the temporal dimension to obtain the audiovisual modal representation that integrates audiovisual emotional information. The audiovisual modal representations are then connected together and input into the classifier to obtain specific audiovisual emotions. A cross-loss function is set to estimate the quality of emotion prediction during training.

[0012] Preferably, the mutual attention module includes:

[0013] Self-attention unit is used to capture the second visual feature F v and the second audio feature F a The time dependency of the third visual feature H is obtained v→v and the third audio feature H a→a ;

[0014] Cross-attention unit to capture the third visual feature H v→v and the third audio feature H a→a The time and feature level interactions and potential relationships between them are used to obtain the first fusion feature H a→v and the second fusion feature H v→a ;

[0015] Adaptive fusion strategy for adaptively learning the third visual feature H v→v and the third audio feature H a→a Independent weights, the first fusion feature H a→v and the second fusion feature H v→a The weight between .

[0016] Preferably, the second visual feature F of the self-attention unit is used v and the second audio feature F a The time dependency of the third visual feature H is obtained v→v and the third audio feature H a→a , including the following steps:

[0017] Visual feature F v and audio feature F a Input into two self-attention modules respectively. The self-attention module projects the input features into three different subspaces: query Q, key K, and value V, with a given weight matrix;

[0018] Then, the dot product between the query Q and the key K is calculated to measure the relevance at the temporal level, and the attention score is obtained. Softmax normalization is then performed on it to obtain the attention weight α.

[0019] In the visual modality, if the expression changes significantly at a certain time step, it is automatically given a higher weight. In the speech modality, if the pitch, speaking rate, or energy changes dramatically at a certain moment, attention is automatically paid to that time step. Finally, the attention weight α is used to perform a weighted summation of the value V, dynamically focusing on important information at different time steps.

[0020] After calculating the attention, in order to prevent excessive attention to a certain feature or a certain time, residual connection and normalization are used, and then an adaptive weight fusion strategy is applied within the modality to obtain the third visual feature H v→v and the third audio feature H a→a .

[0021] Preferably, a cross attention unit is used to capture the third visual feature H v→v and the third audio feature H a→a The time and feature level interactions and potential relationships between them are used to obtain the first fusion feature H a→v and the second fusion feature H v→a , including the following steps:

[0022] For the third visual feature H v→v and the third audio feature H a→a They are projected into different spaces, so that they cross-project into each other's Q and themselves are projected into K and V;

[0023] Then, residual connection and normalization are applied to obtain weighted fusion, and an adaptive weight fusion strategy is applied between modalities to make the third visual feature H v→v and the third audio feature H a→a The first fusion feature H is obtained by interaction at the time level and feature level a→v and the second fusion feature H v→a .

[0024] Preferably, the calculation method of the cross attention unit is

[0025]

[0026] Where, CAX represents cross-modal attention, Softmax() represents the normalized exponential function, T represents transpose, d represents the aligned feature dimension, V represents the value vector, Q represents the query vector, Indicates that the feature space belongs to t×d, t represents the time dimension after alignment, X represents the modality, F represents the visual modality, and S represents the audio modality.

[0027] Preferably, the classifier is calculated as

[0028]

[0029] in, The feature space of the weight vector W belongs to 2d×C, where C is the number of emotion categories. is the emotion prediction probability, M represents the fusion feature after splicing, and N represents the number of samples;

[0030] The cross entropy loss function is

[0031]

[0032] Among them, N represents the number of samples, y i,j represents the true category, Represents the predicted category.

[0033] An audiovisual emotion recognition device, comprising

[0034] The feature extraction module is used to extract the first visual feature F′ and the first audio feature S′ from the original video clip; after obtaining the first visual feature F′ and the first audio feature S′, they are convolved separately to align them in the time dimension and feature dimension and locate them in the same space, and obtain the aligned second visual feature F v and the second audio feature F a ;

[0035] Mutual attention module for the second visual feature F after interactive alignment v and the second audio feature F a , obtain the emotional representation that integrates the audiovisual emotional information, weight the emotional representation that integrates the audiovisual emotional information to the input of the mutual attention module and pool it in the time dimension to obtain the audiovisual modal representation that integrates the audiovisual emotional information, and then connect the audiovisual modal representations together;

[0036] The classifier is used to classify the connected audiovisual modal representations to obtain specific audiovisual emotions and set the cross loss function to estimate the quality of emotion prediction during training.

[0037] Therefore, the present invention adopts the above-mentioned audio-visual emotion recognition method and device, and the technical effects are as follows:

[0038] A feature interaction and adaptive fusion model based on mutual attention module (MAAF) is proposed. The model consists of a feature extraction module and a mutual attention module, which aims to capture the interaction and information complementarity within and between modalities, and introduces an adaptive fusion strategy to automatically learn the inter-modal weights.

[0039] A new cross-modal attention calculation method is proposed, which focuses on capturing information interaction at the feature level between modalities. This exploration fills the gaps in previous research on fine-grained correlation mining.

[0040] Extensive experiments are conducted on two basic multimodal sentiment analysis datasets, RAVDESS and IEMOCAP, and the experimental results verify the superior performance of the proposed model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of an audiovisual emotion recognition method of the present invention;

[0042] Figure 2 This is the diagram of the intra-modal adaptive fusion structure;

[0043] Figure 3 This is the structure diagram of inter-modal adaptive fusion. DETAILED DESCRIPTION

[0044] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0045] Unless otherwise defined, technical or scientific terms used in the present invention shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0046] Example 1

[0047] like Figure 1 As shown, a method for audio-visual emotion recognition includes the following steps:

[0048] Step 1: Extracting the first visual feature F′ and the first audio feature S′ from the original video clip, including the following steps:

[0049] Audiovisual emotion analysis aims to detect the emotional state of people in video clips using both visual and audio modalities. Given a video clip, it consists of two components: audio (a) and visual sequences (v). The textual modality is typically a transcription of the audio, but this is not considered for this example. The focus is on emotion recognition by combining facial expressions with audio signals.

[0050] Use Efficientface to extract the face frame sequence I=(i1,i2,…,i k ,i∈R H×W×3) is represented by the following formula:

[0051]

[0052] Among them, θ represents the set of adjustable parameters of the model, k represents the face of k frame or the face of k moment, d v Represents the dimension of the visual feature vector.

[0053] Similarly, for the audio part, let the original input audio signal be S∈R T , after audio feature extraction network or manual feature extraction, the audio features can be expressed as:

[0054]

[0055] Where θ represents the set of adjustable parameters of the model, T represents the time dimension, and d a Represents the dimension of the audio feature vector.

[0056] Step 2: After obtaining the first visual feature F′ and the first audio feature S′, convolution is performed on them respectively to align them in the time dimension and feature dimension and locate them in the same space, and the aligned second visual feature F is obtained. v and the second audio feature F a ;

[0057] To ensure that the representations of unimodal sequences of different modalities are in the same space, the first visual feature F′ and the first audio feature S′ are input into a one-dimensional convolutional layer:

[0058]

[0059] Where d is the common feature dimension, and t is the aligned time feature dimension. The time dimension of the audio is aligned with the time dimension of the face frame sequence to represent the facial expression when speaking at time t.

[0060] Table 1 1D convolution structure table

[0061]

[0062] Facial expression sequence F v ={f1,f2,…,f t}where f t is the facial image captured at time t. The speech signal sequence F a ={s1,s2,…,s t}where S tis the audio feature (MFCC or fbank) extracted at time t. This embodiment explores the potential correlation between facial expressions (especially changes in lips, eyebrows, and eye movements) and speech signals, and mines the complementary information of facial expressions and sounds (as shown in Table 1) to improve the accuracy of emotion recognition. Since facial expressions usually show continuous dynamic changes when people express emotions, it is necessary to effectively capture the evolution of expressions between different frames. At the same time, speech signals also have temporal continuity, so it is necessary to study the relationship between speech signals and corresponding facial frames to more accurately model the interaction of audiovisual modalities, thereby improving the effect of emotion recognition.

[0063] Table 2 Emotion-expression-sound correspondence table

[0064]

[0065]

[0066] Step 2: Then, the aligned second visual feature F v and the second audio feature F a The data is sent to the mutual attention module for interaction to obtain the emotional representation that integrates the audiovisual emotional information;

[0067] The mutual attention module includes:

[0068] Self-attention unit is used to capture the second visual feature F v and the second audio feature F a The time dependency of the third visual feature H is obtained v→v and the third audio feature H a→a ;

[0069] Cross-attention unit to capture the third visual feature H v→v and the third audio feature H a→a The time and feature level interactions and potential relationships between them are used to obtain the first fusion feature H a→v and the second fusion feature H v→a ;

[0070] Adaptive fusion strategy for adaptively learning the third visual feature H v→v and the third audio feature H a→a Independent weights, the first fusion feature H a→v and the second fusion feature H v→a The weight between .

[0071] The second visual feature F is used by the self-attention unit v and the second audio feature F a The time dependency of the third visual feature H is obtained v→v and the third audio feature Ha→a .

[0072] Self-Attention: Self-attention is a mechanism for capturing internal dependencies in sequential data and is commonly used in tasks such as natural language processing (NLP) and computer vision (CV). It considers all elements of the entire input sequence, not just local information, during processing, making it suitable for modeling temporal dependencies and global features. For example, in the visual modality (sequence of face frames), it can analyze the dynamic changes in facial expressions over time and understand the trends of expression changes. In the audio modality (speech signal), it can capture the temporal dynamic characteristics of speech and understand the changing patterns of emotion in speech.

[0073] The following steps are involved:

[0074] Visual feature F v and audio feature F a They are input into two self-attention modules respectively. The self-attention module projects the input features into three different subspaces: query Q, key K, and value V. Given the weight matrix The query Q, key K, and value V in cross-modal attention are calculated as:

[0075]

[0076] Where a represents the audio modality and v represents the visual modality.

[0077] Then, the dot product between the query Q and the key K is calculated to measure the relevance at the temporal level, and the attention score is obtained. Softmax normalization is then performed on it to obtain the attention weight α.

[0078] In the visual modality, if the expression changes significantly at a certain time step, it is automatically given a higher weight. In the speech modality, if the pitch, speaking rate, or energy changes dramatically at a certain moment, attention is automatically paid to that time step. Finally, the attention weight α is used to perform a weighted summation of the value V, dynamically focusing on important information at different time steps.

[0079]

[0080] The self-attention mechanism can capture temporal emotional dynamics. In facial frame sequences, it helps the model understand how facial expressions change over time. For example, facial features like lips, eyes, and eyebrows, as well as facial muscles, exhibit distinct movement patterns depending on emotion. In audio, the self-attention mechanism can learn variations in the rhythm, pitch, and energy of speech, enabling more accurate emotion recognition. In summary, the self-attention mechanism can enhance the weight of information at key time steps. For example, when a person is happy, a key frame containing a smile may be more emotionally expressive than other frames. In speech, a person may raise their voice at certain points to express emotion, and self-attention can focus on these key points. The self-attention mechanism also reduces redundancy and improves information utilization. Traditional RNN structures (such as LSTM) often suffer from long-range dependencies. The self-attention mechanism directly allows information from different time steps to interact, thus avoiding these long-range dependencies. In emotion recognition, this means that even if the emotional information in the current frame is weak, the model can still refer to information from previous and subsequent frames to reach an accurate emotional judgment.

[0081] Residual and Normalization (A&N): After calculating the attention, residual connection and normalization are used to prevent excessive attention to a certain feature or a certain time.

[0082] X SA ′=layernorm(X+SA X ),X∈{F,S} (7);

[0083] Where layernorm() represents layer normalization.

[0084] Then an adaptive weight fusion strategy is applied within the modality, such as Figure 2 As shown, σ is the sigmoid function, and the third visual feature H is obtained v→v and the third audio feature H a→a The function of this layer is to filter the most important information within each modality and prevent irrelevant information from interfering with the fusion process. The calculation process is:

[0085] X SA =X SA ′⊙sigmoid(WX SA ′+b),X∈{F,S} (8);

[0086] Where ⊙ represents the element-by-element product.

[0087] Intra-modal adaptive weight fusion allows the visual and audio modalities to extract the most effective information, preventing irrelevant features from interfering with the final decision. For example, if a certain dimension of the visual feature is unimportant, that information is automatically suppressed. If an audio feature at a certain time step is irrelevant to emotion, its contribution is automatically reduced.

[0088] After obtaining temporal dependencies and global features across modalities, cross-modal attention is used to model the interactions between modalities, specifically the similarities and complementarities between the visual modality (sequence of facial frames) and the audio modality (speech signal). This allows the model to extract information from one modality that is helpful for the other, thereby enhancing sentiment analysis. Unlike self-attention, cross-modal attention's core concept is to use features from one modality as the "query (Q)" and features from the other modality as the "key (K)" and "value (V)" to learn intermodal associations.

[0089] Capturing the third visual feature H using crisscross attention units v→v and the third audio feature H a→a The time and feature level interactions and potential relationships between them are used to obtain the first fusion feature H a→v and the second fusion feature H v→a , including the following steps:

[0090] For the third visual feature H v→v and the third audio feature H a→a They are projected into different spaces, so that they cross-project into each other's Q and themselves are projected into K and V;

[0091]

[0092] n, m represent the visual and audio modalities respectively.

[0093] Most methods using cross-modal attention can only capture the temporal correlation between modalities, but fail to capture the interaction at the feature level. To address this problem, we propose a cross-modal attention calculation method.

[0094]

[0095] Then, residual connection and normalization are applied to obtain weighted fusion, and an adaptive weight fusion strategy is applied between modalities to make the third visual feature H v→v and the third audio feature H a→a The first fusion feature H is obtained by interaction at the time level and feature level a→v and the second fusion feature H v→a .

[0096] Apply residual connection and standardization to obtain the modal features of the interaction between the time level and feature level after weighted fusion.

[0097] X CA ′=layernorm(X SA +CA X ),X∈{F,S} (11);

[0098] Cross-modal attention based on feature interaction can promote cross-modal information fusion. If the visual channel and audio channel have strong correlations in certain dimensions, feature attention can help the model better integrate this information, thereby improving the accuracy of emotion classification. For example, the emotional characteristics of the voice expressed during speech will be intuitively reflected in the visual features (facial expressions of the face), and different emotions will have different vocal expressions and facial expression changes. When high-energy audio fluctuations are detected in the audio features, the visual features may enhance the degree of eye and mouth openness, thereby forming a complementary emotional representation.

[0099] Inter-modality adaptive weight fusion: Figure 3 As shown in , the role of this layer is to learn the complementarity of vision and audio, ensuring that the final fusion feature can fully utilize the information of the two modalities. The calculation process is:

[0100] M=[F CA ′,S CA ′]⊙Softmax([W v F CA ′,W a S CA ′]+b) (12);

[0101] Where F' CA Represents the visual modality features after weighted fusion, S' CA Represents the audio modal features after weighted fusion, W v Represents the adaptive weight vector of visual modality features, W a The adaptive weight vector represents the audio modal features, b represents the bias term, M represents the fused features, and [,] represents concatenation. Inter-modal adaptive weight fusion ensures that the final fused features fully utilize information from both modalities without losing key emotional cues. It also allows the sentiment analysis model to adaptively adjust modal weights for different input conditions, rather than relying solely on information from a single modality.

[0102] The visual modality can provide clues to changes in facial expressions, such as features of the mouth and eye areas, while the audio modality can provide clues such as voice intensity, pitch, and speaking rate. For example, the pitch rises with anger and falls with sadness. Using cross-modal attention based on feature interactions to fuse visual and audio features allows the visual modality to focus on key information in the audio modality, while the audio modality also focuses on key information in the visual modality. This, combined with the complementary properties of facial expressions and voice, improves the robustness of emotion classification.

[0103] Through the mutual attention module, temporal dependencies within the modalities are first modeled to capture the dynamic features of the visual and audio modalities. The visual and audio modalities are then allowed to interact to identify the most relevant features, enabling better information fusion. Finally, adaptive weight fusion is applied to allow the visual and audio modalities to complement each other, resulting in higher-quality fused features. This enables the model to dynamically adjust modal weights, extract key information, and enhance inter-modal complementarity, thereby improving the accuracy and robustness of sentiment classification.

[0104] Step 3: The emotional representation that integrates audiovisual emotional information is weighted and added to the input of the mutual attention module and pooled in the temporal dimension to obtain an audiovisual modal representation that integrates audiovisual emotional information. The audiovisual modal representations are then concatenated and input into the classifier to obtain the specific audiovisual emotion.

[0105]

[0106] in C is the number of emotion categories, Predict the probability for the emotion.

[0107] Set the cross-loss function to estimate the quality of sentiment prediction during training.

[0108]

[0109] Where C represents the number of categories, N represents the number of samples, and y i,j represents the true category, Represents the predicted category.

[0110] An audiovisual emotion recognition device, comprising

[0111] The feature extraction module is used to extract the first visual feature F′ and the first audio feature S′ from the original video clip; after obtaining the first visual feature F′ and the first audio feature S′, they are convolved separately to align them in the time dimension and feature dimension and locate them in the same space, and obtain the aligned second visual feature F v and the second audio feature F a ;

[0112] Mutual attention module for the second visual feature F after interactive alignment v and the second audio feature F a , obtain the emotional representation that integrates the audiovisual emotional information, weight the emotional representation that integrates the audiovisual emotional information to the input of the mutual attention module and pool it in the time dimension to obtain the audiovisual modal representation that integrates the audiovisual emotional information, and then connect the audiovisual modal representations together;

[0113] The classifier is used to classify the connected audiovisual modal representations to obtain specific audiovisual emotions and set the cross loss function to estimate the quality of emotion prediction during training.

[0114] Example 2

[0115] Using the method of Example 1, experiments were conducted on RAVDESS and IEMOCAP, two benchmark datasets widely used in multimodal sentiment analysis, to verify the effectiveness of the proposed method.

[0116] RAVDESS dataset: This dataset selects speech data from the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). This dataset consists of 1,440 audio and video samples recorded by 24 actors (12 male and 12 female) speaking with varying emotions. The dataset proposes a task for classifying emotional states into eight categories: neutral, calm, happy, sad, angry, fearful, surprised, and disgusted. Each actor recorded 60 video sequences. We cropped or zero-padded these sequences to an average sequence length of 3.6 seconds.

[0117] IEMOCAP (Interactive Emotional Dyadic Motion Capture Database) is a larger-scale emotional dialogue dataset featuring dialogues performed by 10 actors. The dataset covers 10 emotion categories: neutral, happiness, sadness, anger, surprise, fear, disgust, frustration, excitement, and others, encompassing audio, video, and text modalities. Because this dataset is used for audio-visual emotion recognition, only the video and audio modalities were extracted. Six emotion categories with a relatively balanced number of categories were selected: neutral, happiness, sadness, anger, excitement, and frustration. These were then cropped or zero-padded to an average sequence length of 5.0 seconds.

[0118] Evaluation Metrics: We use a variety of quantitative metrics to provide a comprehensive assessment of the overall model performance. These metrics include the widely used F1 score and overall accuracy (WA), which are used to account for class imbalance and better capture the average performance of different classes. We report the overall accuracy (WA) and weighted average w-F1 score to measure overall performance.

[0119] Visual Branch: Facial expressions in videos involve both facial appearance and temporal dynamics in video sequences. Effectively modeling these spatial and temporal dynamics is crucial for extracting discriminative and robust features, thereby improving overall system performance. In the audiovisual emotion recognition model proposed in this paper, the visual branch is primarily responsible for extracting dynamic facial expression features. First, the raw emotional videos are preprocessed (frame extraction, MTCNN face detection, cropping, and data augmentation) to produce a sequence of emotionally rich facial frames. Subsequently, the visual branch extracts high-level facial expression features from the frame sequence and aligns the features of different modalities using 1D convolution. To achieve an end-to-end trainable model capable of learning from raw emotional videos, the feature extraction module is integrated into the network and optimized jointly with the fusion module. Unlike most existing methods, which separate feature extraction from multimodal fusion and primarily utilize pre-extracted features such as facial landmark locations, facial action units (AUs), or head pose information, this approach does not.

[0120] To construct an efficient and robust visual feature extraction module, we selected the recently proposed lightweight and efficient facial expression recognition architecture, EfficientFace. This architecture, based on the ShuffleNet-V2 architecture, enhances facial expression recognition capabilities by introducing two key modules: a local feature extractor and a channel-spatial modulator. The local feature extractor divides the low-level global feature map, which has undergone preliminary convolution, into multiple local regions and extracts features from each of these regions using depthwise separable convolutions. This approach captures subtle facial expression details and effectively mitigates information loss caused by facial occlusion or partial distortion, thereby improving the model's robustness in complex scenes. The channel-spatial modulator, on the other hand, uses channel-attention and spatial-attention to weight high-level features. This emphasizes the most discriminative global feature regions (such as the eyes and mouth corners) while reducing interference from irrelevant information, effectively improving recognition challenges caused by pose changes or background clutter.

[0121] In the visual feature extraction module, the fully connected classification layer of EfficientFace is removed, and the high-dimensional feature vector (usually 512 or 1024 dimensions) of its penultimate layer (or feature extraction layer) is extracted. This vector contains rich information such as facial muscle movement patterns and key expression changes, which serves as the input for subsequent multimodal fusion.

[0122] The 2D-CNN + 1D convolution approach was chosen over direct 3D convolution because it offers several advantages for the given task. The first is that 2D convolution layers incur lower computational overhead compared to 3D convolution, and applying a 1D convolution operation in the temporal dimension is sufficient to capture this information. The second advantage is that video frames are processed separately to ensure that only the face region is extracted, thereby reducing background interference.

[0123] 2) Audio Branch: In the audio feature extraction phase, the raw audio signal is first preprocessed, primarily including noise removal, silence detection, and volume normalization, to ensure the reliability of subsequent feature extraction. The preprocessed audio signal is divided into several overlapping short-time frames and processed using an appropriate window function (such as a Hamming window) to capture the signal's stable characteristics over short periods of time. Next, low-level acoustic features, such as Mel-Fequency Cepstral Coefficients (MFCCs), fundamental frequency, energy, and other spectral features, are extracted from each frame. These features effectively reflect the emotional information in speech. To capture the temporal dynamics and higher-level emotional semantics of the audio signal, the extracted low-level features are then refined using deep neural networks (such as convolutional neural networks and recurrent neural networks) to obtain semantically rich high-level feature representations. Ultimately, these audio features, after deep feature learning, are integrated with facial expression features extracted from the visual branch in multimodal sentiment analysis to enhance the accuracy and robustness of emotion recognition.

[0124] For audio processing, 13 mel-frequency cepstral coefficients were extracted for further processing. For visual data, 15 evenly distributed frames were selected from the video and the actor's face was cropped using the face detection algorithm MTCNN. The image was resized to 224x224 pixels. We trained the model on the original 15-frame video and split the data into training, validation, and test sets to ensure that the identity of the participants was not repeated between sets. The video was scaled to [0,1] and data augmented using random horizontal flipping and random rotation. The audio data was randomly added with noise intensity and random audio transformation. All models were trained for 50 epochs with the optimizer being SGD, a learning rate of 0.04, a momentum of 0.9, and a weight decay of 1e-3, with the learning rate decreasing every 20 epochs.

[0125] On the RAVDESS and IEMOCAP datasets, we compared MFFA with the overall accuracy (WA) of various methods. In addition, our model also reports the weighted F1 score, but the weighted average W-F1 score cannot be obtained for methods that do not provide source code.

[0126] Table 3 Comparative experiments on the RAVDESS dataset

[0127]

[0128]

[0129] Table 4 Comparative experiments on IEMOCAP dataset

[0130]

[0131] Tables 3 and 4 show the performance of the baselines and MAAF on the RAVDESS and IEMOCAP datasets, respectively. On the RAVDESS dataset, MAAF outperforms all baseline models, with an overall accuracy (WA) score that is 2.44% higher than the nearest competitor, MultiMAE-DER. On the IEMOCAP dataset, MFFA outperforms all baseline models in terms of overall accuracy (WA) and weighted average W-F1 score, with an overall accuracy (WA) score that is 1.25% higher than the nearest competitor, HeadFusion.

[0132] The processing and hyperparameter settings for both datasets are identical, but the table shows a difference in their accuracy. This is because IEMOCAP has lower facial resolution than RAVDESS, and the IEMOCAP dataset consists of conversational videos, so most faces are angled. Furthermore, the emotional expressions in the IEMOCAP dataset are more realistic, resulting in lower accuracy for MFFA on the IEMOCAP dataset. Therefore, using higher-quality datasets and adding noise to the data to simulate real-world data instability are crucial for model performance and robustness.

[0133] Most methods use the attention mechanism, indicating that the attention mechanism is widely used in multimodal fusion. Adaptive fusion first filters out irrelevant information at the unimodal level and highlights important information. Then, it dynamically learns the weights between different modalities at the multimodal level, thereby more effectively fusing multimodal representations and having huge advantages in modal fusion.

[0134] Ablation experiments were performed on RAVDESS and IEMOCAP. Table 5 reports the results under different ablation settings.

[0135] Table 5 Ablation experiment results

[0136]

[0137]

[0138] Self-attention, cross-modal attention based on feature interaction, and adaptive fusion are the three key components of our proposed mutual attention module. We remove only one component at a time to evaluate the effectiveness of the component. In addition, we compare the ordinary cross-modal attention with our proposed cross-modal attention based on feature interaction. From Table 5, we conclude that: (1) All components are useful, as removing one of them leads to performance degradation. (2) Adaptive fusion plays an important role. After removing this module, the overall accuracy (WA) and weighted average W-F1 decrease by 2.04% and 1.84%, respectively. (3) The cross-modal attention based on feature interaction performs better than the ordinary cross-modal attention classification, with an improvement of 1.78% and 2.15% in the overall accuracy (WA) and weighted average W-F1 score, respectively, indicating the effectiveness of our proposed cross-modal attention based on feature interaction. (4) The cross-modal attention based on feature interaction has a relatively large impact, as the results show that after removing this module, the overall accuracy (WA) and weighted average W-F1 decrease by 2.71% and 2.28%, respectively, which means that it is valuable to capture the feature-level correlation between feature sequences.

[0139] Effects of different modalities: To compare the effects of different modalities, we compared the effects of single modality classification and audiovisual modality classification. From Table 5, we can conclude that: (1) For single modality, the sound modality and the visual modality are equally important, because they have similar overall accuracy (WA) and weighted average W-F1 scores in single modality emotion recognition. (2) The audiovisual modality results are better than the single modality results. From Table 5, we can conclude that audiovisual dual-modality emotion recognition has a significant improvement in performance compared to single-modality emotion recognition.

[0140] In summary, the above results demonstrate the effectiveness of MFFA.

[0141] Therefore, the present invention adopts the above-mentioned audio-visual emotion recognition method and device, which shows superior performance in emotion recognition tasks and effectively improves the accuracy and robustness of multimodal sentiment analysis.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for audio-visual emotion recognition, characterized in that: The following steps are involved: Extracting a first visual feature F′ and a first audio feature S′ from the original video clip; After obtaining the first visual feature F′ and the first audio feature S′, convolution is performed on them respectively to align them in the time dimension and feature dimension and locate them in the same space, and the aligned second visual feature F is obtained. v and the second audio feature F a ; Then the aligned second visual feature F v and the second audio feature F a The data is sent to the mutual attention module for interaction to obtain the emotional representation that integrates the audiovisual emotional information; The emotional representation that integrates audiovisual emotional information is weighted into the input of the mutual attention module and pooled in the temporal dimension to obtain the audiovisual modal representation that integrates audiovisual emotional information. The audiovisual modal representations are then connected together and input into the classifier to obtain specific audiovisual emotions. A cross-loss function is set to estimate the quality of emotion prediction during training.

2. The audiovisual emotion recognition method according to claim 1, characterized in that: The mutual attention module includes: Self-attention unit is used to capture the second visual feature F v and the second audio feature F a The time dependency of the third visual feature H is obtained v→v and the third audio feature H a→a ; Cross-attention unit to capture the third visual feature H v→v and the third audio feature H a→a The time and feature level interactions and potential relationships between them are used to obtain the first fusion feature H a→v and the second fusion feature H v→a ; Adaptive fusion strategy for adaptively learning the third visual feature H v→v and the third audio feature H a→a Independent weights, the first fusion feature H a→v and the second fusion feature H v→a The weight between .

3. The audiovisual emotion recognition method according to claim 2, characterized in that: Use the second visual feature F of the self-attention unit v and the second audio feature F a The time dependency of the third visual feature H is obtained v→v and the third audio feature H a→a , The following steps are involved: Visual feature F v and audio feature F a Input into two self-attention modules respectively. The self-attention module projects the input features into three different subspaces: query Q, key K, and value V, with a given weight matrix; Then, the dot product between the query Q and the key K is calculated to measure the relevance at the temporal level, and the attention score is obtained. Softmax normalization is then performed on it to obtain the attention weight α. In the visual modality, if the expression changes significantly at a certain time step, it is automatically given a higher weight. In the speech modality, if the pitch, speaking rate, or energy changes dramatically at a certain moment, attention is automatically paid to that time step. Finally, the attention weight α is used to perform a weighted summation of the value V, dynamically focusing on important information at different time steps. After calculating the attention, in order to prevent excessive attention to a certain feature or a certain time, residual connection and normalization are used, and then an adaptive weight fusion strategy is applied within the modality to obtain the third visual feature H v→v and the third audio feature H a→a .

4. The audiovisual emotion recognition method according to claim 2, wherein: Capturing the third visual feature H using crisscross attention units v→v and the third audio feature H a→a The time and feature level interactions and potential relationships between them are used to obtain the first fusion feature H a→v and the second fusion feature H v→a , including the following steps: For the third visual feature H v→a and the third audio feature H a→a They are projected into different spaces, so that they cross-project into each other's Q and themselves are projected into K and V; Then, residual connection and normalization are applied to obtain weighted fusion, and an adaptive weight fusion strategy is applied between modalities to make the third visual feature H v→v and the third audio feature H a→a The first fusion feature H is obtained by interaction at the time level and feature level a→v and the second fusion feature H v→a .

5. The audiovisual emotion recognition method according to claim 2, characterized in that: The calculation method of the cross attention unit is Where, CA X represents cross-modal attention, Softmax() represents the normalized exponential function, T represents transpose, d represents the aligned feature dimension, V represents the value vector, Q represents the query vector, Indicates that the feature space belongs to t×d, t represents the time dimension after alignment, X represents the modality, F represents the visual modality, and S represents the audio modality.

6. The audiovisual emotion recognition method according to claim 1, characterized in that: The classifier is calculated as in, The feature space of the weight vector W belongs to 2d×C, where C is the number of emotion categories. is the emotion prediction probability, M represents the fusion feature after splicing, and N represents the number of samples; The cross entropy loss function is Among them, N represents the number of samples, y i,j represents the true category, Represents the predicted category.

7. An audiovisual emotion recognition device, characterized in that: include The feature extraction module is used to extract the first visual feature F′ and the first audio feature S′ from the original video clip; after obtaining the first visual feature F′ and the first audio feature S′, they are convolved separately to align them in the time dimension and feature dimension and locate them in the same space, and obtain the aligned second visual feature F v and the second audio feature F a ; Mutual attention module for the second visual feature F after interactive alignment v and the second audio feature F a , obtain the emotional representation that integrates the audiovisual emotional information, weight the emotional representation that integrates the audiovisual emotional information to the input of the mutual attention module and pool it in the time dimension to obtain the audiovisual modal representation that integrates the audiovisual emotional information, and then connect the audiovisual modal representations together; The classifier is used to classify the connected audiovisual modal representations to obtain specific audiovisual emotions and set the cross loss function to estimate the quality of emotion prediction during training.

Images