Missing modal emotion recognition method and system based on cross-modal attention fusion

Abstract

The invention provides a missing modal emotion recognition method and system based on cross-modal attention fusion, and belongs to the technical field of emotion recognition, and the method comprises the steps: obtaining multi-modal data of a to-be-recognized emotion; marking a missing mode indication for the missing mode of the multi-mode data, and carrying out feature preprocessing to obtain a primary feature and a preprocessing feature; performing feature extraction on the primary features of each mode to obtain advanced features; a cross-modal attention fusion layer is utilized to calculate attention scores between the modals for the advanced features of all the modals, weighted summation is carried out on the advanced features according to the attention scores, meanwhile, the fusion degree of the features of all the modals is dynamically adjusted through a gating fusion mechanism, and preliminary fusion features are obtained; fusing the preliminary fusion features and the pre-processed features through a residual connection module to obtain full-modal features; and performing emotion recognition on the full-modal features by adopting a deep classification network. According to the method, the mutual relationship among different modal data is learned to realize effective identification of multi-modal emotion.

Description

Technical Field

[0001] The present invention belongs to the field of emotion recognition technology, and in particular relates to a missing modality emotion recognition method and system based on cross-modal attention fusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Emotion recognition, a key branch of artificial intelligence, has become a core research direction by integrating cross-modal information to analyze emotional states. Compared to single-modal approaches, multimodal fusion strategies offer significant advantages in the completeness and accuracy of emotion representation, demonstrating irreplaceable technical value in scenarios such as intelligent service systems, human-computer collaborative interaction, and clinical psychological assessment.

[0004] In recent years, the rapid development of the Internet and multimedia technologies has spawned a large amount of multimodal data, and multimodal emotion recognition has also received widespread attention. However, in actual application scenarios, the problem of missing modal data is very common, that is, some modal data may be unavailable or of poor quality, which further increases the difficulty of emotion recognition. Current multimodal models are basically learned and trained based on complete data, but they often perform poorly when faced with incomplete data. Single modal data often has the limitation of incomplete information representation: due to the implicitness of text language, it is difficult to accurately convey emotions; although audio tone and speed can reflect emotional tendencies, the lack of visual assistance can easily lead to misjudgment; intuitive video facial expressions and body movements lack semantic information, which makes it difficult to achieve comprehensive emotional understanding.

[0005] Early research employed traditional machine learning methods, such as support vector machines (SVMs) and hidden Markov models (HMMs), to achieve sentiment classification through handcrafted features. However, these methods suffer from insufficient feature association mining when processing unstructured multimodal data. Furthermore, fine-tuning traditional pre-trained models on small datasets can lead to instability. Therefore, when any modal data—text, audio, or video—is missing, traditional methods are unable to effectively process the missing information, leading to a sharp deterioration in recognition performance. Furthermore, existing cross-modal interaction mechanisms in deep learning approaches are limited to bimodal attention, making it difficult to fully capture the complex collaborative relationships between text, audio, and video. Furthermore, fixed-weight feature fusion methods cannot dynamically suppress noise modal interference, resulting in inefficient information fusion. Summary of the Invention

[0006] In order to overcome the shortcomings of the above-mentioned existing technologies, the present invention proposes a missing modality emotion recognition method and system based on cross-modal attention fusion. By constructing a hierarchical missing modality emotion recognition model, the relationship between different modal data is learned, and the relevant features are fully utilized to realize the effective recognition of multimodal emotions.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, the present invention discloses a method for missing modality emotion recognition based on cross-modal attention fusion, comprising: Obtain multimodal data of the emotion to be recognized; Marking a missing mode of the multimodal data with a missing mode indication and obtaining a primary feature through an input unit, and performing feature preprocessing on the multimodal data to obtain a preprocessed feature; Using the Transformer encoder to extract the primary features of each modality to obtain high-level features; The cross-modal attention fusion layer is used to calculate the attention score between the modalities for the high-level features of each modality. The high-level features are weighted and summed according to the attention score. At the same time, the fusion degree of each modality feature is dynamically adjusted through the gated fusion mechanism to obtain the preliminary fusion feature. The preliminary fusion features and the pre-processing features are fused through a residual connection module to obtain full-modal features; A deep classification network is used to perform emotion recognition on the omnimodal features to obtain emotion categories.

[0008] In a second aspect, the present invention discloses a missing modality emotion recognition system based on cross-modal attention fusion, comprising: A data acquisition module is configured to: acquire multimodal data of the emotion to be recognized; a primary feature extraction module configured to: mark a missing mode indication for the missing mode of the multimodal data and obtain primary features through an input unit, and simultaneously perform feature preprocessing on the multimodal data to obtain preprocessed features; A high-level feature extraction module is configured to: extract the primary features of each modality using a Transformer encoder to obtain high-level features; The attention weighting module is configured to calculate the attention score between the modalities using the cross-modal attention fusion layer for the high-level features of each modality, perform weighted summation of the high-level features based on the attention score, and dynamically adjust the fusion degree of each modality feature through the gated fusion mechanism to obtain the preliminary fused features; A feature fusion module is configured to: fuse the preliminary fusion features with the pre-processed features through a residual connection module to obtain full-modal features; The emotion recognition module is configured to: use a deep classification network to perform emotion recognition on the omnimodal features to obtain emotion categories.

[0009] In a third aspect, the present invention discloses an electronic device comprising a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, the steps of the above-mentioned missing modality emotion recognition method based on cross-modal attention fusion are completed.

[0010] In a fourth aspect, the present invention discloses a computer-readable storage medium for storing computer instructions, which, when executed by a processor, completes the steps of the above-mentioned missing modality emotion recognition method based on cross-modal attention fusion.

[0011] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes the use of a Transformer module to capture the long-distance dependencies within each modality through a multi-head self-attention mechanism, and realizes nonlinear transformation of features through a feedforward neural network, thereby generating feature representations of each modality. The triplet cross-modal attention is based on these features to calculate the attention scores between modalities, thereby realizing cross-modal interaction and fusion.

[0012] The present invention extracts triplet attention through the Transformer module to establish the basis for cross-modal association representation. Triplet attention realizes inter-modal interaction. At the same time, gated fusion guides the dynamic weighted fusion of cross-modal features. Dynamic weighted fusion can selectively fuse the generated features and original features, effectively utilize the available modal information, and reduce the information loss of the missing modality.

[0013] The present invention proposes a missing modality module that generates approximate features by averaging and pooling available modal features through a modality generation mechanism, and dynamically adjusts weights through gated fusion to compensate for missing information and maintain feature integrity.

[0014] The present invention improves training stability and avoids gradient vanishing through residual connections, thereby further ensuring the classification performance of gated fusion and MLP.

[0015] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0017] Figure 1Schematic diagram of the missing modality emotion recognition model described in Example 1 of the present invention.

[0018] Figure 2 These are the experimental results when the number of attention heads of the CAEHM model described in Example 1 of the present invention is 8 and the key dimension is 48.

[0019] Figure 3 These are the experimental results when the number of attention heads of the CAEHM model described in Example 1 of the present invention is 8 and the key dimension is 64.

[0020] Figure 4 These are the experimental results when the number of attention heads of the CAEHM model described in Example 1 of the present invention is 12 and the key dimension is 48.

[0021] Figure 5 These are the experimental results when the number of attention heads of the CAEHM model described in Example 1 of the present invention is 12 and the key dimension is 64.

[0022] Figure 6 These are the experimental results when the number of attention heads of the CAEHM model described in Example 1 of the present invention is 16 and the key dimension is 48.

[0023] Figure 7 These are the experimental results when the number of attention heads of the CAEHM model described in Example 1 of the present invention is 16 and the key dimension is 64. DETAILED DESCRIPTION

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0025] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0026] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0027] Explanation of terms: Cross-modality refers to different types of modal data, which can be text, images, audio, video, etc. For example, in multimedia content, videos contain visual modalities (images) and auditory modalities (sounds), while news information involves both textual modalities (words) and image modalities (pictures).

[0028] The attention mechanism is an artificial neural network structure that mimics the selective nature of human visual attention. It allows the model to automatically focus on the most relevant parts of the input data when processing information, while giving less attention to less important parts. In emotion recognition scenarios, when analyzing emotional data containing text and video, the attention mechanism can help the model identify certain words in the text, or certain images, shots, and sound clips in the video, which plays an important role in accurately determining emotional tendencies. Fusion refers to the integration of data or information from different modalities to produce a more comprehensive and accurate representation than that of a single modality. Fusion can be performed in a variety of ways, ranging from simple concatenation and weighted summation to more complex multi-layer feature fusion in deep learning. In cross-modal emotion recognition, the goal of fusion is to break down barriers between modalities and leverage the strengths of each modality, enabling the model to understand and perceive emotions from multiple dimensions and compensate for the lack of information or ambiguity inherent in a single modality.

[0029] In real-world data scenarios, data from certain modalities is often missing. When conducting sentiment analysis on a video-based social media platform, some videos may only have visuals but no sound, or vice versa. Alternatively, during multimodal data collection, data from certain modalities may be unavailable due to equipment failure, data transmission loss, or other reasons. Missing modalities pose a challenge to emotion recognition, as models that rely on multimodal fusion to improve accuracy may not function properly. Therefore, it is necessary to study how to accurately recognize emotions despite modalities being missing. Emotion recognition is a key task in the field of artificial intelligence. It aims to analyze various data (such as text, speech, images, and videos) to identify the underlying emotions. These include basic emotions like joy, sadness, anger, surprise, fear, and disgust, as well as more complex and nuanced mixed emotions. Emotion recognition technology has a wide range of applications, including intelligent customer service (understanding user emotions and responding appropriately), film and television production (analyzing audiences' emotional reactions to film plots), and mental health monitoring (analyzing patients' speech and behavior to assist in determining their mental state).

[0030] Example 1 In one or more embodiments, a method for missing modality emotion recognition based on cross-modal attention fusion is disclosed, and a missing modality emotion recognition model (Cross-modal Attention Enhanced Hierarchical Model, CAEHM) is constructed. The model adopts a hierarchical architecture design, such as Figure 1 As shown, including the following: Step S1: Acquire multimodal data of the emotion to be recognized; Multimodal data includes text, audio, and video data; the text, audio, and video data are input into the multimodal input interface and received through independent channels.

[0031] Step S2: Mark the missing mode of the multimodal data with a missing mode indication and obtain primary features through the input unit, and at the same time perform feature preprocessing on the multimodal data to obtain preprocessed features.

[0032] Three independent input channels are set up to receive text, audio, and video data respectively, supporting the parallel access of three modal data. It receives data in three modalities, text, audio, and video, and can handle the missing modality that may occur in these data.

[0033] In this embodiment, a missing mode indicator is added Marking missing data locations. Missing mode indicators. In addition to receiving the original modal data, missing mode indicators are also introduced during input. These indicators are used to mark which modal data are missing in each time step to assist in subsequent feature processing.

[0034] The parallel input and missing mode indication are input processed and can be expressed as: (1) Where, It is the input feature of text, audio and video modalities; LayerNrom is layer normalization; are the parameters of the fully connected layer.

[0035] Furthermore, it is passed to the feature preprocessing unit for padding, alignment, mask generation, layer normalization and Dropout regularization to obtain primary features to ensure the consistency of data dimension.

[0036] First, for missing modal data, zero filling is used to fill in the missing modal indicator. The marked missing data positions are filled with zero data, that is, zero vectors are filled in the missing positions so that the input data is consistent with the original data in dimension.

[0037] Then, the features of each modal data are aligned, and the data of different modalities are adjusted to the same time step through interpolation and other methods to ensure the consistency of the features of each modality in the time dimension.

[0038] In the feature preprocessing stage, the input features are masked to generate a mask corresponding to the input data, and the features are projected to a uniform dimension through a fully connected layer: (2) Where, Filling values ​​to mask input features; is a fully connected layer with a dimension of 512; LayerNrom is layer normalization.

[0039] Finally, layer normalization and Dropout are used to normalize the projected feature data, eliminating dimensional differences between features and accelerating model training. At the same time, Dropout regularization is applied to randomly discard some features, improving the model's generalization and robustness.

[0040] Feature preprocessing is used to eliminate the fact that data of different modalities have different time lengths and sampling rates.

[0041] Step S3: Using a Transformer encoder, extract the primary features of each modality to obtain high-level features.

[0042] The multi-head self-attention neural network utilizes the Transformer model, the core module of the CAEHM model. The Transformer comprises a nine-layer cascaded Transformer encoder, a cross-modal attention fusion layer, a feature fusion residual connection module, and a deep classification network. The primary features obtained after preprocessing are fed into modality-specific Transformer modules and processed through Transformer blocks. Each block comprises a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism captures long-range dependencies in the sequence, while the feedforward neural network implements nonlinear feature transformations. Features are collected every three layers for subsequent fusion.

[0043] Specifically, nine Transformer blocks are applied to each modality, each consisting of a multi-head self-attention mechanism and a feedforward neural network. The final output is the raw feature vector of the last layer of each modality, which is fed into the subsequent gated fusion module. Hierarchical features are output every three layers for subsequent missing modality generation and ternary cross-attention fusion. The multi-head self-attention mechanism is used to capture long-range dependencies in the sequence, while the feedforward neural network implements nonlinear feature transformation.

[0044] The formula for MultiHeadAttention is as follows: (3) (4) Where, They are query (Query), key (Key), and value (Value). By entering Linear transformation is obtained; Indicates the Learnable parameters of size; represents the output projection matrix, and , The dimension of the feature vector for each time step; head i For the The output of the attention head, MultHead(·) is the multi-head attention output, Concat is the splicing operation, which splices the keys and values ​​of the two key-value pair modes together to form a joint key and value matrix, and Attention(·) is the scaled dot-product attention function, which is implemented as scaled dot-product attention (Scaled Dot-Product Attention).

[0045] Among them, the calculation formula of scaled dot product attention is as follows: (5) Where Q is the query matrix, and its shape is ,in is the number of queries, is the key / query dimension of each attention head; K is the key matrix, with shape ,in is the number of keys; V is the value matrix, with shape ,in is the dimension of the value; is a scaling factor used to prevent the dot product result from being too large, which causes the gradient of the softmax function to disappear; softmax is an activation function that converts the attention score into a probability distribution, and T is the transpose.

[0046] Multi-head attention mechanism followed by residual connection and layer normalization: (6) (7) Where Dropout is the dropout rate, which is set to 0.4; is the original input of the residual connection, X att is the output feature after processing by the multi-head self-attention mechanism, X out1 is the output feature after multi-head self-attention mechanism, residual connection and layer normalization.

[0047] Feed-Forward Network (FFN): (8) (9) (10) Where, is the first layer weight; is the second layer weight; L2 is the regularization coefficient , Xffn is the feature after multi-head self-attention mechanism and residual connection, X final It is the output feature after being processed by the feedforward neural network. This feature will serve as the final output of the Transformer block and as the input of the next Transformer block.

[0048] Step S4: Use the cross-modal attention fusion layer to calculate the attention scores between the modalities for the high-level features of each modality, perform weighted summation on the high-level features according to the attention scores, and dynamically adjust the fusion degree of each modality feature through the gated fusion mechanism to obtain the preliminary fusion features.

[0049] High-level features from text, audio, and video are fed into the cross-modal attention fusion layer. This layer utilizes a triplet cross-modal attention mechanism to calculate the attention score of each modality relative to the other two modalities, and then takes a weighted sum of these scores to produce the fused features. Furthermore, a gated fusion mechanism dynamically adjusts the degree of fusion between the features of each modality to minimize information loss from missing modalities.

[0050] When dealing with missing modalities, the model first generates features for the missing modality from the available modalities using the Missing Modality Generator module. The generated features are dynamically fused with the original features through a gating mechanism, thus reducing the impact of the missing modality on performance.

[0051] When a modality is missing (marked by a missing indicator), the model generates features of the missing modality using features of other modalities. The output of the modality generation module is expressed as follows: (11) Where, Generate features for missing modes, and is a learnable parameter, Mean(AvailableModalities) represents the average pooling of available modal features, and Tanh is the activation function.

[0052] The gated fusion mechanism selectively fuses the missing modality generation features with the high-level features obtained through the Transformer module, effectively utilizing the available modal information and reducing the information loss of the missing modality. When the modality is complete, the high-level features directly enter the gated fusion; when the modality is missing, after passing through the modality generation module, the missing modality generation features and the high-level features enter the gated fusion together.

[0053] The specific formula of the gated fusion mechanism is as follows: (12) Where, is the high-level feature output by the feedforward network, 、 are the weight matrices used to process high-level features and generate features for missing modalities, respectively; 、 are the bias items corresponding to the weight matrix, and siogmiod is the activation function.

[0054] Subsequently, the data of the three modalities are fused through the triple cross-modal attention mechanism. The triple cross-modal attention is used to calculate the attention score between each modal data and the other two modal data, and the features are weighted and summed according to the attention score to obtain the fused features. The input of the triple cross-modal attention includes the query modality and two key-value modalities. When there is a missing modality, the key-value modality contains the generated features. The formula is as follows: (13) (14) (15) Where, is the triplet cross-modal attention, Q is the query modality, the modal data that currently needs to be paid attention to, Q1 and Q2 are and 、 and The stitching matrix; Represents the first key-value modality (Key-Value Modality 1), providing feature information related to the query modality; It represents the second key-value modality (Key-Value Modality 2), which provides another part of feature information related to the query modality. Concat is a splicing operation that concatenates the keys and values ​​of the two key-value modalities together to form a joint key and value matrix.

[0055] The initial fusion features are obtained by using triplet cross-modal attention for multimodal fusion (multi_modal_fusion): (16) Where, is the initial fusion feature, d att represents the cross-modal attention output dimension, and Same as input dimension, d Xis the feature dimension; [;] indicates concatenation along the feature dimension; X, Y, and Z represent the three modal features adjusted by the gated fusion mechanism; CrossAttn is the triplet cross-modal attention function, which is used to calculate the attention scores between different modalities and perform weighted summation.

[0056] The triplet cross-modal attention mechanism simultaneously calculates the cross-attention of text, audio, and video, capturing complex dependencies and resolving information fragmentation. The gated fusion mechanism dynamically adjusts the degree of integration of features from each modality, determining the importance of each modality based on specific circumstances, enabling more flexible and accurate feature fusion. The multimodal fusion layer integrates features from each modality into a unified feature representation, integrating the strengths of each modality and helping the model learn complex emotional patterns.

[0057] Step S5: The preliminary fusion features and the preprocessing features are fused through a residual connection module to obtain full-modal features.

[0058] The preprocessed features and the initial fusion features are spliced ​​together to retain more information. At the same time, residual connections are used to alleviate the gradient vanishing problem and improve the training efficiency of the model.

[0059] First, after residual splicing: (17) Where, is the feature after residual splicing, and ; Then, perform full modal splicing to obtain full modal features: (18) Among them, H is the full modal feature; X conbined 、Y conbined 、Z conbined The feature combinations are the residual concatenation of the preliminary fusion features and preprocessing features of the three modalities.

[0060] Step S6: Use a deep classification network to perform emotion recognition on the full modal features to obtain emotion categories.

[0061] The fused full-modal feature vector is input into the deep classification network, processed by a 5-layer multi-layer perceptron (MLP), and uses ReLU and softmax activation functions to output the probability distribution of emotion categories to complete multimodal emotion recognition.

[0062] Specifically, the deep classification network uses a five-layer multi-layer perceptron (MLP) for final sentiment classification. The MLP layer contains multiple fully connected layers and activation functions, which are used to further nonlinearly transform and classify the fused features. The ReLU activation function is used to implement high-order feature mapping, and the output layer uses a softmax activation function to output the probability distribution of each sentiment category for the sentiment classification task.

[0063] The deep classification network MLP multi-layer perceptron can be expressed as: (19) (20) (twenty one) (twenty two) (twenty three) Where W3, W4, W5, W6, and W7 are the weight matrices of each fully connected layer, and 、 、 、 、 , C is the number of classification categories; b3, b4, b5, b6, b7 are the corresponding bias vectors, h1, h2, h3, h4 are the outputs of the middle hidden layer, and P is the final probability distribution of the emotion category; all layers contain L2 regularization.

[0064] The 5-layer MLP can learn complex decision boundaries and adapt to the strong nonlinearity of multimodal sentiment features. The MLP's fully connected structure effectively handles high-dimensional inputs. L2 regularization is applied to each layer to prevent overfitting, and dropout is combined to improve generalization.

[0065] This embodiment uses the Transformer module to capture the long-distance dependencies within each modality through a multi-head self-attention mechanism, and implements nonlinear transformation of features through a feedforward neural network to generate feature representations of each modality. The triplet cross-modal attention calculates the attention scores between modalities based on these features, thereby achieving cross-modal interaction and fusion. The representation extracted by the Transformer is the basis for the triplet attention to establish cross-modal associations. The various modules work together to achieve accurate missing modality emotion recognition.

[0066] This example compares the proposed missing modality emotion recognition method based on cross-modal attention fusion with several existing methods: 1. Lower Bound (LB): This baseline serves as the lower bound for performance comparison. It uses the most basic implementation approach, training multiple independent models to cover all modal combinations, including audio, video, and text only, as well as various bimodal and trimodal combinations. During testing, the corresponding model is selected for prediction based on the available modal combinations. This approach completely ignores the possibility of interactions between different modalities and cannot handle modality-missing patterns not seen during training. As a result, its performance is poor when modality-missing patterns are present in the test data. This baseline is primarily used to highlight other methods.

[0067] 2. Modality Substitution (MS): This method is a simple and direct strategy for handling missing modal data. When a modal data set is missing, this method replaces the feature input of the missing modal with a preset default value (an all-zero vector). The model uses the complete modal data during training and directly accepts the replacement value as input during testing.

[0068] 3. Modality Dropout (MD): This method randomly drops data from certain modalities with a fixed probability, forcing the model to learn to extract valid information from the remaining modalities. After the training phase, the model can directly handle incomplete modal input during testing. While this method is effective in improving adaptability to missing modalities, it does not explicitly model the process of generating the missing information.

[0069] 4. Memory Fusion Network (MFN): This method integrates intra-view and cross-view interaction mechanisms. It uses an LSTM system to assign an independent LSTM function to each view to learn intra-view interactions. It also introduces the Delta-memory Attention Network (DMAN) mechanism to identify cross-view interactions by comparing the differences in LSTM memory states between adjacent time steps.

[0070] 5. MISA (Modality-Invariant and Specific Representations for Multimodal Sentiment Analysis): This method maps each modality into two independent subspaces. The first subspace narrows the modality gap by learning common features across modal representations; the second subspace captures the characteristic information of each modality.

[0071] 6. MPLMM (Multimodal Prompt Learning with Missing Modalities): This method designs three types of prompts: generative prompts, missing signal prompts, and missing type prompts to generate missing modal features.

[0072] Table 1 Comparison results between CAEHM model and other baselines

[0073] Table 1 shows experimental results on the CMU-MOSI dataset. Comparing the baseline methods MS and MD reveals that randomly dropping data modalities during model training improves the model's generalization ability, thereby reducing its sensitivity to missing modal data during testing. The proposed CAEHM model achieved an average accuracy of 73.01% and an F1 score of 70.08%, demonstrating its effectiveness in multimodal emotion recognition tasks. The experiment compared the effects of multiple modal combinations. Among single modalities, the video modality (v) had the best overall performance in terms of accuracy (Acc) and F1, followed by the text modality (t). Perhaps due to the limited emotional information of the single modality, the audio modality (a) and video modality (v) performed better than other methods overall. In the combination under the dual-modality missing scenario, the audio-video (a, v) and audio-text (a, t) effects were similar, with an accuracy of 61%, indicating that the gating mechanism significantly improved the speech-semantic association weight and solved the problem of traditional models' over-reliance on the visual modality. The video-text (v, t) combination results were significantly lower, indicating that the video modality was difficult to effectively integrate with text information. When video was missing, the asynchrony between audio and text semantics intensified, and it could not effectively adapt to video-text-dominated scenarios, highlighting the necessity of multimodal fusion.

[0074] Table 2 Ablation experiment results

[0075] As shown in Table 2, this embodiment conducts an ablation experiment, and verifies the synergistic effect achieved by the interaction between the modules by removing various different modules. If the Transformer module is removed, the modal feature representation capability will degenerate, resulting in the inability of triple attention to effectively model cross-modal associations due to insufficient input feature quality. If triple attention is missing, the interaction between modalities is destroyed, gated fusion degenerates into a simple weighted average due to the lack of cross-modal guidance signals, and the residual connection fails due to the reduction in feature diversity. If the residual connection is removed, the training stability decreases, and the vanishing gradient will weaken the classification performance of gated fusion and MLP. The missing modality module generates approximate features by averaging the available modal features through the modality generation mechanism, and dynamically adjusts the weights through gated fusion to compensate for missing information and maintain feature integrity; if it fails, it will lead to a reduction in feature diversity, and the residual connection cannot effectively alleviate the vanishing gradient, thereby weakening the classification performance of the MLP classifier.

[0076] Table 3 Results of different parameters when the missing rate is 0.1

[0077] Table 4 Results of different parameters when the missing rate is 0.2

[0078] Table 5 Results of different parameters when the missing rate is 0.3

[0079] Table 6 Results of different parameters when the missing rate is 0.4

[0080] Table 7 Results of different parameters when the missing rate is 0.5

[0081] Table 8 Results of different parameters when the missing rate is 0.6

[0082] Table 9 Results of different parameters when the missing rate is 0.7

[0083] Table 10 Results of different parameters when the missing rate is 0.8

[0084] Table 11 Results of different parameters when the missing rate is 0.9

[0085] As shown in Tables 3-11, the model has stable performance under low missing rates and also performs well under medium missing rates. Different parameter settings have a greater impact on the model. The optimal combination of missing rate, number of attention heads, and key dimension is (0.1, 8, 64), with Acc reaching 76.06 and F1 being 75.04. However, in actual scenarios, the case of modal missingness of 10% is relatively rare, and a more representative missing rate needs to be selected. Comparing Tables 5 and 6, it is found that when the missing rate increases from 0.3 to 0.4, the results drop significantly, and the missing rate of 0.3 is closer to the common modal missing situation. Therefore, this simulation selects a missing rate of 0.3 for the experiment and sets the optimal combination (16, 64). As the missing rate increases further, Acc and F1 drop significantly. When the missing rate is 0.7, Acc drops to 57.31 and F1 is 57.10. When the missing rate is as high as 0.9, Acc and F1 drop sharply to 37.90 and 27.48, respectively. This shows that when the missing rate is too high, it has a serious negative impact on model performance.

[0086] like Figure 2-Figure 7 As shown in FIG, the experimental results of this embodiment for various structural parameters are shown. Figure 1 、 Figure 2 This is the experimental result when the number of attention heads is 8. Figure 3 、 Figure 4 This is the experimental result when the number of attention heads is 12. Figure 5 、 Figure 6 This is the experimental result when the number of attention heads is 16; Figure 1 、 Figure 3 、 Figure 5 This is the experimental result when the key dimension is 48. Figure 2 、 Figure 4 、 Figure 6 These are the experimental results when the key dimension is 64.

[0087] Example 2 In one or more embodiments, a system for missing modality emotion recognition based on cross-modal attention fusion is disclosed, specifically including: A data acquisition module is configured to: acquire multimodal data of emotions to be recognized; a primary feature extraction module configured to: mark a missing mode indication for the missing mode of the multimodal data and obtain primary features through an input unit, and simultaneously perform feature preprocessing on the multimodal data to obtain preprocessed features; A high-level feature extraction module is configured to: extract the primary features of each modality using a Transformer encoder to obtain high-level features; The attention weighting module is configured to calculate the attention score between the modalities using the cross-modal attention fusion layer for the high-level features of each modality, perform weighted summation of the high-level features based on the attention score, and dynamically adjust the fusion degree of each modality feature through the gated fusion mechanism to obtain the preliminary fused features; A feature fusion module is configured to: fuse the preliminary fusion features and the pre-processed features through a residual connection module to obtain full-modal features; The emotion recognition module is configured to: use a deep classification network to perform emotion recognition on the omnimodal features to obtain emotion categories.

[0088] Example 3 This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above-mentioned missing modality emotion recognition method based on cross-modal attention fusion are completed.

[0089] Example 4 This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned missing modality emotion recognition method based on cross-modal attention fusion are completed.

[0090] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0093] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0094] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A missing modality emotion recognition method based on cross-modal attention fusion, characterized by: include: Obtain multimodal data of the emotion to be recognized; Marking a missing mode of the multimodal data with a missing mode indication and obtaining a primary feature through an input unit, and performing feature preprocessing on the multimodal data to obtain a preprocessed feature; Using the Transformer encoder to extract the primary features of each modality to obtain high-level features; The cross-modal attention fusion layer is used to calculate the attention score between the modalities for the high-level features of each modality. The high-level features are weighted and summed according to the attention score. At the same time, the fusion degree of each modality feature is dynamically adjusted through the gated fusion mechanism to obtain the preliminary fusion feature. The preliminary fusion features and the pre-processing features are fused through a residual connection module to obtain full-modal features; A deep classification network is used to perform emotion recognition on the omnimodal features to obtain emotion categories.

2. The method for missing modality emotion recognition based on cross-modal attention fusion according to claim 1, characterized in that: The feature preprocessing includes: For missing modal data, zero filling is used to fill the missing data position marked by the missing modal indicator; Perform feature alignment on each modal data and adjust the data of different modalities to the same time step; After the features are aligned, they are masked to generate a mask corresponding to the input data, and the features are projected to a unified dimension through a fully connected layer; Layer normalization and Dropout are used to perform layer normalization on the projected feature data to eliminate the dimensional differences between different features.

3. The method for missing modality emotion recognition based on cross-modal attention fusion according to claim 1, characterized in that: The Transformer encoder includes a multi-head self-attention mechanism, residual connections, layer normalization and a feedforward neural network; The multi-head self-attention mechanism is expressed as: Where, They are query, key, and value respectively; Indicates the Learnable parameters of size; represents the output projection matrix; Attention(·) is the scaled dot product attention function; head i For the The output of an attention head; MultHead(·) is the output of multiple attention heads; Concat is the concatenation operation.

4. The method for missing modality emotion recognition based on cross-modal attention fusion according to claim 1, wherein: The cross-modal attention fusion layer uses triple cross-modal attention to calculate the attention score between each modality data and the other two modal data, and performs weighted summation of features according to the attention score to obtain the preliminary fusion feature, which is expressed as: Where, X fused is the preliminary fusion feature, [;] represents splicing along the feature dimension; X, Y, and Z represent the three modal features adjusted by the gated fusion mechanism; CrossAttn is the triplet cross-modal attention function.

5. The method for missing modality emotion recognition based on cross-modal attention fusion according to claim 1, wherein: The gated fusion mechanism dynamically adjusts the fusion degree of each modality feature, specifically: Selectively fuse missing modality generative features and high-level features through a gated fusion mechanism: Where, is the high-level feature output by the feedforward network, Generate features for missing modes, 、 are the weight matrices used to process high-level features and generate features for missing modalities, respectively; 、 are the bias items corresponding to the weight matrix respectively; Tanh and siogmiod are both activation functions.

6. The method for missing modality emotion recognition based on cross-modal attention fusion according to claim 1, wherein: When dealing with missing modalities, the Missing Modality Generator module uses the available modalities to generate the features of the missing modality. When a modality is missing, the model generates the features of the missing modality through the features of other modalities. The output of the modality generator is expressed as: Where, and is a learnable parameter, Mean(Available Modalities) represents the average pooling of available modal features, and tanh is the activation function.

7. The method for missing modality emotion recognition based on cross-modal attention fusion according to claim 1, wherein: The deep classification network uses a 5-layer multilayer perceptron for final sentiment classification. The multilayer perceptron layer contains multiple fully connected layers and activation functions for further nonlinear transformation and classification of the fused features. The ReLU activation function is used to implement high-order feature mapping, and the output layer uses the softmax activation function to output the probability distribution of each sentiment category for the sentiment classification task, which is expressed as: Where W3, W4, W5, W6, and W7 are the weight matrices of each fully connected layer, b3, b4, b5, b6, and b7 are the corresponding bias vectors, h1, h2, h3, and h4 are the outputs of the middle hidden layer, P is the final probability distribution of the emotion category, and Dropout is the discard layer.

8. A missing modality emotion recognition system based on cross-modal attention fusion, characterized by: include: A data acquisition module is configured to: acquire multimodal data of emotions to be recognized; a primary feature extraction module configured to: mark a missing mode indication for the missing mode of the multimodal data and obtain primary features through an input unit, and simultaneously perform feature preprocessing on the multimodal data to obtain preprocessed features; A high-level feature extraction module is configured to: extract the primary features of each modality using a Transformer encoder to obtain high-level features; The attention weighting module is configured to calculate the attention score between the modalities using the cross-modal attention fusion layer for the high-level features of each modality, perform weighted summation of the high-level features based on the attention score, and dynamically adjust the fusion degree of each modality feature through the gated fusion mechanism to obtain the preliminary fused features; A feature fusion module is configured to: fuse the preliminary fusion features and the pre-processed features through a residual connection module to obtain full-modal features; The emotion recognition module is configured to: use a deep classification network to perform emotion recognition on the omnimodal features to obtain emotion categories.

9. An electronic device, characterized in that: The invention comprises a memory and a processor and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the missing modality emotion recognition method based on cross-modal attention fusion as described in any one of claims 1 to 7 is completed.

10. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, complete the missing modality emotion recognition method based on cross-modal attention fusion as described in any one of claims 1-7.

Images