Method and system for missing modality sentiment recognition based on cross-modal attention fusion

Abstract

This invention proposes a missing modality emotion recognition method and system based on cross-modal attention fusion, belonging to the field of emotion recognition technology. The method includes: acquiring multimodal data of the emotion to be recognized; marking missing modalities in the multimodal data with missing modality indicators and performing feature preprocessing to obtain primary features and preprocessed features; extracting high-level features from the primary features of each modality; calculating attention scores between modalities using a cross-modal attention fusion layer, weighting and summing the high-level features based on the attention scores, and dynamically adjusting the fusion degree of each modality feature through a gating fusion mechanism to obtain preliminary fused features; fusing the preliminary fused features and preprocessed features through a residual connection module to obtain full-modal features; and using a deep classification network to perform emotion recognition on the full-modal features. This invention learns the interrelationships between different modalities to achieve effective multimodal emotion recognition.

Description

Technical Field

[0001] This invention belongs to the field of emotion recognition technology, and in particular relates to a missing modality emotion recognition method and system based on cross-modal attention fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Emotion recognition, as an important branch of artificial intelligence, integrates cross-modal information to analyze emotional states, and has become a core research direction in this field. Compared with single-modal methods, multimodal fusion strategies have significant advantages in the completeness and accuracy of emotion representation, especially in scenarios such as intelligent service systems, human-computer collaborative interaction, and clinical psychological assessment, demonstrating irreplaceable technical value.

[0004] In recent years, the rapid development of the internet and multimedia technologies has generated a massive amount of multimodal data, and multimodal emotion recognition has received widespread attention. However, in practical applications, the problem of missing modal data is very common; that is, some modal data may be unavailable or of poor quality, which further increases the difficulty of emotion recognition. Current multimodal models are basically learned and trained based on complete data, but they often perform poorly when faced with incomplete data. Single-modal data often suffers from the limitation of incomplete information representation: the implicit nature of textual language makes it difficult to accurately convey emotions; although audio tone and speed can reflect emotional tendencies, the lack of visual aids can easily lead to misjudgment; intuitive video facial expressions and body movements lack semantic information, making it difficult to achieve comprehensive emotion understanding.

[0005] Early research employed traditional machine learning methods, such as Support Vector Machines (SVMs) and Hidden Markov Models (HMMs), to achieve sentiment classification through manually designed features. However, these methods suffer from insufficient feature association mining when dealing with unstructured multimodal data. Furthermore, fine-tuning traditional pre-trained models on small datasets leads to instability. Therefore, when any of the text, audio, or video modalities is missing, traditional methods suffer from a sharp decline in recognition performance due to their inability to effectively handle the missing information. In addition, existing cross-modal interaction mechanisms in deep learning methods are limited to bimodal attention, making it difficult to fully capture the complex collaborative relationships among text, audio, and video. Moreover, fixed-weight feature fusion methods cannot dynamically suppress noisy modal interference, resulting in low information fusion efficiency. Summary of the Invention

[0006] To overcome the shortcomings of the existing technologies, this invention proposes a missing modality emotion recognition method and system based on cross-modal attention fusion. By constructing a hierarchical missing modality emotion recognition model, the method learns the interrelationships between different modal data and makes full use of relevant features to achieve effective recognition of multimodal emotions.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] In a first aspect, this invention discloses a missing modality emotion recognition method based on cross-modal attention fusion, comprising:

[0009] Acquire multimodal data of the emotion to be identified;

[0010] The missing modalities of the multimodal data are marked with missing modal indicators and primary features are obtained through the input unit. At the same time, the multimodal data is preprocessed to obtain preprocessed features.

[0011] Advanced features are obtained by extracting features from the primary features of each modality using a Transformer encoder.

[0012] The attention scores between modalities are calculated using a cross-modal attention fusion layer on the high-level features of each modality. The high-level features are then weighted and summed based on the attention scores. At the same time, the degree of fusion of each modality feature is dynamically adjusted through a gating fusion mechanism to obtain preliminary fused features.

[0013] The preliminary fusion features and preprocessed features are fused together using a residual connection module to obtain full-modal features;

[0014] A deep classification network is used to perform emotion recognition on the full-modal features to obtain the emotion category.

[0015] Secondly, this invention discloses a missing modality emotion recognition system based on cross-modal attention fusion, comprising:

[0016] The data acquisition module is configured to acquire multimodal data of the emotion to be identified.

[0017] The primary feature extraction module is configured to: mark missing modalities in the multimodal data as missing modalities and obtain primary features through the input unit; and simultaneously perform feature preprocessing on the multimodal data to obtain preprocessed features.

[0018] The advanced feature extraction module is configured to extract advanced features from the primary features of each modality using a Transformer encoder.

[0019] The attention weighting module is configured to: calculate the attention score between modalities using the cross-modal attention fusion layer on the high-level features of each modality, perform weighted summation of the high-level features based on the attention score, and dynamically adjust the fusion degree of each modality feature through a gating fusion mechanism to obtain the preliminary fused features;

[0020] The feature fusion module is configured to fuse the preliminary fused features and the preprocessed features through a residual connection module to obtain full-modal features;

[0021] The emotion recognition module is configured to use a deep classification network to perform emotion recognition on the full-modal features to obtain the emotion category.

[0022] Thirdly, the present invention discloses an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when run by the processor, complete the steps of the above-mentioned missing modality emotion recognition method based on cross-modal attention fusion.

[0023] Fourthly, the present invention discloses a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the aforementioned missing modality emotion recognition method based on cross-modal attention fusion.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0025] This invention proposes to use a Transformer module to capture long-range dependencies within each modality through a multi-head self-attention mechanism, and to achieve nonlinear transformation of features through a feedforward neural network to generate feature representations for each modality. Triple cross-modal attention is based on these features to calculate attention scores between modalities, thereby achieving cross-modal interaction and fusion.

[0026] This invention establishes the basis for cross-modal association representation by extracting triple attention through the Transformer module. Triple attention enables inter-modal interaction. At the same time, gating fusion guides the dynamic weighted fusion of cross-modal features. Dynamic weighted fusion can selectively fuse generated features and original features, effectively utilize available modal information, and reduce information loss from missing modalities.

[0027] This invention proposes a missing modality module that uses the available modality features to generate approximate features through average pooling via a modality generation mechanism, and dynamically adjusts the weights through gating fusion to compensate for missing information and maintain feature integrity.

[0028] This invention improves training stability and avoids gradient vanishing by using residual connections, thereby further ensuring the classification performance of gated fusion and MLP.

[0029] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0030] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0031] Figure 1 This is a schematic diagram of the missing modality emotion recognition model described in Embodiment 1 of the present invention.

[0032] Figure 2 The results are experimental results for the CAEHM model described in Embodiment 1 of this invention when the number of attention heads is 8 and the key dimension is 48.

[0033] Figure 3 The results are experimental results for the CAEHM model described in Embodiment 1 of this invention when the number of attention heads is 8 and the key dimension is 64.

[0034] Figure 4 The results are experimental results for the CAEHM model described in Embodiment 1 of this invention when the number of attention heads is 12 and the key dimension is 48.

[0035] Figure 5 The results are experimental results for the CAEHM model described in Embodiment 1 of this invention when the number of attention heads is 12 and the key dimension is 64.

[0036] Figure 6 The results are experimental results for the CAEHM model described in Embodiment 1 of this invention when the number of attention heads is 16 and the key dimension is 48.

[0037] Figure 7 The results are experimental results for the CAEHM model described in Embodiment 1 of this invention when the number of attention heads is 16 and the key dimension is 64. Detailed Implementation

[0038] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0040] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0041] Terminology Explanation:

[0042] Cross-modal refers to data in different modalities, such as text, images, audio, and video. For example, in multimedia content, video includes visual modalities (images) and auditory modalities (sound), while news information involves both text modalities (text) and image modalities (images).

[0043] Attention mechanisms are artificial neural network structures that mimic the selective attention characteristics of human vision. They allow models to automatically focus on the most relevant parts of the input data when processing information, while assigning less attention weight to less important parts. In sentiment recognition scenarios, when analyzing sentiment data containing text and video, attention mechanisms can help models identify certain words in the text or certain scenes and sound clips in the video, playing a crucial role in accurately judging sentiment.

[0044] Fusion refers to integrating data or information from different modalities to produce a more comprehensive and accurate representation than a single modality. There are various fusion methods, such as simple concatenation, weighted summation, and more complex multi-layer feature fusion in deep learning. In cross-modal emotion recognition, the goal of fusion is to break down barriers between modalities, fully utilize the advantages of each modality, and enable the model to understand and perceive emotions from multiple dimensions, compensating for the lack of information or ambiguity inherent in a single modality.

[0045] In real-world data scenarios, data for certain modalities is often missing. When performing sentiment analysis on a video-centric social media platform, some videos may only have visuals without audio, or audio without visuals. Similarly, during multimodal data collection, equipment malfunctions or data loss may prevent the acquisition of data for certain modalities. Missing modalities pose a challenge to sentiment recognition because models that rely on multimodal fusion to improve accuracy may fail to function properly. Therefore, it is necessary to investigate how to accurately recognize sentiment even with missing modalities.

[0046] Emotion recognition is a crucial task in the field of artificial intelligence. It aims to determine the emotional states contained within various data sources (such as text, speech, images, and videos), including basic emotions like joy, sadness, anger, surprise, fear, and disgust, as well as more complex and nuanced emotional mixtures. Emotion recognition technology has wide applications in numerous fields, such as intelligent customer service (understanding user emotions and responding appropriately), film production (analyzing audiences' emotional reactions to film plots), and mental health monitoring (analyzing patients' speech and behavior to help determine their psychological state).

[0047] Example 1

[0048] In one or more embodiments, a missing modality sentiment recognition method based on cross-modal attention fusion is disclosed, and a missing modality sentiment recognition model (Cross-modal Attention Enhanced Hierarchical Model, CAEHM) is constructed. The model adopts a hierarchical architecture design, such as... Figure 1 As shown, it includes the following:

[0049] Step S1: Obtain multimodal data of the emotion to be identified;

[0050] Multimodal data includes text, audio, and video data; text, audio, and video data are input into the multimodal input interface and received through independent channels.

[0051] Step S2: Mark missing modalities in the multimodal data as missing modalities and obtain primary features through input units. At the same time, perform feature preprocessing on the multimodal data to obtain preprocessed features.

[0052] Three independent input channels are set up to receive text, audio, and video data respectively, supporting parallel input of the three modalities. It can receive text, audio, and video data and handle any missing modalities that may occur.

[0053] In this embodiment, a missing mode indicator is added. Missing data locations are marked. In addition to receiving the raw modal data, missing modal indicators are introduced during input. These indicators are used to mark which modal data are missing at each time step to aid subsequent feature processing.

[0054] Parallel input and missing modality indication for input processing can be represented as:

[0055] (1)

[0056] In the formula, Input features are provided for text, audio, and video modalities; LayerNrom is used for layer normalization. These are the parameters for the fully connected layer.

[0057] Furthermore, the data is passed to the feature preprocessing unit for padding, alignment, mask generation, layer normalization, and Dropout regularization to obtain primary features, ensuring data dimensionality consistency.

[0058] First, for missing modal data, zero-padding is used to fill the missing modal indicators. The marked missing data locations are filled with zeros, that is, zero vectors are filled at the missing locations to make the input data consistent with the original data in terms of dimension.

[0059] Then, feature alignment is performed on the data of each modality. By using interpolation and other methods, the data of different modalities are adjusted to the same time step to ensure the consistency of the features of each modality in the time dimension.

[0060] In the feature preprocessing stage, the input features are masked to generate a mask corresponding to the input data, and the features are projected onto a unified dimension through a fully connected layer.

[0061] (2)

[0062] In the formula, To mask the input features; It is a fully connected layer with dimension 512; LayerNrom is the layer normalization.

[0063] Finally, layer normalization and Dropout are used to normalize the projected feature data, eliminating the dimensional differences between different features and accelerating model training. Simultaneously, Dropout regularization is applied to randomly discard some features, improving the model's generalization ability and robustness.

[0064] Feature preprocessing eliminates the differences in time length and sampling rate between data of different modalities.

[0065] Step S3: Use the Transformer encoder to extract high-level features from the primary features of each modality.

[0066] The multi-head self-attention mechanism neural network employs the Transformer model, the core module of the CAEHM model, which includes a 9-layer cascaded Transformer encoder, a cross-modal attention fusion layer, a feature fusion residual connection module, and a deep classification network. The preprocessed primary features are fed into their respective modality-specific Transformer modules, where they undergo further processing. Each Transformer block contains a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism captures long-range dependencies in the sequence, while the feedforward neural network performs non-linear feature transformation. Features are collected every 3 layers for subsequent fusion.

[0067] Specifically, nine Transformer blocks are applied to each modality's data, each consisting of a multi-head self-attention mechanism and a feedforward neural network. The final output is the original feature vector of the last layer of each modality, used as input to the subsequent gating fusion module; and hierarchical features are output every three layers, used for subsequent missing modality generation and ternary cross-attention fusion. The multi-head self-attention mechanism is used to capture long-range dependencies in the sequence, and the feedforward neural network performs non-linear feature transformation.

[0068] The formula for multi-head attention is as follows:

[0069] (3)

[0070] (4)

[0071] In the formula, These are respectively Query, Key, and Value, accessed by inputting... Obtained by linear transformation; Indicates the first Learnable parameters for size; This represents the output projection matrix, and , The dimension of the feature vector for each time step; head i For the first The output of each attention head, MultHead(·) is the multi-head attention output, Concat is the concatenation operation, which concatenates the keys and values ​​of the two key-value pairs into a joint key-value matrix, and Attention(·) is the scaled dot-product attention function, which is implemented as Scaled Dot-Product Attention.

[0072] The formula for calculating the scaled dot product attention is as follows:

[0073] (5)

[0074] In the formula, Q is the query matrix with shape as follows: ,in It is the number of queries. K is the key / query dimension for each attention head; K is the key matrix with shape [missing information]. ,in V is the number of keys; V is the value matrix with shape [missing information]. ,in It is the dimension of the value; is a scaling factor used to prevent the gradient of the softmax function from vanishing due to an excessively large dot product result; softmax is the activation function that converts the attention score into a probability distribution; and T is the transpose.

[0075] After multi-head attention mechanism, residual connections and layer normalization are performed:

[0076] (6)

[0077] (7)

[0078] In the formula, Dropout is the drop rate, set to 0.4; X is the original input for the residual connection. att X represents the output features after processing by a multi-head self-attention mechanism. out1 The output features are obtained after multi-head self-attention mechanism, residual connection and layer normalization.

[0079] Feedforward Network (FFN):

[0080] (8)

[0081] (9)

[0082] (10)

[0083] In the formula, The first layer weights; The second layer weights; L2 is the regularization coefficient. X ffn For features after multi-head self-attention mechanism and residual connection, X final The output feature after processing by the feedforward neural network will be used as the final output of the Transformer block and as the input of the next Transformer block.

[0084] Step S4: Calculate the attention scores between modalities using the cross-modal attention fusion layer for the high-level features of each modality, perform weighted summation of the high-level features based on the attention scores, and dynamically adjust the fusion degree of each modality feature through a gating fusion mechanism to obtain preliminary fused features.

[0085] High-level features from text, audio, and video are input into a cross-modal attention fusion layer. This layer utilizes a triplet cross-modal attention mechanism to calculate the attention score of each modality with the other two modalities, and then sums these scores in a weighted manner to obtain the fused features. Simultaneously, a gating fusion mechanism dynamically adjusts the degree of fusion for each modality feature, reducing the loss of missing modal information.

[0086] When dealing with missing modalities, the model first uses the Missing Modality Generator to generate features for the missing modalities from the available modalities. The generated features are then dynamically fused with the original features through a gating mechanism, thereby reducing the impact of missing modalities on performance.

[0087] When a modality is missing (marked by a missing indicator), the model generates features for the missing modality using features from other modalities. The output of the modality generation module is expressed by the following formula:

[0088] (11)

[0089] In the formula, To generate features for missing modes, and These are learnable parameters. Mean(AvailableModalities) represents the average pooling of available modal features, and Tanh is the activation function.

[0090] By selectively fusing features generated from missing modalities and high-level features obtained through the Transformer module using a gating fusion mechanism, available modal information is effectively utilized while reducing information loss from missing modalities. When a modality is complete, high-level features directly enter the gating fusion; when a modality is missing, after passing through the modality generation module, the features generated from the missing modality and high-level features enter the gating fusion together.

[0091] The specific formula for the gating fusion mechanism is as follows:

[0092] (12)

[0093] In the formula, These are the high-level features output by the feedforward network. , These are the weight matrices used to process high-level features and generate features from missing modalities, respectively. , These are the bias terms corresponding to the weight matrix, and siogmiod is the activation function.

[0094] Subsequently, the data from the three modalities are fused using a triplet cross-modal attention mechanism. Triplet cross-modal attention calculates the attention score between each modal and the other two modalities. The features are then weighted and summed based on these attention scores to obtain the fused features. The input to triplet cross-modal attention includes the query modality and two key-value modalities. When a missing modality exists, the key-value modalities contain the generated features. The formula is as follows:

[0095] (13)

[0096] (14)

[0097] (15)

[0098] In the formula, This is a triple cross-modal attention mechanism, where Q represents the query modality and the modality data that needs to be focused on. Q1 and Q2 are respectively... and , and The splicing matrix; This indicates the first key-value modality (Key-Value Modality 1), providing feature information related to the query modality; This indicates the second key-value modality, which provides another part of the feature information related to the query modality; concat is a concatenation operation that concatenates the keys and values ​​of the two key-value modalities together to form a joint key-value matrix.

[0099] Preliminary fusion features are obtained by using triplet cross-modal attention for multimodal fusion:

[0100] (16)

[0101] In the formula, For preliminary fusion features, d att This represents the cross-modal attention output dimension, and Same as the input dimension, d X is the feature dimension; [;] indicates concatenation along the feature dimension; X, Y, and Z represent the three modal features adjusted by the gating fusion mechanism, respectively; CrossAttn is the triple cross-modal attention function, used to calculate the attention scores between different modalities and perform a weighted summation.

[0102] The triplet cross-modal attention mechanism simultaneously computes the cross-attention of text, audio, and video, capturing complex dependencies and solving the problem of information fragmentation. The gating fusion mechanism dynamically adjusts the fusion degree of features from each modality, determining the importance of each feature based on specific circumstances, achieving more flexible and accurate feature fusion. The multimodal fusion layer integrates features from each modality into a unified feature representation, combining the advantages of each modality to help the model learn complex emotional patterns.

[0103] Step S5: The preliminary fusion features and preprocessed features are fused together through the residual connection module to obtain full-modal features.

[0104] Preprocessed features and initially fused features are concatenated to retain more information. Simultaneously, residual connections are used to mitigate the vanishing gradient problem and improve the model's training efficiency.

[0105] First, after residual splicing:

[0106] (17)

[0107] In the formula, These are the features after residual stitching, and ;

[0108] Then, full-modal concatenation is performed to obtain full-modal features:

[0109] (18)

[0110] Where H represents the full-modal feature; X conbined Y conbined Z conbined These are feature combinations obtained by residual splicing of the preliminary fusion features and preprocessed features of the three modalities, respectively.

[0111] Step S6: Use a deep classification network to perform emotion recognition on the full-modal features to obtain the emotion category.

[0112] The fused multimodal feature vectors are input into a deep classification network, processed by a 5-layer multilayer perceptron (MLP), and activated by ReLU and softmax functions to output the probability distribution of sentiment categories, thus completing multimodal sentiment recognition.

[0113] Specifically, the deep classification network uses a 5-layer Multilayer Perceptron (MLP) for final sentiment classification. The MLP layers contain multiple fully connected layers and activation functions to perform further non-linear transformations and classification on the fused features. Higher-order feature mapping is achieved using the ReLU activation function, and the output layer uses the softmax activation function to output the probability distribution for each sentiment category, used for the sentiment classification task.

[0114] The deep classification network MLP (Multilayer Perceptron) can be expressed as:

[0115] (19)

[0116] (20)

[0117] (twenty one)

[0118] (twenty two)

[0119] (twenty three)

[0120] In the formula, W3, W4, W5, W6, and W7 are the weight matrices of each fully connected layer, and , , , , C represents the number of categories; b3, b4, b5, b6, and b7 are the corresponding bias vectors; h1, h2, h3, and h4 are the outputs of the intermediate hidden layers; and P is the final sentiment category probability distribution. All layers contain L2 regularization.

[0121] A 5-layer MLP can learn complex decision boundaries and adapt to the strong nonlinearity of multimodal sentiment features. The fully connected structure of the MLP can effectively handle high-dimensional inputs. L2 regularization is applied to each layer to prevent overfitting, and Dropout is combined to improve generalization.

[0122] This embodiment employs a Transformer module to capture long-range dependencies within each modality through a multi-head self-attention mechanism. A feedforward neural network is then used to perform nonlinear transformations of features, generating feature representations for each modality. Triple cross-modal attention calculates attention scores between modalities based on these features, thereby achieving cross-modal interaction and fusion. The representations extracted by the Transformer are the foundation for establishing cross-modal associations using triple attention. The joint efforts of various modules achieve accurate missing modality sentiment recognition.

[0123] This embodiment compares the proposed missing modality emotion recognition method based on cross-modal attention fusion with several existing methods:

[0124] 1. Lower Bound (LB): This baseline serves as a lower limit for performance comparison. It employs a basic implementation, training multiple independent models to cover all modal combinations, including audio, video, and text-only scenarios, various bimodal combinations, and trimodal combinations. During testing, the appropriate model is selected for prediction based on the available modal combinations. This approach completely ignores the possibility of interactions between different modalities and cannot handle modality missing patterns not seen during training. Therefore, its performance is poor when modality missing patterns appear in the test data. This baseline is primarily used to highlight other methods.

[0125] 2. Modality Substitution (MS): This method is a simple and direct strategy for handling missing modalities. When a modality's data is missing, this method uses a preset default value (an all-zero vector) to replace the missing modality's feature input. The model uses complete modality data during training and directly accepts the substitute values ​​as input during testing.

[0126] 3. Modality Dropout (MD): This method randomly discards data from certain modalities with a fixed probability, forcing the model to learn to extract useful information from the remaining modalities. After the training phase, the model can directly process incomplete modal inputs during the testing phase. Although this method is effective in improving adaptability to missing modalities, it does not explicitly model the process of generating missing information.

[0127] 4. MFN (Memory Fusion Network): This method integrates intra-view interaction and cross-view interaction mechanisms. It uses an LSTM system to assign an independent LSTM function to each view to learn intra-view interactions. It introduces the DMAN (Delta-memory Attention Network) mechanism to identify cross-view interactions by comparing the differences in LSTM memory states at adjacent time steps.

[0128] 5. MISA (Modality-Invariant and Specific Representations for Multimodal Sentiment Analysis): This method maps each modality to two independent subspaces. The first subspace learns common features through cross-modal representations to narrow the modality gap; the second subspace captures the feature information of each modality.

[0129] 6. MPLMM (Multimodal Prompt Learning with Missing Modalities): This method designs three types of prompts: generative prompts, missing signal prompts, and missing type prompts to generate missing modal features.

[0130] Table 1. Comparison results of CAEHM model with other baselines

[0131]

[0132] Table 1 shows the experimental results on the CMU-MOSI dataset. By comparing with the baseline methods MS and MD, it was found that randomly discarding data modalities during the model training phase improves the model's generalization ability, thereby reducing its sensitivity to missing modal data during the testing phase. The CAEHM model proposed in this embodiment achieves an average accuracy of 73.01% and an F1 score of 70.08%, demonstrating its effectiveness in multimodal emotion recognition tasks. Experiments comparing the effects of various modal combinations revealed that in single-modal scenarios, the video modality (v) showed the best overall performance in terms of accuracy (Acc) and F1 score, followed by the text modality (t). This may be due to the limited emotional information in single-modal scenarios. The audio modality (a) and video modality (v) generally outperformed other methods. In the combination of modal missing scenarios, audio-video (a,v) and audio-text (a,t) performed similarly, both achieving an accuracy of 61%. This indicates that the gating mechanism significantly improves the speech-semantic association weights, addressing the problem of traditional models' over-reliance on the visual modality. The video-text (v,t) combination showed significantly lower results, indicating that the video modality is difficult to effectively integrate with text information. When video is missing, the asynchronicity between audio and text semantics intensifies, making it unable to effectively adapt to video-text-dominated scenarios, highlighting the necessity of multimodal fusion.

[0133] Table 2 Ablation Experiment Results

[0134]

[0135] As shown in Table 2, this embodiment conducted ablation experiments, verifying the synergistic effect achieved by the interaction between different modules by removing various modules. Removing the Transformer module degrades the modal feature representation capability, causing the triplet attention to fail to effectively model cross-modal associations due to insufficient input feature quality. Without the triplet attention, intermodal interactions are disrupted, and gated fusion degenerates into a simple weighted average due to the lack of cross-modal guidance signals; residual connections also fail due to reduced feature diversity. Removing residual connections decreases training stability, and gradient vanishing weakens the classification performance of gated fusion and MLP. The missing modal module generates approximate features through modality generation mechanisms using average pooling of available modal features, and dynamically adjusts weights through gated fusion to compensate for missing information and maintain feature integrity. If it fails, feature diversity decreases, residual connections cannot effectively alleviate gradient vanishing, and thus weaken the classification performance of the MLP classifier.

[0136] Table 3 Results of different parameters when the missing rate is 0.1

[0137]

[0138] Table 4 Results of different parameters when the missing rate is 0.2

[0139]

[0140] Table 5 Results of different parameters when the missing rate is 0.3

[0141]

[0142] Table 6 Results of different parameters when the missing data rate is 0.4

[0143]

[0144] Table 7 Results of different parameters when the missing data rate is 0.5

[0145]

[0146] Table 8 Results of different parameters when the missing rate is 0.6

[0147]

[0148] Table 9 Results of different parameters when the missing rate is 0.7

[0149]

[0150] Table 10 Results of different parameters when the missing rate is 0.8

[0151]

[0152] Table 11 Results of different parameters when the missing data rate is 0.9

[0153]

[0154] As shown in Tables 3-11, the model performs stably under low missing rates and also shows good performance under moderate missing rates. Different parameter settings have a significant impact on the model. The optimal combination of missing rate, number of attention heads, and key dimension is (0.1, 8, 64), achieving an Acc of 76.06 and an F1 score of 75.04. However, in real-world scenarios, modal missing rates of 10% are relatively rare, requiring the selection of a more representative missing rate. Comparing Tables 5 and 6, it is found that the results decrease significantly when the missing rate increases from 0.3 to 0.4, and the missing rate of 0.3 is closer to common modal missing situations. Therefore, this simulation uses a missing rate of 0.3 for experiments and sets the optimal combination (16, 64). As the missing rate further increases, Acc and F1 decrease significantly. When the missing rate is 0.7, Acc drops to 57.31 and F1 is 57.10. When the missing rate is as high as 0.9, Acc and F1 drop even more sharply to 37.90 and 27.48, respectively. This shows that when the missing rate is too high, it has a serious negative impact on the model performance.

[0155] like Figures 2-7The image shows the experimental results for various structural parameters in this embodiment. Figure 1 , Figure 2 The results are from an experiment with 8 attention heads. Figure 3 , Figure 4 The results are from an experiment with 12 attention heads. Figure 5 , Figure 6 The results are from an experiment with 16 attention heads. Figure 1 , Figure 3 , Figure 5 These are the experimental results when the key dimension is 48. Figure 2 , Figure 4 , Figure 6 The results are for an experimental setup with a key dimension of 64.

[0156] Example 2

[0157] In one or more embodiments, a missing modality emotion recognition system based on cross-modal attention fusion is disclosed, specifically including:

[0158] The data acquisition module is configured to acquire multimodal data of the emotion to be identified.

[0159] The primary feature extraction module is configured to: mark missing modalities in the multimodal data as missing modalities and obtain primary features through the input unit; and simultaneously perform feature preprocessing on the multimodal data to obtain preprocessed features.

[0160] The advanced feature extraction module is configured to extract advanced features from the primary features of each modality using a Transformer encoder.

[0161] The attention weighting module is configured to: calculate the attention score between modalities using the cross-modal attention fusion layer on the high-level features of each modality, perform weighted summation of the high-level features based on the attention score, and dynamically adjust the fusion degree of each modality feature through a gating fusion mechanism to obtain the preliminary fused features;

[0162] The feature fusion module is configured to fuse the preliminary fused features and the preprocessed features through a residual connection module to obtain full-modal features;

[0163] The emotion recognition module is configured to use a deep classification network to perform emotion recognition on the full-modal features to obtain the emotion category.

[0164] Example 3

[0165] This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps of the missing modality emotion recognition method based on cross-modal attention fusion described above.

[0166] Example 4

[0167] This embodiment provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the aforementioned missing modality emotion recognition method based on cross-modal attention fusion.

[0168] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0169] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0170] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0171] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0172] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A missing modality sentiment recognition method based on cross-modal attention fusion, characterized in that, include: Acquire multimodal data of the emotion to be identified; The missing modalities of the multimodal data are marked with missing modal indicators and primary features are obtained through the input unit. At the same time, the multimodal data is preprocessed to obtain preprocessed features. Advanced features are obtained by extracting features from the primary features of each modality using a Transformer encoder. The attention scores between modalities are calculated using a cross-modal attention fusion layer on the high-level features of each modality. The high-level features are then weighted and summed based on the attention scores. At the same time, the degree of fusion of each modality feature is dynamically adjusted through a gating fusion mechanism to obtain preliminary fused features. The preliminary fusion features and preprocessed features are fused together using a residual connection module to obtain full-modal features; A deep classification network is used to perform emotion recognition on the full-modal features to obtain the emotion category.

2. The missing modality emotion recognition method based on cross-modal attention fusion as described in claim 1, characterized in that, The feature preprocessing includes: For missing modal data, zero-padding is used to fill the missing data positions marked by the missing modal indicator. Perform feature alignment on the data of each modality to adjust the data of different modalities to the same time step; After feature alignment, a masking process is performed to generate a mask corresponding to the input data, and the features are projected onto a unified dimension through a fully connected layer; Layer normalization and Dropout are used to perform layer normalization on the projected feature data to eliminate the dimensional differences between different features.

3. The missing modality emotion recognition method based on cross-modal attention fusion as described in claim 1, characterized in that, The Transformer encoder includes a multi-head self-attention mechanism, residual connections, layer normalization, and a feedforward neural network. The multi-head self-attention mechanism is expressed as follows: In the formula, These are respectively query, key, and value; Indicates the first Learnable parameters for size; Represents the output projection matrix; Attention(·) is the scaling dot product attention function; head i For the first The output of a single attention head; MultHead(·) is the output of a multi-head attention head; Concat is the concatenation operation.

4. The missing modality emotion recognition method based on cross-modal attention fusion as described in claim 1, characterized in that, The cross-modal attention fusion layer uses triplet cross-modal attention to calculate the attention score between each modal data and the other two modal data. The features are then weighted and summed based on these attention scores to obtain preliminary fused features, expressed as: In the formula, X fused For the initial feature fusion, [;] indicates concatenation along the feature dimension; X, Y, and Z represent the three modal features adjusted by the gating fusion mechanism, respectively; CrossAttn is the triple cross-modal attention function.

5. The missing modality emotion recognition method based on cross-modal attention fusion as described in claim 1, characterized in that, The method of dynamically adjusting the fusion degree of each modal feature through a gating fusion mechanism is as follows: The missing modality generation features and high-level features are selectively fused through a gated fusion mechanism: In the formula, These are the high-level features output by the feedforward network. Features are generated for missing modes. , These are the weight matrices used to process high-level features and generate features from missing modalities, respectively. , These are the bias terms corresponding to the weight matrix; Tanh and siogmiod are both activation functions.

6. The missing modality emotion recognition method based on cross-modal attention fusion as described in claim 1, characterized in that, When handling missing modalities, the Missing Modality Generator (MMG) module utilizes available modalities to generate features for the missing modalities. When a modality is missing, the model generates features for the missing modality using features from other modalities. The output of the MMG module is represented as follows: In the formula, and These are learnable parameters. Mean (Available Modalities) represents the average pooling of available modal features, and tanh is the activation function.

7. The missing modality emotion recognition method based on cross-modal attention fusion as described in claim 1, characterized in that, The deep classification network uses a 5-layer multilayer perceptron for final sentiment classification. Each multilayer perceptron layer contains multiple fully connected layers and activation functions to perform further nonlinear transformations and classification on the fused features. The ReLU activation function is used to implement high-order feature mapping, and the output layer uses the softmax activation function to output the probability distribution for each sentiment category. This distribution is used for the sentiment classification task and is expressed as follows: In the formula, W3, W4, W5, W6, and W7 are the weight matrices of each fully connected layer, b3, b4, b5, b6, and b7 are the corresponding bias vectors, h1, h2, h3, and h4 are the outputs of the intermediate hidden layers, P is the final sentiment category probability distribution, and Dropout is the dropout layer.

8. A missing modality emotion recognition system based on cross-modal attention fusion, characterized in that, include: The data acquisition module is configured to acquire multimodal data of the emotion to be identified. The primary feature extraction module is configured to: mark missing modalities in the multimodal data as missing modalities and obtain primary features through the input unit; and simultaneously perform feature preprocessing on the multimodal data to obtain preprocessed features. The advanced feature extraction module is configured to extract advanced features from the primary features of each modality using a Transformer encoder. The attention weighting module is configured to: calculate the attention score between modalities using the cross-modal attention fusion layer on the high-level features of each modality, perform weighted summation of the high-level features based on the attention score, and dynamically adjust the fusion degree of each modality feature through a gating fusion mechanism to obtain the preliminary fused features; The feature fusion module is configured to fuse the preliminary fused features and the preprocessed features through a residual connection module to obtain full-modal features; The emotion recognition module is configured to use a deep classification network to perform emotion recognition on the full-modal features to obtain the emotion category.

9. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the missing modality emotion recognition method based on cross-modal attention fusion as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the missing modality emotion recognition method based on cross-modal attention fusion as described in any one of claims 1-7.

Images