Multi-modal information fusion emotion recognition method and system based on uncertainty perception

Abstract

The invention discloses a multi-modal information fusion emotion recognition method and system based on uncertainty perception. The method comprises the following steps of: modeling features of each modal to generate Gaussian distribution representation; establishing a cross-modal distribution attention mechanism; semantic association is carried out on the Gaussian distribution representation of each mode and the Gaussian distribution representations of the other modes in sequence, and cross-mode association distribution is obtained. And performing weighted fusion on a plurality of cross-modal association distributions obtained by semantic association to generate fusion distributions. And sampling the fusion distribution to obtain feature embedding. According to the method, the uncertainty of each mode is dynamically quantified through an uncertainty distribution modeling module, and semantic association between the modes is captured by using a cross-mode distribution attention mechanism. Finally, the correlated modal distribution is integrated through a cross-modal correlation distribution fusion module, and the fused cross-modal correlation distribution has smaller variance and more robust performance, which is an important breakthrough in the prior art.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal sentiment analysis, and particularly relates to a sentiment recognition method and system based on uncertainty-aware multimodal information fusion. Background Art

[0002] Sentiment recognition aims to identify and classify emotional states from human behaviors or languages. Early sentiment recognition methods mainly relied on a single modality, such as using only facial expressions, speech features, or text content to recognize emotions. Although these methods have achieved success in specific situations, due to the complexity and diversity of emotional expressions, it is often difficult for a single modality to comprehensively capture an individual's emotional state. With the in-depth research, researchers have begun to explore multimodal fusion technologies, which comprehensively utilize information from different sensory channels (such as videos, audios, texts, etc.) to improve the accuracy and robustness of sentiment recognition. Multimodal fusion technologies can understand and infer emotional information from multiple perspectives and dimensions, thereby achieving more accurate emotional classification.

[0003] The core of multimodal fusion lies in how to efficiently integrate data from different modalities to comprehensively and accurately capture emotional information. Common fusion strategies include feature-level fusion, decision-level fusion, and model-level fusion. Among them, model-level fusion based on deep learning has become the mainstream of multimodal sentiment recognition. However, existing methods mainly focus on the semantic associations between modalities and often ignore the uncertainty of the modality data itself. Although these methods can effectively model cross-modal semantic information, in the case of noisy or missing data, they lack the quantification of modality reliability and adaptive processing, resulting in a decline in the accuracy and robustness of sentiment recognition. Some studies have tried to adjust the modality fusion weights through uncertainty quantification, but usually fail to fully consider the semantic relationships between modalities. In sentiment analysis tasks, cross-modal semantic associations are crucial. Ignoring this may lead to limited information fusion effects and thus affect the overall recognition performance.

[0004] To address the above challenges, the present invention proposes a sentiment recognition method based on uncertainty-aware multimodal information fusion. First, the features of each modality are extracted, and the reliability of each modality is quantified by modeling its distribution; and an uncertainty-aware cross-modal distribution attention mechanism is designed to simultaneously capture the semantic associations and uncertainty information between modalities. Finally, feature embeddings are generated through Bayesian weighted fusion and reparameterized sampling for sentiment recognition, thereby effectively improving the robustness and accuracy of sentiment recognition. Summary of the Invention

[0005] The object of the present invention is to propose an emotion recognition method based on uncertainty-aware multimodal information fusion in view of the neglect and deficiencies of the prior art, so that in the process of multimodal data fusion, not only the semantic associations between different modalities are considered, but also the uncertainty of the data is fully taken into account, thereby effectively improving the robustness and accuracy of emotion recognition. First, text, visual, and audio modality data are acquired, initial features are extracted, and Gaussian distribution representations containing the mean and variance are generated by modeling the unimodal feature distributions to quantify the uncertainty of the data. Subsequently, an uncertainty-aware cross-modal distribution attention mechanism is designed to optimize the modality interaction process by capturing the semantic associations between modalities and utilizing the uncertainty information of the distribution representations while capturing the semantic associations between modalities. Finally, a Bayesian weighted fusion is adopted based on the modality uncertainty to generate a fusion distribution, and feature embeddings are obtained through reparameterized sampling for emotion classification.

[0006] In a first aspect, the present invention provides an emotion recognition method based on uncertainty modeling for multimodal information fusion, and the process is as follows:

[0007] Acquire multimodal data and extract features.

[0008] Model the features of each modality respectively to generate Gaussian distribution representations.

[0009] Establish a cross-modal distribution attention mechanism; for the Gaussian distribution representation of each modality, semantic associations are successively made with the Gaussian distribution representations of the other modalities to obtain cross-modal association distributions. The semantic associations finely model the fine-grained associations between modalities from both global and local levels, not only capturing semantic relationships, but also fully considering modality uncertainty, and optimizing the modality interaction process through the confidence information of the distribution representations.

[0010] Perform weighted fusion on the multiple cross-modal association distributions obtained from the semantic associations to generate a fusion distribution.

[0011] Sample the fusion distribution to obtain feature embeddings.

[0012] Input the feature embeddings into an emotion recognition model to obtain emotion recognition results.

[0013] Preferably, the process of modeling the features of the target modality to generate a Gaussian distribution representation is as follows:

[0014] Initialize the query matrix Q, key matrix K, and value matrix V in the self-attention mechanism to the features of the target modality.

[0015] Use the self-attention mechanism to model the features of the target modality to obtain the embedding of the target modality.

[0016] Perform distribution modeling on the embedding of the target modality to generate an initial Gaussian distribution.

[0017] Preferably, the process of semantically associating two distributions is as follows:

[0018] The two distributions are the original distribution and the associated distribution respectively; a cross-modal distribution attention mechanism is established, and the query distribution is initialized with the original distribution; the key distribution and the value distribution are initialized with the associated distribution.

[0019] Calculate the similarity between the query distribution and the key distribution as the distribution attention.

[0020] Sample the query distribution and the key distribution multiple times respectively to obtain multiple feature embeddings; calculate the dot product attention between the feature embeddings corresponding to the query distribution and the key distribution;

[0021] Perform weighted fusion on the distribution attention and the dot product attention to obtain the comprehensive attention.

[0022] Multiply the normalized value of the comprehensive attention by the mean and variance of the associated distribution to obtain the mean and variance of the distribution obtained after semantic association.

[0023] Preferably, in the weighted fusion of the distribution attention and the dot product attention, the weight parameter of the dot product attention is dynamically adjusted according to the uncertainty of the distribution. The higher the uncertainty, the larger the value of the weight parameter of the dot product attention.

[0024] Preferably, the process of weighted fusion of multiple cross-modal associated distributions is as follows: successively perform the fusion of two cross-modal associated distributions until the fusion of all cross-modal associated distributions is completed.

[0025] The process of fusing two cross-modal associated distributions is as follows: perform weighted average calculation on the means of the cross-modal associated distributions to obtain the mean of the distribution obtained by fusing the two cross-modal associated distributions; reconcile the variances of the cross-modal associated distributions to obtain the variance of the distribution obtained by fusing the two cross-modal associated distributions.

[0026] Preferably, the overall loss function of the emotion recognition model includes a reconstruction loss function for generating Gaussian distribution representations by feature modeling of each modality, a KL divergence loss function for cross-modal associated distributions generated by semantic association, and a multi-class cross-entropy loss function for emotion recognition.

[0027] Preferably, feature embeddings are obtained by reparameterized sampling of the fused distribution.

[0028] In a second aspect, the present invention provides an emotion recognition system for performing the foregoing emotion recognition method; the emotion recognition system includes a data acquisition module, a feature extraction module, a modal distribution modeling module, a cross-modal distribution attention module, a Bayesian fusion module, a reparameterized sampling module, and an identification module.

[0029] The data acquisition module is used to acquire multi-modal data including text, audio, and visual modalities.

[0030] The feature extraction module is used to preprocess the multi-modal data and extract text features, audio features, and visual features.

[0031] The modal distribution modeling module is used to model the uncertainty of multi-modal features and obtain the Gaussian distribution representation corresponding to each modality.

[0032] The cross-modal distribution attention module is used to capture the semantic associations between the distributions of each modality and generate corresponding cross-modal association distributions for the distribution of each modality.

[0033] The Bayesian fusion module is used to fuse multiple cross-modal association distributions.

[0034] The reparameterized sampling module is used to sample the fused distribution to obtain feature embeddings.

[0035] The recognition module is used to perform sentiment recognition on the feature embeddings.

[0036] In a third aspect, the present invention provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. The memory stores the computer program; the processor executes the foregoing sentiment recognition method.

[0037] In a fourth aspect, the present invention provides a readable storage medium, which stores a computer program; when the computer program is executed by a processor, it is used to implement the foregoing sentiment recognition method.

[0038] The beneficial effects of the present invention are as follows:

[0039] 1. The present invention provides a multi-modal information fusion strategy based on uncertainty perception. This strategy dynamically quantifies the uncertainty of each modality through an uncertainty distribution modeling module and uses a cross-modal distribution attention mechanism to capture the semantic associations between modalities. Finally, the associated modal distributions are integrated through a cross-modal association distribution fusion module. The fused cross-modal association distribution has a smaller variance and more robust performance, which is an important breakthrough in the prior art.

[0040] 2. The present invention designs an innovative cross-modal distribution attention mechanism, which can capture the semantic associations between modalities and optimize the modal interaction process by combining uncertainty information, significantly improving the information fusion effect in multi-modal sentiment analysis.

[0041] 3. The present invention is significantly superior to the baseline model in the sentiment recognition task, especially in dealing with categories with fewer samples, and shows stronger robustness and stability.. Description of the Drawings

[0042] Figure 1 This is the overall flowchart of Embodiment 1 of the present invention.

[0043] Figure 2 This is the model framework diagram of Embodiment 1 of the present invention.

[0044] Figure 3 This is the model block diagram for performing distribution modeling on a single modality in step S2 of Embodiment 1 of the present invention.

[0045] Figure 4 This is the model block diagram for the modality distribution interaction process in step S3 of Embodiment 1 of the present invention. Detailed implementation manners

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0047] The terms "include" and "have" and any variations thereof mentioned in the embodiments of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes other steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0048] Embodiment 1

[0049] A sentiment recognition method for multi-modal information fusion based on uncertainty perception, the overall process is as follows:

[0050] First, perform uncertainty modeling on the feature of multiple modal data through a distribution modeling module, represent it as a Gaussian distribution, so as to obtain distribution features including mean and variance, and use them to quantify the main representation and uncertainty information of the modality.

[0051] Secondly, design a cross-modal distribution attention mechanism. While capturing the semantic association between modalities, use the uncertainty information of the distribution representation to optimize the modality interaction process. This mechanism uses the optimal transport distance (i.e., Wasserstein distance) to evaluate the global distribution similarity between modalities, and combines feature sampling to capture local semantic associations, dynamically adjusts the attention weights, and suppresses the interference of high-uncertainty modalities.

[0052] Then, based on the brain-inspired mechanism of weighted integration of multi-sensory information, the Bayesian weighted fusion method is used to fuse the distributions of each modality, generate the final fused distribution features, and convert the fused distribution into an emotional feature representation through reparameterized sampling, thereby improving the robustness and accuracy of emotion recognition.

[0053] As Figure 1 and Figure 2 shown, the specific steps of the emotion recognition method are as follows:

[0054] S1: Obtain multi-modal data of the same object, including text, visual, and audio modalities, and extract the initial feature representations of each modality to obtain the text feature h t , visual feature h v , audio feature h a ; Each sample of the multi-modal data is labeled with an emotion type.

[0055] S2: Model the features of each modality through a distribution modeling module, and generate Gaussian distribution representations including means and variances corresponding to the three modality data respectively to quantify the uncertainty of the data.

[0056] As Figure 3 shown, the process of generating a Gaussian distribution representation corresponding to one modality is as follows: Using the features of one modality as the original modality, perform modeling based on the self-attention mechanism on the original modality to obtain the embedding of the features, and then perform distribution modeling on the embedding of the features to obtain the initial Gaussian distribution; Optimize the Gaussian distribution using the reconstruction loss function.

[0057] Taking the text feature h t as the original modality as an example, the specific process of generating the Gaussian distribution representation is as follows:

[0058] (1) Initialize the query matrix Q, key matrix K, and value matrix V in the self-attention mechanism to the text feature h t .

[0059] (2) Use the self-attention mechanism to model the text feature h t to obtain the in-modal text embedding f t .

[0060]

[0061] where B is the batch size, N is the sequence length, C is the feature dimension, d k is the dimension of the key vector, and softmax(·) is the normalized activation function.

[0062] (3) Perform distribution modeling on the text embedding f t to generate a Gaussian distribution with a mean μ t and variance The text feature Gaussian distribution

[0063]

[0064] Among them, Reshape(·) represents the reshaping operation, which reshapes the feature embedding into a form containing the mean and variance, and f t ′[:,:,i,:] represents slicing operation at the i-th position of the third dimension of the text embedding f t and and are the mean and variance taken out after slicing f t . Linear(·) is the projection function used to map the mean and variance to the final target dimension, and MLP(·) is the multi-layer perceptron used to learn useful feature representations from the original data to obtain the text feature distribution

[0065] (4) Sample the text feature distribution obtained in step (3) to generate multiple feature embeddings. To optimize the distribution representation of the modality, this embodiment adopts the reconstruction loss function (MSE loss) to ensure that the sampled feature embeddings can reconstruct the original features as much as possible. Specifically, the decoder reconstructs the sampled features to obtain the reconstructed features and calculates the mean square error between the original features and the reconstructed features, thereby optimizing the mean and variance of the Gaussian distribution of each modality. The purpose of this process is to retain the key information of the original features, and at the same time, by enhancing the uncertainty representation, improve the accuracy of the interaction between modalities, so as to improve the robustness and accuracy of the sentiment recognition model. The reconstruction loss function Loss REC is as follows:

[0066]

[0067] Among them, X m is the modality feature vector, is the feature reconstruction vector obtained by sampling the modality feature distribution. ‖·‖ 2 represents the Euclidean square norm of the vector.

[0068] By minimizing the reconstruction loss function Loss REC , the parameters of the text feature distribution mean μ t and variance can be effectively optimized, so that the sampled feature embeddings can better reproduce the original modality features, thereby ensuring the accuracy and reliability of the distribution representation.

[0069] By obtaining the text feature distribution The same method is used to analyze the audio features h a and visual features h v Perform operations to obtain audio feature distribution and visual feature distribution

[0070] S3: If Figure 4 As shown in the figure, a cross-modal distributed attention mechanism is used to capture the semantic association between modalities, and the uncertainty information of the distribution representation is used to optimize the modal interaction process.

[0071] The Gaussian distribution representation of each modality is semantically associated with the Gaussian distribution representations of the other two modalities in turn to generate the corresponding perceptual cross-modal association distribution. The process is as follows: the Gaussian distribution representation corresponding to one of the modalities is used as the original distribution; the Gaussian distribution representations of the other two modalities are used as the first association distribution and the second association distribution respectively; the original distribution is semantically associated twice using the first association distribution and the second association distribution in turn to obtain the perceptual cross-modal association distribution corresponding to the original distribution.

[0072] Specifically, the text feature distribution is semantically associated with the audio feature distribution through the first-layer cross-modal distributed attention module. Subsequently, the associated distribution is further semantically associated with the visual feature distribution through the second-layer cross-modal distributed attention module, ultimately resulting in a text-perceived cross-modal association distribution. Similarly, the audio feature distribution is first associated with the text modality and then with the visual modality to form an audio-perceived cross-modal association distribution; the visual feature distribution is first associated with the text modality and then with the audio modality to generate a visual-perceived cross-modal association distribution.

[0073] Distribution based on text features As the original distribution, the audio feature distribution As the first associated distribution, the visual feature distribution As an example of the second correlation distribution, generate text-aware cross-modal correlation distribution The specific process is as follows:

[0074] S3.1: Establish a cross-modal distributed attention mechanism consisting of two layers of cross-modal distributed attention modules. Q 1 Using text feature distribution Initialization, key distribution P K 1 Sum distribution P V 1 Using audio feature distribution initialization:

[0075]

[0076] S3.2: Calculate the query distribution P Q 1 and the key distribution P K 1 to obtain the global difference between them, and get the attention weights between cross-modal distributions:

[0077]

[0078] where WessAttention is the distribution attention calculated by the Wasserstein distance, and is the Wasserstein function for calculating the distance between distributions, and its formula is as follows:

[0079]

[0080] where, represents the Wasserstein distance between the distribution P Q 1 and the distribution P K 1 in the first-layer distribution attention mechanism, which is used to measure the global difference between distributions. μ1 and μ2 are the means of P Q 1 and P K 1 respectively, and σ t and σ a are the standard deviations of P Q 1 and P K 1 respectively, and ‖·‖ represents calculating the Euclidean distance.

[0081] S3.3: Sample the query distribution P Q 1 and the key distribution P K 1 N times respectively to obtain the feature embeddings Z QN and Z KN , and calculate the dot product attention between the feature embeddings to measure the local correlation between modalities:

[0082]

[0083] where Z Qi is the feature embedding obtained by sampling the query distribution P Q 1 for the i-th time, and Z Kj is the feature embedding obtained by sampling the key distribution P K 1 for the j-th time. Dot(·) represents the dot product attention function for calculating the feature embeddings, and DotAttention represents the dot product attention based on the feature embeddings ZQN and Z KN Calculate the result of taking the average of the dot product attention summation.

[0084] S3.4: Perform attention weighted summation processing on WessAttention and DotAttention:

[0085] Attention ta =(1 - α)WassAttention ta +α*DotAttention ta (11)

[0086] Among them, Attention is the result of the attention summation; α is the weight parameter that controls the contribution sizes of WassAttention and DotAttention, and is also the key to optimizing semantic decoding using modal uncertainty information.

[0087] Specifically, when the uncertainty σ of a certain modality 2 increases, the Wasserstein distance grows non-linearly, meaning that the distribution difference between this modality and other modalities is greater, and the corresponding local correlation is weaker. Therefore, in this embodiment, the attention allocation is optimized by dynamically adjusting the size of the weight parameter α: when the modal uncertainty is high, the weight parameter α is increased to increase the weight of the dot product attention DotAttention in the local correlation calculation, and at the same time reduce the influence of the Wasserstein attention WassAttention in the global difference calculation. This mechanism can adaptively suppress the global influence of high-uncertainty modalities, enhance the semantic decoding ability between modalities, and make the model more robust when dealing with modal uncertainty.

[0088] S3.5: Normalize the attention summation result Attention through the Softmax function, and then multiply it by the mean μ V 1 of the value distribution P a and the variance to obtain the result as the new parameter of the distribution associated with the text modality and the audio modality.

[0089]

[0090] In the present invention, the modal correlation distribution obtained through the cross-modal distribution attention mechanism is the core part of multi-modal fusion. To prevent the correlation distribution from degenerating into a deterministic point distribution and thus losing the uncertainty information, the Kullback-Leibler (KL) divergence is introduced as a regularization term to constrain this distribution. After the modal correlation distribution is obtained through the cross-modal attention mechanism, in order to maintain the diversity of this distribution and avoid its degeneration into a deterministic point distribution, the KL divergence is applied to the difference between this correlation distribution and the standard normal distribution N(0, I). The KL divergence Loss KL is formulated as follows:

[0091]

[0092] S3.6: Use the correlation distribution obtained in step S3.5 as the new query distribution P Q 2 in the second-layer cross-modal distribution attention module, while the key distribution P K 2 and the value distribution P V 2 are initialized with the visual feature distribution and repeat steps S3.2 - S3.5 to obtain the text-aware cross-modal correlation distributions of the text modality associated with the audio and visual modalities

[0093]

[0094] Attention tav =(1 - α)WassAttention tav +α*DotAttention tav (18)

[0095]

[0096] By the same method of obtaining the text-aware cross-modal correlation distribution , respectively operate on the audio feature distribution and the visual feature distribution to obtain the audio-aware cross-modal correlation distribution and the visual-aware cross-modal correlation distribution In this embodiment, since each modality will be associated with the other two modalities, there is no requirement for the order of initializing the key distribution and the value distribution in the first-layer and second-layer cross-modal distribution attention modules with that modality.

[0097] S4: Generate a fusion distribution using Bayesian weighted fusion based on modal uncertainty.

[0098] The text-aware cross-modal association distribution obtained in step S3 The audio-aware cross-modal association distribution and the visual-aware cross-modal association distribution are weighted and fused through the Bayesian mechanism, and the means of the three cross-modal association distributions are weighted and averaged, and the variances are harmonized:

[0099]

[0100] where μ fuSed is the mean parameter of the fused distribution, is the variance parameter of the fused distribution.

[0101] In this step, first fuse the two cross-modal association distributions, and then fuse the fused result with the third cross-modal association distribution. The order of fusion of the three cross-modal association distributions is not required:

[0102]

[0103] Obtain the fused distribution after weighted fusion of the three modal distributions

[0104] S5: Use the reparameterization technique to sample the fused distribution to obtain the final feature embedding Z fuSed , and perform sentiment recognition.

[0105] The process of reparameterized sampling of the fused distribution is as follows:

[0106]

[0107] where Z fuSed is the feature embedding obtained by sampling, which is used for sentiment recognition; is the mean parameter of the fused distribution; is the variance parameter of the fused distribution; ⊙ represents element-wise multiplication; ∈ is a random variable of the standard normal distribution.

[0108] Use the final feature embedding Z fuSed , and input it into the sentiment classification model for classification prediction. The multi-class cross-entropy loss function is used to constrain the model training, and the formula of the multi-class cross-loss function Loss CE is as follows:

[0109]

[0110] where C is the number of sentiment classification task categories, y i is the indicator function of category i in the actual label, p iis the probability of class i predicted by the model.

[0111] The calculation result of the overall loss function Loss is as follows:

[0112] Loss = Loss CE + Loss REc + Loss KL (24)

[0113] The emotion recognition method provided in this embodiment was tested on two publicly available datasets related to multi-modal emotions and recognized in the industry, and relevant experiments were conducted. The dataset and parameter settings are as follows:

[0114] In emotion recognition, the present invention uses the IEMOCAP and MELD datasets to verify the effectiveness of this embodiment.

[0115] (1) IEMOCAP: The IEMOCAP dataset contains approximately 12 hours of two-person dialogue videos, which are segmented into 7433 utterances and 151 conversations. Each utterance is labeled with one of six emotion tags: happy, sad, neutral, angry, excited, and frustrated. This dataset is labeled by multiple emotion experts to reduce subjectivity in the labeling process.

[0116] (2) MELD: The MELD dataset is a multi-character dataset containing 13708 utterances and 1433 conversations from the TV series "Friends". Each utterance is labeled with one of seven emotion categories: angry, disgusted, fearful, joyful, neutral, sad, and surprised. Similar to IEMOCAP, the MELD dataset is also labeled by multiple emotion experts to ensure the accuracy and objectivity of the labeling.

[0117] During the experiment, for the IEMOCAP dataset (six-class classification) and the MELD dataset (seven-class classification), a comprehensive and detailed evaluation method was adopted to accurately measure the performance of the model in the emotion classification task. Specifically, the present invention calculated the F1 score for each emotion category to evaluate the classification accuracy of the model on different emotion categories. In addition, to comprehensively reflect the overall performance of the model and consider the imbalance in the class distribution in the dataset, the present invention further calculated the weighted average F1 score (WeightedF1Score). This evaluation strategy can not only accurately describe the classification effect of the model on each category but also effectively measure the impact of class imbalance on the overall performance, thus providing a scientific basis for the optimization and improvement of the model.

[0118] Analysis of experimental results:

[0119] The method proposed by the present invention has been verified on the IEMOCAP dataset (six-class classification) and the MELD dataset (seven-class classification), and a comparative analysis has been conducted with existing advanced models. In the IEMOCAP dataset, the method of the present invention has achieved better performance in all emotion categories. The overall weighted F1 score has increased by 2.43%. Among them, the F1 scores of the Happiness and Sadness categories have increased by 8.89% and 4.3% respectively. For the Frustration and Anger categories with fewer samples and more confusion, they have increased by 2.31% and 2.08% respectively, showing stronger discriminative ability. Especially under the condition of data imbalance, the method of the present invention still shows strong generalization ability in the categories with scarce samples. In the MELD dataset, the overall weighted F1 score of the method of the present invention has increased by 0.83%. Among them, it has increased by 23.63% and 7.87% in the low-resource categories Fear and Disgust respectively, and by 0.66% and 0.71% in the main categories Neutral and Joy respectively. Through the uncertainty-aware mechanism, this method can still accurately capture emotional features in the context of noise interference or semantic ambiguity, thus improving the accuracy and robustness of emotion classification.

[0120] In the cross-modal distribution attention module, the parameter α is the key factor for balancing the contributions of DotAttention and WassAttention. To verify its effectiveness in dealing with modal uncertainty, in this embodiment, different intensities of Gaussian noise are introduced into the IEMOCAP dataset. As the noise intensity increases, the F1 accuracy gradually decreases. Higher uncertainty will weaken the quality of modal information and affect the model performance. At the same time, α also gradually increases with the increase of noise. In a low-noise environment, the value of α is small (for example, when the noise intensity is 1, α = 0.50), and the model relies on WassAttention to maintain global semantic consistency. When the noise increases (for example, when the noise intensity is 10, α = 0.90), α gradually increases, and the model turns to DotAttention to enhance local correlation, while suppressing the interference of WassAttention in global semantic modeling, thereby reducing the negative impact of high-uncertainty modalities. The experimental results verify the dynamic attention adjustment mechanism driven by modal uncertainty: when the modal uncertainty is high, increase α to strengthen the modeling of local correlation by DotAttention, while weakening the influence of WassAttention on global features. This mechanism enables the model to adaptively adjust the attention allocation when facing high-uncertainty modalities, enhances the cross-modal semantic decoding ability, and thus improves the robustness of the emotion analysis task.

[0121] Embodiment 2

[0122] An emotion recognition system, comprising:

[0123] Data acquisition module: responsible for acquiring multimodal data; the multimodality includes text modality, audio modality, and visual modality.

[0124] Feature extraction module: responsible for preprocessing the multimodal data and extracting text features, audio features, and visual features.

[0125] Modal distribution modeling module: responsible for performing uncertainty modeling on multiple modal features respectively to obtain a Gaussian distribution representation including mean and variance to quantify the uncertainty of the data.

[0126] Two-layer cross-modal distribution attention module: responsible for capturing semantic associations between modalities and optimizing the modal interaction process using the uncertainty information of the distribution representation.

[0127] Bayesian fusion module: responsible for fusing the distributions obtained after multiple cross-modal associations.

[0128] Reparameterized sampling module: responsible for sampling the fused distribution to obtain the final feature embedding.

[0129] Recognition module: responsible for inputting the multimodal fusion result obtained by sampling into the classifier of the sentiment recognition model to obtain the sentiment recognition result.

[0130] Finally, it should be noted that the above-described embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting them. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the technical field of the present invention can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for emotion recognition based on uncertainty-aware multimodal information fusion, characterized in that: The emotion recognition method is as follows: Model the features of each modality respectively to generate Gaussian distribution representations; Establish a cross-modal distribution attention mechanism; for the Gaussian distribution representation of each modality, perform semantic association with the Gaussian distribution representations of the remaining modalities in sequence to obtain cross-modal association distributions; Perform weighted fusion on the multiple cross-modal association distributions obtained by semantic association to generate a fusion distribution; Sample the fusion distribution to obtain feature embeddings; Input the feature embeddings into an emotion recognition model to obtain emotion recognition results.

2. The emotional recognition method according to claim 1, characterized in that: The process of modeling the features of the target modality to generate a Gaussian distribution representation is as follows: Initialize the query matrix, key matrix, and value matrix in the self-attention mechanism with the features of the target modality; Use the self-attention mechanism to model the features of the target modality to obtain embeddings of the target modality; Perform distribution modeling on the embeddings of the target modality to generate an initial Gaussian distribution.

3. The emotion recognition method according to claim 1, characterized in that: The process of performing semantic association on two distributions is as follows: The two distributions are the original distribution and the associated distribution respectively; establish a cross-modal distribution attention mechanism, initialize the query distribution with the original distribution, and initialize the key distribution and value distribution with the associated distribution; Calculate the similarity between the query distribution and the key distribution as the distribution attention; Perform multiple samplings on the query distribution and the key distribution respectively to obtain multiple feature embeddings; Calculate the dot product attention between the feature embeddings corresponding to the query distribution and the key distribution; Perform weighted fusion on the distribution attention and the dot product attention to obtain comprehensive attention; Multiply the normalized value of the comprehensive attention by the mean and variance of the associated distribution to obtain the mean and variance of the distribution obtained after semantic association.

4. The emotional recognition method according to claim 3, wherein: In the weighted fusion of the distribution attention and the dot product attention, the weight parameter of the dot product attention is dynamically adjusted according to the uncertainty of the distribution. The higher the uncertainty, the larger the value of the weight parameter of the dot product attention.

5. The emotion recognition method according to claim 1, wherein: The process of performing weighted fusion on multiple cross-modal association distributions is: perform the fusion of two cross-modal association distributions successively until the fusion of all cross-modal association distributions is completed; The process of fusing two cross-modal association distributions is: perform weighted average calculation on the means of the cross-modal association distributions to obtain the mean of the distribution obtained by fusing the two cross-modal association distributions; perform harmonic adjustment on the variances of the cross-modal association distributions to obtain the variance of the distribution obtained by fusing the two cross-modal association distributions.

6. The emotional recognition method according to claim 1, characterized in that: The overall loss function of the emotion recognition model includes the reconstruction loss function for generating Gaussian distribution representations by modeling the features of each modality, the KL divergence loss function for cross-modal association distributions generated by semantic association, and the multi-class cross-entropy loss function for emotion recognition.

7. The emotion recognition method according to claim 1, wherein: Feature embeddings are obtained by reparameterized sampling of the fusion distribution.

8. An emotion recognition system, characterized in that: It is used to execute the emotion recognition method according to any one of claims 1-7; the emotion recognition system includes a data acquisition module, a feature extraction module, a modality distribution modeling module, a cross-modal distribution attention module, a Bayesian fusion module, a reparameterized sampling module, and an identification module; The data acquisition module is used to acquire multi-modal data including text, audio, and visual modalities; The feature extraction module is used to preprocess the multi-modal data and extract text features, audio features, and visual features; The modal distribution modeling module is used to model the uncertainty of multi-modal features and obtain the Gaussian distribution representation corresponding to each modality; The cross-modal distribution attention module is used to capture the semantic associations between the distributions of each modality and generate corresponding cross-modal association distributions for the distributions of each modality respectively; The Bayesian fusion module is used to fuse multiple cross-modal association distributions; The reparameterized sampling module is used to sample the fused distribution to obtain feature embeddings; The recognition module is used to perform sentiment recognition on the feature embeddings.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that: The memory stores a computer program; the processor executes the sentiment recognition method according to any one of claims 1-8.

10. A readable storage medium stores a computer program; characterized in that: When the computer program is executed by the processor, it is used to implement the sentiment recognition method according to any one of claims 1-8.

Images