Multi-modal dialogue emotion recognition method and system based on fine-grained modal alignment

Abstract

The invention relates to the technical field of multi-modal emotion recognition, and provides a multi-modal dialogue emotion recognition method and system based on fine-grained modal alignment, firstly, redundant information possibly introduced by a simple fusion mode is avoided through fine-grained modal alignment, so that the calculation burden is reduced, and the emotion recognition efficiency is improved; and secondly, through alignment of potential variables between modes, the problem of semantic gaps between the modes is solved, and complementarity and synergistic effect of information between different modes are improved. And thirdly, through adaptive fusion of the Transform model, the model can automatically adjust the fusion weight between modals, the fusion precision is further improved, and finally the emotion recognition accuracy is improved.

Description

Technical Field

[0001] This disclosure relates to the field of multimodal emotion recognition technology, specifically to a multimodal dialogue emotion recognition method and system based on fine-grained modal alignment. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Multimodal dialogue sentiment recognition aims to analyze and understand the emotional state in a dialogue by integrating data from different modalities (such as speech, text, and video). Unlike single-sentence sentiment analysis, sentiment recognition requires not only analyzing information from each modality but also considering the context of the dialogue, the interaction between participants, and the dynamic changes in emotions. Therefore, effectively fusing information from different modalities is key to improving the accuracy of sentiment analysis. In multimodal sentiment recognition, modality alignment is a crucial step. It refers to converting information from different modalities (such as text, speech, and images) into a unified representation so that these different modalities can interact and merge effectively. The purpose of modality alignment is to generate a unified multimodal representation, enabling the model to comprehensively analyze information from different modalities and perform more accurate sentiment recognition.

[0004] The core problem with existing technologies is that different modalities contribute differently to the task. Simple fusion strategies may introduce a large amount of redundant information. For example, fusion methods such as simple weighting or concatenation may fuse unnecessary information (redundant information) from different modalities. This redundant information does not make a substantial contribution to the emotion recognition task; instead, it may make the model more complex, increase the computational burden, and affect the final accuracy of emotion recognition.

[0005] Furthermore, current modality alignment methods typically align the entire modality as a unit, neglecting the differences in fine-grained features within each modality. This prevents the full extraction of the unique emotional information provided by each modality, thus affecting the fusion effect. In multimodal emotion recognition, each modality (such as speech, text, and images) has different characteristics in expressing emotions. Speech primarily conveys emotion through tone and speech rate, while text expresses emotion through semantic content. This difference can lead to a "semantic gap," i.e., semantic inconsistencies between modalities. If cross-modal information interaction is performed directly without considering these fine-grained differences and without meticulous processing of these complex and high-dimensional emotional signals, the model will struggle to accurately understand the mutual influence and enhancement between modalities. This may result in poor information fusion and affect the accuracy of emotion recognition.

[0006] It is evident that current multimodal emotion recognition methods often fail to adequately consider the semantic differences between different modalities, simply aligning the modalities as a whole into a common space while ignoring the fine-grained features within each modality. This approach struggles to effectively capture complementary information between modalities, affecting the fusion of emotion signals and resulting in lower accuracy in emotion recognition. Summary of the Invention

[0007] To address the aforementioned issues, this disclosure proposes a multimodal dialogue emotion recognition method and system based on fine-grained modal alignment. By using fine-grained modal alignment and independently encoding different emotion dimensions, it effectively avoids the fusion of redundant information. Furthermore, by adaptively learning the fusion weights between modalities through a Transformer model, it improves the accuracy and efficiency of multimodal emotion recognition.

[0008] To achieve the above objectives, the present disclosure adopts the following technical solution: One or more embodiments provide a multimodal dialogue emotion recognition method based on fine-grained modality alignment, comprising the following steps: Acquire multimodal information and generate common-sense knowledge representations related to the input text in the multimodal information; Independent VAE models are set up for each modality information and common sense knowledge representation. Different sentiment dimensions of each modality are independently encoded to obtain latent variables that correspond to each modality and contain multiple sentiment dimensions. The sentiment dimensions of the latent variables of different modalities are aligned by minimizing the KL divergence between modalities. For the latent variables of the aligned sentiment dimension, resampling is performed on each sentiment dimension to obtain a feature vector, resulting in the aligned reconstructed representation. ; Reconstructed representation of each aligned modality Adaptive modality fusion is performed using the Transformer model, which learns the fusion weights between modalities. By obtaining unified fusion features, emotion classification is performed to obtain emotion recognition results.

[0009] One or more embodiments provide a multimodal dialogue emotion recognition system based on fine-grained modality alignment, including: The common sense knowledge generation module is configured to acquire multimodal information and generate common sense knowledge representations related to the input text in the multimodal information; The sentiment dimension alignment module is configured to set up independent VAE models for each modality information and common sense knowledge representation, independently encode different sentiment dimensions for each modality, and obtain latent variables that contain multiple sentiment dimensions corresponding to each modality. The sentiment dimensions of the latent variables of different modalities are aligned by minimizing the KL divergence between modalities. The reconstruction module is configured to resample the latent variables of the aligned sentiment dimension for each sentiment dimension to obtain a feature vector, thus obtaining the aligned reconstructed representation. ; The adaptive modality fusion module is configured to reconstruct the representation of each aligned modality. Adaptive modality fusion is performed using the Transformer model, which learns the fusion weights between modalities. By obtaining unified fusion features, emotion classification is performed to obtain emotion recognition results.

[0010] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps in the above-described multimodal dialogue emotion recognition method based on fine-grained modality alignment.

[0011] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps in the above-described multimodal dialogue emotion recognition method based on fine-grained modality alignment.

[0012] Compared with the prior art, the beneficial effects of this disclosure are as follows: This disclosed multimodal dialogue sentiment recognition method firstly avoids redundant information that may be introduced by simple fusion methods through fine-grained modality alignment, thereby reducing the computational burden and improving the efficiency of sentiment recognition. Secondly, by aligning latent variables between modalities, it solves the "semantic gap" problem between modalities, enhancing the complementarity and synergy of information between different modalities. Thirdly, through adaptive fusion using the Transformer model, the model can automatically adjust the fusion weights between modalities, further improving the fusion accuracy and ultimately enhancing the accuracy of sentiment recognition.

[0013] The advantages of this disclosure, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0014] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute a limitation thereof.

[0015] Figure 1 This is a flowchart of the multimodal dialogue emotion recognition method of Embodiment 1 of this disclosure; Figure 2 This is a flowchart illustrating the multimodal dialogue emotion recognition method using the emotion prediction model of Embodiment 1 of this disclosure; Detailed Implementation The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0016] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0017] It should be noted that the terminology used herein is for descriptive purposes only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0018] This invention discloses a fine-grained modality-aligned multimodal dialogue sentiment recognition method. The method uses a variational autoencoder to decouple the sentiment features of each modality using the VAD dimension, and achieves modality alignment by minimizing the KL divergence of the Gaussian distributions between modalities. Common sense knowledge representation based on the COMET model is introduced, and common sense modality information is incorporated through a cross-attention mechanism during the modality fusion stage. Simultaneously, a learnable modality weight mechanism is designed to achieve adaptive fusion between modalities within a Transformer structure, thereby improving the accuracy and robustness of sentiment recognition. Specific embodiments are described below.

[0019] Example 1 In one or more of the technical solutions disclosed in the embodiments, such as Figures 1 to 2 As shown, a multimodal dialogue emotion recognition method based on fine-grained modality alignment includes the following steps: Step 1: Obtain multimodal information and generate common sense knowledge representations related to the input text in the multimodal information; Step 2: Set up independent VAE models for each modality information and common sense knowledge representation, and independently encode the different sentiment dimensions VAD for each modality to obtain latent variables that correspond to each modality and contain multiple sentiment dimensions. Align the sentiment dimensions of the latent variables of different modalities by minimizing the KL divergence between modalities. Step 3: For the latent variables of the aligned sentiment dimensions, resample for each sentiment dimension to obtain a feature vector, thus obtaining the aligned reconstructed representation. ; Step 4: Reconstruct the representation of each aligned modality. Adaptive modality fusion is performed using the Transformer model, which learns the fusion weights between modalities. By obtaining unified fusion features, emotion classification is performed to obtain emotion recognition results; This embodiment is based on the idea of ​​fine-grained modality alignment. It reduces the "semantic gap" between modalities by independently encoding the sentiment dimension of each modality and performing fine-grained alignment of latent variables between modalities. In step 1, common-sense knowledge representation helps bridge the semantic differences between different modal information. In step 2, a variational autoencoder (VAE) is used to independently encode the sentiment dimension of each modality, and KL divergence minimization is used to align latent variables between modalities, ensuring that no redundant information is introduced during the alignment process. In step 3, resampling is used to further refine the latent variables of each sentiment dimension, ensuring that the aligned latent variables are more accurate at a fine-grained level. Finally, in step 4, a Transformer model is used for modality fusion. By learning the fusion weights between each modality, the feature information of each modality is effectively integrated, ultimately obtaining the sentiment recognition result.

[0020] In this embodiment, firstly, fine-grained modality alignment avoids redundant information that may be introduced by simple fusion methods, thereby reducing the computational burden and improving the efficiency of emotion recognition. Secondly, by aligning latent variables between modalities, the "semantic gap" problem between modalities is solved, enhancing the complementarity and synergy of information between different modalities. Thirdly, through adaptive fusion using the Transformer model, the model can automatically adjust the fusion weights between modalities, further improving the fusion accuracy and ultimately enhancing the accuracy of emotion recognition.

[0021] In some embodiments, in dialogue emotion recognition, multimodal information includes text features, acoustic features, and visual features; First, we define the problem: in dialogue sentiment recognition, a dialogue consists of a sequence of utterances spoken by S speakers. , where n is the number of discourses.

[0022] Every word Corresponding to an emotional tag ,in , A predefined set of labels. Sentiment recognition in dialogue tasks aims to predict the sentiment label for each utterance in a conversation.

[0023] Typically, in a multimodal environment, each sentence includes data from three modalities, represented as follows: ; in, Representing text features, Indicates acoustic characteristics, Representing visual features, their dimensions are respectively , , ; t, a, and v represent the text modality, acoustic modality, and visual modality, respectively; Step 1 also includes the step of acquiring common sense knowledge. In dialogues, many emotional expressions need to be understood through common sense reasoning in context. Common sense knowledge can help the model understand the differences and characteristics of emotional expressions in different cultures or environments.

[0024] In order to introduce common sense knowledge modality into multimodal emotion recognition, in step 1, common sense knowledge related to the input text features is generated based on the acquired multimodal information. A trained COMET model can be used to extract relevant events based on the input text features, transmit the relevant events to the trained COMET model, and output the corresponding common sense knowledge. Specifically, first, relevant events are extracted from the input text, and then these events are used to generate related common-sense knowledge, using the following formula: (1); in, This represents the generated common-sense knowledge. This represents the parameters of the trained model. Represents a given event, It indicates a reasoning relationship.

[0025] Optionally, the COMET model can be trained using the ATOMIC dataset to generate knowledge representations related to the features of the input text.

[0026] ATOMIC is a knowledge base containing a large amount of event-based reasoning, which can provide rich information about the causal relationships and reasoning logic of social interactions.

[0027] COMET is a neural network-based reasoning model capable of generating these knowledge representations. By combining ATOMIC and COMET, more contextual and reasoning-based common-sense knowledge modalities can be provided to the input text, enhancing the model's understanding of sentiment. Step 2, feature processing and sentiment decoupling: An independent variational autoencoder (VAE) model is used for each modality to encode the sentiment features of each modality; the sentiment features of each modality are decoupled into multiple latent variables. In this embodiment, the latent variables are set to four: Valence, Arousal, Dominance, and Content. A VAE model is constructed for each modality, and the different emotional dimensions (VAD) of each modality are independently encoded. Specifically, VAE is a generative model that includes an encoder and a decoder; An encoder maps input data x to a latent space, and the distribution parameters of the output latent variable z can be the mean and variance. The goal of an encoder is to learn a latent representation of the input data.

[0028] Decoder: Generates reconstructed input data x from the latent variable z. The decoder generates samples similar to the input data by sampling the latent variable. VAEs can sample and generate new data from the latent space, enhancing the model's generalization ability.

[0029] To obtain a high-quality VAE model, the optimization objectives during the VAE model training process include: (1) The VAE model adopts variational inference during the training process. With the goal of minimizing reconstruction error and KL divergence, it constructs variational lower bound and maximizes variational lower bound so that the VAE model can better fit the data and approach the true prior distribution. Each VAE encodes the input data, converting the input features... The values ​​mapped to the latent space include the mean. and logarithmic variance ,Right now: (2); in, V, A, D, C represent pleasure, arousal, dominance, and emotion-irrelevant content, respectively. Let k represent the common sense knowledge mode; The latent variables output by the encoder of each VAE model Follows a Gaussian distribution: (3); Where I is the identity matrix; The core of the VAE model lies in maximizing the marginal likelihood p(x) of the data, as follows: (4); The goal of the VAE model is to learn to make the generated data (data x generated from the latent space z) as similar as possible to the real data. Since directly calculating the marginal log-likelihood is not feasible in high-dimensional spaces and is computationally difficult, this embodiment optimizes the VAE model by introducing the variational lower bound (ELBO). Specifically, the variational lower bound (ELBO) is constructed as follows: (5); Part One The reconstruction error component is used to measure the quality of the data x generated from the latent variable z, and is actually used to evaluate the performance of the generative model. Part Two To reconstruct the error part of the KL divergence, let represent the variational posterior distribution. With prior distribution The difference between the two distributions. KL divergence measures the difference between the two distributions and is often used to control the distribution of the latent variable to be close to the prior distribution.

[0030] This embodiment optimizes both the reconstruction quality and the consistency between the latent distribution and the prior distribution by maximizing the ELBO. Since directly calculating the marginal likelihood (the integral in formula (4) above) is very difficult, the above scheme in this embodiment uses variational inference to introduce a lower bound (ELBO, Evidence Lower Bound). This optimizes an approximation without having to directly calculate the difficult integral. (2) During the training process, the output accuracy of the sentiment dimension of the latent variables of the VAE model encoder is evaluated. Sentiment labels are set for each sentiment dimension, and the mean squared error (MSE) is used as the loss function for training. Specifically, during the VAE model training process, three sentiment dimensions (VAD, Arousal, Dominance) are constructed. Sentiment labels for each word are generated through a sentiment dictionary. A rating predictor is connected to the encoder of the VAE model to predict the sentiment dimension output of the encoder. The mean squared error (MSE) is used as the loss function during the training process to optimize the difference between the VAE model's prediction results and the true labels, thereby improving the accuracy of sentiment analysis. Specifically, the rating predictor consists of a linear layer and an activation layer connected in sequence. The activation layer outputs a score using a sigmoid activation function. Limited to the range (0,1); A sentiment tag is generated for each word using a sentiment dictionary. Optionally, the NRC-VAD sentiment dictionary can be used, which contains over 20,000 English words and provides detailed descriptions of the emotional intensity of words across three independent dimensions: Valence, Arousal, and Dominance. Then, an aggregation process is used to calculate the VAD (Valence Ability Difference) for each predefined tag, ranging from 0 to 1. This VAD score is used as the sentiment tag for each word. pseudo-tags . Based on the mean vector μ output by the VAE model encoder, a predicted score for the sentiment dimension is generated by the rating predictor.

[0031] The rating predictor is a linear layer, which is then activated by a sigmoid function to produce the output score. The range is limited to (0,1). To measure the difference between the model's predicted sentiment dimension and the pseudo-label, Mean SquaredError (MSE) is used to quantify this difference: (6); Where N represents the batch size.

[0032] (3) Decoupling of the sentiment dimension: Construct an optimization objective that minimizes the mutual information between latent variables in order to reduce the dependency between latent variables output by the VAE model encoder; Mutual information (MI) measures the degree of dependence between two random variables. For two random variables Zi and Zj, their mutual information is defined as: (7); In this embodiment, four VAE models are set up, and the latent variables output by the encoders respectively satisfy Gaussian distributions, so that the mutual information between the means of the four Gaussian distributions is minimized: (8); Specifically, by minimizing the mutual information among these latent variables, the model is driven to capture the unique information of each latent variable, thereby achieving feature decoupling and independence.

[0033] Furthermore, in step 2, fine-grained mode alignment is performed by calculating the KL divergence of all modes of the latent variables output by the VAE model to minimize the KL divergence between modes and construct the alignment loss, thereby minimizing the differences between different modes. By making the Gaussian distributions of the sentiment dimensions corresponding to each modality more similar, the feature differences between modalities are minimized, thus improving the consistency of modal features in the sentiment space. KL divergence is calculated for all modality pairs (i,j), and the latent representations of each modality i and j are computed by the model through the encoder part, with each modality corresponding to a Gaussian distribution. and The alignment loss of each pair of modes is summed to obtain the total alignment loss: (9); Finally, the overall loss function for training the VAE model is: ; in, , , and These are hyperparameters that control the four types of loss weights.

[0034] In this embodiment, minimizing the KL divergence enhances the consistency of Gaussian distributions across different modalities within the sentiment space, thereby improving model performance, particularly in sentiment analysis and multimodal tasks. This is achieved by optimizing the fine-grained modality alignment loss function. In step 3, feature reconstruction is performed, and a feature vector is obtained by resampling each sentiment dimension: (10); Where ϵ is the noise sampled from the standard normal distribution N(0,1). The resampled features are then concatenated. ; ; ; Obtain the aligned reconstructed representation .

[0035] Different modalities (such as text, audio, and visual) have semantic gaps in expressing emotions, and traditional methods often fail to align their semantic spaces in the emotional dimension. This embodiment aligns the Valence, Arousal, and Dominance (VAD) dimensions of each modality, enabling each modality to have a unified semantic representation in the same emotional dimension.

[0036] Next, common-sense knowledge is incorporated during modality fusion. Common-sense knowledge, as a special text modality, supplements and aids in the fusion of the three modalities. Considering the differences in data quality and reliability among different modalities, and the varying importance of modalities in different scenarios, step 4 of this embodiment introduces an adaptive modality fusion method.

[0037] In step 4, the Transformer model is used for adaptive modality fusion, by learning the fusion weights between modalities. The method for obtaining emotion recognition results by acquiring unified fusion features and performing emotion classification includes the following steps: Step 41, Single-modal fusion under multimodal-assisted enhancement: For each modality m, the reconstructed representation of modality m is constructed as a query, the reconstructed representations of all modalities are constructed as values ​​and keys, and attention is calculated to obtain the attention-processed features corresponding to each modality; Specifically, multimodal interactions are embedded into each attention layer of the Transformer. These Transformer attention layers contain three inputs: queries, keys, and values. The reconstructed representation of each modality is taken as Q, and attention is calculated for its four modalities Q and K respectively to learn richer intermodal relationships. The formula is expressed as: (11); in, It is a feature of fusion. ; In this embodiment, four modalities are included: text modality. Acoustic modes Visual modal And common sense knowledge modality k, such as for text modality Q and K are calculated based on all modalities, resulting in four features. All modalities result in 16 fused features. Step 42: Calculate the learnable first weights based on the obtained fusion features. The fusion features corresponding to each modality are fused to obtain the fused features for each modality. Learnable weights are set to adaptively adjust the weights of the four fusion features within a single modality, thereby weighted connections for fusion: (12); (13); in, These are the learned fusion weights. and These are learnable parameters. It is the sigmoid activation function. It is an element-wise product.

[0038] Step 43: Calculate the learnable second weights for each modality based on the features after modal fusion. Multimodal fusion is performed.

[0039] Since each modality contributes differently to the final result, learnable weights can be used. This is used to adjust the weights of different modalities. Then, the single-modal fusion features obtained for each modality are... exist The connection is adjusted to obtain the final fusion feature H: (14); (15); Traditional modal fusion methods suffer from inconsistent data quality across different modalities, information redundancy, or conflicts during the fusion process, making it impossible to dynamically adjust the importance of each modality. This embodiment introduces a modal interaction mechanism into each layer of the Transformer and sets learnable weights, enabling weighted fusion of each modal feature. This allows for dynamic adjustment of the contribution ratio of each modality to the final result using learnable parameters, preventing weak modalities from affecting overall performance.

[0040] A further technical solution is to make the unified fusion result in step 43 close to the single-modal prediction result. When training the Transformer model, the method of minimizing the KL divergence (Kullback-Leibler Divergence) is used to maximize the mutual information between the fusion result and the input modality, so that the fusion result is close to the single-modal prediction result. Although a good fusion feature has been obtained, the relationship between each individual modality and the fusion feature has not been fully explored. Therefore, the method of minimizing Kullback-Leibler Divergence is used to make the fusion result closer to the individual modality prediction result, thereby reducing noise in the fusion modality and making the model more robust to changes in the input data.

[0041] Specifically, the features obtained from each modality fusion in step 42 and the features of the fused modality are respectively passed through a sentiment classifier to obtain the corresponding sentiment probability distribution, wherein: the features obtained from each modality fusion include text, audio and video modal features; The corresponding sentiment probability distribution is obtained, expressed as: (16); (17); in, It is a unimodal predicted sentiment probability distribution. It is a predicted sentiment probability distribution based on fusion modalities. , , W and b are learnable parameters; Each classifier is trained using cross-entropy loss: (18); (19); Where N represents the number of utterances in the conversation, L represents the number of sentiment categories, and Yi represents the one-hot vector of the true label.

[0042] Using KL divergence to make the fused features approximate the output probabilities of a single modality: (20); Finally, the overall loss function for training the Transformer model is: ; in, These are hyperparameters that control the four types of loss weights.

[0043] The implementation of the above method steps is achieved through the constructed sentiment prediction model, including the COMET model, the VAE model set according to the number of modalities, the reconstruction module, and the Transformer model; during the training process, multiple loss functions need to be optimized simultaneously using the phased training method: In the first stage, within the set number of training times e, that is, when epoch < e, freeze the other parameters in the overall model and train the encoder and decoder of the VAE model: Among them, the loss function formulas (5), (6), (8), and (9) for VAE model training are designed to train a high-quality VAE model to achieve fine-grained modality alignment. To prevent interference in subsequent training and maintain the performance of the high-quality VAE, the overall loss function is: (21) Among them, 、 、 and are hyperparameters that control the weights of the four losses. Preferably, set = = = = 1.

[0044] In the second stage, that is, when epoch >= e, freeze the parameters of the VAE model and use the main loss [[ID=�0]]to train the parameters of the Transformer model and other parameters: (22) / / Among them, is a hyperparameter that controls the weights of the four losses. Through experiments, it can be set .

[0045] To illustrate the effect of the method in this embodiment, experiments were conducted; the experimental results show the effectiveness of the method (abbreviated as FGMAM) proposed in this embodiment.

[0046] The FGMAM of this embodiment was compared with the following baseline models: BC-LSTM encodes context semantic information through a Bi-LSTM network to perform sentiment prediction.

[0047] DialogueRNN introduces a novel RNN architecture that uses global and specific sentiment GRUs to effectively capture the states and context dynamics of individual speakers in the dialogue to improve the effect of sentiment detection.

[0048] DialogueGCN uses graph convolutional neural networks (GCN) to model the relationships between interlocutors in a dialogue.

[0049] CTNet introduces a transformer-based architecture that effectively captures internal modalities and cross-modal interactions in multimodal inputs.

[0050] COGMEN is a graph neural network-based model that can capture both local (internal and inter-speaker dependencies) and global contextual information.

[0051] SCMM addresses two major limitations of existing methods: the lack of diverse dependency modeling (including long-short-term and context-specific representations) and a consistent approach to handling contributions from different modalities.

[0052] AdaIGN employs directed graph structures, node and edge selection strategies, and graph-level selection strategies to improve performance, while using task-specific loss functions to prioritize certain modalities.

[0053] GraphCFC introduces a directed graph-based module that can efficiently model context and interaction information in dialogues. It addresses the shortcomings of existing methods by employing multiple subspace extractors and pairwise cross-modal complementarity strategies.

[0054] MGLRA proposes an iterative alignment mechanism to progressively improve modal alignment and noise reduction for multimodal emotion recognition in dialogue. The model applies memory-based recursive alignment and a mask graph convolutional network (GCN) to achieve robust feature fusion.

[0055] 1) Comparative experiment; Specifically, for the IEMOCAP dataset in Table 1, the method achieved an accuracy of 72.95% on the WAA dataset and 73.31% on the WAF1 dataset, outperforming the aforementioned existing baselines. While the method in this embodiment achieves a lower F1 score for a specific emotion category compared to some other methods—for example, the method in this embodiment achieves an F1 score of 80.68% for sadness, while MGLRA achieves 84.99% for sadness—this is because sadness and neutral emotion features are very similar. For instance, in sadness, speech may be monotonous, slow, and low-pitched, with minimal facial changes, while neutral emotion features are typically characterized by little emotional fluctuation and a bland appearance. The model is prone to mispredicting sadness as neutral. However, the FGMAM proposed in this embodiment outperforms the baseline results in other emotion categories. The proposed method improves performance on WAA by approximately 13% compared to BC-LSTM on WAF1 and by approximately 14% compared to WAA. This observation demonstrates the importance of cross-modal fusion strategies. Furthermore, compared to CTNet, which employs a multimodal fusion method, it achieves improvements of 5.4% and 6.3% on WAA and WAF1, respectively. The main reason is that these existing methods only focus on multimodal fusion, neglecting the alignment between modalities. Moreover, the method proposed in this embodiment is superior to the existing method COGMEN, as it highlights speaker and context-related information while ignoring the introduction of multimodal information.

[0056] WAA is the weighted average accuracy, and WF1 is the weighted average F1 score; For the MELD dataset, it provides more fine-grained classification. Compared with other methods, the method proposed in this embodiment achieves the highest metrics (Table 2), with an accuracy of 67.66% on WAA and 66.81% on WAF1. The proposed FGMAM improves WAA and WAF1 accuracy by 0.6% and 1.2% respectively compared to the newly proposed MGLRA. Furthermore, compared to text modality-dependent Bc-LSTM and DialogueRNN, the proposed WAF1 improves by approximately 9%. This enhancement is due to a cross-modal unified design utilizing information from different sources. Compared to CTNet, which considers multimodal fusion, the method that achieves high alignment in the sentiment space through sentiment representations of each modality improves WAF1 by 23.5%, validating the effectiveness of fine-grained modality alignment and adaptive modality fusion.

[0057] Table 1. Experimental results on the IEMOCAP dataset;

[0058] Table 2. Experimental results on the MELD dataset;

[0059] 2) Ablation experiment; In this section, ablation studies were conducted on two datasets to investigate the contribution of each module in FGMAM. Consider the following settings: w / o CKA: Do not use the common sense knowledge assistance module.

[0060] Only with text: Combines common-sense knowledge with textual modal features using cross-attention.

[0061] Only with fusion: Only common sense knowledge is combined with the fused features using cross attention. w / o FGMA: Do not use the fine-grained modal alignment module.

[0062] w / o AMF: Modality fusion is performed using only cross-attention instead of the adaptive modality fusion module.

[0063] Table 3. Results of ablation studies

[0064] Table 3 shows the results of the ablation experiment, from which we can conclude that: (1) Without the help of common sense knowledge, the model cannot better understand the implicit emotions in sentences, nor can it provide auxiliary information in different fields (such as medical, educational, social media, etc.) or different cultural backgrounds, resulting in a decline in performance. (2) Different modalities may apply common sense knowledge in different ways when expressing emotions. If common sense knowledge is only applied to text features, it may miss the potential important emotional information in other modalities, thus affecting the accuracy, richness and robustness of sentiment analysis. (3) If common sense knowledge is only supplemented to the features after modality fusion, it may also miss the unique contributions between each modality. For example, the visual modality may convey emotional information through details in the image (such as facial expressions, body posture, etc.), while the audio modality conveys emotions through elements such as tone, pauses, and speed of speech. Simply fusing modalities and then applying common sense knowledge may not be able to effectively capture the differences in emotional expression between modalities. (4) It can be observed that not using FGMA leads to poor results. This can be attributed to the model's difficulty in handling noise or interference information from different modalities, resulting in unstable emotion recognition results or oversensitivity to irrelevant factors. (5) If the adaptive modality fusion module is not used, it is impossible to capture the interrelationships and interaction information between the various modalities and ignore the differences in the importance of the modalities, thereby affecting the model's performance.

[0065] Example 2 Based on Example 1, this example provides a multimodal dialogue emotion recognition system based on fine-grained modality alignment, including: The common sense knowledge generation module is configured to acquire multimodal information and generate common sense knowledge representations related to the input text in the multimodal information; The sentiment dimension alignment module is configured to set up independent VAE models for each modality information and common sense knowledge representation, independently encode different sentiment dimensions for each modality, and obtain latent variables that contain multiple sentiment dimensions corresponding to each modality. The sentiment dimensions of the latent variables of different modalities are aligned by minimizing the KL divergence between modalities. The reconstruction module is configured to resample the latent variables of the aligned sentiment dimension for each sentiment dimension to obtain a feature vector, thus obtaining the aligned reconstructed representation. ; The adaptive modality fusion module is configured to reconstruct the representation of each aligned modality. Adaptive modality fusion is performed using the Transformer model, which learns the fusion weights between modalities. By obtaining unified fusion features, emotion classification is performed to obtain emotion recognition results.

[0066] It should be noted that each module in this embodiment corresponds one-to-one with each step in embodiment 1, and their specific implementation process is the same, so it will not be repeated here.

[0067] Example 3 This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it completes the steps in the multimodal dialogue emotion recognition method based on fine-grained modal alignment in Embodiment 1.

[0068] Example 4 This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, they complete the steps in the multimodal dialogue emotion recognition method based on fine-grained modal alignment in Embodiment 1.

[0069] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0070] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A multimodal dialogue emotion recognition method based on fine-grained modal alignment, characterized in that, Includes the following steps: Acquire multimodal information and generate common-sense knowledge representations related to the input text in the multimodal information; Independent VAE models are set up for each modality information and common sense knowledge representation. Different sentiment dimensions of each modality are independently encoded to obtain latent variables that correspond to each modality and contain multiple sentiment dimensions. The sentiment dimensions of the latent variables of different modalities are aligned by minimizing the KL divergence between modalities. For the latent variables of the aligned sentiment dimension, resampling is performed on each sentiment dimension to obtain a feature vector, resulting in an aligned reconstructed representation; The reconstructed representation of each aligned modality is used to perform adaptive modality fusion using the Transformer model. By learning the fusion weights between modalities, a unified fusion feature is obtained for sentiment classification to obtain the sentiment recognition result.

2. The multimodal dialogue emotion recognition method based on fine-grained modal alignment as described in claim 1, characterized in that: Based on the features of the input text, relevant events are extracted and transmitted to the trained COMET model to output the corresponding common sense knowledge.

3. The multimodal dialogue emotion recognition method based on fine-grained modal alignment as described in claim 1, characterized in that: For VAE models, the optimization objectives set during the training process include one or more of the following: The VAE model employs variational inference during training to minimize reconstruction error and KL divergence, constructing a variational lower bound and maximizing the variational lower bound to enable the VAE model to better fit the data and approach the true prior distribution. To improve the accuracy of the sentiment dimension output of the encoder of the VAE model, sentiment labels are set for each sentiment dimension, and mean squared error is used as the loss function for training. An optimization objective is constructed to minimize the mutual information among latent variables in order to reduce the dependencies between latent variables output by the VAE model encoder; The KL divergence of all modes of the latent variables output by the VAE model is calculated to minimize the KL divergence between modes and construct an alignment loss to minimize the differences between different modes.

4. The multimodal dialogue emotion recognition method based on fine-grained modal alignment as described in claim 1, characterized in that: The method of using the Transformer model for adaptive modality fusion, which learns the fusion weights between modalities to obtain unified fusion features for sentiment classification and thus obtains sentiment recognition results, includes the following steps: For each modality m, the reconstructed representation of modality m is constructed as a query, and the reconstructed representations of all modalities are constructed as values ​​and keys. Attention is then calculated to obtain the attention-processed features corresponding to each modality. Based on the obtained fusion features, the learnable first weight is calculated, and the attention-processed features corresponding to each modality are fused to obtain the fused features for each modality. Based on the features obtained after modal fusion, a learnable second weight corresponding to each modality is calculated, and multimodal fusion is performed.

5. The multimodal dialogue emotion recognition method based on fine-grained modal alignment as described in claim 1, characterized in that: When training the Transformer model, the KL divergence minimization method is used to maximize the mutual information between the fusion result and the input modality, so that the fusion result is close to the single-modality prediction result.

6. The multimodal dialogue emotion recognition method based on fine-grained modal alignment as described in claim 1, characterized in that: It also includes the constructed sentiment prediction model, which includes the COMET model, the VAE model set according to the number of modalities, the reconstruction module, and the Transformer model.

7. The multimodal dialogue emotion recognition method based on fine-grained modal alignment as described in claim 6, characterized in that: The sentiment prediction model training process employs a phased training method: In the first stage, within the set number of training iterations e, other parameters in the overall sentiment prediction model are frozen, and the encoder and decoder of the VAE model are trained. In the second stage, after exceeding the set number of training iterations e, the parameters of the VAE model are frozen, and the parameters of the Transformer model and other parameters are trained.

8. A multimodal dialogue emotion recognition system based on fine-grained modal alignment, characterized in that, include: The common sense knowledge generation module is configured to acquire multimodal information and generate common sense knowledge representations related to the input text in the multimodal information; The sentiment dimension alignment module is configured to set up independent VAE models for each modality information and common sense knowledge representation, independently encode different sentiment dimensions for each modality, and obtain latent variables that contain multiple sentiment dimensions corresponding to each modality. The sentiment dimensions of the latent variables of different modalities are aligned by minimizing the KL divergence between modalities. The reconstruction module is configured to resample the latent variables of the aligned sentiment dimension for each sentiment dimension to obtain a feature vector, thus obtaining the aligned reconstructed representation. ; The adaptive modality fusion module is configured to reconstruct the representation of each aligned modality. Adaptive modality fusion is performed using the Transformer model, by learning the fusion weights between modalities. By obtaining unified fusion features, emotion classification is performed to obtain emotion recognition results.

9. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps in the multimodal dialogue emotion recognition method based on fine-grained modal alignment as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps in the multimodal dialogue emotion recognition method based on fine-grained modal alignment as described in any one of claims 1-7.

Images