Emotion recognition method and device based on multi-modal consensus and diversity decoupling

Abstract

The invention relates to an emotion recognition method and device based on multi-modal consensus and diversity decoupling. The method comprises the following steps: firstly, collecting multi-modal input data including language, vision and audio signals and carrying out corresponding preprocessing; then, constructing a multi-modal consensus and diversity decoupling emotion recognition model which comprises a multi-modal decoupling coding module, a prototype-Gram unification module, a feature enhancement module, a diversity classification module and an emotion prediction head; then, inputting the preprocessed multi-modal input data into the multi-modal consensus and diversity decoupling emotion recognition model, and performing model training optimization based on a total loss function formed by emotion prediction task loss, decoupling loss, unified target loss and diversity loss; and finally, inputting the multi-modal data to be recognized into the trained multi-modal consensus and diversity decoupling emotion recognition model, and outputting an emotion recognition result. And the accuracy, robustness and interpretability of the multi-modal emotion recognition system are improved.

Description

Technical Field

[0001] This application relates to the field of multimodal emotion computing and semantic understanding technology, and in particular to an emotion recognition method and apparatus based on multimodal consensus and diversity decoupling. Background Technology

[0002] Against the backdrop of continuous breakthroughs in artificial intelligence and multimodal semantic understanding technologies, emotion recognition has become a core research direction in the construction of human-computer interaction systems and intelligent data analysis. Its technological achievements have been widely applied in many fields such as intelligent customer service, psychological assessment, and intelligent cockpits, and are of great significance for improving the naturalness of human-computer interaction and the accuracy of intelligent analysis.

[0003] Traditional emotion recognition technologies often rely on single-modal data for identification, such as depending solely on intonation features in speech signals or on emotional vocabulary in text content. However, in real-world applications, complex situations often arise, such as multi-modal interference (e.g., background noise in public environments), modal information occlusion (e.g., obscured facial areas in videos), or partial modality loss (e.g., only audio data is available without corresponding visual data). In these cases, the recognition performance and environmental robustness of single-modal emotion recognition methods significantly decrease, making it difficult to meet the accuracy requirements of practical applications.

[0004] To address the inherent limitations of single-modal recognition methods and improve the accuracy and stability of emotion recognition, multimodal emotion recognition (MER) technology has increasingly become a research hotspot in this field. The core idea of ​​this technology lies in integrating multimodal information such as language (e.g., text content), vision (e.g., facial expressions, body language), and audio (e.g., tone of voice, speech rate) to jointly characterize human emotional states from multiple dimensions. Theoretically, this can effectively improve the perceptual ability and scene generalization performance of emotion recognition models.

[0005] Although multimodal emotion recognition technology has significant advantages over traditional single-modal methods, existing multimodal emotion recognition methods still have many technical problems that urgently need to be solved, specifically in the following aspects: First, most mainstream multimodal emotion recognition methods rely on modal data alignment or direct feature concatenation as their core techniques. These methods often overlook the inherent semantic differences between different modalities and the significant contribution of each modality's independent features to emotion recognition. This approach can easily lead to over-alignment of modalities, causing the fused multimodal features to lose their unique emotional representation cues within a homogenized feature space, thus reducing the model's ability to distinguish complex emotions.

[0006] Second, existing multimodal emotion recognition models generally lack a systematic modeling mechanism for "consensus information" and "diversity information" among modalities. In achieving multimodal collaborative work, these models struggle to effectively balance the extraction of common features among modalities with the preservation of their differential expressions, leading to a "risk of shared error"—that is, when information from one modality is distorted due to noise interference or missing due to equipment failure, it can cause synchronous deviations in the recognition results of other modalities, severely impacting the system's reliability.

[0007] Third, some existing methods only perform shallow fusion at the feature level, failing to delve into the geometric structure and semantic space levels to achieve precise alignment of multimodal sentiment distribution. This shallow fusion approach prevents the model from establishing a clear correlation between each modality of data and sentiment categories. This not only results in a lack of interpretability in the model's sentiment recognition results and difficulty in tracing the basis for recognition, but also makes the model's generalization ability severely insufficient when faced with unfamiliar new scenarios or types of data, limiting the practical application scope of the technology.

[0008] Therefore, there is an urgent need in related technologies for a way to improve the accuracy, robustness and interpretability of multimodal emotion recognition systems. Summary of the Invention

[0009] Therefore, it is necessary to provide an emotion recognition method and apparatus based on multimodal consensus and diversity decoupling that can improve the accuracy, robustness and interpretability of multimodal emotion recognition systems, in order to address the above-mentioned technical problems.

[0010] Firstly, this application provides an emotion recognition method based on multimodal consensus and diversity decoupling. The method includes: Collect multimodal input data including speech, visual and audio signals and perform corresponding preprocessing; A multimodal consensus and diversity decoupling emotion recognition model is constructed, including a multimodal decoupling coding module, a prototype-Gram unification module, a feature enhancement module, a diversity classification module, and an emotion prediction head; The preprocessed multimodal input data is input into the multimodal consensus and diversity decoupling emotion recognition model, and the model is trained and optimized based on the total loss function composed of emotion prediction task loss, decoupling loss, unified goal loss and diversity loss. The multimodal data to be identified is input into the trained multimodal consensus and diversity decoupled emotion recognition model, and the emotion recognition result is output.

[0011] Optionally, in one embodiment of this application, the corresponding preprocessing includes: The language signal is segmented into words, and the text is converted into a high-dimensional semantic feature vector using a pre-trained language encoder to obtain a language feature sequence. The visual signal is processed frame by frame to perform face detection and alignment, extract facial region or action unit features, and obtain a visual feature sequence. The audio signal is framed, windowed, and analyzed in the frequency domain to extract acoustic parameter vectors and obtain audio feature sequences. The language feature sequence, visual feature sequence, and audio feature sequence are time-aligned and normalized to obtain the aligned multimodal feature sequence.

[0012] Optionally, in one embodiment of this application, the multimodal decoupling coding module includes: The aligned multimodal feature sequences are then subjected to shared encoding and modality-specific encoding to obtain shared features and modality-specific features, as follows:

[0013]

[0014] in, For 1D convolutional shared encoder, A proprietary 1D convolutional encoder. Given the input feature sequence, L For language modality, V For visual modality, A For audio modality, For shared features, These are modal-specific features; Apply orthogonal constraint loss between shared features and modality-specific features; The shared features and modality-specific features are concatenated and then reconstructed to obtain the reconstructed features, and the reconstruction loss is calculated. The reconstructed features are then input again into the 1D convolutional private encoder to obtain re-encoded features, and the cyclic constraint loss is calculated. The decoupling loss is calculated based on the orthogonal constraint loss, reconstruction loss, and cyclic constraint loss.

[0015] Optionally, in one embodiment of this application, the prototype-Gram unification module includes a Gram volume constraint submodule and a prototype anchoring submodule, wherein: The Gram volume constraint submodule is used to normalize the shared features of the three modalities and construct Gram volume constraints. The prototype anchoring submodule is used to build a set of category prototypes for each emotion category and apply prototype constraints to the normalized shared features; The unified target loss is calculated based on the loss due to the Gram volume constraint and the loss due to the prototype constraint.

[0016] Optionally, in one embodiment of this application, the feature enhancement module includes a shared feature enhancement submodule and a unique feature enhancement submodule, wherein: The shared feature enhancement submodule is used to perform sequence modeling based on self-attention on the shared features of the three modalities using a single-modal Transformer encoder to obtain the enhanced shared features. The single-modal Transformer encoder is composed of several layers stacked based on multi-head self-attention and feedforward networks. The feature enhancement submodule is used to perform joint modeling and cross-modal interactive attention calculation on the modal-specific features of the three modalities using a multimodal Transformer encoder to obtain an enhanced multimodal feature sequence. The multimodal Transformer encoder consists of several layers stacked based on multi-head self-attention and feedforward networks. In the attention calculation, features from language, vision and audio modalities are used simultaneously as queries, keys and values.

[0017] Optionally, in one embodiment of this application, the diversity classification module includes a modality classifier, a confidence product suppression submodule, and a confidence interval constraint submodule, wherein: The enhanced multimodal feature sequence is input into the corresponding modality classifier to obtain the predicted probability distribution on the set of sentiment categories; The confidence product suppression submodule is used to penalize the confidence product of different modalities on non-target categories in order to suppress consistent high-confidence predictions of multimodalities on wrong categories; The confidence interval constraint submodule is used to apply an interval constraint to the difference between the target class confidence and the highest non-target class confidence for each modality; The diversity loss is calculated based on the cross-entropy loss, confidence product suppression loss, and confidence interval constraint loss of the modality classifier prediction results.

[0018] Optionally, in one embodiment of this application, the loss of the sentiment prediction task is measured using a cross-entropy function to measure the difference between the model output and the true sentiment label, expressed as:

[0019] in, For batch size, For the number of emotion categories, This represents the unique heat representation of the true label. To predict probabilities.

[0020] Secondly, this application also provides an emotion recognition device based on multimodal consensus and diversity decoupling. The device includes: The multimodal data acquisition module is used to acquire multimodal input data, including speech, visual and audio signals, and perform corresponding preprocessing. A multimodal consensus and diversity decoupling emotion recognition model construction module is used to build a multimodal consensus and diversity decoupling emotion recognition model, including a multimodal decoupling encoding module, a prototype-Gram unification module, a feature enhancement module, a diversity classification module, and a sentiment prediction head; The multimodal consensus and diversity decoupling emotion recognition model training module is used to input the preprocessed multimodal input data into the multimodal consensus and diversity decoupling emotion recognition model, and to perform model training optimization based on the total loss function composed of emotion prediction task loss, decoupling loss, unified target loss and diversity loss; The emotion recognition module is used to input the multimodal data to be recognized into the trained multimodal consensus and diversity decoupling emotion recognition model, and output the emotion recognition result.

[0021] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the steps of the methods described in the various embodiments above.

[0022] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the methods described in the various embodiments above.

[0023] The aforementioned emotion recognition method and apparatus based on multimodal consensus and diversity decoupling first collects multimodal input data including language, visual, and audio signals and performs corresponding preprocessing. Then, a multimodal consensus and diversity decoupling emotion recognition model is constructed, including a multimodal decoupling encoding module, a prototype-Gram unification module, a feature enhancement module, a diversity classification module, and an emotion prediction head. Next, the preprocessed multimodal input data is input into the multimodal consensus and diversity decoupling emotion recognition model, and the model is trained and optimized based on a total loss function composed of emotion prediction task loss, decoupling loss, unification target loss, and diversity loss. Finally, the multimodal data to be recognized is input into the trained multimodal consensus and diversity decoupling emotion recognition model, and the emotion recognition result is output. In other words, a multimodal decoupling coding module decomposes language, visual, and audio signals into shared features and modality-specific features. Decoupling loss, reconstruction constraints, and cyclic constraints are introduced between the shared and specific branches. This reduces redundant correlations and over-alignment between different modalities, preserving independent emotional cues for each modality. Furthermore, it ensures that the original semantic information can still be reconstructed after decoupling, thus maintaining stable emotional representation capabilities and high recognition performance even when a single modality is affected by noise or partially missing data. The proposed Prototype-Gram Unification (PGU) module normalizes the shared features of the three modalities and constructs Gram volume constraints to achieve consistent alignment in geometric space. Simultaneously, by combining prototype anchoring of category prototypes and exponential moving average updates, a stable multimodal emotional consensus representation is constructed in the semantic space. This effectively alleviates the semantic shift problem caused by differences in the distribution of different modalities and improves the consistency and interpretability of cross-modal emotional representation. This paper introduces a diversity classification (DC) module into the modality-specific branch. A confidence product suppression (CPS) term is used to suppress consistent high-confidence predictions across different modalities on the wrong category, while a confidence margin constraint (HMC) term enhances the confidence margin between the target and non-target categories for each modality. This reduces multimodal co-error phenomena and enhances the discriminative power of single-modal predictions, enabling the model to adaptively prioritize more reliable modal information even in cases of modality quality imbalance or contradictory modal information. A Transformer-based shared feature and modality-specific feature enhancement submodule is employed, combined with an end-to-end multi-objective joint optimization strategy. Task loss, decoupling loss, unification loss, and diversity loss are trained collaboratively, achieving multimodal consensus modeling and modality diversity preservation within a unified framework. This method is compatible with existing text, audio, and video acquisition and encoding modules, possesses good modularity and scalability, and can be flexibly deployed in various application scenarios such as emotional human-computer interaction, intelligent customer service, online education, and virtual assistants, demonstrating high engineering application value and promising prospects for widespread adoption. Attached Figure Description

[0024] Figure 1This is a flowchart illustrating an emotion recognition method based on multimodal consensus and diversity decoupling in one embodiment. Figure 2 This is a schematic diagram of the structure of a multimodal consensus and diversity decoupling emotion recognition model in one embodiment; Figure 3 This is a schematic diagram of a prototype-Gram unified module in one embodiment; Figure 4 This is a schematic diagram of a diversity classification module in one embodiment; Figure 5 This is a structural block diagram of an emotion recognition device based on multimodal consensus and diversity decoupling in one embodiment; Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] In one embodiment, such as Figure 1 As shown, an emotion recognition method based on multimodal consensus and diversity decoupling is provided, including the following steps: S101: Acquire multimodal input data including speech, visual and audio signals and perform corresponding preprocessing.

[0027] In this embodiment, firstly, multimodal input data, including language, visual and audio signals, is collected, and corresponding modal preprocessing is performed to obtain multimodal feature sequences for model training.

[0028] Specifically, in one embodiment of this application, the corresponding preprocessing includes: S201: Segment the language signal into words, and use a pre-trained language encoder to convert the text into a high-dimensional semantic feature vector to obtain a language feature sequence.

[0029] S202: Perform video frame-by-frame processing on the visual signal, perform face detection and alignment, extract facial region or action unit features, and obtain a visual feature sequence.

[0030] S203: Perform frame segmentation, windowing, and frequency domain analysis on the audio signal to extract acoustic parameter vectors and obtain audio feature sequences.

[0031] S204: Perform time alignment and normalization on the language feature sequence, visual feature sequence, and audio feature sequence to obtain the aligned multimodal feature sequence.

[0032] In one embodiment of this application, raw language signals are acquired using a language acquisition device and preprocessed, including word segmentation and encoding. Word segmentation involves tokenizing the raw text signal, and encoding involves converting the text into a high-dimensional semantic feature vector using a pre-trained language encoder (such as BERT). Through these processing steps, a language feature sequence after word segmentation and encoding is obtained. This is used for subsequent training.

[0033] Synchronous facial or lip-sync video frame sequences are captured using video capture equipment; each video frame is processed frame by frame, first performing face detection and alignment to ensure standardization of facial regions; then, facial region or motion unit features are extracted to obtain the visual feature representation of each frame. These feature sequences are then used for training and inference, preserving key facial dynamic information.

[0034] Raw audio signals are acquired using audio acquisition equipment and processed through operations including framing, windowing, and Fourier transform. These are then analyzed in the frequency domain to extract acoustic parameters (such as Mel-frequency coefficients). Through an audio preprocessing workflow, an audio feature sequence with temporal structure and frequency domain distribution characteristics is obtained. This is used for subsequent audio analysis and training.

[0035] The obtained language, visual, and audio signal feature sequences are subjected to time alignment and normalization. Specifically, firstly, the timestamps of each modality are aligned so that the feature sequences of each modality are matched at the same time step. For the audio and visual modalities, interpolation methods are used to resample the signals to align them with the timestamps of the language modality. Then, the feature sequences of each modality are normalized to eliminate scale differences between different modalities, ensuring that each modality is trained and inferred at the same scale.

[0036] S102: Construct a multimodal consensus and diversity decoupling sentiment recognition model, including a multimodal decoupling coding module, a prototype-Gram unification module, a feature enhancement module, a diversity classification module, and a sentiment prediction head.

[0037] In this embodiment of the application, a construction is performed as follows: Figure 2The multimodal consensus and diversity decoupling emotion recognition model shown includes a multimodal decoupling encoding module, a prototype-Gram unification module, a feature enhancement module, a diversity classification module, and an emotion prediction head. The multimodal decoupling encoding module encodes each modality feature sequence using a shared encoder and a modality-specific encoder, respectively, to obtain corresponding shared features and modality-specific features. The shared features are input to the prototype-Gram unification (PGU) module, which constructs a Gram matrix for the shared features of the three modalities and calculates a volume metric, achieving geometric consistency constraints by minimizing the three-modal Gram volume. In the PGU module, a category prototype set is constructed for each emotion category, and the shared features are aligned with the corresponding category prototypes to achieve prototype anchoring of the shared features. Simultaneously, the shared features and modality-specific features are input to the feature enhancement module for feature enhancement. The enhanced modality-specific features are input to the diversity classification (DC) module, which outputs a category confidence score for each modality through a modality classifier. The enhanced shared features and enhanced modality-specific features are fused to obtain a fused representation, which is then input to the emotion prediction head to output the emotion recognition result.

[0038] Specifically, in one embodiment of this application, the multimodal decoupling coding module includes: S301: Perform shared encoding and modality-specific encoding on the aligned multimodal feature sequences to obtain shared features and modality-specific features, represented as follows:

[0039]

[0040] in, For 1D convolutional shared encoder, A proprietary 1D convolutional encoder. Given the input feature sequence, L For language modality, V For visual modality, A For audio modality, For shared features, These are modal-specific features.

[0041] S302: Apply orthogonal constraint loss between shared features and modality-specific features.

[0042] S303: Concatenate the shared features with the modality-specific features and then reconstruct the features to obtain the reconstructed features, and calculate the reconstruction loss.

[0043] S304: Input the reconstructed features back into the 1D convolutional private encoder to obtain re-encoded features, and calculate the cyclic constraint loss.

[0044] S305: Calculate the decoupling loss based on the orthogonal constraint loss, reconstruction loss, and cyclic constraint loss.

[0045] In one embodiment of this application, the aligned multimodal feature sequences are shared-encoded: for each modality The corresponding input feature sequence Input 1D convolutional shared encoder Shared features are obtained:

[0046] Wherein, the shared encoder It is a 1D convolutional neural network that extracts common features from language, visual, and audio signals by performing convolution operations and nonlinear transformations along the time dimension.

[0047] Modality-private encoding is performed on the aligned multimodal feature sequences: for each modality The corresponding input feature sequence Input 1D convolutional modal proprietary encoder Modal-specific features are obtained:

[0048] Among them, the modal private encoder It is a 1D convolutional neural network, where each encoder processes only the feature sequence of a single modality to extract the unique information of that modality.

[0049] An orthogonal constraint loss is applied between shared features and modality-specific features to constrain their correlation:

[0050] in It represents the cosine similarity between shared features and modality-specific features.

[0051] A reconstruction constraint loss is introduced to preserve input information: for each mode Configure the corresponding decoder The shared features and modality-specific features are concatenated and then reconstructed to obtain the reconstructed features:

[0052] Reconstruction loss is defined as:

[0053] in It is the Frobenius norm.

[0054] A cyclic constraint loss is introduced to constrain the stability of modality-specific features during reconstruction and re-encoding: the reconstructed features are then input again into the corresponding modality-specific encoder. The re-encoded features are obtained:

[0055] Cyclic constraint loss is defined as:

[0056] Multimodal decoupling coding module The overall decoupling loss is:

[0057] in, The weighting coefficients for orthogonal constraint loss.

[0058] In one embodiment of this application, the prototype-Gram unification module includes a Gram volume constraint submodule and a prototype anchoring submodule, wherein: S401: Gram volume constraint submodule, used to normalize the shared features of the three modes and construct Gram volume constraints.

[0059] S402: The prototype anchoring submodule is used to build a set of category prototypes for each emotion category and to impose prototype constraints on the normalized shared features.

[0060] S403: Calculate the unified target loss based on the loss of the Gram volume constraint and the loss of the prototype constraint.

[0061] In one embodiment of this application, such as Figure 3 As shown, the Gram volume constraint submodule is used to normalize the shared features of the three modes and construct Gram volume constraints; specifically, for the first... Shared features of each sample across language, visual, and audio modalities , , First, Normalization yields: , ,

[0062] in, Representing language, visual, and audio modalities respectively. For the first Each sample in modality Shared features under normalized vectors. Constructing a three-modal Gram matrix And using its determinant as a volume measure, the first... The trimodal shared feature volume of each sample is:

[0063] The Gram volume constraint loss is obtained by averaging the samples within the batch.

[0064] in, This represents the number of samples in the batch.

[0065] The prototype anchoring submodule is used to construct a set of category prototypes for each emotion category and apply prototype constraints to the normalized shared features to maintain the separability between emotion categories and prevent the collapse of shared features. Specifically, it includes: For each emotion category Pre-set several prototype vectors This forms a category prototype library; for the first Normalized shared features of each sample and its category labels Define the prototype anchoring loss as:

[0066] in, For the number of emotion categories, The number of prototypes for each category. For negative category weights, Indicates the first Each sample shares any one of the three modal normalization features.

[0067] For samples Matching prototype The exponential moving average (EMA) method is used for online updates.

[0068] in, The momentum coefficient, This represents the prototype index that is closest to the mean of the normalized shared features of the current sample.

[0069] By minimizing the This allows the shared features of the three modalities within the same emotion category to aggregate within the prototype space, while maintaining a distance from prototypes of other categories, in accordance with the definition in claim 3. Together, we will achieve the unified optimization goal of the PGU module. .

[0070] The unified objective loss of the PGU module is defined as:

[0071] in, For the loss of the Gram volume-constrained submodule, For the loss of the prototype anchoring submodule, and These are the corresponding weighting coefficients.

[0072] In one embodiment of this application, the feature enhancement module includes a shared feature enhancement submodule and a unique feature enhancement submodule, wherein: S501: Shared feature enhancement submodule, used to perform self-attention-based sequence modeling on the shared features of the three modalities using a single-modal Transformer encoder to obtain enhanced shared features. The single-modal Transformer encoder is composed of several layers stacked based on multi-head self-attention and feedforward networks.

[0073] S502: Specific Feature Enhancement Submodule, used to perform joint modeling and cross-modal interactive attention calculation on the modal-specific features of the three modalities using a multimodal Transformer encoder to obtain an enhanced multimodal specific feature sequence. The multimodal Transformer encoder consists of several layers stacked based on multi-head self-attention and feedforward networks. In the attention calculation, features from language, vision and audio modalities are used simultaneously as queries, keys and values.

[0074] In one embodiment of this application, a shared feature enhancement submodule is used to perform self-attention-based sequence modeling on the shared features of the three modalities; for each modality Shared feature sequences Input the corresponding single-mode Transformer encoder The enhanced shared features are obtained:

[0075] Among them, the single-mode Transformer encoder It consists of several stacked layers based on Multi-Head Self-Attention (MHSA) and Feed-Forward Network (FFN); in each layer, for the input sequence representation The computational form of the self-attention sublayer is as follows:

[0076] in, , , , For learnable weight matrix, This is a feature dimension used to model the dependencies between different time steps within the same modality.

[0077] The modality-specific feature enhancement submodule is used to jointly model the modality-specific features of the three modalities and calculate cross-modal interactive attention; the obtained modality-specific features are aligned along the time dimension and concatenated along the sequence dimension to form a multimodal input sequence.

[0078] And input it into the multimodal Transformer encoder. The enhanced multimodal-specific feature sequence is obtained:

[0079] Among them, the multimodal Transformer encoder It consists of several stacked layers based on multi-head attention (MHA) and feedforward networks, and uses features from language, vision, and audio modalities as queries simultaneously in attention computation. ),key( ) and value ( To achieve cross-modal interactive information modeling; the output sequence The segments are cut according to the original splicing order, and the enhanced modality-specific features corresponding to the three modalities are obtained respectively: , , This is used in the subsequent diversity classification module.

[0080] In one embodiment of this application, the diversity classification module includes a modality classifier, a confidence product suppression submodule, and a confidence interval constraint submodule, wherein: S601: Input the enhanced multimodal feature sequence into the corresponding modality classifier to obtain the predicted probability distribution on the sentiment category set.

[0081] S602: The confidence product suppression submodule is used to penalize the confidence product of different modalities on non-target categories in order to suppress consistent high-confidence predictions of multimodalities on wrong categories.

[0082] S603: The confidence interval constraint submodule is used to apply an interval constraint to the difference between the target class confidence and the highest non-target class confidence for each modality.

[0083] S604: Calculate diversity loss based on cross-entropy loss, confidence product suppression loss, and confidence interval constraint loss from the prediction results of the modality classifier.

[0084] In one embodiment of this application, such as Figure 4 As shown, the enhanced modality-specific features are input into the corresponding modality classifier to obtain the predicted probability distribution on the set of sentiment categories:

[0085] in, For modality The logits output by the classifier, This indicates that the modality relates to the emotion category. The prediction confidence level.

[0086] The Confidence Product Suppression (CPS) submodule penalizes the confidence product of different modalities on non-target categories to suppress consistent high-confidence predictions by multimodalities on incorrect categories; for the first in a batch... One sample, labeled as CPS loss is defined as

[0087] in, For batch size, Indicates the first Each sample in modality Top category confidence level These are the weighting coefficients for the three-modal product terms.

[0088] The Confidence Interval Constraint (HMC) submodule applies an interval constraint to the difference between the target class confidence and the highest non-target class confidence for each modality, thereby improving the discriminative power of single-modality predictions; for each modality and the first in the batch A sample, recorded Confidence level for the target category The highest non-target class confidence. The HMC loss is defined as:

[0089] in, This is the preset confidence interval threshold.

[0090] Based on this, the diversity loss of the diversity classification (DC) module is defined as...

[0091] in, The cross-entropy loss is calculated based on the prediction results of the three modality classifiers. and These are the weighting coefficients for CPS loss and HMC loss, respectively.

[0092] Enhanced shared features Enhanced modal-specific features The features are concatenated and fused along the feature dimension to obtain the final fused representation. The fused representation is then input into the sentiment prediction head. Output the predicted probability distribution: .

[0093] S103: Input the preprocessed multimodal input data into the multimodal consensus and diversity decoupling emotion recognition model, and perform model training and optimization based on the total loss function composed of emotion prediction task loss, decoupling loss, unified goal loss and diversity loss.

[0094] In one embodiment of this application, the loss of the sentiment prediction task is measured using a cross-entropy function to measure the difference between the model output and the true sentiment label, expressed as:

[0095] in, For batch size, For the number of emotion categories, This represents the unique heat representation of the true label. To predict probabilities.

[0096] In one embodiment of this application, the total loss function for model training is defined as:

[0097] in, It serves as the decoupling loss for the multimodal decoupling coding module, used to constrain the independence between shared features and modality-specific features, and maintains the integrity of input information through orthogonal constraints, reconstruction constraints, and cyclic constraints; To achieve a unified objective loss for the PGU module, the Gram volume loss and prototype anchoring loss are minimized to realize a unified constraint on multimodal shared features at the geometric and semantic prototype levels. To mitigate the diversity loss of the DC module, a combined confidence product suppression (CPS) term and confidence interval constraint (HMC) term are used to suppress common confidence across different modalities in error categories and enhance the discriminative power of single-modal predictions. , , The weighting coefficients for each loss term are used to balance the contributions of decoupling constraints, consensus constraints, and diversity constraints to the overall optimization, thereby achieving synergistic optimization that integrates and unifies multimodal features and complements their differences.

[0098] By minimizing the total loss function during training The model parameters are iteratively updated, and after training, an emotion recognition model based on multimodal consensus and diversity decoupling is obtained. This model can perform end-to-end emotion recognition and prediction on the input language, visual and audio signals during the inference stage.

[0099] S104: Input the multimodal data to be identified into the trained multimodal consensus and diversity decoupling emotion recognition model, and output the emotion recognition result.

[0100] In the aforementioned emotion recognition method based on multimodal consensus and diversity decoupling, firstly, multimodal input data including language, visual, and audio signals are collected and preprocessed accordingly; then, a multimodal consensus and diversity decoupling emotion recognition model is constructed, including a multimodal decoupling encoding module, a prototype-Gram unification module, a feature enhancement module, a diversity classification module, and an emotion prediction head; next, the preprocessed multimodal input data is input into the multimodal consensus and diversity decoupling emotion recognition model, and the model is trained and optimized based on a total loss function composed of emotion prediction task loss, decoupling loss, unification target loss, and diversity loss; finally, the multimodal data to be recognized is input into the trained multimodal consensus and diversity decoupling emotion recognition model, and the emotion recognition result is output. In other words, a multimodal decoupling coding module decomposes language, visual, and audio signals into shared features and modality-specific features. Decoupling loss, reconstruction constraints, and cyclic constraints are introduced between the shared and specific branches. This reduces redundant correlations and over-alignment between different modalities, preserving independent emotional cues for each modality. Furthermore, it ensures that the original semantic information can still be reconstructed after decoupling, thus maintaining stable emotional representation capabilities and high recognition performance even when a single modality is affected by noise or partially missing data. The proposed Prototype-Gram Unification (PGU) module normalizes the shared features of the three modalities and constructs Gram volume constraints to achieve consistent alignment in geometric space. Simultaneously, by combining prototype anchoring of category prototypes and exponential moving average updates, a stable multimodal emotional consensus representation is constructed in the semantic space. This effectively alleviates the semantic shift problem caused by differences in the distribution of different modalities and improves the consistency and interpretability of cross-modal emotional representation. This paper introduces a diversity classification (DC) module into the modality-specific branch. A confidence product suppression (CPS) term is used to suppress consistent high-confidence predictions across different modalities on the wrong category, while a confidence margin constraint (HMC) term enhances the confidence margin between the target and non-target categories for each modality. This reduces multimodal co-error phenomena and enhances the discriminative power of single-modal predictions, enabling the model to adaptively prioritize more reliable modal information even in cases of modality quality imbalance or contradictory modal information. A Transformer-based shared feature and modality-specific feature enhancement submodule is employed, combined with an end-to-end multi-objective joint optimization strategy. Task loss, decoupling loss, unification loss, and diversity loss are trained collaboratively, achieving multimodal consensus modeling and modality diversity preservation within a unified framework. This method is compatible with existing text, audio, and video acquisition and encoding modules, possesses good modularity and scalability, and can be flexibly deployed in various application scenarios such as emotional human-computer interaction, intelligent customer service, online education, and virtual assistants, demonstrating high engineering application value and promising prospects for widespread adoption.

[0101] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0102] Based on the same inventive concept, this application also provides an emotion recognition device based on multimodal consensus and decoupling of diversity for implementing the emotion recognition method based on multimodal consensus and decoupling of diversity described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the emotion recognition device based on multimodal consensus and decoupling of diversity provided below can be found in the limitations of the emotion recognition method based on multimodal consensus and decoupling of diversity described above, and will not be repeated here.

[0103] In one embodiment, such as Figure 5 As shown, an emotion recognition device 500 based on multimodal consensus and diversity decoupling is provided, including: a multimodal data acquisition module 501, a multimodal consensus and diversity decoupling emotion recognition model construction module 503, a multimodal consensus and diversity decoupling emotion recognition model training module 505, and an emotion recognition module 507, wherein: The multimodal data acquisition module 501 is used to acquire multimodal input data including language, visual and audio signals and perform corresponding preprocessing.

[0104] The multimodal consensus and diversity decoupling emotion recognition model construction module 503 is used to construct a multimodal consensus and diversity decoupling emotion recognition model, including a multimodal decoupling encoding module, a prototype-Gram unification module, a feature enhancement module, a diversity classification module, and a sentiment prediction head.

[0105] The multimodal consensus and diversity decoupling emotion recognition model training module 505 is used to input the preprocessed multimodal input data into the multimodal consensus and diversity decoupling emotion recognition model, and to perform model training and optimization based on the total loss function composed of emotion prediction task loss, decoupling loss, unified target loss and diversity loss.

[0106] The emotion recognition module 507 is used to input the multimodal data to be recognized into the trained multimodal consensus and diversity decoupling emotion recognition model, and output the emotion recognition result.

[0107] In one embodiment of this application, the corresponding preprocessing includes: The language signal is segmented into words, and the text is converted into a high-dimensional semantic feature vector using a pre-trained language encoder to obtain a language feature sequence. The visual signal is processed frame by frame to perform face detection and alignment, extract facial region or action unit features, and obtain a visual feature sequence. The audio signal is framed, windowed, and analyzed in the frequency domain to extract acoustic parameter vectors and obtain audio feature sequences. The language feature sequence, visual feature sequence, and audio feature sequence are time-aligned and normalized to obtain the aligned multimodal feature sequence.

[0108] In one embodiment of this application, the multimodal decoupling coding module includes: The aligned multimodal feature sequences are then subjected to shared encoding and modality-specific encoding to obtain shared features and modality-specific features, as follows:

[0109]

[0110] in, For 1D convolutional shared encoder, A proprietary 1D convolutional encoder. For the input feature sequence, L For language modality, V For visual modality, A For audio modality, For shared features, These are modal-specific features; Apply orthogonal constraint loss between shared features and modality-specific features; The shared features and modality-specific features are concatenated and then reconstructed to obtain the reconstructed features, and the reconstruction loss is calculated. The reconstructed features are then input again into the 1D convolutional private encoder to obtain re-encoded features, and the cyclic constraint loss is calculated. The decoupling loss is calculated based on the orthogonal constraint loss, reconstruction loss, and cyclic constraint loss.

[0111] In one embodiment of this application, the prototype-Gram unification module includes a Gram volume constraint submodule and a prototype anchoring submodule, wherein: The Gram volume constraint submodule is used to normalize the shared features of the three modalities and construct Gram volume constraints. The prototype anchoring submodule is used to build a set of category prototypes for each emotion category and apply prototype constraints to the normalized shared features; The unified target loss is calculated based on the loss due to the Gram volume constraint and the loss due to the prototype constraint.

[0112] In one embodiment of this application, the feature enhancement module includes a shared feature enhancement submodule and a unique feature enhancement submodule, wherein: The shared feature enhancement submodule is used to perform sequence modeling based on self-attention on the shared features of the three modalities using a single-modal Transformer encoder to obtain the enhanced shared features. The single-modal Transformer encoder is composed of several layers stacked based on multi-head self-attention and feedforward networks. The feature enhancement submodule is used to perform joint modeling and cross-modal interactive attention calculation on the modal-specific features of the three modalities using a multimodal Transformer encoder to obtain an enhanced multimodal feature sequence. The multimodal Transformer encoder consists of several layers stacked based on multi-head self-attention and feedforward networks. In the attention calculation, features from language, vision and audio modalities are used simultaneously as queries, keys and values.

[0113] In one embodiment of this application, the diversity classification module includes a modality classifier, a confidence product suppression submodule, and a confidence interval constraint submodule, wherein: The enhanced multimodal feature sequence is input into the corresponding modality classifier to obtain the predicted probability distribution on the set of sentiment categories; The confidence product suppression submodule is used to penalize the confidence product of different modalities on non-target categories in order to suppress consistent high-confidence predictions of multimodalities on wrong categories; The confidence interval constraint submodule is used to apply an interval constraint to the difference between the target class confidence and the highest non-target class confidence for each modality; The diversity loss is calculated based on the cross-entropy loss, confidence product suppression loss, and confidence interval constraint loss of the modality classifier prediction results.

[0114] In one embodiment of this application, the loss of the sentiment prediction task is measured using a cross-entropy function to measure the difference between the model output and the true sentiment label, expressed as:

[0115] in, For batch size, For the number of emotion categories, This represents the unique heat representation of the true label. To predict probabilities.

[0116] The modules in the aforementioned emotion recognition device based on multimodal consensus and diversity decoupling can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0117] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals (audio acquisition devices for collecting audio signals from the environment, video acquisition devices for collecting video data of a speaker's face or upper body, etc.). Wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an emotion recognition method based on multimodal consensus and diversity decoupling. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0118] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0119] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0120] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0121] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0123] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0124] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An emotion recognition method based on multimodal consensus and diversity decoupling, characterized in that, The method includes: Collect multimodal input data including speech, visual and audio signals and perform corresponding preprocessing; A multimodal consensus and diversity decoupling emotion recognition model is constructed, including a multimodal decoupling coding module, a prototype-Gram unification module, a feature enhancement module, a diversity classification module, and an emotion prediction head; The preprocessed multimodal input data is input into the multimodal consensus and diversity decoupling emotion recognition model, and the model is trained and optimized based on the total loss function composed of emotion prediction task loss, decoupling loss, unified goal loss and diversity loss. The multimodal data to be identified is input into the trained multimodal consensus and diversity decoupled emotion recognition model, and the emotion recognition result is output.

2. The emotion recognition method based on multimodal consensus and diversity decoupling according to claim 1, characterized in that, The corresponding preprocessing includes: The language signal is segmented into words, and the text is converted into a high-dimensional semantic feature vector using a pre-trained language encoder to obtain a language feature sequence. The visual signal is processed frame by frame to perform face detection and alignment, extract facial region or action unit features, and obtain a visual feature sequence. The audio signal is framed, windowed, and analyzed in the frequency domain to extract acoustic parameter vectors and obtain audio feature sequences. The language feature sequence, visual feature sequence, and audio feature sequence are time-aligned and normalized to obtain the aligned multimodal feature sequence.

3. The emotion recognition method based on multimodal consensus and diversity decoupling according to claim 1, characterized in that, The multimodal decoupling coding module includes: The aligned multimodal feature sequences are then subjected to shared encoding and modality-specific encoding to obtain shared features and modality-specific features, as follows: in, For 1D convolutional shared encoder, A proprietary 1D convolutional encoder. Given the input feature sequence, L For language modality, V For visual modality, A For audio modality, For shared features, These are modal-specific features; Apply orthogonal constraint loss between shared features and modality-specific features; The shared features and modality-specific features are concatenated and then reconstructed to obtain the reconstructed features, and the reconstruction loss is calculated. The reconstructed features are then input again into the 1D convolutional private encoder to obtain re-encoded features, and the cyclic constraint loss is calculated. The decoupling loss is calculated based on the orthogonal constraint loss, reconstruction loss, and cyclic constraint loss.

4. The emotion recognition method based on multimodal consensus and diversity decoupling according to claim 1, characterized in that, The prototype-Gram unified module includes a Gram volume constraint submodule and a prototype anchoring submodule, wherein: The Gram volume constraint submodule is used to normalize the shared features of the three modalities and construct Gram volume constraints. The prototype anchoring submodule is used to build a set of category prototypes for each emotion category and apply prototype constraints to the normalized shared features; The unified target loss is calculated based on the loss due to the Gram volume constraint and the loss due to the prototype constraint.

5. The emotion recognition method based on multimodal consensus and diversity decoupling according to claim 1, characterized in that, The feature enhancement module includes a shared feature enhancement submodule and a unique feature enhancement submodule, wherein: The shared feature enhancement submodule is used to perform sequence modeling based on self-attention on the shared features of the three modalities using a single-modal Transformer encoder to obtain the enhanced shared features. The single-modal Transformer encoder is composed of several layers stacked based on multi-head self-attention and feedforward networks. The feature enhancement submodule is used to perform joint modeling and cross-modal interactive attention calculation on the modal-specific features of the three modalities using a multimodal Transformer encoder to obtain an enhanced multimodal feature sequence. The multimodal Transformer encoder consists of several layers stacked based on multi-head self-attention and feedforward networks. In the attention calculation, features from language, vision and audio modalities are used simultaneously as queries, keys and values.

6. The emotion recognition method based on multimodal consensus and diversity decoupling according to claim 1, characterized in that, The diversity classification module includes a modality classifier, a confidence product suppression submodule, and a confidence interval constraint submodule, wherein: The enhanced multimodal feature sequence is input into the corresponding modality classifier to obtain the predicted probability distribution on the set of sentiment categories; The confidence product suppression submodule is used to penalize the confidence product of different modalities on non-target categories in order to suppress consistent high-confidence predictions of multimodalities on wrong categories; The confidence interval constraint submodule is used to apply an interval constraint to the difference between the target class confidence and the highest non-target class confidence for each modality; The diversity loss is calculated based on the cross-entropy loss, confidence product suppression loss, and confidence interval constraint loss of the modality classifier prediction results.

7. The emotion recognition method based on multimodal consensus and diversity decoupling according to claim 1, characterized in that, The loss of the sentiment prediction task is measured using the cross-entropy function to determine the difference between the model output and the true sentiment label, expressed as: in, For batch size, For the number of emotion categories, This represents the unique heat representation of the true label. To predict probabilities.

8. An emotion recognition device based on multimodal consensus and diversity decoupling, characterized in that, The device includes: The multimodal data acquisition module is used to acquire multimodal input data, including speech, visual and audio signals, and perform corresponding preprocessing. A multimodal consensus and diversity decoupling emotion recognition model construction module is used to build a multimodal consensus and diversity decoupling emotion recognition model, including a multimodal decoupling encoding module, a prototype-Gram unification module, a feature enhancement module, a diversity classification module, and a sentiment prediction head; The multimodal consensus and diversity decoupling emotion recognition model training module is used to input the preprocessed multimodal input data into the multimodal consensus and diversity decoupling emotion recognition model, and to perform model training optimization based on the total loss function composed of emotion prediction task loss, decoupling loss, unified target loss and diversity loss; The emotion recognition module is used to input the multimodal data to be recognized into the trained multimodal consensus and diversity decoupling emotion recognition model, and output the emotion recognition result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images