Multi-modal emotion recognition method

Abstract

The invention belongs to the technical field of multi-modal information processing, and particularly relates to a multi-modal emotion recognition method, which can give consideration to modal generality and modal difference at the same time, realizes adaptive fusion between modals, and is suitable for emotion understanding and analysis of video, voice and text multi-source data. Comprising the following steps: 1, preprocessing a CMU-MOSI data set and a CMU-MOSEI data set to generate training data; 2, processing data by adopting a pre-training encoder to obtain a high-quality initial feature sequence; 3, inputting each modal initial feature into a sharing and private coding unit through a feature decoupling module, and respectively obtaining a cross-modal public emotion sharing feature and a modal specific emotion private feature; according to the multi-modal emotion recognition method provided by the invention, the emotion recognition capability is remarkably improved, and effective data support can be provided for scenes such as psychological analysis, medical assistance and human-computer interaction.

Description

Technical Field

[0001] This invention belongs to the field of multimodal information processing technology, specifically a multimodal emotion recognition method. Background Technology

[0002] With the rapid development of artificial intelligence and deep learning technologies, sentiment analysis methods based on multimodal information have gradually become an important research direction in the fields of intelligent perception and human-computer interaction. Multimodal sentiment analysis typically utilizes multi-source information such as text, speech, and vision to identify and understand an individual's emotional state, and has been widely applied in scenarios such as intelligent customer service, public opinion analysis, psychological assessment, educational assistance, and human-computer interaction. However, due to significant differences in information expression forms, time scales, and semantic levels among different modalities, multimodal sentiment analysis still faces many technical challenges in practical applications. Specifically, the text modality focuses on high-level semantics and logical expression, the speech modality reflects more emotional intensity, tone, and rhythm changes, while the visual modality includes intuitive emotional cues such as facial expressions, posture, and scene. These differences result in significant inconsistencies in the distribution space of features from different modalities. Directly splicing or simply fusing features often introduces a large amount of redundant information or even noise interference, thus affecting the accuracy of sentiment recognition. Existing emotion recognition methods still generally suffer from the following shortcomings: On the one hand, in the process of cross-modal alignment, all modal features are often uniformly constrained, failing to distinguish between shared information between modalities and the differential information contained in the modality itself, which can easily lead to the loss of key emotional cues; on the other hand, existing fusion methods mostly adopt fixed weights or static structures, lacking the ability to dynamically adjust the contribution of different modalities according to the specific context, making it difficult to adapt to complex and ever-changing emotional expression scenarios.

[0003] Therefore, we propose a multimodal emotion recognition method that can simultaneously take into account both modal commonalities and modal differences, and achieve adaptive fusion between modalities to solve the above problems. Summary of the Invention To address the shortcomings of existing technologies, this invention proposes a multimodal emotion recognition method that can simultaneously consider modal commonalities and modal differences, achieving adaptive fusion between modalities and solving the problems mentioned in the background technology.

[0004] To achieve the above objectives, the present invention specifically adopts the following technical solution: A multimodal emotion recognition method that can simultaneously consider modal commonalities and modal differences to achieve adaptive fusion between modalities includes the following steps: S1: Preprocess the CMU-MOSI and CMU-MOSEI datasets to generate training data; S2: Use a high-performance pre-trained encoder to process the data to obtain a high-quality initial feature sequence; S3: The modal feature decoupling module is used to input the initial features of each modality into the shared coding unit and the private coding unit to obtain the shared features representing cross-modal common sentiment information and the private features representing modality-specific sentiment information, respectively. S4: Using a cross-modal alignment module, the shared features of speech and video modalities are guided to align to the text semantic space through a contrastive learning mechanism, thereby achieving cross-modal semantic consistency constraints. S5: The gating attention fusion module is used to generate multimodal gating weights based on the aligned shared features, and the private features of each modality are weighted and fused to obtain a unified multimodal sentiment representation; S6: Input the fused multimodal features into the sentiment analysis module and output the corresponding sentiment category or sentiment intensity result.

[0005] Furthermore, in S1, when preprocessing the CMU-MOSI and CMU-MOSEI datasets, temporal interpolation was used to insert the sequence size to a specified length of 8 frames for the video. The speech signal in the video was uniformly resampled to 16 kHz, and the text modality was directly input to the language model. Furthermore, in S2, a high-performance pre-trained encoder is used to obtain a high-quality initial feature sequence. The text modality uses BERT to encode semantic information, the video modality relies on CLIP-ViT to extract visual representations, and the audio modality uses HuBERT to model speech and prosodic features. Furthermore, in S3, a modal feature decoupling module is used to input the initial features of each modality into a shared encoding unit and a private encoding unit, respectively obtaining shared features representing cross-modal common sentiment information and private features representing modality-specific sentiment information. The core idea is to achieve the separation of modal commonality and individuality through shared-private dual-path modeling. Specifically, for the input modal feature sequence... ,in These represent text, audio, and video, respectively. First, through the modal projection head... After unifying the dimensions, the input is fed into two parallel encoders: one via a shared encoder that shares parameters. This yields a shared representation across modalities: Capture common emotional patterns across modalities. On the other hand, through modality-specific proprietary encoders. The private representation is obtained: It preserves the unique rhythm, semantic rhetoric, and visual details of each modality. To ensure decoupling between shared and private subspaces, an orthogonality constraint loss is introduced. The basic idea is to minimize the correlation between the two in the vector space, avoiding information redundancy. Specifically, sequential features are pooled into vectors. , And define the loss as: This constraint drives shared representations to focus on modality-independent public sentiment semantics, while private representations focus on modality-unique expressive features, thereby achieving an effective separation of semantic commonality and modality differences in the feature space.

[0006] Furthermore, in S4, a cross-modal alignment module is employed to guide the shared features of the speech and video modalities to align with the text semantic space through a contrastive learning mechanism, thereby achieving cross-modal semantic consistency constraints. This strategy focuses on the text modality because text is typically more explicit and higher-level in semantic expression, providing a reference for the visual and audio modalities. Therefore, we use text shared representations... Set as a "query" or anchor point, and force the shared representation of visual and audio to converge toward it in the semantic space through contrastive learning.

[0007] Specifically, in a batch of size N, the first... Each sample contains text-shared features. Visual sharing features Features shared with audio We construct positive and negative sample pairs: Positive sample pairs: and That is, text and visual / audio features from the same sample are shared; Negative sample pairs: for any , and This refers to the visual / audio shared features between the text and other samples within the batch.

[0008] During training, the model needs to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs to achieve cross-modal alignment. The alignment loss uses the multimodal extended version of the InfoNCE loss function. Taking text-visual modal alignment as an example, the formula is as follows: in, Represents the cosine similarity function. is a hyperparameter used to adjust the smoothness of the distribution. Similarly, the text-to-audio alignment loss is defined as: The final cross-modal alignment loss is the sum of the two: This loss function achieves semantically consistent cross-modal shared representation learning by bringing positive sample pairs closer together and pushing negative sample pairs further apart in the feature space. Its core advantages are threefold: First, it is asymmetric, using only text as the query anchor point, avoiding the noise accumulation problem that may arise from pairwise alignment between modalities; second, it enhances discriminability by leveraging InfoNCE's softmax positive-negative contrast mechanism, enabling the model to learn more discriminative cross-modal representations; finally, this method is scalable, requiring no explicit pairwise alignment of all modalities, but only the establishment of a unified semantic anchor point around the text to achieve global alignment between multiple modalities.

[0009] Furthermore, in step S5, a gated attention fusion module is used to generate multimodal gate weights based on the aligned shared features, and weighted fusion of the private features of each modality is performed to obtain a unified multimodal sentiment representation. The core idea is to guide the weight allocation of private features of different modalities through a gating mechanism, thereby achieving adaptive fusion of multimodal information.

[0010] The design of this network is inspired by the gated cross-attention mechanism. Specifically, it first aggregates the aligned shared features within a batch, using text-shared features. Using average pooling as the core, a globally shared semantic vector is obtained: This globally shared semantic vector can be seen as the semantic center of the current batch, representing consistent sentiment trends and core information across different modalities. Next, the model uses this global semantic vector to generate three modality-specific gating signals: in, and Learnable parameters specific to the mode. Use a sigmoid activation function to ensure the gated signal. The value of is located in the interval [0,1]. This design allows the model to perform soft selection among private features of different modalities, rather than a simple linear superposition. Subsequently, the gating signal is used to weight the private features of each modality. Let For modality The vector obtained by average pooling the private feature sequences is then defined as follows: in, The concatenation operation is represented by MLP, which is used for further dimensionality reduction and nonlinear transformation to make the fused representation more discriminative. This gated attention fusion mechanism can dynamically adjust the contribution of different modalities based on global context information. For example, when When capturing strong intonation or emotional fluctuations, the model may increase the gating weights of the audio modality. To highlight the prosodic features of speech; while in scenarios where visual expression plays a dominant role, the weight of visual modalities is... This will be amplified. In this way, the model can avoid a "one-size-fits-all" fixed weighting in multimodal fusion, and instead achieve context-aware adaptive information integration.

[0011] Furthermore, in step S6, the fused multimodal features are input into the sentiment analysis module, which outputs the corresponding sentiment category or sentiment intensity result. Specifically, this is achieved by using a fully connected network combined with normalization and activation operations to perform discriminative modeling on the fused features, and then using softmax classification to output the corresponding sentiment category.

[0012] Compared with existing technologies, this invention proposes a multimodal emotion recognition method that can simultaneously take into account both modal commonalities and modal differences, achieving adaptive fusion between modalities, and has the following beneficial effects: By designing a unified feature representation and differential information capture approach, and simultaneously modeling the complementarity and differences between modalities, we effectively alleviate the information loss and imbalance problems inherent in previous models during multimodal alignment and fusion. Therefore, this method not only achieves new state-of-the-art results in overall performance but also demonstrates outstanding advantages in task generalization and robustness. On the CMU-MOSI and CMU-MOSEI datasets, it achieves 85.6% and 87.2% accuracy in binary classification, respectively, and 53.6% and 58.2% accuracy in seven-class classification, respectively, far exceeding other methods. Applying this method to scenarios such as intelligent customer service, public opinion analysis, psychological assessment, educational assistance, and human-computer interaction can provide effective sentiment data support. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the overall network structure of the present invention; Figure 3 This is a comparative experimental diagram of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Example like Figure 1-2 As shown in the figure, an embodiment of the present invention proposes a multimodal emotion recognition method that can simultaneously take into account both modal commonalities and modal differences, and achieve adaptive fusion between modalities, including the following steps: S1: Perform preprocessing operations on the CMU-MOSI and CMU-MOSEI datasets to generate data for training.

[0016] Temporal interpolation was applied to the video to insert the sequence size to a specified length of 8 frames. The audio signal in the video was uniformly resampled to 16 kHz, and the text modality was directly input to the language model. S2: Use a high-performance pre-trained encoder to process the data to obtain a high-quality initial feature sequence.

[0017] The text modality uses BERT to encode semantic information, the video modality relies on CLIP-ViT to extract visual representations, and the audio modality uses HuBERT to model speech and prosodic features.

[0018] S3: The modal feature decoupling module is used to input the initial features of each modality into the shared coding unit and the private coding unit to obtain the shared features representing cross-modal common sentiment information and the private features representing modality-specific sentiment information, respectively.

[0019] Multimodal features achieve the separation of modal commonality and individuality through a shared-private dual-path modeling approach. Specifically, for the input modal feature sequence... ,in These represent text, audio, and video, respectively. First, through the modal projection head... After unifying the dimensions, the input is fed into two parallel encoders: one via a shared encoder that shares parameters. This yields a shared representation across modalities: Capture common emotional patterns across modalities. On the other hand, through modality-specific proprietary encoders. The private representation is obtained: It preserves the unique rhythm, semantic rhetoric, and visual details of each modality. To ensure decoupling between shared and private subspaces, an orthogonality constraint loss is introduced. The basic idea is to minimize the correlation between the two in the vector space, avoiding information redundancy. Specifically, sequential features are pooled into vectors. , And define the loss as: This constraint drives shared representations to focus on modality-independent public sentiment semantics, while private representations focus on modality-unique expressive features, thereby achieving an effective separation of semantic commonality and modality differences in the feature space.

[0020] S4: Employing a cross-modal alignment module, a contrastive learning mechanism is used to guide the shared features of speech and video modalities to align with the text semantic space, thereby achieving cross-modal semantic consistency constraints.

[0021] This strategy centers on the text modality because text is typically more explicit and higher-level in its semantic expression, providing a reference for both visual and audio modalities. Therefore, we share text representations. Set as a "query" or anchor point, and force the shared representation of visual and audio to converge toward it in the semantic space through contrastive learning.

[0022] Specifically, in a batch of size N, the first... Each sample contains text-shared features. Visual sharing features Features shared with audio We construct positive and negative sample pairs: Positive sample pairs: and That is, text and visual / audio features from the same sample are shared; Negative sample pairs: for any , and This refers to the visual / audio shared features between the text and other samples within the batch.

[0023] During training, the model needs to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs to achieve cross-modal alignment. The alignment loss uses the multimodal extended version of the InfoNCE loss function. Taking text-visual modal alignment as an example, the formula is as follows: in, Represents the cosine similarity function. is a hyperparameter used to adjust the smoothness of the distribution. Similarly, the text-to-audio alignment loss is defined as: The final cross-modal alignment loss is the sum of the two: This loss function achieves semantically consistent cross-modal shared representation learning by bringing positive sample pairs closer together and pushing negative sample pairs further apart in the feature space. Its core advantages are threefold: First, it is asymmetric, using only text as the query anchor point, avoiding the noise accumulation problem that may arise from pairwise alignment between modalities; second, it enhances discriminability by leveraging InfoNCE's softmax positive-negative contrast mechanism, enabling the model to learn more discriminative cross-modal representations; finally, this method is scalable, requiring no explicit pairwise alignment of all modalities, but only the establishment of a unified semantic anchor point around the text to achieve global alignment between multiple modalities.

[0024] S5: The gating attention fusion module is used to generate multimodal gating weights based on the aligned shared features, and the private features of each modality are weighted and fused to obtain a unified multimodal sentiment representation.

[0025] The design of this network is inspired by the gated cross-attention mechanism. Specifically, it first aggregates the aligned shared features within a batch, using text-shared features. Using average pooling as the core, a globally shared semantic vector is obtained: This globally shared semantic vector can be seen as the semantic center of the current batch, representing consistent sentiment trends and core information across different modalities. Next, the model uses this global semantic vector to generate three modality-specific gating signals: in, and Learnable parameters specific to the mode. Use a sigmoid activation function to ensure the gated signal. The value of is located in the interval [0,1]. This design allows the model to perform soft selection among private features of different modalities, rather than a simple linear superposition. Subsequently, the gating signal is used to weight the private features of each modality. Let For modality The vector obtained by average pooling the private feature sequences is then defined as follows: in, The concatenation operation is represented by MLP, which is used for further dimensionality reduction and nonlinear transformation to make the fused representation more discriminative. This gated attention fusion mechanism can dynamically adjust the contribution of different modalities based on global context information. For example, when When capturing strong intonation or emotional fluctuations, the model may increase the gating weights of the audio modality. To highlight the prosodic features of speech; while in scenarios where visual expression plays a dominant role, the weight of visual modalities is... This will be amplified. In this way, the model can avoid a "one-size-fits-all" fixed weighting in multimodal fusion, and instead achieve context-aware adaptive information integration.

[0026] S6: Input the fused multimodal features into the sentiment analysis module and output the corresponding sentiment category or sentiment intensity result. A fully connected network, combined with normalization and activation operations, is used to perform discriminative modeling on the fused features, and softmax classification is employed to output the corresponding sentiment category. Figure 3 The experimental results analysis of our method compared with other methods demonstrates that our method outperforms several methods in terms of Acc2, Acc7, and MAE metrics.

[0027] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal emotion recognition method, characterized in that, Includes the following steps: S1: Preprocess the CMU-MOSI and CMU-MOSEI datasets to generate training data; S2: Use a high-performance pre-trained encoder to process the data to obtain a high-quality initial feature sequence; S3: The feature decoupling module is used to input the initial features of each modality into the shared coding unit and the private coding unit to obtain the shared features representing cross-modal common sentiment information and the private features representing modality-specific sentiment information, respectively. S4: Using a cross-modal alignment module, the shared features of speech and video modalities are guided to align to the text semantic space through a contrastive learning mechanism, thereby achieving cross-modal semantic consistency constraints. S5: The gating attention fusion module is used to generate multimodal gating weights based on the aligned shared features, and the private features of each modality are weighted and fused to obtain a unified multimodal sentiment representation; S6: Input the fused multimodal features into the sentiment analysis module and output the corresponding sentiment category or sentiment intensity result.

2. The multimodal emotion recognition method according to claim 1, characterized in that: When preprocessing the CMU-MOSI and CMU-MOSEI datasets, temporal interpolation was used to insert the sequence size to a specified length of 8 frames for the video. The speech signal in the video was uniformly resampled to 16 kHz, and the text modality was directly input into the language model.

3. The multimodal emotion recognition method according to claim 1, characterized in that: The S2 section employs a high-performance pre-trained encoder to obtain a high-quality initial feature sequence. The text modality uses BERT to encode semantic information, the video modality relies on CLIP-ViT to extract visual representations, and the audio modality uses HuBERT to model speech and prosodic features.

4. The multimodal emotion recognition method according to claim 1, characterized in that: In S3, the modal feature decoupling module is used to input the initial features of each modality into the shared coding unit and the private coding unit, respectively obtaining the shared features representing cross-modal common emotional information and the private features representing modality-specific emotional information; It achieves the separation of modal commonality and individuality through shared-private dual-path modeling; for the input modal feature sequence ,in Representing text, audio, and video respectively; first through a modal projection head After unifying the dimensions, the input is fed into two parallel encoders: one via a shared encoder that shares parameters. This yields a shared representation across modalities: ； Capture common emotional patterns across modalities; via modality-specific proprietary encoders. The private representation is obtained: ； It preserves the unique rhythm, semantic rhetoric, and visual details of each modality; to ensure the decoupling of shared and private subspaces, an orthogonality constraint loss is introduced. The basic idea is to minimize the correlation between the two in the vector space and avoid information redundancy; specifically, the sequence features are pooled into vectors. , And define the loss as: 。 5. The multimodal emotion recognition method according to claim 1, characterized in that: The S4 module employs a cross-modal alignment module, which guides the shared features of the speech and video modalities to align with the text semantic space through a contrastive learning mechanism, thereby achieving cross-modal semantic consistency constraints. This strategy focuses on the text modality because text is typically more explicit and higher-level in semantic expression, providing a reference for the visual and audio modalities. Therefore, we share text representations... Set as a "query" or anchor point, and force the shared representation of vision and audio to converge toward it in the semantic space through contrastive learning; Positive sample pairs: and That is, text and visual / audio features from the same sample are shared; Negative sample pairs: for any , and That is, the visual / audio features shared between the text and other samples in the batch; During training, the model needs to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs to achieve cross-modal alignment. The alignment loss uses the multimodal extended version of the InfoNCE loss function. Taking text-visual modal alignment as an example, the formula is as follows: in, Represents the cosine similarity function. The hyperparameter is used to adjust the smoothness of the distribution; the text-to-audio alignment loss is defined as: The final cross-modal alignment loss is the sum of the two: 。 6. The multimodal emotion recognition method according to claim 1, characterized in that: In S5, a gated attention fusion module is used to generate multimodal gated weights based on the aligned shared features, and weighted fusion of private features of each modality is performed to obtain a unified multimodal sentiment representation. It guides the weight allocation of private features of different modalities through a gating mechanism, thereby achieving adaptive fusion of multimodal information; First, the aligned shared features within the batch are aggregated to obtain text shared features. Using average pooling as the core, a globally shared semantic vector is obtained: This globally shared semantic vector represents consistent sentiment trends and core information across different modalities; next, the model uses this global semantic vector to generate three modality-specific gating signals: ； in, and Learnable parameters specific to the mode. Use a sigmoid activation function to ensure the gated signal. The value of is located in the interval [0,1]; this design enables the model to perform soft selection among private features of different modalities, rather than simple linear superposition; subsequently, the gating signal is used to weight the private features of each modality; let For modality The vector obtained by average pooling the private feature sequences is then defined as follows: in, This indicates a splicing operation, while MLP is used for further dimensionality reduction and nonlinear transformation.

7. The multimodal emotion recognition method according to claim 1, characterized in that: In step S6, the fused multimodal features are input into the sentiment analysis module, and the corresponding sentiment category or sentiment intensity result is output. It uses a fully connected network combined with normalization and activation operations to perform discriminative modeling on fused features and uses softmax classification to output the corresponding sentiment category.

Images