A cue-driven two-stage multimodal sentiment representation learning method

Abstract

This invention discloses a cue-driven two-stage multimodal sentiment representation learning method, comprising: collecting raw video data from multiple publicly available multimodal sentiment analysis datasets; performing preprocessing and feature extraction to obtain vectorized multimodal feature representations of visual, audio, and textual data, and acquiring multi-source emotional cues; in the training phase, constructing a cue-driven two-stage multimodal sentiment representation learning model, enhancing inter-class separability and preserving intra-class intensity features through an sentiment anchor comparison and alignment stage, including cue-based sentiment anchor learning and comparison and alignment between joint representations and sentiment anchors; capturing dynamic changes in emotional expression through a sentiment intensity shift estimation stage; and in the inference phase, predicting the sentiment category and the final sentiment state. This invention's method can stably capture shared sentiment features across multiple modalities, effectively suppress interference from individual differences, and significantly improve the accuracy and robustness of sentiment prediction.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision, and in particular to a cue-driven two-stage multimodal emotion representation learning method. Background Technology

[0002] Sentiment analysis aims to identify an individual's inner emotions through external behaviors and has wide applications in online education, smart healthcare, and human-computer interaction. Human inner emotions are manifested in various external behaviors, such as facial expressions, tone of voice, and linguistic content. Traditional sentiment analysis methods rely on only one of these signals. However, due to facial occlusion, semantic ambiguity, and cross-linguistic variability, unimodal sentiment analysis based on facial expressions, text, or audio often fails to accurately identify an individual's emotional state. Considering that these behavioral signals are driven by the same inner emotion and are therefore inherently interconnected, recent research has shifted from unimodal sentiment analysis to multimodal sentiment analysis. The key challenge lies in capturing cross-modal interaction information from multiple signals.

[0003] Fusion-based and decoupling-based approaches are two mainstream multimodal analysis methods, both attempting to capture cross-modal interactions from input signals to predict an individual's emotional state. Fusion-based methods integrate multiple signals into a joint representation using strategies such as tensor operations, gate units, attention mechanisms, and graph fusion. However, cross-modal signals differ in their constituent elements, element distribution, noise levels, and task relevance—a phenomenon known as modal heterogeneity—which hinders effective cross-modal interactions and leads to the fragility of the joint representation. Decoupling-based methods posit that cross-modal interactions include both modality-shared and modality-specific components and employ various techniques to extract these components from single modalities before integrating them into a joint representation for prediction.

[0004] While fusion-based and decoupling-based approaches have made significant progress, their implementations primarily focus on modal-level operations, neglecting the individual differences in human emotional expression. Due to significant differences in cultural background, socially learned behavior, physiological basis, and even personality traits, different individuals often exhibit highly differentiated overt behavioral patterns when faced with the same emotional stimuli. For example, the same level of pleasure may correspond to a behavioral spectrum ranging from a subtle smile to loud laughter. This complexity of emotional expression, driven by individual differences, has not only failed to be effectively modeled by existing models but also limits the algorithms' ability to generalize to specific groups. Summary of the Invention

[0005] The problem to be solved by this invention is to provide a prompt-driven two-stage multimodal sentiment representation learning method. By constructing prompts based on sentiment semantics to generate sentiment anchors with category discrimination, the inter-class separation of multimodal joint representation is enhanced, making it easier to determine the sentiment category. Furthermore, the method can be used to mine the fine-grained offset between the sentiment state of the sample joint representation and the sentiment anchor, thereby achieving accurate sentiment prediction.

[0006] This invention adopts the following technical solution: a cue-driven two-stage multimodal emotion representation learning method, the specific steps of which are as follows:

[0007] Step 1: Collect raw video data from multiple publicly available multimodal sentiment analysis datasets, label the data, and divide it into training, validation, and test sets;

[0008] Step 2: Preprocess and extract features from the raw video data to obtain vectorized multimodal feature representations of visual, audio, and text data, and acquire multi-source emotional cues;

[0009] Step 3: In the training phase, construct a cue-driven two-stage multimodal sentiment representation learning model (PD-MSR), including a sentiment anchor comparison and alignment phase and a sentiment intensity shift estimation phase, to learn the interaction relationships between modalities and perform sentiment analysis;

[0010] The emotional anchor point comparison and alignment stage, used to enhance inter-class separability and preserve intra-class intensity features, includes:

[0011] The prompt-based sentiment anchor learning utilizes the semantic information of sentiment labels and is trained under the constraint of cross-entropy loss to generate sentiment anchors with class discrimination.

[0012] The joint representation is aligned with the sentiment anchor by contrast, and the inter-class discriminativeness of the sentiment anchor is used to strengthen the joint representation, so as to determine the sentiment category of the sample at a coarse-grained level.

[0013] The emotional intensity shift estimation stage is used to capture the dynamic changes in emotional expression, calculate the fine-grained shift between the emotional state jointly represented by the samples and the emotional anchor point, and predict the intensity change relative to emotional polarity.

[0014] Step 4: In the reasoning stage, based on the similarity scores between the joint representation and each emotional anchor, the intensity shift of the prediction is calculated to predict the emotional category and the final emotional state.

[0015] As a preferred option, in step 1, there are three publicly available multimodal sentiment analysis datasets: MOSI dataset, MOSEI dataset, and IEMOCAP dataset.

[0016] As a preferred embodiment, step 2, the preprocessing and feature extraction, includes the following sub-steps:

[0017] Step 201: Decompose the video data to obtain the original multimodal features, including: visual image frame sequence signals, audio pulse waveform signals, and text statements.

[0018] Step 202: Set up the MSA task for multimodal sentiment analysis and the MER task for multimodal sentiment recognition;

[0019] The MSA task aims to obtain continuous emotional state values, ranging from -3 to 3, based on visual image frame sequence signals, audio pulse waveform signals, and text statements of individual samples. Positive and negative values ​​represent positive / negative emotions, and the absolute value of the value represents the intensity of the emotion. The MER task aims to obtain discrete emotion categories, including basic emotions such as happiness, sadness, anger, and neutrality, based on visual image frame sequence signals, audio pulse waveform signals, and text statements of individual samples.

[0020] Step 203: Extract vectorized features from each modality of data:

[0021] For video signals, the Facet toolkit is used for visual feature extraction, including: detection and quantification of facial motion units, facial key points, head posture, and expression-related features.

[0022] For speech signals, the COVAREP toolkit is used to extract low-level descriptors as audio features to model the dynamic changes in emotion in the speech signal. These include: Mel-frequency cepstral coefficients, which reflect the speech spectral envelope; fundamental frequency features, which describe the frequency of vocal cord vibration; spectral envelope, which represents the smooth shape of the speech spectrum; and speech quality features, including jitter and tremolo, which reflect sound stability.

[0023] For the language expression of individuals in the video, the pre-trained language model BERT is used to extract text features in the MSA task; and GloVe is used as the text feature representation method in the MER task; thus obtaining deep semantic feature representations.

[0024] As a preferred option, in step 3, the emotional anchor point comparison and alignment stage utilizes the semantic information of emotional tags to label continuous emotional state regression values, thereby enhancing the ability of sample representation to express coarse-grained emotional categories.

[0025] For the MSA task, continuous labels are rounded and quantified to obtain seven coarse-grained sentiment category labels, including: highly negative (corresponding label values ​​in the range of [-3, -2.5]), negative (corresponding label values ​​in the range of (-2.5, -1.5]), slightly negative (corresponding label values ​​in the range of (-1.5, -0.5]), neutral (corresponding label values ​​in the range of (-0.5, 0.5)), slightly positive (corresponding label values ​​in the range of [0.5, 1.5)), positive (corresponding label values ​​in the range of [1.5, 2.5)), and highly positive (corresponding label values ​​in the range of [2.5, 3]).

[0026] For the MER task, four emotion category labels are used: happy, sad, angry, and neutral.

[0027] Furthermore, in step 3, the cue-based emotion anchor learning method is as follows:

[0028] Step 301: Construct a prompt template based on the sentiment category labels of the samples. This represents the core semantic information of the emotion category;

[0029] Step 302: Utilize a pre-trained large language model , will prompt template Encode to obtain category semantic embeddings containing general knowledge. And map it to a specific dimension: ;

[0030] in, It is a linear layer. Represents the learnable parameters of a linear layer. Represents pre-trained large language models Parameters;

[0031] Step 303: Train the semantic embeddings corresponding to each emotion category, using cross-entropy loss. Train the linear mapping layer under constraints;

[0032] in, and They represent the first The true and predicted categories of each sample;

[0033] Step 304: Use the embedded representation of the emotion category as the corresponding The emotional anchor features of each category are represented , These are generated sentiment anchors used to reflect category characteristics.

[0034] Furthermore, in step 3, the method for comparing and aligning the joint representation with the emotional anchor is as follows:

[0035] Step 311: For the visual, speech, and text multimodal features of the sample input. A multimodal representation learning method is employed for fusion and decoupling, and joint representations are obtained through cross-modal interaction. ; where subscript These represent the text modality, visual modality, and audio modality, respectively.

[0036] Step 312: Construct contrastive learning between joint representations and emotional anchors, and integrate them with joint representations. Anchor points belonging to the same emotional category Considered a positive sample, denoted as ,Will Other emotional anchors Treat them as negative samples and use contrastive loss. As the objective function, it maximizes the similarity of positive sample pairs and minimizes the similarity of negative sample pairs.

[0037] Furthermore, in step 3, the emotion intensity shift estimation stage is used for the MSA task, and the specific processing is as follows:

[0038] Step 321: Calculate the joint representation Anchor points corresponding to its actual category labels Differences between characteristics ;

[0039] Step 322: Model differential features using non-linear relationships. and emotional intensity shift The subtle emotional connections between them ;

[0040] Step 323: Combine the true class label with the intensity shift predicted in this stage to obtain the final sentiment state predicted by the model during the training phase. .

[0041] As a preferred option, in step 4, for the MER task, the joint representation of the samples is calculated. With each anchor point The cosine similarity score is used to select the category corresponding to the anchor point with the highest score as the prediction result. .

[0042] For the MSA task, the joint representation of the samples is calculated. With each anchor point The similarity score is used to select the category corresponding to the anchor with the highest score as the prediction result of the sentiment category in the first stage. Based on sample characterization And the anchor with the highest similarity Calculate the difference features The predicted intensity shift is obtained through nonlinear mapping. The final emotional state predicted by the model is obtained. .

[0043] The present invention also provides: an electronic device, comprising:

[0044] One or more processors;

[0045] A storage device on which one or more programs are stored;

[0046] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned cue-driven two-stage multimodal sentiment representation learning method.

[0047] The present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the aforementioned prompt-driven two-stage multimodal emotion representation learning method.

[0048] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0049] 1. This invention introduces a prompt-driven two-stage multimodal sentiment representation learning model. In the first stage, the sentiment anchor comparison and alignment module is used to align features of different modalities in a unified sentiment semantic space, reducing the intermodal shift caused by individual expression differences. In the second stage, the sentiment intensity shift estimation module is used to correct the intensity of the prediction results, thereby further reducing the error caused by the inconsistency of sentiment strength expression.

[0050] 2. Compared with existing methods based solely on feature fusion or feature decoupling, the method of this invention can more stably capture shared sentiment features among multiple modalities and effectively suppress interference caused by individual differences, significantly improving the accuracy and robustness of prediction; at the same time, it can serve as a general framework and be combined with any multimodal representation learning model, greatly enhancing the scalability of this invention. Attached Figure Description

[0051] Figure 1 This is a block diagram of the two-stage multimodal emotion representation learning model driven by prompts in this invention;

[0052] Figure 2 This is a schematic diagram showing the feature visualization results of the joint representation of samples obtained by the model of this invention on the MOSI dataset. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0054] In one embodiment of the present invention, a prompt-driven two-stage multimodal sentiment representation learning method is proposed. By constructing prompts based on sentiment semantics, sentiment anchors with class discrimination are generated, enhancing the inter-class separation of multimodal joint representation, facilitating the determination of sentiment category, and further mining the fine-grained offset between the sentiment state of the sample joint representation and the sentiment anchor, so as to achieve accurate sentiment prediction.

[0055] Step 1: Dataset Preparation

[0056] First, we obtained multiple publicly available multimodal sentiment analysis datasets to ensure the comparability and universality of the experiments, and to provide a standardized data foundation for subsequent model training, validation and performance evaluation.

[0057] In this embodiment, three datasets are used, including: MOSI and MOSEI, which are multimodal datasets for multimodal sentiment analysis tasks, labeled with continuous sentiment state regression values ​​(-3 to 3, positive and negative values ​​represent positive and negative sentiment polarities respectively, and the absolute value represents the sentiment intensity); and IEMOCAP, which is a dataset for multimodal emotion recognition tasks, labeled with discrete sentiment categories (happy, sad, angry, neutral, etc.).

[0058] The MOSI and MOSEI datasets consist of personal monologue comments facing a camera collected from websites, with each sample being a video clip ranging from 2 to 5 seconds. The IEMOCAP dataset, on the other hand, consists of video clips of two-person dialogues collected in a controlled laboratory environment.

[0059] Subsequently, the dataset was divided into training, validation, and test sets, allowing the model to learn during training and perform final performance evaluation during testing, thus ensuring the reliability and reproducibility of the experimental results. Data loading was accomplished using PyTorch's DataLoader tool.

[0060] Step 2: Input Processing and Feature Extraction

[0061] In order to obtain data features from video data that can be input into the model, the video needs to be preprocessed and features extracted to obtain visual, audio and text feature representations.

[0062] The first step is to decompose the video data to obtain the visual image frame sequence, audio pulse waveforms, and text sentences, which are the original multimodal features. Since the original data formats of text, speech, and video modalities differ significantly, they are difficult to use directly as model input. Therefore, feature extraction tools are needed to extract vectorized features from each modal data separately.

[0063] 1. Visual modality, which mainly reflects the speaker's facial expressions, posture, and movement changes during expression, is an important source of signals for emotion recognition. This embodiment uses the Facet toolkit for visual feature extraction.

[0064] Facet is a multimodal behavior analysis tool that automatically detects and quantifies facial action units (AUs), facial landmarks, head pose, and other expression-related features. These features are closely related to emotional expression in psychology. Facial action units correspond to specific movements of facial muscles; for example, AU12 (upturned corners of the mouth) is associated with "happiness," and AU4 (frowned brows) is associated with "anger." Facial landmarks are the coordinates of facial feature points, such as the corners of the eyes, mouth, and brow peaks, reflecting facial deformation. Head pose includes head tilt, pitch, and roll angles, reflecting the speaker's attention direction and changes in body posture.

[0065] For different datasets, this embodiment extracts the following features based on their acquisition conditions and task characteristics: MOSI: 47-dimensional visual features; MOSEI: 35-dimensional visual features; IEMOCAP: 35-dimensional visual features.

[0066] 2. Audio modalities provide rich emotion-related cues, such as speech rate, pitch, energy, and timbre. This embodiment uses the COVAREP toolkit to extract low-level descriptors (LLDs) from the speech signal as audio features. COVAREP is an open-source speech signal processing library specifically designed for emotion recognition and speaker recognition tasks, capable of efficiently extracting various audio features.

[0067] In this embodiment, the main features extracted include:

[0068] Mel-frequency cepstral coefficients (MFCCs): 12 dimensions, reflecting the spectral envelope of speech and simulating human auditory perception;

[0069] Fundamental frequency (Pitch) characteristics: describes the frequency of vocal cord vibration and is related to pitch.

[0070] Spectral envelope: Represents the smooth shape of the speech spectrum, reflecting timbre characteristics;

[0071] Voice quality characteristics include jitter, tremor, etc., which reflect the stability of the voice.

[0072] Based on the above metrics, 74-dimensional audio feature vectors were extracted in both the MSA and MER tasks to model the dynamic changes in emotion in speech signals.

[0073] 3. Language and Text: For the language expression of individuals in the video, the pre-trained language model BERT is used to extract text features for the MSA task; and GloVe is used as the text feature representation method for the MER task.

[0074] BERT is based on the Transformer architecture and captures deep semantic dependencies between contexts through pre-training on a large corpus.

[0075] In the specific implementation, the input text is passed through the BERT encoding layer to obtain the hidden vector representation of each token, and the sentence-level representation (such as the [CLS] vector) is extracted to finally obtain a 768-dimensional text feature vector. This feature can well represent sentence-level semantic information and potential sentiment features, providing a high-level semantic representation for subsequent multimodal fusion.

[0076] GloVe is a word vector learning model based on global statistical information. By performing matrix decomposition on the co-occurrence matrix of a large-scale corpus, the resulting word vectors can capture both local context and global statistical relationships.

[0077] In the specific implementation, each word is mapped to a 300-dimensional dense vector representation, and words with similar semantics are closer in the vector space; by averaging or weighting the word vectors in the sentence, a sentence-level sentiment representation is obtained, providing basic semantic input for the MER model.

[0078] The aforementioned multimodal feature extraction schemes can acquire multi-source emotional cues at the text, visual, and audio levels. BERT and GloVe provide deep semantic representations, Facet captures facial expressions and posture information, and COVAREP characterizes the prosody and phonological features of speech. The combination of multimodal features helps the model comprehensively understand and recognize emotional states from three perspectives: linguistic content, nonverbal behavior, and speech signals.

[0079] Step 3: Train the network model

[0080] Based on the previously extracted visual, speech, and text multimodal feature vectors, a network model is constructed and trained to learn the interaction relationships between modalities and complete the sentiment analysis task.

[0081] In this embodiment, the constructed cue-driven two-stage multimodal emotion representation learning model has the following structure: Figure 1 As shown, it includes the emotional anchor point comparison and alignment stage and the emotional intensity shift estimation stage.

[0082] Specifically, the inherent diversity of emotional expression poses a significant challenge to learning discriminative joint representations for fine-grained emotion analysis. To address this fundamental limitation, this embodiment designs an emotion anchor comparison and alignment stage, utilizing the semantic information of emotion labels and annotating continuous emotion state regression values ​​(e.g., -2.5 corresponds to "highly negative" emotion semantics, and 2.5 corresponds to "highly positive" emotion semantics) to enhance the expressive power of sample representations on coarse-grained emotion categories.

[0083] For the multimodal sentiment analysis (MSA) task, rounding continuous labels and quantifying them yields seven coarse-grained sentiment categories: highly negative (corresponding to label values ​​in the range of -3, -2.5), negative (corresponding to label values ​​in the range of -2.5, -1.5), slightly negative (corresponding to label values ​​in the range of -1.5, -0.5), neutral (corresponding to label values ​​in the range of -0.5, 0.5), slightly positive (corresponding to label values ​​in the range of [0.5, 1.5)), positive (corresponding to label values ​​in the range of [1.5, 2.5)), and highly positive (corresponding to label values ​​in the range of [2.5, 3]).

[0084] Multimodal emotion recognition (MER) is itself a classification task, using labels such as happy, sad, angry, and neutral.

[0085] Specifically, the emotional anchor contrast alignment phase consists of two steps: cue-based emotional anchor learning and contrast alignment between joint representations and emotional anchors.

[0086] 1. Cue-based affective anchor learning, the specific methods are as follows:

[0087] First, based on the sentiment categories of the samples mentioned above, a cue template is constructed. "The sentiment of <class>“” is used to represent the core semantic information of this emotion category.

[0088] Then, using existing pre-trained large language models Encode the prompt template to obtain a category semantic embedding containing general knowledge. Map it to a specific dimension: ;

[0089] in, It is a linear layer. Represents the learnable parameters of a linear layer. Represents pre-trained large language models The parameters.

[0090] To improve the class discriminative power of sentiment anchor features, cross-entropy loss is used... Train the linear mapping layer under constraints;

[0091] in, and They represent the first The true class and predicted class of each sample.

[0092] Furthermore, after several rounds of training, the semantic embeddings corresponding to each emotion category gradually separate due to the constraint of cross-entropy loss. Ultimately, the embedding representations of each category are used as the corresponding... The emotional anchor features of each category are represented This is used in the subsequent comparison and alignment steps.

[0093] 2. Contrast and alignment between joint representations and emotional anchors, the specific methods are as follows:

[0094] Based on prompt-based emotion anchor learning, reliable emotion anchors were generated. In the comparison alignment step between joint representations and sentiment anchors, due to the optimization of cross-entropy loss, the sentiment anchors... It can clearly reflect category characteristics and has good inter-class separability. Therefore, it is hoped that the inter-class distinguishability of the emotion anchor point can be used to strengthen the joint representation, so as to facilitate the judgment of the emotion category of the sample at a coarse-grained level.

[0095] Specifically, for the visual, speech, and text multimodal features of the sample input , Representing the input text modality, visual modality, and audio modality data respectively, we first employ existing multimodal representation learning methods to obtain a joint representation through cross-modal interaction. .

[0096] Multimodal representation learning aims to obtain joint representations from multiple modalities and can be divided into fusion-based methods and decoupling-based methods. Fusion-based methods utilize tensor operations, gating units, attention mechanisms, and graph structures to integrate multimodal information into a joint representation for sentiment analysis, such as TFN and MulT. Decoupling-based methods, on the other hand, recognize that fusion-based methods cannot capture the interactions between modalities. Therefore, they decompose cross-modal interactions into modality-general and modality-specific parts, such as MISA.

[0097] After obtaining joint characterization Then, a contrastive learning model is constructed between the joint representation and the sentiment anchor. In the contrastive learning, for each sample, the features learned by the model are expected to meet the following requirements: samples that are semantically similar or belong to the same category (positive samples) should be closer in the feature space, and similarly, samples that are semantically different or belong to other categories (negative samples) should be farther apart in the feature space.

[0098] Based on this idea, this embodiment will combine characterization with... Anchor points belonging to the same emotional category Considered as a positive sample, denoted as At the same time, other emotional anchors Treat them as negative samples and use contrastive loss. As the objective function, it maximizes the similarity of positive sample pairs and minimizes the similarity of negative sample pairs;

[0099] in, Represents an exponential function. This indicates the calculation of cosine similarity. This represents the temperature coefficient, used to control the smoothness of the category distribution.

[0100] In this way, the sentiment anchor comparison and alignment stage enhances inter-class separability and preserves intra-class intensity features for subsequent sentiment intensity shift estimation.

[0101] Compared to predicting coarse-grained emotion categories, fine-grained emotion states are better suited to capturing the dynamic changes in emotion expression, thus providing timely decision-making guidance. In the multimodal sentiment analysis (MSA) task, to further obtain fine-grained emotion states, this embodiment designs an emotion intensity shift estimation to predict intensity changes relative to emotion polarity.

[0102] Specifically, the true label of the sample With quantified category labels Subtracting the values, the calculated intensity shift This is the goal of this stage.

[0103] In the previous stage, we have narrowed the gap in the joint representation of the samples by using contrastive loss. Emotional anchors built with its true category labels This distances it from other categories of emotional anchors, aligning the distribution of joint representations and their corresponding emotional anchors.

[0104] Furthermore, compute joint representations Anchor points corresponding to its actual category labels Differences between characteristics : Used to reflect the sample representation within the sentiment category With anchor point The differences were identified, and nonlinear relationships were used to model the difference features. and emotional intensity shift The subtle emotional connections between them: ;in, and This represents the parameters of the linear layer.

[0105] Finally, by combining the true class label with the intensity shift predicted in this stage, we obtain the final sentiment state predicted by the model during the training phase. : .

[0106] Step 4: Prediction Results

[0107] In the model inference stage, since multimodal emotion recognition (MER) is a classification task, this embodiment removes the emotion intensity shift estimation stage designed for the MSA task, and obtains the emotion anchor point. Then, the joint characterization of the samples is calculated. With each anchor point The cosine similarity score is used to select the category corresponding to the anchor point with the highest score as the prediction result. :

[0108] .

[0109] For classification tasks, the results are evaluated using binary classification accuracy (Acc2) and F1 score.

[0110] Specifically, based on the prediction results and the actual labels, four scenarios were statistically analyzed: A true example represents a positive example where the prediction is positive. A true negative example indicates a negative example that was predicted to be negative. False positives are negative examples that are predicted to be positive, and false negatives are positive examples that are predicted to be negative.

[0111] Accuracy represents the proportion of samples correctly predicted by the model out of all samples, reflecting the overall classification accuracy of the model. It is calculated using the following formula: .

[0112] The F1 score is more suitable for evaluating classification performance under imbalanced sample conditions. .

[0113] On the IEMOCAP dataset, the model was evaluated using the binary classification accuracy (Accc2 and F1 scores) and average Acc2 and F1 scores for each emotion.

[0114] Furthermore, for the multimodal sentiment analysis (MSA) task, firstly, the joint representation of the samples is still calculated. With each anchor point The similarity score is used to select the category corresponding to the anchor with the highest score as the prediction result of the sentiment category in the first stage. .

[0115] Next, based on sample characterization And the anchor with the highest similarity Calculate the difference features The predicted intensity shift is obtained through nonlinear mapping. Combining the two, we obtain the final emotional state predicted by the model. + .

[0116] To quantitatively and fairly evaluate the model's performance, this embodiment follows recognized standards and uses mean absolute error (MAE), binary classification accuracy (Acc2) and F1 score after binary polarity processing, seven-class classification accuracy (Acc7) after quantization processing, and correlation coefficient with the true label (Corr) as evaluation metrics on the MOSI and MOSEI datasets.

[0117] MAE measures the model's predicted values. With real labels The average absolute difference between them reflects the overall degree of bias in the predictions: The smaller the MAE, the more accurate the model's predictions.

[0118] Acc7 is one of the metrics used in multimodal sentiment analysis for discrete sentiment classification tasks.

[0119] ;

[0120] in, It is an indicator function; it returns 1 if the condition is true, and 0 otherwise. This indicates rounding to the nearest integer.

[0121] Corr measures the linear correlation between model predictions and actual values, i.e., whether they exhibit a consistent linear trend. Unlike MAE, Corr does not focus on absolute error, but rather on whether the trends in the predicted and actual values ​​are consistent.

[0122] ;

[0123] in, and These represent the mean of the true labels and the mean of the predicted values, respectively.

[0124] To verify the effectiveness of the model of this invention, the results of combining the PD-MSR of this invention with several typical multimodal representation learning baseline models (modal addition, modal concatenation, MISA, MulT, MCL and MMCL) on three datasets are compared with the results of the previous state-of-the-art models, as shown in Table 1 and Table 2.

[0125] Table 1: Comparison of the results of the PD-MSR method of this invention with other state-of-the-art models on the MOSI and MOSEI datasets.

[0126]

[0127] Table 2: Comparison of the results of the PD-MSR method of this invention with other state-of-the-art models on the IEMOCAP dataset.

[0128]

[0129] On the MOSI dataset, PD-MSR(Add) and PD-MSR(Concatenation), despite employing only simple modality fusion techniques, achieved highly competitive results. The PD-MSR(MISA) method, combined with MISA, achieved improvements of 16.5%, 3.6%, 3.7%, 10.6%, and 5.4% across all metrics. PD-MSR(MulT) achieved improvements of 19.8%, 4.2%, 4.3%, and 10.6% on Acc7, Acc2, F1, and MAE metrics, significantly outperforming MulT and surpassing previous state-of-the-art (SOTA) results for these metrics. PD-MSR(MCL) outperformed the baseline MCL, improving by 0.6%, 1.5%, 1.6%, 5.0%, and 2.3% across all metrics, while PD-MSR(MMCL) outperformed the previous SOTA model MMCL, producing improvements of 0.6%, 0.5%, 0.6%, 2.0%, and 0.5%. PD-MSR (MCL) and PD-MSR (MMCL) achieved state-of-the-art (SOTA) results across all metrics, outperforming previous SOTA results by 0.7%, 0.7%, and 2.2% in Acc2, F1, and MAE, respectively.

[0130] On the MOSEI dataset, PD-MSR(Add) and PD-MSR(Concatenation) outperform most methods on the Acc7, MAE, and Corr metrics. When used in conjunction with the PD-MSR framework, various MSA methods demonstrate comprehensive improvements across all metrics. Notably, PD-MSR(MMCL) achieves improvements of 0.7%, 0.6%, and 0.9% on the Acc7, MAE, and Corr metrics respectively compared to the previous state-of-the-art model MMCL, setting a new state-of-the-art record. These optimal results indicate that PD-MSR excels in fine-grained analysis of individual emotional states and can be combined with any MSA method.

[0131] On the IEMOCAP dataset, PD-MSR(Add) and PD-MSR(Concatenation) perform comparable to or even better than some earlier methods, demonstrating that our framework effectively enhances joint representations. PD-MSR also improves the performance of existing MER methods. PD-MSR(MISA) achieves significant improvements over MISA, with improvements of 4.0%, 7.0%, 3.6%, 4.5%, and 4.8% on the four emotions and average performance, respectively. Similarly, PD-MSR(MulT) outperforms MulT, with improvements of 2.1%, 3.8%, 0.6%, 4.2%, and 2.6% on the same metrics. For newer methods such as MCL and MMCL, combining them with our representation enhancement framework PD-MSR improves all emotion recognition metrics, resulting in an average performance improvement of 0.9%. PD-MSR (MCL) and PD-MSR (MMCL) achieved state-of-the-art (SOTA) results on all metrics, improving upon previous SOTA results by 0.4%, 0.9%, 0.5%, 1.8%, and 0.9%, respectively. This indicates that PD-MSR can enhance the joint representation with emotion category information obtained from the multimodal representation learning model for use in the MER task.

[0132] Meanwhile, the feature visualization results of the sample representations on the MOSI dataset can also intuitively demonstrate the effectiveness of the method, such as... Figure 2 As shown, different colors are used to distinguish sample representations of different emotion categories. Dimension 1 and Dimension 2 are dimensionality-reduced representations of the sample representations in two-dimensional space. It can be seen that, because PD-MSR enhances the sample representations at both the emotion category and intensity shift stages, the distribution in the feature space exhibits a distribution that is separated between classes and compact within classes. This indicates that the PD-MSR method of this invention can indeed effectively solve the problem of joint representation fragility caused by the diversity of individual emotion expression.

[0133] In this embodiment of the invention, an electronic device is also provided, including: one or more processors; a storage device storing one or more programs thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the prompt-driven two-stage multimodal emotion representation learning method described in the above embodiments.

[0134] In this embodiment of the invention, a computer-readable storage medium is also provided, on which a computer program is stored. When the program is executed by a processor, it implements the steps of the prompt-driven two-stage multimodal emotion representation learning method in the above embodiments.

[0135] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.< / class>

Claims

1. A cue-driven two-stage multimodal emotion representation learning method, characterized in that, The specific steps are as follows: Step 1: Collect raw video data from multiple publicly available multimodal sentiment analysis datasets, label the data, and divide it into training, validation, and test sets; Step 2: Preprocess and extract features from the raw video data to obtain vectorized multimodal feature representations of visual, audio, and text data, and acquire multi-source emotional cues; Step 3: In the training phase, construct a cue-driven two-stage multimodal sentiment representation learning model, including a sentiment anchor comparison and alignment stage and a sentiment intensity shift estimation stage, to learn the interaction relationships between modalities and perform sentiment analysis. The emotional anchor point comparison and alignment stage, used to enhance inter-class separability and preserve intra-class intensity features, includes: The prompt-based sentiment anchor learning utilizes the semantic information of sentiment labels and is trained under the constraint of cross-entropy loss to generate sentiment anchors with class discrimination. The joint representation is aligned with the sentiment anchor by comparison. The inter-class discriminative power of the sentiment anchor is used to strengthen the joint representation and determine the sentiment category of the sample at a coarse-grained level. The method is as follows: Step 311: For the visual, speech, and text multimodal features of the sample input. A multimodal representation learning method is employed for fusion and decoupling, and joint representations are obtained through cross-modal interaction. ; Among them, subscript These represent the text modality, visual modality, and audio modality, respectively. Step 312: Construct contrastive learning between joint representations and emotional anchors, and integrate them with joint representations. Anchor points belonging to the same emotional category Considered a positive sample, denoted as ,Will Other emotional anchors Treat them as negative samples and use contrastive loss. As the objective function, it maximizes the similarity of positive sample pairs and minimizes the similarity of negative sample pairs; in, Represents an exponential function. This indicates the calculation of cosine similarity. This represents the temperature coefficient, used to control the smoothness of category distribution; The emotional intensity shift estimation stage is used to capture the dynamic changes in emotional expression, calculate the fine-grained shift between the emotional state jointly represented by the samples and the emotional anchor point, and predict the intensity change relative to emotional polarity. Step 4: In the reasoning stage, based on the similarity scores between the joint representation and each emotional anchor, the intensity shift of the prediction is calculated to predict the emotional category and the final emotional state.

2. The two-stage multimodal emotion representation learning method according to claim 1, characterized in that, In step 1, there are three publicly available multimodal sentiment analysis datasets: MOSI dataset, MOSEI dataset, and IEMOCAP dataset. The MOSI and MOSEI datasets were obtained from the internet and consist of personal monologue comments facing a camera. Each sample is a video clip of varying length. For the multimodal sentiment analysis task, continuous sentiment state regression values ​​were labeled on the MOSI and MOSEI datasets. The IEMOCAP dataset was collected in a controlled laboratory environment. Each sample is a video clip of a dialogue between two people. For the multimodal emotion recognition task, the IEMOCAP dataset is labeled with discrete emotion categories.

3. The two-stage multimodal emotion representation learning method according to claim 1, characterized in that, Step 2, the preprocessing and feature extraction, includes the following sub-steps: Step 201: Decompose the video data to obtain the original multimodal features, including: visual image frame sequence signals, audio pulse waveform signals, and text sentences; Step 202: Set up the MSA task for multimodal sentiment analysis and the MER task for multimodal emotion recognition; the MSA task obtains continuous sentiment state values ​​based on the visual image frame sequence signal, audio pulse waveform signal and text sentence of individual samples, with positive and negative values ​​representing positive / negative, and the absolute magnitude of the value representing the sentiment intensity; the MER task obtains discrete sentiment categories based on the visual image frame sequence signal, audio pulse waveform signal and text sentence of individual samples; Step 203: Extract vectorized features from each modality of data: For video signals, the Facet toolkit is used for visual feature extraction, including: detection and quantification of facial motion units, facial key points, head posture and expression-related features; For speech signals, the COVAREP toolkit is used to extract low-level descriptors as audio features to model the dynamic changes in emotion in speech signals. These include: Mel-frequency cepstral coefficients, which reflect the speech spectral envelope; fundamental frequency features, which describe the frequency of vocal cord vibration; spectral envelope, which represents the smooth shape of the speech spectrum; and speech quality features, including jitter and tremolo, which reflect sound stability. For the language expression of individuals in the video, the pre-trained language model BERT is used to extract text features in the MSA task; and GloVe is used as the text feature representation method in the MER task; thus obtaining deep semantic feature representations.

4. The two-stage multimodal emotion representation learning method according to claim 3, characterized in that, In step 3, the emotional anchor point comparison and alignment stage utilizes the semantic information of emotional tags to label continuous emotional state regression values, thereby enhancing the ability of sample representation to express coarse-grained emotional categories. For the MSA task, continuous labels are rounded and quantified to obtain seven coarse-grained sentiment category labels, including: highly negative, negative, slightly negative, neutral, slightly positive, positive, and highly positive. For the MER task, four emotion category labels are used: happy, sad, angry, and neutral.

5. The two-stage multimodal emotion representation learning method according to claim 4, characterized in that, In step 3, the cue-based emotion anchor learning method is as follows: Step 301: Construct a prompt template based on the sentiment category labels of the samples. This represents the core semantic information of the emotion category; Step 302: Utilize a pre-trained large language model , will prompt template Encode to obtain category semantic embeddings containing general knowledge. And map it to a specific dimension: ; in, It is a linear layer. Represents the learnable parameters of a linear layer. Represents pre-trained large language models Parameters; Step 303: Train the semantic embeddings corresponding to each emotion category, using cross-entropy loss. Train the linear mapping layer under constraints; in, and They represent the first The true and predicted categories of each sample; Step 304: Use the embedded representation of the emotion category as the corresponding The emotional anchor features of each category are represented , These are generated sentiment anchors used to reflect category characteristics.

6. The two-stage multimodal emotion representation learning method according to claim 5, characterized in that, In step 3, the sentiment intensity shift estimation stage is used for the MSA task, and the specific processing is as follows: Step 321: Calculate the joint representation Anchor points corresponding to its actual category labels Differences between characteristics : ； Step 322: Model differential features using non-linear relationships. and emotional intensity shift The subtle emotional connections between them : ； in, and Parameters representing linear layers; Step 323: Combine the true class label with the intensity shift predicted in this stage to obtain the final sentiment state predicted by the model during the training phase. : ； in, This represents the quantified category label.

7. The two-stage multimodal emotion representation learning method according to claim 6, characterized in that, In step 4: For the MER task, the joint representation of the samples is calculated. With each anchor point The cosine similarity score is used to select the category corresponding to the anchor point with the highest score as the prediction result. : ； For the MSA task, the joint representation of the samples is calculated. With each anchor point The similarity score is used to select the category corresponding to the anchor with the highest score as the prediction result of the sentiment category in the first stage. Based on sample characterization And the anchor with the highest similarity Calculate the difference features The predicted intensity shift is obtained through nonlinear mapping. The final emotional state predicted by the model is obtained. + .

8. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the two-stage multimodal emotion representation learning method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the two-stage multimodal emotion representation learning method according to any one of claims 1 to 7.

Images