A multimodal sentiment analysis method based on contrast-enhanced heterogeneous modal fusion

Abstract

The present invention provides a multimodal sentiment analysis method for contrast-enhanced non-isomorphic modal fusion, which relates to the technical field of sentiment analysis. The present invention compensates for the difference in information density between visual modal features and audio modal features and text modalities by fusing them. The present invention designs joint contrastive learning, and obtains intra-sample loss and inter-sample loss through intra-sample contrastive learning and inter-sample contrastive learning. The intra-sample contrastive learning aims to enhance inter-modal consistency, and the inter-sample contrastive learning ensures the independence and uniqueness of each modal feature to avoid information loss caused by excessive fusion. Therefore, the present invention takes into account the complementarity and difference between modalities, compensates for the difference in information density between visual and audio modalities and text modalities, and improves the performance and generalization ability of the sentiment analysis model.

Description

Technical Field

[0001] The present invention relates to the technical field of sentiment analysis, and in particular to a multimodal sentiment analysis method for contrast-enhanced non-isomorphic modal fusion. Background Art

[0002] Background: With the development of social media platforms and technological advancements, people have begun recording videos and uploading them to social media platforms such as Weibo, TikTok, YouTube, and Twitter to express their opinions. More and more people are sharing their opinions online in the form of videos, rather than just text. This diversification of information sources has enabled researchers to analyze sentiment across modalities such as text, audio, and visuals, making multimodal sentiment analysis (MSA) an important research field.

[0003] In multimodal sentiment analysis, audio captures the speaker's intonation and audio features; text provides semantic information about the language content, where vocabulary, syntax, and context all convey emotion; and video includes facial expressions, body language, and background information. These three modalities, despite their different sources, express consistent emotions and possess strong correlations, enriching the diverse perspectives of sentiment analysis. The combined validation of these modalities compensates for their limitations and ultimately improves the accuracy and comprehensiveness of sentiment analysis. Previous work has demonstrated that trimodal sentiment analysis is more accurate than single-modal or even bimodal analysis.

[0004] In reality, different modal data contain different information, which researchers refer to as inter-modal diversity. At the same time, there is also relatively consistent information between modalities, which researchers refer to as inter-modal complementarity. Simultaneously considering both modal complementarity and diversity to obtain effective features has always been a challenge in the multimodal field. Existing research indicates that traditional text sentiment analysis relies solely on words, phrases, and the semantic connections between them to determine sentiment. Models relying solely on the information contained in a single modality are insufficient to discern complex emotional information. In the trimodal context, however, by incorporating acoustic and visual information in addition to textual information, leveraging the connections between multimodal data can reveal potentially hidden emotional information within the text, thereby achieving more accurate emotion recognition. This demonstrates that inter-modal complementary information plays a crucial role in prediction. Given the inseparable complementarity and diversity between modalities, how to better integrate and fuse information from each modality has become a research hotspot in multimodal sentiment analysis.

[0005] Inspired by emotional arousal models in psychology, Zhang et al. proposed a Deep Emotional Arousal Network (DEAN) to simulate the entire human communication process under multimodal input. On the input side, an improved multi-head attention mechanism is used to explore the interaction between each modality. Subsequently, the authors employed a multimodal BiLSTM to extract contextual features from each modality through a bidirectional LSTM network and model the time series to simulate emotional consistency. The final module is a multimodal gating module, which implicitly fuses modalities by selecting information to output based on modality importance. These modules respectively simulate the functions of the perceptual analysis system, cognitive comparator, and activation mechanism in human emotional arousal.

[0006] Zhang F, Li X, Lim CP, et al.Deep emotional arousal network for multimodal sentiment analysis and emotion recognition[J], Information Fusion, 2022,88:296–304.

[0007] Li et al. proposed a global acoustic feature-enhanced sequential perception network (AMOA). After extracting features from each modality, they used a cross-modal transformer (CMT) for two-stage fusion. The first stage fused the auditory modality with the text modality as the core, and the second stage added the visual modality. To reflect the overall acoustic information of the overall pitch change, the authors added a global acoustic feature (GAF). Contrastive learning was used to align and concatenate the two features. Finally, the contrastive loss and classification loss were added together to guide model training, and classification was performed through a multi-layer perceptron layer.

[0008] Li Z, Zhou Y, Zhang W, et al. AMOA: Global acoustic feature enhanced modal-order-aware network for multimodal sentiment analysis, Proceedings of the 29th International Conference on Computational Linguistics, 2022, Gyeongju, Republic of Korea, 2022, 7136–7146.

[0009] The Text Feature Enhancement-based Transformer Fusion Network (TETFN) proposed by Wang et al. uses text to enhance pairwise cross-modal feature interactions to obtain an effective unified multimodal representation. It integrates textual information into learned emotion-related non-textual representations through text-based multi-head attention and preserves the difference information between modes through unimodal label prediction. In addition, the vision pre-trained model Vision Transformer is used to extract visual features from the original video to preserve the global and local information of the face. The advantage of this model is that it can integrate discourse information, improve the effectiveness of non-text modalities in MSA, and maintain inter-modal and intra-modal relationships.

[0010] Wang D,Guo X,Tian Y,et al.TETFN:A text enhanced transformer fusionnetwork for multimodal sentiment analysis[J].Pattern Recognition,2023,136:109259.

[0011] Shi et al. proposed a Co-spatial Representation Interaction Network (CRNet), which leverages distinct acoustic and visual representation subspaces to interact with language modalities. They separated the video and audio modalities into modality-invariant and modality-specific subspaces, and enhanced the corresponding modality-invariant and modality-specific representations through a gradient-based representation enhancement mechanism. Furthermore, they designed a novel multi-task strategy to optimize the training process and improve the performance of different combinations of representations from these two spaces.

[0012] Shi H,Pu Y,Zhao Z,et al.Co-space Representation Interaction Network for multimodal sentiment analysis[J].Knowledge-Based Systems,2024,283:111149.

[0013] Compared to single-modal data, multimodal data often contains richer emotional information. The core of multimodal sentiment analysis lies in how to effectively fuse information from multiple modalities for classification. Many current fusion methods employ simple strategies, lacking scientific rationality in the selection of fusion features, and overlook the complementarity between modal data and the importance of the emotional intensity of different modal data. However, in real-world environments, human behavior is often complex, and relying on simple model fusion strategies cannot achieve accurate emotion prediction.

[0014] The flaw of the Deep Emotional Arousal Network (DEAN) is that it does not explicitly consider the information density differences between visual and audio modalities and text modalities in the initial feature modeling stage. Direct fusion and interaction of the three may cause the large amount of noise information contained in the visual and audio modalities to interfere with the prediction results; finally, the weight design of the gating mechanism may be too dependent on data distribution and lacks explicit modeling of the differences and complementarities between modalities, resulting in insufficient information utilization.

[0015] A drawback of the global acoustic feature-enhanced sequential perception network (AMOA) is its two-stage fusion strategy: the first stage integrates the acoustic modality with the text modality as the core, and the second stage incorporates the visual modality. While this step-by-step fusion approach can gradually enhance the text modality, it can also cause the feature information of subsequent modalities to be obscured by the previously fused modality, and in particular, the importance of the visual modality may be underestimated.

[0016] The Text-Enhanced Transformer Fusion Network (TETFN) suffers from the drawback that visual and audio modalities typically contain more redundant information and have lower information density. Relying solely on text guidance may not fully tap into the potential emotion-related features inherent in these modalities. The TETFN model preserves intermodal differences through single-modal label prediction. However, during the fusion process, the modalities may become overly dependent, weakening their differences and failing to fully preserve their relative independence.

[0017] The drawback of the Co-spatial Representation Interaction Network (CRNet) is that its modal fusion strategy is relatively simple, directly fusing the text modality with the visual / acoustic modality without considering the difference in information density between different modalities, which may lead to the inability to fully explore the deep interaction information between modalities. Summary of the Invention

[0018] In view of the shortcomings of the existing technology, the purpose of the present invention is to propose a multimodal sentiment analysis method for contrast-enhanced non-isomorphic modal fusion, comprising:

[0019] Step 1: Construct a sample set, which includes multiple target samples. The target samples include input samples and output samples. The input samples include the temporally aligned visual features X v , audio features X after time alignment a and time-aligned text features X t , the output sample is the label corresponding to the input sample, the label is the emotion category corresponding to the input sample, the sample set is divided into a training set, a validation set and a test set;

[0020] Step 2: Through the feedforward network, the time-aligned visual features X in the target samples in the test set are vand the time-aligned audio features X a Processing is performed to obtain the visual modality feature F v and audio modal features F a , extract the time-aligned text features X through two layers of LSTM t The temporal relationship of t ;

[0021] Step 3: Based on the visual modality feature F v and audio modal features F a , obtain non-text modal features F av ;

[0022] Step 3.1: Targeting the visual modality feature F v , organize and condense information through the encoder of the first type L layer Transformer to obtain the concentrated visual modality features For audio modal features F a , organize and condense the information through the encoder of the first type L layer Transformer to obtain the concentrated audio modal features

[0023] Step 3.2: Get the shared token, which is represented by Among them, f represents the total amount of data in the shared Token; the shared Token F b and the concentrated audio modal features Splice and get Will Through the encoder of the first type L layer Transformer, we get Will and the concentrated visual modality features Splice and get Will Obtain non-text modality features through the encoder of the first L-layer Transformer

[0024] Step 4: Use the cross-modal Transformer encoder to transform the non-text features F av and text modality features F t Perform cross-modal attention interaction to obtain enhanced text features and enhanced non-text features

[0025] Step 5: Enhanced text features and enhanced non-text features Splice and get the fused feature R. The fused feature R is input into a fully connected network and the sentiment prediction result y of the input sample is output. This is achieved by the following formula:

[0026] y=W2R+b2;

[0027] Among them, W2 and b2 are the weight and bias of the fully connected layer;

[0028] Step 6: Calculate the intra-sample contrast loss, inter-sample contrast loss, and classification task loss.

[0029] Step 6.1: For all target samples in the current batch, the text modality feature F in the target sample t and non-text modality features F av , calculate the in-sample contrast loss value L intra ;

[0030] Step 6.2: For each sample, determine the negative sample set and positive sample set of the sample according to the label of the sample, and generate the negative sample pair set N according to the negative sample set i , generate a positive sample pair set P based on the positive sample set i , according to the negative sample set N i and the set of positive sample pairs P i , calculate the contrast loss value L between samples inter ;

[0031] Step 6.3: Based on the output sample and the sentiment prediction results corresponding to the input sample obtained in step 5, use the mean square error loss function to calculate the loss value L of the classification task pred ;

[0032] Step 7: According to the loss value L of the classification task pred , inter-sample contrast loss value L inter And the in-sample comparison loss value L intra , calculate the total loss value L all , which is specifically achieved through the following formula:

[0033] L all =L pred +λ intra L intra +λ inter L inter ;

[0034] Among them, λ intra and λ inter are the two contrastive learning loss weights;

[0035] According to the total loss value, the parameters of the feedforward network and the LSTM in step 2, the parameters of the encoder of the first type L-layer Transformer in step 3, the parameters of the cross-modal Transformer encoder in step 4, and the parameters of the fully connected network in step 5 are updated.

[0036] Optionally, step 1 specifically includes:

[0037] Step 1.1: Obtaining original features for multimodal sentiment analysis, wherein the original features include video features, acoustic features, and language features;

[0038] Step 1.2: For the video features, use FaceNet to process the video features to obtain visual features, for the acoustic features, use COVAERP to process the acoustic features to obtain audio features, and for the language features, use the BERT model to process the language features to obtain text features;

[0039] Step 1.3: Based on the timestamp, align the visual features, the audio features, and the text features in time sequence to obtain the time-aligned visual features X v , the audio feature X after time alignment a and the text feature X after time alignment t ;

[0040] Step 1.4: Temporally aligned visual features X v , audio features after time alignment X a and time-aligned text features X t As the input sample, obtain the label corresponding to the input sample, and use the label corresponding to the input sample as the output sample; the input sample and the output sample constitute the target sample, and multiple target samples constitute a sample set. According to a preset ratio, the sample set is divided into a training set, a validation set, and a test set.

[0041] Optionally, the visual modality feature F in step 2 v and audio modal features F a It is calculated by the following formula:

[0042] F v =ReLU(W1X v +b1);

[0043] F a =ReLU(W1X a +b1);

[0044] Among them, W1 is the weight of the feedforward network, b1 is the bias of the feedforward network;

[0045] The text modality feature F in step 2 t It is calculated by the following formula:

[0046] F t =LSTM2(LSTM1(X t )).

[0047] Optionally, the encoder of the first type of L-layer Transformer in step 3.1 includes L Transformer encoding layers, and each Transformer layer consists of a multi-head self-attention (MSA), a layer normalization (LN), and a feed-forward neural network (MLP);

[0048] Based on this, in step 3.1, the visual modality feature F v , organize and condense information through the encoder of the first type L layer Transformer to obtain the concentrated visual modality features This is achieved specifically through the following formula:

[0049]

[0050] in, is the visual modality feature output by the lth layer, is the intermediate variable of the lth layer, is the visual modality feature output by the l+1th layer;

[0051] Based on this, after the Lth layer, the concentrated visual modal features are obtained

[0052] Optionally, in step 3.1, the audio modal feature F a , organize and condense the information through the encoder of the first type L layer Transformer to obtain the concentrated audio modal features This is achieved specifically through the following formula:

[0053]

[0054] in, is the audio modality feature output by the lth layer, is the intermediate variable of the lth layer, is the audio modality feature output by the l+1th layer;

[0055] Based on this, after the Lth layer, the concentrated visual modal features are obtained

[0056] Optional, enhanced text features from step 4 It is calculated by the following formula:

[0057]

[0058] in, represents the enhanced text features output by the l+1th layer, represents the enhanced text features output by the lth layer, represents the enhanced non-text features output by the lth layer, Represents the intermediate variable of the l-th layer output.

[0059] Optional, non-text features enhanced in step 4 It is calculated by the following formula:

[0060]

[0061] in, represents the enhanced non-text features output by the l+1th layer, represents the enhanced text features output by the lth layer, represents the enhanced non-text features output by the lth layer, Represents the intermediate variable of the l-th layer output.

[0062] Optionally, step 6.1 is specifically implemented by the following formula:

[0063]

[0064] Among them, L intra is the intra-sample contrast loss value, N represents the total number of samples in the current batch, represents the text modality feature of the i-th sample in the current batch, represents the non-text modality feature of the i-th sample in the current batch, represents the non-text modal feature of the jth sample in the current batch, τ is the temperature parameter, express and The cosine similarity of .

[0065] Optionally, step 6.2 specifically includes:

[0066] Step 6.2.1: Based on the label of the sample, from all samples in the current batch, obtain multiple samples with the same label as the sample to form the positive sample set, and obtain multiple samples with different labels from the sample to form the negative sample set;

[0067] Step 6.2.2: Calculate the cosine similarity between the sample and each sample in the negative sample set, obtain the two samples with the highest cosine similarity as the negative sample pair, and obtain the two samples with the lowest cosine similarity as the negative sample pair, and obtain the negative sample pair set N i; Calculate the cosine similarity between the sample and each sample in the positive sample set, obtain the two samples with the highest cosine similarity as the positive sample pair, and obtain the positive sample pair set P i ;

[0068] Step 6.2.3: According to the negative sample set N i and the set of positive sample pairs P i , the calculation of the contrast loss value between samples is implemented by the following formula:

[0069]

[0070] Among them, L inter Indicates the contrast loss value between samples, N represents the total number of samples in the current batch, F i represents the i-th sample in the current batch, F c Represents the positive sample pair set P i The cth sample in F d Represents the negative sample pair set N i The dth sample in , τ is the temperature parameter.

[0071] Optionally, step 6.3 is specifically implemented by the following formula:

[0072]

[0073] Among them, y i represents the output sample, Indicates the sentiment prediction result corresponding to the input sample, L pred Represents the loss value of the classification task.

[0074] The beneficial effects of adopting the above technical solution are:

[0075] The present invention provides a multimodal sentiment analysis method for contrast-enhanced non-isomorphic modal fusion, which compensates for the difference in information density between visual modal features and audio modal features and text modalities. The present invention designs joint contrastive learning, and obtains intra-sample loss and inter-sample loss through intra-sample contrastive learning and inter-sample contrastive learning. The intra-sample contrastive learning aims to enhance inter-modal consistency, and inter-sample contrastive learning ensures the independence and uniqueness of each modal feature to avoid information loss caused by excessive fusion. Therefore, the present invention takes into account the complementarity and difference between modalities, compensates for the difference in information density between visual and audio modalities and text modalities, and improves the performance and generalization ability of the sentiment analysis model. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 Schematic diagram of a flow chart of a multimodal sentiment analysis method for contrast-enhanced non-isomorphic modality fusion in an embodiment of the present invention;

[0077] Figure 2 Schematic diagram of the process of obtaining non-text modality features in an embodiment of the present invention;

[0078] Figure 3 Schematic diagram of the cross-modal Transformer process in an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0080] In response to the problems existing in the prior art, the present invention proposes a contrast-enhanced non-isomorphic modal fusion multimodal sentiment analysis method (CHAMF) to address the deficiencies of the above-mentioned prior art. Existing studies often treat each modality equally, ignoring the difference in information density between visual and audio modalities and text modalities. Text is a signal generated by humans, which is highly semantic and information-intensive. In contrast, visual and audio modalities are mainly natural signals containing a large amount of spatial redundancy, most of which are low-level semantics and single features. If this difference is not processed in the modal fusion process, it will limit the performance of the fusion network. Therefore, this method designs a Transformer-based feature fusion module to first fuse the visual and audio modalities to make up for the difference in information density between them and the text modalities.

[0081] Existing methods for fusing different modalities often overemphasize complementarity and ignore differences between them, or focus on only one modality, failing to balance them. Therefore, this method designs a joint contrastive learning task to standardize the fusion process of non-text and text modalities. This includes within-sample contrastive learning to enhance inter-modal consistency and between-sample contrastive learning to ensure the independence and uniqueness of each modality's features, avoiding information loss caused by over-fusion.

[0082] The purpose of this invention is to provide an effective multimodal data fusion method for the field of multimodal sentiment analysis tasks, taking into account the complementarity and differences between modalities, and compensating for the difference in information density between visual and audio modalities and textual modalities, thereby improving the performance and generalization ability of sentiment analysis models.

[0083] Specifically, this method effectively improves the performance and efficiency of multimodal sentiment analysis through a series of innovative designs. The network structure of the method is as follows Figure 1As shown. The method consists of the following parts: (1) Modal feature modeling. Generate preliminary feature representations of each modality; (2) Non-text modal feature fusion. Design a Transformer-based feature fusion module to fuse the feature information of visual and audio modalities to obtain non-text modal features. This overcomes the difference in information density between visual and audio modalities and text modalities; (3) Input non-text modal and text modal features into the cross-modal Transformer to further promote the interaction and fusion of deep features between modalities; (4) Design a joint contrastive learning task: intra-sample contrastive learning and inter-sample contrastive learning to enhance the fusion of non-text modalities and text modalities. Intra-sample contrastive learning aims to enhance inter-modal consistency; inter-sample contrastive learning refers to contrastive learning between the same modal features of samples, which standardizes the intra-class aggregation and inter-class separation of modal features through more fine-grained comparison. (5) Fully connected output classification layer. The mutually enhanced features of non-text modalities and text modalities are input into the fully connected network to complete the sentiment classification task.

[0084] Specifically, the present invention provides a multimodal sentiment analysis method for contrast-enhanced heterogeneous modal fusion, combining Figure 1 , which may include the following steps:

[0085] Step 1: Construct a sample set, which includes multiple target samples. The target samples include input samples and output samples. The input samples include the temporally aligned visual features X v , audio features X after time alignment a and time-aligned text features X t , the output sample is the label corresponding to the input sample, the label is the emotion category corresponding to the input sample, the sample set is divided into a training set, a validation set and a test set;

[0086] Step 1.1: Obtaining original features for multimodal sentiment analysis, wherein the original features include video features, acoustic features, and language features; wherein the original features are obtained from a public dataset;

[0087] Step 1.2: For the video features, FaceNet is used to process the video features to obtain visual features, wherein FaceNet is an analysis tool based on the Facial Action Coding System (FACS), and 35 visual features are obtained by processing; for the acoustic features, COVAERP is used to process the acoustic features to obtain audio features, which are 74 dimensions; for the language features, BERT model is used to process the language features to obtain text features, which are 768 dimensions;

[0088] Step 1.3: Based on the timestamp, align the visual features, the audio features, and the text features in time sequence to obtain the time-aligned visual features X v , the audio feature X after time alignment a and the text feature X after time alignment t ;

[0089] Step 1.4: Temporally aligned visual features X v , audio features after time alignment X a and time-aligned text features X t As the input sample, obtain the label corresponding to the input sample, and use the label corresponding to the input sample as the output sample; the input sample and the output sample constitute the target sample, and multiple target samples constitute a sample set. According to a preset ratio, the sample set is divided into a training set, a validation set and a test set, for example, according to a ratio of 8:1:1.

[0090] Step 2: Through the feedforward network, the time-aligned visual features X in the target samples in the test set are v and the time-aligned audio features X a Processing is performed to obtain the visual modality feature F v and audio modal features F a , thus, the time-aligned visual feature X v and the aligned audio features X a Mapping to a new feature space for better fusion processing is done by the following formula:

[0091] F v =ReLU(W1X v +b1);

[0092] F a =ReLU(W1X a +b1);

[0093] Among them, W1 is the weight of the feedforward network, b1 is the bias of the feedforward network;

[0094] The aligned text features X are extracted through two layers of LSTM t The temporal relationship of the text can better capture the contextual information in the text and obtain the text modality feature F t , specifically calculated by the following formula:

[0095] F t =LSTM2(LSTM1(X t )).

[0096] Step 3: Based on the visual modality feature F vand audio modal features F a , obtain non-text modal features F av , combined with Figure 1 , it can also be understood that this step is carried out in the feature fusion module based on Transformer. Specifically, combined with Figure 2 , which may include the following steps:

[0097] Step 3.1: Targeting the visual modality feature F v , organize and condense information through the encoder of the first type L layer Transformer to obtain the concentrated visual modality features For audio modal features F a , organize and condense the information through the encoder of the first type L layer Transformer to obtain the concentrated audio modal features

[0098] The first type of L-layer Transformer encoder includes L Transformer encoding layers, and each Transformer layer consists of multi-head self-attention MSA, layer normalization LN and feedforward neural network MLP;

[0099] Based on this, in step 3.1, the visual modality feature F v , organize and condense information through the encoder of the first type L layer Transformer to obtain the concentrated visual modality features This is achieved specifically through the following formula:

[0100]

[0101]

[0102] in, is the visual modality feature output by the lth layer, is the intermediate variable of the lth layer, is the visual modality feature output by the l+1th layer;

[0103] Based on this, after the Lth layer, the concentrated visual modal features are obtained

[0104] Among them, for the audio modal feature F a , organize and condense the information through the encoder of the first type L layer Transformer to obtain the concentrated audio modal features This is achieved specifically through the following formula:

[0105]

[0106] in, is the audio modality feature output by the lth layer, is the intermediate variable of the lth layer, is the audio modality feature output by the l+1th layer;

[0107] Based on this, after the Lth layer, the concentrated visual modal features are obtained

[0108] Among them, MSA is the dot product attention, and the query, key, and value are all F m The linear projection can be expressed by the following formula:

[0109] MSA(X)=Attention(W Q X,W K X,W V X);

[0110] Among them, W Q represents the query weight, W K Represents the weight of the key, W V The weight of the value;

[0111] Step 3.2: Get the shared token, which is represented by Among them, f represents the total amount of data in the shared Token, f< <d m , that is to say, the length of the shared Token is much smaller than the dimension, and the shared information is carried between modalities by sharing the Token; the shared Token F b and the concentrated audio modal features Splice and get Will Through the encoder of the first type L layer Transformer, we get Will and the concentrated visual modality features Splice and get Will Obtain non-text modality features through the encoder of the first L-layer Transformer In this step, only necessary information can be shared, thereby improving the efficiency of inter-modal feature fusion and reducing computational complexity.

[0112] Step 4: Use the Cross-modal Transformer (CMT) encoder to transform the non-text features F av and text modality features F t Perform cross-modal attention interaction to obtain enhanced text features and enhanced non-text features The core idea of ​​CMT is to achieve information interaction between modalities through a cross-modal attention mechanism, thereby obtaining enhanced multi-modal feature representation. Unlike the traditional Transformer encoder, the multi-head attention mechanism in CMT is designed as cross-modal attention (MCA), and the calculation formula is: MCA(X,Y)=Attention(W Q X,W K Y,W V Y), in addition, CMT is a module composed of multiple stacked layers, the structure of each layer remains unchanged, and the input features are continuously updated in each layer to capture deeper cross-modal interaction information.

[0113] Specific, combined Figure 3 , the text features enhanced in step 4 It is calculated by the following formula:

[0114]

[0115] in, represents the enhanced text features output by the l+1th layer, represents the enhanced text features output by the lth layer, represents the enhanced non-text features output by the lth layer, Represents the intermediate variable of the l-th layer output.

[0116] Non-text features enhanced in step 4 It is calculated by the following formula:

[0117]

[0118] in, represents the enhanced non-text features output by the l+1th layer, represents the enhanced text features output by the lth layer, represents the enhanced non-text features output by the lth layer, Represents the intermediate variable of the l-th layer output.

[0119] like Figure 3 , this step is processed by L layers, each layer is processed according to Figure 3 The right side is processed, where, from bottom to top, Figure 3 The multi-head attention in the formula is MCA, the first residual connection and normalization are LN in the formula, the feedforward network is MLP in the formula, and the second residual connection and normalization are LN in MLP in the formula.

[0120] Furthermore, the above enhanced non-text features The calculation formula and enhanced text features The calculation formula can also be expressed as:

[0121]

[0122] Step 5: Enhanced text features and enhanced non-text features Splice and get the fused feature R. The fused feature R is input into a fully connected network and the sentiment prediction result y of the input sample is output. This is achieved by the following formula:

[0123] y=W2R+b2;

[0124] Among them, W2 and b2 are the weight and bias of the fully connected layer;

[0125] Step 6: Design a joint contrastive learning task to enhance the integration of non-text modalities and text modalities. The joint contrastive learning task consists of intra-sample contrastive learning and inter-sample contrastive learning tasks. From this, calculate the intra-sample contrastive loss, inter-sample contrastive loss, and classification task loss.

[0126] First, a dynamic queue is set up to store the text modality features and non-text modality features of historical batches, and the text modality dynamic queue Q is obtained. t and non-text modal dynamic queue Q av , text modal dynamic queue Q t and non-text modal dynamic queue Q av The definition is as follows:

[0127]

[0128]

[0129] in, Represents the text modality features of the i-th sample in the historical batch, K represents the non-text modality feature of the i-th sample in the historical batch. t represents the total number of text modality features in the text modality dynamic queue, K av represents the total number of non-text modality features in the non-text modality dynamic queue, and d represents the dimension;

[0130] Dynamic queue update rule: After each iteration, the features of the current batch are added to the queue while ensuring that the queue capacity does not exceed the maximum value:

[0131]

[0132] Where B represents the number of samples in the current batch. max_size represents the maximum capacity of the dynamic queue. When the capacity is exceeded, the oldest data is removed.

[0133] It should be noted that the i in the formula before the update is the i-th in the historical batch, and the i in the formula after the update is the i-th in the current batch. The two i's only represent the numbers, and the value ranges can be the same or different.

[0134] Step 6.1: In-sample contrastive learning by optimizing the text modality F of the same sample t and non-text modality features F av The similarity of the text modality feature F in the target sample is obtained by distinguishing the modality feature pairs between different samples, thereby improving the fusion effect between the text modality and the non-text modality. Specifically, for all target samples in the current batch, the text modality feature F in the target sample is obtained by t and non-text modality features F av , calculate the in-sample contrast loss value L intra ;

[0135] This is achieved specifically through the following formula:

[0136]

[0137] Among them, L intra is the intra-sample contrast loss value, N represents the total number of samples in the current batch, represents the text modality feature of the i-th sample in the current batch, represents the non-text modality feature of the i-th sample in the current batch, represents the non-text modality feature in the j-th sample in the current batch; τ is a temperature parameter used to control the distribution sharpness of contrastive learning; express and The cosine similarity of .

[0138] Step 6.2: Contrastive learning between samples refers to the contrastive learning between the same modal features of the samples to further optimize the feature expression. Specifically, for each sample, according to the label of the sample, determine the negative sample set and positive sample set of the sample, and generate the negative sample pair set N based on the negative sample set. i , generate a positive sample pair set P based on the positive sample set i , according to the negative sample set N i and the set of positive sample pairs P i , calculate the contrast loss value L between samples inter ;

[0139] Step 6.2.1: According to the label of the sample, obtain multiple samples with the same label as the sample from all samples in the current batch to form a positive sample set Get multiple samples whose labels are different from the label of the sample to form a negative sample set

[0140] Step 6.2.2: Calculate the cosine similarity between the sample and each sample in the negative sample set, obtain the two samples with the highest cosine similarity as the negative sample pair, and obtain the two samples with the lowest cosine similarity as the negative sample pair, and obtain the negative sample pair set N i ; Calculate the cosine similarity between the sample and each sample in the positive sample set, obtain the two samples with the highest cosine similarity as the positive sample pair, and obtain the positive sample pair set P i .

[0141] Step 6.2.3: According to the negative sample set N i and the set of positive sample pairs P i , the calculation of the contrast loss value between samples is implemented by the following formula:

[0142]

[0143] Among them, L inter Indicates the contrast loss value between samples, N represents the total number of samples in the current batch, F i represents the i-th sample in the current batch, F c Represents the positive sample pair set P i The cth sample in F d Represents the negative sample pair set N i The dth sample in , τ is the temperature parameter.

[0144] Step 6.3: Based on the output sample and the sentiment prediction results corresponding to the input sample obtained in step 5, use the mean square error loss function to calculate the loss value L of the classification task pred ;

[0145] This is achieved specifically through the following formula:

[0146]

[0147] Among them, y i represents the output sample, Indicates the sentiment prediction result corresponding to the input sample, L pred Represents the loss value of the classification task.

[0148] Step 7: According to the loss value L of the classification task pred , inter-sample contrast loss value L inter And the in-sample comparison loss value L intra , calculate the total loss value L all , which is specifically achieved through the following formula:

[0149] L all =L pred +λ intra L intra+λ inter L inter ;

[0150] Among them, λ intra and λ inter are two contrastive learning loss weights used to control the impact of the two losses;

[0151] According to the total loss value, the parameters of the feedforward network and the LSTM in step 2, the parameters of the encoder of the first type L-layer Transformer in step 3, the parameters of the cross-modal Transformer encoder in step 4, and the parameters of the fully connected network in step 5 are updated.

[0152] The key points of the present invention are: (1) The present invention proposes an innovative contrast-enhanced non-isomorphic modal fusion network that can efficiently integrate trimodal information to complete the multimodal sentiment analysis task. (2) A Transformer-based feature fusion module is designed to fuse visual and audio modal features to obtain non-text modal features, making up for the difference in information density between visual and audio modalities and text modalities. (3) A joint contrast learning task is designed in the network to effectively enhance the interaction of different modal information. This task not only promotes a closer integration of text and non-text information of the same sample in the feature space, but also ensures the uniqueness of each modality in its own feature space. (4) Experiments show that compared with the mainstream models in the field of multimodal sentiment analysis, this method has significant improvements in key indicators such as accuracy, indicating that this method can more effectively capture and fuse multimodal sentiment features, showing strong potential in multimodal sentiment analysis tasks.

[0153] In view of the above scheme, the present invention conducted the following experiments:

[0154] Dataset: The datasets used in this experiment, CMU-MOSI and CMU-MOSEI, are two popular benchmark datasets for multimodal sentiment analysis research. (1) CMU-MOSI consists of 93 Youtube movie review videos, which are divided into 2199 video clips. The sentiment scores of the video clips range from -3 to +3, representing very negative, negative, somewhat negative, neutral, somewhat positive, positive, and very positive. (2) The CMU-MOSEI dataset has a larger amount of data than the CMU-MOSI dataset. It consists of 5000 videos, including 1000 different speakers and 250 different topics. The videos are divided into 22856 video clips. The sentiment scores of the video clips are marked between -3 and +3, corresponding to the sentiment polarity from very negative to very positive. The division of the dataset is shown in Table 1

[0155] Table 1 Dataset division

[0156]

[0157] Evaluation Metrics: Two evaluation tasks were established: regression and classification. For the regression task, mean absolute error (MAE) and Pearson correlation coefficient (Corr) were used. For the classification task, binary classification accuracy (Acc-2) and F1 score were used as evaluation metrics.

[0158] (1) Mean absolute error (MAE): It represents the average absolute value of the error between the predicted value and the true value. It is used to measure the overall deviation of the model prediction. The smaller the value, the better the model performance.

[0159] (2) Pearson correlation coefficient (Corr): measures the linear correlation between the predicted value and the true value, with a value range of [-1, 1]. The closer the value is to 1, the stronger the linear relationship between the predicted value and the true value.

[0160] (3) Binary classification accuracy (Acc-2): After dividing emotions into positive and negative categories, the proportion of correctly classified samples to the total number of samples.

[0161] (4) F1-Score: It is the harmonic mean of precision and recall. The higher the value, the better the overall performance of the model for the classification task.

[0162] Hyperparameters and training: (1) CMU-MOSI dataset: The optimizer is Adam, the learning rate is set to 0.001, the batch_size is set to 16, the attention heads are set to 8, the Transformer is set to 4 layers, and the queue length in the joint contrastive learning task is set to 32. (2) CMU-MOSEI dataset: The optimizer is Adam, the learning rate is set to 0.001, the batch_size is set to 32, the attention heads are set to 8, the Transformer is set to 4 layers, and the queue length in the joint contrastive learning task is set to 64. λ in the training loss function intra Take 0.2, λ inter Take 0.1.

[0163] Table 2 Hyperparameter settings

[0164]

[0165] Experimental results:

[0166] Comparative analysis with the baseline model:

[0167] On the CMU-MOSI and CMU-MOSEI datasets, our proposed method significantly outperformed the CHAMF network and the Deep Emotional Arousal Network (DEAN) on all evaluation metrics. This may be because DEAN aims to provide a framework that guides learning along a human-like path to gradually acquire a complex understanding of emotions. However, its simple implementation of modal fusion fails to fully exploit deep feature interactions between modalities, limiting its performance. Our method, however, employs a Transformer-based feature fusion module and a joint contrastive learning task to more effectively extract and fuse multimodal features, significantly improving analysis capabilities. Our proposed method also outperformed the global acoustic feature-enhanced sequential perception network (AMOA) and the text-enhanced Transformer fusion network (TETFN) on all evaluation metrics. This may be because our method first fuses the visual and audio modalities to compensate for the information density difference between them and the text modality. Furthermore, the joint contrastive learning task enhances intermodal collaboration and feature interactions, fully unleashing the potential of multimodal information. Compared with the current state-of-the-art model CRNet, the proposed method still shows superior performance in all evaluation indicators. This further proves the effectiveness of the proposed network structure and fusion strategy, providing a new and more powerful solution for multimodal sentiment analysis tasks.

[0168] Table 3 Results of CMU-MOSI dataset

[0169]

[0170] Table 4 Results of CMU-MOSEI dataset

[0171]

[0172] Ablation experiment:

[0173] A series of ablation experiments were designed and implemented on the CMU-MOSEI dataset to verify the effectiveness and necessity of each module in the proposed method.

[0174] Single-modal experiments: We first conducted experiments focusing on extracted single-modal features, using only one modality: text, vision, or audio. A single-layer LSTM network was used to predict sentiment. The results showed that text significantly outperformed other modalities, confirming that text, as a human-generated signal, is highly semantically and information-dense. In contrast, visual and audio modalities exhibited lower single-modal prediction performance due to their greater spatial redundancy.

[0175] Non-text modality fusion: Using the Transformer-based feature fusion module proposed by CHAMF, we fuse visual and audio modalities to obtain non-text modality features, which are then used for prediction through a single-layer LSTM network. Experimental results show that the fused non-text modality performs nearly as well as the text modality in sentiment prediction, demonstrating that the Transformer-based feature fusion module can effectively bridge the information density gap between visual and audio modalities and text modalities.

[0176] Removing the Transformer-based feature fusion module (without AFM): When fusing visual and audio modal features, the Transformer-based feature fusion module was removed. Instead, the features of the two modalities were simply concatenated and then fused through a single fully connected network. Experimental results showed that the insufficient depth of feature fusion between audio and visual modalities reduced the ability to coordinate information between the modalities and significantly degraded prediction performance.

[0177] Without the contrast enhancement fusion module (CHF): When fusing non-text and text modalities, the contrast enhancement non-text and text modality fusion module was removed, and fusion was completed using only a single cross-modal Transformer layer. Results show that the model struggles to effectively distinguish features from samples of different categories, and lacks inter-modal collaboration and consistency, resulting in reduced prediction performance. This demonstrates the importance of joint contrastive learning for enhancing inter-modal interaction and consistency.

[0178] Table 5 Ablation experiment results of CMU-MOSEI dataset

[0179]

[0180]

[0181] The above description is merely a preferred embodiment of the present disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by mutually replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A multimodal sentiment analysis method for contrast-enhanced heterogeneous modal fusion, characterized in that: include: Step 1: Construct a sample set, which includes multiple target samples. The target samples include input samples and output samples. The input samples include the temporally aligned visual features X v , audio features X after time alignment a and time-aligned text features X t , the output sample is the label corresponding to the input sample, the label is the emotion category corresponding to the input sample, the sample set is divided into a training set, a validation set and a test set; Step 2: Through the feedforward network, the time-aligned visual features X in the target samples in the test set are v and the time-aligned audio features X a Processing is performed to obtain the visual modality feature F v and audio modal features F a , extract the time-aligned text features X through two layers of LSTM t The temporal relationship of t ; Step 3: Based on the visual modality feature F v and audio modal features F a , obtain non-text modal features F av ; Step 3.1: Targeting the visual modality feature F v , organize and condense information through the encoder of the first type L layer Transformer to obtain the concentrated visual modality features For audio modal features F a , organize and condense the information through the encoder of the first type L layer Transformer to obtain the concentrated audio modal features Step 3.2: Get the shared token, which is represented by Among them, f represents the total amount of data in the shared Token; the shared Token F b and the concentrated audio modal features Splice and get Will Through the encoder of the first type L layer Transformer, we get Will and the concentrated visual modality features Splice and get Will Obtain non-text modality features through the encoder of the first L-layer Transformer Step 4: Use the cross-modal Transformer encoder to transform the non-text features F av and text modality features F t Perform cross-modal attention interaction to obtain enhanced text features and enhanced non-text features Step 5: Enhanced text features and enhanced non-text features Splice and get the fused feature R. The fused feature R is input into a fully connected network and the sentiment prediction result y of the input sample is output. This is achieved by the following formula: y=W2R+b2; Among them, W2 and b2 are the weight and bias of the fully connected layer; Step 6: Calculate the intra-sample contrast loss, inter-sample contrast loss, and classification task loss. Step 6.1: For all target samples in the current batch, the text modality feature F in the target sample t and non-text modality features F av , calculate the in-sample contrast loss value L intra ; Step 6.2: For each sample, determine the negative sample set and positive sample set of the sample according to the label of the sample, and generate the negative sample pair set N according to the negative sample set i , generate a positive sample pair set P based on the positive sample set i , according to the negative sample set N i and the set of positive sample pairs P i , calculate the contrast loss value L between samples inter ; Step 6.3: Based on the output sample and the sentiment prediction results corresponding to the input sample obtained in step 5, use the mean square error loss function to calculate the loss value L of the classification task pred ; Step 7: According to the loss value L of the classification task pred , inter-sample contrast loss value L inter And the in-sample comparison loss value L intra , calculate the total loss value L all , which is specifically achieved through the following formula: L all =L pred +λ intra L intra +λ inter L inter ; Among them, λ intra and λ inter are the two contrastive learning loss weights; According to the total loss value, the parameters of the feedforward network and the LSTM in step 2, the parameters of the encoder of the first type L-layer Transformer in step 3, the parameters of the cross-modal Transformer encoder in step 4, and the parameters of the fully connected network in step 5 are updated.

2. The multimodal sentiment analysis method for contrast-enhanced heterogeneous modality fusion according to claim 1, characterized in that: Step 1 specifically includes: Step 1.1: Obtaining original features for multimodal sentiment analysis, wherein the original features include video features, acoustic features, and language features; Step 1.2: For the video features, use FaceNet to process the video features to obtain visual features, for the acoustic features, use COVAERP to process the acoustic features to obtain audio features, and for the language features, use the BERT model to process the language features to obtain text features; Step 1.3: Based on the timestamp, align the visual features, the audio features, and the text features in time sequence to obtain the time-aligned visual features X v , the audio feature X after time alignment a and the text feature X after time alignment t ; Step 1.4: Temporally aligned visual features X v , audio features after time alignment X a and time-aligned text features X t As the input sample, obtain the label corresponding to the input sample, and use the label corresponding to the input sample as the output sample; the input sample and the output sample constitute the target sample, and multiple target samples constitute a sample set. According to a preset ratio, the sample set is divided into a training set, a validation set, and a test set.

3. The multimodal sentiment analysis method for contrast-enhanced heterogeneous modality fusion according to claim 1, characterized in that: The visual modality feature F in step 2 v and audio modal features F a It is calculated by the following formula: F v =ReLU(W1X v +b1); F a =ReLU(W1X a +b1); Among them, W1 is the weight of the feedforward network, b1 is the bias of the feedforward network; The text modality feature F in step 2 t It is calculated by the following formula: F t =LSTM2(LSTM1(X t ))。 4. The method for multimodal sentiment analysis based on contrast-enhanced heterogeneous modality fusion according to claim 1, characterized in that: The first type of L-layer Transformer encoder in step 3.1 contains L Transformer encoding layers, each of which consists of a multi-head self-attention (MSA), a layer normalization (LN), and a feed-forward neural network (MLP). Based on this, in step 3.1, the visual modality feature F v , organize and condense information through the encoder of the first type L layer Transformer to obtain the concentrated visual modality features This is achieved specifically through the following formula: in, is the visual modality feature output by the lth layer, is the intermediate variable of the lth layer, is the visual modality feature output by the l+1th layer; Based on this, after the Lth layer, the concentrated visual modal features are obtained 5. The multimodal sentiment analysis method for contrast-enhanced heterogeneous modality fusion according to claim 1, characterized in that: In step 3.1, the audio modal feature F a , organize and condense the information through the encoder of the first type L layer Transformer to obtain the concentrated audio modal features This is achieved specifically through the following formula: in, is the audio modality feature output by the lth layer, is the intermediate variable of the lth layer, is the audio modality feature output by the l+1th layer; Based on this, after the Lth layer, the concentrated visual modal features are obtained 6. The method for multimodal sentiment analysis based on contrast-enhanced heterogeneous modality fusion according to claim 1, characterized in that: The enhanced text features in step 4 It is calculated by the following formula: in, represents the enhanced text features output by the l+1th layer, represents the enhanced text features output by the lth layer, represents the enhanced non-text features output by the lth layer, Represents the intermediate variable of the l-th layer output.

7. The multimodal sentiment analysis method for contrast-enhanced heterogeneous modality fusion according to claim 1, characterized in that: Non-text features enhanced in step 4 It is calculated by the following formula: in, represents the enhanced non-text features output by the l+1th layer, represents the enhanced text features output by the lth layer, represents the enhanced non-text features output by the lth layer, Represents the intermediate variable of the l-th layer output.

8. The method for multimodal sentiment analysis based on contrast-enhanced heterogeneous modality fusion according to claim 1, characterized in that: Step 6.1 is specifically implemented by the following formula: Among them, L intra is the intra-sample contrast loss value, N represents the total number of samples in the current batch, represents the text modality feature of the i-th sample in the current batch, represents the non-text modality feature of the i-th sample in the current batch, represents the non-text modal feature of the jth sample in the current batch, τ is the temperature parameter, express and The cosine similarity of .

9. The method for multimodal sentiment analysis based on contrast-enhanced heterogeneous modality fusion according to claim 1, characterized in that: Step 6.2 specifically includes: Step 6.2.1: Based on the label of the sample, from all samples in the current batch, obtain multiple samples with the same label as the sample to form the positive sample set, and obtain multiple samples with different labels from the sample to form the negative sample set; Step 6.2.2: Calculate the cosine similarity between the sample and each sample in the negative sample set, obtain the two samples with the highest cosine similarity as the negative sample pair, and obtain the two samples with the lowest cosine similarity as the negative sample pair, and obtain the negative sample pair set N i ; Calculate the cosine similarity between the sample and each sample in the positive sample set, obtain the two samples with the highest cosine similarity as the positive sample pair, and obtain the positive sample pair set P i ; Step 6.2.3: According to the negative sample set N i and the set of positive sample pairs P i , the calculation of the contrast loss value between samples is implemented by the following formula: Among them, L inter Indicates the contrast loss value between samples, N represents the total number of samples in the current batch, F i represents the i-th sample in the current batch, F c Represents the positive sample pair set P i The cth sample in F d Represents the negative sample pair set N i The dth sample in , τ is the temperature parameter.

10. The method for multimodal sentiment analysis based on contrast-enhanced heterogeneous modality fusion according to claim 1, characterized in that: Step 6.3 is specifically implemented by the following formula: Among them, y i represents the output sample, Indicates the sentiment prediction result corresponding to the input sample, L pred Represents the loss value of the classification task.

Images