Multi-modal sentiment analysis method for contrast enhancement of non-isomorphic modal fusion

Abstract

The invention provides a multi-modal sentiment analysis method for contrast enhancement non-isomorphic modal fusion, and relates to the technical field of sentiment analys.According to the method, a visual modal feature and an audio modal feature are fused to make up for the difference of information density with text modalities, and joint contrast learning is designed to make up for the difference of information density with text modalities. The loss in the samples, the loss between the samples and the contrast learning in the samples are obtained through the contrast learning in the samples and the contrast learning between the samples, so that the consistency between the modes and the contrast learning between the samples are enhanced, the independence and the uniqueness of each mode feature are guaranteed, and information loss caused by excessive fusion is avoided. According to the method, complementarity and difference between modes are considered, the difference of information density between the visual and audio modes and the text mode is made up, and the performance and generalization ability of the sentiment analysis model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of sentiment analysis, and in particular to a multimodal sentiment analysis method of contrast-enhanced non-isomorphic modality fusion. Background Art

[0002] Background With the development of social media platforms and the advancement of technology, people have begun to record videos and upload them to social media platforms such as Weibo, Douyin, YouTube or Twitter to express their opinions. More and more people share their opinions online in the form of videos, not just text. The diversification of information sources allows researchers to analyze emotions through modalities such as text, audio and vision, making Multimodal Sentiment Analysis (MSA) an important research field.

[0003] In multimodal sentiment analysis, audio can capture the speaker's tone and audio features; text provides semantic information of language content, where vocabulary, syntax and context can convey emotions; and video contains facial expressions, body language and background information. The three modal information comes from different sources, but the emotions expressed are indeed consistent and have a strong correlation, enriching the different perspectives of sentiment analysis. Different modalities are combined and verified with each other to make up for the limitations of each modality and improve the accuracy and comprehensiveness of the final sentiment analysis. According to previous work, it has indeed been proved that the accuracy of trimodal sentiment analysis is higher than that of single modality or even bimodality.

[0004] In fact, the information contained in different modal data is different, which researchers call the difference between modalities. At the same time, there is also relatively consistent information between modalities, which researchers call the complementarity between modalities. Considering the complementarity and difference of modalities at the same time and obtaining effective features has always been one of the challenges in the multimodal field. From the existing research situation, traditional text sentiment analysis only judges sentiment through words, phrases and the semantic associations between them. The model is not enough to identify complex emotional information based on the information contained in only one modality. In the case of trimodality, acoustic and visual information are added on the basis of text information. With the help of the association between multimodal data, the emotional information that may be hidden in the text data can be displayed, thereby achieving more accurate emotion recognition. This shows that the complementary information between modalities plays a vital role in prediction. Based on the inseparable complementarity and difference between different modalities, how to better assist each other and then fuse the information of each modality has become a research hotspot in multimodal sentiment analysis.

[0005] Inspired by the emotional arousal model in psychology, Zhang et al. proposed a deep emotional arousal network (DEAN) to simulate the entire communication process of humans under multimodal input. At the input end, the interaction between each two modalities is explored through an improved multi-head attention mechanism. Subsequently, the authors adopted a multimodal BiLSTM to extract context-related features of each modality through a bidirectional LSTM network and modeled the time series to simulate emotional consistency. The last module is a multimodal gating module, which implicitly performs the fusion between modalities by selecting the information to be output according to the importance of the modality. The functions of the perceptual analysis system, cognitive comparator, and activation mechanism in human psychological emotional arousal are simulated respectively.

[0006] Zhang F, Li X, Lim CP, et al.Deep emotional arousal network for multimodal sentiment analysis and emotion recognition[J], Information Fusion, 2022,88:296–304.

[0007] Li et al. proposed a global acoustic feature enhanced sequential perception network (AMOA). After extracting the features of each modality, a cross-modal transformer (CMT) was used for two-stage fusion. The first stage took the text modality as the core and fused the auditory modality, and the second stage added the visual modality. In order to reflect the overall acoustic information of the overall change of the tone, the authors added a global acoustic feature (GAF). And contrastive learning was used to align and connect the two features. Finally, the contrast loss and classification loss were added together to guide the model training, and classification was performed through a multi-layer perceptron layer.

[0008] Li Z, Zhou Y, Zhang W, et al. AMOA: Global acoustic feature enhanced modal-order-aware network for multimodal sentiment analysis, Proceedings of the 29th International Conference on Computational Linguistics, 2022, Gyeongju, Republic of Korea, 2022, 7136–7146.

[0009] The Text Feature Enhancement-based Transformer Fusion Network (TETFN) proposed by Wang et al. uses text to enhance the interaction of two cross-modal features to obtain an effective unified multimodal representation. It integrates text information into learning emotion-related non-text representations through text-based multi-head attention, and retains the difference information between modes through unimodal label prediction. In addition, the vision pre-training model Vision Transformer is used to extract visual features from the original video to preserve the global and local information of the face. The advantage of this model is that it can integrate discourse information and improve the effectiveness of non-text modalities in MSA while maintaining inter-modal and intra-modal relationships.

[0010] Wang D,Guo X,Tian Y,et al.TETFN:A text enhanced transformer fusionnetwork for multimodal sentiment analysis[J].Pattern Recognition,2023,136:109259.

[0011] Shi et al. proposed a co-spatial representation interaction network (CRNet) that uses different acoustic and visual representation subspaces to interact with language modalities. The video and audio modalities are separated into modality-invariant subspaces and modality-specific subspaces, and the corresponding modality-invariant representations and modality-specific representations are enhanced through a gradient-based representation enhancement mechanism. At the same time, a new multi-task strategy is designed to optimize the training process to improve the performance of different representation combinations from these two spaces.

[0012] Shi H,Pu Y,Zhao Z,et al.Co-space Representation Interaction Network for multimodal sentiment analysis[J].Knowledge-Based Systems,2024,283:111149.

[0013] Compared with single modality, multimodal data often contains richer emotional information. The core of multimodal sentiment analysis is how to effectively fuse multiple modal information together for classification. Many current fusion methods only use some simple fusion strategies, which are not scientific for the selected fusion features and ignore the complementarity between modal data and the importance of the emotional level of different modal data. However, in real environments, people's behaviors are often complex, and relying on simple model fusion strategies cannot achieve accurate emotional prediction.

[0014] The defect of the Deep Emotion Arousal Network (DEAN) is that the information density difference between the visual and audio modalities and the text modality is not explicitly considered in the preliminary feature modeling stage. The direct fusion interaction of the three may cause the large amount of noise information contained in the visual and audio modalities to interfere with the prediction results. Finally, the weight design of the gating mechanism may be too dependent on the data distribution and lacks explicit modeling of the differences and complementarities between the modalities, resulting in insufficient information utilization.

[0015] The defect of the global acoustic feature enhanced sequential perception network (AMOA) is that it adopts a two-level fusion strategy. The first stage integrates the acoustic modality with the text modality as the core, and the second stage adds the visual modality. Although this step-by-step fusion method can gradually enhance the text modality, it may also cause the feature information of the subsequent modality to be obscured by the previously fused modality, especially the importance of the visual modality may be underestimated.

[0016] The Text Feature Enhanced Transformer Fusion Network (TETFN) has the disadvantage that visual and audio modalities usually contain more redundant information and have lower information density. Relying solely on text guidance may not fully tap the potential emotion-related features contained in these modalities. The TETFN model only retains the difference information between modalities through unimodal label prediction, but in the fusion process, the modalities may be too dependent on each other, thereby weakening the difference and failing to fully preserve the relative independence between modalities.

[0017] The defect of the Co-spatial Representation Interaction Network (CRNet) is that its modal fusion strategy is relatively simple, directly fusing the text modality with the visual / acoustic modality without considering the difference in information density between different modalities, which may lead to the inability to fully explore the deep interaction information between modalities. Summary of the invention

[0018] In view of the shortcomings of the prior art, the purpose of the present invention is to propose a multimodal sentiment analysis method for contrast-enhanced non-isomorphic modal fusion, comprising:

[0019] Step 1: Construct a sample set, wherein the sample set includes multiple target samples, wherein the target samples include input samples and output samples, wherein the input samples include the temporally aligned visual features X v , audio features X after time alignment a And the time-aligned text feature X t , the output sample is the label corresponding to the input sample, the label is the emotion category corresponding to the input sample, and the sample set is divided into a training set, a validation set, and a test set;

[0020] Step 2: Through the feedforward network, the temporally aligned visual features X in the target samples in the test set are vAnd the time-aligned audio features X a Processing is performed to obtain the visual modality feature F v and audio modal features F a , extract the time-aligned text features X through two layers of LSTM t The temporal relationship of t ;

[0021] Step 3: Based on the visual modality feature F v and audio modal features F a , obtain the non-text modality feature F av ;

[0022] Step 3.1: Targeting the visual modality feature F v , the information is organized and concentrated through the encoder of the first type L-layer Transformer to obtain the concentrated visual modality features For the audio modal feature F a , the information is organized and concentrated through the encoder of the first type L-layer Transformer to obtain the concentrated audio modal features

[0023] Step 3.2: Get the shared token, which is represented by Where f represents the total amount of data in the shared Token; share the shared Token F b And the concentrated audio modal features Splice and get Will Through the encoder of the first type L layer Transformer, we get Will And the condensed visual modality features Splice and get Will Through the encoder of the first type L layer Transformer, non-text modality features are obtained

[0024] Step 4: Use a cross-modal Transformer encoder to train the non-text features F av and text modality feature F t Perform cross-modal attention interaction to obtain enhanced text features And the enhanced non-text features

[0025] Step 5: Enhanced text features And the enhanced non-text features Splice and get the fused feature R. The fused feature R is input into a fully connected network and the sentiment prediction result y of the input sample is output. This is achieved by the following formula:

[0026] y=W2R+b2;

[0027] Among them, W2 and b2 are the weight and bias of the fully connected layer;

[0028] Step 6: Calculate the intra-sample contrast loss, inter-sample contrast loss, and classification task loss.

[0029] Step 6.1: For all target samples in the current batch, the text modality feature F in the target sample t and non-text modality features F av , calculate the in-sample contrast loss value L intra ;

[0030] Step 6.2: For each sample, determine the negative sample set and positive sample set of the sample according to the label of the sample, and generate a negative sample pair set N according to the negative sample set i , generate a positive sample pair set P based on the positive sample set i , according to the negative sample set N i and the set of positive sample pairs P i , calculate the contrast loss value L between samples inter ;

[0031] Step 6.3: Based on the output sample and the sentiment prediction results corresponding to the input sample obtained in step 5, use the mean square error loss function to calculate the loss value L of the classification task pred ;

[0032] Step 7: According to the loss value L of the classification task pred , the contrast loss value L between samples inter And the sample comparison loss value L intra , calculate the total loss value L all , which is specifically achieved through the following formula:

[0033] L all =L pred +λ intra L intra +λ inter L inter ;

[0034] Among them, λ intra and λ inter are two contrastive learning loss weights;

[0035] According to the total loss value, the parameters of the feedforward network and the LSTM in step 2, the parameters of the encoder of the first type L-layer Transformer in step 3, the parameters of the cross-modal Transformer encoder in step 4, and the parameters of the fully connected network in step 5 are updated.

[0036] Optionally, step 1 specifically includes:

[0037] Step 1.1: Obtaining original features for multimodal sentiment analysis, wherein the original features include video features, acoustic features, and language features;

[0038] Step 1.2: For the video features, FaceNet is used to process the video features to obtain visual features, for the acoustic features, COVAERP is used to process the acoustic features to obtain audio features, and for the language features, the BERT model is used to process the language features to obtain text features;

[0039] Step 1.3: Based on the timestamp, align the visual features, the audio features and the text features in time series to obtain the time-aligned visual features X v , the audio feature X after time alignment a and the text feature X after time alignment t ;

[0040] Step 1.4: Temporally aligned visual features X v , audio features after time alignment X a And the time-aligned text feature X t as input samples, obtaining labels corresponding to the input samples, and using labels corresponding to the input samples as output samples; the input samples and the output samples constitute target samples, multiple target samples constitute a sample set, and the sample set is divided into a training set, a validation set, and a test set according to a preset ratio.

[0041] Optionally, the visual modality feature F in step 2 v and audio modal features F a It is calculated by the following formula:

[0042] F v =ReLU(W1X v +b1);

[0043] F a =ReLU(W1X a +b1);

[0044] Among them, W1 is the weight of the feedforward network, b1 is the bias of the feedforward network;

[0045] The text modality feature F in step 2 t It is calculated by the following formula:

[0046] F t =LSTM2(LSTM1(X t )).

[0047] Optionally, the encoder of the first type of L-layer Transformer in step 3.1 includes L Transformer encoding layers, and each Transformer layer consists of a multi-head self-attention MSA, a layer normalization LN, and a feed-forward neural network MLP;

[0048] Based on this, in step 3.1, the visual modality feature F v , the information is organized and concentrated through the encoder of the first type L-layer Transformer to obtain the concentrated visual modality features This is achieved specifically through the following formula:

[0049]

[0050] in, is the visual modality feature output by the lth layer, is the intermediate variable of the lth layer, is the visual modality feature output by the l+1th layer;

[0051] Based on this, after the Lth layer, the concentrated visual modal features are obtained

[0052] Optionally, in step 3.1, the audio modal feature F a , the information is organized and concentrated through the encoder of the first type L-layer Transformer to obtain the concentrated audio modal features This is achieved specifically through the following formula:

[0053]

[0054] in, is the audio modality feature output by the lth layer, is the intermediate variable of the lth layer, is the audio modality feature output by the l+1th layer;

[0055] Based on this, after the Lth layer, the concentrated visual modal features are obtained

[0056] Optionally, the text features enhanced in step 4 It is calculated by the following formula:

[0057]

[0058] in, represents the enhanced text features output by the l+1th layer, represents the enhanced text features output by the lth layer, represents the enhanced non-text features output by the lth layer, Represents the intermediate variable of the output of the lth layer.

[0059] Optionally, non-text features enhanced in step 4 It is calculated by the following formula:

[0060]

[0061] in, represents the enhanced non-text features output by the l+1th layer, represents the enhanced text features output by the lth layer, represents the enhanced non-text features output by the lth layer, Represents the intermediate variable of the output of the lth layer.

[0062] Optionally, step 6.1 is specifically implemented by the following formula:

[0063]

[0064] Among them, L intra is the intra-sample contrast loss value, N represents the total number of samples in the current batch, represents the text modality feature of the i-th sample in the current batch, represents the non-text modality feature in the i-th sample in the current batch, represents the non-text modal feature in the jth sample in the current batch, τ is the temperature parameter, express and The cosine similarity of .

[0065] Optionally, step 6.2 specifically includes:

[0066] Step 6.2.1: According to the label of the sample, among all the samples in the current batch, obtain multiple samples whose labels are the same as the label of the sample to form a positive sample set, and obtain multiple samples whose labels are different from the label of the sample to form a negative sample set;

[0067] Step 6.2.2: Calculate the cosine similarity between the sample and each sample in the negative sample set, obtain the two samples with the highest cosine similarity as the negative sample pair, and obtain the two samples with the lowest cosine similarity as the negative sample pair, and obtain the negative sample pair set N i; Calculate the cosine similarity between the sample and each sample in the positive sample set, obtain the two samples with the highest cosine similarity as the positive sample pair, and obtain the positive sample pair set P i ;

[0068] Step 6.2.3: According to the negative sample set N i and the set of positive sample pairs P i , the calculation of the contrast loss value between samples is implemented by the following formula:

[0069]

[0070] Among them, L inter represents the contrast loss value between samples, N represents the total number of samples in the current batch, and F i represents the i-th sample in the current batch, F c Represents the positive sample pair set P i The cth sample in F d Represents the negative sample pair set N i The dth sample in , τ is the temperature parameter.

[0071] Optionally, step 6.3 is specifically implemented by the following formula:

[0072]

[0073] Among them, y i represents the output sample, Indicates the sentiment prediction result corresponding to the input sample, L pred Represents the loss value of the classification task.

[0074] The beneficial effects of adopting the above technical solution are:

[0075] The present invention provides a contrast-enhanced non-isomorphic modal fusion multimodal sentiment analysis method, which compensates for the difference in information density between visual and audio modalities by fusing visual modal features with audio modal features. The present invention designs joint contrast learning, and obtains the loss within the sample and the loss between samples through contrast learning within the sample and contrast learning between samples. The contrast learning within the sample aims to enhance the consistency between modalities, and the contrast learning between samples ensures the independence and uniqueness of each modal feature to avoid information loss caused by excessive fusion. Therefore, the present invention takes into account the complementarity and difference between modalities, compensates for the difference in information density between visual and audio modalities and text modality, and improves the performance and generalization ability of the sentiment analysis model. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 Schematic diagram of a flow chart of a multimodal sentiment analysis method for contrast-enhanced non-isomorphic modality fusion in an embodiment of the present invention;

[0077] Figure 2 A schematic diagram of a process for obtaining non-text modality features in an embodiment of the present invention;

[0078] Figure 3 Schematic diagram of the process of the cross-modal Transformer in an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0080] In view of the problems existing in the prior art, the present invention proposes a contrast-enhanced non-isomorphic modal fusion multimodal sentiment analysis method (CHAMF) to address the deficiencies of the above-mentioned prior art. In existing studies, each modality is often treated equally, ignoring the difference in information density between visual and audio modalities and text modalities. Text is a signal generated by humans, which is highly semantic and information-intensive; in contrast, visual and audio modalities are mainly natural signals containing a large amount of spatial redundancy, most of which are low-level semantics and single features. If this difference is not processed in the modal fusion process, it will limit the performance of the fusion network. Therefore, this method designs a Transformer-based feature fusion module to first fuse the visual and audio modalities to make up for the difference in information density between the visual and audio modalities and the text modalities.

[0081] When fusing different modalities, existing methods often over-focus on the complementarity between modalities and ignore the differences between modalities; or only focus on one of the two, making it difficult to take both into account. Therefore, this method designs a joint contrastive learning task to regulate the fusion process of non-text modalities and text modalities. It includes contrastive learning within samples, which aims to enhance the consistency between modalities; contrastive learning between samples, which ensures the independence and uniqueness of each modality’s features and avoids information loss caused by excessive fusion.

[0082] The purpose of the present invention is to provide an effective multimodal data fusion method for the field of multimodal sentiment analysis tasks, taking into account the complementarity and differences between modalities, and compensating for the difference in information density between visual and audio modalities and textual modalities, thereby improving the performance and generalization ability of sentiment analysis models.

[0083] Specifically, this method effectively improves the performance and efficiency of multimodal sentiment analysis through a series of innovative designs. Figure 1As shown in the figure, the method consists of the following parts: (1) Modal feature modeling. Generate preliminary feature representations of each modality; (2) Non-text modal feature fusion. Design a Transformer-based feature fusion module to fuse the feature information of visual and audio modalities to obtain non-text modal features. This overcomes the difference in information density between visual and audio modalities and text modalities; (3) Input non-text modal and text modal features into the cross-modal Transformer to further promote the interaction and fusion of deep features between modalities; (4) Design a joint contrastive learning task: intra-sample contrastive learning and inter-sample contrastive learning to enhance the fusion of non-text modalities and text modalities. Intra-sample contrastive learning aims to enhance inter-modal consistency; inter-sample contrastive learning refers to contrastive learning between features of the same modality of samples. Through more fine-grained comparison, the intra-class aggregation and inter-class separation of modal features are standardized. (5) Fully connected output classification layer. The mutually enhanced features of non-text modalities and text modalities are input into the fully connected network to complete the sentiment classification task.

[0084] Specifically, the present invention provides a multimodal sentiment analysis method for contrast-enhanced non-isomorphic modality fusion, combining Figure 1 , which may include the following steps:

[0085] Step 1: Construct a sample set, wherein the sample set includes multiple target samples, wherein the target samples include input samples and output samples, wherein the input samples include the temporally aligned visual features X v , audio features X after time alignment a And the time-aligned text feature X t , the output sample is the label corresponding to the input sample, the label is the emotion category corresponding to the input sample, and the sample set is divided into a training set, a validation set, and a test set;

[0086] Step 1.1: Obtaining original features for multimodal sentiment analysis, wherein the original features include video features, acoustic features, and language features; wherein the original features are obtained from a public dataset;

[0087] Step 1.2: For the video features, FaceNet is used to process the video features to obtain visual features, wherein FaceNet is an analysis tool based on the Facial Action Coding System (FACS), and 35 visual features are obtained through processing; for the acoustic features, COVAERP is used to process the acoustic features to obtain audio features, and the audio features are 74 dimensions; for the language features, the BERT model is used to process the language features to obtain text features, and the text features are 768 dimensions;

[0088] Step 1.3: Based on the timestamp, align the visual features, the audio features and the text features in time series to obtain the time-aligned visual features X v , the audio feature X after time alignment a and the text feature X after time alignment t ;

[0089] Step 1.4: Temporally aligned visual features X v , audio features after time alignment X a And the time-aligned text feature X t as an input sample, obtaining a label corresponding to the input sample, and taking the label corresponding to the input sample as an output sample; the input sample and the output sample constitute a target sample, multiple target samples constitute a sample set, and the sample set is divided into a training set, a validation set, and a test set according to a preset ratio, for example, divided according to a ratio of 8:1:1.

[0090] Step 2: Through the feedforward network, the temporally aligned visual features X in the target samples in the test set are v And the time-aligned audio features X a Processing is performed to obtain the visual modality feature F v and audio modal features F a , thus, the temporally aligned visual feature X v and the aligned audio feature X a Mapped to a new feature space for better fusion processing, specifically calculated by the following formula:

[0091] F v =ReLU(W1X v +b1);

[0092] F a =ReLU(W1X a +b1);

[0093] Among them, W1 is the weight of the feedforward network, b1 is the bias of the feedforward network;

[0094] The aligned text features X are extracted through two layers of LSTM t The temporal relationship of , better captures the contextual information in the text, and obtains the text modality feature F t , which is calculated by the following formula:

[0095] F t =LSTM2(LSTM1(X t )).

[0096] Step 3: Based on the visual modality feature F vand audio modal features F a , obtain the non-text modality feature F av , combined with Figure 1 , it can also be understood that this step is performed in the Transformer-based feature fusion module. Specifically, combined with Figure 2 , which may include the following steps:

[0097] Step 3.1: Targeting the visual modality feature F v , the information is organized and concentrated through the encoder of the first type L-layer Transformer to obtain the concentrated visual modality features For the audio modal feature F a , the information is organized and concentrated through the encoder of the first type L-layer Transformer to obtain the concentrated audio modal features

[0098] The encoder of the first type of L-layer Transformer includes L Transformer encoding layers, and each Transformer layer consists of multi-head self-attention MSA, layer normalization LN and feed-forward neural network MLP;

[0099] Based on this, in step 3.1, the visual modality feature F v , the information is organized and concentrated through the encoder of the first type L-layer Transformer to obtain the concentrated visual modality features This is achieved specifically through the following formula:

[0100]

[0101]

[0102] in, is the visual modality feature output by the lth layer, is the intermediate variable of the lth layer, is the visual modality feature output by the l+1th layer;

[0103] Based on this, after the Lth layer, the concentrated visual modal features are obtained

[0104] Among them, for the audio modal feature F a , the information is organized and concentrated through the encoder of the first type L-layer Transformer to obtain the concentrated audio modal features This is achieved specifically through the following formula:

[0105]

[0106] in, is the audio modality feature output by the lth layer, is the intermediate variable of the lth layer, is the audio modality feature output by the l+1th layer;

[0107] Based on this, after the Lth layer, the concentrated visual modal features are obtained

[0108] Among them, MSA is the dot product attention, and the query, key, and value are all F m The linear projection can be expressed by the following formula:

[0109] MSA(X)=Attention(W Q X,W K X,W V X);

[0110] Among them, W Q represents the query weight, W K represents the weight of the key, W V The weight of the value;

[0111] Step 3.2: Get the shared token, which is represented by Where f represents the total amount of data in the shared Token, f< <d m , that is to say, the length of the shared token is much smaller than the dimension, and the shared token is used to carry shared information between modalities; the shared token F b And the concentrated audio modal features Splice and get Will Through the encoder of the first type L layer Transformer, we get Will And the condensed visual modality features Splice and get Will Through the encoder of the first type L layer Transformer, non-text modality features are obtained In this step, only necessary information can be shared, thereby improving the efficiency of feature fusion between modalities and reducing computational complexity.

[0112] Step 4: Use the Cross-Modal Transformer (CMT) encoder to train the non-text features F av and text modality feature F t Perform cross-modal attention interaction to obtain enhanced text features And the enhanced non-text features The core idea of ​​CMT is to achieve information interaction between modalities through a cross-modal attention mechanism, thereby obtaining enhanced multi-modal feature representation. Different from the traditional Transformer encoder, the multi-head attention mechanism in CMT is designed as cross-modal attention (MCA), and the calculation formula is: MCA(X,Y)=Attention(W Q X,W K Y,W V Y), in addition, CMT is a module composed of multiple stacked layers, the structure of each layer remains unchanged, and the input features are continuously updated in each layer to capture deeper cross-modal interaction information.

[0113] Specific, combined Figure 3 , the text features enhanced in step 4 It is calculated by the following formula:

[0114]

[0115] in, represents the enhanced text features output by the l+1th layer, represents the enhanced text features output by the lth layer, represents the enhanced non-text features output by the lth layer, Represents the intermediate variable of the output of the lth layer.

[0116] Non-text features enhanced in step 4 It is calculated by the following formula:

[0117]

[0118] in, represents the enhanced non-text features output by the l+1th layer, represents the enhanced text features output by the lth layer, represents the enhanced non-text features output by the lth layer, Represents the intermediate variable of the output of the lth layer.

[0119] like Figure 3 This step is processed by L layers, and each layer is processed according to Figure 3 The right side is processed, where, from bottom to top, Figure 3 The multi-head attention in is MCA in the formula, the first residual connection and normalization are LN in the formula, the feedforward network is MLP in the formula, and the second residual connection and normalization are LN in MLP in the formula.

[0120] Furthermore, the above enhanced non-text features The calculation formula and enhanced text features The calculation formula can also be expressed as:

[0121]

[0122] Step 5: Enhanced text features And the enhanced non-text features Splice and get the fused feature R. The fused feature R is input into a fully connected network and the sentiment prediction result y of the input sample is output. This is achieved by the following formula:

[0123] y=W2R+b2;

[0124] Among them, W2 and b2 are the weight and bias of the fully connected layer;

[0125] Step 6: Design a joint contrastive learning task to enhance the integration of non-text modalities and text modalities. The joint contrastive learning task consists of intra-sample contrastive learning and inter-sample contrastive learning tasks, thereby calculating the intra-sample contrast loss value, inter-sample contrast loss value, and classification task loss value;

[0126] First, a dynamic queue is set up to store the text modality features and non-text modality features of historical batches, and the text modality dynamic queue Q is obtained. t and non-text modal dynamic queue Q av , text modal dynamic queue Q t and non-text modal dynamic queue Q av The definition is as follows:

[0127]

[0128]

[0129] in, represents the text modality feature of the i-th sample in the historical batch, K represents the non-text modality feature of the i-th sample in the historical batch. t represents the total number of text modality features in the text modality dynamic queue, K av represents the total number of non-text modality features in the non-text modality dynamic queue, and d represents the dimension;

[0130] Dynamic queue update rule: After each iteration, add the features of the current batch to the queue while ensuring that the queue capacity does not exceed the maximum value:

[0131]

[0132] Where B represents the number of samples in the current batch. max_size represents the maximum capacity of the dynamic queue. When the capacity is exceeded, the earliest data is removed.

[0133] It should be noted that the i in the formula before the update is the i-th in the historical batch, and the i in the formula after the update is the i-th in the current batch. The two i's only represent the numbers, and the value ranges can be the same or different.

[0134] Step 6.1: In-sample contrastive learning by optimizing the text modality F of the same sample t With non-text modality feature F av , and distinguish the modal feature pairs between different samples, thereby improving the fusion effect between text modality and non-text modality. Specifically, for all target samples in the current batch, for the text modality feature F in the target sample t and non-text modality features F av , calculate the in-sample contrast loss value L intra ;

[0135] This is achieved specifically through the following formula:

[0136]

[0137] Among them, L intra is the intra-sample contrast loss value, N represents the total number of samples in the current batch, represents the text modality feature of the i-th sample in the current batch, represents the non-text modality feature in the i-th sample in the current batch, represents the non-text modality feature in the jth sample in the current batch; τ is the temperature parameter used to control the distribution sharpness of contrastive learning; express and The cosine similarity of .

[0138] Step 6.2: Contrastive learning between samples refers to the contrastive learning between the same modality features of samples to further optimize feature expression. Specifically, for each sample, according to the label of the sample, determine the negative sample set and positive sample set of the sample, and generate a negative sample pair set N according to the negative sample set. i , generate a positive sample pair set P based on the positive sample set i , according to the negative sample set N i and the set of positive sample pairs P i , calculate the contrast loss value L between samples inter ;

[0139] Step 6.2.1: According to the label of the sample, obtain multiple samples with the same label as the sample from all samples in the current batch to form a positive sample set Get multiple samples whose labels are different from the label of this sample to form a negative sample set

[0140] Step 6.2.2: Calculate the cosine similarity between the sample and each sample in the negative sample set, obtain the two samples with the highest cosine similarity as the negative sample pair, and obtain the two samples with the lowest cosine similarity as the negative sample pair, and obtain the negative sample pair set N i ; Calculate the cosine similarity between the sample and each sample in the positive sample set, obtain the two samples with the highest cosine similarity as the positive sample pair, and obtain the positive sample pair set P i .

[0141] Step 6.2.3: According to the negative sample set N i and the set of positive sample pairs P i , the calculation of the contrast loss value between samples is implemented by the following formula:

[0142]

[0143] Among them, L inter represents the contrast loss value between samples, N represents the total number of samples in the current batch, and F i represents the i-th sample in the current batch, F c Represents the positive sample pair set P i The cth sample in F d Represents the negative sample pair set N i The dth sample in , τ is the temperature parameter.

[0144] Step 6.3: Based on the output sample and the sentiment prediction results corresponding to the input sample obtained in step 5, use the mean square error loss function to calculate the loss value L of the classification task pred ;

[0145] This is achieved specifically through the following formula:

[0146]

[0147] Among them, y i represents the output sample, Indicates the sentiment prediction result corresponding to the input sample, L pred Represents the loss value of the classification task.

[0148] Step 7: According to the loss value L of the classification task pred , the contrast loss value L between samples inter And the sample comparison loss value L intra , calculate the total loss value L all , which is specifically achieved through the following formula:

[0149] L all =L pred +λ intra L intra+λ inter L inter ;

[0150] Among them, λ intra and λ inter are two contrastive learning loss weights, used to control the impact of the two losses;

[0151] According to the total loss value, the parameters of the feedforward network and the LSTM in step 2, the parameters of the encoder of the first type L-layer Transformer in step 3, the parameters of the cross-modal Transformer encoder in step 4, and the parameters of the fully connected network in step 5 are updated.

[0152] The key points of the present invention are: (1) The present invention proposes an innovative contrast-enhanced non-isomorphic modal fusion network, which can efficiently integrate trimodal information to complete the multimodal sentiment analysis task. (2) A Transformer-based feature fusion module is designed to fuse visual and audio modal features to obtain non-text modal features, making up for the difference in information density between visual and audio modalities and text modalities. (3) A joint contrast learning task is designed in the network to effectively enhance the interaction of different modal information. This task not only promotes a closer combination of text and non-text information of the same sample in the feature space, but also ensures the uniqueness of each modality in its own feature space. (4) Experiments show that compared with the mainstream models in the field of multimodal sentiment analysis, this method has significant improvements in key indicators such as accuracy, indicating that this method can more effectively capture and fuse multimodal sentiment features, showing great potential in multimodal sentiment analysis tasks.

[0153] In view of the above scheme, the present invention conducted the following experiments:

[0154] Datasets: The datasets used in this experiment, CMU-MOSI and CMU-MOSEI, are two popular benchmark datasets for multimodal sentiment analysis research. (1) CMU-MOSI consists of 93 Youtube movie review videos, which are divided into 2199 video clips. The sentiment scores of the video clips range from -3 to +3, representing very negative, negative, somewhat negative, neutral, somewhat positive, positive, and very positive. (2) Compared with the CMU-MOSI dataset, the CMU-MOSEI dataset has a larger amount of data. It consists of 5000 videos, including 1000 different speakers and 250 different topics. The videos are divided into 22856 video clips. The sentiment scores of the video clips are marked between -3 and +3, corresponding to sentiment polarities from very negative to very positive. The division of the dataset is shown in Table 1.

[0155] Table 1 Dataset division

[0156]

[0157] Evaluation indicators: Two evaluation tasks were established, namely regression and classification tasks. For the regression task, the mean absolute error (MAE) and Pearson correlation coefficient (Corr) were used. For the classification task, the binary classification accuracy (Acc-2) and F1 score were used as evaluation indicators.

[0158] (1) Mean absolute error (MAE): It represents the average absolute value of the error between the predicted value and the true value. It is used to measure the overall deviation of the model prediction. The smaller the value, the better the model performance.

[0159] (2) Pearson correlation coefficient (Corr): measures the linear correlation between the predicted value and the true value, with a value range of [-1,1]. The closer the value is to 1, the stronger the linear relationship between the predicted value and the true value.

[0160] (3) Binary classification accuracy (Acc-2): After classifying emotions into positive and negative categories, the ratio of correctly classified samples to the total number of samples.

[0161] (4) F1-Score: It is the harmonic mean of precision and recall. The higher the value, the better the overall performance of the model for the classification task.

[0162] Hyperparameters and training: (1) CMU-MOSI dataset: The optimizer is Adam, the learning rate is set to 0.001, the batch_size is set to 16, the attention head is set to 8, the Transformer is set to 4 layers, and the queue length in the joint contrastive learning task is set to 32. (2) CMU-MOSEI dataset: The optimizer is Adam, the learning rate is set to 0.001, the batch_size is set to 32, the attention head is set to 8, the Transformer is set to 4 layers, and the queue length in the joint contrastive learning task is set to 64. λ in the training loss function intra Take 0.2, λ inter Take 0.1.

[0163] Table 2 Hyperparameter settings

[0164]

[0165] Experimental results:

[0166] Comparative analysis with the baseline model:

[0167] On the CMU-MOSI and CMU-MOSEI datasets, the proposed method has achieved significant improvements in all evaluation indicators compared with the CHAMF network and the Deep Emotion Arousal Network (DEAN). This may be because DEAN aims to provide a framework that guides learning to gradually gain a complex understanding of emotions along a human-like path. However, its implementation of modal fusion is relatively simple and fails to fully explore the deep-level feature interactions between modalities, resulting in limited performance. The proposed method uses a Transformer-based feature fusion module and a joint contrastive learning task to more effectively extract and fuse multimodal features from a structural perspective, thereby significantly improving the analysis ability. Compared with the global acoustic feature enhanced sequential perception network (AMOA) and the text feature enhanced Transformer fusion network (TETFN), the proposed method also achieves better results in all evaluation indicators. The reason may be that the proposed method first fuses the visual and audio modalities to make up for the difference in information density between them and the text modality. At the same time, the joint contrastive learning task enhances the collaboration and feature interaction between modalities and fully releases the potential information of multimodal. Compared with the current advanced model CRNet, the proposed method still shows better performance in all evaluation indicators. This further proves the effectiveness of the network structure and fusion strategy proposed in this paper, and provides a new and more powerful solution for multimodal sentiment analysis tasks.

[0168] Table 3 Results of CMU-MOSI dataset

[0169]

[0170] Table 4 Results of CMU-MOSEI dataset

[0171]

[0172] Ablation experiment:

[0173] A series of ablation experiments were designed and implemented on the CMU-MOSEI dataset to verify the effectiveness and necessity of each module in the proposed method.

[0174] Single-modal experiment: First, we conducted an experiment on the extracted single-modal features, using only one of the modalities, text, vision, or audio, to predict sentiment through a layer of LSTM network. The experimental results show that the prediction performance of the text modality is significantly better than that of other modalities, which verifies that text, as a human-generated signal, has the characteristics of high semantics and information density. In contrast, the performance of single-modal prediction is lower for visual and audio modalities because they contain more spatial redundancy.

[0175] Non-text modality fusion: The Transformer-based feature fusion module proposed by CHAMF is used to fuse the visual and audio modalities to obtain non-text modality features, and then predict them through a layer of LSTM network. The experimental results show that the performance of the fused non-text modality in sentiment prediction is close to that of the text modality, proving that the Transformer-based feature fusion module can effectively make up for the difference in information density between the visual and audio modalities and the text modality.

[0176] Remove the Transformer-based feature fusion module (w / o AFM): When fusing visual and audio modal features, remove the Transformer-based feature fusion module and instead simply concat the two modal features and fuse them through a fully connected network. Experimental results show that the feature fusion depth of audio and visual modalities is insufficient, resulting in a decrease in the ability to coordinate information between modalities and a significant reduction in prediction performance.

[0177] Remove contrast enhancement fusion module (w / o CHF): When fusing non-text modalities with text modalities, the contrast enhancement non-text and text modal fusion module is removed, and the fusion is completed through only one layer of cross-modal Transformer. The results show that the model is difficult to effectively distinguish the features of samples of different categories, and the lack of coordination and consistency between modalities leads to a decrease in prediction performance. This verifies the importance of joint contrastive learning tasks for enhancing interaction and consistency between modalities.

[0178] Table 5 Ablation experiment results of CMU-MOSEI dataset

[0179]

[0180]

[0181] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A multimodal sentiment analysis method for contrast-enhanced non-isomorphic modality fusion, characterized in that: include: Step 1: Construct a sample set, wherein the sample set includes multiple target samples, wherein the target samples include input samples and output samples, wherein the input samples include the temporally aligned visual features X v , audio features X after time alignment a And the time-aligned text feature X t , the output sample is the label corresponding to the input sample, the label is the emotion category corresponding to the input sample, and the sample set is divided into a training set, a validation set, and a test set; Step 2: Through the feedforward network, the temporally aligned visual features X in the target samples in the test set are v And the time-aligned audio features X a Processing is performed to obtain the visual modality feature F v and audio modal features F a , extract the time-aligned text features X through two layers of LSTM t The temporal relationship of t ; Step 3: Based on the visual modality feature F v and audio modal features F a , obtain the non-text modality feature F av ; Step 3.1: Targeting the visual modality feature F v , the information is organized and concentrated through the encoder of the first type L-layer Transformer to obtain the concentrated visual modality features For the audio modal feature F a , the information is organized and concentrated through the encoder of the first type L-layer Transformer to obtain the concentrated audio modal features Step 3.2: Get the shared token, which is represented by Where f represents the total amount of data in the shared Token; share the shared Token F b And the concentrated audio modal features Splice and get Will Through the encoder of the first type L layer Transformer, we get Will And the condensed visual modality features Splice and get Will Through the encoder of the first type L layer Transformer, non-text modality features are obtained Step 4: Use a cross-modal Transformer encoder to train the non-text features F av and text modality feature F t Perform cross-modal attention interaction to obtain enhanced text features And the enhanced non-text features Step 5: Enhanced text features And the enhanced non-text features Splice and get the fused feature R. The fused feature R is input into a fully connected network and the sentiment prediction result y of the input sample is output. This is achieved by the following formula: y=W2R+b2; Among them, W2 and b2 are the weight and bias of the fully connected layer; Step 6: Calculate the intra-sample contrast loss, inter-sample contrast loss, and classification task loss. Step 6.1: For all target samples in the current batch, the text modality feature F in the target sample t and non-text modality features F av , calculate the in-sample contrast loss value L intra ; Step 6.2: For each sample, determine the negative sample set and positive sample set of the sample according to the label of the sample, and generate a negative sample pair set N according to the negative sample set i , generate a positive sample pair set P based on the positive sample set i , according to the negative sample set N i and the set of positive sample pairs P i , calculate the contrast loss value L between samples inter ; Step 6.3: Based on the output sample and the sentiment prediction results corresponding to the input sample obtained in step 5, use the mean square error loss function to calculate the loss value L of the classification task pred ; Step 7: According to the loss value L of the classification task pred , the contrast loss value L between samples inter And the sample comparison loss value L intra , calculate the total loss value L all , which is specifically achieved through the following formula: L all =L pred +λ intra L intra +λ inter L inter ; Among them, λ intra and λ inter are two contrastive learning loss weights; According to the total loss value, the parameters of the feedforward network and the LSTM in step 2, the parameters of the encoder of the first type L-layer Transformer in step 3, the parameters of the cross-modal Transformer encoder in step 4, and the parameters of the fully connected network in step 5 are updated.

2. The method for multimodal sentiment analysis based on contrast-enhanced non-isomorphic modality fusion according to claim 1, characterized in that: Step 1 specifically includes: Step 1.1: Obtaining original features for multimodal sentiment analysis, wherein the original features include video features, acoustic features, and language features; Step 1.2: For the video features, FaceNet is used to process the video features to obtain visual features, for the acoustic features, COVAERP is used to process the acoustic features to obtain audio features, and for the language features, the BERT model is used to process the language features to obtain text features; Step 1.3: Based on the timestamp, align the visual features, the audio features and the text features in time series to obtain the time-aligned visual features X v , the audio feature X after time alignment a and the text feature X after time alignment t ; Step 1.4: Temporally aligned visual features X v , audio features after time alignment X a And the time-aligned text feature X t as input samples, obtaining labels corresponding to the input samples, and using labels corresponding to the input samples as output samples; the input samples and the output samples constitute target samples, multiple target samples constitute a sample set, and the sample set is divided into a training set, a validation set, and a test set according to a preset ratio.

3. The method for multimodal sentiment analysis based on contrast-enhanced non-isomorphic modality fusion according to claim 1, characterized in that: The visual modality feature F in step 2 v and audio modal features F a It is calculated by the following formula: F v =ReLU(W1X v +b1); F a =ReLU(W1X a +b1); Among them, W1 is the weight of the feedforward network, b1 is the bias of the feedforward network; The text modality feature F in step 2 t It is calculated by the following formula: F t =LSTM2(LSTM1(X t ))。 4. The method for multimodal sentiment analysis based on contrast-enhanced non-isomorphic modality fusion according to claim 1, characterized in that: In step 3.1, the encoder of the first type of L-layer Transformer includes L Transformer encoding layers, and each Transformer layer consists of multi-head self-attention MSA, layer normalization LN and feed-forward neural network MLP; Based on this, in step 3.1, the visual modality feature F v , the information is organized and concentrated through the encoder of the first type L-layer Transformer to obtain the concentrated visual modality features This is achieved specifically through the following formula: in, is the visual modality feature output by the lth layer, is the intermediate variable of the lth layer, is the visual modality feature output by the l+1th layer; Based on this, after the Lth layer, the concentrated visual modal features are obtained 5. The method for multimodal sentiment analysis based on contrast-enhanced non-isomorphic modality fusion according to claim 1, characterized in that: In step 3.1, for the audio modal feature F a , the information is organized and concentrated through the encoder of the first type L-layer Transformer to obtain the concentrated audio modal features This is achieved specifically through the following formula: in, is the audio modality feature output by the lth layer, is the intermediate variable of the lth layer, is the audio modality feature output by the l+1th layer; Based on this, after the Lth layer, the concentrated visual modal features are obtained 6. The method for multimodal sentiment analysis based on contrast-enhanced non-isomorphic modality fusion according to claim 1, characterized in that: The enhanced text features in step 4 It is calculated by the following formula: in, represents the enhanced text features output by the l+1th layer, represents the enhanced text features output by the lth layer, represents the enhanced non-text features output by the lth layer, Represents the intermediate variable of the output of the lth layer.

7. The method for multimodal sentiment analysis based on contrast-enhanced non-isomorphic modality fusion according to claim 1, characterized in that: Non-text features enhanced in step 4 It is calculated by the following formula: in, represents the enhanced non-text features output by the l+1th layer, represents the enhanced text features output by the lth layer, represents the enhanced non-text features output by the lth layer, Represents the intermediate variable of the output of the lth layer.

8. The method for multimodal sentiment analysis with contrast enhancement and non-isomorphic modality fusion according to claim 1, characterized in that: Step 6.1 is specifically implemented by the following formula: Among them, L intra is the intra-sample contrast loss value, N represents the total number of samples in the current batch, represents the text modality feature of the i-th sample in the current batch, represents the non-text modality feature in the i-th sample in the current batch, represents the non-text modal feature in the jth sample in the current batch, τ is the temperature parameter, express and The cosine similarity of .

9. The method for multimodal sentiment analysis based on contrast-enhanced non-isomorphic modality fusion according to claim 1, characterized in that: Step 6.2 specifically includes: Step 6.2.1: According to the label of the sample, among all the samples in the current batch, obtain multiple samples whose labels are the same as the label of the sample to form a positive sample set, and obtain multiple samples whose labels are different from the label of the sample to form a negative sample set; Step 6.2.2: Calculate the cosine similarity between the sample and each sample in the negative sample set, obtain the two samples with the highest cosine similarity as the negative sample pair, and obtain the two samples with the lowest cosine similarity as the negative sample pair, and obtain the negative sample pair set N i ; Calculate the cosine similarity between the sample and each sample in the positive sample set, obtain the two samples with the highest cosine similarity as the positive sample pair, and obtain the positive sample pair set P i ; Step 6.2.3: According to the negative sample set N i and the set of positive sample pairs P i , the calculation of the contrast loss value between samples is implemented by the following formula: Among them, L inter represents the contrast loss value between samples, N represents the total number of samples in the current batch, and F i represents the i-th sample in the current batch, F c Represents the positive sample pair set P i The cth sample in F d Represents the set of negative sample pairs N i The dth sample in , τ is the temperature parameter.

10. The method for multimodal sentiment analysis with contrast enhancement and non-isomorphic modality fusion according to claim 1, characterized in that: Step 6.3 is specifically implemented by the following formula: Among them, y i represents the output sample, Indicates the sentiment prediction result corresponding to the input sample, L pred Represents the loss value of the classification task.

Images