Multi-modal sentiment analysis method and device

Abstract

The invention discloses a multi-modal sentiment analysis method and device, and relates to the technical field of sentiment analysis, and the method comprises the steps: carrying out the feature extraction and modal alignment of to-be-detected multi-modal sentiment data, and determining an aligned text feature, an aligned video feature and an aligned audio feature; performing feature enhancement on the aligned video features and the aligned audio features based on the aligned text features to generate enhanced video features and enhanced audio features; performing cross-modal cross attention fusion by adopting the aligned text features, the enhanced video features and the enhanced audio features, and constructing interactive text features, interactive video features and interactive audio features; and performing later fusion on the interactive text features, the interactive video features and the interactive audio features, performing emotion prediction, and outputting an emotion analysis result. The feature quality of audio and video modals is improved by adopting an audio and video feature enhancement mechanism, and cross-modal feature interaction and multi-modal late fusion are effectively combined, so that the reliability of multi-modal sentiment analysis is improved on the whole.

Description

Technical Field

[0001] The present invention relates to the technical field of sentiment analysis, and in particular to a multimodal sentiment analysis method and device. Background Art

[0002] With the rapid development of artificial intelligence technology, affective computing, as an important part of human-computer interaction, is becoming one of the research hotspots. As a core aspect of affective computing, affective analysis aims to automatically identify an individual's affective state by analyzing signals such as voice, facial expressions, gestures, and language text. Multimodal Sentiment Analysis (MSA) mainly combines different modal signals to obtain richer affective information, which has been proven to effectively improve the accuracy of affective analysis.

[0003] Most of the current multimodal sentiment analysis methods rely on early fusion, that is, splicing different modal features in the input layer or close to the input layer. Although this method can capture the association between multimodal information, its fusion at the feature level has problems of information redundancy and sensitivity to noise, and fails to fully utilize the complementary information between different modalities, resulting in low reliability of multimodal sentiment analysis. Summary of the invention

[0004] The present invention provides a multimodal sentiment analysis method and device, which solves the technical problem that the current multimodal sentiment analysis method fails to fully utilize the complementary information between different modalities, resulting in low reliability of multimodal sentiment analysis.

[0005] A first aspect of the present invention provides a multimodal sentiment analysis method, comprising:

[0006] Acquire multimodal emotion data to be tested, and input the multimodal emotion data to be tested into a trained multimodal emotion analysis network; the trained multimodal emotion analysis network includes a feature extraction aligner, an audio and video feature enhancer, a cross-modal converter and an emotion predictor;

[0007] Inputting the multimodal emotion data to be tested into the feature extraction aligner to perform feature extraction and modality alignment, and determining alignment text features, alignment video features, and alignment audio features;

[0008] Based on the aligned text features, the aligned video features and the aligned audio features are enhanced by the audio and video feature enhancer to generate enhanced video features and enhanced audio features;

[0009] The aligned text features, the enhanced video features and the enhanced audio features are used by the cross-modal converter to perform cross-modal cross-attention fusion to construct interactive text features, interactive video features and interactive audio features;

[0010] The interactive text features, the interactive video features and the interactive audio features are post-fused to perform sentiment prediction and output a sentiment analysis result.

[0011] Optionally, the step of inputting the multimodal emotion data to be tested into the feature extraction aligner to perform feature extraction and modality alignment, and determining the aligned text features, aligned video features, and aligned audio features, comprises:

[0012] Obtaining the text modality emotion data to be tested, the video modality emotion data to be tested, and the audio modality emotion data to be tested;

[0013] Using a text extraction model to extract features from the text modality sentiment data to be tested, and determining original text features;

[0014] Extracting features of the video modality emotion data to be tested through a video extraction model to generate original video features;

[0015] Extracting features of the audio modality emotion data to be tested based on the audio extraction model to construct original audio features;

[0016] The original text features, the original video features and the original audio features are respectively input into a one-dimensional temporal convolutional layer for modality alignment, and the corresponding aligned text features, aligned video features and aligned audio features are output.

[0017] Optionally, the step of enhancing the aligned video features and the aligned audio features by the audio and video feature enhancer based on the aligned text features to generate enhanced video features and enhanced audio features includes:

[0018] The aligned text features, the aligned video features and the aligned audio features are respectively encoded by a Relu activation function, and an encoded text feature, an encoded video feature and an encoded audio feature are correspondingly generated;

[0019] Residual connections are respectively performed on the aligned text features and the encoded text features, the aligned video features and the encoded video features, and the aligned audio features and the encoded audio features, and residual text features, residual video features, and residual audio features are output;

[0020] Respectively splicing the residual text feature with the residual video feature, and the residual text feature with the residual audio feature, and correspondingly determining a spliced ​​text video feature and a spliced ​​text audio feature;

[0021] Based on the Sigmoid activation function, the weights of the spliced ​​text video features and the spliced ​​text audio features are adaptively allocated, and the video weight and the audio weight are output;

[0022] Determine a weighted video feature and a weighted audio feature by respectively multiplying the residual video feature by the video weight and the residual audio feature by the audio weight element by element;

[0023] After multiplying the residual text feature by the complement of the video weight element by element, the residual text feature is added element by element to the weighted video feature to generate an auxiliary video feature;

[0024] Multiplying the residual text feature by the complement of the audio weight element by element, and adding the residual text feature to the weighted audio feature element by element to construct an auxiliary audio feature;

[0025] The auxiliary video feature and the aligned video feature, and the auxiliary audio feature and the aligned audio feature are respectively multiplied element by element to output an enhanced video feature and an enhanced audio feature.

[0026] Optionally, the post-fusion of the interactive text features, the interactive video features and the interactive audio features to perform sentiment prediction and output a sentiment analysis result includes:

[0027] The interactive text feature, the interactive video feature and the interactive audio feature are concatenated and low-dimensionally mapped to output a low-dimensional fusion feature;

[0028] A Relu activation function is used to perform nonlinear mapping on the low-dimensional fusion features, and output late fusion features;

[0029] The sentiment prediction layer performs sentiment prediction on the late fusion features to determine the sentiment analysis result.

[0030] Optionally, it also includes:

[0031] Obtain training multimodal sentiment data and real sentiment analysis results;

[0032] Inputting the training multimodal emotion data into the multimodal emotion analysis network to be trained, determining the training interactive text features, training interactive video features, training interactive audio features and training emotion analysis results, and calculating the overall loss function value in combination with the real emotion analysis results;

[0033] The multimodal sentiment analysis network to be trained is iteratively optimized according to the overall loss function value until the overall loss function value converges, thereby determining a trained multimodal sentiment analysis network.

[0034] Optionally, the process of calculating the overall loss function value includes:

[0035] ;

[0036] In the formula, is the overall loss function, is the task loss function, is the weight parameter of contrast loss, is the contrast loss function;

[0037] The contrast loss functions include:

[0038] ;

[0039] in, ;

[0040] In the formula, is the distance-aware contrast loss function, It is the fusion feature of interactive audio features and interactive text features. It is the fusion feature of interactive video features and interactive text features. is the cosine similarity score function, is the index of the anchor sample, is the index of the positive sample, is the index of the negative sample, is the index set of positive samples selected according to the label distance, is the temperature parameter;

[0041] The task loss functions include:

[0042] ,or, ;

[0043] In the formula, is the batch size, For the The real sentiment analysis results, For the Training sentiment analysis results.

[0044] A second aspect of the present invention provides a multimodal sentiment analysis device, comprising:

[0045] A data acquisition module, used for acquiring multimodal emotion data to be tested, and inputting the multimodal emotion data to be tested into a trained multimodal emotion analysis network; the trained multimodal emotion analysis network includes a feature extraction aligner, an audio and video feature enhancer, a cross-modal converter and an emotion predictor;

[0046] A feature alignment module, used for inputting the multimodal emotion data to be tested into the feature extraction aligner for feature extraction and modality alignment, and determining alignment text features, alignment video features and alignment audio features;

[0047] A feature enhancement module, used for enhancing the aligned video feature and the aligned audio feature through the audio and video feature enhancer based on the aligned text feature, to generate enhanced video feature and enhanced audio feature;

[0048] A feature cross-module, configured to perform cross-modal cross-attention fusion using the aligned text features, the enhanced video features, and the enhanced audio features through the cross-modal converter to construct interactive text features, interactive video features, and interactive audio features;

[0049] The emotion prediction module is used to perform emotion prediction after post-fusion of the interactive text features, the interactive video features and the interactive audio features, and output the emotion analysis result.

[0050] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the multimodal sentiment analysis method as described in any one of the above items.

[0051] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, the multimodal sentiment analysis method as described in any one of the above items is implemented.

[0052] A fifth aspect of the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the multimodal sentiment analysis method as described in any one of the above items.

[0053] It can be seen from the above technical solutions that the present invention has the following advantages:

[0054] The above scheme of the present invention provides a multimodal sentiment analysis method, including: obtaining multimodal sentiment data to be tested, and inputting the multimodal sentiment data to be tested into a trained multimodal sentiment analysis network; the trained multimodal sentiment analysis network includes a feature extraction aligner, an audio and video feature enhancer, a cross-modal converter and a sentiment predictor; inputting the multimodal sentiment data to be tested into the feature extraction aligner for feature extraction and modal alignment, and determining aligned text features, aligned video features and aligned audio features; based on the aligned text features, the audio and video feature enhancer is used to enhance the aligned video features and the aligned audio features respectively, to generate enhanced video features and enhanced audio features; the aligned text features, enhanced video features and enhanced audio features are used to perform cross-modal cross-attention fusion through the cross-modal converter to construct interactive text features, interactive video features and interactive audio features; the interactive text features, interactive video features and interactive audio features are post-fused for sentiment prediction, and the sentiment analysis results are output. Based on the above scheme, according to multiple modal information, the audio and video feature enhancement mechanism is adopted to improve the feature quality of audio and video modalities, and effectively combine cross-modal feature interaction and multimodal late fusion, which not only ensures the deep interaction between different modal information, but also can retain the independence of each modal feature to the greatest extent, which helps to improve the accuracy and robustness of multimodal sentiment analysis, enhance the ability to capture complex emotional expressions, thereby improving the reliability of multimodal sentiment analysis as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0056] Figure 1 A flowchart of the steps of a multimodal sentiment analysis method provided by an embodiment of the present invention;

[0057] Figure 2 A network architecture diagram of a multimodal sentiment analysis network provided by an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of the structure of an audio and video feature enhancer provided by an embodiment of the present invention;

[0059] Figure 4 A schematic diagram of the structure of the cross-modal converter provided by an embodiment of the present invention taking the text modality as an example;

[0060] Figure 5 A structural block diagram of a multimodal sentiment analysis device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The embodiments of the present invention provide a multimodal sentiment analysis method and device, which are used to solve the technical problem that the current multimodal sentiment analysis method fails to fully utilize the complementary information between different modalities, resulting in low reliability of multimodal sentiment analysis.

[0062] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] See also Figure 1 , Figure 1 A flowchart of the steps of a multimodal sentiment analysis method provided by an embodiment of the present invention.

[0064] The present invention provides a multimodal sentiment analysis method, comprising:

[0065] Step 101: obtain the multimodal emotion data to be tested, and input the multimodal emotion data to be tested into a trained multimodal emotion analysis network; the trained multimodal emotion analysis network includes a feature extraction aligner, an audio and video feature enhancer, a cross-modal converter and an emotion predictor.

[0066] It should be noted that the network architecture of the multimodal sentiment analysis network is as follows Figure 2 As shown, it can be divided into feature extraction aligner, audio and video feature enhancer (AVFEM), cross-modal converter (Crossmodal Transformer Fusion) and emotion predictor. The trained multimodal emotion analysis network is determined by performing model training on the built multimodal emotion analysis network to be trained. When the multimodal emotion data to be tested is obtained, the trained multimodal emotion analysis network is used to perform emotion analysis.

[0067] Step 102: input the multimodal emotion data to be tested into a feature extraction aligner for feature extraction and modality alignment, and determine the aligned text features, aligned video features, and aligned audio features.

[0068] Step 102 includes the following sub-steps:

[0069] S11, obtaining the text modality emotion data to be tested, the video modality emotion data to be tested, and the audio modality emotion data to be tested.

[0070] It should be noted that the multimodal emotion data to be tested in this embodiment includes text modal emotion data to be tested in text modality (t), video modal emotion data to be tested in video modality (v) and audio modal emotion data to be tested in audio modality (a), and the feature extraction aligner includes a text extraction model, a video extraction model, an audio extraction model and a one-dimensional time convolution layer (Conv1D).

[0071] S12. Use the text extraction model to extract features from the text modality sentiment data to determine the original text features.

[0072] It should be noted that for the text modality, since the pre-trained language model has achieved great success, this embodiment uses a pre-trained 12-layer BERT model (Pre-trained Bert) as a text extraction model for text feature extraction. BERT is a two-way pre-trained language model based on Transformer, which can effectively capture the emotional information in the context. The emotional data of the text modality to be tested is processed by BERT's Tokenization, and the length of the sentence sequence is padded or truncated with a fixed length of 50. The original text features extracted by BERT contain the emotional information of the text modality:

[0073] ;

[0074] In the formula, is the original text feature, For text extraction models, is the text modality sentiment data, parameters (weights) for the text extraction model, is the text feature dimension.

[0075] S13. Perform feature extraction on the emotion data of the video modality to be tested through the video extraction model to generate original video features.

[0076] It should be noted that, for the video modality, frames are first extracted from the video clips of the emotional data of the video modality to be tested at a sampling rate of 30 Hz. Subsequently, the OpenFace2.0 facial behavior analysis toolkit is used as the video extraction model (Viedo Feature Extraction) to extract the face coordinates, facial action units, head posture, head direction, and eye annotation direction in each video frame. The above are aggregated into a feature set as the original video features of the video modality.

[0077] S14. Perform feature extraction on the audio modal emotion data to be tested based on the audio extraction model to construct original audio features.

[0078] It should be noted that for audio modality, the LibROSA library and the COVAREP tool can usually be used as an audio feature extraction model (Audio Feature Extraction) to extract various feature sets of the audio modality emotion data to be tested and construct them as original audio features; in one implementation, through the LibROSA library, we extract audio features at a sampling frequency of 22050Hz, including Mel-frequency cepstral coefficients (MFCC), logarithmic frequency features (logF0) and constant Q transform features (CQT); in another implementation, using the COVAREP tool, we obtain the pitch features, voiced and unvoiced segmentation features of the audio segment, etc. These feature sets can well reflect the emotional state of the speaker and can provide important audio feature clues for multimodal sentiment analysis.

[0079] S15. Input the original text features, original video features and original audio features into the one-dimensional temporal convolution layer respectively for modality alignment, and output the corresponding aligned text features, aligned video features and aligned audio features.

[0080] It should be noted that temporal convolution is a variant of convolutional neural network, which is very effective in processing time series data. It can capture local features in the input data and perform convolution operations through sliding windows to obtain context-dependent representations. In addition, it can effectively reduce the dimension of sequence data through convolution and downsampling operations (such as pooling layers), thereby reducing the computational complexity of the model while retaining important features. We use this feature to further process the original features of the three modalities.

[0081] Since the emotion data to be tested corresponding to the original features of each modality are collected at different sampling rates, in order to ensure that each element of the original feature fully understands its neighboring elements, consider using the original features through a one-dimensional temporal convolution layer (Conv1D) for modality alignment processing:

[0082] ;

[0083] ;

[0084] ;

[0085] In the formula, To align text features, is the original text feature, To align audio features, is the original audio feature, To align video features, is the original video feature, For modal The size of the convolution kernel, For modal The characteristic dimension of is the common feature dimension after convolution, It is a one-dimensional temporal convolution layer;

[0086] At this point, the information of each modality not only fully combines the characteristics of its local structure, but also aligns with other modalities in the feature dimension, so as to better serve as the input of the audio and video feature enhancer and cross-modal converter.

[0087] Step 103: Based on the aligned text features, the aligned video features and the aligned audio features are enhanced by an audio and video feature enhancer to generate enhanced video features and enhanced audio features.

[0088] Step 103 includes the following sub-steps:

[0089] The aligned text features, aligned video features and aligned audio features are encoded respectively through the Relu activation function, and the encoded text features, the encoded video features and the encoded audio features are generated accordingly;

[0090] Perform residual connections on the aligned text features and the encoded text features, the aligned video features and the encoded video features, and the aligned audio features and the encoded audio features, respectively, and output residual text features, residual video features, and residual audio features;

[0091] Respectively concatenate the residual text features with the residual video features, and the residual text features with the residual audio features, and correspondingly determine the concatenated text video features and the concatenated text audio features;

[0092] Based on the Sigmoid activation function, the weights of the spliced ​​text video features and the spliced ​​text audio features are adaptively allocated, and the video weight and the audio weight are output;

[0093] The residual video feature and the video weight, the residual audio feature and the audio weight are respectively multiplied element by element to determine the weighted video feature and the weighted audio feature;

[0094] After multiplying the residual text feature by the complement of the video weight element by element, the feature is added element by element to the weighted video feature to generate the auxiliary video feature.

[0095] Multiply the residual text feature by the complement of the audio weight element by element, and add it to the weighted audio feature element by element to construct the auxiliary audio feature;

[0096] The auxiliary video features are multiplied element-by-element with the aligned video features, and the auxiliary audio features are multiplied element-by-element with the aligned audio features to output enhanced video features and enhanced audio features.

[0097] It should be noted that in multimodal sentiment analysis, since text modal features contain richer and more accurate emotional information, text modal features are often superior to non-text modal features. Therefore, in order to improve the recognition accuracy of the model, multimodal sentiment analysis usually strengthens the role of text modal features. However, this also leads to the fact that the features of audio modality and video modality have not received sufficient attention in extraction and utilization. To address this problem, this embodiment designs an audio and video feature enhancement mechanism (AVFEM) in the audio and video feature enhancer. This mechanism enhances the emotional information in the audio and video modal features by taking advantage of the advantages of text modal features, thereby suppressing redundant features and improving the quality of audio and video modal features. This mechanism can significantly improve the feature expression ability of audio and video modalities, better capture the emotional information association between multimodal modalities in sentiment analysis tasks, and lay the foundation for the efficient fusion of multimodal features.

[0098] The structure of the audio and video feature enhancer is as follows Figure 3 As shown, it includes Relu activation function and Sigmoid activation function, and its working process is described as follows: 1) If video feature enhancement is performed, Represents aligned video features ,First, align the text features The modal features are encoded through the Relu activation function to determine the encoded text features and the residual text features are obtained through the residual connection (RC). At the same time, the encoded video features of the aligned video features are determined based on the Relu activation function, and then the residual video features are constructed by residual connection. Then, the residual text features and residual video features are spliced ​​to generate spliced ​​text video features. The Sigmoid activation function is used to adaptively allocate the proportions between text modality features and video modality features based on the spliced ​​text video features, thereby outputting the video proportion. During the allocation process, due to the advantages of text modality features, they will occupy a higher proportion, while the proportion of video modality features is relatively low. Then, based on the video proportion, the residual video features are weighted to output weighted video features. The residual text features are weighted by the complement of the video proportion and then spliced ​​with the weighted video features, thereby generating the corresponding enhanced auxiliary video features. Finally, the auxiliary video features are multiplied by the input aligned video features through residual connection to obtain the final enhanced enhanced video features; 2) If audio feature enhancement is performed, at this time Represents aligned audio features , according to the above video feature enhancement process, based on the alignment text feature The encoded text features and the aligned audio features are respectively constructed by the Relu activation function, the residual text features and the residual audio features are determined by the residual connection, the concatenated text audio features after the residual text features and the residual audio features are concatenated are weighted by the Sigmoid activation function to determine the audio weight, the residual text features and the residual audio features are weighted and concatenated based on the audio weight and the complement of the audio weight to generate auxiliary audio features, and the auxiliary audio features are multiplied element by element with the input aligned audio features to output the enhanced audio features; the mathematical principle of the working process of the audio and video feature enhancer can be referred to as follows:

[0099] ;

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] in, ;

[0105] In the formula, is the residual text feature, is the Relu activation function, To align video features Or align audio features , is the residual video feature or the residual audio feature, For the modal, is the video weight or the audio weight, is the Sigmoid activation function, is an auxiliary video feature or an auxiliary audio feature, To enhance video features Or enhance audio features .

[0106] Step 104: Use the aligned text features, enhanced video features, and enhanced audio features through a cross-modal converter to perform cross-modal cross-attention fusion to construct interactive text features, interactive video features, and interactive audio features.

[0107] Step 104 includes the following sub-steps:

[0108] It should be noted that the core of the Crossmodal Transformer Fusion is the cross-attention mechanism. As an extension of the attention mechanism, cross-attention achieves information fusion between modalities by calculating the attention weights between two different inputs (i.e., from two different modalities). Its essence is to replace the key-value pairs of the modality in the attention mechanism with the key-value pairs of another modality for matrix calculation. By introducing other modalities as context information, calculating the attention weights between modalities, dynamically adjusting the information complementary relationship between modalities, and generating a fused cross-modal feature representation, it can effectively capture deep correlations while reducing redundant information and enhancing noise resistance. Taking the text modality as an example, Figure 4 As shown, the cross-modal transformer aligns the text features Respectively and Enhanced Video Features and enhanced audio features After performing cross-modal attention mechanism operation, video text cross features are obtained and audio-text cross-features , the two are spliced ​​together to obtain the interactive text features after cross-modal cross-attention fusion of the text modality ,During the whole fusion process, the feature information from other modalities (audio and video modalities in this case) is converted into different sets of key-value pairs, and interacts with the target modality (text modality in this case). Each target modality continuously fuses features through feature information from multi-head cross-modal attention and continuously updates its own sequence, making the feature information of the target modality more comprehensive. It is understandable that for the convenience of display, Figure 4 In Alignment text feature , only the form of expression changes:

[0109] ;

[0110] ;

[0111] ;

[0112] ;

[0113] ;

[0114] in, ;

[0115] In the formula, is layer normalization (LayerNorm), For multi-head cross attention (Multi-Head), is the intermediate audio-text cross feature, Reason Parameterized Positionwise Feed-Forward layer, is the audio-text cross feature, is the intermediate video text cross feature, It is the video text cross feature;

[0116] For the enhanced video features and enhanced audio features of audio and video modalities, the interactive video features are obtained in the same way. and interactive audio features .

[0117] Step 105: perform sentiment prediction after post-fusion of the interactive text features, interactive video features, and interactive audio features, and output the sentiment analysis result.

[0118] Step 105 includes the following sub-steps:

[0119] The interactive text features, interactive video features and interactive audio features are concatenated and low-dimensionally mapped to output low-dimensional fusion features;

[0120] The Relu activation function is used to perform nonlinear mapping on the low-dimensional fusion features and output the late fusion features;

[0121] The sentiment prediction layer is used to perform sentiment prediction on the late fusion features to determine the sentiment analysis results.

[0122] It should be noted that late fusion technology, as a method of modal fusion in the model decision stage, can alleviate the information loss and interference problems caused by early fusion to a certain extent by independently processing each modality and finally integrating the prediction results; in the late fusion process, the interactive features of each single modality are concatenated and projected into a low-dimensional space. In the , we get the low-dimensional fusion features:

[0123] ;

[0124] in, ;

[0125] In the formula, is a low-dimensional fusion feature. is the weight matrix used to project high-dimensional features to low dimensions, is the late fusion bias term;

[0126] Finally, the low-dimensional fusion features are used to input the sentiment prediction layer to predict multimodal sentiment, and the final output sentiment analysis result is:

[0127] ;

[0128] in, ;

[0129] In the formula, is the sentiment analysis result, is the prediction weight matrix, is the prediction bias term.

[0130] It can be understood that, in one implementation, the sentiment prediction layer may adopt a linear layer.

[0131] Optionally, the model training process of the multimodal sentiment analysis network includes:

[0132] Obtain training multimodal sentiment data and real sentiment analysis results;

[0133] Input the training multimodal sentiment data into the multimodal sentiment analysis network to be trained, determine the training interactive text features, training interactive video features, training interactive audio features and training sentiment analysis results, and calculate the overall loss function value in combination with the real sentiment analysis results;

[0134] The multimodal sentiment analysis network to be trained is iteratively optimized according to the overall loss function value until the overall loss function value converges, and the trained multimodal sentiment analysis network is determined.

[0135] It should be noted that the overall loss function used to calculate the overall loss function value in this embodiment includes a task loss function and a comparison loss function. The calculation process of the overall loss function value includes:

[0136] ;

[0137] In the formula, is the overall loss function, is the task loss function, is the weight parameter for contrast loss (the size of the weight parameter can be dynamically adjusted to adapt to different task scenarios), is the contrast loss function;

[0138] In traditional multimodal sentiment analysis, supervised contrastive learning is usually used. By dividing sentiment labels into discrete categories and sampling positive and negative samples based on the categories, the multimodal sentiment analysis task is simplified to a classification problem. However, this method ignores the continuity of sentiment labels and the distance information between them, which may lead to false positive or false negative samples in the sampling process. Based on the above problems, this embodiment introduces the distance information of the label into the distance-aware contrastive learning to effectively avoid the generation of false positive and false negative samples. Specifically, the positive and negative samples are divided by calculating the label distance between the anchor sample and other samples in the batch. If the label distance is less than a threshold c, the sample is regarded as a positive sample, and if the label distance is greater than c, the sample is regarded as a negative sample. This embodiment uses different fusion feature pairs as anchor samples and applies contrastive learning to the mixed mode fused with text to calculate the correlation between the mixed modes; the contrast loss function includes:

[0139] ;

[0140] in, ;

[0141] In the formula, is the distance-aware contrast loss function, It is the fusion feature of interactive audio features and interactive text features. It is the fusion feature of interactive video features and interactive text features. is the cosine similarity score function, is the index of the anchor sample, is the index of the positive sample, is the index of the negative sample, is the index set of positive samples selected according to the label distance, is the temperature parameter;

[0142] The task loss function is used to design different tasks and serves as the main training target of the model. In this embodiment, two tasks, regression task and classification task, are mainly considered; therefore, the task loss function includes:

[0143] ,or, ;

[0144] In the formula, is the batch size, For the The real sentiment analysis results, For the Training sentiment analysis results;

[0145] During the model training process, feature processing is performed in the multimodal sentiment analysis network to be trained according to steps 102 to 105 based on the training multimodal sentiment data, so as to obtain training interactive text features, training interactive video features, training interactive audio features and training sentiment analysis results, and the overall loss function value is calculated according to the design of the overall loss function combined with the real sentiment analysis results. The model parameters of the multimodal sentiment analysis network to be trained are iteratively optimized with the goal of minimizing the overall loss function value until the overall loss function value converges, thereby determining the trained multimodal sentiment analysis network.

[0146] To verify the effectiveness of the above method, we conducted experiments on the CMU-MOSI and CMU-MOSEI datasets. The experimental results are shown in Table 1:

[0147] Table 1 Comparison of experimental results of different models

[0148]

[0149] The TFN, LMF, MFN, MulT, and MISA data in Table 1 are from Mao et al. (2022). According to the scores measured on the dataset, the index scores of our method are relatively higher than those of other commonly used models. Therefore, after experimental verification, it can be determined that our method has achieved certain results.

[0150] In an embodiment of the present invention, based on multiple modal information, an audio and video feature enhancement mechanism is adopted to improve the feature quality of audio and video modalities, and effectively combines cross-modal feature interaction and multimodal late fusion, which not only ensures the deep interaction between different modal information, but also can retain the independence of each modal feature to the greatest extent, which helps to improve the accuracy and robustness of multimodal sentiment analysis, enhance the ability to capture complex emotional expressions, thereby improving the reliability of multimodal sentiment analysis as a whole, and is suitable for various multimodal sentiment analysis scenarios such as artificial intelligence, emotional computing, and human-computer interaction.

[0151] See also Figure 5 , Figure 5 A structural block diagram of a multimodal sentiment analysis device provided in an embodiment of the present invention.

[0152] The present invention provides a multimodal sentiment analysis device, comprising:

[0153] The data acquisition module 501 is used to acquire the multimodal emotion data to be tested, and input the multimodal emotion data to be tested into the trained multimodal emotion analysis network; the trained multimodal emotion analysis network includes a feature extraction aligner, an audio and video feature enhancer, a cross-modal converter and an emotion predictor;

[0154] A feature alignment module 502 is used to input the multimodal emotion data to be tested into a feature extraction aligner for feature extraction and modality alignment, and determine alignment text features, alignment video features, and alignment audio features;

[0155] A feature enhancement module 503 is used to enhance the aligned video features and the aligned audio features through an audio and video feature enhancer based on the aligned text features, thereby generating enhanced video features and enhanced audio features;

[0156] A feature cross-module 504 is used to perform cross-modal cross-attention fusion using aligned text features, enhanced video features, and enhanced audio features through a cross-modal converter to construct interactive text features, interactive video features, and interactive audio features;

[0157] The sentiment prediction module 505 is used to perform sentiment prediction after post-fusion of the interactive text features, interactive video features and interactive audio features, and output the sentiment analysis results.

[0158] Optionally, the feature alignment module 502 is specifically configured to:

[0159] Obtaining the text modality emotion data to be tested, the video modality emotion data to be tested, and the audio modality emotion data to be tested;

[0160] Use the text extraction model to extract features from the text modality sentiment data to determine the original text features;

[0161] The video extraction model is used to extract features from the emotion data of the video modality to be tested, and the original video features are generated;

[0162] Based on the audio extraction model, feature extraction is performed on the emotional data of the audio modality to be tested, and original audio features are constructed;

[0163] The original text features, original video features, and original audio features are respectively input into the one-dimensional temporal convolutional layer for modality alignment, and the corresponding aligned text features, aligned video features, and aligned audio features are output.

[0164] Optionally, the feature enhancement module 503 is specifically used for:

[0165] The aligned text features, aligned video features and aligned audio features are encoded respectively through the Relu activation function, and the encoded text features, the encoded video features and the encoded audio features are generated accordingly;

[0166] Perform residual connections on the aligned text features and the encoded text features, the aligned video features and the encoded video features, and the aligned audio features and the encoded audio features, respectively, and output residual text features, residual video features, and residual audio features;

[0167] Respectively concatenate the residual text features with the residual video features, and the residual text features with the residual audio features, and correspondingly determine the concatenated text video features and the concatenated text audio features;

[0168] Based on the Sigmoid activation function, the weights of the spliced ​​text video features and the spliced ​​text audio features are adaptively allocated, and the video weight and the audio weight are output;

[0169] The residual video feature and the video weight, the residual audio feature and the audio weight are respectively multiplied element by element to determine the weighted video feature and the weighted audio feature;

[0170] After multiplying the residual text feature by the complement of the video weight element by element, the feature is added element by element to the weighted video feature to generate the auxiliary video feature.

[0171] Multiply the residual text feature by the complement of the audio weight element by element, and add it to the weighted audio feature element by element to construct the auxiliary audio feature;

[0172] The auxiliary video features are multiplied element-by-element with the aligned video features, and the auxiliary audio features are multiplied element-by-element with the aligned audio features to output enhanced video features and enhanced audio features.

[0173] Optionally, the emotion prediction module 505 is specifically used for:

[0174] The interactive text features, interactive video features and interactive audio features are concatenated and low-dimensionally mapped to output low-dimensional fusion features;

[0175] The Relu activation function is used to perform nonlinear mapping on the low-dimensional fusion features and output the late fusion features;

[0176] The sentiment prediction layer is used to perform sentiment prediction on the late fusion features to determine the sentiment analysis results.

[0177] Optionally, a network training module is also included, for:

[0178] Obtain training multimodal sentiment data and real sentiment analysis results;

[0179] Input the training multimodal sentiment data into the multimodal sentiment analysis network to be trained, determine the training interactive text features, training interactive video features, training interactive audio features and training sentiment analysis results, and calculate the overall loss function value in combination with the real sentiment analysis results;

[0180] The multimodal sentiment analysis network to be trained is iteratively optimized according to the overall loss function value until the overall loss function value converges, and the trained multimodal sentiment analysis network is determined.

[0181] Optionally, the calculation process of the overall loss function value includes:

[0182] ;

[0183] In the formula, is the overall loss function, is the task loss function, is the weight parameter of contrast loss, is the contrast loss function;

[0184] The contrast loss functions include:

[0185] ;

[0186] in, ;

[0187] In the formula, is the distance-aware contrast loss function, It is the fusion feature of interactive audio features and interactive text features. It is the fusion feature of interactive video features and interactive text features. is the cosine similarity score function, is the index of the anchor sample, is the index of the positive sample, is the index of the negative sample, is the index set of positive samples selected according to the label distance, is the temperature parameter;

[0188] The task loss functions include:

[0189] ,or, ;

[0190] In the formula, is the batch size, For the The real sentiment analysis results, For the Training sentiment analysis results.

[0191] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the multimodal sentiment analysis method as described in any of the above embodiments.

[0192] An embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the multimodal sentiment analysis method as described in any of the above embodiments are implemented.

[0193] An embodiment of the present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements the steps of the multimodal sentiment analysis method as described in any of the above embodiments.

[0194] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0195] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0196] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0197] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0198] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0199] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal sentiment analysis method, characterized in that: include: Acquire multimodal emotion data to be tested, and input the multimodal emotion data to be tested into a trained multimodal emotion analysis network; The trained multimodal sentiment analysis network includes a feature extraction aligner, an audio and video feature enhancer, a cross-modal converter and a sentiment predictor; Inputting the multimodal emotion data to be tested into the feature extraction aligner to perform feature extraction and modality alignment, and determining alignment text features, alignment video features, and alignment audio features; Based on the aligned text features, the aligned video features and the aligned audio features are enhanced by the audio and video feature enhancer to generate enhanced video features and enhanced audio features; The aligned text features, the enhanced video features and the enhanced audio features are used by the cross-modal converter to perform cross-modal cross-attention fusion to construct interactive text features, interactive video features and interactive audio features; The interactive text features, the interactive video features and the interactive audio features are post-fused to perform sentiment prediction and output a sentiment analysis result.

2. The multimodal sentiment analysis method according to claim 1, characterized in that: The step of inputting the multimodal emotion data to be tested into the feature extraction aligner to perform feature extraction and modality alignment, and determining the aligned text features, aligned video features, and aligned audio features, comprises: Obtaining the text modality emotion data to be tested, the video modality emotion data to be tested, and the audio modality emotion data to be tested; Using a text extraction model to extract features from the text modality sentiment data to be tested, and determining original text features; Extracting features of the video modality emotion data to be tested through a video extraction model to generate original video features; Extracting features of the audio modality emotion data to be tested based on the audio extraction model to construct original audio features; The original text features, the original video features and the original audio features are respectively input into a one-dimensional temporal convolutional layer for modality alignment, and the corresponding aligned text features, aligned video features and aligned audio features are output.

3. The multimodal sentiment analysis method according to claim 1, characterized in that: The step of enhancing the aligned video features and the aligned audio features based on the aligned text features by using the audio and video feature enhancer to generate enhanced video features and enhanced audio features includes: The aligned text features, the aligned video features and the aligned audio features are respectively encoded by a Relu activation function, and an encoded text feature, an encoded video feature and an encoded audio feature are correspondingly generated; Residual connections are respectively performed on the aligned text features and the encoded text features, the aligned video features and the encoded video features, and the aligned audio features and the encoded audio features, and residual text features, residual video features, and residual audio features are output; Respectively splicing the residual text feature with the residual video feature, and the residual text feature with the residual audio feature, and correspondingly determining a spliced ​​text video feature and a spliced ​​text audio feature; Based on the Sigmoid activation function, the weights of the spliced ​​text video features and the spliced ​​text audio features are adaptively allocated, and the video weight and the audio weight are output; Respectively multiplying the residual video feature by the video weight, and the residual audio feature by the audio weight element by element, to determine a weighted video feature and a weighted audio feature; After multiplying the residual text feature by the complement of the video weight element by element, the residual text feature is added element by element to the weighted video feature to generate an auxiliary video feature; Multiplying the residual text feature by the complement of the audio weight element by element, and adding the residual text feature to the weighted audio feature element by element to construct an auxiliary audio feature; The auxiliary video feature and the aligned video feature, and the auxiliary audio feature and the aligned audio feature are respectively multiplied element by element to output an enhanced video feature and an enhanced audio feature.

4. The multimodal sentiment analysis method according to claim 1, characterized in that: The post-fusion of the interactive text features, the interactive video features and the interactive audio features is performed to perform sentiment prediction and output a sentiment analysis result, including: The interactive text feature, the interactive video feature and the interactive audio feature are concatenated and low-dimensionally mapped to output a low-dimensional fusion feature; A Relu activation function is used to perform nonlinear mapping on the low-dimensional fusion features, and a late fusion feature is output; The sentiment prediction layer performs sentiment prediction on the late fusion features to determine the sentiment analysis result.

5. The multimodal sentiment analysis method according to claim 1, characterized in that: Also includes: Obtain training multimodal sentiment data and real sentiment analysis results; Inputting the training multimodal emotion data into the multimodal emotion analysis network to be trained, determining the training interactive text features, training interactive video features, training interactive audio features and training emotion analysis results, and calculating the overall loss function value in combination with the real emotion analysis results; The multimodal sentiment analysis network to be trained is iteratively optimized according to the overall loss function value until the overall loss function value converges, thereby determining a trained multimodal sentiment analysis network.

6. The multimodal sentiment analysis method according to claim 5, characterized in that: The calculation process of the overall loss function value includes: ； In the formula, is the overall loss function, is the task loss function, is the weight parameter of contrast loss, is the contrast loss function; The contrast loss functions include: ； in, ; In the formula, is the distance-aware contrast loss function, It is the fusion feature of interactive audio features and interactive text features. It is the fusion feature of interactive video features and interactive text features. is the cosine similarity score function, is the index of the anchor sample, is the index of the positive sample, is the index of the negative sample, is the index set of positive samples selected according to the label distance, is the temperature parameter; The task loss functions include: ,or, ; In the formula, is the batch size, For the The real sentiment analysis results, For the Training sentiment analysis results.

7. A multimodal sentiment analysis device, characterized in that: include: A data acquisition module is used to acquire the multimodal emotion data to be tested, and input the multimodal emotion data to be tested into a trained multimodal emotion analysis network; The trained multimodal sentiment analysis network includes a feature extraction aligner, an audio and video feature enhancer, a cross-modal converter and a sentiment predictor; A feature alignment module, used for inputting the multimodal emotion data to be tested into the feature extraction aligner for feature extraction and modality alignment, and determining alignment text features, alignment video features and alignment audio features; A feature enhancement module, used for enhancing the aligned video feature and the aligned audio feature through the audio and video feature enhancer based on the aligned text feature, to generate enhanced video feature and enhanced audio feature; A feature cross-module, configured to perform cross-modal cross-attention fusion using the aligned text features, the enhanced video features, and the enhanced audio features through the cross-modal converter to construct interactive text features, interactive video features, and interactive audio features; The emotion prediction module is used to perform emotion prediction after post-fusion of the interactive text features, the interactive video features and the interactive audio features, and output the emotion analysis result.

8. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the multimodal sentiment analysis method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the multimodal sentiment analysis method as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the multimodal sentiment analysis method as described in any one of claims 1 to 6 are implemented.

Images