Video multi-modal sentiment analysis method and device based on multi-layer perception fusion

Abstract

The application discloses a video multi-modal sentiment analysis method and device based on a multi-layer perception fusion, relates to the technical field of sentiment analysis, and comprises the following steps: S1, text features, image features and audio features in a video are extracted; time sequence information in the image features and the audio features is extracted to obtain time sequence image features and time sequence audio features; S2, a video multi-modal sentiment analysis model comprising a multi-modal feature capturing module, a multi-layer perception fusion module and a sentiment classifier is constructed, and a loss function is constructed according to modal similarity and modal heterogeneity between modes; S3, the model is trained; and S4, the text features, the time sequence image features and the time sequence audio features are input into the trained model to obtain a sentiment polarity probability distribution. By constructing similarity loss and heterogeneity loss and using a multi-layer perception to perform sequence, channel and modal dimension fusion, the application reduces the calculation complexity and memory consumption and improves the integrity and discriminability of multi-modal sentiment features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sentiment analysis, and particularly relates to a video multi-modal sentiment analysis method and device based on multi-layer perception fusion. BACKGROUND

[0002] In today's digital era, social media platforms and short video platforms are showing explosive growth. Users have uploaded a large amount of video data on these platforms. These video data are not just a simple stack of images and sounds. They contain rich emotional information, which is of great value in understanding user intentions and grasping the direction of social discussion hotspots.

[0003] However, traditional sentiment analysis methods mainly focus on text data, and determine sentiment orientation by analyzing words, sentence structures, etc. in the text. When faced with videos containing multiple modal data such as images and audio, traditional methods are not up to the task. Because visual elements in images, tone, rhythm, etc. in audio may contain unique emotional expressions, which are difficult for traditional text sentiment analysis methods to fully capture, resulting in a great limitation on the effect of traditional methods in processing real social video content.

[0004] At present, multi-modal sentiment analysis has gradually become a research hotspot in the field of sentiment computing. Most mainstream multi-modal sentiment analysis methods use deep learning models to extract features of different modalities (such as images, audio, text, etc.) respectively, and then fuse these features in the subsequent stage to determine sentiment. Unfortunately, existing research has obvious shortcomings in the fusion stage, often ignoring the modeling of relationships between modalities, especially the similarity and heterogeneity of modalities. For example, different modalities may have similar patterns when expressing the same emotion, but at the same time they have their own unique expression methods, and these heterogeneity features also contain important emotional information. Due to the neglect of these aspects of modeling, the model cannot effectively mine the potential complementary information between modalities, resulting in problems such as information redundancy and low fusion efficiency, which seriously affect the accuracy and efficiency of sentiment analysis.

[0005] In addition, many current high-performance video multi-modal sentiment analysis methods rely on large Transformer architectures or pre-trained large models. Although these models perform well in terms of performance and can accurately analyze emotions, their large computational load and significant training resource overhead have become a bottleneck restricting their widespread application. In resource-constrained edge devices, such as some small intelligent terminal devices, or in actual deployment environments, such as real-time sentiment analysis scenarios that require fast response and limited resources, these large models are difficult to run effectively, greatly limiting the popularization of sentiment analysis technology in practical applications. SUMMARY

[0006] To solve the above problems, the application provides a video multi-modal sentiment analysis method and device based on multi-layer perception fusion. A multi-objective optimization mechanism is constructed by combining the similarity loss and heterogeneity loss of the modalities. On the basis of capturing the similarity of the modal distribution, the unique features of each modality are explicitly learned. The multi-layer perception fusion module is introduced into the multi-layer perception fusion module. A three-dimensional lightweight fusion framework is used in sequence, channel and modality. The matrix operation replaces the self-attention mechanism of the traditional Transformer. Compared with the complex fusion strategy, the computational complexity and memory consumption are significantly reduced. Compared with the single-dimensional fusion method, the integrity and discriminability of the multi-modal sentiment features are improved.

[0007] In one aspect, the video multi-modal sentiment analysis method based on multi-layer perception fusion includes the following specific steps:

[0008] S1, using a text model, an image model and an audio model to extract features from text, image and audio data in the video respectively to obtain static text features, image features and audio features; using a time sequence feature extraction model to extract time sequence information in the image features and the audio features respectively to obtain time sequence image features and time sequence audio features;

[0009] S2, constructing a video multi-modal sentiment analysis model including a multi-modal feature capturing module, a multi-layer perception fusion module and a sentiment classifier;

[0010] The multi-modal feature capturing module captures the similarity and heterogeneity between the text features, the time sequence image features and the time sequence audio features to obtain modality sentiment features including text similarity features, text heterogeneity features, image similarity features, image heterogeneity features, audio similarity features and audio heterogeneity features;

[0011] The multi-layer perception fusion module fuses all the modality sentiment features to obtain fused multi-modal sentiment features;

[0012] The sentiment classifier classifies the fused multi-modal sentiment features to output a sentiment polarity probability distribution;

[0013] S3, based on a total loss function, using labeled data to train the video multi-modal sentiment analysis model to obtain a trained video multi-modal sentiment analysis model; the total loss function includes a similarity loss constructed according to the modality similarity, a heterogeneity loss constructed according to the modality heterogeneity and a classification loss constructed according to the labeled data;

[0014] S4, inputting the text features, the time sequence image features and the time sequence audio features into the trained video multi-modal sentiment analysis model to obtain a sentiment polarity probability distribution, and realizing video sentiment prediction.

[0015] Preferably, the multi-layer perception fusion module uses a multi-layer perception to sequentially interact and fuse all the modal sentiment features from the sequence dimension, the channel dimension and the modal dimension to obtain the fused multi-modal sentiment features.

[0016] Preferably, the multi-layer perception fusion module uses a multi-layer perception to sequentially interact and fuse all the modal sentiment features from the sequence dimension, the channel dimension and the modal dimension to obtain the fused multi-modal sentiment features, and the specific process is as follows:

[0017] All the modal sentiment features are arranged in time sequence as a sequence tensor, a sequence-axis fusion unit of a sequence-oriented multi-layer perception is used to perform sequence-axis fusion on the sequence tensor, and sequence fusion features are output; which is represented as:

[0018] ;

[0019] wherein, the sequence fusion features are represented as, a fixed modal and a channel are fused on the sequence axis; layer normalization is represented as; a full connection layer is represented as; non-linear activation is represented as; a feature vector on the sequence axis is represented as; the number of modalities is represented as; the number of channels is represented as; the number of fused sequences is represented as;

[0020] The sequence fusion features are subjected to tensor reconstruction in the channel dimension to obtain a channel tensor, a channel-axis fusion unit of a channel-oriented multi-layer perception is used to perform channel-axis fusion on the channel tensor, and channel fusion features are output; which is represented as:

[0021] ;

[0022] wherein, the channel fusion features are represented as, a fixed sequence and a modal are fused on the channel axis; a feature vector on the channel axis is represented as; the number of fused channels is represented as;

[0023] The modal dimension tensor reconstruction is performed on the channel fusion features to obtain a modal tensor, and a multi-layer perception of a modal direction is used to perform modal axis fusion on the modal tensor to output a fused multi-modal emotion feature; represented as:

[0024] ;

[0025] wherein, represents the fused multi-modal emotion feature, represents a fixed sequence and a channel fused on the modal axis; represents a feature vector on the modal axis; represents the number of fused modalities.

[0026] Preferably, the text model is a BERT model; the image model is an OpenFace image feature processing tool; the audio model is a COVAREP tool; and the time series feature extraction model is a one-way long short-term memory network sLSTM.

[0027] Preferably, the total loss function is represented as:

[0028] ;

[0029] wherein, represents a classification loss; represents a similarity loss; represents a heterogeneity loss; and represents a regularization weight coefficient.

[0030] Preferably, the similarity loss is represented as:

[0031] ;

[0032] wherein, represents a similarity loss; represents a regularized central moment difference, represents an order; represents a similarity feature of a modality ; represents a similarity feature of a modality ; represents a text modality; represents an audio modality; represents a video modality.

[0033] Preferably, the heterogeneity loss is represented as:

[0034] ;

[0035] wherein, represents the heterogeneity loss; represents the square of the Frobenius norm; represents the similarity feature matrix of the modal ; represents the transpose of the similarity feature matrix of the modal ; represents the specificity feature matrix of the modal ; represents the transpose of the specificity feature matrix of the modal ; represents the specificity feature matrix of the modal ; represents the text modality; represents the audio modality; represents the video modality.

[0036] Preferably, the similarities and heterogeneities between the text features, the time-series image features and the time-series audio features are captured as follows:

[0037] The text features, the time-series image features and the time-series audio features are respectively input into feedforward neural networks with shared learnable parameters to learn the common features among the modalities, to obtain the text similarity features, the image similarity features and the audio similarity features;

[0038] The text features, the time-series image features and the time-series audio features are respectively input into feedforward neural networks with independent learnable parameters to learn the respective heterogeneity features of each modality, to obtain the text heterogeneity features, the image heterogeneity features and the audio heterogeneity features.

[0039] Preferably, the classification loss is a cross-entropy loss or a mean absolute error loss; the cross-entropy loss is represented as:

[0040] ;

[0041] wherein, represents the classification loss; represents the number of sentiment categories; represents the real label; represents the predicted label;

[0042] The mean absolute error loss is represented as:

[0043] ;

[0044] wherein, represents the number of sentiment categories; represents the real label; represents the predicted label; represents the square of the L2 norm.

[0045] In another aspect, the video multi-modal sentiment analysis device based on multi-layer perception fusion comprises the following:

[0046] A video multi-modal feature extraction module is configured to extract features of text, image and audio data in a video by using a text model, an image model and an audio model respectively to obtain static text features, image features and audio features; and extract time sequence information in the image features and the audio features by using a time sequence feature extraction model to obtain time sequence image features and time sequence audio features.

[0047] A video multi-modal sentiment analysis model construction module is configured to construct a video multi-modal sentiment analysis model comprising a multi-modal feature capturing module, a multi-layer perception fusion module and a sentiment classifier.

[0048] The multi-modal feature capturing module is configured to capture similarities and heterogeneities among the text features, the time sequence image features and the time sequence audio features to obtain modal sentiment features comprising text similarity features, text heterogeneity features, image similarity features, image heterogeneity features, audio similarity features and audio heterogeneity features.

[0049] The multi-layer perception fusion module is configured to fuse all the modal sentiment features to obtain fused multi-modal sentiment features.

[0050] The sentiment classifier is configured to classify the fused multi-modal sentiment features to output a sentiment polarity probability distribution.

[0051] A video multi-modal sentiment analysis model training module is configured to train the video multi-modal sentiment analysis model by using labeled data based on a total loss function to obtain a trained video multi-modal sentiment analysis model; the total loss function comprises a similarity loss constructed according to modal similarities, a heterogeneity loss constructed according to modal heterogeneities and a classification loss constructed according to the labeled data.

[0052] A video sentiment prediction module is configured to input the text features, the time sequence image features and the time sequence audio features into the trained video multi-modal sentiment analysis model to obtain a sentiment polarity probability distribution and realize video sentiment prediction.

[0053] Compared with the prior art, the present application has the following beneficial effects:

[0054] (1) The present application uses center moment difference (CMD) and soft orthogonal constraint to construct a modal similarity loss function, which explicitly learns unique features of each modality on the basis of capturing modal distribution similarities, thereby enhancing the complementarity and expression ability of cross-modal features compared with traditional multi-modal fusion methods.

[0055] (2) The application introduces a multi-layer perceptron (MLP) into the field of multi-modal feature fusion, and through a three-dimensional lightweight fusion framework of sequence, channel and mode, matrix operation is used to replace the self-attention mechanism of the traditional Transformer, so that the calculation complexity and memory consumption are significantly reduced compared with a complex fusion strategy;

[0056] (3) The application uses the time sequence dynamics and multi-modal interaction features of video data, adopts a three-dimensional MLP fusion module to jointly model the sequence dependency, channel semanticity and mode correlation, and compared with a single-dimensional fusion method, the integrity and discriminability of multi-modal emotional features are improved;

[0057] (4) The application combines the similarity loss and heterogeneity loss of the mode to construct a multi-objective optimization mechanism, associates the text semantic, image visual and audio acoustic features through a unified representation space, and combines the three-dimensional fusion framework to perform emotion analysis from the mode relationship and feature interaction dimensions, so that the emotion prediction accuracy in a complex scene is improved;

[0058] (5) The application can be used for real-time emotion analysis of social media videos, emotion recognition of human-computer interaction systems, learning state evaluation of educational videos and the like, and can also be used as a basic method for multi-modal data noise reduction and feature enhancement, and provides data support in the emotional dimension for intelligent recommendation, social discussion hotspot monitoring and the like. BRIEF DESCRIPTION OF DRAWINGS

[0059] The application will be further described in detail below with reference to the drawings;

[0060] Figure 1 A flowchart of a video multi-modal emotion analysis method based on a multi-layer perceptron fusion of an embodiment of the application;

[0061] Figure 2 A flowchart of a video multi-modal emotion analysis method based on a multi-layer perceptron fusion of an embodiment of the application;

[0062] Figure 3 A fusion process flowchart of a multi-layer perceptron module of a video multi-modal emotion analysis method based on a multi-layer perceptron fusion of an embodiment of the application;

[0063] Figure 4 A multi-layer perceptron unit structure diagram of a video multi-modal emotion analysis method based on a multi-layer perceptron fusion of an embodiment of the application;

[0064] Figure 5 A sample distribution diagram of a MOSI data set used by a video multi-modal emotion analysis method based on a multi-layer perceptron fusion of an embodiment of the application;

[0065] Figure 6A sample distribution diagram of the MOSEI data set used by the video multi-modal emotion analysis method based on multi-layer perception fusion of the embodiment of the present application is shown in the figure.

[0066] Figure 7 A modal correlation ablation experiment result schematic diagram of the video multi-modal emotion analysis method based on multi-layer perception fusion of the embodiment of the present application is shown in the figure.

[0067] Figure 8 A fusion module component ablation experiment result schematic diagram of the video multi-modal emotion analysis method based on multi-layer perception fusion of the embodiment of the present application is shown in the figure.

[0068] Figure 9 An L (sequence) axis output dimension influence on MAE schematic diagram in the video multi-modal emotion analysis method based on multi-layer perception fusion of the embodiment of the present application is shown in the figure.

[0069] Figure 10 An M (modality) axis output dimension influence on MAE schematic diagram in the video multi-modal emotion analysis method based on multi-layer perception fusion of the embodiment of the present application is shown in the figure.

[0070] Figure 11 A D (channel) axis output dimension influence on MAE schematic diagram in the video multi-modal emotion analysis method based on multi-layer perception fusion of the embodiment of the present application is shown in the figure.

[0071] Figure 12 A recall rate, accuracy rate and F1 value schematic diagram of the video multi-modal emotion analysis method based on multi-layer perception fusion of the embodiment of the present application is shown in the figure.

[0072] Figure 13 A structure block diagram of the video multi-modal emotion analysis device based on multi-layer perception fusion of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0073] The present application is further described below through specific embodiments.

[0074] As shown in the figures, Figure 1 and Figure 2 the video multi-modal emotion analysis method based on multi-layer perception fusion has the following specific steps:

[0075] S1, text features, image features and audio features are obtained by using a text model, an image model and an audio model to respectively extract features of text, image and audio data in a video.

[0076] In this embodiment, the BERT pre-training model, OpenFace and COVAREP tools are used to extract the initial features of text, image and audio data respectively, and the image and audio features are further refined by sLSTM. The specific process is as follows:

[0077] S11, text features are extracted by using the BERT pre-training model. The process is as follows:

[0078] S111, the text modal data in the video is input into the BERT pre-training model for Chinese word segmentation and stop word removal preprocessing;

[0079] S112, the global semantic association of aspect words in the text is performed by the self-attention mechanism of the BERT model to obtain the context semantic information of the text;

[0080] S113, the output layer of the BERT model is used to generate the initialization , is the original text input into the pre-training model BERT, is the text semantic feature processed by BERT, is a learnable parameter.

[0081] S12, the image features are preliminarily extracted, and the obtained image features are further processed. The process is as follows:

[0082] S121, the OpenFace image feature processing tool is used to preliminarily extract the image features. OpenFace can segment the image into many small squares, detect the important feature points of the image, and train a deep convolutional neural network to generate image features. The preliminarily extracted image features are represented as ;

[0083] S122, the obtained image features are further processed by using the unidirectional long short-term memory network (sLSTM). The sLSTM increases the depth of the model by using multiple LSTM layers to increase the number of layers, thereby improving the modeling ability of the time sequence. The expression of the LSTM is as follows:

[0084] ;

[0085] ;

[0086] ;

[0087] ;

[0088] ;

[0089] In the above formula, is an input vector representing the current time, is a hidden state vector, is a sigmoid activation function, and is a tanh activation function, is a memory cell, is a forget gate, is an input gate, is an output gate, and is a weight parameter matrix;

[0090] preliminarily extracted image features are processed using sLSTM to obtain final image emotional features, i.e.

[0091] ;

[0092] wherein, is an image feature obtained after further processing by sLSTM, represents a learnable parameter in sLSTM used for extracting image features.

[0093] S13, audio features are preliminarily extracted, and the obtained audio features are further processed to obtain final audio emotional features. The specific process is as follows:

[0094] S131, the COVAREP audio feature processing tool is used to preliminarily extract audio features. The basic audio features include frame energy, fundamental frequency, and short-time jitter parameters, and important audio emotional feature parameters Mel and cepstrum coefficient of scale frequency. Several tens of audio features such as peakSlope, QQQ, and MDQ are calculated, and these features are superimposed to obtain initial audio features ;

[0095] S132, the obtained audio features are further processed using sLSTM to obtain final audio emotional features:

[0096] ;

[0097] wherein, the audio feature is a feature preliminarily extracted by the COVAREP tool, is an audio feature obtained after processing by sLSTM, represents a learnable parameter of sLSTM used for extracting audio features.

[0098] S2, a video multi-modal emotional analysis model including a multi-modal feature capturing module, a multi-layer perception fusion module, and an emotional classifier is constructed.

[0099] The video multi-modal sentiment analysis model introduces a central moment difference (CMD) as a similarity loss function, learns the distribution similarity of different modal features by calculating the distribution difference between modalities, and simultaneously constructs a heterogeneity loss function through a soft orthogonal constraint to capture the unique features of each modality.

[0100] The process of learning the distribution similarity of different modal features and capturing the unique features of each modality by the multi-modal feature capturing module is as follows:

[0101] Each modality is encoded using a feedforward neural network to extract its original feature representation.

[0102] To learn the common features between modalities, a shared encoder structure is used to generate a similarity feature representation.

[0103] To capture modal-specific features, a separate encoder parameter is used to generate a heterogeneity feature representation. For each modality in a given video , the encoder is used to learn its modal similarity and heterogeneity features , i.e.:

[0104] ;

[0105] ;

[0106] In the formula, and are the learned modal similarity and heterogeneity features, is the input initial video modality feature, is the learning parameter.

[0107] Specifically, to generate six hidden vectors (two for each modality, including similarity and heterogeneity), a simple feedforward neural layer is used to share parameters between all three modalities to obtain their similarity, and separate parameters are assigned to each modality to obtain their heterogeneity.

[0108] The process of calculating the similarity loss function (CMD) is as follows:

[0109] For the similarity vector generated for each modality, the central moment difference (CMD) is calculated. CMD calculates the difference between different distributions based on the central moment of the distribution. The core idea is to use high-order moments to measure the distance between probability distributions. The smaller the CMD distance, the more similar the two distributions are.

[0110] Let X and Y be two bounded random samples with probability distributions p and q on the interval [a, b], the regularized center moment difference CMD K is defined as the empirical estimation of the CMD metric, i.e.,

[0111] ;

[0112] where, denotes the empirical expectation vector of X, denotes the all k-th sample center distance vector of X coordinates.

[0113] The similarity is obtained by reducing the modal difference through CMD, and the CMD loss of each pair of modal is calculated as follows:

[0114] ;

[0115] The CMD loss is used to constrain the consistency of the distribution of different modalities in the common subspace, and reduce the distribution difference between modalities; the CMD loss models the distribution alignment by accumulating the first-order to high-order center moment difference.

[0116] The process of calculating the heterogeneity loss function (orthogonal constraint) is as follows:

[0117] The heterogeneity feature matrix of each modality is subjected to soft orthogonal constraint, so as to ensure that the features of different modalities remain non-redundant in the vector space;

[0118] The Frobenius norm is calculated to control the correlation between modal vectors;

[0119] The heterogeneity loss function aims to capture the different aspects between modalities, thereby supplementing the information other than the similarity features. By applying soft orthogonal constraint between two vectors, the calculation of non-redundancy is realized. In a training batch, let and be the vector matrix, whose row represents the hidden vector of each modality and in each video segment, then the orthogonal constraint of the modal vector pair is expressed as:

[0120] ;

[0121] where, denotes the orthogonal constraint, denotes the square of the Frobenius norm.

[0122] Further, the heterogeneity orthogonal term between modalities is introduced to enhance the independence and discriminability of the modal difference representation. That is:

[0123] ;

[0124] Adding orthogonal constraints between modal heterogeneity vectors helps reduce redundant information, enhances the model's representational power, improves training stability, and promotes generalization ability. Furthermore, it can more effectively capture the unique features of each modality, improving overall performance and robustness.

[0125] like Figure 3 As shown, the multimodal feature fusion process of the multilayer perceptron module in this embodiment follows a three-dimensional progressive logic: sequence axis, channel axis, and modality axis. First, focusing on the sequence axis, the multilayer perceptron performs in-depth mining and fusion of the features of the multimodal input data in the time or sequence dimension. By capturing sequence correlation information, sequence fusion features are generated, which contain the dynamic correlations between different modal data as the sequence changes. Then, based on the sequence fusion features, the process moves to the channel axis. The multilayer perceptron filters, reorganizes, and interactively calculates features in the channel dimension, mining key semantic information within and between each modal channel. The sequence-fused features are further refined to obtain channel fusion features, strengthening the semantic expression of features at the channel level. Finally, based on the channel fusion features, the process enters the modality axis fusion stage. The multilayer perceptron integrates feature correlations between different modalities such as text, images, and audio, breaking down modal barriers, mining cross-modal emotional interaction patterns, and ultimately outputting modality fusion features. By employing a three-dimensional, progressive, and mutually supportive fusion approach, we can comprehensively and meticulously uncover the emotional value of multimodal data, providing more discriminative and complementary feature support for subsequent video sentiment analysis, thereby improving the accuracy and robustness of sentiment analysis.

[0126] like Figure 4 As shown, the internal operation flow of the multilayer perceptron module in this embodiment follows the logic of feature transformation, activation, normalization, and residual fusion. First, the input features enter the first fully connected layer, where the neural network completes dimension mapping and information interaction; the sigmoid activation function introduces nonlinearity, enhancing the flexibility of feature expression; the features flow into the second fully connected layer to further explore deep correlations; layer normalization standardizes the feature distribution, improving training stability; finally, the original input and processed features are fused through residual connections to alleviate the gradient vanishing problem, ensure the integrity of information transmission, and provide more robust feature transformation capabilities for multimodal sentiment analysis.

[0127] The multilayer perceptron fusion module performs interactive fusion of the processed single-modal features from three dimensions: sequence, channel, and modality. First, the sequence-axis multilayer perceptron captures the temporal dynamic features of the video. Then, the channel-axis multilayer perceptron and the modality-axis multilayer perceptron realize the semantic filtering of feature channels and the integration of cross-modal information to generate multimodal emotional features.

[0128] Sequence-axis fusion is performed. All modal emotion features are arranged in time sequence as a sequence tensor; the sequence features are input into a multi-layer perception unit in sequence direction to perform affine transformation and activation; the sequence fused feature representation is output, which preserves the time context and time sequence dependency. The sequence-axis fusion process is represented as:

[0129]

[0130] wherein, and are two matrix represented learnable parameters, is a feature vector in sequence axis.

[0131] The mathematical expression of the complete multi-layer perception module fusion in sequence axis is represented as:

[0132]

[0133] wherein, is a nonlinear activation, is layer normalization, represents the output sequence fused feature, which is obtained after sufficient fusion in different modalities and channels.

[0134] Channel-axis fusion is performed. The sequence fused features are reconstructed in channel dimension tensor; the channel features are input into a multi-layer perception unit in channel direction to extract modal internal channel features; the channel fused features are output, which enhances the expression of each channel feature. The mathematical expression is represented as:

[0135]

[0136] Modality-axis fusion is performed. The channel fusion results are reorganized in modal dimension tensor; the modal dimension features are input into a modal-axis MLP unit to perform deep fusion; the final fused multi-modal emotion features are output, which are provided for the emotion classifier. The mathematical expression is represented as:

[0137]

[0138] wherein, is a feature representing sequence and modality interaction, represents the final multi-modal emotion feature after sufficient fusion in sequence, modality and channel.

[0139] ​​​​The fused multi-modal sentiment feature is input into a Softmax function, and a probability distribution of sentiment polarity is calculated through a fully connected layer to realize video sentiment prediction. The specific process is as follows: the fused three-dimensional multi-modal feature tensor is unfolded into a one-dimensional vector; the unfolded vector is input into a sentiment classifier for subsequent processing; the unfolded vector is input into a fully connected layer; a Softmax activation function is used to generate a probability distribution of each sentiment category; and a predicted sentiment label is selected according to the maximum probability;

[0140] Specifically, the multi-modal sentiment feature after sufficient interactive fusion is flattened as and further input into a sentiment classifier to predict the final multi-modal sentiment tendency. When performing a video multi-modal sentiment classification task, a Softmax activation function is used to feed it to a fully connected layer to generate a probability of sentiment polarity, which is represented as:

[0141] ;

[0142] In the formula, represents the multi-modal sentiment feature after sufficient fusion, W t , b t represent learnable parameters, represents the final prediction result.

[0143] Cross-entropy is used as a classification loss function to optimize the model, and the cross-entropy loss function is represented as:

[0144] ;

[0145] In the formula, represents the number of sentiment categories, represents the real label, represents the predicted label.

[0146] In another embodiment, MAE is used as an evaluation index in a regression task. Mean absolute error (MAE) is used as a loss function for a video multi-modal sentiment regression task, and is represented as:

[0147] ;

[0148] In the formula, represents the number of sentiment categories, represents the real label, represents the predicted label.

[0149] S3, based on the total loss function, the labeled data is used to train the video multi-modal sentiment analysis model to obtain a trained video multi-modal sentiment analysis model.

[0150] The similarity loss function and the heterogeneity loss function The overall learning of the embodiment method is completed by minimizing the following total loss function, that is:

[0151]

[0152] In the formula, and is a regularization weight coefficient for determining the contribution of each loss function to the total loss .

[0153] Based on the total loss function , the video multi-modal sentiment analysis model is trained using the labeled data to obtain the trained video multi-modal sentiment analysis model. S4, the text features, time sequence image features and time sequence audio features are input into the trained video multi-modal sentiment analysis model to obtain the sentiment polarity probability distribution, and video sentiment prediction is realized.

[0154] Experimental verification.

[0155] Two public video multi-modal sentiment analysis datasets CMU-MOSI and CMU-MOSEI are selected as experimental datasets. The sentiment analysis of the MOSI and Mosei datasets is a regression task measured by mean absolute error (MAE) and Pearson correlation coefficient (Corr). In addition, the sentiment classification task is also involved, and the evaluation indicators include seven classification accuracy (Acc7), binary classification accuracy (Acc2) and F1 value from-3 to 3.

[0156] Specifically, as shown in Figure 5 and Figure 6 , the CMU-MOSI dataset used in the embodiment contains 2199 opinion video clips, and each video is labeled with a sentiment intensity ranging from [-3, 3], indicating strong negative to positive emotions. The dataset has strict labeling, including sentiment polarity, sentiment intensity, image features per frame, and audio features per millisecond. Among them, the training samples are 1284, the verification samples are 229, and the test samples are 686. The CMU-MOSEI dataset used in the embodiment is an improvement of the MOSI dataset, with more video clip quantities and more diverse samples. The dataset contains more than 23,500 video clips, covering a wide range of topics, and the data includes text transcription, audio and image information. Each video is labeled with sentiment polarity and intensity. Among them, the training samples are 16326, the verification samples are 1871, and the test samples are 4659.

[0157] MAE is a commonly used evaluation index for regression models, which is used to measure the difference between the predicted value and the actual value, that is:​

[0158] ;

[0159] wherein, represents the number of samples, i.e., the total number of data points; represents the actual value of the sample; represents the predicted value of the sample.

[0160] The Pearson correlation coefficient is used to measure the linear correlation between two variables, i.e.:

[0161] ;

[0162] wherein, and represent the variable values of the sample; and represent the variable mean values; represents the Pearson correlation coefficient, which takes a value in the range [-1, 1], and is negative when negative, and positive when positive.

[0163] The accuracy Acc and F1 value calculation formula is as follows:

[0164] ;

[0165] ;

[0166] The training parameters are shown in Table 1:

[0167] Table 1: Training parameter settings.

[0168]

[0169] As shown in Table 2, compared with the baseline method, the method of the embodiment achieves high experimental performance in each emotion category.

[0170] Table 2: Comparison of experimental results.

[0171]

[0172] The method of the embodiment performs better than the TFN and LMFMulT methods on two public data sets. The method of the embodiment comprehensively considers the complementary relationship between modalities by introducing similarity and heterogeneity loss functions, thereby having stronger robustness and expression ability in capturing modal information. In view of the problem that the Transformer model has a high requirement for computing resources and is difficult to balance efficiency and performance, the method of the embodiment realizes more efficient modal feature fusion through the lightweight design of the multilayer perceptron, avoiding the computational overhead of the Transformer.

[0173] The method of the present example outperforms the SWAFN and MISA methods on two public data sets. The SWAFN and MISA methods address the potential complementarity of different modalities to some extent, but have limited ability to handle redundant features and noise. The method of the present application learns the similarity and heterogeneity between multi-modal features while interacting and sharing information from three axes of sequence, channel and modality, and can extract more comprehensive important emotional features.

[0174] The method of the present example outperforms the CubeMLP and PS-Mixer methods on two public data sets. The method of the present example more deeply analyzes the similarity and heterogeneity relationship between different modalities by introducing a similarity and heterogeneity loss function, proving the effectiveness of fully mining the complex emotional relationship between modalities.

[0175] Compared with the DMD method, the method of the present example outperforms the DMD method on the CMU-MOSI data set, but performs slightly worse than the DMD method on the CMU-MOSEI data set. The possible reason is that the DMD more effectively analyzes the heterogeneity between different modalities of data through an efficient decoupling multi-modal distillation method, and uses a multi-modal Transformer to calculate the distribution difference between different modalities and perform feature alignment. The performance of the Transformer is better than that of the multi-layer perceptron in large-scale data, while the method based on the multi-layer perceptron fusion has better performance in the case of less data. Secondly, the method of the present example additionally explores the similarity between different modalities, which is the reason why the method of the present example outperforms the DMD method on the MOSI data set.

[0176] Spatial complexity is an important indicator for measuring the memory resources consumed by a model at runtime, especially when dealing with large-scale data. Optimization of spatial complexity is particularly important. Taking performance on the CMU-MOSI public data set as an example, the spatial complexity experimental results are shown in Table 3.

[0177] Table 3: Spatial complexity experimental results.

[0178]

[0179] As can be seen from Table 3, compared with other baseline methods, the spatial complexity of the method of the present example is greatly reduced, and the best result is achieved in the CMU-MOSI data set.

[0180] In the TFN method, the fusion of different modalities is performed on the features of all modalities. By representing the features of each modality as a tensor and calculating the tensor product between each modality, the spatial complexity is O(L M ).

[0181] The methods using Transformer (such as LMFMulT and DMD) fuse inter-modal information on sequence L using a self-attention mechanism. Since the self-attention mechanism calculation needs to operate on each pair of sequence elements, the spatial complexity is proportional to the square of the sequence length, and the spatial complexity is O(L 2 ).

[0182] The method of the embodiment fuses multi-modal features in three axes (sequence L, modality M and channel D) respectively by using a multi-layer perception unit. The spatial complexity is determined by the maximum dimension of the input tensor, and the spatial complexity is O(max(L,M,D)).

[0183] Therefore, the method of the embodiment can reduce memory consumption while maintaining high performance. It is shown that multi-modal feature fusion by a small-scale model can achieve good results while greatly reducing computing resources.

[0184] To verify the influence of each modality and the learning of similarity and heterogeneity on the method of the embodiment, five ablation experiments are set. Including: removing the text modality ((-)Text) to verify the influence of introducing text sentiment information on video multi-modal sentiment analysis. Remove the image modality ((-)Visual) to verify the influence of introducing image sentiment information on video multi-modal sentiment analysis. Remove the audio modality ((-)Audio) to verify the influence of introducing audio sentiment information on video multi-modal sentiment analysis. Remove the learning of modality similarity ((-)L sim ) to verify the influence of learning the similarity between each modality on video multi-modal sentiment analysis. Remove the learning of modality heterogeneity ((-)L diff ) to verify the influence of learning the heterogeneity between each modality on video multi-modal sentiment analysis. The results of the modality-related ablation experiment are shown in Tables 4 and 7:

[0185] Table 4: Results of modality-related ablation experiment.

[0186]

[0187] Observe Tables 4 and Figure 7The ablation experiment results show that in the video multi-modal sentiment analysis task, different modalities have different effects on sentiment analysis. In video multi-modal sentiment analysis, the text modality is the most important for sentiment analysis, and the experimental results decrease sharply when the text modality is removed, so the text modality is indispensable in multi-modal sentiment analysis. The audio modality has a certain influence on the sentiment analysis results, and the image modality has a relatively small influence. In addition, learning the similarity between different modalities can obtain more accurate sentiment analysis results, and learning the heterogeneity can also increase the method effect to a certain extent.

[0188] To verify the influence of each component in the multi-layer perception fusion module on the method of the embodiment, three ablation experiments are set. Including: removing the sequence fusion axis ((-) MLP-L) to verify the influence of sequence fusion on each modality on video multi-modal sentiment analysis. Remove the channel fusion axis ((-) MLP-M) to verify the influence of channel fusion on each modality on video multi-modal sentiment analysis. Remove the modality fusion axis ((-) MLP-D) to verify the influence of modality fusion on each modality on video multi-modal sentiment analysis. The fusion module component ablation experiment results are shown in Table 5 and Figure 8 .

[0189] Table 5: Fusion module component ablation experiment results

[0190]

[0191] Different axis fusion has different effects on the final obtained multi-modal sentiment features, among which the sequence axis is the most important, which may be due to the fact that video data is time series data. The fusion on the sequence axis can effectively capture the dynamic changes of emotions, context information, time alignment between modalities, and long and short term dependencies in the video. These factors work together to make the sequence axis feature fusion particularly important in multi-modal sentiment analysis, thereby helping the model to more accurately understand and predict the emotional state.

[0192] The embodiment realizes the sufficient fusion of different modality features by constructing a multi-layer perception fusion module. Thanks to the inherent structure of the multi-layer perception, the model can flexibly adjust the output dimension of each multi-layer perception unit, thereby effectively affecting the final prediction effect. From a mathematical point of view, L', M' and D' represent the dimensions of the input features after affine transformation on the sequence L, modality M and channel D axes. By adjusting the different output dimensions, the performance of the method can be affected to a certain extent. The output dimension experiment results are shown in Figure 9 , Figure 10 and Figure 11As shown in the results of the observation output dimension experiment, when the output dimension of any fusion axis is less than 3 (<3), the performance of the method significantly decreases. However, when the output dimension is greater than or equal to 3 (≥3), the fluctuation of the method performance significantly decreases. The fundamental reason for this phenomenon may be closely related to the three modalities (text, image, and audio) involved in the training process. Specifically, the input multi-modal feature vector dimension is a three-dimensional tensor in the initial stage, and its rank is 3. The recall, accuracy, and F1 value diagram is as shown in Figure 12 Therefore, by using affine transformation on all fusion axes to convert the tensor into, the method can more fully capture the emotional complementary information between modalities, thereby achieving the best performance.

[0193] As shown in Figure 13 The application also discloses a video multi-modal sentiment analysis device based on multi-layer perception fusion, which comprises:

[0194] A video multi-modal feature extraction module 1301 is configured to extract features of text, image, and audio data in a video by using a text model, an image model, and an audio model respectively, to obtain static text features, image features, and audio features; and extract time sequence information in the image features and the audio features by using a time sequence feature extraction model, to obtain time sequence image features and time sequence audio features.

[0195] A video multi-modal sentiment analysis model construction module 1302 is configured to construct a video multi-modal sentiment analysis model comprising a multi-modal feature capturing module, a multi-layer perception fusion module, and a sentiment classifier.

[0196] The multi-modal feature capturing module is configured to capture similarities and heterogeneities among the text features, the time sequence image features, and the time sequence audio features, to obtain modal sentiment features comprising text similarity features, text heterogeneity features, image similarity features, image heterogeneity features, audio similarity features, and audio heterogeneity features.

[0197] The multi-layer perception fusion module is configured to fuse all the modal sentiment features, to obtain fused multi-modal sentiment features.

[0198] The sentiment classifier is configured to classify the fused multi-modal sentiment features, to output a sentiment polarity probability distribution.

[0199] A video multi-modal sentiment analysis model training module 1303 is configured to train the video multi-modal sentiment analysis model based on a total loss function by using labeled data, to obtain a trained video multi-modal sentiment analysis model; the total loss function comprises a similarity loss constructed according to modal similarities, a heterogeneity loss constructed according to modal heterogeneities, and a classification loss constructed according to the labeled data.

[0200] The video emotion prediction module 1304 is configured to input the text features, the time sequence image features and the time sequence audio features into the trained video multi-modal emotion analysis model to obtain an emotion polarity probability distribution, thereby realizing video emotion prediction.

[0201] The specific implementation of the video multi-modal emotion analysis device based on multi-layer perception fusion is the same as the method, and thus will not be repeated here.

[0202] The above merely illustrates the specific embodiments of the present application, but the design concept of the present application is not limited thereto, and any non-essential modification of the present application using the concept shall be deemed as an infringement of the protection scope of the present application.

Claims

1. A video multi-modal sentiment analysis method based on multi-layer perception fusion, characterized in that, The method comprises the following steps: S1, extracting features of text, image and audio data in the video by using a text model, an image model and an audio model respectively to obtain static text features, image features and audio features; extracting time sequence information in the image features and the audio features by using a time sequence feature extraction model to obtain time sequence image features and time sequence audio features; S2, constructing a video multi-modal sentiment analysis model comprising a multi-modal feature capturing module, a multi-layer perception fusion module and a sentiment classifier; the multi-modal feature capturing module captures the similarity and heterogeneity among the text features, the time sequence image features and the time sequence audio features to obtain modal sentiment features comprising text similarity features, text heterogeneity features, image similarity features, image heterogeneity features, audio similarity features and audio heterogeneity features; the multi-layer perception fusion module fuses all the modal sentiment features to obtain fused multi-modal sentiment features; the sentiment classifier classifies the fused multi-modal sentiment features to output a sentiment polarity probability distribution; S3, training the video multi-modal sentiment analysis model based on a total loss function by using labeled data to obtain a trained video multi-modal sentiment analysis model; the total loss function comprises a similarity loss constructed according to modal similarity, a heterogeneity loss constructed according to modal heterogeneity and a classification loss constructed according to the labeled data; S4, inputting the text features, the time sequence image features and the time sequence audio features into the trained video multi-modal sentiment analysis model to obtain a sentiment polarity probability distribution, thereby realizing video sentiment prediction; the total loss function is expressed as: L = L task + aL sim + bL diff ; where L task denotes the classification loss; L sim denotes the similarity loss; L diff denotes the heterogeneity loss; and a and b denote the regularization weight coefficients the multi-layer perception fusion module uses a multi-layer perception to sequentially interact and fuse all the modal sentiment features from a sequence dimension, a channel dimension and a modal dimension to obtain the fused multi-modal sentiment features, and the specific process is as follows: all the modal sentiment features are arranged in a time sequence as a sequence tensor, a sequence-axis fusion unit of a sequence-oriented multi-layer perception is used to perform sequence-axis fusion on the sequence tensor, and a sequence fusion feature is output; which is expressed as: where U *,m,d denotes sequence fusion features, *, m, d denotes fixed modality m and channel d fusing sequence axis; LN denotes layer normalization; Aff L denotes a fully connected layer; σ denotes a nonlinear activation; X *,m,d denotes a feature vector on the sequence axis; M denotes the number of modalities; D denotes the number of channels; L' denotes the number of fused sequences; the sequence fusion feature is subjected to tensor reconstruction in the channel dimension to obtain a channel tensor, and a channel-axis fusion unit of a channel-oriented multi-layer perception is used to perform channel-axis fusion on the channel tensor, and a channel fusion feature is output; which is expressed as: where V l,m,* represents the channel fusion feature, l, m, * represents the fixed sequence l and the modal m to fuse the channel axis; U l,m,* represents the feature vector on the channel axis; D' represents the number of fused channels; the channel fusion feature is subjected to tensor reconstruction in the modal dimension to obtain a modal tensor, and a modal-axis fusion unit of a modal-oriented multi-layer perception is used to perform modal-axis fusion on the modal tensor, and a fused multi-modal sentiment feature is output; which is expressed as: wherein X′ l,*,d represents the fused multimodal sentiment feature, l,*, d represents the fusion of the fixed sequence l and the channel d on the modal axis; V l,*,d represents the feature vector on the modal axis; M′ represents the fused modal number.

2. The multi-layer perceptron fusion-based video multi-modal sentiment analysis method according to claim 1, characterized in that, the text model is a BERT model; the image model is an OpenFace image feature processing tool; the audio model is a COVAREP tool; and the time sequence feature extraction model is a one-way long short-term memory network sLSTM.

3. The multi-layer perceptron fusion-based video multi-modal sentiment analysis method according to claim 1, wherein, the similarity loss is expressed as: wherein L sim represents a similarity loss; CMD K represents a regularized central moment difference, K represents an order; represents a similarity feature of modality m1; represents a similarity feature of modality m2; l represents a text modality; a represents an audio modality; v represents a video modality.

4. The multi-layer perceptron fusion-based video multi-modal sentiment analysis method according to claim 1, characterized in that, the heterogeneity loss is expressed as: where L diff denotes the heterogeneity loss; denotes the square of the Frobenius norm; denotes the transpose of the similarity feature matrix representation of modality m; denotes the transpose of the specificity feature matrix representation of modality m1; denotes the specificity feature matrix representation of modality m; denotes the specificity feature matrix representation of modality m2; l denotes the text modality; a denotes the audio modality; v denotes the video modality.

5. The multi-layer perceptron fusion-based video multi-modal sentiment analysis method according to claim 1, wherein, the similarity and heterogeneity among the text features, the time sequence image features and the time sequence audio features are captured, and the specific process is as follows: inputting the text features, the time-series image features and the time-series audio features into feedforward neural networks with shared learnable parameters to learn inter-modal common features, to obtain text similarity features, image similarity features and audio similarity features; inputting the text features, the time-series image features and the time-series audio features into feedforward neural networks with independent learnable parameters to learn inter-modal heterogeneous features, to obtain text heterogeneous features, image heterogeneous features and audio heterogeneous features.

6. The multi-layer perceptron fusion-based video multi-modal sentiment analysis method according to claim 1, wherein, The classification loss is a cross-entropy loss or a mean absolute error loss; the cross-entropy loss is represented as: wherein L task represents a classification loss; N represents the number of emotion categories; y i represents a true label; represents a predicted label; The mean absolute error loss is represented as: where N denotes the number of emotion categories; y i denotes the true label; denotes the predicted label; denotes the square of L2 norm.

7. A multi-layer perceptron fusion-based video multi-modal sentiment analysis device using the multi-layer perceptron fusion-based video multi-modal sentiment analysis method according to any one of claims 1-6, comprising the following: The video multi-modal feature extraction module is configured to extract features of text, image and audio data in the video by using a text model, an image model and an audio model respectively, to obtain static text features, image features and audio features. extracting time-series information in the image features and the audio features using a time-series feature extraction model to obtain time-series image features and time-series audio features; a video multi-modal sentiment analysis model construction module configured to construct a video multi-modal sentiment analysis model comprising a multi-modal feature capturing module, a multi-layer perceptron fusion module and a sentiment classifier; the multi-modal feature capturing module is configured to capture similarities and heterogeneities among text features, time-series image features and time-series audio features to obtain modal sentiment features including text similarity features, text heterogeneous features, image similarity features, image heterogeneous features, audio similarity features and audio heterogeneous features; the multi-layer perceptron fusion module is configured to fuse all modal sentiment features to obtain fused multi-modal sentiment features; the sentiment classifier is configured to classify the fused multi-modal sentiment features to output a sentiment polarity probability distribution; a video multi-modal sentiment analysis model training module configured to train the video multi-modal sentiment analysis model using labeled data based on a total loss function to obtain a trained video multi-modal sentiment analysis model; the total loss function comprises a similarity loss constructed according to modal similarities, a heterogeneity loss constructed according to modal heterogeneities and a classification loss constructed according to the labeled data; a video sentiment prediction module configured to input the text features, the time-series image features and the time-series audio features into the trained video multi-modal sentiment analysis model to obtain a sentiment polarity probability distribution, thereby realizing video sentiment prediction.

Images