Video multi-mode sentiment analysis method and device based on multi-layer perceptron fusion

Abstract

The invention discloses a video multi-mode sentiment analysis method and device based on multi-layer perceptron fusion, and relates to the technical field of sentiment analysis, and the method comprises the steps: S1, extracting text features, image features and audio features in a video; extracting time sequence information in the image features and the audio features to obtain time sequence image features and time sequence audio features; s2, constructing a video multi-modal sentiment analysis model comprising a multi-modal feature capture module, a multi-layer perceptron fusion module and a sentiment classifier, and constructing a loss function according to modal similarity and modal heterogeneity between modals; s3, training the model; and S4, inputting the text features, the time sequence image features and the time sequence audio features into the trained model to obtain emotion polarity probability distribution. According to the method, similarity loss and heterogeneity loss are constructed, and sequence, channel and modal dimension fusion is carried out by using a multi-layer perceptron, so that the calculation complexity and memory consumption are reduced, and the integrity and discrimination capability of the multi-modal emotion features are improved.

Description

Technical Field

[0001] The present invention relates to the field of sentiment analysis technology, and in particular to a video multimodal sentiment analysis method and device based on multi-layer perceptron fusion. Background Art

[0002] In today's digital age, social media and short video platforms are experiencing explosive growth, with users uploading massive amounts of video data. This video data is more than just a collection of images and sounds; it contains rich emotional information, which is extremely valuable for understanding user intent and identifying the trends in social discussion.

[0003] However, traditional sentiment analysis methods primarily focus on text data, analyzing vocabulary and sentence structure to determine emotional tendencies. When faced with complex content like video, which includes multiple modalities such as images and audio, traditional methods struggle. This is because visual elements in images and the intonation and rhythm of audio can all contain unique emotional expressions that traditional text-based sentiment analysis methods struggle to fully capture. This significantly limits the effectiveness of traditional methods when processing real-world social video content.

[0004] Currently, multimodal sentiment analysis has become a research hotspot in the field of affective computing. Most mainstream multimodal sentiment analysis methods leverage deep learning models to extract features from different modalities (such as images, audio, and text) separately. These features are then subsequently fused to perform sentiment analysis. Unfortunately, existing research has significant shortcomings in the fusion stage, often neglecting the modeling of relationships between modalities, particularly the similarities and heterogeneity between modalities. For example, while different modalities may express the same emotion in similar ways, they also possess unique expressions. These heterogeneous features also contain important emotional information. This neglect of modeling these aspects results in the model's inability to effectively exploit the potential complementary information between modalities, leading to information redundancy and inefficient fusion, severely impacting the accuracy and efficiency of sentiment analysis.

[0005] Furthermore, many current high-performance multimodal video sentiment analysis methods rely on large Transformer architectures or pre-trained models. While these models offer excellent performance and can perform relatively accurate sentiment analysis, their massive computational overhead and significant training resource overhead have become bottlenecks hindering their widespread adoption. These large models struggle to run effectively on resource-constrained edge devices, such as small smart terminals, or in real-world deployments, such as real-time sentiment analysis scenarios that require fast response times and limited resources. This significantly limits the widespread adoption of sentiment analysis technology in real-world applications. Summary of the Invention

[0006] To address the above problems, the present invention proposes a video multimodal sentiment analysis method and device based on multi-layer perceptron fusion. By combining the similarity loss and heterogeneity loss of modalities, a multi-objective optimization mechanism is constructed. On the basis of capturing the similarity of modal distribution, the unique features of each modality are explicitly learned. By introducing the multi-layer perceptron into the multi-layer perceptron fusion module, a lightweight fusion framework of sequence, channel and modality is adopted, and the self-attention mechanism of the traditional Transformer is replaced by matrix operation, compared with complex fusion strategies, the computational complexity and memory consumption are significantly reduced. Compared with the single-dimensional fusion method, the integrity and discriminability of multimodal sentiment features are improved.

[0007] On the one hand, the video multimodal sentiment analysis method based on multi-layer perceptron fusion has the following specific steps:

[0008] S1, using a text model, an image model, and an audio model to extract features of the text, image, and audio data in the video, respectively, to obtain static text features, image features, and audio features; using a temporal feature extraction model to extract temporal information from the image features and the audio features, respectively, to obtain temporal image features and temporal audio features;

[0009] S2, building a video multimodal sentiment analysis model including a multimodal feature capture module, a multi-layer perceptron fusion module and a sentiment classifier;

[0010] The multimodal feature capture module captures the similarities and heterogeneities between text features, time-series image features, and time-series audio features, and obtains modal sentiment features including text similarity features, text heterogeneity features, image similarity features, image heterogeneity features, audio similarity features, and audio heterogeneity features;

[0011] The multi-layer perceptron fusion module fuses all modal emotion features to obtain fused multi-modal emotion features;

[0012] The sentiment classifier classifies the fused multimodal sentiment features and outputs a sentiment polarity probability distribution;

[0013] S3, based on a total loss function, using the labeled data to train the video multimodal sentiment analysis model to obtain a trained video multimodal sentiment analysis model; the total loss function includes a similarity loss constructed according to modality similarity, a heterogeneity loss constructed according to modality heterogeneity, and a classification loss constructed according to the labeled data;

[0014] S4, inputs text features, temporal image features and temporal audio features into the trained video multimodal sentiment analysis model to obtain the sentiment polarity probability distribution and realize video sentiment prediction.

[0015] Preferably, the multi-layer perceptron fusion module uses a multi-layer perceptron to interactively fuse all modal emotion features from the sequence dimension, channel dimension and modal dimension in sequence to obtain a fused multi-modal emotion feature.

[0016] Preferably, the multi-layer perceptron fusion module uses a multi-layer perceptron to interactively fuse all modal emotion features from the sequence dimension, channel dimension and modal dimension in sequence to obtain the fused multi-modal emotion features, as follows:

[0017] Arrange all modal sentiment features into sequence tensors in chronological order, use the sequence axis fusion unit of the sequence-oriented multi-layer perceptron to perform sequence axis fusion on the sequence tensors, and output sequence fusion features; expressed as:

[0018] ;

[0019] in, represents sequence fusion features, Indicates fixed mode and channel Fuse the sequence axes; Representation layer normalization; represents a fully connected layer; represents nonlinear activation; Represents the eigenvector on the sequence axis; represents the modal number; Indicates the number of channels; Indicates the number of sequences after fusion;

[0020] The channel dimension tensor of the sequence fusion feature is reconstructed to obtain the channel tensor, and the channel axis is fused on the channel tensor using the channel-oriented multi-layer perceptron, and the channel fusion feature is output; it is expressed as:

[0021] ;

[0022] in, represents the channel fusion feature, Represents a fixed sequence and modal Fuse the channel axes; represents the eigenvector on the channel axis; Indicates the number of channels after fusion;

[0023] The modal tensor is obtained by reconstructing the modal dimension of the channel fusion feature. The modal axis fusion of the modal tensor is performed using a modal multi-layer perceptron, and the fused multi-modal sentiment feature is output; it is expressed as:

[0024] ;

[0025] in, Represents the fused multimodal sentiment features, Represents a fixed sequence and channel Fusion of modal axes; represents the eigenvector on the modal axis; Indicates the number of modalities after fusion.

[0026] Preferably, the text model is a BERT model; the image model is an OpenFace image feature processing tool; the audio model is a COVAREP tool; and the temporal feature extraction model is a unidirectional long short-term memory network sLSTM.

[0027] Preferably, the total loss function is expressed as:

[0028] ;

[0029] in, represents the classification loss; Represents similarity loss; represents heterogeneity loss; and represents the regularization weight coefficient.

[0030] Preferably, the similarity loss is expressed as:

[0031] ;

[0032] in, Represents similarity loss; represents the regularized central moment difference, Indicates the order; Indicates modality similarity characteristics; Indicates modality similarity characteristics; Indicates text mode; Indicates audio mode; Indicates video mode.

[0033] Preferably, the heterogeneity loss is expressed as:

[0034] ;

[0035] in, represents heterogeneity loss; represents the square of the Frobenius norm; Indicates modality The similarity feature matrix representation of The transpose of Indicates modality The specific feature matrix representation of The transpose of Indicates modality The specific feature matrix representation of Indicates modality The specific feature matrix representation of Indicates text mode; Indicates audio mode; Indicates video mode.

[0036] Preferably, the capturing of similarities and heterogeneities among text features, time-series image features, and time-series audio features is specifically as follows:

[0037] The text features, time-series image features, and time-series audio features are respectively input into a feedforward neural network with shared learnable parameters to learn the common features between modalities and obtain text similarity features, image similarity features, and audio similarity features.

[0038] The text features, time-series image features and time-series audio features are respectively input into a feedforward neural network with independent learnable parameters to learn the heterogeneous features of each modality and obtain text heterogeneity features, image heterogeneity features and audio heterogeneity features.

[0039] Preferably, the classification loss is a cross entropy loss or a mean absolute error loss; the cross entropy loss is expressed as:

[0040] ;

[0041] in, represents the classification loss; Indicates the number of sentiment categories; represents the true label; represents the predicted label;

[0042] The mean absolute error loss is expressed as:

[0043] ;

[0044] in, Indicates the number of sentiment categories; represents the true label; represents the predicted label; represents the L2 norm squared.

[0045] On the other hand, the video multimodal sentiment analysis device based on multi-layer perceptron fusion includes the following:

[0046] The video multimodal feature extraction module is used to extract features of the text, image and audio data in the video using a text model, an image model and an audio model, respectively, to obtain static text features, image features and audio features; and to use a temporal feature extraction model to extract the temporal information in the image features and the audio features, respectively, to obtain temporal image features and temporal audio features;

[0047] Video multimodal sentiment analysis model construction module, used to build a video multimodal sentiment analysis model including a multimodal feature capture module, a multi-layer perceptron fusion module and a sentiment classifier;

[0048] The multimodal feature capture module captures the similarities and heterogeneities between text features, time-series image features, and time-series audio features, and obtains modal sentiment features including text similarity features, text heterogeneity features, image similarity features, image heterogeneity features, audio similarity features, and audio heterogeneity features;

[0049] The multi-layer perceptron fusion module fuses all modal emotion features to obtain fused multi-modal emotion features;

[0050] The sentiment classifier classifies the fused multimodal sentiment features and outputs a sentiment polarity probability distribution;

[0051] A video multimodal sentiment analysis model training module is used to train the video multimodal sentiment analysis model using labeled data based on a total loss function to obtain a trained video multimodal sentiment analysis model; the total loss function includes a similarity loss constructed based on modal similarity, a heterogeneity loss constructed based on modal heterogeneity, and a classification loss constructed based on labeled data;

[0052] The video emotion prediction module is used to input text features, time-series image features, and time-series audio features into the trained video multimodal emotion analysis model to obtain the emotion polarity probability distribution and realize video emotion prediction.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] (1) This paper uses the central moment difference (CMD) and soft orthogonal constraints to construct a modal similarity loss function. On the basis of capturing the similarity of modal distribution, it explicitly learns the unique features of each modality. Compared with traditional multimodal fusion methods, it enhances the complementarity and expressiveness of cross-modal features.

[0055] (2) This paper introduces the multi-layer perceptron (MLP) into the field of multimodal feature fusion. Through a lightweight fusion framework of sequence, channel and modality, it replaces the traditional Transformer's self-attention mechanism with matrix operations. Compared with complex fusion strategies, it significantly reduces computational complexity and memory consumption.

[0056] (3) This paper utilizes the temporal dynamics and multimodal interaction characteristics of video data and adopts a three-dimensional MLP fusion module to jointly model sequence dependency, channel semantics, and modal relevance. Compared with the single-dimensional fusion method, it improves the integrity and discriminability of multimodal emotional features.

[0057] (4) This paper combines modal similarity loss and heterogeneity loss to construct a multi-objective optimization mechanism, associates text semantics, image vision, and audio acoustic features through a unified representation space, and combines it with a three-dimensional fusion framework to perform sentiment analysis from the dual dimensions of modal relationship and feature interaction, thereby improving the accuracy of sentiment prediction in complex scenarios.

[0058] (5) The present invention can be used in scenarios such as real-time sentiment analysis of social media videos, emotion recognition in human-computer interaction systems, and learning status assessment in educational videos. It can also serve as a basic method for multimodal data noise reduction and feature enhancement, providing data support in the emotional dimension for applications such as intelligent recommendation and monitoring of social discussion hotspots. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The present invention will be described in further detail below with reference to the accompanying drawings;

[0060] Figure 1 Flowchart of a video multimodal sentiment analysis method based on multi-layer perceptron fusion according to an embodiment of the present invention;

[0061] Figure 2 Schematic diagram of the process of a video multimodal sentiment analysis method based on multi-layer perceptron fusion according to an embodiment of the present invention;

[0062] Figure 3 A flowchart of the fusion process of the multi-layer perceptron module of the video multimodal sentiment analysis method based on multi-layer perceptron fusion according to an embodiment of the present invention;

[0063] Figure 4 A multi-layer perceptron unit structure diagram of a video multimodal sentiment analysis method based on multi-layer perceptron fusion according to an embodiment of the present invention;

[0064] Figure 5 A sample distribution diagram of the MOSI dataset used in the video multimodal sentiment analysis method based on multi-layer perceptron fusion in an embodiment of the present invention;

[0065] Figure 6This is a sample distribution diagram of the MOSEI dataset used in the video multimodal sentiment analysis method based on multi-layer perceptron fusion in an embodiment of the present invention;

[0066] Figure 7 Schematic diagram of modality-related ablation experiment results of a video multimodal sentiment analysis method based on multi-layer perceptron fusion according to an embodiment of the present invention;

[0067] Figure 8 Schematic diagram of ablation experiment results of fusion module components of the video multimodal sentiment analysis method based on multi-layer perceptron fusion according to an embodiment of the present invention;

[0068] Figure 9 Schematic diagram of the effect of the L (sequence) axis output dimension on MAE in the video multimodal sentiment analysis method based on multi-layer perceptron fusion according to an embodiment of the present invention;

[0069] Figure 10 Schematic diagram of the effect of the M (modality) axis output dimension on MAE in the video multimodal sentiment analysis method based on multi-layer perceptron fusion according to an embodiment of the present invention;

[0070] Figure 11 Schematic diagram of the effect of the D (channel) axis output dimension on MAE in the video multimodal sentiment analysis method based on multi-layer perceptron fusion according to an embodiment of the present invention;

[0071] Figure 12 This is a schematic diagram of the recall rate, accuracy, and F1 value of the video multimodal sentiment analysis method based on multi-layer perceptron fusion in an embodiment of the present invention.

[0072] Figure 13 This is a structural block diagram of a video multimodal sentiment analysis device based on multi-layer perceptron fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The present invention is further described below through specific embodiments.

[0074] like Figure 1 and Figure 2 As shown in the figure, the video multimodal sentiment analysis method based on multi-layer perceptron fusion has the following specific steps:

[0075] S1, use the text model, image model and audio model to extract features of the text, image and audio data in the video respectively, and obtain static text features, image features and audio features; use the time series feature extraction model to extract the time series information in the image features and the audio features respectively, and obtain time series image features and time series audio features.

[0076] In this example, the BERT pre-trained model, OpenFace, and COVAREP tools were used to perform preliminary feature extraction on text, image, and audio data, respectively. The image and audio features were then further refined using sLSTM. The details are as follows:

[0077] S11, use the BERT pre-trained model to extract text features. The process is:

[0078] S111: Input the text modality data in the video into the BERT pre-training model for Chinese word segmentation and stop word removal preprocessing;

[0079] S112, through the self-attention mechanism of the BERT model, global semantic association is performed on the aspect words in the text to obtain the contextual semantic information of the text;

[0080] S113, initialization of text generation using the output layer of the BERT model , is the original text input to the pre-trained model BERT, It is the semantic feature of the text after BERT processing. are learnable parameters.

[0081] S12, perform preliminary extraction of image features, and then further process the acquired image features. The process is:

[0082] S121, use OpenFace image feature processing tools to perform preliminary extraction of image features. OpenFace can divide the image into many small squares, detect important feature points of the image, and train a deep convolutional neural network to generate image features. The preliminary extracted image features are expressed as ;

[0083] S122 uses a unidirectional long short-term memory network (sLSTM) to further process the acquired image features. sLSTM uses multiple LSTM layers to increase the number of layers to enhance the depth of the model, thereby improving the modeling ability of time series. The expression of LSTM is as follows:

[0084] ;

[0085] ;

[0086] ;

[0087] ;

[0088] ;

[0089] In the above formula, represents the input vector at the current moment, represents the hidden state vector, represents the sigmoid activation function, and represents the tanh activation function, Represents a memory unit, represents the forget gate, represents the input gate, represents the output gate, and represents the weight parameter matrix;

[0090] Preliminary extracted image features The final image emotion features are obtained by processing with sLSTM, namely:

[0091] ;

[0092] Where, is the image feature obtained after further processing by sLSTM, Represents the learnable parameters in sLSTM used to extract image features.

[0093] S13, preliminarily extract the audio features, and then further process the obtained audio features to obtain the final audio emotion features. The specific process is:

[0094] S131, using the COVAREP audio feature processing tool to perform preliminary audio feature extraction. This includes basic audio features: frame energy, fundamental frequency, short-term jitter parameters, as well as important audio emotional feature parameters Mel and scale frequency cepstral coefficients. Dozens of audio features such as peakSlope, QQQ, and MDQ are calculated and superimposed to obtain the initial audio features. ;

[0095] S132: Use sLSTM to further process the acquired audio features to obtain the final audio emotion features:

[0096] ;

[0097] Where, audio features It is the feature initially extracted by COVAREP tool. It is the audio feature obtained after sLSTM processing. Represents the learnable parameters of the sLSTM used to extract audio features.

[0098] S2, constructs a video multimodal sentiment analysis model including a multimodal feature capture module, a multi-layer perceptron fusion module and a sentiment classifier.

[0099] The video multimodal sentiment analysis model introduces the central moment difference (CMD) as a similarity loss function. It learns the distribution similarity of features of different modalities by calculating the distribution differences between modalities. At the same time, it constructs a heterogeneous loss function through soft orthogonal constraints to capture the unique characteristics of each modality.

[0100] The multimodal feature capture module learns the distribution similarity of features of different modalities and captures the unique features of each modality in the following process:

[0101] Use a feedforward neural network to encode each modality and extract its original feature representation;

[0102] To learn common features between modalities, a shared encoder structure is used to generate similarity feature representations;

[0103] To capture modality-specific features, separate encoder parameters are used to generate heterogeneous feature representations. For each modality in a given video , using the encoder to learn its modality similarity and heterogeneous characteristics ,Right now:

[0104] ;

[0105] ;

[0106] Where, and are the learned modality similarity and heterogeneity features, is the initial video modality feature of the input, To learn the parameters.

[0107] Specifically, in order to generate six hidden vectors (two for each modality, including similarity and heterogeneity), directly using a simple feed-forward neural layer to share parameters across all three modalities , obtain their similarity by assigning separate parameters to each mode , and obtain its heterogeneity.

[0108] The process of calculating the similarity loss function (CMD) is:

[0109] For the similarity vectors generated by each modality, the central moment discrepancy (CMD) is calculated. CMD calculates the difference between different distributions based on the central moment of the distribution. The core idea is to use higher-order moments to measure the distance between probability distributions. The smaller the CMD distance, the more similar the two distributions are.

[0110] Let X and Y be bounded random samples from the interval [a, b] with probability distributions p and q, respectively, and the regularized central moment difference CMD K is defined as an empirical estimate of the CMD metric, namely:

[0111] ;

[0112] Where, represents the empirical expectation vector of X, Represents the center distance vector of all k samples of X coordinates.

[0113] CMD is used to reduce modal differences and obtain similarity. The CMD loss for each pair of modalities is calculated as follows:

[0114] ;

[0115] CMD loss is used to constrain the distribution consistency of different modes in the common subspace and reduce the distribution differences between modes; CMD loss explicitly models distribution alignment by accumulating the differences from first-order to high-order central moments.

[0116] The process of calculating the heterogeneity loss function (orthogonality constraint) is:

[0117] Apply soft orthogonal constraints to the heterogeneous feature matrix of each mode to ensure that the features of different modes remain non-redundant in the vector space;

[0118] Calculate the correlation between the Frobenius norm-controlled mode vectors;

[0119] The heterogeneity loss function aims to capture the different aspects between modalities, thereby supplementing information other than similarity features. By imposing a soft orthogonal constraint between two vectors, non-redundant calculations can be achieved. In a training batch, let and is a vector matrix whose rows represent each modality in each video clip The hidden vector and , then the orthogonality constraint of the modal vector pair is expressed as:

[0120] ;

[0121] in, represents the orthogonality constraint, represents the square of the Frobenius norm.

[0122] Furthermore, the heterogeneous orthogonal terms between modalities are introduced to enhance the independence and discriminability of modal difference representation. That is:

[0123] ;

[0124] Adding orthogonal constraints between modal heterogeneity vectors helps reduce redundant information, enhance the model's representation ability, improve training stability, and promote generalization. It can also more effectively capture the unique characteristics of each modality, improving overall performance and robustness.

[0125] like Figure 3 As shown, the multimodal feature fusion process of the multi-layer perceptron module in this embodiment follows the three-dimensional progressive logic of the sequence axis, channel axis, and modal axis. First, focus on the sequence axis and use the multi-layer perceptron to deeply mine and fuse the features of the multimodal input data in the time or sequence dimension. By capturing the sequence association information, a sequence fusion feature is generated. The feature contains the dynamic association of different modal data with the sequence change. Then, based on the sequence fusion feature, turn to the channel axis processing. The multi-layer perceptron screens, reorganizes and interactively calculates the features in the channel dimension, mines the key semantic information within and between each modal channel, further refines the features after sequence fusion, obtains the channel fusion feature, and strengthens the semantic expression of the feature at the channel level. Finally, based on the channel fusion feature, enter the modal axis fusion stage. The multi-layer perceptron integrates the feature associations between different modalities such as text, images, and audio, breaks the modal barriers, mines the cross-modal emotional interaction pattern, and finally outputs the modal fusion feature. Through a three-dimensional progressive and mutually coordinated fusion approach, the emotional value of multimodal data is comprehensively and meticulously explored, providing more discriminative and complementary feature support for subsequent video sentiment analysis, thereby improving the accuracy and robustness of sentiment analysis.

[0126] like Figure 4 As shown, the internal operational flow of the multilayer perceptron module in this embodiment follows the logic of feature transformation, activation, normalization, and residual fusion. First, input features enter the first fully connected layer, where a neural network completes dimensional mapping and information exchange. The sigmoid activation function introduces nonlinearity, enhancing the flexibility of feature expression. Features flow into the second fully connected layer to further explore deep-level associations. Layer normalization (LN) standardizes the feature distribution and improves training stability. Finally, residual connections are used to fuse the original input with the processed features, alleviating the vanishing gradient problem and ensuring information integrity, providing more robust feature transformation capabilities for multimodal sentiment analysis.

[0127] The multi-layer perceptron fusion module interactively fuses the processed single-modal features from three dimensions: sequence, channel, and modality. First, the sequence-axis multi-layer perceptron is used to capture the temporal dynamic features of the video, and then the channel-axis multi-layer perceptron and the modality-axis multi-layer perceptron are used to realize feature channel semantic screening and cross-modal information integration to generate multimodal emotional features.

[0128] Perform sequence axis fusion. Arrange all modal sentiment features in chronological order as a sequence tensor; input the sequence features to the sequence-oriented multi-layer perceptron unit, perform affine transformation and activation; output the fused feature representation of the sequence, preserving the temporal context and temporal dependencies. The sequence axis fusion process is expressed as:

[0129] ;

[0130] Where, and are the learnable parameters represented by two matrices, is the eigenvector on the sequence axis.

[0131] The mathematical expression of the fusion of the complete multi-layer perceptron module on the sequence axis is expressed as:

[0132] ;

[0133] Where, is a nonlinear activation, is layer normalization, It represents the output sequence fusion feature, which is obtained by fully fusing different modalities and channels.

[0134] Perform channel axis fusion. Reconstruct the channel dimension tensor of the sequence fusion features; input channel features to the channel-oriented multi-layer perceptron unit to extract the internal channel features of the modality; output the channel fusion features to enhance the expressiveness of each channel feature. The mathematical expression is:

[0135] ;

[0136] Perform modal axis fusion. Reorganize the channel fusion results into tensors according to the modal dimension; input the modal dimension features into the modal axis MLP unit for deep fusion; output the final fused multimodal sentiment features and provide them to the sentiment classifier. The mathematical expression is:

[0137] ;

[0138] Where, To represent features that have undergone sequence and modality interactions, It represents the final multimodal sentiment feature after full fusion of sequence, modality and channel.

[0139] The fused multimodal emotional features are input into the Softmax function, and the emotional polarity probability distribution is calculated through the fully connected layer to achieve video emotion prediction. The specific process is as follows: the fused three-dimensional multimodal feature tensor is expanded into a one-dimensional vector; the dimensionality is flattened and input into the emotion classifier for subsequent processing; the flattened vector is input into the fully connected layer; the Softmax activation function is used to generate the probability distribution of each emotion category; and the predicted emotion label is selected based on the maximum probability.

[0140] Specifically, after full interactive fusion, the multimodal emotional features was flattened , which is further input into the sentiment classifier to predict the final multimodal sentiment tendency. When performing the video multimodal sentiment classification task, the Softmax activation function is used to feed it into the fully connected layer to generate the probability of sentiment polarity, which is expressed as:

[0141] ;

[0142] Where, represents the fully integrated multimodal sentiment features, W t 、b t represents the learnable parameters, Indicates the final prediction result.

[0143] Cross entropy is used as the classification loss function for model optimization. The cross entropy loss function is expressed as:

[0144] ;

[0145] Where, represents the number of sentiment categories, represents the true label, Represents the predicted label.

[0146] In another embodiment, MAE is used as an evaluation metric in the regression task. The mean absolute error (MAE) is used as the loss function for the video multimodal emotion regression task, which is expressed as:

[0147] ;

[0148] Where, represents the number of sentiment categories, represents the true label, Represents the predicted label.

[0149] S3, based on the total loss function, uses the labeled data to train the video multimodal sentiment analysis model to obtain a trained video multimodal sentiment analysis model.

[0150] Combining the above similarity loss function and heterogeneity loss function , the overall learning of the method in this embodiment is completed by minimizing the following total loss function, namely:

[0151] ;

[0152] Where, and is the regularization weight coefficient, which is used to determine the contribution of each loss function to the total loss contribution.

[0153] Based on the total loss function , use the labeled data to train the video multimodal sentiment analysis model to obtain a trained video multimodal sentiment analysis model. S4, input the text features, time-series image features and time-series audio features into the trained video multimodal sentiment analysis model to obtain the sentiment polarity probability distribution and realize video sentiment prediction.

[0154] Experimental verification.

[0155] Two public multimodal video sentiment analysis datasets, CMU-MOSI and CMU-MOSEI, were selected as experimental datasets. Sentiment analysis on the MOSI and MOSEI datasets is a regression task using mean absolute error (MAE) and Pearson correlation coefficient (Corr) as metrics. Furthermore, sentiment classification tasks are also involved, with evaluation metrics including the seven-category accuracy (Acc7) ranging from -3 to 3, the two-category accuracy (Acc2), and the F1 score.

[0156] Specifically, such as Figure 5 and Figure 6 As shown, the CMU-MOSI dataset used in this example contains 2,199 viewpoint video clips. Each video is annotated with a sentiment intensity ranging from [−3, 3], representing strong negative to positive sentiment. This dataset is rigorously annotated, including sentiment polarity, sentiment intensity, per-frame and per-viewpoint image features, and per-millisecond audio features. It contains 1,284 training samples, 229 validation samples, and 686 test samples. The CMU-MOSEI dataset used in this example is an improvement on the MOSI dataset, with a larger number of video clips and a more diverse sample size. This dataset contains over 23,500 video clips covering a wide range of topics, and the data includes text transcriptions, audio, and image information. Each video is annotated with sentiment polarity and intensity. It contains 16,326 training samples, 1,871 validation samples, and 4,659 test samples.

[0157] MAE is a commonly used regression model evaluation indicator that measures the difference between the predicted value and the actual value, namely:

[0158] ;

[0159] Where, Indicates the number of samples, that is, the total number of data points; represents the actual value of the sample; Represents the predicted value of the sample.

[0160] The Pearson correlation coefficient is used to measure the linear correlation between two variables, namely:

[0161] ;

[0162] in, and The variable value representing the sample; and represents the mean of the variable; Represents the Pearson correlation coefficient, with a value range of [-1,1]. A negative number indicates a negative correlation, and a positive number indicates a positive correlation.

[0163] The calculation formulas for accuracy Acc and F1 value are as follows:

[0164] ;

[0165] ;

[0166] The training parameters are shown in Table 1:

[0167] Table 1: Training parameter settings.

[0168]

[0169] As shown in Table 2, compared with the baseline method, the method of this embodiment achieves high experimental performance in each emotion category.

[0170] Table 2: Comparative experimental results.

[0171]

[0172] The method in this embodiment outperformed the TFN and LMFMuLT methods on two public datasets. By introducing similarity and heterogeneity loss functions, the method in this embodiment comprehensively considers the complementary relationship between modalities, resulting in greater robustness and expressiveness in capturing modal information. Furthermore, the lightweight design of a multi-layer perceptron achieves more efficient modal feature fusion, addressing the Transformer model's high computational resource requirements and difficulty in achieving a balance between efficiency and performance, thereby avoiding the Transformer's computational overhead.

[0173] The method in this example outperforms the SWAFN and MISA methods on two public datasets. While SWAFN and MISA address the potential complementarity of different modalities to a certain extent, their ability to handle redundant features and noise is limited. The method in this paper, while learning the similarities and heterogeneity between multimodal features, interactively shares information across three axes: sequence, channel, and modality, enabling the extraction of more comprehensive and important emotional features.

[0174] The method in this example outperforms CubeMLP and PS-Mixer on two public datasets. By introducing similarity and heterogeneity loss functions, the method in this example further analyzes the similarity and heterogeneity relationships between different modalities, demonstrating its effectiveness in fully exploring the complex emotional relationships between modalities.

[0175] Compared to the DMD method, the method in this example outperforms the DMD method on the CMU-MOSI dataset, but performs slightly worse on the CMU-MOSEI dataset. This may be because DMD more effectively analyzes the heterogeneity between different modal data through an efficient decoupled multimodal distillation method, and uses a multimodal Transformer to calculate the distribution differences between different modalities and perform feature alignment. The Transformer performs better than the multilayer perceptron on large-scale data, while methods based on multilayer perceptron fusion have better performance when the amount of data is small. Secondly, the method in this example also explores the similarities between different modalities, which is why it outperforms the DMD method on the MOSI dataset.

[0176] Space complexity is an important indicator for measuring the memory resources consumed by a model during runtime. Optimizing space complexity is particularly important when processing large-scale data. Taking the performance on the CMU-MOSI public dataset as an example, the space complexity experimental results are shown in Table 3:

[0177] Table 3: Space complexity experimental results.

[0178]

[0179] From Table 3, we can see that compared with other baseline methods, the space complexity of the method in this embodiment is greatly reduced, and the best results are achieved in the CMU-MOSI dataset.

[0180] In the TFN method, the fusion of different modalities is performed on the features of all modalities. By expressing the features of each modality as a tensor and calculating the tensor product between the modalities, the space complexity is O(L M ).

[0181] Transformer methods (such as LMFMuLT and DMD) use the self-attention mechanism to fuse inter-modal information on the sequence L. Since the self-attention mechanism calculation needs to operate on each pair of sequence elements, the space complexity is proportional to the square of the sequence length, so the space complexity is O(L 2 ).

[0182] This embodiment uses a multi-layer perceptron to perform multimodal feature fusion on three axes: sequence L, modality M, and channel D. The spatial complexity is determined by the maximum dimension of the input tensor and is O(max(L,M,D)).

[0183] Therefore, the method of this embodiment can reduce memory consumption while maintaining high performance. This shows that multimodal feature fusion through a small-scale model can achieve good results while significantly reducing computing resources.

[0184] In order to verify the influence of each modality and learning similarity and heterogeneity on the method of this embodiment, five ablation experiments are set up. Including: removing the text modality ((-)Text) to verify the influence of introducing text emotional information on video multimodal emotional analysis. Removing the image modality ((-)Visual) to verify the influence of introducing image emotional information on video multimodal emotional analysis. Removing the audio modality ((-)Audio) to verify the influence of introducing audio emotional information on video multimodal emotional analysis. Removing modality similarity learning ((-)L sim ) to verify the influence of learning the similarity between different modalities on video multimodal sentiment analysis. diff ) to verify the influence of learning the heterogeneity between different modalities on video multimodal sentiment analysis. The results of modality-related ablation experiments are shown in Table 4 and 7 shows:

[0185] Table 4: Modality-related ablation experimental results.

[0186]

[0187] Observe Table 4 and Figure 7Ablation experiment results show that different modalities contribute differently to sentiment analysis in multimodal video sentiment analysis. In multimodal video sentiment analysis, the text modality is the most important. Removing the text modality significantly reduces the performance, making it indispensable for multimodal sentiment analysis. The audio modality also has a certain impact on sentiment analysis results, while the image modality has a relatively smaller impact. Furthermore, learning similarities between different modalities can lead to more accurate sentiment analysis results, and learning heterogeneity can also improve the effectiveness of the method to a certain extent.

[0188] In order to verify the influence of each component in the multi-layer perceptron fusion module on the method of this embodiment, three ablation experiments are set up. Including: removing the sequence fusion axis ((-)MLP-L) to verify the influence of sequence fusion of each modality on video multimodal sentiment analysis. Removing the channel fusion axis ((-)MLP-M) to verify the influence of channel fusion of each modality on video multimodal sentiment analysis. Removing the modal fusion axis ((-)MLP-D) to verify the influence of modal fusion of each modality on video multimodal sentiment analysis. The results of the ablation experiment of the fusion module components are shown in Tables 5 and Figure 8 shown.

[0189] Table 5: Fusion module component ablation experiment results.

[0190]

[0191] Fusion along different axes has varying impacts on the resulting multimodal sentiment features. The sequence axis is the most important, likely due to the temporal nature of video data. Fusion along the sequence axis effectively captures the dynamics of emotion, contextual information, temporal alignment between modalities, and long- and short-term dependencies within the video. These factors combine to make feature fusion along the sequence axis particularly crucial in multimodal sentiment analysis, helping the model more accurately understand and predict emotional states.

[0192] This embodiment achieves full fusion of different modal features by constructing a multi-layer perceptron fusion module. Thanks to the inherent structure of the multi-layer perceptron, the model can flexibly adjust the output dimension of each multi-layer perceptron unit, thereby effectively affecting the final prediction effect. From a mathematical point of view, L', M' and D' represent the dimensions of the input features after affine transformation on the sequence L, modality M and channel D axes. By adjusting different output dimensions, it can have a certain impact on the performance of the method. The experimental results of the output dimension are shown in Figure 2. Figure 9 、 Figure 10 and Figure 11As shown in the figure, observing the results of the output dimension experiment, it can be seen that when the output dimension of any fusion axis is less than 3 (<3), the performance of the method decreases significantly. However, when the output dimension is greater than or equal to 3 (≥3), the fluctuation of the method performance is significantly reduced. The root cause of this phenomenon may be closely related to the three modalities (text, image and audio) involved in the training process. Specifically, the dimension of the input multimodal feature vector is a three-dimensional tensor with a rank of 3 in the initial stage. The diagram of recall rate, accuracy rate and F1 value is shown in the figure. Figure 12 Therefore, when the tensor is transformed into using affine transformation on all fusion axes, this method can more fully capture the complementary emotional information between the modalities, thus achieving the best performance.

[0193] like Figure 13 As shown, the present invention also discloses a video multimodal sentiment analysis device based on multi-layer perceptron fusion, comprising:

[0194] The video multimodal feature extraction module 1301 is configured to extract features from the text, image, and audio data in the video using a text model, an image model, and an audio model, respectively, to obtain static text features, image features, and audio features; and to extract time sequence information from the image features and the audio features using a time sequence feature extraction model, respectively, to obtain time sequence image features and time sequence audio features;

[0195] A video multimodal sentiment analysis model construction module 1302 is used to construct a video multimodal sentiment analysis model including a multimodal feature capture module, a multi-layer perceptron fusion module, and a sentiment classifier;

[0196] The multimodal feature capture module captures the similarities and heterogeneities between text features, time-series image features, and time-series audio features, and obtains modal sentiment features including text similarity features, text heterogeneity features, image similarity features, image heterogeneity features, audio similarity features, and audio heterogeneity features;

[0197] The multi-layer perceptron fusion module fuses all modal emotion features to obtain fused multi-modal emotion features;

[0198] The sentiment classifier classifies the fused multimodal sentiment features and outputs a sentiment polarity probability distribution;

[0199] The video multimodal sentiment analysis model training module 1303 is configured to train the video multimodal sentiment analysis model using the labeled data based on a total loss function to obtain a trained video multimodal sentiment analysis model; the total loss function includes a similarity loss constructed based on modal similarity, a heterogeneity loss constructed based on modal heterogeneity, and a classification loss constructed based on the labeled data;

[0200] The video emotion prediction module 1304 is used to input text features, time-series image features, and time-series audio features into a trained video multimodal emotion analysis model to obtain emotion polarity probability distribution and implement video emotion prediction.

[0201] The specific implementation of the video multimodal sentiment analysis device based on multi-layer perceptron fusion is the same as the video multimodal sentiment analysis method based on multi-layer perceptron fusion, and will not be repeated in this embodiment.

[0202] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited to this. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A video multimodal sentiment analysis method based on multi-layer perceptron fusion, characterized in that: The steps include: S1, using the text model, image model and audio model to extract features of the text, image and audio data in the video respectively, to obtain static text features, image features and audio features; Using a time series feature extraction model to extract time series information from the image features and the audio features, respectively, to obtain time series image features and time series audio features; S2, building a video multimodal sentiment analysis model including a multimodal feature capture module, a multi-layer perceptron fusion module and a sentiment classifier; The multimodal feature capture module captures the similarities and heterogeneities between text features, time-series image features, and time-series audio features, and obtains modal sentiment features including text similarity features, text heterogeneity features, image similarity features, image heterogeneity features, audio similarity features, and audio heterogeneity features; The multi-layer perceptron fusion module fuses all modal emotion features to obtain fused multi-modal emotion features; The sentiment classifier classifies the fused multimodal sentiment features and outputs a sentiment polarity probability distribution; S3, based on a total loss function, using the labeled data to train the video multimodal sentiment analysis model to obtain a trained video multimodal sentiment analysis model; the total loss function includes a similarity loss constructed according to modality similarity, a heterogeneity loss constructed according to modality heterogeneity, and a classification loss constructed according to the labeled data; S4, inputs text features, temporal image features and temporal audio features into the trained video multimodal sentiment analysis model to obtain the sentiment polarity probability distribution and realize video sentiment prediction.

2. The video multimodal sentiment analysis method based on multi-layer perceptron fusion according to claim 1 is characterized in that: The multi-layer perceptron fusion module uses a multi-layer perceptron to interactively fuse all modal emotion features from the sequence dimension, channel dimension and modal dimension in turn to obtain fused multi-modal emotion features.

3. The video multimodal sentiment analysis method based on multi-layer perceptron fusion according to claim 2 is characterized in that: The multi-layer perceptron fusion module uses a multi-layer perceptron to interactively fuse all modal sentiment features from the sequence dimension, channel dimension, and modal dimension in turn to obtain the fused multi-modal sentiment features, as follows: Arrange all modal sentiment features into sequence tensors in chronological order, use the sequence axis fusion unit of the sequence-oriented multi-layer perceptron to perform sequence axis fusion on the sequence tensors, and output sequence fusion features; expressed as: ； in, represents sequence fusion features, Indicates fixed mode and channel Fuse the sequence axes; Representation layer normalization; represents a fully connected layer; represents nonlinear activation; Represents the eigenvector on the sequence axis; represents the modal number; Indicates the number of channels; Indicates the number of sequences after fusion; The channel dimension tensor of the sequence fusion feature is reconstructed to obtain the channel tensor, and the channel axis is fused on the channel tensor using the channel-oriented multi-layer perceptron, and the channel fusion feature is output; it is expressed as: ； in, represents the channel fusion feature, Represents a fixed sequence and modal Fuse the channel axes; represents the eigenvector on the channel axis; Indicates the number of channels after fusion; The modal tensor is obtained by reconstructing the modal dimension of the channel fusion feature. The modal axis fusion of the modal tensor is performed using a modal multi-layer perceptron, and the fused multi-modal sentiment feature is output; it is expressed as: ； in, Represents the fused multimodal sentiment features, Represents a fixed sequence and channel Fusion of modal axes; represents the eigenvector on the modal axis; Indicates the number of modalities after fusion.

4. The video multimodal sentiment analysis method based on multi-layer perceptron fusion according to claim 1 is characterized in that The text model is the BERT model; the image model is the OpenFace image feature processing tool; the audio model is the COVAREP tool; and the temporal feature extraction model is the unidirectional long short-term memory network sLSTM.

5. The video multimodal sentiment analysis method based on multi-layer perceptron fusion according to claim 1 is characterized in that: The total loss function is expressed as: ； in, represents the classification loss; Represents similarity loss; represents heterogeneity loss; and represents the regularization weight coefficient.

6. The video multimodal sentiment analysis method based on multi-layer perceptron fusion according to claim 1 is characterized in that: The similarity loss is expressed as: ； in, Represents similarity loss; represents the regularized central moment difference, Indicates the order; Indicates modality similarity characteristics; Indicates modality similarity characteristics; Indicates text mode; Indicates audio mode; Indicates video mode.

7. The video multimodal sentiment analysis method based on multi-layer perceptron fusion according to claim 1 is characterized in that: The heterogeneity loss is expressed as: ； in, represents heterogeneity loss; represents the square of the Frobenius norm; Indicates modality The similarity feature matrix representation of The transpose of Indicates modality The specific feature matrix representation of The transpose of Indicates modality The specific feature matrix representation of Indicates modality The specific feature matrix representation of Indicates text mode; Indicates audio mode; Indicates video mode.

8. The video multimodal sentiment analysis method based on multi-layer perceptron fusion according to claim 1 is characterized in that: The capturing of similarities and heterogeneities among text features, temporal image features, and temporal audio features is as follows: The text features, time-series image features, and time-series audio features are respectively input into a feedforward neural network with shared learnable parameters to learn the common features between modalities and obtain text similarity features, image similarity features, and audio similarity features. The text features, time-series image features and time-series audio features are respectively input into a feedforward neural network with independent learnable parameters to learn the heterogeneous features of each modality and obtain text heterogeneity features, image heterogeneity features and audio heterogeneity features.

9. The video multimodal sentiment analysis method based on multi-layer perceptron fusion according to claim 1 is characterized in that: The classification loss is cross entropy loss or mean absolute error loss; the cross entropy loss is expressed as: ； in, represents the classification loss; Indicates the number of sentiment categories; represents the true label; represents the predicted label; The mean absolute error loss is expressed as: ； in, Indicates the number of sentiment categories; represents the true label; represents the predicted label; represents the L2 norm squared.

10. A video multimodal sentiment analysis device based on multi-layer perceptron fusion, comprising the following: The video multimodal feature extraction module is used to extract features from the text, image and audio data in the video using the text model, image model and audio model respectively, to obtain static text features, image features and audio features; Using a time series feature extraction model to extract time series information from the image features and the audio features, respectively, to obtain time series image features and time series audio features; Video multimodal sentiment analysis model construction module, used to build a video multimodal sentiment analysis model including a multimodal feature capture module, a multi-layer perceptron fusion module and a sentiment classifier; The multimodal feature capture module captures the similarities and heterogeneities between text features, time-series image features, and time-series audio features, and obtains modal sentiment features including text similarity features, text heterogeneity features, image similarity features, image heterogeneity features, audio similarity features, and audio heterogeneity features; The multi-layer perceptron fusion module fuses all modal emotion features to obtain fused multi-modal emotion features; The sentiment classifier classifies the fused multimodal sentiment features and outputs a sentiment polarity probability distribution; A video multimodal sentiment analysis model training module is used to train the video multimodal sentiment analysis model using labeled data based on a total loss function to obtain a trained video multimodal sentiment analysis model; the total loss function includes a similarity loss constructed based on modal similarity, a heterogeneity loss constructed based on modal heterogeneity, and a classification loss constructed based on labeled data; The video emotion prediction module is used to input text features, time-series image features, and time-series audio features into the trained video multimodal emotion analysis model to obtain the emotion polarity probability distribution and realize video emotion prediction.

Images