A multi-modal sentiment analysis method based on a multi-modal fusion network

Abstract

The application discloses a multi-modal sentiment analysis method based on a multi-modal fusion network, which acquires a sentiment information mode to be processed, performs data extraction on the sentiment information mode to be processed, inputs the sentiment information mode after data extraction into a trained multi-modal sentiment analysis model, and outputs a sentiment prediction result of the sentiment information mode to be processed; the trained multi-modal sentiment analysis model captures diversified features of mode data through a multi-scale feature extraction module, fuses different modes by using a text multi-modal feature fusion module, introduces unsupervised contrast learning to mine information between the fused features and the multi-scale features, and further optimizes feature representation; the method enhances extraction of different scale features by the model, fully utilizes a text mode, and improves the accuracy of sentiment analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal sentiment analysis, and in particular to a multimodal sentiment analysis method based on a multimodal fusion network. Background Art

[0002] Multimodal sentiment analysis (MSA) is a comprehensive emotion recognition technology that integrates multiple information sources, including language, facial expressions, voice, music, and body movements, with the goal of deeply revealing and analyzing user emotions. In the early stages of MSA research, researchers primarily relied on handcrafted feature extraction methods, such as bag-of-words models, local binary patterns, and scale-invariant feature transforms. With the rapid development of deep learning, convolutional neural networks, recurrent neural networks, and long short-term memory networks have become key research tools in data processing. To further improve the accuracy of MSA, researchers have explored strategies for integrating information from different modalities. Early fusion methods focused on integrating data from different sources into a unified representation. For example, TFN and LMF integrated information by constructing high-dimensional tensors using LSTM. Subsequently, modeling modal interactions has become a research focus. For example, MCTN optimizes modal transitions using a cycle consistency loss, MulT uses a Transformer architecture for intermodal encoding, and ICCN uses deep learning to improve intermodal consistency in high-dimensional nonlinear spaces. Recently, researchers have begun to focus on methods that emphasize both modal consistency and differentiation. For example, Yu et al. employed a self-supervised learning strategy to design an automatic label generation module that reduces inter-modal differences and is applicable to both multimodal and unimodal training tasks. Han et al. introduced the concept of mutual information and proposed a learning framework that maximizes mutual information to avoid losing task-relevant information. Hazarika et al. projected modal vectors into different spaces and used a regularization component to learn representations of shared and unique modal features.

[0003] While existing methods have made some progress in improving performance, they typically treat all modalities equally, which limits further improvement. For example, on the CMU-MOSI dataset, a sentiment prediction strategy that assumes equal contributions from all three modalities yielded suboptimal results. This is because in multimodal sentiment analysis, the contributions of text, visual, and auditory modalities vary. Text, in particular, has a significant impact on the accuracy of sentiment analysis due to its rich emotional cues.

[0004] Therefore, how to make full use of text modalities to improve the accuracy and efficiency of sentiment analysis is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0005] In response to the above-mentioned deficiencies in the existing technology, the present invention provides a multimodal sentiment analysis method based on a multimodal fusion network. Through a pyramid-structured multi-scale feature extraction module, a text multimodal feature fusion module, and unsupervised comparative learning, the model enhances the extraction of features at different scales, fully utilizes the text modality, and improves the accuracy of sentiment analysis.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A multimodal sentiment analysis method based on a multimodal fusion network includes the following steps:

[0008] S1. Obtaining an emotional information modality to be processed and performing data extraction on the emotional information modality to be processed, wherein the emotional information modality includes a text modality, an audio modality, and a video modality;

[0009] S2. Inputting the emotional information modality after data extraction into the trained multimodal emotion analysis model, and outputting the emotion prediction result of the emotional information modality to be processed; the multimodal emotion analysis model includes a multi-scale feature extraction module, a text multimodal feature fusion module and a multimodal emotion prediction module;

[0010] The training steps of the multimodal sentiment analysis model are as follows:

[0011] S201, the multimodal sentiment analysis model uses the emotional information modality extracted from the input data as the input of the multiscale feature extraction module, outputs multiscale features with text modality, audio modality and video modality, and inputs them into the text multimodal feature fusion module;

[0012] S202: The text multimodal feature fusion module fuses the input multi-scale features of different modalities through a text gating unit with the text modality as the core to generate fused features, designs a text channel attention Transformer unit with a multimodal channel attention mechanism, uses the fused features as input to the text channel attention Transformer unit, and outputs modal fusion features as input to the multimodal sentiment prediction module;

[0013] S203, the multimodal emotion prediction module connects the input modal fusion features to obtain a fusion feature, and obtains an emotion prediction result of the emotion information modality to be processed according to the fusion feature;

[0014] S204, optimizing and updating the model parameters of the multimodal sentiment analysis model with the goal of minimizing the total loss function constructed by the model basic loss function and the unsupervised contrastive learning loss function, thereby completing the training of the multimodal sentiment analysis model;

[0015] S205. Repeat steps S201 to S204 to perform iterative training until the multimodal sentiment analysis model converges or reaches a preset number of iterations.

[0016] As a preferred solution, the data extraction specifically includes:

[0017] The step of extracting data from the text modality includes: extracting features from the input text sequence using a pre-trained BERT model to obtain a global representation and a local representation of the text modality, wherein the global representation is a corresponding vector marked by a special tag [CLS], and the [CLS] tag vector corresponding to the last layer of the BERT model is used as a comprehensive representation of the text sequence. The expression for data extraction from text mode is as follows:

[0018]

[0019] Where; Represents the text features extracted by the BERT model, l t is the maximum length of the text sequence, d t represents the dimension of comprehensive representation of text sequence, Represents the learnable parameters of text modality in BERT;

[0020] The step of extracting data from the audio modality includes: extracting features from the audio modality using the pre-trained COVAREP model to obtain an initial vector feature I a Then, the initial vector feature I a Through the sLSTM model, the characteristics of its time series are deeply captured. Finally, the end-state hidden vector is selected as the comprehensive representation of the audio sequence. The expression for data extraction of the audio modality is as follows:

[0021]

[0022] Where, Represents the audio features extracted by the pre-trained COVAREP model, l a Indicates the sequence length of the audio, d a represents the dimension of the comprehensive representation of the audio sequence, Represents the learnable parameters of the audio modality in the sLSTM model;

[0023] The step of extracting data from the video modality includes: extracting features from the video modality using the pre-trained FACET model to obtain an initial vector I vThen, the initial vector feature I v Through the sLSTM model, the features in its time series are deeply captured. Finally, the end-state hidden vector is selected as the comprehensive representation of the video sequence. The expression for data extraction of the video modality is as follows:

[0024]

[0025] Where, represents the video features extracted by the pre-trained FACET model, l v Indicates the sequence length of the video, d v represents the dimension of the comprehensive representation of the video sequence, Represents the learnable parameters of the video modality in sLSTM.

[0026] As a preferred solution, in step S201, the multi-scale feature extraction module includes four convolution blocks and a channel attention unit in parallel, wherein the four convolution blocks in parallel are a first convolution block, a second convolution block, a third convolution block, and a fourth convolution block, and the parallel connection result of the second convolution block and the third convolution block is cascaded with the channel attention unit;

[0027] The first convolution block is a 1×1 convolution layer, the second convolution block is a 3×3 convolution layer, the third convolution block is a 5×5 convolution layer, and the fourth convolution block is a global pooling layer; the outputs of the second and third convolution blocks in parallel are used as the input of the channel attention unit, and the output of the channel attention unit is then added with the outputs of the first and fourth convolution blocks to obtain the overall output of different modalities in the multi-scale feature extraction module.

[0028] As a preferred solution, the channel attention unit includes a cascaded global average pooling operation, an excitation operation, and a scaling operation; in the channel attention unit, the outputs of the second and third convolution blocks of the input are compressed along the sequence length using a global average pooling operation to calculate global spatial information, and then an excitation operation is performed to generate weights for each feature channel. Finally, the outputs of different channels are controlled by scaling operations and weight parameters, thereby outputting features that have passed through the channel attention unit;

[0029] The processing process of the channel attention unit is expressed as:

[0030]

[0031] η=Excitation(Z,W)=σ(g(Z,W))=σ(W2ReLU(W1Z));

[0032] X′ sn =Scale(Xsn ,η)=X sn ×η;

[0033] Where Z represents the global spatial information, Squeeze(·) represents the global average pooling operation, and X sn express After the output features of the second and third convolution blocks, Represents the features obtained by data extraction of different modalities, s∈{a,t,v}, n∈{2,3}, η represents the weight parameter, Excitation(·,w) represents the excitation operation, g(·) is the nonlinear function, σ(·) represents the sigmoid function, W1 and W2 both represent the weight matrix of the fully connected layer, ReLU represents the activation function, X' vn represents the output feature after channel attention enhancement, and Scale(·) represents the scaling operation;

[0034] The processing process of the multi-scale feature extraction module is expressed as:

[0035]

[0036] Where, X a represents the multi-scale features of audio modality, X t represents the multi-scale features of text modality, X v represents the multi-scale features of the video modality, Concat(·) represents the connection, X a1 、X t1 、X v1 Represent the output of audio, text and video modalities after the first convolutional block, X a4 、X t4 、X v4 Represent the outputs of the fourth convolutional block for audio, text, and video modalities, respectively.

[0037] As a preferred solution, in step S202, the text gating unit includes an audio text gating subunit and a video text gating subunit, and the audio text gating subunit has the same structure as the video text gating subunit, including a cascaded convolution normalization unit, a convolution activation unit, a normalization activation unit, and a Sigmoid activation function layer, wherein the convolution normalization unit includes a cascaded first 1×1 convolution layer and a first batch normalization layer, the convolution activation unit includes a cascaded first ReLU activation function layer and a second 1×1 convolution layer, and the normalization activation unit includes a cascaded second batch normalization layer and a second ReLU activation function layer;

[0038] In the video-text gating subunit, the input text modality multi-scale features and the video modality multi-scale features are processed by the convolution normalization unit respectively, and the processed features are added to obtain a fused feature map. Then, the fused feature map is passed through the convolution activation unit to obtain a single-channel feature map, and then processed by the normalization activation unit. Then, the processed single-channel feature map is converted into an attention coefficient through a Sigmoid activation function layer. Finally, the video modality multi-scale features as the input are element-wise multiplied with the output of the Sigmoid activation function layer to obtain the features output by the video-text gating subunit;

[0039] In the audio-text gating subunit, the input text modality multi-scale features and the audio modality multi-scale features are used as inputs of the audio-text gating subunit, and are processed by the convolution normalization unit respectively, and the processed features are added to obtain a fused feature map, and then the fused feature map is passed through the convolution activation unit to obtain a single-channel feature map, and then processed by the normalization activation unit. Then, the processed single-channel feature map is converted into an attention coefficient through a Sigmoid activation function layer, and finally, the video modality multi-scale features as the input are element-wise multiplied with the output of the Sigmoid activation function layer to obtain the features output by the audio-text gating subunit;

[0040] The features output by the video text gating subunit and the features output by the audio text gating subunit are used as the overall output features of the text gating unit;

[0041] The video text gating sub-unit processing process is expressed as follows:

[0042] X′ tv =BN(Conv 1×1 (X t ))+BN(Conv 1×1 (X v ));

[0043] X tv =X v ⊙σ(δ(BN(Conv 1×1 (δ(X′ tv )))));

[0044] Where X′ tv Represents the fusion feature map of the multi-scale features of the text modality and the multi-scale features of the video modality, BN(·) represents batch normalization, Conv 1×1 (·) represents a one-dimensional convolutional layer, X t represents the multi-scale features of text modality, X v represents the multi-scale features of video modality, X tvrepresents the fusion of the multi-scale features of the video modality and the multi-scale features of the text modality output by the video-text gating subunit, ⊙ represents element-wise multiplication, and δ(·) represents the ReLU activation function;

[0045] The audio text gating sub-unit processing process is expressed as follows:

[0046] X′ ta =BN(Conv 1×1 (X t ))+BN(Conv 1×1 (X a ));

[0047] X ta =X v ⊙σ(δ(BN(Conv 1×1 (δ(X′ ta )))));

[0048] Where X′ ta Represents the fusion feature map of the multi-scale features of the text modality and the multi-scale features of the audio modality, X t represents the multi-scale features of text modality, X a represents the multi-scale features of the audio modality, X ta Represents the features of the audio modality and text modality fused at the output of the audio-text gating subunit.

[0049] As a preferred solution, in step S202, the text channel attention Transformer unit with a multimodal channel attention mechanism includes N layers of network layers consisting of a cascade of a multimodal channel attention module and a fully connected feedforward module; the multimodal channel attention module includes a first normalization layer and a multi-head attention unit connected in sequence, and the fully connected feedforward module includes a second normalization layer and a fully connected feedforward unit connected in sequence;

[0050] In the text channel attention Transformer unit, the input feature map of the text channel attention Transformer unit is used as the input of the first normalization layer in the multimodal channel attention module in the first network layer, the output of the first normalization layer is used as the input of the multi-head attention unit, the output of the modal channel attention unit is added to the input of the channel attention Transformer unit, and the output is used as the input of the second normalization layer in the fully connected feedforward module. The output of the second normalization layer is used as the input of the fully connected feedforward unit, the input of the fully connected feedforward unit is added to the output of the multimodal channel attention module, and its output is used as the input of the next network layer. After N layers of feedforward calculation, its output is used as the overall output of the text channel attention Transformer unit;

[0051] The processing process is represented as:

[0052]

[0053]

[0054] In the formula, represents the fusion feature representation of the text modality and the video modality after being processed by the multi-head attention unit at the i-1th layer, represents the fusion feature representation of the text modality and the audio modality after being processed by the multi-head attention unit at the i-1th layer, all represent the multi-head attention unit of the i-th layer, and LN(·) represents a layer normalization layer, represents the fusion feature representation of the text modality and the video modality at the i-1th layer, represents the fusion feature representation of the text modality and the audio modality at the i-1th layer, represents the full connection feedforward unit of the i-th layer, represents the fusion feature representation of the text modality and the video modality after being processed by the full connection feedforward module at the i-th layer, represents the fusion feature representation of the text modality and the audio modality after being processed by the full connection feedforward module at the i-th layer, and TCAT(·) represents a text channel attention Transformer unit, ta represents the feature after the audio text gate sub-unit outputs the fusion of the audio modality and the text modality, a represents the audio modality multi-scale feature, v represents the video modality multi-scale feature, represents the modality fusion feature after the text modality and the audio modality are fused by the text channel attention Transformer unit, represents the modality fusion feature after the text modality and the video modality are fused by the text channel attention Transformer unit.

[0055] As a preferred solution, the multi-head attention unit projects the fused feature obtained after being processed by the text gate unit and the multi-scale feature into a query, key, and value vector space, and then applies a scaled dot-product attention mechanism in the channel dimension to generate an output vector;

[0056] The X tv The processing process of the multi-head attention unit is represented as:

[0057]

[0058] In the formula, Y tv represents X tvThe output after attention mechanism processing, Attention(·) represents the attention mechanism, Q tv Represents X tv The query matrix, K v Represents X v The bond matrix, V v Represents X v The value matrix of , Softmax(·) represents the Softmax function, Represents the weight of the query matrix, W K represents the weight of the bond matrix, represents the weight of the value matrix, Represents the output of each head in the multi-head attention unit of the video modality, and W0 represents the weight matrix of the output after splicing the multi-head attention unit.

[0059] The X ta The processing process of the multi-head attention unit is expressed as:

[0060]

[0061] Where Y ta Represents X ta The output after attention mechanism processing, Q ta Represents X ta The query matrix, K a Represents X a The bond matrix, V a Represents X a The value matrix of represents the weight of the value matrix, The output of each head in the multi-head attention unit representing the audio modality.

[0062] As a preferred solution, in step S203, the processing of the multimodal emotion prediction module includes:

[0063] The audio and text modality fusion features as input are connected with the video and text modality fusion features, and then the features are mapped to a lower-dimensional feature space through the first linear layer of the activation function to obtain the fusion feature X m , the fused features are passed through the second linear layer to generate the sentiment prediction result as output;

[0064] The processing process is expressed as:

[0065]

[0066] Where, X m represents the fusion feature obtained after the first linear layer processing, ReLU(·) represents the activation function, represents the weight parameter of the first linear layer, represents the bias parameter of the first linear layer, represents the sentiment prediction result, represents the weight parameter of the second linear layer, Represents the bias parameters of the second linear layer.

[0067] As a preferred solution, in step S204, the unsupervised contrastive learning loss function is:

[0068]

[0069] Where Point(·) represents the correlation quantization function, exp(·) represents the exponential function, and X s Represents multi-scale features of different modalities, Represents X s The normalized eigenvector of Indicates that the parameter is neural network, Represents X m With X s The contrast loss function between s represents the expected value of different modes, represents the fusion feature of the jth sample, Represents the multi-scale features of different modalities of the j-th sample, represents the unsupervised contrastive learning loss function, represents the contrast loss between the fusion feature and the text modality, represents the contrast loss between the fused features and the audio modality, represents the contrast loss between the fused features and the video modality;

[0070] The basic loss function of the model is:

[0071]

[0072] Where, Represents the basic loss function of the model, N represents the total number of training samples, Represents the predicted value of the multimodal sentiment analysis model for the i-th sample, Represents the true value of the i-th sample;

[0073] As a preferred solution, the total loss function is:

[0074]

[0075] Where, represents the total loss function, α represents The weight of The weight of .

[0076] Compared with the prior art, the present invention has the following technical effects:

[0077] (1) The present invention constructs a multimodal sentiment analysis model, which takes into account the multi-scale features of modal data that are often overlooked in the prior art through a multi-scale feature extraction module, and designs a text multimodal feature fusion module to fully utilize the potential of text modality in multimodal sentiment analysis. To further enhance the performance of the model, the model of the present invention also adopts an unsupervised contrastive learning method to deeply explore the intrinsic connection between multi-scale features and fusion features, and optimize the representation of fusion features.

[0078] (2) The pyramid-structured multi-scale feature extraction module proposed in this paper uses four convolution kernels of different sizes to gradually capture the features of modal data at different scales, and introduces a channel attention mechanism through the second and third convolution blocks to further enhance the expressiveness of features. The channel attention mechanism strengthens the features of key channels while suppressing non-key information, thereby enhancing the model's ability to capture important information.

[0079] (3) The present invention designs a text multimodal feature fusion module, which strengthens the guiding role of text information in the feature fusion process through the text gating unit, so that the model can pay more attention to the key emotional clues in the text, realize the effective fusion of modal feature map information, and capture the relationship between different modalities through the text channel attention Transformer unit, realizing effective cross-modal feature interaction, which helps the model capture and utilize the complementary information between different modalities and improve the accuracy of sentiment analysis; secondly, through the multimodal channel attention mechanism, the model can adaptively adjust the weights of different modal features, highlight features that are more relevant to the sentiment analysis task, and suppress unimportant or noise features. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to make the purpose, technical solutions and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0081] Figure 1 This is a framework diagram of the multimodal sentiment analysis model of the present invention;

[0082] Figure 2 This is a structural diagram of the channel attention module of the present invention;

[0083] Figure 3 This is a structural diagram of the text multimodal feature fusion module of the present invention;

[0084] Figure 4 This is the t-SNE visualization result of UCL in the embodiment of the present invention.

[0085] Figure 5This is a performance comparison chart of various models on CMU-OSI in an embodiment of the present invention. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0087] The present invention will be described in further detail below with reference to the accompanying drawings.

[0088] Multimodal sentiment analysis (MSA), a core component of human-computer interaction, has rapidly developed into a highly anticipated research hotspot in recent years. This rapid development is driven by its significant role in understanding human emotions, enhancing interactive experiences, and optimizing decision-making. With the continuous advancement of artificial intelligence (AI), multimodal sentiment analysis is becoming a crucial bridge connecting human emotion and machine intelligence. However, this field also faces numerous challenges, including feature extraction, feature fusion, and processing the contributions of different modalities.

[0089] To address the above problems, the present invention constructs a new multimodal sentiment analysis model (TMFN) and proposes a multimodal sentiment analysis method based on a multimodal fusion network based on the model. The method includes the following steps:

[0090] S1. Obtaining an emotional information modality to be processed and performing data extraction on the emotional information modality to be processed, wherein the emotional information modality includes a text modality, an audio modality, and a video modality;

[0091] S2. Inputting the emotional information modality after data extraction into the trained multimodal emotion analysis model, and outputting the emotion prediction result of the emotional information modality to be processed; the multimodal emotion analysis model includes a multi-scale feature extraction module, a text multimodal feature fusion module and a multimodal emotion prediction module;

[0092] The training steps of the multimodal sentiment analysis model are as follows:

[0093] S201, the multimodal sentiment analysis model uses the emotional information modality extracted from the input data as the input of the multiscale feature extraction module, outputs multiscale features with text modality, audio modality and video modality, and inputs them into the text multimodal feature fusion module;

[0094] S202: The text multimodal feature fusion module fuses the input multi-scale features of different modalities through a text gating unit with the text modality as the core to generate fused features, designs a text channel attention Transformer unit with a multimodal channel attention mechanism, uses the fused features as input to the text channel attention Transformer unit, and outputs modal fusion features as input to the multimodal sentiment prediction module;

[0095] S203: The multimodal emotion prediction module connects the input modal fusion features to obtain fusion features, and obtains an emotion prediction result of the emotion information modality to be processed based on the fusion features;

[0096] S204, optimizing and updating the model parameters of the multimodal sentiment analysis model with the goal of minimizing the total loss function constructed by the model basic loss function and the unsupervised contrastive learning loss function, thereby completing the training of the multimodal sentiment analysis model;

[0097] S205. Repeat steps S201 to S204 to perform iterative training until the multimodal sentiment analysis model converges or reaches a preset number of iterations.

[0098] The present application firstly proposes a pyramid structure multi-scale feature extraction method, which uses convolution kernels of different sizes to capture diversified features of modal data, and through a channel attention mechanism, the features of key channels are strengthened while non-key information is suppressed, thereby enhancing the ability of the model to capture important information. Secondly, in view of the advantages of the text modality in multi-modal sentiment analysis, the present application designs a text multi-modal feature fusion module, which includes a text gating unit and a text channel attention transformer (TCAT). The text gating unit dynamically adjusts the feature weight of the text content, effectively guiding the fusion with other modal information. This strategy ensures that the features closely related to sentiment analysis are properly emphasized, thereby significantly improving the accuracy and efficiency of sentiment analysis. TCAT is used to perform cross-modal information fusion, which uses a multimodal channel attention mechanism (MCA) to capture the mutual relationship between different modal features. By adaptively adjusting the weights between different channels, TCAT can more accurately identify and utilize cross-modal correlation information, thereby promoting effective interaction between modal features. Finally, in order to optimize the model performance and improve the quality of the fused features, unsupervised contrastive learning is used to deeply mine the mutual information between the fused features and the multi-scale features, optimize the fused feature representation, and further improve the performance of the model.

[0099] The multi-modal sentiment analysis method based on the multi-modal fusion network of the present application will be described in more detail below.

[0100] 1. Data extraction

[0101] For multi-modal sentiment analysis, text, audio and video are the three main information modalities.

[0102] The data extraction step for the text modality includes using a pre-trained BERT model to perform feature extraction on the input text sequence I t ={w C ,w0,…,w i ,…,w S}, where w C and w S represent special tokens [CLS] and [SEP] respectively, to obtain global and local representations of the text modality. The global representation is the corresponding vector marked by the special token [CLS], and the corresponding [CLS] token vector of the last layer of the BERT model is taken as the comprehensive representation of the text sequence The expression for data extraction of the text modality is as follows:

[0103]

[0104] Where; Represents the text features extracted by the BERT model, l t is the maximum length of the text sequence, d t represents the dimension of comprehensive representation of text sequence, Represents the learnable parameters of text modality in BERT;

[0105] The step of extracting data from the audio modality includes: extracting features from the audio modality using the pre-trained COVAREP model to obtain an initial vector feature. Then, the initial vector feature I a Through the sLSTM model, the characteristics of its time series are deeply captured. Finally, the end-state hidden vector is selected as the comprehensive representation of the audio sequence. The expression for data extraction of the audio modality is as follows:

[0106]

[0107] Where, Represents the audio features extracted by the pre-trained COVAREP model, l a Indicates the sequence length of the audio, d a represents the dimension of the comprehensive representation of the audio sequence, Represents the learnable parameters of the audio modality in sLSTM;

[0108] The step of extracting data from the video modality includes: extracting features from the video modality using a pre-trained FACET model to obtain an initial vector Then, the initial vector feature I v The sLSTM model (single directional Long Short-Term Memory) is used to deeply capture the characteristics of its time series. Finally, the end-state hidden vector is selected as the comprehensive representation of the video sequence. The expression for data extraction of the video modality is as follows:

[0109]

[0110] Where, represents the video features extracted by the pre-trained FACET model, l v Indicates the sequence length of the video, d v represents the dimension of the comprehensive representation of the video sequence, Represents the learnable parameters of the video modality in sLSTM.

[0111] 2. Multimodal Sentiment Analysis Model

[0112] like Figure 1 As shown in Figure 2, the multimodal sentiment analysis model includes a multi-scale feature extraction module, a text multimodal feature fusion module, and a multimodal sentiment prediction module. The following describes each of these modules.

[0113] 2.1 Multi-scale feature extraction module

[0114] Multi-scale feature extraction technology plays a key role in many fields such as image processing, computer vision and pattern recognition. By extracting image features at different scales, it can comprehensively capture image information and significantly improve the model's ability to understand and represent image content.

[0115] This invention proposes a new multi-scale feature extraction method of pyramid structure. Different from the prior art, this invention takes into account the initial features. The contextual semantic information has been fully captured through the pre-training model and sLSTM (singledirectional Long Short-Term Memory) technology. Based on this, we use four convolution blocks of different sizes to fully extract the multi-scale features of the target object, and further enhance the expressive power of the features by using the channel attention mechanism in the second and third convolution blocks. Specifically, the multi-scale feature extraction module of the present invention includes four convolution blocks and a channel attention unit in parallel, wherein the four convolution blocks in parallel are the first convolution block, the second convolution block, the third convolution block and the fourth convolution block, and the parallel result of the second convolution block and the third convolution block is cascaded with the channel attention unit;

[0116] The first convolution block is a 1×1 convolution layer, the second convolution block is a 3×3 convolution layer, the third convolution block is a 5×5 convolution layer, and the fourth convolution block is a global pooling layer; the outputs of the second and third convolution blocks in parallel are used as the input of the channel attention unit, and the output of the channel attention unit is then added with the outputs of the first and fourth convolution blocks to obtain the overall output of different modalities in the multi-scale feature extraction module.

[0117] like Figure 2As shown, the channel attention unit includes a cascaded global average pooling operation, an excitation operation, and a scaling operation; in the channel attention unit, the outputs of the second and third convolution blocks of the input are compressed along the sequence length using a global average pooling operation to calculate the global spatial information, and then the excitation operation is performed to generate weights for each feature channel. Finally, the outputs of different channels are controlled by scaling operations and weight parameters, thereby outputting the features that have passed the channel attention unit.

[0118] Below, taking the video modality as an example, the processing process of the multi-scale feature extraction module is as follows:

[0119] First, the initial features are trained by using convolution blocks of different sizes. Processed to capture its multi-scale features:

[0120]

[0121] In the formula, k represents the convolution kernel size, d represents the expansion rate, and X v1 、X v2 、X v3 、X v4 Represent the features extracted by the first, second, third, and fourth convolutional blocks respectively;

[0122] It is worth noting that the 1×1 convolution and global average pooling branches are designed to preserve the inherent properties of the input features through the minimum and maximum receptive fields, so we do not apply the channel attention mechanism to these two branches. Next, the features extracted from the second and third layers are (where n∈{2,3}) further feature enhancement is performed through the channel attention mechanism;

[0123] Next, global average pooling is used to compress X along the sequence length vn , to calculate global spatial information:

[0124]

[0125] Then, the excitation operation F is performed ex (·,w) generates weights for each feature channel, similar to the gating mechanism in LSTM, F ex (·,w) uses a fully connected layer to learn the weight parameter η:

[0126] η=Excitation(Z,W)=σ(g(Z,W))=σ(W2ReLU(W1Z));

[0127] Then, through F scale(·) And weight parameter η to control the output of different channels, and obtain the feature X′ enhanced by channel attention vn :

[0128] X′ vn =Scale(X vn ,η)=X vn ×η;

[0129] Finally, we connect X v1 、X' vn and X v4 Get the final multi-scale feature X v :

[0130]

[0131] Where Z represents the global spatial information, Squeeze(·) represents the global average pooling operation, and X vn Represents the output features of the second and third convolutional blocks of the video modality, η represents the weight parameter, Excitation(·,w) represents the excitation operation, g(·) is the nonlinear function, σ(·) represents the sigmoid function, W1 and W2 both represent the weight matrix of the fully connected layer, r is the channel transformation ratio, ReLU represents the activation function, X' vn represents the output feature after channel attention enhancement, Scale(·) represents the scaling operation, X v represents the multi-scale features of the video modality, and Concat(·) represents the connection.

[0132] 2.2 Text Multimodal Feature Fusion Module

[0133] Through the multi-scale feature extraction module, we obtain multi-scale features X of different modalities s , s∈{a,t,v}, where X a represents the multi-scale features of audio modality, X t represents the multi-scale features of text modality, X v Representing multi-scale features of video modalities. In multimodal sentiment analysis, text information is considered to be the modality that most reflects the speaker's emotions due to its intuitiveness. Therefore, the present invention designs a text multimodal feature fusion module. This module mainly consists of two parts: the Text Gating Unit (TGU) and the Text Channel Attention Unit (TCAT). These two parts are centered on the text modality and work together to deepen the understanding and utilization of other modal information, effectively improving the accuracy of multimodal sentiment analysis.

[0134] Below, when implementing this specific application, the specific process of multimodal feature fusion is explained in detail by taking the fusion of text and video modalities as an example.

[0135] 2.2.1 Text Gating Unit

[0136] The text gating unit includes an audio text gating subunit and a video text gating subunit. The audio text gating subunit has the same structure as the video text gating subunit, including a cascaded convolution normalization unit, a convolution activation unit, a normalization activation unit and a Sigmoid activation function layer, wherein the convolution normalization unit includes a cascaded first 1×1 convolution layer and a first batch normalization layer, the convolution activation unit includes a cascaded first ReLU activation function layer and a second 1×1 convolution layer, and the normalization activation unit includes a cascaded second batch normalization layer and a second ReLU activation function layer;

[0137] In the video-text gating subunit, the input text modality multi-scale features and the video modality multi-scale features are processed by the convolution normalization unit respectively, and the processed features are added to obtain a fused feature map. Then, the fused feature map is passed through the convolution activation unit to obtain a single-channel feature map, and then processed by the normalization activation unit. Then, the processed single-channel feature map is converted into an attention coefficient through a Sigmoid activation function layer. Finally, the video modality multi-scale features as the input are element-wise multiplied with the output of the Sigmoid activation function layer to obtain the features output by the video-text gating subunit;

[0138] In the audio-text gating subunit, the input text modality multi-scale features and the audio modality multi-scale features are used as the input of the audio-text gating subunit, and the output is obtained as the output feature of the audio-text gating subunit; the features output by the video-text gating subunit and the features output by the audio-text gating subunit are used as the overall output feature of the text gating unit;

[0139] The video text gating sub-unit processing process is expressed as follows:

[0140] X′ tv =BN(Conv 1×1 (X t ))+BN(Conv 1×1 (X v ));

[0141] X tv =X v ⊙σ(δ(BN(Conv 1×1 (δ(X′ tv )))));

[0142] Where X′tv Represents the fusion feature map of the multi-scale features of the text modality and the multi-scale features of the video modality, BN(·) represents batch normalization, Conv 1×1 (·) represents a one-dimensional convolutional layer, X t represents the multi-scale features of text modality, X v represents the multi-scale features of video modality, X tv represents the fusion of the multi-scale features of the video modality and the multi-scale features of the text modality output by the video-text gating subunit, ⊙ represents element-wise multiplication, and δ(·) represents the ReLU activation function.

[0143] 2.2.2 Text Channel Attention Transformer Unit

[0144] like Figure 3 As shown, the text channel attention Transformer unit includes N layers of network layers consisting of a cascade of multimodal channel attention modules and fully connected feedforward modules; the multimodal channel attention module includes a first normalization layer and a multi-head attention unit connected in sequence, and the fully connected feedforward module includes a second normalization layer and a fully connected feedforward unit connected in sequence;

[0145] In the text channel attention Transformer unit, the input feature map of the text channel attention Transformer unit is used as the input of the first normalization layer in the multimodal channel attention module in the first network layer, the output of the first normalization layer is used as the input of the multi-head attention unit, the output of the modal channel attention unit is added to the input of the channel attention Transformer unit, and the output is used as the input of the second normalization layer in the fully connected feedforward module. The output of the second normalization layer is used as the input of the fully connected feedforward unit, the input of the fully connected feedforward unit is added to the output of the multimodal channel attention module, and its output is used as the input of the next network layer. After N layers of feedforward calculation, its output is used as the overall output of the text channel attention Transformer unit;

[0146] The processing process is expressed as:

[0147]

[0148] Where, They represent the text-video modality fusion features and text-audio modality fusion features after processing at the i-1 layer, Both represent the multi-head attention unit of layer i, LN(·) represents the layer normalization layer, represents the i-1th layer text-video fusion features and text-audio modality fusion features, represents the fully connected feedforward unit of layer i, represents the text-video fusion feature and text-audio modality fusion feature after the i-th layer is processed by the fully connected feedforward module, TCAT(·) represents the text channel attention Transformer unit, X ta represents the fusion feature of the audio modality and text modality output by the audio-text gating subunit, X a represents the multi-scale features of audio modality, X v Represents the multi-scale features of video modality, Represents the modal fusion features of the text modality and audio modality after being fused by the text channel attention Transformer unit, Represents the modality fusion features of the text modality and video modality after being fused by the text channel attention Transformer unit.

[0149] In the above TCAT unit, the multi-head attention unit projects the fused features and multi-scale features obtained after processing by the text gating unit into the query, key, and value vector space: Q tv =X tv W Q ,K v =X v W K ,V v =X v W V , where W(·) is a trainable parameter matrix. A scaled dot-product attention mechanism is then applied across the channel dimension. This mechanism generates an output vector by associating a query vector with a series of key-value pairs. In this process, queries, keys, values, and outputs are all represented as vectors. Specifically, the output vector is computed as a weighted sum of value vectors, where the weights are determined by a compatibility function between the query vector and the corresponding key vector.

[0150] The processing process is expressed as:

[0151]

[0152] Where Y tv It represents the output of the fusion features of text and video modalities after being processed by the attention mechanism. Attention(·) represents the attention mechanism, Q tv represents the query matrix, K v represents the bond matrix, V v represents the value matrix, Softmax(·) represents the Softmax function, Represents the weight of the query matrix, W K represents the weight of the bond matrix, represents the weight of the value matrix, The output of each head in the multi-head attention unit is represented by Y, and W0 represents the weight matrix of the output of the multi-head attention unit after splicing.

[0153] 2.2.3, Multimodal sentiment prediction module

[0154] The processing process of the multimodal sentiment prediction module includes: connecting the audio and text modal fusion features as input with the video and text modal fusion features, then mapping the features to a lower-dimensional feature space through the first linear layer of the activation function to obtain the fusion features X m The fusion features are processed through the second linear layer to generate the sentiment prediction results as output;

[0155] The processing process is represented as:

[0156]

[0157] In the formula, X m represents the fusion features obtained by processing through the first linear layer, ReLU(·) represents the activation function, represents the weight parameters of the first linear layer, represents the bias parameters of the first linear layer, represents the sentiment prediction results, represents the weight parameters of the second linear layer, represents the bias parameters of the second linear layer.

[0158] 3, Unsupervised contrastive learning

[0159] In the present application, through the text multimodal feature fusion module, a more optimal multimodal fusion feature representation is obtained. However, the relationship between the multi-scale feature X s (wherein s belongs to {a, t, v}) and the fusion feature X m has not been fully explored. Therefore, the present application uses unsupervised contrastive learning to explore the relationship between the two and optimize the fusion feature representation. We maximize the mutual information between the fusion feature and the input modal to promote the model to efficiently integrate the multi-scale feature into the fusion feature.

[0160] The present application optimizes by the unsupervised contrastive learning method, which includes the following steps:

[0161] A correlation quantization function is used to measure the correlation between the fusion feature and the multi-scale feature, all other modal representations are taken as negative samples, and the contrastive loss between a single modal and the fusion feature is calculated. Then, an unsupervised contrastive learning loss function composed of the loss of the fusion feature with the text, video and audio modalities is constructed, and the fusion feature is optimized with the goal of minimizing the unsupervised contrastive learning loss function, thereby obtaining the optimized fusion feature.

[0162] The processing process is expressed as:

[0163]

[0164] Where Point(·) represents the correlation quantization function, exp(·) represents the exponential function, Represents X s The normalized eigenvector of Indicates that the parameter is neural network, Represents X m With X s The contrast loss function between s represents the expected value of different modes, represents the fusion feature of the jth sample, Represents the multi-scale features of different modalities of the j-th sample, represents the unsupervised contrastive learning loss function, represents the contrast loss between the fusion feature and the text modality, represents the contrast loss between the fused features and the audio modality, represents the contrastive loss between the fused features and the video modality.

[0165] 4. Model training

[0166] The multimodal sentiment analysis model is trained in the following way:

[0167] The emotional information modality of the data extraction completed in advance is input into the multimodal sentiment analysis model, and a total loss function constructed by the model basic loss function and the unsupervised contrastive learning loss function is constructed. The model parameters of the multimodal sentiment analysis model are optimized and updated with the goal of minimizing the loss function, thereby completing the training of the multimodal sentiment analysis model.

[0168] Among them, the basic loss function of the model is:

[0169]

[0170] Where, Represents the basic loss function of the model, N represents the total number of training samples, Represents the predicted value of the multimodal sentiment analysis model for the i-th sample, Represents the true value of the i-th sample;

[0171] The total loss function is:

[0172]

[0173] Where, denotes the total loss function, a denotes the weight of and b denotes the weight of .

[0174] 5. Example

[0175] 5.1. Implementation details

[0176] This example implements the proposed model using the PyTorch framework on two benchmark datasets, CMU-MOSI and CMU-MOSEI. Table 1 shows the data distribution of the two datasets.

[0177] Table 1. Dataset distribution of CMU-MOSI and CMU-MOSEI

[0178] Dataset Train Valid Test Total CMU-MOSI 1284 229 686 2299 CMU-MOSEI 16326 1871 4659 22856

[0179] During the training process, this example uses the Adam optimizer and implements an early-stop strategy for 8 epochs to monitor the performance of the model. In addition, in order to find the best combination of hyperparameters, a random search is performed. Table 2 details the configuration of the CMU-MOSI and CMU-MOSEI datasets. All training and testing processes are completed on a single NVIDIA GeForce RTX3060Ti GPU, and the performance evaluation of the model is based on the average score of five random runs on the test set.

[0180] where, according to previous research work, the results are presented in the form of classification and regression. In the classification task, the focus is on the weighted F1 score (F1-score) and the binary accuracy (Acc-2). For the MOSI and MOEI datasets, two different evaluation methods are used: one considers the score as negative / non-negative (non-exclude zero), and the other considers the score as negative / positive (exclude zero). In the regression task, the mean absolute error (MAE) and the Pearson correlation coefficient (Corr) are reported. It should be noted that, except for MAE, a high value of these metrics generally means better model performance.

[0181] Table 2. Main hyperparameters used in the two datasets

[0182] Hyper-parameter CMU-MOSI CMU-MOSEI EarlyStop 8 8 BatchSize 32 32 NumHeads 8 4 LRforBERT 5e-5 5e-5 LRforOthers 0.01 0.001 EncoderLayer 4 4 HiddenLayer 5 5 OutputDropout 0.3 0.1

[0183] 5.2. Results and analysis

[0184] State-of-the-art multimodal sentiment analysis models were compared on the CMU-MOSI and CMU-MOSEI datasets. The specific comparison results are shown in Tables 3 and 4. In multimodal sentiment analysis, datasets are typically categorized into aligned and unaligned types. Although aligned datasets generally yield better performance, unaligned datasets provide more diverse modal information, helping to improve the model's generalization capabilities in real-world scenarios. Therefore, unaligned datasets were chosen for training and evaluating this model.

[0185] On the CMU-MOSI dataset, as shown in Table 3, the The model representation of comes from MMIM, and the mark The model is represented as derived from the SUGRM model. All models with * are reproduced under the same conditions for fair comparison. The proposed model shows significant performance improvement compared with models using aligned datasets (such as RAVEN, Mult, MISA, MAG-BERT). In particular, compared with the best-performing MAG-BERT model, the proposed model improves by 0.60 / 1.28, 0.73 / 1.37 and 0.025 in Acc-2, F1-Score and Corr indicators, respectively, and MAE is reduced by 9.73%. In addition, compared with other models using non-aligned datasets (such as Self-MM, MMIM, SUGRM, M3SA), the method of this embodiment also shows a competitive advantage. In order to further verify the effectiveness of the model, the three optimal benchmark models of Self-MM, MMIM and SUGRM were reproduced under the same experimental environment, and it was found that among the four evaluation indicators, our model was slightly inferior to MMIM only in the Corr indicator.

[0186] On the CMU-MOSEI dataset, our model performs equally well, as shown in Table 4, which is marked as The model representation of comes from MMIM, and the mark The model representations are derived from the SUGRM model. All models marked with an * were replicated under the same conditions for fair comparison. Compared to the best-performing model, MAG-BERT, using an aligned dataset, our model improves the Acc-2, F1-Score, and Corr metrics by 1.76 / 1.30, 1.42 / 1.47, and 0.033, respectively, while the MAE decreases by 4.64%. Similarly, we followed the parameter settings in the original paper and replicated the three best baseline models, Self-MM, MMIM, and SUGRM, under the same experimental conditions. The results show that our model surpasses these baselines on all evaluation metrics, achieving state-of-the-art performance.

[0187] Table 3 Experimental results on the CMU-MOSI dataset

[0188]

[0189] Table 4 Experimental results on the CMU-MOSEI dataset

[0190]

[0191]

[0192] 5.3 Ablation Study

[0193] 5.3.1. Influence of various modes

[0194] Table 5 shows in detail the performance of the proposed model under different modal combinations. After in-depth analysis, several key conclusions were drawn. (1) In terms of single-modal performance, the text modality performs best, while the video modality is relatively weak. The reason why text can perform well is that it usually contains rich emotion-related information, such as emotional vocabulary and semantic structure, which is crucial for sentiment analysis. (2) Multimodal combinations are generally better than single modalities. Regardless of the combination method, the comprehensive effect of multimodality is always better than the use of a single modality. In particular, when the text modality is included in the multimodal combination, the performance improvement is particularly significant, which is more effective than using audio or video modality alone. This finding further confirms the core role of the text modality in multimodal sentiment analysis and is consistent with the text-dominated multimodal fusion design concept proposed in this embodiment. (3) The combination method (i.e., T∝A∝V) that regards the representation capabilities of the three modalities as equivalent is compared with the feature fusion method adopted in this embodiment. The results show that the method of this embodiment has more advantages. This is mainly due to the strategic positioning of the text modality in the method of this embodiment, which makes it the core hub of the multimodal information fusion process. Through this strategy, information from different modalities can be more effectively integrated, thereby significantly improving the overall representation ability and performance.

[0195] Table 5 Performance of TFMN under different multimodal settings

[0196]

[0197] 5.3.2 Impact of Multi-scale Feature Extraction

[0198] To explore the impact of multi-scale feature extraction on model performance, this example conducted ablation experiments on two datasets to evaluate the contributions of different modules in multi-scale feature extraction. This example tested the following situations:

[0199] w / o 1×1: 1×1 convolutional blocks are not used to extract features in TMFN.

[0200] w / o 3×3: 3×3 convolutional blocks are not used to extract features in TMFN.

[0201] w / o 5×5: 5×5 convolutional blocks are not used to extract features in TMFN.

[0202] w / o Avgpool: avgpool is not used in TMFN to extract features.

[0203] w / o Channel Attention: Channel attention is not used in TMFN for further feature enhancement.

[0204] As shown in Table 6, by analyzing Table 6, the following conclusions can be drawn: (1) The absence of the feature extraction module has an adverse effect on the model performance, which verifies the effectiveness of the multi-scale feature extraction strategy proposed in this embodiment. In particular, removing the Channel Attention module leads to the most obvious performance degradation, while removing the 1×1 convolution module has a relatively small impact. (2) Although the 1×1 convolution helps capture local features, its contribution to global features is limited, so its removal only leads to a slight performance degradation. In contrast, removing the average pooling layer leads to a significant performance degradation because average pooling retains the key part of spatial information while reducing the feature dimension. (3) The absence of the Channel Attention module has the greatest impact on the model performance, which emphasizes the importance of implementing an effective feature enhancement mechanism based on multi-size convolution kernels. (4) Removing the 3×3 or 5×5 convolution modules also leads to performance degradation because these modules play a core role in extracting local and neighborhood features. Therefore, the channel attention mechanism is applied to the 3×3 and 5×5 convolution outputs to strengthen the features and further improve the overall performance of the model.

[0205] Table 6 The influence of different modules in multi-scale feature extraction

[0206]

[0207] 5.4. Example Study

[0208] 5.4.1 Unsupervised Contrastive Learning

[0209] In order to deeply analyze Unsupervised contrastive learning (UCL), this embodiment uses the t-SNE algorithm to convert multimodal feature representation into two-dimensional feature points for visualization. Figure 4 As shown, Figure 4 (b) (right sub-figure) shows the embedding space after applying UCL learning, while Figure 4(a) (left subgraph) shows the embedding space without the application of UCL. By comparison, we find that in the case without UCL, the data points in the embedding space are randomly distributed, and there is a lack of obvious distinction between different categories. After applying UCL, the distance between data points is significantly reduced, forming a clearer clustering effect, which indicates that our UCL design effectively realizes the division of sample categories. However, it can be noted that after optimization, there are still some data points misclassified, resulting in overlapping of some clusters. This phenomenon prompts us that in the future, we need to further optimize the learning strategy of the model to reduce the occurrence of misclassification and improve the accuracy of the model.

[0210] 5.4.2, Model performance

[0211] To comprehensively evaluate the performance of the TMFN model we proposed, we conducted a series of experiments on the CMU-MOSI dataset. The experimental results are shown in the table below, which shows the factual sentiment label of each sample and the prediction results of each model. From the table, we can see that compared with advanced models such as MMIN, Self-MM, and SUGRM, our model performs better in sentiment prediction accuracy, which is mainly due to its innovative multi-scale feature extraction technology and text-guided multi-modal feature fusion strategy. These technologies enable the model to more accurately capture and analyze sentiment information in multi-modal data, thereby effectively improving performance. Figure 5 Figure 5

[0212] 6, Review

[0213] The embodiment constructs a multi-modal sentiment analysis model, which captures multi-scale features of modal data using different sizes of convolution kernels and strengthens the recognition of key information through channel attention mechanism. In order to fully exploit the advantages of the text modality in multi-modal sentiment analysis, a text multi-modal feature fusion module is designed, which regulates the fusion of other modal information through a text gating unit, and applies the text multi-modal feature fusion module to deepen the understanding of the relationship between different modal features, effectively promoting feature fusion. Finally, an unsupervised contrastive learning method is introduced, which further optimizes the feature representation by mining the mutual information between the fused features and the multi-scale features. The experimental results on two standard datasets prove that the multi-modal sentiment analysis model outperforms existing baseline models in the task of multi-modal sentiment analysis.

[0214] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described by referring to the preferred embodiments of the present application, those skilled in the art should understand that various changes can be made in form and detail without departing from the spirit and scope of the present application as defined in the appended claims.​​

Claims

1. A multimodal sentiment analysis method based on a multimodal fusion network, characterized in that: The steps include: S1. Obtaining an emotional information modality to be processed and performing data extraction on the emotional information modality to be processed, wherein the emotional information modality includes a text modality, an audio modality, and a video modality; S2. Inputting the emotional information modality after data extraction into the trained multimodal emotion analysis model, and outputting the emotion prediction result of the emotional information modality to be processed; the multimodal emotion analysis model includes a multi-scale feature extraction module, a text multimodal feature fusion module and a multimodal emotion prediction module; The training steps of the multimodal sentiment analysis model are as follows: S201, the multimodal sentiment analysis model uses the emotional information modality extracted from the input data as the input of the multiscale feature extraction module, outputs multiscale features with text modality, audio modality and video modality, and inputs them into the text multimodal feature fusion module; S202: The text multimodal feature fusion module fuses the input multi-scale features of different modalities through a text gating unit with the text modality as the core to generate fused features, designs a text channel attention Transformer unit with a multimodal channel attention mechanism, uses the fused features as input to the text channel attention Transformer unit, and outputs modal fusion features as input to the multimodal sentiment prediction module; S203, the multimodal emotion prediction module connects the input modal fusion features to obtain a fusion feature, and obtains an emotion prediction result of the emotion information modality to be processed according to the fusion feature; S204, optimizing and updating the model parameters of the multimodal sentiment analysis model with the goal of minimizing the total loss function constructed by the model basic loss function and the unsupervised contrastive learning loss function, thereby completing the training of the multimodal sentiment analysis model; S205. Repeat steps S201 to S204 to perform iterative training until the multimodal sentiment analysis model converges or reaches a preset number of iterations.

2. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 1, characterized in that: The data extraction specifically includes: The step of extracting data from the text modality includes: extracting features from the input text sequence using a pre-trained BERT model to obtain a global representation and a local representation of the text modality, wherein the global representation is a corresponding vector marked by a special tag [CLS], and the [CLS] tag vector corresponding to the last layer of the BERT model is used as a comprehensive representation of the text sequence. The expression for data extraction from text modality is as follows: Where; Represents the text features extracted by the BERT model, l t is the maximum length of the text sequence, d t represents the dimension of comprehensive representation of text sequence, Represents the learnable parameters of text modality in BERT; The step of extracting data from the audio modality includes: extracting features from the audio modality using the pre-trained COVAREP model to obtain an initial vector feature I a Then, the initial vector feature I a Through the sLSTM model, the characteristics of its time series are deeply captured. Finally, the end-state hidden vector is selected as the comprehensive representation of the audio sequence. The expression for data extraction of the audio modality is as follows: Where, Represents the audio features extracted by the pre-trained COVAREP model, l a Indicates the sequence length of the audio, d a represents the dimension of the comprehensive representation of the audio sequence, Represents the learnable parameters of the audio modality in the sLSTM model; The step of extracting data from the video modality includes: extracting features from the video modality using the pre-trained FACET model to obtain an initial vector I v Then, the initial vector feature I v Through the sLSTM model, the features in its time series are deeply captured. Finally, the end-state hidden vector is selected as the comprehensive representation of the video sequence. The expression for data extraction of the video modality is as follows: Where, represents the video features extracted by the pre-trained FACET model, l v Indicates the sequence length of the video, d v represents the dimension of the comprehensive representation of the video sequence, Represents the learnable parameters of the video modality in sLSTM.

3. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 1, characterized in that: In step S201, the multi-scale feature extraction module includes four convolution blocks and a channel attention unit in parallel, wherein the four convolution blocks in parallel are a first convolution block, a second convolution block, a third convolution block, and a fourth convolution block, and the parallel connection result of the second convolution block and the third convolution block is cascaded with the channel attention unit; The first convolution block is a 1×1 convolution layer, the second convolution block is a 3×3 convolution layer, the third convolution block is a 5×5 convolution layer, and the fourth convolution block is a global pooling layer; the outputs of the second and third convolution blocks in parallel are used as the input of the channel attention unit, and the output of the channel attention unit is then added with the outputs of the first and fourth convolution blocks to obtain the overall output of different modalities in the multi-scale feature extraction module.

4. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 3 is characterized in that: The channel attention unit includes a cascaded global average pooling operation, an excitation operation, and a scaling operation. In the channel attention unit, the outputs of the second and third convolution blocks are compressed along the sequence length using a global average pooling operation to calculate global spatial information. Then, an excitation operation is performed to generate weights for each feature channel. Finally, the outputs of different channels are controlled by scaling operations and weight parameters, thereby outputting features that have passed through the channel attention unit. The processing process of the channel attention unit is expressed as: η=Excitation(Z,W)=σ(g(Z,W))=σ(W2ReLU(W1Z)); X′ sn =Scale(X sn ,η)=X sn ×η; Where Z represents the global spatial information, Squeeze(·) represents the global average pooling operation, and X sn express After the output features of the second and third convolution blocks, Represents the features obtained by data extraction of different modalities, s∈{a,t,v}, n∈{2,3}, η represents the weight parameter, Excitation(·,w) represents the excitation operation, g(·) is the nonlinear function, σ(·) represents the sigmoid function, W1 and W2 both represent the weight matrix of the fully connected layer, ReLU represents the activation function, X' vn represents the output feature after channel attention enhancement, and Scale(·) represents the scaling operation; The processing process of the multi-scale feature extraction module is expressed as: Where, X a represents the multi-scale features of audio modality, X t represents the multi-scale features of text modality, X v represents the multi-scale features of the video modality, Concat(·) represents the connection, X a1 、X t1 、X v1 Represent the output of audio, text and video modalities after the first convolutional block, X a4 、X t4 、X v4 Represent the outputs of the fourth convolutional block for audio, text, and video modalities, respectively.

5. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 1, characterized in that: In step S202, the text gating unit includes an audio text gating subunit and a video text gating subunit. The audio text gating subunit has the same structure as the video text gating subunit, including a cascaded convolution normalization unit, a convolution activation unit, a normalization activation unit, and a Sigmoid activation function layer, wherein the convolution normalization unit includes a cascaded first 1×1 convolution layer and a first batch normalization layer, the convolution activation unit includes a cascaded first ReLU activation function layer and a second 1×1 convolution layer, and the normalization activation unit includes a cascaded second batch normalization layer and a second ReLU activation function layer; In the video-text gating subunit, the input text modality multi-scale features and the video modality multi-scale features are processed by the convolution normalization unit respectively, and the processed features are added to obtain a fused feature map. Then, the fused feature map is passed through the convolution activation unit to obtain a single-channel feature map, and then processed by the normalization activation unit. Then, the processed single-channel feature map is converted into an attention coefficient through a Sigmoid activation function layer. Finally, the video modality multi-scale features as the input are element-wise multiplied with the output of the Sigmoid activation function layer to obtain the features output by the video-text gating subunit; In the audio-text gating subunit, the input text modality multi-scale features and the audio modality multi-scale features are used as inputs of the audio-text gating subunit, and are processed by the convolution normalization unit respectively, and the processed features are added to obtain a fused feature map, and then the fused feature map is passed through the convolution activation unit to obtain a single-channel feature map, and then processed by the normalization activation unit. Then, the processed single-channel feature map is converted into an attention coefficient through a Sigmoid activation function layer, and finally, the video modality multi-scale features as the input are element-wise multiplied with the output of the Sigmoid activation function layer to obtain the features output by the audio-text gating subunit; The features output by the video text gating subunit and the features output by the audio text gating subunit are used as the overall output features of the text gating unit; The video text gating sub-unit processing process is expressed as follows: X′ tv =BN(Conv 1×1 (X t ))+BN(Conv 1×1 (X v )); X tv =X v ⊙σ(δ(BN(Conv 1×1 (δ(X′ tv ))))); Where, X' tv Represents the fusion feature map of the multi-scale features of the text modality and the multi-scale features of the video modality, BN(·) represents batch normalization, Conv 1×1 (·) represents a one-dimensional convolutional layer, X t represents the multi-scale features of text modality, X v represents the multi-scale features of video modality, X tv represents the fusion of the multi-scale features of the video modality and the multi-scale features of the text modality output by the video-text gating subunit, ⊙ represents element-wise multiplication, and δ(·) represents the ReLU activation function; The audio text gating sub-unit processing process is expressed as follows: X′ ta =BN(Conv 1×1 (X t ))+BN(Conv 1×1 (X a )); X ta =X v ⊙σ(δ(BN(Conv 1×1 (δ(X′ ta ))))); Where, X' ta Represents the fusion feature map of the multi-scale features of the text modality and the multi-scale features of the audio modality, X t represents the multi-scale features of text modality, X a represents the multi-scale features of audio modality, X ta Represents the features of the audio modality and text modality fused at the output of the audio-text gating subunit.

6. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 1, characterized in that: In step S202, the text channel attention Transformer unit with a multimodal channel attention mechanism includes N layers of network layers consisting of a cascade of a multimodal channel attention module and a fully connected feedforward module; the multimodal channel attention module includes a first normalization layer and a multi-head attention unit connected in sequence, and the fully connected feedforward module includes a second normalization layer and a fully connected feedforward unit connected in sequence; In the text channel attention Transformer unit, the input feature map of the text channel attention Transformer unit is used as the input of the first normalization layer in the multimodal channel attention module in the first network layer, the output of the first normalization layer is used as the input of the multi-head attention unit, the output of the modal channel attention unit is added to the input of the channel attention Transformer unit, and the output is used as the input of the second normalization layer in the fully connected feedforward module. The output of the second normalization layer is used as the input of the fully connected feedforward unit, the input of the fully connected feedforward unit is added to the output of the multimodal channel attention module, and its output is used as the input of the next network layer. After N layers of feedforward calculation, its output is used as the overall output of the text channel attention Transformer unit; The processing process is expressed as: Where, represents the fusion feature representation of text modality and video modality after being processed by the multi-head attention unit at the i-1 layer, represents the fusion feature representation of the text modality and audio modality after being processed by the multi-head attention unit at the i-1 layer, Both represent the multi-head attention unit of layer i, LN(·) represents the layer normalization layer, represents the fusion feature representation of the i-1th layer text modality and video modality, represents the fusion feature representation of the i-1th layer text modality and audio modality, represents the fully connected feedforward unit of layer i, represents the fusion feature representation of the text modality and video modality after the i-th layer is processed by the fully connected feed-forward module, represents the fusion feature representation of the text modality and audio modality after the i-th layer is processed by the fully connected feedforward module, TCAT(·) represents the text channel attention Transformer unit, X ta represents the fusion feature of the audio modality and text modality output by the audio-text gating subunit, X a represents the multi-scale features of audio modality, X v Represents the multi-scale features of video modality, Represents the modal fusion features of the text modality and audio modality after being fused by the text channel attention Transformer unit, Represents the modality fusion features of the text modality and video modality after being fused by the text channel attention Transformer unit.

7. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 6, characterized in that: The multi-head attention unit projects the fused features and multi-scale features obtained after processing by the text gating unit into the query, key, and value vector space, and then applies the scaled dot product attention mechanism in the channel dimension to generate an output vector; The X tv The processing process of the multi-head attention unit is expressed as: Where Y tv Represents X tv The output after attention mechanism processing, Attention(·) represents the attention mechanism, Q tv Represents X tv The query matrix, K v Represents X v The bond matrix, V v Represents X v The value matrix of , Softmax(·) represents the Softmax function, Represents the weight of the query matrix, W K represents the weight of the bond matrix, represents the weight of the value matrix, Represents the output of each head in the multi-head attention unit of the video modality, and W0 represents the weight matrix of the output after splicing the multi-head attention unit; The X ta The processing process of the multi-head attention unit is expressed as: Where Y ta Represents X ta The output after attention mechanism processing, Q ta Represents X ta The query matrix, K a Represents X a The bond matrix, V a Represents X a The value matrix of represents the weight of the value matrix, The output of each head in the multi-head attention unit representing the audio modality.

8. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 1, characterized in that: In step S203, the processing of the multimodal emotion prediction module includes: The audio and text modality fusion features as input are connected with the video and text modality fusion features, and then the features are mapped to a lower-dimensional feature space through the first linear layer of the activation function to obtain the fusion feature X m , the fused features are passed through the second linear layer to generate the sentiment prediction result as output; The processing process is expressed as: Where, X m represents the fusion feature obtained after the first linear layer processing, ReLU(·) represents the activation function, represents the weight parameter of the first linear layer, represents the bias parameter of the first linear layer, represents the sentiment prediction result, represents the weight parameter of the second linear layer, Represents the bias parameters of the second linear layer.

9. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 1, characterized in that: In step S204, the unsupervised contrastive learning loss function is: Where Point(·) represents the correlation quantization function, exp(·) represents the exponential function, and X s Represents multi-scale features of different modalities, Represents X s The normalized eigenvector of Indicates that the parameter is neural network, Represents X m With X s The contrast loss function between s represents the expected value of different modes, represents the fusion feature of the jth sample, Represents the multi-scale features of different modalities of the j-th sample, represents the unsupervised contrastive learning loss function, represents the contrast loss between the fusion feature and the text modality, represents the contrast loss between the fused features and the audio modality, represents the contrast loss between the fused features and the video modality; The basic loss function of the model is: Where, Represents the basic loss function of the model, N represents the total number of training samples, Represents the predicted value of the multimodal sentiment analysis model for the i-th sample, Represents the true value of the i-th sample.

10. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 9, characterized in that: The total loss function is: Where, represents the total loss function, α represents The weight of The weight of .

Images