Multi-modal sentiment analysis method based on multi-modal fusion network

Abstract

The invention discloses a multi-modal sentiment analysis method based on a multi-modal fusion network, and the method comprises the steps: obtaining a to-be-processed sentiment information mode, carrying out the data extraction of the to-be-processed sentiment information mode, inputting the sentiment information mode after the data extraction into a trained multi-modal sentiment analysis model, and obtaining a multi-modal sentiment analysis model; outputting and obtaining an emotion prediction result of the emotion information mode to be processed; the trained multi-modal sentiment analysis model captures diversified features of modal data through a multi-scale feature extraction module, fuses different modals by using a text multi-modal feature fusion module, introduces unsupervised comparative learning to mine information between fused features and multi-scale features, further optimizes feature representation, and improves the sentiment analysis accuracy. According to the method, the extraction of different scale features by the model is enhanced, the text mode is fully utilized, and the sentiment analysis accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal sentiment analysis, and in particular to a multimodal sentiment analysis method based on a multimodal fusion network. Background Art

[0002] Multimodal Sentiment Analysis (MSA) is a comprehensive emotion recognition technology that integrates multiple information sources such as language, facial expressions, sounds, music, and body movements. The purpose is to deeply reveal and analyze user emotions. In the early research stage of MSA, researchers mainly relied on hand-crafted feature extraction methods, such as bag-of-words models, local binary patterns, and scale-invariant feature transforms. With the rapid development of deep learning technology, convolutional neural networks, recurrent neural networks, and long short-term memory networks have become the main research tools in the field of data processing. To further improve the accuracy of MSA, researchers have explored information fusion strategies between different modalities. Early fusion methods focused on integrating data from different sources into a unified representation, for example, TFN and LMF integrated information by constructing high-dimensional tensors through LSTM. Later, modal interaction modeling became the focus of research, such as MCTN optimizing modal conversion through cycle consistency loss, MulT using Transformer architecture for mutual encoding between modalities, and ICCN using deep learning to improve the consistency between modalities in high-dimensional nonlinear space. Recently, researchers have begun to pay extensive attention to methods that emphasize modal consistency and difference. For example, Yu et al. adopted a self-supervised learning strategy to design an automatic label generation module to reduce the difference between modalities, which is suitable for multimodal and unimodal training tasks. Han et al. introduced the concept of mutual information and proposed a learning framework to maximize mutual information to avoid losing task-related information. Hazarika et al. projected the modal vectors into different spaces and used regularization components to learn the representation of common and unique modal features.

[0003] Although existing methods have made some progress in improving performance, they usually treat all modalities equally, which limits further performance improvement to a certain extent. For example, on the CMU-MOSI dataset, the sentiment prediction strategy with equal contributions from the three modalities did not produce satisfactory results. The reason is that in multimodal sentiment analysis, the contributions of text, visual, and auditory modalities are different. In particular, the text modality has an important impact on the accuracy of sentiment analysis because it contains rich emotional clues.

[0004] Therefore, how to make full use of text modality to improve the accuracy and efficiency of sentiment analysis is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multimodal sentiment analysis method based on a multimodal fusion network. Through a pyramid-structured multi-scale feature extraction module, a text multimodal feature fusion module and unsupervised comparative learning, the model's extraction of features of different scales is enhanced, and the text modality is fully utilized to improve the accuracy of sentiment analysis.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A multimodal sentiment analysis method based on a multimodal fusion network comprises the following steps:

[0008] S1. Obtaining a to-be-processed emotional information modality, and performing data extraction on the to-be-processed emotional information modality, wherein the emotional information modality includes a text modality, an audio modality, and a video modality;

[0009] S2, inputting the emotional information modality after data extraction into the trained multimodal emotional analysis model, and outputting the emotional prediction result of the emotional information modality to be processed; the multimodal emotional analysis model includes a multi-scale feature extraction module, a text multimodal feature fusion module and a multimodal emotional prediction module;

[0010] The training steps of the multimodal sentiment analysis model are as follows:

[0011] S201, the multimodal sentiment analysis model uses the sentiment information modality extracted from the input data as the input of the multiscale feature extraction module, outputs the multiscale features with text modality, audio modality and video modality, and inputs them into the text multimodal feature fusion module;

[0012] S202, the text multimodal feature fusion module fuses the input multi-scale features of different modalities through a text gating unit with the text modality as the core to generate fused features, designs a text channel attention Transformer unit with a multimodal channel attention mechanism, uses the fused features as input of the text channel attention Transformer unit, outputs the modal fusion features, and uses them as input of the multimodal sentiment prediction module;

[0013] S203, the multimodal emotion prediction module connects the input modal fusion features to obtain fusion features, and obtains the emotion prediction result of the emotion information modality to be processed according to the fusion features;

[0014] S204, optimizing and updating the model parameters of the multimodal sentiment analysis model with the goal of minimizing the total loss function constructed by the model basic loss function and the unsupervised contrastive learning loss function, thereby completing the training of the multimodal sentiment analysis model;

[0015] S205. Repeat steps S201 to S204 to perform iterative training until the multimodal sentiment analysis model converges or reaches a preset number of iterations.

[0016] As a preferred solution, the data extraction specifically includes:

[0017] The step of extracting data from the text modality includes: extracting features from the input text sequence using a pre-trained BERT model to obtain a global representation and a local representation of the text modality, wherein the global representation is a corresponding vector marked by a special marker [CLS], and the [CLS] marker vector corresponding to the last layer of the BERT model is used as a comprehensive representation of the text sequence. The expression for data extraction from text mode is as follows:

[0018]

[0019] Where: Represents the text features extracted by the BERT model. l t is the maximum length of a text sequence, d t represents the dimension of comprehensive representation of text sequence, Represents the learnable parameters of the text modality in BERT;

[0020] The step of extracting data from the audio modality comprises: extracting features from the audio modality using the pre-trained COVAREP model to obtain an initial vector feature I a Then, the initial vector feature I a Through the sLSTM model, the features in its time series are deeply captured. Finally, the end-state hidden vector is selected as the comprehensive representation of the audio sequence. The expression for data extraction of the audio modality is as follows:

[0021]

[0022] In the formula, represents the audio features extracted by the pre-trained COVAREP model, l a Indicates the sequence length of the audio, d a represents the dimension of the comprehensive representation of the audio sequence, Represents the learnable parameters of the audio modality in the sLSTM model;

[0023] The step of extracting data from the video modality includes: extracting features from the video modality using a pre-trained FACET model to obtain an initial vector I vThen, the initial vector feature I v Through the sLSTM model, the features in its time series are deeply captured. Finally, the end-state hidden vector is selected as the comprehensive representation of the video sequence. The expression for data extraction of the video modality is as follows:

[0024]

[0025] In the formula, represents the video features extracted by the pre-trained FACET model, l v Indicates the sequence length of the video, d v represents the dimension of the comprehensive representation of the video sequence, Represents the learnable parameters of the video modality in sLSTM.

[0026] As a preferred solution, in step S201, the multi-scale feature extraction module includes four convolution blocks and a channel attention unit in parallel, wherein the four convolution blocks in parallel are respectively a first convolution block, a second convolution block, a third convolution block and a fourth convolution block, and the parallel connection result of the second convolution block and the third convolution block is cascaded with the channel attention unit;

[0027] The first convolution block is a 1×1 convolution layer, the second convolution block is a 3×3 convolution layer, the third convolution block is a 5×5 convolution layer, and the fourth convolution block is a global pooling layer; the outputs of the second and third convolution blocks in parallel are used as the input of the channel attention unit, and the output of the channel attention unit is added with the outputs of the first and fourth convolution blocks to obtain the overall output of different modalities in the multi-scale feature extraction module.

[0028] As a preferred solution, the channel attention unit includes a cascaded global average pooling operation, an excitation operation, and a scaling operation; in the channel attention unit, the outputs of the second convolution block and the third convolution block are compressed along the sequence length using a global average pooling operation to calculate global spatial information, and then, an excitation operation is performed to generate weights for each feature channel, and finally, the outputs of different channels are controlled by scaling operations and weight parameters, so as to output the features that have passed through the channel attention unit;

[0029] The processing process of the channel attention unit is expressed as:

[0030]

[0031] η=Excitation(Z,W)=σ(g(Z,W))=σ(W2ReLU(W1Z));

[0032] X′ sn =Scale(Xsn ,η)=X sn ×η;

[0033] In the formula, Z represents the global spatial information, Squeeze(·) represents the global average pooling operation, and X sn express After the output features of the second and third convolution blocks, Represents the features obtained by data extraction of different modalities, s∈{a,t,v}, n∈{2,3}, η represents the weight parameter, Excitation(·,w) represents the excitation operation, g(·) is the nonlinear function, σ(·) represents the sigmoid function, W1 and W2 both represent the weight matrix of the fully connected layer, ReLU represents the activation function, and X' vn represents the output feature after channel attention enhancement, Scale(·) represents the scaling operation;

[0034] Then the processing process of the multi-scale feature extraction module is expressed as:

[0035]

[0036] Where, X a represents the multi-scale features of the audio modality, X t represents the multi-scale features of text modality, X v represents the multi-scale features of the video modality, Concat(·) represents the connection, and X a1 , X t1 , X v1 Respectively represent the output of the audio, text and video modalities after the first convolutional block, X a4 , X t4 , X v4 Represent the outputs of the fourth convolutional block for audio, text, and video modalities, respectively.

[0037] As a preferred solution, in step S202, the text gating unit includes an audio text gating subunit and a video text gating subunit, and the audio text gating subunit has the same structure as the video text gating subunit, including a cascaded convolution normalization unit, a convolution activation unit, a normalization activation unit, and a Sigmoid activation function layer, wherein the convolution normalization unit includes a cascaded first 1×1 convolution layer and a first batch normalization layer, the convolution activation unit includes a cascaded first ReLU activation function layer and a second 1×1 convolution layer, and the normalization activation unit includes a cascaded second batch normalization layer and a second ReLU activation function layer;

[0038] In the video text gating subunit, the input text modality multi-scale features and the video modality multi-scale features are processed by the convolution normalization unit respectively, and the processed features are added to obtain a fused feature map, and then the fused feature map is passed through the convolution activation unit to obtain a single-channel feature map, and then processed by the normalization activation unit, and then the processed single-channel feature map is converted into an attention coefficient through a Sigmoid activation function layer, and finally, the video modality multi-scale features as input and the output of the Sigmoid activation function layer are element-wise multiplied to obtain features as outputs of the video text gating subunit;

[0039] In the audio-text gating subunit, the input text modality multi-scale features and the audio modality multi-scale features are used as inputs of the audio-text gating subunit, and are processed by convolution normalization units respectively, and the processed features are added to obtain a fused feature map, and then the fused feature map is passed through a convolution activation unit to obtain a single-channel feature map, and then processed by a normalization activation unit, and then the processed single-channel feature map is converted into an attention coefficient through a Sigmoid activation function layer, and finally, the video modality multi-scale features as input and the output of the Sigmoid activation function layer are element-wise multiplied to obtain features as outputs of the audio-text gating subunit;

[0040] The features output by the video text gating subunit and the features output by the audio text gating subunit are used as the overall output features of the text gating unit;

[0041] The video text gating subunit processing process is expressed as:

[0042] X′ tv =BN(Conv 1×1 (X t ))+BN(Conv 1×1 (X v ));

[0043] X tv =X v ⊙σ(δ(BN(Conv 1×1 (δ(X′ tv )))));

[0044] In the formula, X′ tv represents the fusion feature map of the multi-scale features of the text modality and the multi-scale features of the video modality, BN(·) represents batch normalization, Conv 1×1 (·) represents a one-dimensional convolutional layer, X t represents the multi-scale features of text modality, X v represents the multi-scale features of video modality, X tvrepresents the fusion of the multi-scale features of the video modality and the multi-scale features of the text modality output by the video-text gating subunit, ⊙ represents element-wise multiplication, and δ(·) represents the ReLU activation function;

[0045] The audio text gating subunit processing process is expressed as:

[0046] X′ ta =BN(Conv 1×1 (X t ))+BN(Conv 1×1 (X a ));

[0047] X ta =X v ⊙σ(δ(BN(Conv 1×1 (δ(X′ ta )))));

[0048] In the formula, X′ ta Represents the fusion feature map of the multi-scale features of the text modality and the multi-scale features of the audio modality, X t represents the multi-scale features of text modality, X a represents the multi-scale features of the audio modality, X ta Represents the features after the fusion of audio modality and text modality output by the audio-text gating subunit.

[0049] As a preferred solution, in step S202, the text channel attention Transformer unit with a multimodal channel attention mechanism includes N layers of network layers composed of a cascade of a multimodal channel attention module and a fully connected feedforward module; the multimodal channel attention module includes a first normalization layer and a multi-head attention unit connected in sequence, and the fully connected feedforward module includes a second normalization layer and a fully connected feedforward unit connected in sequence;

[0050] In the text channel attention Transformer unit, the input feature map of the text channel attention Transformer unit is used as the input of the first normalization layer in the multimodal channel attention module in the first network layer, the output of the first normalization layer is used as the input of the multi-head attention unit, the output of the modal channel attention unit is added to the input of the channel attention Transformer unit, and the output is used as the input of the second normalization layer in the fully connected feedforward module, the output of the second normalization layer is used as the input of the fully connected feedforward unit, the input of the fully connected feedforward unit is added to the output of the multimodal channel attention module, and its output is used as the input of the next network layer, and after N layers of feedforward calculation, its output is used as the overall output of the text channel attention Transformer unit;

[0051] The processing process is expressed as:

[0052]

[0053]

[0054] In the formula, represents the fused feature representation of the text modality and the video modality after being processed by the multi-head attention unit at the i-1 layer, represents the fused feature representation of the text modality and the audio modality after being processed by the multi-head attention unit at the i-1 layer, Both represent the multi-head attention unit of the i-th layer, LN(·) represents the layer normalization layer, represents the fusion feature representation of the i-1th layer text modality and video modality, represents the fusion feature representation of the i-1th layer text modality and audio modality, represents the fully connected feed-forward unit of the i-th layer, represents the fusion feature representation of the text modality and the video modality after the i-th layer is processed by the fully connected feed-forward module, represents the fusion feature representation of the text modality and audio modality after the i-th layer is processed by the fully connected feed-forward module, TCAT(·) represents the text channel attention Transformer unit, X ta represents the fusion feature of the audio modality and text modality output by the audio-text gating subunit, X a represents the multi-scale features of the audio modality, X v Represents the multi-scale features of video modality, Represents the modality fusion features of the text modality and the audio modality after being fused by the text channel attention Transformer unit, Represents the modality fusion features of the text modality and the video modality after being fused by the text channel attention Transformer unit.

[0055] As a preferred solution, the multi-head attention unit projects the fused features and multi-scale features obtained after processing by the text gating unit into the query, key, and value vector space, and then applies the scaled dot product attention mechanism on the channel dimension to generate an output vector;

[0056] The X tv The processing process of the multi-head attention unit is expressed as:

[0057]

[0058] Where Y tv Represents X tvThe output after attention mechanism processing, Attention(·) represents the attention mechanism, Q tv Represents X tv The query matrix, K v Represents X v The bond matrix, V v Represents X v The value matrix of , Softmax(·) represents the Softmax function, represents the weight of the query matrix, W K represents the weight of the bond matrix, represents the weight of the value matrix, Represents the output of each head in the multi-head attention unit of the video modality, and W0 represents the weight matrix of the output after splicing the multi-head attention unit.

[0059] The X ta The processing process of the multi-head attention unit is expressed as:

[0060]

[0061] Where Y ta Represents X ta The output after attention mechanism processing, Q ta Represents X ta The query matrix, K a Represents X a The bond matrix, V a Represents X a The value matrix of represents the weight of the value matrix, The output of each head in the multi-head attention unit representing the audio modality.

[0062] As a preferred solution, in step S203, the processing process of the multimodal emotion prediction module includes:

[0063] The audio and text modality fusion features as input are connected with the video and text modality fusion features, and then the features are mapped to a lower-dimensional feature space through the first linear layer of the activation function to obtain the fusion feature X m , passing the fused features through the second linear layer to generate the sentiment prediction result as output;

[0064] The processing process is expressed as:

[0065]

[0066] Where, X m represents the fusion feature obtained after the first linear layer processing, ReLU(·) represents the activation function, represents the weight parameter of the first linear layer, represents the bias parameter of the first linear layer, represents the sentiment prediction result, represents the weight parameter of the second linear layer, Represents the bias parameters of the second linear layer.

[0067] As a preferred solution, in step S204, the unsupervised contrastive learning loss function is:

[0068]

[0069] In the formula, Point(·) represents the correlation quantization function, exp(·) represents the exponential function, and X s Represents multi-scale features of different modalities, Represents X s The normalized eigenvector of Indicates that the parameter is The neural network, Represents X m With X s The contrast loss function between s represents the expected value of different modes, represents the fusion feature of the jth sample, represents the multi-scale features of different modalities of the j-th sample, represents the unsupervised contrastive learning loss function, represents the contrast loss between the fusion feature and the text modality, represents the contrast loss between the fused features and the audio modality, represents the contrast loss between the fused features and the video modality;

[0070] The basic loss function of the model is:

[0071]

[0072] In the formula, represents the basic loss function of the model, N represents the total number of training samples, represents the predicted value of the multimodal sentiment analysis model for the i-th sample, Represents the true value of the i-th sample;

[0073] As a preferred solution, the total loss function is:

[0074]

[0075] In the formula, represents the total loss function, α represents The weight of The weight of .

[0076] Compared with the prior art, the present invention has the following technical effects:

[0077] (1) The present invention constructs a multimodal sentiment analysis model, which takes into account the multi-scale features of modal data that are often overlooked in the prior art through a multi-scale feature extraction module, and designs a text multimodal feature fusion module to fully utilize the potential of text modality in multimodal sentiment analysis. To further enhance the performance of the model, the model of the present invention also adopts an unsupervised contrastive learning method to deeply explore the intrinsic connection between multi-scale features and fused features, and optimize the representation of fused features.

[0078] (2) The pyramid-structured multi-scale feature extraction module proposed in the present invention utilizes four different-sized convolution kernels to gradually capture the features of the modal data at different scales, and introduces a channel attention mechanism through the second and third convolution blocks to further enhance the expressiveness of the features. The channel attention mechanism is used to strengthen the features of key channels while suppressing non-key information, thereby enhancing the model's ability to capture important information.

[0079] (3) The present invention designs a text multimodal feature fusion module, which strengthens the guiding role of text information in the feature fusion process through a text gating unit, so that the model can pay more attention to the key emotional clues in the text, realize the effective fusion of modal feature map information, and capture the relationship between different modalities through the text channel attention Transformer unit, so as to realize effective cross-modal feature interaction, which helps the model capture and utilize the complementary information between different modalities and improve the accuracy of sentiment analysis; secondly, through the multimodal channel attention mechanism, the model can adaptively adjust the weights of different modal features, highlight the features that are more relevant to the sentiment analysis task, and suppress unimportant or noise features. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0081] Figure 1 This is a framework diagram of the multimodal sentiment analysis model of the present invention;

[0082] Figure 2 Schematic diagram of the channel attention module of the present invention;

[0083] Figure 3 This is a structural diagram of the text multimodal feature fusion module of the present invention;

[0084] Figure 4 This is the t-SNE visualization result of UCL in the embodiment of the present invention.

[0085] Figure 5The figure is a performance comparison result chart of various models on CMU-OSI in an embodiment of the present invention. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0087] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0088] As a core component of human-computer interaction, multimodal sentiment analysis (MSA) has rapidly developed into a hot research topic in recent years. The rapid development of this field is due to its significant role in understanding human emotional states, improving interactive experience, and optimizing decision-making processes. With the continuous advancement of artificial intelligence technology, multimodal sentiment analysis is gradually becoming an important bridge connecting human emotions and machine intelligence. However, this field also faces many challenges, including feature extraction, feature fusion, and processing of contributions of different modalities.

[0089] In view of the above problems, the present invention constructs a new multimodal sentiment analysis model (TMFN), and proposes a multimodal sentiment analysis method based on a multimodal fusion network based on the model, which comprises the following steps:

[0090] S1. Obtaining a to-be-processed emotional information modality, and performing data extraction on the to-be-processed emotional information modality, wherein the emotional information modality includes a text modality, an audio modality, and a video modality;

[0091] S2, inputting the emotional information modality after data extraction into the trained multimodal emotional analysis model, and outputting the emotional prediction result of the emotional information modality to be processed; the multimodal emotional analysis model includes a multi-scale feature extraction module, a text multimodal feature fusion module and a multimodal emotional prediction module;

[0092] The training steps of the multimodal sentiment analysis model are as follows:

[0093] S201, the multimodal sentiment analysis model uses the sentiment information modality extracted from the input data as the input of the multiscale feature extraction module, outputs the multiscale features with text modality, audio modality and video modality, and inputs them into the text multimodal feature fusion module;

[0094] S202, the text multimodal feature fusion module fuses the input multi-scale features of different modalities through a text gating unit with the text modality as the core to generate fused features, designs a text channel attention Transformer unit with a multimodal channel attention mechanism, uses the fused features as input of the text channel attention Transformer unit, outputs the modal fusion features, and uses them as input of the multimodal sentiment prediction module;

[0095] S203: the multimodal emotion prediction module connects the input modal fusion features to obtain fusion features, and obtains the emotion prediction result of the emotion information modality to be processed according to the fusion features;

[0096] S204, optimizing and updating the model parameters of the multimodal sentiment analysis model with the goal of minimizing the total loss function constructed by the model basic loss function and the unsupervised contrastive learning loss function, thereby completing the training of the multimodal sentiment analysis model;

[0097] S205. Repeat steps S201 to S204 to perform iterative training until the multimodal sentiment analysis model converges or reaches a preset number of iterations.

[0098] Firstly, this paper proposes a pyramid-structured multi-scale feature extraction method, which uses convolution kernels of different sizes to capture the diversified features of modal data, and strengthens the features of key channels through the channel attention mechanism, while suppressing non-key information, thereby enhancing the model's ability to capture important information; secondly, in view of the advantages of text modality in multimodal sentiment analysis, this paper designs a text multimodal feature fusion module, which includes a text gating unit and a text channel attention transformer unit (text-based channel-wise attention transformer, TCAT). The text gating unit effectively guides the fusion with other modal information by dynamically adjusting the feature weights of the text content. This strategy ensures that the features closely related to sentiment analysis are properly emphasized, thereby significantly improving the accuracy and efficiency of sentiment analysis. TCAT is used to perform cross-modal information fusion. It uses a multimodal channel-wise attention mechanism (multimodal channel-wise attention, MCA) to capture the relationship between different modal features. By adaptively adjusting the weights between different channels, TCAT can more accurately identify and utilize cross-modal correlation information, thereby promoting effective interaction between modal features. Finally, in order to optimize the model performance and improve the quality of fused features, unsupervised contrastive learning was used to deeply explore the mutual information between fused features and multi-scale features, optimize the fused feature representation, and further improve the performance of the model.

[0099] The multimodal sentiment analysis method based on the multimodal fusion network of the present invention is described in more detail below.

[0100] 1. Data extraction

[0101] For multimodal sentiment analysis, text, audio, and video are the three main information modalities.

[0102] The step of extracting data from the text modality comprises: using a pre-trained BERT model to extract the input text sequence I t ={w C ,w0,…,w i ,…,w S} for feature extraction, where w C and w S Represent the special tags [CLS] and [SEP] respectively, and obtain the global representation and local representation of the text modality. The global representation is the corresponding vector marked by the special tag [CLS]. The [CLS] tag vector corresponding to the last layer of the BERT model is used as the comprehensive representation of the text sequence. The expression for data extraction from text mode is as follows:

[0103]

[0104] Where: Represents the text features extracted by the BERT model. l t is the maximum length of a text sequence, d t represents the dimension of comprehensive representation of text sequence, Represents the learnable parameters of the text modality in BERT;

[0105] The step of extracting data from the audio modality includes: extracting features from the audio modality using the pre-trained COVAREP model to obtain an initial vector feature Then, the initial vector feature I a Through the sLSTM model, the features in its time series are deeply captured. Finally, the end-state hidden vector is selected as the comprehensive representation of the audio sequence. The expression for data extraction of the audio modality is as follows:

[0106]

[0107] In the formula, represents the audio features extracted by the pre-trained COVAREP model, l a Indicates the sequence length of the audio, d a represents the dimension of the comprehensive representation of the audio sequence, Represents the learnable parameters of the audio modality in sLSTM;

[0108] The step of extracting data from the video modality includes: extracting features from the video modality using a pre-trained FACET model to obtain an initial vector Then, the initial vector feature I v The sLSTM model (single directional Long Short-Term Memory) is used to deeply capture the features in its time series. Finally, the end-state hidden vector is selected as the comprehensive representation of the video sequence. The expression for data extraction of the video modality is as follows:

[0109]

[0110] In the formula, represents the video features extracted by the pre-trained FACET model, l v Indicates the sequence length of the video, d v represents the dimension of the comprehensive representation of the video sequence, Represents the learnable parameters of the video modality in sLSTM.

[0111] 2. Multimodal sentiment analysis model

[0112] like Figure 1 As shown in Figure 1, the multimodal sentiment analysis model includes a multi-scale feature extraction module, a text multimodal feature fusion module, and a multimodal sentiment prediction module. The following introduces each of the above modules.

[0113] 2.1 Multi-scale feature extraction module

[0114] Multi-scale feature extraction technology plays a key role in many fields such as image processing, computer vision and pattern recognition. By extracting image features at different scales, it can comprehensively capture image information and significantly improve the model's ability to understand and represent image content.

[0115] The present invention proposes a new multi-scale feature extraction method of pyramid structure. Different from the prior art, the present invention takes into account the initial feature The contextual semantic information has been fully captured through the pre-trained model and sLSTM (singledirectional Long Short-Term Memory) technology. Based on this, we used four convolution blocks of different sizes to fully extract the multi-scale features of the target object, and further enhanced the expressiveness of the features by using the channel attention mechanism in the second and third convolution blocks. Specifically, the multi-scale feature extraction module of the present invention includes four convolution blocks and a channel attention unit in parallel, wherein the four convolution blocks in parallel are respectively a first convolution block, a second convolution block, a third convolution block and a fourth convolution block, and the parallel result of the second convolution block and the third convolution block is cascaded with the channel attention unit;

[0116] The first convolution block is a 1×1 convolution layer, the second convolution block is a 3×3 convolution layer, the third convolution block is a 5×5 convolution layer, and the fourth convolution block is a global pooling layer; the outputs of the second and third convolution blocks in parallel are used as the input of the channel attention unit, and the output of the channel attention unit is added with the outputs of the first and fourth convolution blocks to obtain the overall output of different modalities in the multi-scale feature extraction module.

[0117] like Figure 2As shown, the channel attention unit includes a cascaded global average pooling operation, an excitation operation, and a scaling operation; in the channel attention unit, the outputs of the second and third convolution blocks of the input are compressed along the sequence length using a global average pooling operation to calculate the global spatial information, and then, an excitation operation is performed to generate weights for each feature channel, and finally, the outputs of different channels are controlled by scaling operations and weight parameters, thereby outputting the features that have passed through the channel attention unit.

[0118] Below, taking the video modality as an example, the actual processing process of the multi-scale feature extraction module is as follows:

[0119] First, the initial features are transformed by using convolution blocks of different sizes. Processed to capture its multi-scale features:

[0120]

[0121] In the formula, k represents the convolution kernel size, d represents the expansion rate, and X v1 , X v2 , X v3 , X v4 Represent the features extracted by the first, second, third, and fourth convolutional blocks respectively;

[0122] It is worth noting that the 1×1 convolution and global average pooling branches are designed to preserve the inherent properties of the input features through the minimum and maximum receptive fields, so we do not apply the channel attention mechanism to these two branches. Next, the features extracted from the second and third layers are (where n∈{2,3}) further feature enhancement is performed through the channel attention mechanism;

[0123] Next, global average pooling is used to compress X along the sequence length vn , to calculate global spatial information:

[0124]

[0125] Then, the excitation operation F is performed. ex (·,w) generates weights for each feature channel, similar to the gating mechanism in LSTM, F ex (·,w) uses a fully connected layer to learn the weight parameter η:

[0126] η=Excitation(Z,W)=σ(g(Z,W))=σ(W2ReLU(W1Z));

[0127] Then, through F scale(·) And weight parameter η to control the output of different channels, and obtain the feature X′ enhanced by channel attention vn :

[0128] X′ vn =Scale(X vn ,η)=X vn ×η;

[0129] Finally, we connect X v1 , X' vn and X v4 Get the final multi-scale feature X v :

[0130]

[0131] In the formula, Z represents the global spatial information, Squeeze(·) represents the global average pooling operation, and X vn The output features of the second and third convolutional blocks representing the video modality, η represents the weight parameter, Excitation(·,w) represents the excitation operation, g(·) represents the nonlinear function, σ(·) represents the sigmoid function, W1 and W2 both represent the weight matrices of the fully connected layer, r is the channel transformation ratio, ReLU represents the activation function, X' vn represents the output feature after channel attention enhancement, Scale(·) represents the scaling operation, X v represents the multi-scale features of the video modality, and Concat(·) represents the connection.

[0132] 2.2 Text Multimodal Feature Fusion Module

[0133] Through the multi-scale feature extraction module, we obtain multi-scale features X of different modalities s , s∈{a,t,v}, where X a represents the multi-scale features of the audio modality, X t represents the multi-scale features of text modality, X v Represents multi-scale features of video modality. In multimodal sentiment analysis, text information is considered to be the most informative modality reflecting the speaker's emotions due to its intuitiveness. Therefore, the present invention designs a text multimodal feature fusion module. This module mainly consists of two parts: a text gating unit (TGU) and a text channel attention unit (Transformer text-based channel-wise attention transformer, TCAT). These two parts are centered on the text modality and work together to deepen the understanding and utilization of other modal information, effectively improving the accuracy of multimodal sentiment analysis.

[0134] Below, when implementing this specific application, the specific process of multimodal feature fusion is explained in detail by taking the fusion of text and video modalities as an example.

[0135] 2.2.1 Text Gating Unit

[0136] The text gating unit includes an audio text gating subunit and a video text gating subunit, and the audio text gating subunit has the same structure as the video text gating subunit, including a cascaded convolution normalization unit, a convolution activation unit, a normalization activation unit and a Sigmoid activation function layer, wherein the convolution normalization unit includes a cascaded first 1×1 convolution layer and a first batch normalization layer, the convolution activation unit includes a cascaded first ReLU activation function layer and a second 1×1 convolution layer, and the normalization activation unit includes a cascaded second batch normalization layer and a second ReLU activation function layer;

[0137] In the video text gating subunit, the input text modality multi-scale features and the video modality multi-scale features are processed by the convolution normalization unit respectively, and the processed features are added to obtain a fused feature map, and then the fused feature map is passed through the convolution activation unit to obtain a single-channel feature map, and then processed by the normalization activation unit, and then the processed single-channel feature map is converted into an attention coefficient through a Sigmoid activation function layer, and finally, the video modality multi-scale features as input and the output of the Sigmoid activation function layer are element-wise multiplied to obtain features as outputs of the video text gating subunit;

[0138] In the audio-text gating subunit, the input text modality multi-scale features and the audio modality multi-scale features are used as the input of the audio-text gating subunit, and the output is obtained as the output feature of the audio-text gating subunit; the features output by the video-text gating subunit and the features output by the audio-text gating subunit are used as the overall output feature of the text gating unit;

[0139] The video text gating subunit processing process is expressed as:

[0140] X′ tv =BN(Conv 1×1 (X t ))+BN(Conv 1×1 (X v ));

[0141] X tv =X v ⊙σ(δ(BN(Conv 1×1 (δ(X′ tv )))));

[0142] In the formula, X′tv represents the fusion feature map of the multi-scale features of the text modality and the multi-scale features of the video modality, BN(·) represents batch normalization, Conv 1×1 (·) represents a one-dimensional convolutional layer, X t represents the multi-scale features of text modality, X v represents the multi-scale features of video modality, X tv represents the features after the fusion of the multi-scale features of the video modality and the multi-scale features of the text modality output by the video-text gating subunit, ⊙ represents element-wise multiplication, and δ(·) represents the ReLU activation function.

[0143] 2.2.2 Text Channel Attention Transformer Unit

[0144] like Figure 3 As shown, the text channel attention Transformer unit includes N layers of network layers composed of a cascade of a multimodal channel attention module and a fully connected feedforward module; the multimodal channel attention module includes a first normalization layer and a multi-head attention unit connected in sequence, and the fully connected feedforward module includes a second normalization layer and a fully connected feedforward unit connected in sequence;

[0145] In the text channel attention Transformer unit, the input feature map of the text channel attention Transformer unit is used as the input of the first normalization layer in the multimodal channel attention module in the first network layer, the output of the first normalization layer is used as the input of the multi-head attention unit, the output of the modal channel attention unit is added to the input of the channel attention Transformer unit, and the output is used as the input of the second normalization layer in the fully connected feedforward module, the output of the second normalization layer is used as the input of the fully connected feedforward unit, the input of the fully connected feedforward unit is added to the output of the multimodal channel attention module, and its output is used as the input of the next network layer, and after N layers of feedforward calculation, its output is used as the overall output of the text channel attention Transformer unit;

[0146] The processing process is expressed as:

[0147]

[0148] In the formula, They represent the text-video modality fusion features and the text-audio modality fusion features after processing at the i-1th layer, Both represent the multi-head attention unit of the i-th layer, LN(·) represents the layer normalization layer, represents the i-1th layer of text-video fusion features and text-audio modality fusion features, represents the fully connected feed-forward unit of the i-th layer, represents the text-video fusion feature and text-audio modality fusion feature after the i-th layer is processed by the fully connected feed-forward module, TCAT(·) represents the text channel attention Transformer unit, X ta represents the fusion feature of the audio modality and text modality output by the audio-text gating subunit, X a represents the multi-scale features of the audio modality, X v Represents the multi-scale features of video modality, Represents the modality fusion features of the text modality and the audio modality after being fused by the text channel attention Transformer unit, Represents the modality fusion features of the text modality and the video modality after being fused by the text channel attention Transformer unit.

[0149] In the above TCAT unit, the multi-head attention unit projects the fused features and multi-scale features obtained after processing by the text gating unit into the query, key, and value vector space: Q tv =X tv W Q ,K v =X v W K ,V v =X v W V , where W(·) is a trainable parameter matrix. Then, a scaled dot product attention mechanism is applied on the channel dimension. The scaled dot product attention mechanism generates an output vector by associating the query vector with a series of key-value pairs. In this process, the query, key, value, and output are all represented in the form of vectors. Specifically, the calculation of the output vector is based on the weighted sum of the value vectors, and the weights are determined by the compatibility function between the query vector and the corresponding key vector.

[0150] The processing process is expressed as:

[0151]

[0152] Where Y tv represents the output of the fusion features of text and video modalities after being processed by the attention mechanism. Attention(·) represents the attention mechanism, Q tv represents the query matrix, K v represents the bond matrix, V v represents the value matrix, Softmax(·) represents the Softmax function, represents the weight of the query matrix, W K represents the weight of the bond matrix, represents the weight of the value matrix, Represents the output of each head in the multi-head attention unit, and W0 represents the weight matrix of the output after splicing the multi-head attention unit.

[0153] 2.2.3 Multimodal Sentiment Prediction Module

[0154] The processing of the multimodal emotion prediction module includes: connecting the audio and text modality fusion features as input with the video and text modality fusion features, and then mapping the features to a lower-dimensional feature space through the first linear layer of the activation function to obtain the fusion feature X m , passing the fused features through the second linear layer to generate the sentiment prediction result as output;

[0155] The processing process is expressed as:

[0156]

[0157] Where, X m represents the fusion feature obtained after the first linear layer processing, ReLU(·) represents the activation function, represents the weight parameter of the first linear layer, represents the bias parameter of the first linear layer, represents the sentiment prediction result, represents the weight parameter of the second linear layer, Represents the bias parameters of the second linear layer.

[0158] 3. Unsupervised Contrastive Learning

[0159] In the present invention, a better multimodal fusion feature representation is obtained through the text multimodal feature fusion module. s (where s∈{a,t,v}) and the fusion feature X m The relationship between the two has not been fully explored. To this end, the present invention uses unsupervised contrastive learning to explore the relationship between the two and optimize the fused feature representation. We iterate between the fused features and the input modality by maximizing the mutual information to promote the model to efficiently integrate multi-scale features into the fused features.

[0160] The present invention optimizes by unsupervised contrastive learning method and comprises the following steps:

[0161] A correlation quantization function is used to measure the correlation between the fused features and the multi-scale features. The representations of all other modalities are used as negative samples to calculate the contrast loss between a single modality and the fused features. Then, an unsupervised contrastive learning loss function is constructed, which is composed of the losses of the fused features and the text, video, and audio modalities. The fused features are optimized with the goal of minimizing the unsupervised contrastive learning loss function, and the optimized fused features are obtained.

[0162] The processing process is expressed as:

[0163]

[0164] In the formula, Point(·) represents the correlation quantization function, exp(·) represents the exponential function, Represents X s The normalized eigenvector of Indicates that the parameter is The neural network, Represents X m With X s The contrast loss function between s represents the expected value of different modes, represents the fusion feature of the jth sample, represents the multi-scale features of different modalities of the j-th sample, represents the unsupervised contrastive learning loss function, represents the contrast loss between the fusion feature and the text modality, represents the contrast loss between the fused features and the audio modality, represents the contrast loss between the fused features and the video modality.

[0165] 4. Model training

[0166] The multimodal sentiment analysis model is trained in the following way:

[0167] The emotional information modality of the pre-extracted data is input into the multimodal sentiment analysis model, and a total loss function constructed by the basic loss function of the model and the unsupervised contrastive learning loss function is constructed. The model parameters of the multimodal sentiment analysis model are optimized and updated with the goal of minimizing the loss function, thereby completing the training of the multimodal sentiment analysis model.

[0168] Among them, the basic loss function of the model is:

[0169]

[0170] In the formula, represents the basic loss function of the model, N represents the total number of training samples, represents the predicted value of the multimodal sentiment analysis model for the i-th sample, Represents the true value of the i-th sample;

[0171] The total loss function is:

[0172]

[0173] In the formula, represents the total loss function, α represents The weight of The weight of .

[0174] 5. Embodiment

[0175] 5.1 Implementation Details

[0176] This example uses the PyTorch framework to implement the proposed model on two benchmark datasets, CMU-MOSI and CMU-MOSEI. Table 1 shows the data distribution of the two datasets.

[0177] Table 1. Dataset distribution of CMU-MOSI and CMU-MOSEI

[0178] Dataset Train Valid Test Total CMU-MOSI 1284 229 686 2299 CMU-MOSEI 16326 1871 4659 22856

[0179] During the training process, this example uses the Adam optimizer and implements an 8-cycle early-stop strategy to monitor model performance. In addition, a random search is performed to find the best hyperparameter combination. Table 2 shows the configuration of the CMU-MOSI and CMU-MOSEI datasets in detail. All training and testing processes are completed on a single NVIDIA GeForce RTX3060Ti GPU, and the performance evaluation of the model is based on the average score of five random runs on the test set.

[0180] Among them, according to previous research work, the results are presented in the form of classification and regression. In the classification task, the focus is on the weighted F1 score (F1-score) and the binary classification accuracy (Acc-2). For the MOSI and MOEI datasets, two different evaluation methods are used: one treats the score as negative / non-negative (non-exclude zero), and the other treats the score as negative / positive (exclude zero). In the regression task, the mean absolute error (MAE) and the Pearson correlation coefficient (Corr) are reported. It should be pointed out that, except for MAE, high values ​​of these metrics usually mean better model performance.

[0181] Table 2 Main hyperparameters used in the two datasets

[0182] Hyper-parameter CMU-MOSI CMU-MOSEI EarlyStop 8 8 BatchSize 32 32 NumHeads 8 4 LRforBERT 5e-5 5e-5 LRforOthers 0.01 0.001 EncoderLayer 4 4 HiddenLayer 5 5 OutputDropout 0.3 0.1

[0183] 5.2 Results and Analysis

[0184] The most advanced multimodal sentiment analysis models are compared on the CMU-MOSI and CMU-MOSEI datasets. The specific comparison results are shown in Tables 3 and 4. In multimodal sentiment analysis, datasets are usually divided into two types: aligned and non-aligned. Although using aligned datasets usually leads to better performance, considering that non-aligned datasets can provide more diverse modal information and help improve the generalization ability of the model in the real world, non-aligned datasets are selected here to train and evaluate the model.

[0185] On the CMU-MOSI dataset, as shown in Table 3, the The model representation of comes from MMIM, and the label is The model is represented as derived from the SUGRM model. All models with * are reproduced under the same conditions for fair comparison. The proposed model shows significant performance improvement compared with the models using aligned datasets (such as RAVEN, Mult, MISA, MAG-BERT). In particular, compared with the best performing MAG-BERT model, the proposed model improves the Acc-2, F1-Score and Corr indicators by 0.60 / 1.28, 0.73 / 1.37 and 0.025 respectively, and the MAE is reduced by 9.73%. In addition, compared with other models using non-aligned datasets (such as Self-MM, MMIM, SUGRM, M3SA), the method of this embodiment also shows a competitive advantage. In order to further verify the effectiveness of the model, the three optimal benchmark models of Self-MM, MMIM and SUGRM were reproduced under the same experimental environment, and it was found that among the four evaluation indicators, our model was slightly inferior to MMIM only in the Corr indicator.

[0186] On the CMU-MOSEI dataset, our model performs equally well, as shown in Table 4, which is marked as The model representation of comes from MMIM, and the label is The model representation is derived from the SUGRM model, and all models with * are reproduced under the same conditions for fair comparison. Compared with the best performing model MAG-BERT using aligned datasets, our model improves by 1.76 / 1.30, 1.42 / 1.47, and 0.033 in Acc-2, F1-Score, and Corr indicators, respectively, and MAE decreases by 4.64%. Similarly, we follow the parameter settings in the original paper and reproduce the three best baseline models Self-MM, MMIM, and SUGRM under the same experimental conditions. The results show that our model surpasses these baselines in all evaluation indicators and reaches the best level.

[0187] Table 3 Experimental results on the CMU-MOSI dataset

[0188]

[0189] Table 4 Experimental results on the CMU-MOSEI dataset

[0190]

[0191]

[0192] 5.3 Ablation study

[0193] 5.3.1. Influence of various modes

[0194] Table 5 shows in detail the performance of the proposed model under different modal combinations. After in-depth analysis, several key conclusions were drawn. (1) In terms of single-modal performance, the text modality performs best, while the video modality is relatively weak. The reason why text can perform well is that it usually contains rich emotion-related information, such as emotional vocabulary and semantic structure, which is crucial for sentiment analysis. (2) Multimodal combinations are generally better than single modalities. Regardless of the combination method, the comprehensive effect of multimodality is always better than the use of a single modality. In particular, when the text modality is included in the multimodal combination, the performance improvement is particularly significant, which is more effective than using audio or video modalities alone. This finding further confirms the core role of the text modality in multimodal sentiment analysis and is consistent with the text-dominated multimodal fusion design concept proposed in this embodiment. (3) The combination method (i.e., T∝A∝V) that regards the representation capabilities of the three modalities as equivalent is compared with the feature fusion method used in this embodiment. The results show that the method of this embodiment has more advantages. This is mainly because the method of this embodiment strategically positions the text modality as the core hub in the multimodal information fusion process. Through this strategy, information from different modalities can be more effectively integrated, thereby significantly improving the overall representation ability and performance.

[0195] Table 5 Performance of TFMN under different multimodal settings

[0196]

[0197] 5.3.2 Impact of multi-scale feature extraction

[0198] In order to explore the impact of multi-scale feature extraction on model performance, this embodiment conducted an ablation experiment on two data sets to evaluate the contribution of different modules in multi-scale feature extraction. This embodiment tested the following situations:

[0199] w / o 1×1: 1×1 convolutional blocks are not used to extract features in TMFN.

[0200] w / o 3×3: 3×3 convolutional blocks are not used to extract features in TMFN.

[0201] w / o 5×5: 5×5 convolutional blocks are not used to extract features in TMFN.

[0202] w / o Avgpool: Avgpool is not used to extract features in TMFN.

[0203] w / o Channel Attention: Channel attention is not used in TMFN for further feature enhancement.

[0204] As shown in Table 6, by analyzing Table 6, the following conclusions can be drawn: (1) The absence of the feature extraction module has an adverse effect on the model performance, which verifies the effectiveness of the multi-scale feature extraction strategy proposed in this embodiment. In particular, removing the Channel Attention module leads to the most obvious performance degradation, while removing the 1×1 convolution module has a relatively small impact. (2) Although the 1×1 convolution helps capture local features, its contribution to global features is limited, so its removal only leads to a small performance degradation. In contrast, removing the average pooling layer leads to a significant performance degradation because average pooling retains the key part of spatial information while reducing the feature dimension. (3) The absence of the Channel Attention module has the greatest impact on the model performance, which emphasizes the importance of implementing an effective feature enhancement mechanism based on multi-size convolution kernels. (4) Removing the 3×3 or 5×5 convolution modules also leads to performance degradation because these modules play a core role in extracting local and neighborhood features. Therefore, the channel attention mechanism is applied on the 3×3 and 5×5 convolution outputs to strengthen the features and further improve the overall performance of the model.

[0205] Table 6 The influence of different modules in multi-scale feature extraction

[0206]

[0207] 5.4. Example Study

[0208] 5.4.1 Unsupervised Contrastive Learning

[0209] In order to deeply analyze Unsupervised contrastive learning (UCL), this embodiment uses the t-SNE algorithm to convert multimodal feature representation into two-dimensional feature points to achieve visualization. Figure 4 As shown, Figure 4 (b) (right sub-figure) shows the embedding space after applying UCL learning, while Figure 4(a) (left sub-figure) shows the embedding space when UCL is not applied. By comparison, we found that when UCL is not used, the data points in the embedding space are distributed more randomly, and there is a lack of obvious distinction between different categories. After applying UCL, the distance between data points is significantly reduced, forming a clearer clustering effect, which shows that our UCL design effectively achieves the division of sample categories. Nevertheless, it can be noticed that after optimization, some data points are still misclassified, resulting in some clusters overlapping. This phenomenon suggests that we need to further optimize the learning strategy of the model in the future to reduce the occurrence of misclassification and improve the accuracy of the model.

[0210] 5.4.2 Model Performance

[0211] In order to comprehensively evaluate the performance of our proposed TMFN model, we conducted a series of experiments on the CMU-MOSI dataset. The experimental results are shown in Figure 5 The table shows the factual sentiment labels of each sample and the prediction results of each model. Figure 5 From the table, we can see that compared with advanced models such as MMIN, Self-MM and SUGRM, our model performs better in sentiment prediction accuracy, mainly due to its innovative multi-scale feature extraction technology and text-guided multimodal feature fusion strategy. These technologies enable the model to more accurately capture and analyze sentiment information in multimodal data, thereby effectively improving performance.

[0212] 6. Overview

[0213] This embodiment constructs a multimodal sentiment analysis model, which uses convolution kernels of different sizes to capture the multi-scale features of modal data and strengthens the recognition of key information through the channel attention mechanism. In order to give full play to the advantages of text modality in multimodal sentiment analysis, a text multimodal feature fusion module is designed. This module regulates the fusion of other modal information through a text gating unit, and applies the text multimodal feature fusion module to deepen the understanding of the relationship between different modal features, effectively promoting feature fusion. Finally, an unsupervised contrastive learning method is introduced to further optimize the feature representation by mining the mutual information between fused features and multi-scale features. Experimental results on two standard data sets show that the performance of the multimodal sentiment analysis model on multimodal sentiment analysis tasks exceeds that of existing baseline models.

[0214] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made in form and details without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A multimodal sentiment analysis method based on a multimodal fusion network, characterized in that: The steps include: S1. Obtaining a to-be-processed emotional information modality, and performing data extraction on the to-be-processed emotional information modality, wherein the emotional information modality includes a text modality, an audio modality, and a video modality; S2, inputting the emotional information modality after data extraction into the trained multimodal emotional analysis model, and outputting the emotional prediction result of the emotional information modality to be processed; the multimodal emotional analysis model includes a multi-scale feature extraction module, a text multimodal feature fusion module and a multimodal emotional prediction module; The training steps of the multimodal sentiment analysis model are as follows: S201, the multimodal sentiment analysis model uses the sentiment information modality extracted from the input data as the input of the multiscale feature extraction module, outputs the multiscale features with text modality, audio modality and video modality, and inputs them into the text multimodal feature fusion module; S202, the text multimodal feature fusion module fuses the input multi-scale features of different modalities through a text gating unit with the text modality as the core to generate fused features, designs a text channel attention Transformer unit with a multimodal channel attention mechanism, uses the fused features as input of the text channel attention Transformer unit, outputs the modal fusion features, and uses them as input of the multimodal sentiment prediction module; S203, the multimodal emotion prediction module connects the input modal fusion features to obtain fusion features, and obtains the emotion prediction result of the emotion information modality to be processed according to the fusion features; S204, optimizing and updating the model parameters of the multimodal sentiment analysis model with the goal of minimizing the total loss function constructed by the model basic loss function and the unsupervised contrastive learning loss function, thereby completing the training of the multimodal sentiment analysis model; S205. Repeat steps S201 to S204 to perform iterative training until the multimodal sentiment analysis model converges or reaches a preset number of iterations.

2. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 1, characterized in that: The data extraction specifically includes: The step of extracting data from the text modality includes: extracting features from the input text sequence using a pre-trained BERT model to obtain a global representation and a local representation of the text modality, wherein the global representation is a corresponding vector marked by a special marker [CLS], and the [CLS] marker vector corresponding to the last layer of the BERT model is used as a comprehensive representation of the text sequence. The expression for data extraction from text mode is as follows: Where: Represents the text features extracted by the BERT model. l t is the maximum length of a text sequence, d t represents the dimension of comprehensive representation of text sequence, Represents the learnable parameters of the text modality in BERT; The step of extracting data from the audio modality comprises: extracting features from the audio modality using the pre-trained COVAREP model to obtain an initial vector feature I a Then, the initial vector feature I a Through the sLSTM model, the features in its time series are deeply captured. Finally, the end-state hidden vector is selected as the comprehensive representation of the audio sequence. The expression for data extraction of the audio modality is as follows: In the formula, represents the audio features extracted by the pre-trained COVAREP model, l a Indicates the sequence length of the audio, d a represents the dimension of the comprehensive representation of the audio sequence, Represents the learnable parameters of the audio modality in the sLSTM model; The step of extracting data from the video modality includes: extracting features from the video modality using a pre-trained FACET model to obtain an initial vector I v Then, the initial vector feature I v Through the sLSTM model, the features in its time series are deeply captured. Finally, the end-state hidden vector is selected as the comprehensive representation of the video sequence. The expression for data extraction of the video modality is as follows: In the formula, represents the video features extracted by the pre-trained FACET model, l v Indicates the sequence length of the video, d v represents the dimension of the comprehensive representation of the video sequence, Represents the learnable parameters of the video modality in sLSTM.

3. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 1, characterized in that: In step S201, the multi-scale feature extraction module includes four convolution blocks and a channel attention unit in parallel, wherein the four convolution blocks in parallel are respectively a first convolution block, a second convolution block, a third convolution block and a fourth convolution block, and the parallel connection result of the second convolution block and the third convolution block is cascaded with the channel attention unit; The first convolution block is a 1×1 convolution layer, the second convolution block is a 3×3 convolution layer, the third convolution block is a 5×5 convolution layer, and the fourth convolution block is a global pooling layer; the outputs of the second and third convolution blocks in parallel are used as the input of the channel attention unit, and the output of the channel attention unit is added with the outputs of the first and fourth convolution blocks to obtain the overall output of different modalities in the multi-scale feature extraction module.

4. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 3 is characterized in that: The channel attention unit includes a cascaded global average pooling operation, an excitation operation, and a scaling operation; in the channel attention unit, the outputs of the second convolution block and the third convolution block are compressed along the sequence length using a global average pooling operation to calculate global spatial information, and then an excitation operation is performed to generate weights for each feature channel, and finally, the outputs of different channels are controlled by scaling operations and weight parameters, so as to output the features that have passed through the channel attention unit; The processing process of the channel attention unit is expressed as: η=Excitation(Z,W)=σ(g(Z,W))=σ(W2ReLU(W1Z)); X′ sn =Scale(X sn ,η)=X sn ×η; In the formula, Z represents the global spatial information, Squeeze(·) represents the global average pooling operation, and X sn express After the output features of the second and third convolution blocks, Represents the features obtained by data extraction of different modalities, s∈{a,t,v}, n∈{2,3}, η represents the weight parameter, Excitation(·,w) represents the excitation operation, g(·) is the nonlinear function, σ(·) represents the sigmoid function, W1 and W2 both represent the weight matrix of the fully connected layer, ReLU represents the activation function, and X' vn represents the output feature after channel attention enhancement, Scale(·) represents the scaling operation; Then the processing process of the multi-scale feature extraction module is expressed as: Where, X a represents the multi-scale features of the audio modality, X t represents the multi-scale features of text modality, X v represents the multi-scale features of the video modality, Concat(·) represents the connection, and X a1 , X t1 , X v1 Respectively represent the output of the audio, text and video modalities after the first convolutional block, X a4 , X t4 , X v4 Represent the outputs of the fourth convolutional block for audio, text, and video modalities, respectively.

5. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 1, characterized in that: In step S202, the text gating unit includes an audio text gating subunit and a video text gating subunit, and the audio text gating subunit has the same structure as the video text gating subunit, including a cascaded convolution normalization unit, a convolution activation unit, a normalization activation unit, and a Sigmoid activation function layer, wherein the convolution normalization unit includes a cascaded first 1×1 convolution layer and a first batch normalization layer, the convolution activation unit includes a cascaded first ReLU activation function layer and a second 1×1 convolution layer, and the normalization activation unit includes a cascaded second batch normalization layer and a second ReLU activation function layer; In the video text gating subunit, the input text modality multi-scale features and the video modality multi-scale features are processed by the convolution normalization unit respectively, and the processed features are added to obtain a fused feature map, and then the fused feature map is passed through the convolution activation unit to obtain a single-channel feature map, and then processed by the normalization activation unit, and then the processed single-channel feature map is converted into an attention coefficient through a Sigmoid activation function layer, and finally, the video modality multi-scale features as input and the output of the Sigmoid activation function layer are element-wise multiplied to obtain features as outputs of the video text gating subunit; In the audio-text gating subunit, the input text modality multi-scale features and the audio modality multi-scale features are used as inputs of the audio-text gating subunit, and are processed by convolution normalization units respectively, and the processed features are added to obtain a fused feature map, and then the fused feature map is passed through a convolution activation unit to obtain a single-channel feature map, and then processed by a normalization activation unit, and then the processed single-channel feature map is converted into an attention coefficient through a Sigmoid activation function layer, and finally, the video modality multi-scale features as input and the output of the Sigmoid activation function layer are element-wise multiplied to obtain features as outputs of the audio-text gating subunit; The features output by the video text gating subunit and the features output by the audio text gating subunit are used as the overall output features of the text gating unit; The video text gating subunit processing process is expressed as: X′ tv =BN(Conv 1×1 (X t ))+BN(Conv 1×1 (X v )); X tv =X v ⊙σ(δ(BN(Conv 1×1 (δ(X′ tv ))))); Where X' tv represents the fusion feature map of the multi-scale features of the text modality and the multi-scale features of the video modality, BN(·) represents batch normalization, Conv 1×1 (·) represents a one-dimensional convolutional layer, X t represents the multi-scale features of text modality, X v represents the multi-scale features of video modality, X tv represents the fusion of the multi-scale features of the video modality and the multi-scale features of the text modality output by the video-text gating subunit, ⊙ represents element-wise multiplication, and δ(·) represents the ReLU activation function; The audio text gating subunit processing process is expressed as: X′ ta =BN(Conv 1×1 (X t ))+BN(Conv 1×1 (X a )); X ta =X v ⊙σ(δ(BN(Conv 1×1 (δ(X′ ta ))))); Where X' ta Represents the fusion feature map of the multi-scale features of the text modality and the multi-scale features of the audio modality, X t represents the multi-scale features of text modality, X a represents the multi-scale features of the audio modality, X ta Represents the features of the audio modality and text modality fused at the output of the audio-text gating subunit.

6. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 1, characterized in that: In step S202, the text channel attention Transformer unit with a multimodal channel attention mechanism includes N layers of network layers composed of a cascade of a multimodal channel attention module and a fully connected feedforward module; the multimodal channel attention module includes a first normalization layer and a multi-head attention unit connected in sequence, and the fully connected feedforward module includes a second normalization layer and a fully connected feedforward unit connected in sequence; In the text channel attention Transformer unit, the input feature map of the text channel attention Transformer unit is used as the input of the first normalization layer in the multimodal channel attention module in the first network layer, the output of the first normalization layer is used as the input of the multi-head attention unit, the output of the modal channel attention unit is added to the input of the channel attention Transformer unit, and the output is used as the input of the second normalization layer in the fully connected feedforward module, the output of the second normalization layer is used as the input of the fully connected feedforward unit, the input of the fully connected feedforward unit is added to the output of the multimodal channel attention module, and its output is used as the input of the next network layer, and after N layers of feedforward calculation, its output is used as the overall output of the text channel attention Transformer unit; The processing process is expressed as: In the formula, represents the fused feature representation of the text modality and the video modality after being processed by the multi-head attention unit at the i-1 layer, represents the fused feature representation of the text modality and the audio modality after being processed by the multi-head attention unit at the i-1 layer, Both represent the multi-head attention unit of the i-th layer, LN(·) represents the layer normalization layer, represents the fusion feature representation of the i-1th layer text modality and video modality, represents the fusion feature representation of the i-1th layer text modality and audio modality, represents the fully connected feed-forward unit of the i-th layer, represents the fusion feature representation of the text modality and the video modality after the i-th layer is processed by the fully connected feed-forward module, represents the fusion feature representation of the text modality and audio modality after the i-th layer is processed by the fully connected feed-forward module, TCAT(·) represents the text channel attention Transformer unit, X ta represents the fusion feature of the audio modality and text modality output by the audio-text gating subunit, X a represents the multi-scale features of the audio modality, X v Represents the multi-scale features of video modality, Represents the modality fusion features of the text modality and the audio modality after being fused by the text channel attention Transformer unit, Represents the modality fusion features of the text modality and the video modality after being fused by the text channel attention Transformer unit.

7. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 6 is characterized in that: The multi-head attention unit projects the fused features and multi-scale features obtained after processing by the text gating unit into the query, key, and value vector space, and then applies the scaled dot product attention mechanism on the channel dimension to generate an output vector; The X tv The processing process of the multi-head attention unit is expressed as: Where Y tv Represents X tv The output after attention mechanism processing, Attention(·) represents the attention mechanism, Q tv Represents X tv The query matrix, K v Represents X v The bond matrix, V v Represents X v The value matrix of , Softmax(·) represents the Softmax function, represents the weight of the query matrix, W K represents the weight of the bond matrix, represents the weight of the value matrix, Represents the output of each head in the multi-head attention unit of the video modality, and W0 represents the weight matrix of the output after splicing the multi-head attention unit. The X ta The processing process of the multi-head attention unit is expressed as: Where Y ta Represents X ta The output after attention mechanism processing, Q ta Represents X ta The query matrix, K a Represents X a The bond matrix, V a Represents X a The value matrix of represents the weight of the value matrix, The output of each head in the multi-head attention unit representing the audio modality.

8. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 1, characterized in that: In step S203, the processing process of the multimodal emotion prediction module includes: The audio and text modality fusion features as input are connected with the video and text modality fusion features, and then the features are mapped to a lower-dimensional feature space through the first linear layer of the activation function to obtain the fusion feature X m , passing the fused features through the second linear layer to generate the sentiment prediction result as output; The processing process is expressed as: Where, X m represents the fusion feature obtained after the first linear layer processing, ReLU(·) represents the activation function, represents the weight parameter of the first linear layer, represents the bias parameter of the first linear layer, represents the sentiment prediction result, represents the weight parameter of the second linear layer, Represents the bias parameters of the second linear layer.

9. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 1, characterized in that: In step S204, the unsupervised contrastive learning loss function is: In the formula, Point(·) represents the correlation quantization function, exp(·) represents the exponential function, and X s Represents multi-scale features of different modalities, Represents X s The normalized eigenvector of Indicates that the parameter is The neural network, Represents X m With X s The contrast loss function between s represents the expected value of different modes, represents the fusion feature of the jth sample, represents the multi-scale features of different modalities of the j-th sample, represents the unsupervised contrastive learning loss function, represents the contrast loss between the fusion feature and the text modality, represents the contrast loss between the fused features and the audio modality, represents the contrast loss between the fused features and the video modality; The basic loss function of the model is: In the formula, represents the basic loss function of the model, N represents the total number of training samples, represents the predicted value of the multimodal sentiment analysis model for the i-th sample, Represents the true value of the i-th sample.

10. The multimodal sentiment analysis method based on a multimodal fusion network according to claim 9, characterized in that: The total loss function is: In the formula, represents the total loss function, α represents The weight of The weight of .

Images