A multimodal sentiment analysis method based on modal interaction multi-task learning
Through the modal interaction multi-task learning method, text, speech, and visual features are extracted from videos, and a multi-task learning framework is constructed. This solves the problems of model complexity and computational overhead in multimodal sentiment analysis and achieves more efficient multimodal sentiment analysis.
Patent Information
- Application Number
- CN202311117032.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-31
- Publication Date
- 2025-09-23
- Estimated Expiration
- 2043-08-31
AI Technical Summary
Existing multimodal sentiment analysis methods ignore the effectiveness of modal information, resulting in complex models and high computational overhead, and fail to effectively utilize the complementary information between multiple modalities.
A multi-task learning method based on modal interaction is adopted to extract text, speech, and visual unimodal features from videos, construct a multi-task learning framework, and use a neural network model with text as the dominant modality, combined with feature representation, fusion, and output modules, to learn the information between text and non-text modalities, and use outer product operations and loss functions to optimize the model.
It improves the accuracy and generalization ability of multimodal sentiment analysis, reduces computing resources and time overhead, and adapts to different video scenarios.
Smart Images
Figure CN117150358B_ABST
Abstract
Description
Technical Field
[0001] The present invention discloses a multimodal sentiment analysis method based on modal interaction multi-task learning, which relates to the technical field of natural language processing. Background Art
[0002] With the development of the information age, users are generating an increasing amount of online content, encompassing text, voice, and image data. Analyzing and studying the sentiment polarity of multimodal data plays a crucial role in improving and enhancing the human-machine and machine-machine interaction environments of the information age. Multimodal sentiment analysis has attracted increasing attention in recent years. Compared to unimodal sentiment analysis, multimodal sentiment analysis studies the mutual complementation and enhancement between multiple modalities, achieving significant improvements when processing social media data.
[0003] Traditional sentiment analysis methods, such as text-based and speech-based sentiment analysis, focus solely on a single modality. The text, speech, and visual modalities in video data originate from multiple heterogeneous sources and differ significantly from one another. Therefore, the core challenge of multimodal sentiment analysis is how to fuse information from these different modalities and fully leverage their complementary information. Furthermore, current multimodal sentiment analysis methods mostly focus solely on modal fusion, ignoring the fact that not all modal information is useful, which can negatively impact sentiment analysis. Furthermore, most current models are end-to-end, resulting in relatively complex and time-consuming models. Summary of the Invention
[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a multimodal sentiment analysis method based on modal interaction multi-task learning, and it has achieved ideal results in multimodal sentiment analysis and can effectively improve the effect of multimodal tasks.
[0005] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:
[0006] A multimodal sentiment analysis method based on modal interaction multi-task learning includes: preprocessing the video; obtaining text, speech, and visual unimodal original features from the preprocessed data; further obtaining unimodal feature representations from the three unimodal original features; constructing a multi-task learning framework based on a neural network model according to the characteristic that multimodal data is dominated by text: the model consists of two parts, namely a multimodal task and two subtasks for learning information between text modalities and non-text modalities. The multimodal task adopts a classic multimodal sentiment analysis architecture, which includes three modules: a feature representation module, a feature fusion module, and an output module. After the feature representation module obtains the unimodal features, it splices the three unimodal feature representations and projects them into a low-dimensional space. Finally, the multimodal fusion feature representation is used to predict the multimodal sentiment; the subtask part projects the previously obtained unimodal feature representation into a new feature space, interacts the text modality with the non-text modality, learns the information between the text modality and the non-text modality, and thus effectively improves the effect of the multimodal task.
[0007] A multimodal sentiment analysis method based on modal interaction multi-task learning includes the following steps:
[0008] S1. Preprocess the video: obtain the corresponding text from the video; separate the audio from the video and transcribe the audio; frame the video, and perform face detection and alignment on the video frames;
[0009] S2. Obtaining unimodal raw features from preprocessed data: Extracting audio and video features using pre-trained toolkits;
[0010] S3. Based on the unimodal original features, further extract unimodal feature representation, including text feature representation, audio feature representation, and video feature representation;
[0011] S4. Based on the characteristic that text is the dominant modality of multimodal data, a multi-task learning framework based on a neural network model is constructed;
[0012] S5. The multimodal sentiment analysis task uses a classic multimodal sentiment analysis architecture, which includes three modules: feature representation module, feature fusion module, and output module. After the feature representation module obtains the unimodal features, it concatenates the three unimodal feature representations and projects them into a low-dimensional space. Finally, the multimodal fusion feature representation is used to predict the multimodal sentiment.
[0013] S6, the subtask part projects the previously obtained unimodal feature representation into a new feature space, interacts the text modality with the non-text modality, learns the information of the text modality and the non-text modality, and thus effectively improves the results of the multimodal task.
[0014] Furthermore, the step S3 specifically includes the following steps:
[0015] S31. For the original text features, use the pre-trained 12-layer BERT to extract the feature representation of the text. Based on experience, the first word vector of the last layer is selected as the representation of the entire sentence;
[0016] S32. For the original audio features, a unidirectional long short-term memory neural network is used to obtain the feature representation of the audio, and the final hidden layer vector is used as the full sequence representation;
[0017] S33. For the original video features, a unidirectional long short-term memory neural network is used to obtain the feature representation of the video, and the last hidden layer vector is used as the full sequence representation.
[0018] Furthermore, the text feature representation, audio feature representation, and video feature representation are specifically:
[0019]
[0020] Among them, F t Represents the text modal feature representation, which is the original text information I t The word vectors selected from the text feature representation extracted by BERT; is the weight parameter used in the text modality feature extraction process; d t’ is the dimension of the last layer of BERT;
[0021]
[0022] Among them, F a Represents the audio modal feature representation, which is the original speech feature I a Obtain audio feature representation through LSTM; is the weight parameter used in the process of extracting audio modal features; d a’ is the hidden layer dimension of LSTM;
[0023]
[0024] Among them, F v Represents the video modality feature representation, which is the original visual feature I v Obtain the feature representation of the video through LSTM; is the weight parameter used in the process of extracting video modality features; d v’ is the hidden layer dimension of LSTM.
[0025] Furthermore, the step S4 specifically includes the following steps:
[0026] The multi-task learning framework consists of two parts: a multimodal task and two subtasks for learning information between textual and non-textual modalities. The subtasks combine the characteristics of multimodal data with text as the dominant modality and learn information between textual and non-textual modalities to guide the final multimodal sentiment analysis task.
[0027] Furthermore, the step S5 specifically includes the following steps:
[0028] All unimodal representations are concatenated and projected into a low-dimensional space:
[0029]
[0030] in Represents the feature representation of multimodal fusion, is the learnable weight matrix, Indicates offset; F t represents the text modality feature representation, F a represents the audio modality feature representation, F v Represents video modality feature representation;
[0031] Finally, the multimodal fusion feature representation F m To predict multimodal sentiment:
[0032]
[0033] Among them, y m is the final multimodal sentiment prediction result, represents the learnable weight matrix, Indicates the offset.
[0034] Furthermore, the step S6 specifically includes the following steps:
[0035] In order to avoid the impact of dimensionality differences between different modalities, the previously obtained single-modal feature representation is directly projected into a new feature space;
[0036] An additional constant 1 is added to the dimensions of the three unimodal feature representations so that the fusion feature generated by the outer product of the two unimodal feature representations contains not only the information of the interaction between the two modalities, but also the information of the unimodal feature representations:
[0037] f t ∈[F t 1] T
[0038] f v ∈[F v 1] T
[0039] f a ∈[F a 1] T
[0040] Among them, f t 、f v 、f a To add an additional constant 1 feature to the dimension of the three unimodal feature representations of the projection; F t represents the text modality feature representation, F a represents the audio modality feature representation, F v Represents video modality feature representation;
[0041] The text modality feature f t and speech modal features f a , visual modality features f v Outer product:
[0042]
[0043]
[0044] in, represents the outer product operation, Represents the result of outer product of text modality with visual modality and audio modality respectively;
[0045] The outer product fusion is passed through several linear layers to facilitate the subsequent learning process:
[0046]
[0047]
[0048]
[0049] in, represents the learnable weight matrix; Both represent offset;
[0050] Finally, the information between text features and non-text features is learned through a learning method that reduces the loss function, thereby improving the overall performance of the multimodal sentiment analysis model corresponding to this method;
[0051] The overall learning method of the multimodal sentiment analysis model corresponding to this method is achieved by minimizing the total loss function result:
[0052] l=l task +2αl MSE
[0053] Among them, α is the weight of each loss function of the subtask to the total loss function, l task represents the multimodal task loss, l MSE Indicates text mode Z t With two fusion modes Z tv , Z ta The loss function between is as follows:
[0054]
[0055]
[0056] Where N is the number of training samples, represents the multimodal emotional truth results of the i-th batch, is the multimodal sentiment prediction result of the i-th batch, k∈(tv,ta), Represents the fusion mode Z of the i-th batch tv and Z ta , Represents the text modality of the i-th batch.
[0057] Compared with the prior art, the present invention has the following beneficial effects:
[0058] 1. The present invention extracts three unimodal features from the video; and adds an extra constant 1 to the dimension of the feature representation of the three modalities, so that the fusion feature generated after the outer product of the two modal features not only contains the information of the interaction between the two modalities, but also contains the information of the unimodal features; combined with the characteristic that multimodal data is dominated by text, the text modal feature representation is fused with the audio modal feature representation and the video modal feature representation by outer product for multi-task learning, thereby improving the generalization of the model and the accuracy of sentiment analysis.
[0059] 2. Both the multi-task part and the sub-task part of the model of the present invention adopt a typical framework, which can not only effectively reduce time overhead and save computing resources, but also enable the model to adapt well to different video scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 Schematic diagram of the model structure of the present invention;
[0061] Figure 2 Schematic diagram of the outer product fusion of two modal features in the present invention. DETAILED DESCRIPTION
[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings.
[0063] A multimodal sentiment analysis method based on multi-task learning with modal interaction includes: preprocessing videos; obtaining text, speech, and visual unimodal raw features from the preprocessed data; and further obtaining unimodal feature representations from each of the three unimodal raw features. Based on the text-dominant modality of multimodal data, a multi-task learning framework based on a neural network model is constructed. The model consists of two parts: a multimodal task and two subtasks that learn information between textual and non-textual modalities. The multimodal task adopts a classic multimodal sentiment analysis architecture, comprising a feature representation module, a feature fusion module, and an output module. After obtaining unimodal features, the feature representation module concatenates the three unimodal feature representations and projects them into a low-dimensional space. Finally, the fused multimodal feature representation is used to predict multimodal sentiment. The subtasks project the previously obtained unimodal feature representations into a new feature space, interacting the textual and non-textual modalities to learn information between the textual and non-textual modalities, thereby effectively improving the performance of the multimodal task.
[0064] The present invention is described in detail below:
[0065] like Figure 1 As shown in FIG, a schematic diagram of the model structure of the present invention, a multimodal sentiment analysis method based on modal interaction multi-task learning has the following specific steps:
[0066] S1. Preprocess the video: obtain the corresponding text from the video; separate the audio from the video and transcribe the audio; frame the video, and perform face detection and alignment on the video frames;
[0067] S2. Obtaining unimodal raw features from preprocessed data: Extracting audio and video features using pre-trained toolkits;
[0068] S3. Based on the unimodal original features, further extract unimodal feature representation, including text feature representation, audio feature representation, and video feature representation;
[0069] S31. For the original text features, use the pre-trained 12-layer BERT to extract the feature representation of the text. Based on experience, the first word vector of the last layer is selected as the representation of the entire sentence:
[0070]
[0071] Among them, F t Represents the text modal feature representation, which is the original text information I t The word vectors selected from the text feature representation extracted by BERT; is the weight parameter used in the text modality feature extraction process; dt’ is the dimension of the last layer of BERT;
[0072] S32. For the original audio features, a unidirectional long short-term memory neural network is used to obtain the feature representation of the audio, and the final hidden layer vector is used as the full sequence representation:
[0073]
[0074] Among them, F a Represents the audio modal feature representation, which is the original speech feature I a Obtain audio feature representation through LSTM; is the weight parameter used in the process of extracting audio modal features; d a’ is the hidden layer dimension of LSTM;
[0075] S33. For the original video features, a unidirectional long short-term memory neural network is used to obtain the feature representation of the video, and the final hidden layer vector is used as the full sequence representation:
[0076]
[0077] Among them, F v Represents the video modality feature representation, which is the original visual feature I v Obtain the feature representation of the video through LSTM; is the weight parameter used in the process of extracting video modality features; d v’ is the hidden layer dimension of LSTM.
[0078] S4. Based on the characteristic that text is the dominant modality of multimodal data, a multi-task learning framework based on a neural network model is constructed;
[0079] like Figure 1 As shown in the figure, the multi-task learning framework mainly consists of two parts: a multimodal task and two sub-tasks for learning the information between text modality and non-text modality: the sub-tasks combine the characteristics of multimodal data with text as the dominant modality, and learn the information between text modality and non-text modality to guide the final multimodal sentiment analysis task.
[0080] S5. The multimodal sentiment analysis task uses a classic multimodal sentiment analysis architecture, which includes three modules: feature representation module, feature fusion module, and output module. After the feature representation module obtains the unimodal features, it concatenates the three unimodal feature representations and projects them into a low-dimensional space. Finally, the multimodal fusion feature representation is used to predict the multimodal sentiment:
[0081] S51, concatenate all unimodal representations and project them into a low-dimensional space:
[0082]
[0083] in Represents the feature representation of multimodal fusion, is the learnable weight matrix, Indicates offset; F t represents the text modality feature representation, F a represents the audio modality feature representation, F v Represents video modality feature representation;
[0084] S52, finally use the multimodal fusion feature representation F m To predict multimodal sentiment:
[0085]
[0086] Among them, y m is the final multimodal sentiment prediction result, represents the learnable weight matrix, Indicates the offset.
[0087] S6, the subtask part projects the previously obtained unimodal feature representation into a new feature space, interacts the text modality with the non-text modality, learns the information of the text modality and the non-text modality, and thus effectively improves the results of the multimodal task.
[0088] S61. To avoid the influence of dimensionality differences between different modalities, the previously obtained single-modal feature representation is directly projected into a new feature space.
[0089] S62. Add an extra constant 1 to the dimensions of the three unimodal feature representations. The purpose is to make the fusion feature generated after the outer product of the two unimodal feature representations contain not only the information of the interaction between the two modalities, but also the information of the unimodal feature representations:
[0090] f t ∈[F t 1] T
[0091] f v ∈[F v 1] T
[0092] f a ∈[F a 1] T
[0093] Among them, f t 、f v 、f aTo add an additional constant 1 feature to the dimension of the three unimodal feature representations of the projection; F t represents the text modality feature representation, F a represents the audio modality feature representation, F v Represents video modality feature representation;
[0094] The text modality feature f t and speech modal features f a , visual modality features f v Outer product:
[0095]
[0096]
[0097] in, represents the outer product operation, Represents the result of outer product of text modality with visual modality and audio modality respectively;
[0098] S63, after outer product fusion, passes through several linear layers to facilitate the subsequent learning process:
[0099]
[0100]
[0101]
[0102] in, represents the learnable weight matrix; Both represent offset;
[0103] S64,
[0104] Finally, the information between text features and non-text features is learned through a learning method that reduces the loss function, thereby improving the overall performance of the multimodal sentiment analysis model corresponding to this method;
[0105] The overall learning method of the multimodal sentiment analysis model corresponding to this method is achieved by minimizing the total loss function result:
[0106] l=l task +2αl MSE
[0107] Among them, α is the weight of each loss function of the subtask to the total loss function, l task represents the multimodal task loss, l MSE Indicates text mode Z t With two fusion modes Z tv , Z taThe loss function between is as follows:
[0108]
[0109]
[0110] Where N is the number of training samples, represents the multimodal emotional truth results of the i-th batch, is the multimodal sentiment prediction result of the i-th batch, k∈(tv,ta), Represents the fusion mode Z of the i-th batch tv and Z ta , Represents the text modality of the i-th batch.
[0111] In order to illustrate the effect of the present invention, the present invention was fairly compared with the most advanced models in the three public MOSI, MOSEI, and SIMS datasets, and achieved ideal results. The MOSI dataset has a total of 93 videos, including 89 speakers and 2199 subjective speech video clips. It is labeled with seven categories of emotional tendencies ranging from -3 to +3; the MOSEI dataset contains 3228 videos, 23453 sentences, 1000 narrators, 250 topics, and a total length of 65 hours. The emotional annotation is a 7-category emotional annotation of each sentence, between [-3 and 3]; the SIMS dataset is a special Chinese MSA benchmark, which contains 2281 video clips collected from different movies, TV series and variety shows. Human annotators label each sample with an emotional score ranging from -1 (strongly negative) to 1 (strongly positive). The experimental results of the datasets used are shown in Table 1:
[0112] Table 1 Model test results
[0113]
[0114] Since the SIMS dataset only contains unaligned data, our model is only compared with the TFN and LMF models in the baseline models on the SIMS dataset, and our model also achieves better results.
[0115] Experimental results show that the proposed model outperforms other baseline models in all four evaluation indicators and in different data scenarios, achieving good results. This fully demonstrates the effectiveness of the model in multimodal sentiment analysis tasks.
[0116] In summary, the present invention combines the characteristics of multimodal data with text as the dominant mode, and uses the outer product method to fully learn the information between text mode and non-text mode, and combines this information with the advantages of multi-task learning data enhancement. A multimodal sentiment analysis method based on modal interaction multi-task learning is proposed, which not only achieves good results in different data venues, but also adopts the classic multimodal sentiment analysis architecture in the multimodal task part, which reduces time overhead and computing resources to a certain extent.
[0117] In order to analyze the effectiveness of the various parts proposed in the model of the present invention, the present invention considers the following three questions:
[0118] Question 1: In the modal representation interaction subtask, does fusing the text modality with the outer product interaction of the audio modality and the video modality respectively play an effective role in the overall effect of the model?
[0119] Question 2: In the modal representation interaction subtask, does learning the loss function for the text modality and the two outer product fusion modalities separately play an effective role in the overall effect of the model?
[0120] Question 3: Does the added modal representation interaction subtask play an effective role in the overall effect of the model?
[0121] To explain the above three questions, we conducted some ablation experiments on the MOSI dataset. The experimental results are shown in Table 2:
[0122] Table 2 Ablation experiment results on the MOSI dataset
[0123]
[0124] +loss(ta, tv) means adding a loss function to learn the information of the two fusion modes, -loss(t, tv) and -loss(t, ta) mean subtracting a loss function and not learning the information of the text and one of the fusion modes. It means removing one outer product operation and replacing it with a linear layer. It means that both outer product operations are removed and replaced by two linear layers, and -subtask means that the subtask part in the entire model framework is removed. From the ablation experiment results, it can be seen that the loss function learning of the text modality and the two outer product fusion modalities plays an effective role in the overall effect of the model. Among them, adding an additional loss function to learn the information between the two outer product fusion features has a negative effect on the overall model; using the text modality as the main modality to fuse and learn the outer products of the speech modality and the visual modality respectively further improves the overall performance of the model. Compared with the case of simply replacing the outer product with a linear layer, the effect is better. It proves that using the text modality as the main modality to interact with the outer products of the speech modality and the visual modality respectively plays an effective role in the overall effect of the model. The experimental results explain the above problems and also illustrate the effectiveness of the part proposed in this invention.
[0125] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.
Claims
1. A multimodal sentiment analysis method based on modal interaction multi-task learning, characterized in that: The following steps are involved: S1. Preprocess the video: obtain the corresponding text from the video; separate the audio from the video and transcribe the audio; frame the video, and perform face detection and alignment on the video frames; S2. Obtaining unimodal raw features from preprocessed data: Extracting audio and video features using pre-trained toolkits; S3. Based on the unimodal original features, further extract unimodal feature representation, including text feature representation, audio feature representation, and video feature representation; S4. Based on the characteristic that text is the dominant modality of multimodal data, a multi-task learning framework based on a neural network model is constructed; S5. The multimodal sentiment analysis task uses a classic multimodal sentiment analysis architecture, which includes three modules: feature representation module, feature fusion module, and output module. After the feature representation module obtains the unimodal features, it concatenates the three unimodal feature representations and projects them into a low-dimensional space. Finally, the multimodal fusion feature representation is used to predict the multimodal sentiment. S6: The subtask projects the previously obtained unimodal feature representation into a new feature space, interacts the text modality with the non-text modality, learns the information of the text modality and the non-text modality, and thus effectively improves the results of the multimodal task; The specific implementation of S6 includes: Project the previously obtained unimodal feature representation into a new feature space; An additional constant 1 is added to the dimensions of the three unimodal feature representations: f t ∈[F t 1] T f v ∈[F v 1] T f a ∈[F a 1] T Among them, f t 、f v 、f a To add an additional constant 1 feature to the dimension of the three unimodal feature representations of the projection; F t represents the text modality feature representation, F a represents the audio modality feature representation, F v Represents video modality feature representation; The text modality feature f t and speech modal features f a , visual modality features f v Outer product: in, represents the outer product operation, Represents the result of outer product of text modality with visual modality and audio modality respectively; The outer product fusion is passed through several linear layers to facilitate the subsequent learning process: in, represents the learnable weight matrix; Both represent offset; The overall learning method of the multimodal sentiment analysis model corresponding to the multimodal sentiment analysis method based on modal interaction multi-task learning is achieved by minimizing the total loss function result: l＝l task +2αl MSE Among them, α is the weight of each loss function of the subtask to the total loss function, l task represents the multimodal task loss, l MSE Indicates text mode Z t With two fusion modes Z tv , Z ta The loss function between is as follows: Where N is the number of training samples, represents the multimodal emotional truth results of the i-th batch, is the multimodal sentiment prediction result of the i-th batch, k∈(tv,ta), Represents the fusion mode Z of the i-th batch tv and Z ta , Represents the text modality of the i-th batch.
2. The multimodal sentiment analysis method based on modal interaction multi-task learning according to claim 1, characterized in that: The step S3 specifically includes the following steps: S31. For the original text features, use the pre-trained 12-layer BERT to extract the feature representation of the text. Based on experience, the first word vector of the last layer is selected as the representation of the entire sentence; S32. For the original audio features, a unidirectional long short-term memory neural network is used to obtain the feature representation of the audio, and the final hidden layer vector is used as the full sequence representation; S33. For the original video features, a unidirectional long short-term memory neural network is used to obtain the feature representation of the video, and the last hidden layer vector is used as the full sequence representation.
3. The multimodal sentiment analysis method based on modal interaction multi-task learning according to claim 2, characterized in that: The text feature representation, audio feature representation, and video feature representation are specifically: Among them, F t Represents the text modal feature representation, which is the original text information I t The word vector selected from the text feature representation extracted by BERT; θ t bert is the weight parameter used in the text modality feature extraction process; d t’ is the dimension of the last layer of BERT; Among them, F a Represents the audio modal feature representation, which is the original speech feature I a Obtain audio feature representation through LSTM; is the weight parameter used in the process of extracting audio modal features; d a ' is the hidden layer dimension of LSTM; Among them, F v Represents the video modality feature representation, which is the original visual feature I v Obtain the feature representation of the video through LSTM; is the weight parameter used in the process of extracting video modality features; d v’ is the hidden layer dimension of LSTM.
4. The multimodal sentiment analysis method based on modal interaction multi-task learning according to claim 1, characterized in that: The specific implementation of S4 includes: The multi-task learning framework consists of two parts: a multimodal task and two subtasks for learning information between textual and non-textual modalities. The subtasks combine the characteristics of multimodal data with text as the dominant modality and learn information between textual and non-textual modalities to guide the final multimodal sentiment analysis task.
5. The multimodal sentiment analysis method based on modal interaction multi-task learning according to claim 1, characterized in that: The specific implementation of S5 includes: All unimodal representations are concatenated and projected into a low-dimensional space: in Represents the feature representation of multimodal fusion, is the learnable weight matrix, Indicates offset; F t represents the text modality feature representation, F a represents the audio modality feature representation, F v Represents video modality feature representation; Finally, the multimodal fusion feature representation F m To predict multimodal sentiment: Among them, y m is the final multimodal sentiment prediction result, represents the learnable weight matrix, Indicates the offset.