Anti-noise multi-modal representation method based on coarse-to-fine progressive cross-modal attention

Abstract

The present application relates to an anti-noise multi-modal representation method (CFP-CMA) based on coarse-to-fine progressive cross-modal attention, to solve the problem of uncertain noise in multi-modal sentiment analysis. First, the present application roughly browses multi-modal information to obtain coarse-grained global information with cross-modal attention, and uses a modal-specific gating unit to dynamically filter the uncertainty noise of each modal. Then, a modal-specific progressive update unit is used to gradually fade the influence of noise in multi-modal data and learn fine-grained features. Finally, an inter-modal adaptive fusion mechanism is used to learn robust multi-modal fusion representation for multi-modal sentiment analysis. The present application uses public datasets MOSI, MOSEI and SIMS to train the model. Experimental results show that the performance of the proposed model in the presence of noise and full modal conditions exceeds that of previous work.

Description

Technical Field

[0001] The present invention relates to a noise-resistant multimodal representation method based on coarse-to-fine progressive cross-modal attention, and belongs to the field of natural language processing. Background Art

[0002] Multimodal sentiment analysis (MSA) has become a mainstream approach to sentiment analysis, as it can analyze richer sentiment by leveraging the complementary power of three modalities (text, audio, and vision). The completeness of all modalities is a common assumption in MSA tasks. However, in real-world scenarios, multimodal data noise can occur at uncertain times and modalities. Modal features often contain significant noise, making sentiment predictions unstable.

[0003] like Figure 2 As shown in Figure 1, (a) shows only the back of the person in this scene, resulting in a loss of facial visual form. (b) In this scene, too many people and occlusions create a lot of noise both visually and auditorily. (c) Intermittent speech creates a lot of noise in the text and audio.

[0004] Under the omnimodal assumption, one of the core issues in multimodal sentiment analysis is how to extract useful features for multimodal fusion. Many methods fuse multimodal features by capturing the interaction between different modalities. For example, additive fusion methods (Sikka et al., 2013; Kim et al., 2013), tensor fusion methods (Zadeh et al., 2017), low-rank fusion (Liu et al., 2018), and attention-based fusion mechanisms (Choi et al., 2018; Yoon et al., 2019). However, the problem of noisy modalities in MSA tasks has received increasing attention. Existing solutions to the noisy modal problem can be roughly divided into two categories: one (Chen et al., 2017; Mittal et al., 2020) uses a gating mechanism to distinguish between noisy and noise-free modalities, removing the noisy modalities (setting them to zero) to keep the modal features noise-free; the second (Pham et al., 2019; Zhao et al., 2021) uses the noise-free modalities to reconstruct the noisy modalities for multimodal sentiment analysis. Most of the above studies assume that the missing modalities are known and replace all missing modalities with zero. However, multimodal data noise can actually appear at uncertain times or in uncertain modalities, and its features are not always all zero.

[0005] In order to solve the problem of multi-modal sentiment analysis in the presence of uncertain time and uncertain modal noise, an anti-noise multi-modal representation method based on coarse-to-fine progressive cross-modal attention (CFP-CMA) is proposed to learn anti-noise multi-modal representation based on modal-specific gating and inter-modal incremental update. Firstly, the multi-modal information is roughly browsed to obtain coarse-grained multi-modal fusion features, and the modal-specific gating unit is used to dynamically filter the uncertain noise in the single modal. Then, the modal-specific progressive update unit is used to gradually update and improve the coarse-grained multi-modal fusion features to obtain anti-noise fine-grained multi-modal features. Finally, the inter-modal adaptive fusion mechanism is used to fuse the coarse-grained multi-modal fusion features and the fine-grained features to obtain robust multi-modal fusion representation, which is used for sentiment classification. SUMMARY

[0006] The present application provides an anti-noise multi-modal representation method based on coarse-to-fine progressive cross-modal attention (CFP-CMA) to solve the problem of multi-modal sentiment analysis in the presence of uncertain time and uncertain modal noise, which improves the shortcomings of some existing methods and improves the performance of multi-modal sentiment analysis.

[0007] The technical scheme of the present application is: an anti-noise multi-modal representation method based on coarse-to-fine progressive cross-modal attention, which comprises extracting features of several modalities by pre-training a language model; fusing the features of several modalities to obtain coarse-grained multi-modal fusion features, using a modal-specific gating unit to filter the noise in the single modal features using coarse-grained multi-modal fusion features; using a modal-specific progressive update unit to gradually update and refine the coarse-grained multi-modal fusion features to obtain fine-grained features; then generating robust multi-modal fusion representation through inter-modal adaptive fusion mechanism for multi-modal sentiment analysis.

[0008] As a further scheme of the present application, the specific steps of the method are as follows:

[0009] Step 1: Use Python language to write program to process data and labels in data set in batches, align data, and use pre-training language model to encode;

[0010] Step 2: Fuse the processed data to obtain coarse-grained multi-modal fusion features; in the modal-specific gating unit, use coarse-grained multi-modal fusion features to filter the noise contained in the single modal data to obtain gating features;

[0011] Step 3: In the modal-specific progressive update unit, use gating features to gradually update and refine coarse-grained multi-modal fusion features to weaken the influence of noise, and obtain updated coarse-grained features;

[0012] Step4: using the obtained coarse-grained multi-modal fusion features and the updated coarse-grained features, through an inter-modal adaptive fusion mechanism, obtaining fine-grained features, dynamically fusing the fine-grained features and the coarse-grained multi-modal fusion features, and finally obtaining robust multi-modal fusion representation;

[0013] Step5: sending the finally obtained robust multi-modal fusion representation into a sentiment classifier, and finally obtaining a sentiment prediction result.

[0014] As a further scheme of the application, the specific steps of Step1 are:

[0015] Step1.1: downloading online public datasets MOSI, MOSEI and SIMS; MOSI is an English multi-modal sentiment analysis benchmark data and contains 2,199 annotated segments; the MOSEI dataset is an expansion of the MOSI dataset and contains 23,453 annotated segments; SIMS is a unique Chinese multi-modal sentiment analysis benchmark dataset and contains 2,281 annotated segments. For the text modality in the dataset, a pre-trained 12-layer BERT is used to extract sentence representation, and the first word vector of the last layer is used as the representation of the entire sentence, denoted by symbol F t ; for the speech and visual modalities in the dataset, a pre-trained toolkit is used to extract initial vector features from the original data, and then sLSTM is used to capture time sequence features, and finally the hidden layer state vector is used as the representation of the entire sequence, and a linear layer is used to align the speech, visual and text modalities, denoted by symbols F a , v respectively.

[0016] As a further scheme of the application, the specific steps of Step2 are:

[0017] Step2.1: concatenating the processed data according to the length dimension to obtain coarse-grained multi-modal fusion features, denoted as:

[0018]

[0019] wherein, denotes concatenation; ReLU is a nonlinear activation function; W c1 is a trainable parameter matrix for mapping modal features to a low-dimensional space; F t is the text modality, F a is the speech modality, and F v is the visual modality; b c1 is a bias term; F c denotes coarse-grained multi-modal fusion features;

[0020] Step2.2: In order to cope with the noise contained in the multi-modal data, the modal specific gating unit is adopted as follows:

[0021]

[0022]

[0023]

[0024] Wherein, S t , S a , S v represent the gating attention value scores of text, speech, visual modalities; f t-gate , f a-gate , f v-gate represent the gating units of text, speech, visual modalities. represent the learnable parameters of the text, speech, visual modality specific gating module.

[0025] Step2.3: After each gating attention score value is multiplied by the single modal feature, the gating feature is obtained as follows:

[0026] G t = F t × S t G a = F a × S a G v = F v × S v

[0027] Wherein, G t , G a , G v represent the gating features of text, speech, visual modalities, which are obtained after the coarse-grained multi-modal fusion features are filtered by the modal specific gating unit.

[0028] As a further scheme of the present application, the Step3 includes:

[0029] Step3.1: In order to gradually update and refine the coarse-grained multi-modal fusion features, the modal specific progressive update unit is adopted as follows:

[0030] F t '= β1× F t + β2× F c

[0031] F a '= β1× F a + β2× F c

[0032] Fv F' = β1xF v + β2xF c

[0033]

[0034] where β1, β2 are hyper-parameters, representing the update ratio between the coarse-grained multi-modal fusion feature and the gating feature; W c2 is a learnable parameter matrix; b c2 is a bias term; F c ' represents the updated coarse-grained feature, F t is the text modality, F a is the speech modality, and F v is the visual modality; F c represents the coarse-grained multi-modal fusion feature.

[0035] As a further scheme of the present application, the Step 4 comprises the following steps:

[0036] Step 4.1: using an inter-modal adaptive fusion mechanism to adaptively fuse the coarse-grained multi-modal feature F c and the updated coarse-grained feature F c ' to obtain a fine-grained feature, denoted as follows:

[0037] F f = δxF c + (1-δ)xF c

[0038] where F f represents the fine-grained feature, and δ is used to refine the coarse-grained multi-modal fusion feature, denoted as:

[0039] δ = sigmoid (W1 T xF c + W2 T xF c ')

[0040] where sigmoid represents a nonlinear activation function; W1, W2 are two learnable parameter matrices.

[0041] Step 4.2: then, using the coarse-grained multi-modal fusion feature F c and the fine-grained feature F f to learn a robust multi-modal representation, specifically as follows:

[0042] F = αxF c + (1-α)xF f

[0043] Wherein, F is the final learned robust multimodal fusion representation, and alpha is used for adaptive fusion of coarse-grained multimodal fusion features F c and fine-grained features F f , expressed as:

[0044] Alpha = tanh (W3 T * F c + W4 T * F f )

[0045] Wherein, tanh represents a nonlinear activation function; W3, W4 are two learnable parameter matrices.

[0046] As a further scheme of the application, the Step5 comprises the following steps:

[0047] Step5.1: the finally obtained robust multimodal fusion representation is sent into a sentiment classifier, and then a probability distribution is obtained after Softmax, which is used for multimodal sentiment classification, and is expressed as follows:

[0048] Q = Softmax (f cls (F))

[0049] Wherein, F represents a robust multimodal fusion representation; f cls is a sentiment classifier composed of a full connection layer; and q represents a probability distribution.

[0050] The application has the following beneficial effects:

[0051] 1. A noise-robust multimodal fusion method is proposed, which has a gradual cross-modal attention from coarse to fine and is used for MSA. The application first attempts to solve the problem of multimodal sentiment analysis in uncertain time and uncertain modal with noise.

[0052] 2. A gradual multimodal fusion method from coarse to fine is proposed, which learns noise-robust multimodal representation through a gradual updating unit of a specific modal and an inter-modal adaptive fusion mechanism.

[0053] 3. Extensive experiments on three benchmark datasets demonstrate the effectiveness of the proposed model in multimodal sentiment analysis under noisy and full modal conditions. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is the anti-noise multimodal representation (CFP-CMA) method process schematic diagram proposed by the application, which has a gradual cross-modal attention from coarse to fine;

[0055] Figure 2 is a schematic diagram of the application in the presence of noise in MOSEI;

[0056] Figure 3 Schematic diagram of the results of SIMS and MOSI under 30% noise conditions of the present invention;

[0057] Figure 4 Schematic diagram of the results of SIMS and MOSI under the random replacement and noise strategy of the present invention;

[0058] Figure 5 FIG. 4 is a schematic diagram of an example study of the MOSEI dataset of the present invention. DETAILED DESCRIPTION

[0059] Example 1: Figures 1-5 As shown in the figure, a noise-resistant multimodal representation method based on coarse-to-fine progressive cross-modal attention is used to train the model using the MOSI dataset as an example. The specific steps of the method are as follows:

[0060] Step 1: Use Python to write a program to batch and align the data and labels in the dataset, and encode them using the pre-trained language model;

[0061] Step 2: The processed data are spliced ​​and fused to obtain coarse-grained multimodal fusion features. In the modality-specific gating unit, the coarse-grained multimodal fusion features are used to filter the noise contained in the single-modal data to obtain the gating features.

[0062] Step 3: In the modality-specific progressive update unit, the gated features are used to gradually update and refine the coarse-grained multimodal fusion features, reduce the influence of noise, and obtain the updated coarse-grained features.

[0063] Step 4: Using the obtained coarse-grained multimodal fusion features and the updated coarse-grained features, fine-grained features are obtained through an inter-modal adaptive fusion mechanism. The fine-grained features and coarse-grained multimodal fusion features are dynamically fused to finally obtain a robust multimodal fusion representation.

[0064] Step 5: The final robust multimodal fusion representation is fed into the sentiment classifier to finally obtain the sentiment prediction result.

[0065] As a further solution of the present invention, the specific steps of Step 1 are:

[0066] Step 1.1: Download the public datasets MOSI, MOSEI, and SIMS from the Internet; for the text modality in the dataset, use the pre-trained 12-layer BERT to extract sentence representations, and use the first word vector of the last layer as the representation of the entire sentence, symbolized by F tdenotes; for the speech and visual modalities in the dataset, initial vector features are extracted from the raw data using pre-trained toolboxes, followed by sLSTM to capture the temporal features, and finally the hidden state vectors are used as the representation of the entire sequence, and a linear layer is used to align the speech, visual modalities and text modalities, respectively denoted by F a v denotes.

[0067] As a further scheme of the present application, the specific steps of Step2 are:

[0068] Step2.1: The processed data is spliced according to the length dimension to obtain coarse-grained multi-modal fusion features, denoted as:

[0069]

[0070] wherein, denotes splicing; ReLU is a nonlinear activation function; W c1 is a trainable parameter matrix that maps modal features to a low-dimensional space; F t is the text modality, F a is the speech modality, and F v is the visual modality; b c1 is a bias term; F c denotes coarse-grained multi-modal fusion features;

[0071] Step2.2: In order to deal with the noise contained in the multi-modal data, a modal-specific gating unit is used as follows:

[0072]

[0073]

[0074]

[0075] wherein, S t , S a , S v denote the gating attention value scores of the text, speech, and visual modalities; f t-gate , f a-gate , f v-gate denote the gating units of the text, speech, and visual modalities; denotes the learnable parameters of the text, speech, and visual modality-specific gating modules;

[0076] Step2.3: After each gating attention score value is multiplied by the single-modal feature, the gating feature is obtained as follows:

[0077] G t = F t x S​t G a =F a ×S a G v =F v ×S v

[0078] wherein, G t , G a , G v represent the gating features of text, speech, visual modalities, which are obtained after the coarse-grained multi-modal fusion features are filtered by the modal-specific gating units.

[0079] As a further scheme of the present application, the Step3 comprises:

[0080] Step3.1: In order to update and refine the coarse-grained multi-modal fusion features step by step, a modal-specific progressive updating unit is adopted as follows:

[0081] F t ′=β1×F t +β2×F c

[0082] F a ′=β1×F a +β2×F c

[0083] F v ′=β1×F v +β2×F c

[0084]

[0085] wherein, β1, β2 are hyperparameters, representing the updating ratio between the coarse-grained multi-modal fusion features and the gating features; W c2 is a learnable parameter matrix; b c2 is a bias term; F c ′ represents the updated coarse-grained features, F t is the text modality, F a is the speech modality, and F v is the visual modality; F c represents the coarse-grained multi-modal fusion features.

[0086] As a further scheme of the present application, the Step4 comprises the following steps:

[0087] Step4.1: A modal-specific adaptive fusion mechanism is adopted to fuse the coarse-grained multi-modal features F c and the updated coarse-grained features F cAdaptively fused to obtain fine-grained features, denoted as follows:

[0088] F f = δ × F c + (1-δ) × F c

[0089] wherein F f denotes fine-grained features, δ is used to refine coarse-grained multi-modal fusion features, denoted as:

[0090] δ = sigmoid (W1 T × F c + W2 T × F c ′)

[0091] wherein sigmoid denotes a nonlinear activation function; W1, W2 are two learnable parameter matrices.

[0092] Step4.2: Then, the coarse-grained multi-modal fusion features F c and the fine-grained features F f are used to learn robust multi-modal representation, specifically as follows:

[0093] F = α × F c + (1-α) × F f

[0094] wherein F is the final learned robust multi-modal fusion representation, α is used to adaptively fuse the coarse-grained multi-modal fusion features F c and the fine-grained features F f , denoted as:

[0095] α = tanh (W3 T × F c + W4 T × F f )

[0096] wherein tanh denotes a nonlinear activation function; W3, W4 are two learnable parameter matrices.

[0097] As a further scheme of the present application, the Step5 includes the following steps:

[0098] Step5.1: The finally obtained robust multi-modal fusion representation is sent into a sentiment classifier, and then a probability distribution is obtained after Softmax, which is used for multi-modal sentiment classification, denoted as follows:

[0099] q = Softmax (f cls (F))

[0100] where F denotes the robust multimodal fusion representation; f cls is the sentiment classifier composed of fully connected layers; q denotes the probability distribution.

[0101] To illustrate the effect of the present application, four groups of comparative experiments are set up. The first group is the main experiment, which is compared with the previous work on full modal data for three data sets; the second group of experiments is compared with the baseline model on noisy data; the third group of experiments is an ablation experiment to verify the effectiveness of the highlighted method; and the fourth group of experiments analyzes the sentiment analysis performance of the model.

[0102] (1) Main experiment results

[0103] First, the model on full modal data sets (CMU-MOSI and CMU-MOSEI) is compared with several state-of-the-art multimodal sentiment analysis models on full modal data sets. The results in Table 1 show that the present application is superior to other state-of-the-art models. This proves that the present application can extract effective representations for multimodal sentiment analysis. The present application also performs full modal testing on the SIMS data set. The results in Table 2 prove that the present application is also effective for Chinese data.

[0104] Table 1: Results of MOSI and MOSEI without random modal noise

[0105]

[0106] Note: In Acc-2 and F1-Score, the left side of “ / ” is calculated as “negative / non-negative”, and the right side is calculated as “negative / positive”.

[0107] Table 2: Results of SIMS under full modal conditions

[0108]

[0109] (2) Experimental results of noisy data

[0110] Table 3 is the experimental results of the present application on noisy data under the condition of the same noise ratio for different noise strategies, and Table 4 is the experimental results of the present application on noisy data under the condition of different noise ratios for the same noise strategy. It can be seen that the present application has good performance and is superior to baseline-Self-MM under uncertain noise modal test conditions. Experiments prove that the present application can effectively filter out uncertain noise in multimodal data, and the final multimodal fusion representation has good robustness.

[0111] Table 3: SIMS and MOSI test results under 20% noise conditions

[0112]

[0113] Table 4: SIMS and MOSI results under the noise strategy of random replacement

[0114]

[0115] (3) Ablation experiment

[0116] Ablation of components: In order to explore the influence of the modal-specific gating unit on the performance of the model, an ablation experiment of removing the modal-specific gating unit is performed. The experimental results, as shown in Figure 3 , 4 , prove the effectiveness of the modal-specific gating unit.

[0117] (4) Case analysis

[0118] In order to further prove the effectiveness of the present application, Figure 5 two examples of uncertain noise and missing modalities are given. In (a), the speaker causes noise in the text and audio forms in the intermittent speech. In (b), due to the change of the scene, only the side face of the person is first shot, causing the lack of visual form. Later, the scene shifts to many people, causing the visual form to contain a lot of noise. In these two scenarios, the present application can effectively filter out the noise in the modal data, and finally obtain the correct prediction result.

[0119] In summary, in order to solve the problem of uncertain noise and uncertain modal noise in multi-modal sentiment analysis, an anti-noise multi-modal representation method with coarse-to-fine progressive cross-modal attention (CFP-CMA) is proposed. The modal-specific gating unit is used to filter the noise in the multi-modal features, and the modal-specific progressive update unit and the intra-modal adaptive update mechanism are used to learn robust multi-modal fusion representation. A large number of experiments verify that the modal-specific gating mechanism can weaken the influence of uncertain noise in multi-modal data, and the modal-specific progressive update unit and the intra-modal adaptive update mechanism can learn robust multi-modal fusion representation.

[0120] The specific embodiments of the present application are described in detail above in combination with the drawings, but the present application is not limited to the above-mentioned embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present application.

Claims

1. A noise-resistant multimodal representation method based on coarse-to-fine progressive cross-modal attention, characterized by: The method includes extracting features of several modalities through a pre-trained language model; The coarse-grained multimodal fusion features are obtained by fusing the features of several modalities. The coarse-grained multimodal fusion features are then used to filter the noise in the single-modal features using a modality-specific gating unit. The coarse-grained multimodal fusion features are gradually updated and refined using a modality-specific progressive updating unit to obtain fine-grained features. Subsequently, a robust multimodal fusion representation is generated through an inter-modal adaptive fusion mechanism for multimodal sentiment analysis. The specific steps of the method are as follows: Step 1: Use Python to write a program to batch and align the data and labels in the dataset, and encode them using the pre-trained language model; Step 2: The processed data are spliced ​​and fused to obtain coarse-grained multimodal fusion features. In the modality-specific gating unit, the coarse-grained multimodal fusion features are used to filter the noise contained in the single-modal data to obtain the gating features. Step 3: In the modality-specific progressive update unit, the gated features are used to gradually update and refine the coarse-grained multimodal fusion features, reduce the influence of noise, and obtain the updated coarse-grained features. Step 4: Using the obtained coarse-grained multimodal fusion features and the updated coarse-grained features, fine-grained features are obtained through an inter-modal adaptive fusion mechanism. The fine-grained features and coarse-grained multimodal fusion features are dynamically fused to finally obtain a robust multimodal fusion representation. Step 5: The final robust multimodal fusion representation is fed into the sentiment classifier to obtain the sentiment prediction result. The Step 3 includes: Step 3.1: In order to gradually update and refine the coarse-grained multimodal fusion features, a modality-specific progressive update unit is adopted as follows: ； ； ； ； in, is a hyperparameter that represents the update ratio between the coarse-grained multimodal fusion features and the gated features; is a learnable parameter matrix; is the bias term; represents the updated coarse-grained features, For text mode, is the speech mode, is the visual modality; Represents coarse-grained multimodal fusion features.

2. The noise-resistant multimodal representation method based on coarse-to-fine progressive cross-modal attention according to claim 1, characterized in that: The specific steps of Step 1 are: Step 1.1: Download the public datasets MOSI, MOSEI, and SIMS from the Internet; for the text modality in the dataset, use the pre-trained 12-layer BERT to extract sentence representations, use the first word vector of the last layer as the representation of the entire sentence, and use the symbol Representation; For the speech and visual modalities in the dataset, the pre-trained toolkit is used to extract the initial vector features from the original data, and then sLSTM is used to capture the temporal features. Finally, the hidden layer state vector is used as the representation of the entire sequence, and the linear layer is used to align the speech, visual and text modalities with symbols respectively. express.

3. The noise-resistant multimodal representation method based on coarse-to-fine progressive cross-modal attention according to claim 1, characterized in that: The specific steps of Step 2 are: Step 2.1: Concatenate the processed data according to the length dimension to obtain coarse-grained multimodal fusion features, expressed as: ； in, Indicates splicing; is a nonlinear activation function; It is a trainable parameter matrix that maps modal features to a low-dimensional space; For text mode, is the speech mode, is the visual modality; is the bias term; Represents coarse-grained multimodal fusion features; Step 2.2: To cope with the noise contained in multimodal data, a modality-specific gating unit is used as follows: ； ； ； in, Represents the gated attention value scores of text, speech, and visual modalities; 、 Gating units representing text, speech, and visual modalities; Represents the learnable parameters of the text, speech, and vision modality-specific gating modules; Step 2.3: After multiplying each gated attention score value with the unimodal feature, the gated features are obtained as follows: ； in, The gated features representing text, speech, and visual modalities are obtained by filtering the noise of coarse-grained multimodal fusion features through modality-specific gating units.

4. The noise-resistant multimodal representation method based on coarse-to-fine progressive cross-modal attention according to claim 1, characterized in that: Step 4 includes the following steps: Step 4.1: Adopting the inter-modal adaptive fusion mechanism, the coarse-grained multimodal features are combined With the updated coarse-grained features Adaptively fuse to obtain fine-grained features, expressed as follows: ； in, Represents fine-grained features, Used to refine the coarse-grained multimodal fusion features, expressed as: ； in, represents a nonlinear activation function; are two learnable parameter matrices; Step 4.2: Then, use the coarse-grained multimodal fusion feature and fine-grained features To learn robust multimodal representations, we can do the following: ； in, is the robust multimodal fusion representation finally learned, For adaptive fusion of coarse-grained multimodal fusion features and fine-grained features , expressed as: ； Among them, tanh represents a nonlinear activation function; are two learnable parameter matrices.

5. The noise-resistant multimodal representation method based on coarse-to-fine progressive cross-modal attention according to claim 1, characterized in that: Step 5 includes the following steps: Step 5.1: The final robust multimodal fusion representation is sent to the sentiment classifier, and then Then we get the probability distribution for multimodal sentiment classification, which is expressed as follows: ； in, Representation of robust multimodal fusion representation; It is a sentiment classifier composed of fully connected layers; Represents a probability distribution.

Images