A multi-modal sentiment analysis method and system based on sequential translation under uncertain missing modalities

Abstract

This invention relates to the field of multimodal sentiment analysis technology, and in particular to a method and system for multimodal sentiment analysis based on sequential translation under uncertain and missing modalities. The method includes: acquiring multimodal sentiment data, including visual, textual, and auditory modalities; constructing a multimodal sentiment analysis model based on sequential translation, firstly by extracting self-attention encoding features of each modality using a Transformer encoder; and then designing a bidirectional sequential translation architecture centered on the textual modality. This invention improves the utilization of textual modal features in the feature fusion process, thereby enhancing the quality of multimodal fusion features and providing strong support for subsequent complex sentiment analysis.

Description

Technical Field

[0001] This invention relates to the field of multimodal sentiment analysis technology, and in particular to a multimodal sentiment analysis method and system based on sequential translation under uncertain missing modalities. Background Technology

[0002] Traditional sentiment analysis primarily relies on textual modalities to accurately identify people's emotions. Its data sources mainly consist of online reviews (movies, books, or consumer products). For example, Long Short-Term Memory (LSTM) networks have been used to perform multi-class predictions of online shopping reviews, travel reviews, and movie reviews. Furthermore, accurate sentiment identification can help create new value in fields such as finance and healthcare.

[0003] Compared to unimodal data, multimodal data contains more emotional information and can integrate complementary features between modalities, giving it a significant advantage in sentiment analysis. To effectively utilize the advantages of multimodal data for more accurate analysis and judgment of people's complex emotions, multimodal sentiment analysis (MSA) has been proposed and has received widespread attention from researchers.

[0004] Multimodal sentiment analysis (MMA) analyzes changes in people's emotions by utilizing multiple modalities in a given video. Currently, MMA is widely used in opinion mining, natural human-computer interaction, product monitoring, and marketing strategy improvement. With the rise of deep learning, many researchers have conducted in-depth research using neural networks. However, many studies are often conducted under the assumption that all multimodal data is always available. In real life, multimodal data may be lost due to interference factors. When a camera is obstructed, visual information cannot be obtained, resulting in the loss of the visual modality in the sample. Similarly, when the ambient noise is too loud, real auditory information is drowned out; or audio equipment malfunctions, leading to the loss of the audio modality; in special cases, users cannot provide textual information for privacy reasons, resulting in the absence of the textual modality. Such samples inevitably affect the results of sentiment analysis. Therefore, how to handle the problem of uncertain modal loss has become an important challenge in the field of multimodal sentiment analysis.

[0005] Furthermore, current research attempts to develop completion strategies to mitigate the impact of missing modalities, but this requires considering complex missing modalities, which significantly increases model complexity. Therefore, existing research lacks a unified MSA model for addressing missing modalities.

[0006] Current research primarily captures cross-modal mappings by translating between any two modalities. However, this mapping breaks down when one modality in a modality pair is missing. Therefore, existing research has failed to utilize available third modalities to enhance this cross-modal mapping.

[0007] Current research ignores the fact that text modalities contribute the most to sentiment classification tasks. This causes the MSA model to fail to fully utilize the high-quality sentiment features within the text modalities, resulting in a decline in the quality of multimodal fusion features.

[0008] Therefore, there is an urgent need for a multimodal sentiment analysis method based on sequential translation. Summary of the Invention

[0009] To address the aforementioned problems, this invention provides a multimodal sentiment analysis method and system based on sequential translation for uncertain missing modalities.

[0010] Firstly, the present invention provides a multimodal sentiment analysis method based on sequential translation under uncertain missing modalities, employing the following technical solution: A multimodal sentiment analysis method based on sequential translation for uncertain missing modalities includes: Multimodal sentiment data was extracted and preprocessed. The data consisted of three modalities: visual, textual, and auditory. A modal translation architecture based on sequential translation is constructed, which includes extracting self-attention encoding features for each modality based on multimodal sentiment data; performing sequential translation on the extracted features through forward and backward translation architectures respectively; obtaining missing joint features and initial joint features through feature fusion; and obtaining joint feature representations through feature learning based on Transformer encoder-decoder. A common space for the joint representation of multimodal features is constructed, which includes processing multimodal features using a dynamic balancing method based on weight matrices. Specifically, by establishing weight matrices and linearly projecting them, initial joint features containing self-attention features of the three modalities and missing joint features containing output features of the sequential translator decoder are formed respectively. Weight matrices are assigned to both to ensure dynamic balance learning of intermodal information during training. Through specific calculations, the interaction between multimodal features and weight information is realized to ensure that the distribution of joint features can approach the complete modality features even when any modality is missing. Finally, the relevant features are concatenated into a joint representation for subsequent use. The constructed MSA model is trained, and the training process is divided into two stages: no pre-training and pre-training. In the no-pre-training stage, the model lacks complete modal features to guide its training; this process is to obtain training features across all modalities for use in the pre-training stage. In the pre-training stage, the model performs training tasks addressing the lack of modalities by loading training features from all modalities. Sentiment analysis is performed using a trained model; Output sentiment analysis results.

[0011] Furthermore, the step of extracting self-attention encoding features for each modality based on multimodal sentiment data includes feeding samples with uncertain missing modalities into a fully connected layer to obtain multimodal data of the same dimension. The input is then fed into a Transformer encoder for self-attention encoding to extract contextual features, and the encoded features for each modality are output. }, represented as: ; ; , in , , The features of each extracted modality are then residually connected and input into a LayerNorm layer for normalization, as shown below: ; ; , Finally, the normalized single-modal features are input into the feedforward fully connected layer for linear transformation, thereby completing the encoding of the three single-modal data, as follows: , , , in, This is the weight matrix. is the bias, and ReLU is the activation function.

[0012] Furthermore, the extracted features are sequentially translated using both forward and reverse translation architectures, including the use of encoded features from three modalities { The process involves two sequential translation tasks. For the forward translation process VtoTtoA, the Transformer decoder guides the encoder output to approximate the target modality. First, the encoded features of the visual modality are... Encoding features of text modality As input to the decoder; and Quary, as a multi-head attention mechanism, will Decoded as Key and Value, it is represented as: Then After performing residual connection operations, the input to the LayerNorm layer is normalized; then the normalized result is input to the feedforward fully connected layer for linear transformation to complete the Transformer decoder module and obtain... , represented as: , , in It is a weight matrix. is the bias, and ReLU is the activation function.

[0013] Furthermore, the extracted features are sequentially translated using both forward and backward translation architectures. This also includes sequential translation between the three modalities based on the decoder output. Specifically, the text modality interacts directly with the other two modalities. The visual and audio modalities are preferentially translated into the text modality before being translated into the third modality. During translation into the third modality, [further details are needed]. As translation features from visual modality to text modality, a transformer encoder is used to obtain translation features. Encoded output And then translate it into an audio modality, represented as: ; ; , in It is a weight matrix. Here, ReLU is the bias, and ReLU is the activation function; finally, As a multi-head attention mechanism, Query will Using these as keys and values, the translation features from the visual modality to the text modality are made closer to the audio modality, ensuring that the extracted contextual information contains three modalities. Then, the output of the multi-head attention mechanism is used... Normalization and fully connected transformations are performed to obtain the final output of the sequential translation. , represented as: ; , , in It is a weight matrix. is the bias, and ReLU is the activation function.

[0014] Furthermore, the process of obtaining missing joint features and initial joint features through feature fusion includes encoding features of the visual modality, text modality, and audio modality. The initial joint feature IJFs are formed through linear projection; the decoder output of the sequential translation process is then used to generate the corresponding joint feature IJFs. , , , The missing joint feature sets (MJFs) are formed through linear projection, where the independent attention features (IJFs) contain self-attention features from the three modalities, and the MJFs contain decoder output features from the sequential translation process. Appropriate weight matrices are assigned to the IJFs and MJFs to ensure that { }and{ , , , The dynamic equilibrium learning of intermodal information during training is represented as follows: ; ; , Where || represents the concatenation operation. , , These represent the autocorrelation common space of the translated visual modality, the text modality output by the encoder, and the translated audio modality, respectively. ; ; , Where || represents the concatenation operation. , Represent and The autocorrelation space, represent and Cross-related space, finally , , Concatenate them into initial joint features IJFs, and , , The concatenation of missing joint feature MJFs for subsequent use is represented as follows: ; , Furthermore, the step of obtaining a joint feature representation through feature learning based on a Transformer encoder-decoder includes capturing initial joint features using a Transformer encoder-decoder architecture. and missing joint features The conditional representation between them, where the encoder part will As a multi-head attention mechanism, query will As the key and value of a multi-head attention mechanism, the purpose is to allow... near , represented as: ; ; , in , It is a parameter matrix. , It is a learnable bias, and ReLU is the activation function. This invention allows the encoder output to... As input to the decoder, and query, key, and value are all... By making the decoder focus on the multimodal space To maintain context consistency, internal dependencies are represented as follows: ; ; , in , It is a parameter matrix. , It uses learnable biases, ReLU as the activation function, and finally achieves deep modality fusion through decoder loss. It is used to perform sentiment classification tasks.

[0015] Furthermore, the training of the constructed MSA model includes building a pre-trained loss function using Kullback–Leibler (KL) divergence and Jensen–Shannon (JS) divergence. This makes the model-generated joint features with missing features approximate the complete joint features, where, The output of the transformer encoder and pre-trained output The JS divergence is used to determine the pre-trained model, which is trained using the full modality and calculated using the following equation. : , , KL divergence is a commonly used method to measure the difference between two probability distributions; however, its inherent asymmetry can cause the optimization process to be biased in one direction. Therefore, this invention uses JS divergence, which has symmetry, to calculate the loss. The calculation equation is as follows: ; , Furthermore, the training of the constructed MSA model also includes constructing a translation loss function. The model is guided to learn mapping patterns in the modal translation process, and the translation process is supervised to preserve the semantic information and sentiment features of the target modality. Specifically, in sequential translation, visual and audio modalities are preferentially translated into text modalities, with a translation loss... Output from the text modality encoder and the decoder output from visual modality to text modality ( The translation loss is determined by the JS divergence between the two values. Output from the text modality encoder ( ) and the decoder output from audio modality to text modality ( The difference is determined by the JS divergence between them, and is expressed as: ; ; ; , And in and The output is then translated to the remaining third modality, where the decoder output is computed from the visual modality to the text modality and then to the audio modality. and the encoder output of the audio modality JS divergence between them as translation loss ; Calculate the decoder output from audio modality to text modality and then to visual modality and the encoder output of the visual modality JS divergence between them as translation loss , represented as: ; ; ; ; Finally , , , The sum of the two is as follows The equation is as follows: .

[0016] Furthermore, the training of the constructed MSA model also includes constructing a decoder loss. To supervise the reconstruction of joint features and achieve deep modality fusion, the output of the Transformer encoder-decoder architecture is computed. The JS divergence between them is obtained, making... become Guided by the joint representation of MJFs, calculate the equation: ; , Then calculate the classification loss. ,Will Input a fully connected network and use Softmax as the activation function to compute the predicted values. Calculate the equation: , in, This is the weight matrix. The bias is used to calculate the final classification loss, which is expressed as: , Where N is the number of samples, It is the true label of the nth sample. It is a predicted label.

[0017] Secondly, a multimodal sentiment analysis system based on sequential translation for uncertain and missing modalities includes: The data acquisition module is configured to acquire multimodal sentiment data, including visual, textual, and auditory modalities. The model building module is configured to build an MSA model based on sequential translation, which includes extracting self-attention encoding features for each modality based on multimodal sentiment data; performing sequential translation on the extracted features using forward and backward translation architectures respectively; fusing multimodal features through a common space to obtain missing joint features and initial joint features; and learning joint feature representations based on Transformer encoder-decoder. The model training module is configured to train the constructed MSA model. The analysis module is configured to perform sentiment analysis using a trained model. The output module is configured to output sentiment analysis results.

[0018] Thirdly, the present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the aforementioned multimodal sentiment analysis method based on sequential translation under uncertain missing modalities.

[0019] Fourthly, the present invention provides a terminal device, including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to provide a multimodal sentiment analysis method based on sequential translation under uncertain missing modalities.

[0020] In summary, the present invention has the following beneficial technical effects: Unlike existing modal translation methods that only capture cross-modal mappings between two modalities, this invention proposes a sequential translation mechanism. Existing methods become inefficient when modalities are missing due to inter-modal dependencies. Sequential translation improves this by introducing a third modality, extending the mapping to all three modalities. This invention translates the original translation results into a third modality. Furthermore, to fully utilize the high-quality features of the text modality, it directly interacts with the other two modalities, rather than indirectly interacting with a mixture of features from the other two modalities. By constructing a sequential translation mechanism, the model can recover missing information through inter-modal translation paths, significantly enhancing its robustness to uncertain missing modalities. Simultaneously, it dynamically embeds high-quality text modal features into the other two modalities, improving the utilization of text modal features in the feature fusion process and thus enhancing the quality of multimodal fusion features, providing strong support for subsequent complex sentiment analysis.

[0021] Unlike traditional multimodal sentiment analysis methods that often employ unidirectional or pairwise modal translation, leading to insufficient cross-modal interaction and unbalanced semantic transfer, this invention proposes a bidirectional sequential translation architecture based on the text modality. This architecture simultaneously executes two parallel translation paths: forward translation (visual → text → auditory) and backward translation (auditory → text → visual), forming a closed cross-modal semantic enhancement loop. Unlike existing unidirectional translation or simple splicing methods, this invention achieves bidirectional information flow and multi-level semantic alignment among the three modalities during the translation process through a bidirectional path. In the forward path, visual features are calibrated by the text modality and then transferred to the auditory modality; in the backward path, auditory features are also guided to the visual modality through the text modality. The text modality, as the hub of bidirectional translation, not only plays a leading role in sentiment recognition but also promotes consistent semantic conversion and deep feature complementarity between the visual and auditory modalities. This bidirectional sequential translation mechanism ensures that the system can recover robust and discriminative joint features from the interaction information formed by the remaining two modalities in the bidirectional path. Experiments show that this design significantly improves the breadth and depth of cross-modal feature fusion. Whether in single-modal missing or multimodal random missing scenarios, this method consistently outperforms existing mainstream models, verifying the effectiveness and superiority of bidirectional translation structures in improving feature fusion quality.

[0022] Unlike existing methods that require independent completion modules for specific missing conditions (such as generative adversarial networks and residual autoencoders), this invention proposes a unified, end-to-end processing framework based on pre-training without the need for pre-defined completion strategies. This framework achieves unified modeling and dynamic adaptation for any modal missing condition through a two-stage fusion mechanism of common space projection and Transformer encoder-decoder joint learning. In the common space projection stage, the model linearly projects and concatenates the self-attention encoded features and sequential translation output features of each modality using a learnable weight matrix to form initial joint features and missing joint features. This design avoids the complexity of designing dedicated feature completion paths for different missing combinations, allowing the system to dynamically learn the balance and interaction patterns between features under different missing states during training. In the Transformer encoder-decoder joint learning stage, the model further approximates the missing joint features towards the joint distribution of the complete modality using pre-training tasks as supervision. This process does not rely on external completion modules or prior missing assumptions. Instead, it achieves information fusion and reconstruction at the feature level through an adaptive attention mechanism within the encoder-decoder, thereby simplifying model complexity while forming a unified and highly discriminative multimodal joint representation, which helps improve the performance of multimodal sentiment analysis systems. Attached Figure Description

[0023] Figure 1This is a schematic diagram of a multimodal sentiment analysis method based on sequential translation under uncertain missing modalities according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the overall model structure of Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the sequential translation method of Embodiment 1 of the present invention; Figure 4 This is a diagram showing the results of a four-class classification experiment conducted on the IEMOCAP dataset using this invention. Figure 5 This is a diagram showing the results of a seven-class classification experiment conducted on the IEMOCAP dataset using this invention. Detailed Implementation

[0024] The present invention will be further described in detail below with reference to the accompanying drawings.

[0025] Example 1 Reference Figure 1 , Figure 2 and Figure 3 This embodiment of a multimodal sentiment analysis method based on sequential translation under uncertain missing modalities includes the following steps: Acquire and preprocess multimodal sentiment data, including visual, textual, and auditory modalities; A sequential translation-based MSA model is constructed, which includes extracting self-attention encoding features for each modality based on multimodal sentiment data; sequentially translating the extracted features using forward and backward translation architectures respectively; obtaining missing joint features and initial joint features through feature fusion; and obtaining joint feature representations through feature learning based on Transformer encoder-decoder. Train the constructed MSA model; To verify whether the model has converged so that the trained model can be used for subsequent sentiment analysis; Output sentiment analysis results.

[0026] Specifically: S1. Acquire multimodal data and preprocess the data. The multimodal data acquired in this embodiment includes three modes: P=[ ],in and These represent visual, textual, and auditory modalities, respectively. Without loss of generality, this invention uses... Denotes any missing modality, where v, t, a For example, when the visual modality is missing, the multimodal features can be represented as follows: Furthermore, to enable the public space module to dynamically integrate multimodal vectors, this invention establishes six weight matrices, namely... , , , , and The research problem of this invention can be defined as sentiment analysis based on multimodal data with uncertain modal missing features. For ease of representation, the invention will be referred to as... This is used to represent multimodal data with missing uncertain modes.

[0027] For the visual modality, this invention uses the OpenFace 2.0 toolkit to extract facial features. The extraction process provides feedback on three key pieces of information: facial details, head movements, and eye movements. For the text modality, this invention employs a pre-trained BERT model to extract text features, resulting in a 768-dimensional text feature representation. For the audio modality, this invention uses the Librosa tool for multi-dimensional feature extraction. This tool can extract spectral, temporal, and structural features from the original audio signal. This invention mixes audio information from the CMU-MOSI and IEMOCAP datasets into a mono signal, resamples it at 16000Hz, and primarily extracts three features: zero-crossing rate, Mel-frequency cepstral coefficients (MFCC), and constant Q-transform (CQT). Each frame is divided into 512 samples, and the 33-dimensional audio features concatenated from these three features serve as the audio modality. Based on these features, the final multimodal emotional data features are obtained.

[0028] S2. Model Building This embodiment proposes a Sequential Translation-Based MSA (STMSA) model, the overall structure of which is as follows: Figure 2 As shown. The STMSA operation process is as follows: First, multimodal data with missing modes is processed. The initial feature module is used to obtain the self-attention encoded features for each modality. Next, the Sequential Translation Module (STM) performs two translation tasks in parallel: (1) forward process: translating the visual modality to the text modality and then to the audio modality (VtoTtoA); (2) backward process: translating the audio modality to the text modality and then to the visual modality (AtoTtoV). Then, the Common Space Mapping Module fuses the output of the Sequential Translation Module and the self-attention encoded features for each modality with Missing Joint Features (MJFs) and Initial Joint Features (IJFs), respectively. Then, under the supervision of the pre-training task, the Transformer encoder-decoder architecture is used to learn the joint representation of MJFs and IJFs, allowing it to fit the joint distribution of the complete modality. Finally, the output of the joint representation is used to perform the final sentiment analysis task through the Softmax function. The paper first introduces the Transformer, and then provides a detailed introduction to the key modules in the model.

[0029] (1) The Transformer is a deep neural network structure based on an attention mechanism, which has significant advantages in natural language processing. Researchers have extended it to the field of computer vision and demonstrated through experiments that the Transformer also has advantages. This invention utilizes the encoder of the Transformer to realize the modal translation process. A simple attention mechanism requires three parameters: Query, Key, and Value. The lengths of Query and Key are both... The length of Values ​​is For a given input matrix X, this invention defines QueryS as... Keys are defined as Values ​​are defined as .in , and This is the weight matrix. The specific calculation process of the dot product attention is shown in equation (1): , Where Q, K, and V are the query, key, and value, respectively. It is the input dimension.

[0030] The multi-head attention mechanism utilizes dynamic weight matrices to mine feature representations from different semantic spaces and outputs them as a matrix. For each head h, Q, K, and V are projected onto a low-dimensional surface using different learnable weight matrices before the attention function is applied. The calculation process is shown in Equation 2: , in It's the number of heads. It is a weight matrix.

[0031] The i-th head Attention is represented as follows: , in , , Let Q, K, and V represent the weight matrices of the i-th head, respectively.

[0032] (2) Initial feature module First, samples with uncertain missing modalities are fed into a fully connected layer to obtain multimodal data of the same dimension. Then, it is input into a Transformer encoder for self-attention encoding to extract its contextual features. The encoder outputs the encoded features for each modality. The equation is calculated as follows: , , , in , , .

[0033] Next, the features extracted from each modality are residually connected and input into a LayerNorm layer for normalization. The calculation process is as follows: , , , Then, the normalized single-modal features are input into the feedforward fully connected layer for linear transformation, thereby completing the encoding of the three single-modal data, as shown in the equation: , , , in, This is the weight matrix. is the bias, and ReLU is the activation function.

[0034] S3. Constructing the sequential translation module in the model To address the problem of uncertain missing modalities, this invention proposes a sequential translation method. When a modality is missing, the encoded features of the missing modality lack genuine sentiment information, making it difficult for the translation process to capture effective inter-modal complementary features. Therefore, this invention chooses to continue translating the original translation results to a third modality and designs a forward and reverse translation architecture to reduce the impact of missing modality problems on sentiment classification tasks.

[0035] This module utilizes the coding features of three modalities { Two sequential translation tasks are performed, which will be explained below from the perspective of the forward translation process (VtoTtoA).

[0036] For the coding features of the three modalities { This invention utilizes a Transformer decoder to guide the encoder's output to approximate the target modality. Specifically, the calculation process involves encoding the features of the visual modality... Encoding features of text modality As input to the decoder; and Quary, as a multi-head attention mechanism, will Decode the key and value. The calculation process is as shown in the equation: , After that, After performing residual connection operations, the result is input into the LayerNorm layer for normalization. Next, the normalized result is input into the feedforward fully connected layer for linear transformation to complete the Transformer decoder module and obtain... The specific operation process is as shown in the equation: , , in It is a weight matrix. is the bias, and ReLU is the activation function.

[0037] Next, this invention performs sequential translation between the three modalities based on the decoder's output. Furthermore, to fully utilize the high-quality features of the text modality, this invention chooses to directly interact with the other two modalities, rather than indirectly interacting with the mixed features of the other two modalities. Therefore, this invention prioritizes translating the visual and audio modalities to the text modality, and then to the third modality. When translating to the third modality, this invention uses... As translation features from visual modality to text modality. Similarly, a transformer encoder is used to obtain translation features. Encoded output This is then translated into audio modalities. The specific calculation process is as follows: , , , in It is a weight matrix. is the bias, and ReLU is the activation function.

[0038] This invention will As a multi-head attention mechanism, Query will The goal, as the Key and Value, is to further align the translation features from the visual modality to the text modality with the audio modality, ensuring that the extracted contextual information contains all three modalities. The output of the multi-head attention mechanism is then used... Normalization and fully connected transformations are performed to obtain the final output of the sequential translation. The equation is calculated as follows: , , , in It is a weight matrix. is the bias, and ReLU is the activation function.

[0039] The same method can be used to obtain the output of the reverse translation process. and This invention will , , and As a translation feature to improve the accuracy of sentiment classification, among which and It is an output feature that directly interacts with the other two modalities, guided by the text modality. and This is a translation feature that integrates three modal relationships. The next subsection explains how to utilize these translation features.

[0040] S4. Constructing the common space module in the model Set up 6 weight matrices , , , , and This module will { }and{ , , , The initial joint features (IJFs) and missing joint features (MJFs) are generated through linear projection. The mapping process is explained in detail below: IJFs contain self-attention features from three modalities. MJFs contain decoder output features from the sequential translation process. This invention assigns corresponding weight matrices to IJFs and MJFs, thus ensuring that { }and{ , , , During training, it can dynamically learn intermodal information in a balanced manner. The calculation process is as follows: , , , The || symbol represents a concatenation operation. , , These represent the autocorrelation common space of the translated visual modality, the text modality output by the encoder, and the translated audio modality, respectively.

[0041] , , , The || symbol represents a concatenation operation. , Represent and The autocorrelation space. represent and Cross-related space.

[0042] This spatial structure enables information exchange between multimodal features and different weights during the training process, while ensuring that the joint features can approximate the complete modality regardless of which modality is missing.

[0043] Next , , Concatenate them into initial joint features IJFs, and , , The missing joint features (MJFs) are concatenated for later use.

[0044] , .

[0045] S5. Sentiment Classification The Transformer encoder-decoder architecture is used to capture the initial joint features. ) and missing joint features The conditional representation between () is used. The encoder part will... As a multi-head attention mechanism, query will As the key and value in a multi-head attention mechanism, the goal is to... near The equation is calculated as follows: , , , in , It is a parameter matrix. , It is a learnable bias, and ReLU is the activation function. This invention allows the encoder output to... As input to the decoder, and query, key, and value are all... This allows the decoder to focus on the multimodal space. It manages internal dependencies while maintaining context consistency.

[0046] , , , in , It is a parameter matrix. , It is a learnable bias, and ReLU is the activation function. Finally, this invention achieves deep modality fusion through decoder loss, and... It is used to perform sentiment classification tasks. This will be explained in the next subsection.

[0047] S6. Training Objective The model proposed in this invention guides training to approach the target through four losses, namely pre-training loss. Translation loss Decoder loss and classification loss The equation is as follows: , in, , , These are the loss weights. The following sections will explain each loss in detail.

[0048] First, the Kullback–Leibler (KL) divergence is a commonly used method to measure the difference between two probability distributions; however, its inherent asymmetry can cause the optimization process to be biased in one direction. Therefore, this invention uses the symmetric Jensen–Shannon (JS) divergence to calculate the loss, as shown in the following equation: , , Where p and q are two probability distributions.

[0049] (1) Pre-training loss This makes the model-generated joint features with missing features approximate the complete joint features. Therefore... The output of the transformer encoder and pre-trained output The JS divergence is used to determine the pre-trained model, which is trained using the full modality. This invention calculates this using the following equation. : , , (2) Translation loss This invention guides the model to learn mapping patterns in the modal translation process and supervises the translation process to preserve the semantic information and sentiment features of the target modality. In the sequential translation module, the invention prioritizes translating visual and audio modalities into text modalities, and the translation loss in this process is... Output from the text modality encoder and the decoder output from visual modality to text modality ( The translation loss is determined by the JS divergence between the two values. Output from the text modality encoder ( ) and the decoder output from audio modality to text modality ( The difference is determined by the JS divergence between them. The calculation equation is as follows: , , , , To enhance the modality translation method's ability to handle modality loss, this invention... and The output is then translated to the remaining third modality, where the decoder output is computed from the visual modality to the text modality and then to the audio modality. and the encoder output of the audio modality JS divergence between them as translation loss ; Calculate the decoder output from audio modality to text modality and then to visual modality and the encoder output of the visual modality JS divergence between them as translation loss The equation is calculated as follows: , , , , Finally, for ease of explanation, , , , The sum of the two is as follows The equation is as follows: , (3) Decoder loss ): Used to supervise the reconstruction of joint features, achieving deep modality fusion. This invention calculates the output of the Transformer encoder-decoder architecture and The JS divergence between them is obtained, making... become Guide the joint representation of MJFs. The equations are calculated as follows: , , (4) Classification loss This invention will: Input a fully connected network and use Softmax as the activation function to compute the predicted values. The equation is calculated as follows: , in, This is the weight matrix. This is the bias value.

[0050] Cross-entropy loss reduces the probability of overfitting or underfitting the model, making the prediction results more closely match the sample features. Therefore, this invention uses the following equation as the final classification loss: , Where N is the number of samples, It is the true label of the nth sample. It is a predicted label.

[0051] Experimental verification (1) Dataset This invention conducts experiments on two well-known datasets (CMU-MOSI and IEMOCAP) to validate the proposed model. The following section describes the details of these two datasets. The distribution of the datasets is shown in Table 1.

[0052] CMU-MOSI: The videos in the dataset are divided into several discourse-level segments, each labeled with a sentiment level indicating its emotional tendency. The CMU-MOSI dataset contains 2199 opinion videos from online sharing websites such as YouTube, with each video's sentiment label ranging from -3 to +3. Each sentence is annotated as positive or negative.

[0053] IEMOCAP is an emotion dataset that captures fine motor skills in conversations, with emotion labels categorized into 9 classes. It includes 5 recorded videos of two-person dialogues, with a total of 10 speakers. Each session involves a spontaneous conversation between one male and one female speaker, with an average dialogue duration of 5 minutes. The session types include 3 emotion scripts and 8 emotion scenarios.

[0054] This invention follows the emotion labeling process to form an emotion recognition setting, and uses the ACC index and M-F1 score to evaluate the model. The calculation equation is as follows: , , in, This represents the number of samples that made correct predictions. P represents the total number of samples, P represents the positive predictive value, and R represents the recall rate.

[0055] The experimental setup for this invention is shown in Table 1.

[0056] (2) Baseline Model This invention selects the following 10 state-of-the-art benchmark models for comparative experiments and verifies the performance of STMSA. These models are described below: AE: A method for encoding and decoding input vectors using nonlinear mappings, aiming to learn the latent representations in the input vectors.

[0057] CRA: A network that simulates the gap between predicted values ​​and raw data by superimposing residual autoencoders and ultimately estimates missing modes.

[0058] MCTN: A model that translates one modality into another through cyclic propagation (forward and backward) and learns joint features between the two modalities during the translation process.

[0059] TransM: An end-to-end model that uses Transformer to fuse source and target modal information and proposes to perform translation in parallel.

[0060] ICDN: A model that uses a special cross-modal Transformer structure to map other modalities to a target modality, and then uses self-supervision to obtain unimodal sentiment labels to guide sentiment analysis.

[0061] MMIN: A network that models sentiment relevance from sentence-level representations and predicts missing modalities based on available modalities.

[0062] TATE_C: A model that uses label encoding techniques to cover missing cases to assist the model in joint representation learning.

[0063] MTMSA: A model that translates visual and audio modalities into text modalities through modal translation, aiming to improve the quality of feature fusion and thus enhance the robustness of the model by leveraging text modalities.

[0064] TATE_J: This model adds different weights to different modalities to make full use of each modality.

[0065] SMCMSA: A model that uses a pre-built pool of similar samples and cosine similarity to filter similar modalities for missing data completion.

[0066] (3) Performance comparison experiment This invention conducted three-class classification experiments on the CMU-MOSI dataset and two-class classification experiments on the IEMOCAP dataset, comparing the proposed model with 10 benchmark models. To verify the performance of the STMSA model under complex missing modalities, the experiments on each dataset were divided into two parts: the first part conducted experiments under single-modal missing conditions, and the second part conducted experiments under multi-modal missing conditions. Furthermore, the modality missing rate was set to 0–0.5, and M-F1 and ACC were used as performance evaluation metrics. The experimental results are shown in Tables 2 and 3.

[0067] Experiments under unimodal missing value conditions. On the CMU-MOSI dataset, when the missing value is set to 0, STMSA's M-F1 score is 1.76% lower than MMIN's, and its ACC score is 0.08% lower than TATE_J's. When the missing value is set to 0.4, STMSA's M-F1 and ACC scores are 0.44% and 1.04% lower than MTMSA's, respectively. When the missing value is set to 0.1, 0.2, 0.3, and 0.5, STMSA's M-F1 and ACC scores are superior to other models. Specifically, when the missing value is 0.5, STMSA improves the M-F1 score by 0.05% to 13.36% and the ACC score by 0.53% to 19.27%. On the IEMOCAP dataset, when the missing value is set to 0, 0.1, 0.2, 0.3, 0.4, and 0.5, STMSA's M-F1 and ACC scores are significantly better than other benchmark models.

[0068] Experiments under multimodal missing values. On the CMU-MOSI dataset, when the missing value rate is 0%, STMSA's M-F1 score is 1.76% lower than MMIN's, and its ACC score is 0.08% lower than TATE_J's. When the missing value rate is 0.1%, STMSA's M-F1 and ACC scores are 4.04% and 1.04% lower than MTMSA's, respectively. When the missing value rate is 0.5%, the M-F1 and ACC scores are 0.29% and 0.56% lower than TATE_J's, respectively. When the missing value rate is 0.2, 0.3, and 0.4, STMSA's M-F1 and ACC scores outperform the other 10 benchmark models. Specifically, when the missing value rate is 0.3, STMSA improves the M-F1 score by 1.77% to 9.19% and the ACC score by 3.65% to 12.59%. On IEMOCAP, STMSA's M-F1 score was 0.3% lower than MTMSA's only when the missing data rate was 0.2%. Under other missing data rate conditions, STMSA's M-F1 and ACC scores were superior to the other 10 baseline models.

[0069] Theoretical Analysis: Tables 2 and 3 show that the AE and CRA models perform poorly among the selected benchmark models, while MCTN and TransM show improved performance compared to AE and CRA. This suggests that modality translation-based methods are more advantageous than autoencoder-based methods in handling modality loss problems. Furthermore, this invention analyzes that CRA only uses simple cascading operations in the feature fusion process, while TransM uses a parallel end-to-end translation structure to include more inter-modal correlation features in the feature fusion vector. For the ICDN model, this invention finds that when the missing rate exceeds 0.4, the M-F1 and ACC values ​​drop significantly. This is because ICDN achieves feature fusion by establishing projections between modalities; however, when a modality is missing, the mapping process cannot learn effective complementary features, and ICDN struggles to address this issue, resulting in a significant performance degradation.

[0070] Meanwhile, the experimental results in Tables 2 and 3 show that the performance of the MSA model tends to decrease as the missing value increases. However, Table 3 reveals that the proposed model, on the IEMOCAP dataset, shows improved M-F1 and ACC values ​​for STMSA when the missing value is 0.3 compared to when the missing value is 0.2. This invention analyzes STMSA by utilizing high-quality text modalities to establish a mapping relationship between visual and audio modalities to achieve feature fusion. Since the missing modalities are random, there is a possibility that the text modalities may still exist, thus enhancing the mapping process and consequently increasing the M-F1 and ACC values.

[0071] On the CMU-MOSI dataset, STMSA, compared to the benchmark models, ranks only behind the MMIN and TATE_J models in M-F1 and ACC scores when the missing value is 0%. However, STMSA achieves optimal performance at missing values ​​of 0.2 and 0.3, demonstrating its ability to handle missing modalities. This is because TATE_J, through label assistance, only enables the encoder to cover all missing cases, but lacks an effective feature fusion method to further integrate complementary features between modalities. STMSA, on the other hand, enhances the feature fusion process by establishing semantic mapping relationships between the three modalities through sequential translation. Furthermore, STMSA does not have a unique generation and completion strategy, indicating that it is a unified model architecture capable of handling missing features.

[0072]

[0073]

[0074] (4) Multiclass Validation This invention conducted four-class and seven-class classification experiments on the IEMOCAP dataset to test the performance of STMSA in multi-class classification tasks and its robustness under modality missing conditions. Table 4 shows the classification labels and the distribution information of the training and test sets in the two sets of experiments. In these two sets of experiments, the missing rate was set to 0-0.5, and TATE_C, TATE_J, MRAN, ICDN, MTMSA, and SMCMSA were selected as comparison models. The experimental results are as follows. Figure 4 and Figure 5 As shown.

[0075] Figure 4 (a) and Figure 4 (b) Shows the M-F1 and ACC values ​​of the TATE_J, MRAN, ICDN, MTMSA, and SMCMSA models in the four-class classification experiment. The experimental results show that STMSA outperforms the comparison models at missing rates ranging from 0 to 0.5. Regarding the M-F1 value, when the missing rate is 0.4, STMSA's M-F1 value is 1.16% higher than SMCMSA's; when the missing rate is 0.5, the M-F1 value is 0.76% higher than SMCMSA's. Regarding the ACC value, when the missing rates are 0.2 and 0.3, STMSA shows the largest improvement over SMCMSA compared to ICDN, increasing by 1.53% and 1.3%, respectively.

[0076] Figure 5 (a) and Figure 5 (b) Shows the M-F1 and ACC values ​​of the TATE_C, TATE_J, MRAN, ICDN, and MTMSA models in a seven-class classification experiment. Figure 5 The experimental results in (a) show that when the deletion rate is 0.1, the M-F1 value of STMSA is 4.59% higher than that of MTMSA, and when the deletion rate is 0.5, the M-F1 value of STMSA is 2.86% higher than that of MTMSA. Figure 5 The experimental results in (b) show that when the missing rate is 0, 0.1 and 0.2, the ACC value of STMSA is significantly improved compared with that of MTMSA, increasing by 2.14%, 1.77% and 2.62% respectively.

[0077] comprehensive Figure 4 and Figure 5 The experimental results show that the proposed STMSA model has advantages in multi-class classification tasks and is robust under modality loss conditions. Furthermore, multi-class classification tasks place higher demands on the feature fusion and decision modules of the MSA model. The model's performance declines with increasing classification labels. Figure 5Experimental results show that MRAN performs poorly in the seven-class classification task. This invention analyzes that MRAN only models the dependencies between different modalities through mapping methods and lacks the ability to acquire richer emotional features from multiple dimensions. In contrast, other models utilize multi-head attention mechanisms to capture potential contextual features in the sequence from multiple dimensions, which allows the fused vector to contain richer emotional features and thus adapt to multi-class classification tasks.

[0078]

[0079] (5) Ablation test To explore the performance of the STMSA model under different translation strategies and the effectiveness of each module in the model, this invention conducted modality ablation and module ablation experiments on the CMU-MOSI dataset. This invention uses V, A, and T to represent the visual modality, audio modality, and text modality, respectively. The specific experimental setup and results are as follows.

[0080] Module ablation: To verify the effectiveness of each module in STMSA, this invention generated three STMSA model variants based on the original model. Each variant removed a key functional module, and the specific model variants are as follows: (1) STMSA-Pre: STMSA directly analyzes and processes multimodal data without a pre-trained network. (2) STMSA-ST: The sequential translation module in the model is removed, and the self-attention encoding features of each modality are directly fused. (3) STMSA-CS: The common space projection module in the model is removed, and the modal features after sequential translation are directly used for output and classification. The results of the module ablation experiment are shown in Table 5.

[0081] First, for the STMSA-PreTN variant, when the missing data rate is 0%, the M-F1 and ACC values ​​are reduced by 0.48% and 0.52% respectively compared to STMSA. However, when the missing data rate is 0.3%, the M-F1 and ACC values ​​are significantly reduced by 7.13% and 10.42% respectively compared to STMSA. This is because the pre-training task uses complete modality data for training; when a modality is missing, the model learns features for each modality from the pre-training output. Based on these experimental results, the pre-training module can effectively improve STMSA's ability to handle missing modalities.

[0082] Compared to STMSA, the STMSA-ST variant exhibits lower M-F1 and ACC values. The largest decreases in M-F1 and ACC values ​​occur at a missing modality rate of 0.3, with reductions of 2.65% and 3.64%, respectively. Then, at a missing modality rate of 0.2, the M-F1 and ACC values ​​decrease by 2.45% and 3.12%, respectively. This demonstrates that even in missing modality environments, the sequential translation module can still fully mine and utilize available inter-modal features, thereby enhancing the robustness of STMSA.

[0083] For the STMSA-CS variant, when the missing rate is 0%, the M-F1 and ACC values ​​are 3.07% and 4.17% lower than STMSA, respectively. When the missing rate is 0.2%, the M-F1 and ACC values ​​are 2.43% and 4.25% lower than STMSA, respectively. When the missing rate is 0.4%, the M-F1 and ACC values ​​are 3.78% and 5.21% lower than STMSA, respectively. This demonstrates the significant contribution of the common projection module to the performance of STMSA.

[0084]

[0085] Modal ablation: To explore the impact of different modal numbers on STMSA performance during translation, and the effectiveness of sequential translation structures, this invention designs the following new sequential translation task and conducts experiments.

[0086] Task1(VtoA, AtoV): In this case, no third modality is introduced to enhance the translation process.

[0087] Task2(VtoT, AtoT): In this case, no third modality is introduced to enhance the translation process, but a text modality is used to guide the modal translation process.

[0088] Task3(VtoAtoT, AtoVtoT): In this case, a third modality is introduced to enhance the translation process by translating the mixed features of the visual and audio modalities into the text modality.

[0089] Task4(TtoVtoA, TtoAtoV): In this case, a third modality is introduced to enhance the translation process by translating the text modality into the other two low-quality modalities.

[0090] Each task was performed on the CMU-MOSI dataset, with the modality missing rate set to 0–0.5. The experimental results are shown in Table 6.

[0091] The experimental results show that the overall performance of the four tasks is lower than that of STMSA. Specifically, when the missing value rate is 0.3, the M-F1 and ACC values ​​for Task 3 are only 52.60% and 77.60%, respectively. This indicates that in a missing modality environment, different translation orders produce translation results of varying quality, and low-quality translation results affect the effectiveness of fused features, leading to reduced classification accuracy. This invention analyzes that Task 1 did not introduce high-quality text modality features into the fusion process, hence its overall performance is lower than STMSA. Although Task 2 introduces text modality to enhance the fusion process, the lack of a third modality in the translation process limits the translation effect. Task 4 translates the text modality into visual and audio modalities, and the experimental results show that its performance is lower than that of STMSA. In STMSA, the translation results establish direct interaction relationships between the text modality and both the visual and audio modalities. Since the text modality possesses high-quality emotion recognition features, STMSA would achieve the best performance with two modality pairs that directly interact with the text.

[0092]

[0093] A computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device, the aforementioned method for multimodal sentiment analysis based on sequential translation in uncertain missing modalities.

[0094] A terminal device includes a processor and a computer-readable storage medium, the processor being configured to implement various instructions; the computer-readable storage medium being configured to store multiple instructions adapted for loading and execution by the processor of the aforementioned multimodal sentiment analysis method based on sequential translation under uncertain missing modalities.

[0095] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A multimodal sentiment analysis method based on sequential translation under uncertain and missing modalities, characterized in that, include: Acquire multimodal sentiment data, including visual, textual, and auditory modalities; A sequential translation-based MSA model is constructed, which includes extracting self-attention encoding features for each modality based on multimodal sentiment data; sequentially translating the extracted features using forward and backward translation architectures respectively; obtaining missing joint features and initial joint features through feature fusion; and obtaining joint feature representations through feature learning based on Transformer encoder-decoder. Train the constructed MSA model; Sentiment analysis is performed using a trained model; Output sentiment analysis results.

2. The method of claim 1, wherein the method is characterized by, The process of extracting self-attention encoding features for each modality based on multimodal sentiment data includes feeding samples with uncertain missing modalities into a fully connected layer to obtain multimodal data of the same dimension. The input is then fed into a Transformer encoder for self-attention encoding to extract contextual features, and the encoded features for each modality are output. }, represented as: ； ； ， in , , The features of each extracted modality are then residually connected and input into a LayerNorm layer for normalization, as shown below: ； ； ， Finally, the normalized single-modal features are input into the feedforward fully connected layer for linear transformation, thereby completing the encoding of the three single-modal data, as follows: ， ， ， in, This is the weight matrix. is the bias, and ReLU is the activation function.

3. The multimodal sentiment analysis method based on sequential translation under uncertain missing modalities according to claim 2, characterized in that, The extracted features are sequentially translated using both forward and reverse translation architectures, including the use of encoded features from three modalities. The process involves two sequential translation tasks. For the forward translation process VtoTtoA, the Transformer decoder guides the encoder output to approximate the target modality. First, the encoded features of the visual modality are... Encoding features of text modality As input to the decoder; and Quary, as a multi-head attention mechanism, will Decoded as Key and Value, it is represented as: Then After performing residual connection operations, the input to the LayerNorm layer is normalized; then the normalized result is input to the feedforward fully connected layer for linear transformation to complete the Transformer decoder module and obtain... , represented as: ， ， in It is a weight matrix. is the bias, and ReLU is the activation function.

4. The method of claim 3, wherein the method is characterized by, The extracted features are sequentially translated using both forward and backward translation architectures. This also includes sequential translation between the three modalities based on the decoder output. Specifically, the text modality interacts directly with the other two modalities. The visual and audio modalities are preferentially translated to the text modality before being translated to the third modality. During the translation to the third modality, [further details are needed]. As translation features from visual modality to text modality, a transformer encoder is used to obtain translation features. Encoded output And then translate it into an audio modality, represented as: ； ； ， in It is a weight matrix. Here, ReLU is the bias, and ReLU is the activation function; finally, As a multi-head attention mechanism, Query will Using these as keys and values, the translation features from the visual modality to the text modality are made closer to the audio modality, ensuring that the extracted contextual information contains three modalities. Then, the output of the multi-head attention mechanism is used... Normalization and fully connected transformations are performed to obtain the final output of the sequential translation. , is represented as: ； ， ， in It is a weight matrix. is the bias, and ReLU is the activation function.

5. The method of claim 4, wherein, The process of obtaining missing joint features and initial joint features through feature fusion includes encoding features from the visual modality, text modality, and audio modality. The initial joint features IJFs are formed through linear projection; The decoder output of the sequential translation process { , , , The missing joint feature MJFs are formed through linear projection, and corresponding weight matrices are assigned to the IJFs and MJFs to ensure the encoding features of the three modalities { } and the decoder's output features { , , , The dynamic equilibrium learning of intermodal information during training is represented as follows: ； ； ， Where || represents the concatenation operation. , , These represent the autocorrelation common space of the translated visual modality, the text modality output by the encoder, and the translated audio modality, respectively. ； ； ， Where || represents the concatenation operation. , These represent the final outputs of the forward translation process. The final output of the reverse translation process The autocorrelation space, Decoder output representing visual and textual modalities Decoder outputs for audio and text modalities The cross-related space will eventually be , , Concatenate them into initial joint features IJFs, and , , The concatenation of missing joint feature MJFs for subsequent use is represented as follows: ； 。 6. The method of claim 5, wherein the method is characterized by, The feature learning based on the Transformer encoder-decoder to obtain the joint feature representation includes capturing the initial joint features using the Transformer encoder-decoder architecture. and missing joint features The conditional representation between them, where the encoder part will As a multi-head attention mechanism, query will As the key and value of a multi-head attention mechanism, the purpose is to allow... near , is represented as: ； ； ， in , It is a parameter matrix. , It is a learnable bias, and ReLU is the activation function. This invention allows the encoder output to... As input to the decoder, and query, key, and value are all... By making the decoder focus on the multimodal space To maintain context consistency, internal dependencies are represented as follows: ； ； ， in , It is a parameter matrix. , It uses learnable biases, ReLU as the activation function, and finally achieves deep modality fusion through decoder loss. It is used to perform sentiment classification tasks.

7. A multimodal sentiment analysis method based on sequential translation under uncertain missing modalities as described in claim 6, characterized in that, The process of training the constructed MSA model includes building a pre-trained loss function using KL divergence and JS divergence. This allows the model to generate joint features with missing information that approximate the complete joint features. The output of the transformer encoder and pre-trained output The JS divergence is used to determine the pre-trained model, which is trained using the full modality and calculated using the following equation. : ， ； The loss is calculated using the symmetric JS divergence, and the calculation equation is as follows: ； 。 8. A multimodal sentiment analysis method based on sequential translation under uncertain missing modalities as described in claim 7, characterized in that, The training of the constructed MSA model also includes constructing a translation loss function. The model is guided to learn mapping patterns in the modal translation process, and the translation process is supervised to preserve the semantic information and sentiment features of the target modality. Specifically, in sequential translation, visual and audio modalities are preferentially translated into text modalities, with a translation loss... Output from the text modality encoder and the decoder output from visual modality to text modality ( The translation loss is determined by the JS divergence between the two values. Output from the text modality encoder ( ) and the decoder output from audio modality to text modality ( The difference is determined by the JS divergence between them, and is expressed as: ； ； ； ， And in and The output is then translated to the remaining third modality, where the decoder output is computed from the visual modality to the text modality and then to the audio modality. and the encoder output of the audio modality JS divergence between them as translation loss ; Calculate the decoder output from audio modality to text modality and then to visual modality and the encoder output of the visual modality JS divergence between them as translation loss , is represented as: ； ； ； ； Finally , , , The sum of the two is as follows The equation can be expressed as: .

9. A multimodal sentiment analysis method based on sequential translation under uncertain missing modalities as described in claim 8, characterized in that, The training of the constructed MSA model also includes constructing a decoder loss. To supervise the reconstruction of joint features and achieve deep modality fusion, the output of the Transformer encoder-decoder architecture is computed. The JS divergence between them is obtained, making... become Guided by the joint representation of MJFs, calculate the equation: ； ， Then calculate the classification loss. ,Will Input a fully connected network and use Softmax as the activation function to compute the predicted values. Calculate the equation: ， in, This is the weight matrix. The bias is used to calculate the final classification loss, which is expressed as: ， Where N is the number of samples, It is the true label of the nth sample. It is a predicted label.

10. A multimodal sentiment analysis system based on sequential translation under uncertain missing modalities, characterized in that, include: The data acquisition module is configured to acquire multimodal sentiment data, including visual, textual, and auditory modalities. The model building module is configured to build an MSA model based on sequential translation, which includes extracting self-attention encoding features for each modality based on multimodal sentiment data; performing sequential translation on the extracted features using forward and backward translation architectures respectively; obtaining missing joint features and initial joint features through feature fusion; and obtaining joint feature representations through feature learning based on Transformer encoder-decoder. The model training module is configured to train the constructed MSA model. The analysis module is configured to perform sentiment analysis using a trained model. The output module is configured to output sentiment analysis results.

Images