Super multi-modal sentiment analysis method and system based on bidirectional translation interaction

Abstract

The invention relates to the technical field of multi-modal sentiment analysis, in particular to a super-multi-modal sentiment analysis method and system based on bidirectional translation interaction. The method comprises the steps that historical data samples and current data samples of super-multi-mode sentiment analysis are obtained, and the historical data samples and the current data samples comprise five modes including images, audios, texts, electroencephalogram and eye movement; carrying out weighted fusion on similar samples in the historical data and the current data sample to obtain enhanced super-multi-modal features under the condition of uncertain modal deficiency; carrying out self-attention interaction on the super-multi-modal features under the condition of uncertain modal deficiency so as to improve the quality of each modal; performing splicing fusion and self-attention interaction on the four non-electroencephalogram modals to form initial fusion four modals, and translating the electroencephalogram into new fusion four modals through a multi-head attention mechanism; according to the method, the current sample and the similar sample are subjected to weighted average fusion, so that the quality of each mode is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal sentiment analysis, and in particular to a super multimodal sentiment analysis method and system based on two-way translation interaction. Background Art

[0002] Sentiment analysis can be divided into two categories: one is based on external human characteristics, i.e., multimodal sentiment analysis (MSA), which uses the user's text, audio and image modalities to perform sentiment analysis; the other is based on internal human characteristics, i.e., sentiment analysis based on physiological modalities (EEG) and behavioral modalities (eye movement). Multimodal sentiment analysis is a study of mining user opinions and emotional states from multimodal data such as text, images and audio.

[0003] However, in practice, due to some uncontrollable environmental and human factors, multimodal data is easily lost and incompletely collected, resulting in the frequent occurrence of uncertain modal missing problems in real applications. For example, when the camera device is blocked, the user's image cannot be captured. Similar phenomena often lead to the failure of existing MSA models. Therefore, existing research has proposed multimodal sentiment analysis under uncertain modal missing based on multimodal sentiment analysis, and has achieved remarkable results.

[0004] However, most existing methods for solving the lack of uncertain modalities adopt a one-way modal transfer method, that is, using the information of one modality to complete or enhance the information of another modality, ignoring that the enhanced modality can also provide modal information in reverse. The one-way transfer method ignores the potential two-way interaction and information complementarity between modalities. And due to the limitations of one-way transfer, the model often has a strong dependence on a specific modality. When the modality is missing or of low quality, the performance of the model will significantly decrease. Summary of the invention

[0005] In order to solve the above-mentioned problems, the present invention provides a super multimodal sentiment analysis method and system based on two-way translation interaction.

[0006] In a first aspect, the present invention provides a super multimodal sentiment analysis method based on two-way translation interaction, which adopts the following technical solution: A super multimodal sentiment analysis method based on two-way translation interaction, comprising: Obtain historical and current data samples for multimodal sentiment analysis, including five modalities: image, audio, text, EEG, and eye movement; Similar samples in historical data are weightedly fused with current data samples to obtain enhanced ultra-multimodal features under the absence of uncertain modes; The ultra-multimodal features under uncertain modality loss are subjected to self-attention interaction to improve the quality of each modality; The four non-EEG modalities are spliced ​​and fused, and self-attention interaction is performed to form the initial fused four modalities, and the EEG is translated into the new fused four modalities through the multi-head attention mechanism; Replace the features in the initial fused four-modal features with the new fused four-modal features to obtain the optimized initial four-modal fused features; The optimized initial four-modal fusion features are feature translated to obtain a new EEG modality, which is then cross-attended and fused with the initial EEG modality to obtain the fused EEG modality. The fused EEG modalities are fed into the softmax function to obtain the final emotion prediction result.

[0007] Furthermore, similar samples in the historical data are weightedly fused with the current sample to obtain the enhanced ultra-multimodal features under the lack of uncertain modalities, including calculating the cosine similarity of the current sample and the historical sample within the sliding window, selecting the two samples with the highest similarity, calculating the adaptive weight of each similar sample, and weighted fusion of the current sample and the similar sample using the adaptive weight. The fusion formula is expressed as: in, is the fused sample, is the current sample.

[0008] Furthermore, the super multimodal features under the uncertain modality loss are subjected to self-attention interaction to improve the quality of each modality, including linear transformation of the super multimodal features under the uncertain modality loss, and then performing a scaled dot product operation for attention calculation, and enhancing the model's capabilities by performing attention calculation on multiple heads in parallel, and then concatenating the outputs of these heads and generating the final output through linear transformation, which is expressed as: in, Indicates Head, , and Indicates The weight matrix of Query, Key and Value, represents the weight matrix, Represents the number of attention heads.

[0009] Furthermore, the four non-EEG modalities are spliced ​​and fused, and self-attention interacts to form an initial fused four modalities, and the EEG is translated into a new fused four modalities through a multi-head attention mechanism, including using a multi-head attention mechanism and a feedforward neural network to obtain feature representations of the five modalities, splicing and fusing the four non-EEG modalities and self-attention interacts to form an initial fused four modalities, and translating the EEG modality into a new fused four modalities through multi-head attention; initial fused four modalities It is expressed as: , New fusion quad-modality It is expressed as: ; Among them, the EEG feature is e and the four-modal fusion features are v, a, t, and p respectively; the feature representations of the five modalities are recorded as , , , and .

[0010] Further, the features in the initial fused quadrimodal features are replaced with the new fused quadrimodal features to obtain the optimized initial quadrimodal fusion features, including performing optimal feature selection on the initial fused quadrimodal and the new fused quadrimodal to form the optimized initial quadrimodal fusion features, wherein the feature dimensions of the initial fused quadrimodal and the new fused quadrimodal are aligned, and the optimal feature selection is performed by comparing the absolute value sum of the features of the initial fused quadrimodal and the new fused quadrimodal, and the formula for calculating the absolute value sum is expressed as: Compare absolute sums: Finally, select features to complete optimal feature selection: in, is the optimal feature.

[0011] Furthermore, the optimized initial four-modal fusion features are feature translated to obtain a new EEG modality, and cross-attention fused with the initial EEG modality to obtain a fused EEG modality, including fusing the new EEG modality with the initial EEG modality in the form of EEG to eliminate fusion interference caused by modality differences, which is expressed as: Among them, the fused EEG modality .

[0012] Furthermore, the fused EEG modality is transmitted to the softmax function to obtain the final emotion prediction result, including defining classification loss and regularization loss based on the softmax function, wherein the classification loss of each sample is calculated using the Softmax cross entropy loss function, and the losses of all samples are summed to obtain the total classification loss; the total classification loss and the introduced L2 regularization loss are weighted and summed for model optimization.

[0013] In the second aspect, a super multimodal sentiment analysis system based on two-way translation interaction includes: The data acquisition module is configured to acquire historical data samples and current data samples for ultra-multimodal sentiment analysis, including five modalities: image, audio, text, EEG, and eye movement; The adaptive prior enhancement module is configured to perform weighted fusion of similar samples in historical data with current data samples to obtain enhanced ultra-multimodal features under the absence of uncertain modalities; The attention mechanism module is configured to perform self-attention interaction on the multi-modal features under uncertain modality loss to improve the quality of each modality; The two-way translation interaction module is configured to concatenate and fuse the four non-EEG modalities and perform self-attention interaction to form an initial fused four-modality, and translate the EEG into a new fused four-modality through a multi-head attention mechanism; The optimal feature selection module is configured to replace the features in the initial fused four-modal features with the new fused four-modal features to obtain the optimized initial four-modal fused features; The fusion module is configured to perform feature translation on the optimized initial four-modal fusion features to obtain a new EEG modality, and perform cross-attention fusion with the initial EEG modality to obtain a fused EEG modality; The prediction module is configured to feed the fused EEG modality to the softmax function to obtain the final emotion prediction result.

[0014] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, for example, a super-multimodal sentiment analysis method based on bidirectional translation interaction.

[0015] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the ultra-multimodal sentiment analysis method based on bidirectional translation interaction.

[0016] In summary, the present invention has the following beneficial technical effects: (1) The present invention proposes an adaptive prior enhancement method. The method is particularly suitable for the intermediate stage between data loading and model input in the training phase. Specifically, the cosine similarity between the current sample and the historical samples is calculated, the historical similar samples with high similarity are selected, and a weight is assigned to each similar sample based on the cosine similarity. Finally, the current sample and the similar samples are weighted averaged to enhance the quality of each modality. In addition, the method only needs to retain part of the user's historical records and does not introduce additional samples, which is more suitable for practical application scenarios.

[0017] (2) The present invention proposes a two-way translation interaction method. Given that the accuracy of EEG in single-modal sentiment analysis is much higher than that of the other four modalities, this method fuses the four non-EEG modalities to form a fused four-modality, then translates the EEG into the fused four-modality, and replaces the poor quality features in the initial fused four-modality through the optimal feature selection module to improve the quality of the fused four-modality and complete the missing information. Afterwards, the optimized four-modal fusion features are translated into EEG modality, and cross-attention fused with the initial EEG modality to complete the missing EEG and improve its quality. In this way, the modal information is fully utilized and the missing modality is completed through a two-way interactive method.

[0018] (3) The present invention proposes an optimal feature selection module, which uses the absolute value of the feature and the absolute value of the feature as the criterion for selecting the two features, so as to select features containing more information to recompose the four-modal fusion features, thereby using EEG to complete the optimization of the four-modal fusion features, while also avoiding the interference with the optimization of the four-modal fusion features caused by the lack of initial EEG features. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic diagram of a super multimodal sentiment analysis method based on bidirectional translation interaction according to Embodiment 1 of the present invention; Figure 2 is a schematic diagram of the ultra-multimodal sentiment analysis model BTUMSA of Example 1 of the present invention; Figure 3 is a schematic diagram of adaptive priori enhancement according to Embodiment 1 of the present invention; Figure 4 4 is a graph showing the experimental results of four types of sentiment classification according to Example 1 of the present invention; Figure 5 4 is a graph showing the experimental results of seven categories of sentiment classification according to Example 1 of the present invention. DETAILED DESCRIPTION

[0020] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0021] Example 1 Reference Figure 1 , a super multimodal sentiment analysis method based on two-way translation interaction in this embodiment includes: Obtain historical and current data samples for multimodal sentiment analysis, including five modalities: image, audio, text, EEG, and eye movement; Similar samples in historical data are weightedly fused with current data samples to obtain enhanced ultra-multimodal features under the absence of uncertain modes; The ultra-multimodal features under uncertain modality loss are subjected to self-attention interaction to improve the quality of each modality; The four non-EEG modalities are spliced ​​and fused, and self-attention interaction is performed to form the initial fused four modalities, and the EEG is translated into the new fused four modalities through the multi-head attention mechanism; Replace the features in the initial fused four-modal features with the new fused four-modal features to obtain the optimized initial four-modal fused features; The optimized initial four-modal fusion features are feature translated to obtain a new EEG modality, which is then cross-attended and fused with the initial EEG modality to obtain the fused EEG modality. The fused EEG modalities are fed into the softmax function to obtain the final emotion prediction result.

[0022] Among them, in order to give full play to the interaction between the five modalities, this embodiment proposes the concept of super multimodality. By combining EEG and eye movement modalities with multimodality, super multimodality is formed, which is used as a model to provide more modal information from different angles in an uncertain modality missing environment, thereby improving the accuracy of sentiment analysis. This is the first time that the problem of super multimodal sentiment analysis under uncertain modality missing has been clearly proposed in academic literature, and we provide a special data set and model for this purpose.

[0023] In order to solve the problem of insufficient information utilization caused by one-way transmission of modalities, this embodiment proposes a two-way translation interaction method. In view of the fact that the accuracy of EEG in unimodal sentiment analysis is much higher than that of the other four modalities, this method fuses the four non-EEG modalities to form a fused four-modality, and then translates the EEG into a fused four-modality, and replaces the poor quality features in the initial fused four-modality through the optimal feature selection module to improve the quality of the fused four-modality and complete the missing information. Afterwards, the optimized four-modal fusion features are translated into EEG modalities, and cross-attention fusion is performed with the initial EEG modality to complete the missing EEG and improve the quality. In this way, the modal information is fully utilized and the missing modality is completed through a two-way interactive method.

[0024] In order to better use the EEG modality to optimize the four-modal fusion features, this embodiment proposes an optimal feature selection module, which uses the absolute value of the feature as the criterion for selecting the two features, so as to select features containing more information to recompose the four-modal fusion features, thereby using EEG to complete the optimization of the four-modal fusion features, while also avoiding interference with the optimization of the four-modal fusion features caused by the lack of initial EEG features.

[0025] In order to improve the super multimodal quality of the initial input, this embodiment proposes an adaptive prior enhancement method. This method aims to improve the initial super multimodal quality by utilizing similar samples in historical data. Specifically, we select samples with high similarity by calculating the cosine similarity between the current sample and the historical samples, and assign weights to each sample according to the cosine similarity. Afterwards, the current sample is weightedly fused with similar samples according to the weight to improve the initial five-modal quality. In addition, this method only needs to retain part of the user's historical records, without introducing additional samples, which is more in line with the needs of actual application scenarios.

[0026] Specifically: S1. Assume that the ultra-multimodal data for sentiment analysis contains five modalities: ,in , , , and Represent image, audio, text, EEG and eye movement modalities respectively. Without loss of generality, this paper uses to represent any missing modes, where For example, when the image modality does not exist, the super multimodal feature can be expressed as The problem studied in this paper can be defined as user sentiment analysis based on ultra-multimodal data with uncertain modality missing. For the convenience of representation, in the following sections, this paper uses to represent ultra-multimodal data with uncertain missing modalities.

[0027] S2. In the context of the rapid development and maturity of technologies such as smart wearables, the metaverse, and the Internet of Things, it is easy to obtain ultra-multimodal data related to user emotions (images, text, audio, physiological modalities (EEG), behavioral modalities (eye movements)). Ultra-multimodal data can better complement information, which helps to improve the accuracy of sentiment analysis. However, there is currently no research on sentiment analysis based on ultra-multimodality (images, text, audio, EEG, eye movements) in the absence of uncertain modalities. To address this issue, this paper proposes a ultra-multimodal sentiment analysis model based on bidirectional translation interaction in the absence of uncertain modalities (BTUMSA), such as Figure 2As shown in the figure. First, in order to improve the quality of the super-multimodal features of the initial input, this paper proposes an adaptive prior enhancement module that is only applied in the model training stage, that is, the similar samples in the historical data are weightedly fused with the current samples to obtain the super-multimodal features under the lack of uncertain modalities. Then, the five modalities with uncertain modal missing are subjected to self-attention interaction to improve the quality of each modality. Then, the four non-EEG modalities are spliced ​​and fused and self-attention interacted to form a fused quad-modality, and the EEG is translated into the fused quad-modality through multi-head attention. Then, through the optimal feature selection module proposed in this paper, the new fused quad-modal features are used to replace the features with poor quality in the initial fused quad-modality, thereby improving the overall quality of the initial fused quad-modality and completing the missing information. After that, the optimized initial quad-modal fusion features are translated into EEG modality, and cross-attention fusion is performed with the initial EEG modality to complete the missing EEG and improve the quality. Finally, the fused EEG modality is fed to the softmax function to obtain the final emotion prediction result.

[0028] S3. In BTUMSA, the Multi-Head Attention (MHA) mechanism is a key module responsible for mining the potential associations between different modalities (text, image, EEG, audio, eye movement) and performing deep feature interactions. This mechanism can capture global modal interaction features and enhance the model's expressiveness by parallelizing multiple groups of attention calculations. The core principles and calculation formulas of the multi-head attention mechanism can be summarized as follows.

[0029] Assume the input matrix is , using 3 different parameter matrices , and For the input matrix Perform linear transformation and define Query as , Keys are ,Values ​​are , we get the Q (query), K (key), and V (value) matrices, and then perform the scaled dot product operation to calculate the attention. The specific process is shown in formula (1): (1) in, , and is the weight matrix.

[0030] The multi-head attention mechanism first enhances the model's capabilities by performing attention calculations on multiple heads in parallel, then concatenates the outputs of these heads and generates the final output through linear transformation, as shown in formulas (2) and (3).

[0031] (2) (3) in, Indicates Head, , and Indicates A weight matrix of Query, Key and Value. represents the weight matrix, Represents the number of attention heads.

[0032] S4. Adaptive prior reinforcement, like Figure 3 As shown in the figure, the adaptive prior enhancement module is a technology proposed in this paper for selecting and fusing similar samples, which aims to improve the quality of current training samples by utilizing similar samples in historical data. Specifically, this module calculates the cosine similarity between the current sample and the historical sample, selects samples with higher similarity, and performs weighted fusion based on the similarity of these similar samples to generate higher quality training samples. In addition, this method only needs to retain part of the user's historical records, without introducing additional samples, which is more in line with the needs of actual application scenarios. The specific process is as follows: First, considering that the memory usage is limited in actual application scenarios and it is difficult to store all user history records, this article introduces the idea of ​​sliding windows. The sliding window selects samples similar to the current data by selecting historical data within a window range of a fixed size on the timeline. Specifically, we use the current training step as a reference point and set a value of its specific size as the starting and ending position of the historical window. In this way, the sliding window can not only prevent resource exhaustion caused by excessive memory usage (i.e., memory explosion), but also ensure that the selected samples are continuous and relevant in time, so as to be closer to actual applications and select the best similar samples.

[0033] Next, within the sliding window, the cosine similarity between the current sample and the historical sample is calculated, and the two samples with the highest similarity are selected. The formula for calculating cosine similarity is as follows: Among them, a and b represent the current sample vector and the historical sample vector, represents the dot product of vectors, ||a|| and ||b|| represent the norms of a and b respectively.

[0034] Considering that there may be duplicate samples, we first screen out historical samples with a similarity between 0.90 and 0.95, and then select the two historical samples with the highest similarity as similar samples.

[0035] For the similar samples that have been screened, the adaptive weight of each similar sample is calculated. The weight calculation formula is as follows: in is the cosine similarity between the i-th similar sample and the current sample.

[0036] Next, the adaptive weights are used to perform weighted fusion on the current sample and the selected similar samples. The fusion formula is as follows: in, is the fused sample, is the current sample.

[0037] S5. Bidirectional translation interaction, Given that the accuracy of EEG in single-modal sentiment analysis is much higher than that of the other four modalities, we propose to divide the five modalities into two categories, EEG features (e) and four-modal fusion features (v, a, t, p), and then translate the EEG features into four-modal fusion features and perform optimal feature selection to improve the quality of the four-modal fusion features and complete the missing four-modal features. Then, the optimized four-modal fusion features are translated into EEG form and cross-fused with the original EEG features to supplement the initial EEG features and solve the problem of missing initial EEG features. The missing modalities are completed and the quality of each modality is improved through two-way translation interaction. The specific process is as follows: First, we use the multi-head attention mechanism to encode text, image, eye movement, audio, and EEG modalities respectively to obtain the feature representation of each modality. Input features , and its multi-head attention mechanism processing is expressed as: in, Represents the multi-head attention mechanism. In order to further capture the nonlinear relationship within each modality, we use a feedforward neural network (FFN) to perform deep feature extraction based on the output of the multi-head attention: After this step, we obtain the feature representations of five modes, which are recorded as , , , and .

[0038] Then we divide the five modalities into two categories, EEG and non-EEG, and concatenate and fuse the four non-EEG modalities and perform self-attention interaction to form a four-modal fusion feature. : Then, the EEG modality is translated into four-modal fusion features through multi-head attention : The multi-head attention parameters for modal translation are set as follows: num_heads=4, which enables the model to focus more on key features during translation, reduce the risk of overfitting, and improve translation stability. dropout_rate=0.2, which is used to prevent overfitting while maintaining a certain stability of information transmission.

[0039] Then, the initial four-modal fusion features are combined with the four-modal fusion features obtained by EEG translation to form the optimal feature selection (introduced in the next module) , so as to use EEG modality to improve the quality of four-modal fusion features and complete the missing ones. Translation to EEG modalities via multi-head attention : In order to supplement the information of EEG features and complete the missing information (if the EEG features are missing), we propose to cross-fuse the two EEG features and fuse them in the form of EEG to eliminate the fusion interference caused by modality differences.

[0040] The cross-fusion multi-head attention parameters are as follows: num_heads=30, so that the information of different modalities can be captured and integrated from multiple angles, improving the representation ability and performance of the model. Dropout_rate=0, so as to retain as much information as possible during the fusion process, ensuring that the features of different modalities can fully interact.

[0041] S6. Optimal feature selection, In order to better use EEG modality to optimize the four-modal fusion features, we propose an optimal feature selection module, using the absolute value of the features as the standard for selecting the two features, so as to select more informative features to recompose the four-modal fusion features, thereby using EEG to complete the optimization of the four-modal fusion features, while also avoiding interference in the optimization of the four-modal fusion features due to the lack of initial EEG features. The specific process is as follows: First, in order to make the features of different modalities comparable on the same dimension, the features of the new modality (the four-modal fusion features after EEG translation) need to be adjusted to ensure that their dimensions are consistent with those of the original modality (the original four-modal fusion features).

[0042] If the final feature dimensions of the two are inconsistent, we perform the following operations to make them the same: First, obtain the original modal features The shape is , new modal features Shape , where B is the batch size, and are the sequence lengths of the original modality and the new modality, respectively. and are the feature dimensions of the original modality and the new modality respectively. If yes If it is an integer multiple of The feature dimensions are as follows: if no If it is an integer multiple of The feature dimensions are as follows: After the feature dimensions are aligned, the feature absolute value sum of the original modal feature O and the new modal feature N is compared to select the feature with higher information content. The larger the feature absolute value sum, the greater the contribution of this feature in this dimension, that is, the greater the amount of emotional information, so the feature absolute value sum can be used as the criterion for optimal feature selection. By comparing the size of the absolute value sum, it is determined whether the feature at each position retains the feature of the original modality or is replaced by the feature of the new modality. The formula for calculating the absolute value sum is as follows: Compare absolute sums:

[0043] Finally, select features to complete optimal feature selection: S7. Training objectives, The total loss of the BTUMSA model proposed in this example is composed of the classification loss ( ) and L2 regularization loss ( ). Next we will introduce the two parts in detail.

[0044] (1) Classification loss: Classification loss is the main goal of model optimization, which is used to measure the difference between the category predicted by the model and the true label. In this study, we define classification loss based on the Softmax Cross-Entropy Loss function. The specific steps are as follows: For a given input sample, the model first generates the logits (unnormalized probability prediction values) of its corresponding classification, represented as a vector , where C is the number of categories and each element is the predicted value of category i. The label of the sample is represented by one-hot encoding, denoted as .

[0045] The Softmax cross entropy loss function is used to calculate the classification loss of each sample. The formula is as follows: In order to obtain more robust prediction results, we applied the above Softmax cross entropy loss function to the model's logits and further summed the losses of all samples to obtain the total classification loss The calculation formula is as follows: Where N is the number of samples in a batch, and Represent the logits and labels of the nth sample respectively.

[0046] (2) L2 regularization loss: To prevent the model from overfitting, we introduced L2 regularization loss. L2 regularization penalizes the sum of the squares of the model weights, making the model smoother and reducing the risk of overfitting. The specific formula is as follows: Among them, W represents the set of all trainable parameters in the model, and λ is the regularization coefficient, which is used to control the weight of the regularization term in the total loss.

[0047] (3) Total loss function: Total loss function It is the weighted sum of classification loss and L2 regularization loss, which is used to optimize the model during training. The total loss function is defined as follows: .

[0048] Experimental verification: In order to verify the performance of the proposed BTUMSA model, this paper conducts a large number of experiments on two self-built datasets (IEMOPRO and MELDPRO). In the following sections, this paper first introduces the process of self-built datasets and data preprocessing. Then, this paper introduces the experimental settings and baseline models. Finally, the experimental results are given and analyzed.

[0049] In order to conduct research on super-multimodal sentiment analysis, this paper proposes to fuse the EEG and eye movement dataset (SEED-IV) with the multimodal sentiment dataset (IEMOCAP and MELD) to construct two new super-multimodal datasets IEMOPRO and MELDPRO. Specifically, for each data in IEMOCAP and MELD, we randomly select a data with the same or similar sentiment label from SEED-IV, and splice all modal information of these two data (including EEG, eye movement, text, audio and image) together to form a new super-multimodal data entry. Through this method, we splice all entries in IEMOCAP and MELD with the corresponding data in SEED-IV, and finally construct two new datasets IEMOPRO and MELDPRO. The self-built dataset not only retains the multimodal features of the original dataset, but also incorporates EEG and eye movement data, providing comprehensive and rich data support for super-multimodal sentiment analysis research, and can effectively verify the effectiveness of the BTUMSA model proposed in this paper. The following section will explain the relevant datasets and their feature extraction process in detail.

[0050] In order to prove the effectiveness of the BTUMSA model proposed in this paper, this paper compares 6 cutting-edge multimodal baseline models. These baseline models can achieve ultra-multimodal comparison after modal expansion.

[0051] In order to test the multi-classification performance of the BTUMSA model, this paper selects the emotion labels of happiness, anger, sadness and neutral to conduct a four-classification experiment on the IEMOPRO dataset. The emotion labels of happiness, anger, sadness, neutral, frustration, excitement and surprise are selected to conduct a seven-classification experiment on the IEMOPROP dataset. The detailed distribution of multi-class labels of the IEMOPRO dataset is shown in Table 1. Here, this paper selects the models TATE_C, TATE_J, MTMSA, SMCMSA and TransM as baseline models. The experimental results are shown in Figure 4 and Figure 5 As shown, the red bar graph represents the performance of the case proposed in this paper.

[0052] Table 1 Detailed distribution of IEMOPRO dataset The experimental results of four types of sentiment classification are as follows Figure 4 As shown. Figure 4 (a) and Figure 4 It can be found in (b) that the BTUMSA model proposed in this paper outperforms the other five baseline models in both evaluation indicators (ACC and M-F1) when the missing rate is 0.1. Figure 4 (c) and Figure 4 As shown in (d), the BTUMSA model proposed in this paper outperforms the other five baseline models in both evaluation indicators (ACC and M-F1) when the missing rate is 0.3.

[0053] The experimental results of seven types of sentiment classification are as follows Figure 5 As shown. Figure 5 (a) and Figure 5 It can be found in (b) that the BTUMSA model proposed in this paper outperforms the other five baseline models in both evaluation indicators (ACC and M-F1) when the missing rate is 0.1. Figure 5 (c) and Figure 5 As shown in (d), the BTUMSA model proposed in this paper outperforms the other five baseline models in both evaluation indicators (ACC and M-F1) when the missing rate is 0.3.

[0054] Based on the above experimental results, it can be concluded that BTUMSA has better performance in sentiment multi-classification.

[0055] Figure 4 and Figure 5 The results show that in the four-class sentiment analysis and seven-class sentiment analysis tasks, the proposed model BTUMSA always outperforms the other five baseline models in F1 and ACC. In summary, according to the above experimental results, the proposed model BTUMSA has the best performance in the multi-class sentiment classification task with uncertain modality loss.

[0056] Example 2 This embodiment provides a super multimodal sentiment analysis system based on two-way translation interaction, including: The data acquisition module is configured to acquire historical data samples and current data samples for ultra-multimodal sentiment analysis, including five modalities: image, audio, text, EEG, and eye movement; The adaptive prior enhancement module is configured to perform weighted fusion of similar samples in historical data with current data samples to obtain enhanced ultra-multimodal features under the absence of uncertain modalities; The attention mechanism module is configured to perform self-attention interaction on the multi-modal features under uncertain modality loss to improve the quality of each modality; The two-way translation interaction module is configured to concatenate and fuse the four non-EEG modalities and perform self-attention interaction to form an initial fused four-modality, and translate the EEG into a new fused four-modality through a multi-head attention mechanism; The optimal feature selection module is configured to replace the features in the initial fused four-modal features with the new fused four-modal features to obtain the optimized initial four-modal fused features; The fusion module is configured to perform feature translation on the optimized initial four-modal fusion features to obtain a new EEG modality, and perform cross-attention fusion with the initial EEG modality to obtain a fused EEG modality; The prediction module is configured to feed the fused EEG modality to the softmax function to obtain the final emotion prediction result.

[0057] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a super-multimodal sentiment analysis method based on bidirectional translation interaction.

[0058] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; and the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the ultra-multimodal sentiment analysis method based on two-way translation interaction.

[0059] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A super multimodal sentiment analysis method based on two-way translation interaction, characterized in that: include: Obtain historical and current data samples for multimodal sentiment analysis, including five modalities: image, audio, text, EEG, and eye movement; Similar samples in historical data are weightedly fused with current data samples to obtain enhanced ultra-multimodal features under the absence of uncertain modes; The ultra-multimodal features under uncertain modality loss are subjected to self-attention interaction to improve the quality of each modality; The four non-EEG modalities are spliced ​​and fused, and self-attention interaction is performed to form the initial fused four modalities, and the EEG is translated into the new fused four modalities through the multi-head attention mechanism; Replace the features in the initial fused four-modal features with the new fused four-modal features to obtain the optimized initial four-modal fused features; The optimized initial four-modal fusion features are feature translated to obtain a new EEG modality, which is then cross-attended and fused with the initial EEG modality to obtain the fused EEG modality. The fused EEG modalities are fed into the softmax function to obtain the final emotion prediction result.

2. According to claim 1, a super multimodal sentiment analysis method based on two-way translation interaction is characterized in that: The similar samples in the historical data are weightedly fused with the current sample to obtain the enhanced ultra-multimodal features under the uncertain modality loss, including calculating the cosine similarity of the current sample and the historical sample within the sliding window, selecting the two samples with the highest similarity, calculating the adaptive weight of each similar sample, and using the adaptive weight to weightedly fuse the current sample and the similar sample. The fusion formula is expressed as: in, is the fused sample, is the current sample.

3. The method for super multimodal sentiment analysis based on two-way translation interaction according to claim 2, characterized in that: The super multimodal features under the uncertain modality loss are subjected to self-attention interaction to improve the quality of each modality, including linear transformation of the super multimodal features under the uncertain modality loss, and then performing a scaled dot product operation for attention calculation, and enhancing the model's capabilities by performing attention calculation on multiple heads in parallel, and then concatenating the outputs of these heads and generating the final output through linear transformation, which is expressed as: in, Indicates Head, , and Indicates The weight matrix of Query, Key and Value, represents the weight matrix, Represents the number of attention heads.

4. The method for super multimodal sentiment analysis based on two-way translation interaction according to claim 3 is characterized in that: The four non-EEG modalities are spliced ​​and fused, and self-attention interacts to form an initial fused four-modality, and the EEG is translated into a new fused four-modality through a multi-head attention mechanism, including using a multi-head attention mechanism and a feedforward neural network to obtain feature representations of the five modalities, splicing and fusing the four non-EEG modalities and self-attention interacts to form an initial fused four-modality, and translating the EEG modality into a new fused four-modality through multi-head attention; initial fused four-modality It is expressed as: ， New fusion quad-modality It is expressed as: ； Among them, the EEG feature is e and the four-modal fusion features are v, a, t, and p respectively; the feature representations of the five modalities are recorded as , , , and .

5. The method for super multimodal sentiment analysis based on two-way translation interaction according to claim 4 is characterized in that: The method of replacing the features in the initial fused quadruple modality with the new fused quadruple modality features to obtain the optimized initial quadruple modality fusion features includes performing optimal feature selection on the initial fused quadruple modality and the new fused quadruple modality and fusing them to form the optimized initial quadruple modality fusion features, wherein the feature dimensions of the initial fused quadruple modality and the new fused quadruple modality are aligned, and the optimal feature selection is performed by comparing the absolute value sum of the features of the initial fused quadruple modality and the new fused quadruple modality. The formula for calculating the absolute value sum is expressed as: Compare absolute sums: Finally, select features to complete optimal feature selection: in, is the optimal feature.

6. The method for super multimodal sentiment analysis based on two-way translation interaction according to claim 5, characterized in that: The optimized initial four-modal fusion features are feature translated to obtain a new EEG modality, and cross-attention fused with the initial EEG modality to obtain a fused EEG modality, including fusing the new EEG modality with the initial EEG modality in the form of EEG to eliminate fusion interference caused by modality differences, which is expressed as: Among them, the fused EEG modality .

7. The method for super multimodal sentiment analysis based on two-way translation interaction according to claim 6 is characterized in that: The fused EEG modality is transmitted to the softmax function to obtain the final emotion prediction result, including defining the classification loss and regularization loss based on the softmax function, wherein the classification loss of each sample is calculated using the Softmax cross entropy loss function, and the losses of all samples are summed to obtain the total classification loss; the total classification loss and the introduced L2 regularization loss are weighted and summed for model optimization.

8. A super multimodal sentiment analysis system based on two-way translation interaction, characterized in that: include: The data acquisition module is configured to acquire historical data samples and current data samples for ultra-multimodal sentiment analysis, including five modalities: image, audio, text, EEG, and eye movement; The adaptive prior enhancement module is configured to perform weighted fusion of similar samples in historical data with current data samples to obtain enhanced ultra-multimodal features under the absence of uncertain modalities; The attention mechanism module is configured to perform self-attention interaction on the multi-modal features under uncertain modality loss to improve the quality of each modality; The two-way translation interaction module is configured to concatenate and fuse the four non-EEG modalities and perform self-attention interaction to form an initial fused four-modality, and translate the EEG into a new fused four-modality through a multi-head attention mechanism; The optimal feature selection module is configured to replace the features in the initial fused four-modal features with the new fused four-modal features to obtain the optimized initial four-modal fused features; The fusion module is configured to perform feature translation on the optimized initial four-modal fusion features to obtain a new EEG modality, and perform cross-attention fusion with the initial EEG modality to obtain a fused EEG modality; The prediction module is configured to feed the fused EEG modality to the softmax function to obtain the final emotion prediction result.

9. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the method according to claim 1 .

10. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the method as claimed in claim 1 .

Images