Differential and missing robust modeling method for multi-modal sentiment analysis

Abstract

The invention provides a difference and deletion robust modeling method for multi-modal sentiment analysis, belongs to the field of multi-modal sentiment analysis, and aims to improve the feature fusion capability among different modals (including texts, audios and videos) and the robustness of a system under the condition of modal deletion. According to the method, high-quality features are obtained from original multi-modal data through the feature extraction module, and it is ensured that key information is reserved; then designing a modal alignment module (MAAT), realizing semantic alignment of a non-text modal and a text modal, and relieving distribution difference between modals; a modal decoupling and information maximization strategy is further introduced, and the modeling ability of the model for fine-grained emotion features is enhanced; and finally, constructing a difference feature completion module so as to effectively cope with a mode missing situation. According to the method, a large number of comparison and ablation experiment verification are carried out on CMU-MOSI, CMU-MOSEI and CH-SMIS data sets, and the result shows that the method is superior to an existing mainstream multi-modal sentiment analysis method in the aspects of accuracy, robustness and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal sentiment analysis, and in particular to a multi-modal sentiment analysis difference and loss robust modeling method. BACKGROUND

[0002] With the rapid development of social media, people's emotional expression has expanded from traditional single text to multi-modal data including speech, images and videos, which can more intuitively convey opinions and emotions. Therefore, multi-modal sentiment analysis (MSA) has gradually become a research hotspot and has shown wide application value in product evaluation, public opinion monitoring, intelligent customer service and other fields. Compared with single-modal methods, MSA can more accurately identify potential emotions by fusing text, audio and visual information. For example, only relying on text "You are really a good person" may be judged as positive speech, but if combined with facial expressions and tone features, it may be a sarcastic negative expression. The typical process includes feature extraction, modality fusion and sentiment classification, and the modeling quality directly affects the system performance.

[0003] Although various fusion strategies and model improvements have been proposed in existing research, there are still the following technical difficulties in the prior art: first, there are significant distribution differences between different modalities, making it difficult to achieve effective alignment and interaction. Although existing alignment methods have some effect, the heterogeneity problem is still serious, affecting the cross-modal fusion effect.

[0004] Second, even if the method of dividing modalities into shared and specific features is used, or the information maximization technique is used to enhance the feature expression ability, it is difficult to deeply mine the fine-grained sentiment information between modalities.

[0005] Third, due to network delays, device failures and other situations in real environments, modal information is missing, further affecting the accuracy of sentiment recognition. Existing methods use modal reconstruction mechanisms (such as encoder-decoder methods) to complete the missing modalities, but this method often reconstructs a large amount of detail information that is irrelevant to emotions, and the generated features are not verified by emotions, which has a large practical limit.

[0006] In summary, the current technology urgently needs a new multi-modal sentiment analysis method that can eliminate modality distribution differences, deeply mine fine-grained sentiment features, and maintain recognition performance in the case of missing modalities. SUMMARY

[0007] To solve the above technical problems, the present application proposes a multi-modal sentiment analysis difference and loss robust modeling method to eliminate modality distribution differences, deeply mine fine-grained sentiment features, and maintain recognition performance in the case of missing modalities.

[0008] The technical scheme adopted by the present application is: a difference and loss robust modeling method for multi-modal sentiment analysis, comprising the following steps: S1, obtaining an emotion information modality to be processed, performing feature extraction on the emotion information modality to be processed to obtain original modality data, the original modality data comprising a text modality X t , an audio modality X a , and a video modality X v ; S2, inputting the data-extracted emotion information modality into a trained multi-modal sentiment analysis model to output a sentiment prediction result of the emotion information modality to be processed; the multi-modal sentiment analysis model comprises a feature extraction module, an MAAT module, an information maximization analysis module, and a specificity completion module; The training steps of the multi-modal sentiment analysis model are as follows: S201, the multi-modal sentiment analysis model inputs the original modality data through the input of the feature extraction module, outputs modality features with a text modality, an audio modality, and a video modality, and inputs them into the MAAT module; S202, the MAAT module converts the audio and visual distribution through the adversarial cyclic translation network to reduce the modality difference between the text and the non-text, outputs the distribution-aligned modality features, and inputs them into the information maximization analysis module; S203, the information maximization module decouples the input modality features to obtain modality-specific information and invariant features, estimates the invariant features and the specific information using the information maximization mechanism, outputs the information maximization-processed modality features, and inputs them into the specificity completion module; S204, the specificity completion module predicts the specificity representation of the missing modality through the conditional generation network, and improves the generation quality and discriminability by means of the adversarial learning mechanism, thereby realizing modality completion at the feature level; S205, the model parameters of the multi-modal sentiment analysis model are optimized and updated to minimize the total loss function constructed by the model basic loss function and the cross-modal contrast learning loss function, thereby completing the training of the multi-modal sentiment analysis model; S206, repeat steps S201 to S205 for iterative training until the multi-modal sentiment analysis model converges or reaches a preset number of iterations.

[0009] As a preferred scheme, the data extraction specifically comprises: using a model to process the original data to obtain the text modality, the audio modality, and the video modality after feature extraction, and the expression is as follows: Where X represents the original feature vector, θ is the model parameter, and t, a, and v represent text, audio, and video, respectively.

[0010] As a preferred embodiment, in step S202, the MAAT module obtains a uniformly distributed modal representation through distribution transformation, as shown in the following formula: in and This represents the distributional transformation process between visual-text and audio-text; the transformation process is subject to a joint constraint of loss. The converted modal representation is as follows: Represents the transformation from non-text to text distribution, where .

[0011] Furthermore, in step S202, the joint loss Including generator loss Discriminator loss and cyclic loss The constraint formula is as follows: In the formula, This represents a non-text to text distribution transformation. This represents the transformation from text to non-text, where... , and This indicates whether the input is a text representation. The weights represent the cyclic loss.

[0012] As a preferred embodiment, in step S203, the information maximization analysis module estimates specific features and invariant features through modal decoupling and mutual information maximization. The loss function includes the lower bounds of mutual information (MI) and conditional mutual information (CMI); the formula for estimating mutual information is... The corresponding loss function is ; The formula for estimating conditional mutual information is: The corresponding loss function is: ; In the formula, W, Z and Q represent different specific features, f() represents the training function, and p(w, z) represents the joint probability distribution of random variables w and z; Let p(w, z) represent the expectation under the joint distribution p(w, z).

[0013] As a preferred embodiment, the estimation process uses an information maximization analysis module to estimate mode-invariant and mode-specific features of the modal features processed by the MAAT module; wherein the mode-invariant feature h inv Characterizing shared information between individual modes and other modes, mode-specific features h spe This includes acoustic-specific features, visual-specific features, and text-specific features comprised of both, constructed using a modality-invariant loss L... inv Modal-specific loss L spe The total loss function L composed of weights mod To optimize and reconcile the modality-specific feature h by minimizing the total loss function. spe With invariant feature h inv .

[0014] Furthermore, the modally invariant loss L inv Modal-specific loss L spe The total loss function L composed of weights mod The constraint formulas are as follows: ; In the formula, , and This represents the corresponding weight.

[0015] As a preferred embodiment, in step S204, the specific completion module uses a conditional generation network. Predicting specific representations of missing modes based on specific features of existing modes Using a conditional generative adversarial network (GAN) mechanism, a discriminator distinguishes between generated and real features. Adversarial loss and reconstruction loss jointly constrain the generator training, thereby achieving high-quality modality completion at the feature level. This process is also influenced by the generator loss function L. gd The constraints are as follows: After the completed features are aggregated with the existing modal features, they are input into a four-layer perceptron for sentiment prediction. The final calculation formula is as follows: By minimizing the mean square error function L between the predicted value and the true value task To optimize the model, the task loss function is: , where n represents the number of samples, This represents the prediction error for the i-th sample.

[0016] As a preferred embodiment, step S205 specifically involves inputting the pre-extracted emotional information modalities into the multimodal sentiment analysis model, and constructing a model task loss function L. task MAAT module loss function Information maximization analysis module loss function and specific completion loss function The constructed total loss function is as follows: In the formula and The weights represent the losses of different modules.

[0017] The beneficial effects of this invention are: To address the shortcomings of existing technologies, this invention provides a robust modeling method for discrepancies and missing data in multimodal sentiment analysis. Through optimized structural design, this invention achieves the following technical advantages: Firstly, it alleviates the differences in modality distribution and improves the feature fusion effect: This invention introduces a modality alignment module (MAAT), which maps non-textual modalities to textual modality space through a recurrent adversarial mechanism, effectively reducing the differences in representation distribution between modalities, thereby enhancing the collaborative representation ability between modalities, promoting the deep fusion of multimodal features, and significantly improving the overall emotion recognition performance.

[0018] Secondly, it enables fine-grained modeling of features and enhances the ability to characterize emotions: This invention combines the modality decoupling mechanism with the mutual information maximization method to innovatively distinguish between modality-shared and modality-specific features, and enhances the semantic alignment and discrimination ability between modalities by using a mutual information estimator, thereby more comprehensively mining the detailed information in emotional expression and improving the accuracy and robustness of emotion classification.

[0019] Third, improve the robustness and generalization ability of the model under modality missing conditions: In response to the common problem of modality missing information in practical applications, this invention designs a specific completion module based on conditional generative adversarial network (CGAN) to generate sentiment-related representations of missing modalities, so as to maintain the recognition accuracy and stability of the model even when the input is incomplete, and demonstrate good practicality and generalization ability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1This is a diagram illustrating the overall framework of the multimodal sentiment analysis model of this invention. Figure 2 This is a schematic diagram of the MAAT module framework of the present invention; Figure 3 This invention specifically completes the network framework diagram; Figure 4 This is a detailed distribution of the dataset in the embodiments of the present invention; Figure 5 This is an experiment with different weights for the loss function in this embodiment of the invention; Figure 6 This describes how the loss function and evaluation metric change with the number of training steps in this embodiment of the invention. Figure 7 Visualization of shared features under different ablation configurations in the embodiments of the present invention; Figure 8 This is an example analysis of an embodiment of the present invention; Figure 9 This is a flowchart of the present invention. Detailed Implementation

[0022] The present invention will now be described in detail through exemplary embodiments. However, it should be understood that, without further description, elements, structures, and features in one embodiment may be advantageously incorporated into other embodiments.

[0023] It should be noted that, unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "a," "an," or "the," etc., used in this patent application specification do not express a quantity limitation, but rather indicate the presence of at least one; terms such as "comprising" or "including" indicate that the elements or objects preceding "comprising" encompass the elements or objects listed following "comprising" or "including" and their equivalents, but do not exclude other elements or objects having the same function.

[0024] This embodiment includes the following steps: S1. Obtain the emotional information modality to be processed, and perform feature extraction on the emotional information modality to obtain the raw modality data, wherein the raw modality data includes text modality X. t Audio Modality X a and video modal X v ; S2. Input the extracted sentiment information modality into the trained multimodal sentiment analysis model, and output the sentiment prediction result of the sentiment information modality to be processed; the multimodal sentiment analysis model includes a feature extraction module, a MAAT module, an information maximization analysis module, and a specific completion module; The training steps for the multimodal sentiment analysis model are as follows: S201. The multimodal sentiment analysis model takes the original modal data as input to the feature extraction module, outputs modal features with text modality, audio modality and video modality, and inputs them into the MAAT module. S202. The MAAT module transforms audio and visual distributions through an adversarial recurrent translation network to reduce modal differences between text and non-text, outputs distribution-aligned modal features, and uses them as input to the information maximization analysis module. S203. The information maximization module decouples the input modal features to obtain modal-specific information and invariant features. It then uses the information maximization mechanism to estimate the invariant features and specific information, outputs the information-maximized modal features, and uses them as the input of the specific completion module. S204. The specific completion module predicts the specific representation of the missing modality through a conditional generation network and improves the generation quality and discriminability by leveraging an adversarial learning mechanism, thereby achieving feature-level modality completion. S205. With the goal of minimizing the total loss function constructed from the model's basic loss function and the cross-modal contrastive learning loss function, the model parameters of the multimodal sentiment analysis model are optimized and updated, thereby completing the training of the multimodal sentiment analysis model. S206. Repeat steps S201 to S205 to perform iterative training until the multimodal sentiment analysis model converges or reaches the preset number of iterations.

[0025] In this embodiment, step S201 specifically involves: for multimodal sentiment analysis, text, audio, and video are the three main information modalities.

[0026] The steps for extracting data from the text modality include: using The model extracts features from the input text sequence to obtain feature representations of the text modalities. The expression for feature extraction of the text modalities is as follows: ; The audio modality feature extraction step includes: using the Wav2vec 2.0 basic model developed by the Facebook AI team to extract features from the input audio and video data, obtaining the feature representation of the audio. The feature extraction expression is as follows: ; The video modal feature extraction step includes: using Transformer ( Extraction involves dividing each 224x224 pixel input image into 16x16 pixel patches. Then, flattening encoding is performed to obtain the visual feature vector. .

[0027] In this embodiment, the MAAT module mentioned in step S202 is as follows: Figure 2 As shown, it is a modal representation obtained by converting text and non-text to a uniform distribution, and the formula is as follows: in and This represents the distribution transformation process between visual-text and audio-text.

[0028] For non-textual distribution transformations, the transformation process is subject to joint constraints of loss, including generator loss. Discriminator loss and cyclic loss The formula is as follows: in, , This represents a non-text to text distribution transformation. This represents the transformation from text to non-text. and This indicates whether the input is a text representation. The weights represent the cyclic loss.

[0029] pass and The interaction between the two modalities results in a representation distribution between text and non-text. The transformation does not change the semantic information of the modality. The transformed modal representation is as follows: In this embodiment, step s203 specifically involves: combining previous work, estimating MI by maximizing the variational lower bound. There are many types of lower bounds, and the researchers chose DV as the lower bound to estimate MI.

[0030] In the formula, W and Z represent two different specific features, f() represents the training function, and p(w, z) represents the joint probability distribution of random variables w and z, which is used to characterize the statistical correlation between different modal features. Represent the expectation under the joint distribution p(w, z); optimize the function f to make the lower bound... Maximizing this yields the strength of the interdependence between W and Z, and the corresponding loss function is: ; To calculate the Conditional Mutual Information (CMI), the researchers employed the Conditional Mutual Information Neural Estimation (CMINE) method. For different specific features W, Z, and Q, the lower bound of the DV of the CMI is: when When the lower bound is tight, meaning the lower bound estimate approximates the true mutual information, the corresponding loss function is: In the formula, W, Z and Q represent different specific features.

[0031] After MAAT processing, specific and differential features are extracted from the modal features. Taking the text modality as an example, the modality-invariant information between text and other modalities can be represented as follows: The corresponding training loss is: The specific characteristics of acoustics relative to text can be represented as: In the formula, X represents the learning concept, which refers to the abstract semantic features learned by the model during the multimodal alignment process. These features are used to represent the shared semantic information of text, speech, vision, and other modalities in the semantic space. This feature can be generated from the intermediate representation of the model during training and is used to measure the consistency or difference of features between different modalities. , and These represent the modal features corresponding to the acoustic modality, text modality, and video modality, respectively. I represents the initial mutual information, which is used to represent the original information content or reference value between different modal features before MAAT processing, as a benchmark for calculating the difference. Similarly, visual-specific features are represented as: Text-specific features are the sum of acoustic and visual features relative to text-specific features: The loss function in the modality-specific feature process is: In summary, the total loss function in the information maximization analysis process is: In the formula, , and This represents the corresponding weights, which will be determined through subsequent experiments to find the optimal weights. By minimizing the loss function, modality-specific features and invariant features are optimized and coordinated.

[0032] In this embodiment, step s204 specifically involves: To improve the robustness of the model in modality missing scenarios, this invention proposes MSH-Net. This module, based on the specific features of existing modalities, predicts the specific representation of missing modalities through a conditional generation network, and improves the generation quality and discriminability by leveraging an adversarial learning mechanism, thereby achieving feature-level modality completion.

[0033] Taking the lack of visual modalities as an example, the researchers designed a conditional generation network. Given textual and acoustic specific features, the specificity of the predicted visual modality is represented as follows: Furthermore, to enhance the realism and discriminative power of the generated features, researchers employed a conditional generative adversarial network (CGAN) as a completion mechanism. (Generator) The discriminator attempts to generate features for the missing modalities from known modalities. It is responsible for distinguishing the differences between generated features and real features.

[0034] The loss of the generator is defined as follows: in and These are weighting coefficients, which respectively control the impact of adversarial loss and reconstruction loss.

[0035] In this way, researchers can... Replace the real It is combined with other modalities for sentiment prediction.

[0036] The final three modality features and missing modality features are aggregated and appended to a four-layer perceptron for sentiment prediction. The final calculation formula is as follows: Using MSE as the task loss function, the model is trained by minimizing the mean squared error between the base ground truth labels and the predicted labels. The task loss function is: Where LTASK represents the mean squared error loss value, the smaller the value, the closer the model's prediction is to the true value, and n represents the number of samples. This represents the prediction error for the i-th sample.

[0037] In this embodiment, step S205 specifically involves: inputting the pre-extracted emotional information modality into the multimodal sentiment analysis model, and constructing a model task loss function and a MAAT module loss function. Information maximization analysis module loss function and specific completion loss function The overall loss function is constructed, and the model parameters of the multimodal sentiment analysis model are optimized and updated with the goal of minimizing the loss function, thereby completing the training of the multimodal sentiment analysis model.

[0038] Among them, the loss function for: The information maximization analysis module loss function for: In the formula, a and b represent the corresponding weights. By minimizing the loss function, modality-specific features and invariant features are optimized and coordinated.

[0039] The specific completion loss function for: As a preferred embodiment, the total loss function is: In the formula and The weights represent the losses of different modules.

[0040] To more clearly describe a robust modeling method for variance and missing data in multimodal sentiment analysis, combined with the appendix... Figure 1 -Appendix Figure 9 This embodiment is described as follows: The following is an example of a robust modeling method for discrepancies and missing data in multimodal sentiment analysis, including the following steps: (1) Use Python code to extract publicly available datasets (CMU-MOSI, CMU-MOSEI and CH-SMIS) and process the data.

[0041] (2) Construct a multimodal sentiment analysis model and train it using the CMU-MOSI, CMU-MOSEI and CH-SMIS datasets to obtain a trained sentiment analysis model; the training process includes data normalization and data augmentation.

[0042] (3) Obtain the multimodal data to be classified and input it into the trained multimodal sentiment analysis model. After processing by the model, the sentiment analysis results are obtained.

[0043] Step (1) includes the following steps in sequence: (1a) Obtain various public datasets (CMU-MOSI, CMU-MOSEI and CH-SMIS) through the Kaggle website and GitHub.

[0044] (1b) Write a Python program to process the dataset to ensure that it can be used during model training.

[0045] This embodiment implements the proposed model using the PyTorch framework on three benchmark datasets: CMU-MOSI, CMU-MOSEI, and CH-SMIS. Figure 4 The data distribution of the three datasets is shown.

[0046] During model training, the researchers used CUDA 11.2 and PyTorch 1.10.1 on the Ubuntu operating system. The AdamW optimizer was used, and the dimension of all modalities was set to 128. The batch size for the CMUMOSI dataset was set to 32, while the batch size for the CMU-MOSEI dataset was set to 12. The learning rate was set to 0.01, multiplied by 0.1 every 10 training iterations. An early stopping threshold of 15 was set during training. All experiments were conducted on an RTX 4090 DGPU. The computational complexity of the model was 7.688 GFLOPs.

[0047] I. Results and Analysis The table below shows the results of comparing the CMU-MOSI and CMU-MOSEI datasets using different models. MAE, Corr, Acc-2, F1, and Acc-7 are the evaluation metrics, and the left-hand column headers are the various data models.

[0048] The experimental results show that, on both the CMU-MOSI and CMU-MOSEI datasets, the proposed model outperforms existing baseline methods in key evaluation metrics such as Corr, Acc-2, and F1. Specifically, on the CMU-MOSI dataset, the proposed model achieves improvements of 0.6% and 0.5% in Acc-2 and F1 respectively compared to the current best model SPMN, and also surpasses the well-performing model DB-MPCA in Corr, with an improvement of 0.012. On the larger and more diverse CMU-MOSEI dataset, the proposed model continues to perform excellently, especially in Corr, where it achieves a 0.055 improvement compared to the best baseline model, and also leads in Acc-2 and F1 metrics.

[0049] Overall, these experimental results fully demonstrate the effectiveness and sophistication of the researchers' model in multimodal sentiment analysis tasks.

[0050] The table below shows the results of comparing different models used in the CH-SMIS dataset. MAE, Corr, Acc-2, F1, and Acc-7 are the evaluation metrics, and the left-hand column headers are the various data models.

[0051] As shown in the table above, the researchers' method achieved superior or competitive performance compared to several strong baseline methods on the Chinese multimodal sentiment recognition dataset CH-SMIS: the lowest MAE (0.380) indicates the model has the smallest error in sentiment intensity prediction; Acc-2 and Acc-5 scores reached 83.68% and 84.98% respectively, significantly outperforming other methods, indicating stronger discriminative ability in binary and pentatonic sentiment classification; the F1 score reached 46.26%, higher than all baseline models, indicating stronger robustness even under class imbalance; although slightly lower than AV-MC (0.699) in Corr (0.686), its overall performance across all metrics is superior. In conclusion, the proposed method has good generalization ability and practical application value in Chinese multimodal sentiment analysis tasks, especially showing significant advantages in handling multi-class and fine-grained sentiment recognition tasks.

[0052] II. Evaluation of Components in the Embodiments The three modules and some sub-components in this embodiment were evaluated using ablation experiments. The experimental results are detailed in the table below: Removing any component from the model leads to a performance degradation, indicating that each module contributes significantly to the overall performance. The most significant performance drop occurs when the Information Maximization (AMAX) module is removed, primarily because this module facilitates fine-grained interactions between modalities, effectively enhancing the expression of specific and differential information. Similarly, the absence of the MAAT module disrupts the consistency of modality distribution, resulting in decreased fusion performance and impacting model performance. As for the MSH module, its core function is to improve the model's robustness to incomplete or missing modalities; removing it significantly degrades the model's performance when faced with missing modalities. In conclusion, each part of the model plays an indispensable role.

[0053] III. The Impact of the Loss Function In the model, the loss functions used by the researchers include MAAT loss, information maximization loss, and specific reconstruction loss. To study the impact of the loss functions on the model, the following experiments were conducted, and the results are shown in the table below: The results in the table show that removing any loss constraint weakens model performance to varying degrees. The removal of [specific loss constraint name] caused a decrease in the model's Corr and Acc-2 by 0.058 and 1.0%, respectively, exhibiting the most significant impact. The removal of [specific loss constraint name] was the second most significant, increasing the MAE by 0.073 and decreasing the F1 score by 2.3%. Furthermore, [specific loss constraint name] also negatively impacted all other metrics. Only when all loss constraints are applied simultaneously can the model achieve optimal performance. This fully demonstrates the importance of each loss constraint for model training and verifies the effectiveness of the researchers' design.

[0054] IV. The Impact of Loss Function Weights The researchers studied the impact of the loss function weights and conducted numerous experiments, obtaining the following results: Figure 5 As shown, the tests involved extensive tuning of the weights for each loss function between 0.1 and 1, ultimately determining the optimal values. , and The optimal weights are 0.4, 0.2, and 0.6, at which point the model achieves its best performance. In contrast, non-optimal weight settings all lead to a decline in model performance. For example, when the weights are set to 0.3, the model's MAE metric performs the worst; while when all loss weights are set to 1, the model's F1 score decreases by 0.5%, 0.7%, and 0.6% respectively compared to the optimal state. This demonstrates that a reasonable allocation of loss weights has a significant positive impact on model performance; only when all loss weights are adjusted to their optimal values ​​can the model achieve its best results.

[0055] V. Visualization of Loss Trends To monitor training dynamics and evaluate model performance, the researchers plotted loss curves and trend charts of multiple indicators. Figure 6 The figure shows the trends of training, validation, and testing losses as the number of training steps increases. It can be seen that the losses of all three decrease rapidly and gradually stabilize with the increase of training steps, indicating that the model has not overfitted and the training process is as expected. The figure also shows the changes of various evaluation metrics during training. The F1, Acc-2, and Corr metrics all steadily increase with the increase of training steps, indicating that the model performance is continuously optimized and the matching degree between the prediction results and the true values ​​is gradually improved. The decreasing trend of the MAE metric further indicates that the model's predictions are more accurate, and the overall trend verifies that the model is in an ideal training state.

[0056] VI. Visualization of Feature Distribution To further verify the effectiveness of the proposed method in improving modal representation quality and emotion discrimination ability, the researchers used t-SNE to visualize the shared features extracted by the model. Specifically, three models were selected for comparison: the baseline model, the model with modal decoupling, and the model with further information maximization. The results are as follows: Figure 7 As shown.

[0057] VII. Case Analysis To further analyze the model's performance, the researchers provided the following real-world examples, including scenarios where audio or video was missing. The results are as follows: Figure 8 . Figure 8 Four examples are presented, including modal information, ground truth labels, and prediction results. It is evident that the developers' model performs exceptionally well. Even with partial modality loss in examples C and D, the model maintains stable prediction performance. This is attributed to the developers' combination of feature decoupling and information maximization methods, which fully exploits fine-grained modal features, and their effective handling of the modality loss problem through a modality-specific completion mechanism.

[0058] It should be noted that although the present invention has been described through the above embodiments, the present invention may have many other embodiments. Without departing from the spirit and scope of the present invention, those skilled in the art can obviously make various corresponding changes and modifications to the present invention, but all such changes and modifications should fall within the scope of protection of the appended claims and their equivalents.

Claims

1. A method for robust modeling of difference and absence in multi-modal sentiment analysis, the method comprising: The method comprises the following steps: S1, obtain an emotional information modality to be processed, perform feature extraction on the emotional information modality to be processed to obtain original modality data, the original modality data including a text modality X t , an audio modality X a , and a video modality X v ; S2, inputting the extracted emotional information mode of the data into the trained multi-modal emotion analysis model to output an emotional prediction result of the emotional information mode to be processed; the multi-modal emotion analysis model comprises a feature extraction module, an MAAT module, an information maximization analysis module, and a specificity completion module; The training steps of the multi-modal emotion analysis model are as follows: S201, the multi-modal emotion analysis model inputs the original modal data through the input of the feature extraction module, and outputs modal features with text modal, audio modal, and video modal, and inputs them into the MAAT module; S202, the MAAT module converts the audio and visual distribution through the adversarial cycle translation network to reduce the modal difference between the text and the non-text, and outputs the distribution-aligned modal features as the input of the information maximization analysis module; S203, the information maximization module decouples the input modal features to obtain modal-specific information and invariant features, estimates the invariant features and the specific information by using the information maximization mechanism, and outputs the information maximization-processed modal features as the input of the specificity completion module; S204, the specificity completion module predicts the specificity representation of the missing modal through the conditional generation network, and improves the generation quality and discriminability by means of the adversarial learning mechanism, so as to realize the modal completion at the feature level; S205, the model parameters of the multi-modal emotion analysis model are optimized and updated to minimize the total loss function constructed by the model basic loss function and the cross-modal contrast learning loss function, and then the training of the multi-modal emotion analysis model is completed; S206, repeat steps S201 to S205 to perform iterative training until the multi-modal emotion analysis model converges or reaches a preset iteration number.

2. The method as claimed in claim 1, wherein, The data extraction specifically comprises: using a model to process the original data to obtain the text modal, the audio modal, and the video modal after feature extraction, and the expression is as follows: Wherein X represents the original feature vector, θ is the model parameter, t, a, and v represent text, audio, and video respectively.

3. The method as claimed in claim 1, wherein, In step S202, the MAAT module obtains the unified distribution modal representation through distribution conversion, and the formula is as follows: wherein and denotes the distributional conversion process between visual-text and audio-text; the conversion process is constrained by a joint loss of losses , the converted modal representations are as follows: representing a non-text-to-text distribution translation, wherein .

4. The method of claim 1, wherein, In step S203, the information maximization analysis module estimates the specific features and invariant features through modal decoupling and mutual information maximization, and the loss function includes the lower bound of mutual information MI and conditional mutual information CMI; the estimation operation formula of mutual information is as follows: with the corresponding loss function ; The estimation operation formula of conditional mutual information is as follows: with a corresponding loss function of ; where W, Z and Q represent different specific features, f() represents a training function, and p(w, z) denotes a joint probability distribution of random variables w and z; denotes an expectation under the joint distribution p(w, z).

5. The method of claim 4, wherein, The estimation process estimates the modal invariant feature h inv characterizing the shared information between the individual modalities and other modalities, and the modality-specific feature h spe including acoustic-specific feature, visual-specific feature and text-specific feature composed of both, by constructing a total loss function L inv consisting of the modal invariant loss L spe and the modality-specific loss L mod to minimize the total loss function L mod to optimize and coordinate the modality-specific feature h spe with the invariant feature h inv .

6. The method of claim 5, wherein, The total loss function L inv consisting of the modal-invariant loss L spe and the modal-specific loss L mod The constraint formulas of the modal-invariant loss L inv and the modal-specific loss L spe are respectively: ； wherein , and represent the corresponding weights.

7. The method of claim 1, wherein, In step S204, the specific completion module generates the specific feature of the missing modality through the conditional generation network The specific feature of the existing modality is used to predict the specific representation of the missing modality The conditional generation adversarial network mechanism is used, the generated feature is distinguished from the real feature through the discriminator, and the generator is trained with the adversarial loss and the reconstruction loss, so that high-quality modality completion is realized at the feature level, and in this process, the generator loss function L gd is also constrained; after the completed feature is aggregated with the existing modality feature, it is input into a four-layer perceptron for emotion prediction, and the final operation formula is as follows: The model is optimized by minimizing the mean square error function L between the predicted value and the true value task The task loss function is: where n represents the number of samples, represents the prediction error of the i-th sample.

8. The method of claim 1, wherein, The step S205 specifically comprises: inputting the emotion information mode with pre-completed data extraction into the multi-modal emotion analysis model, and constructing a total loss function L task , a MAAT module loss function , an information maximization analysis module loss function , and a specificity completion loss function The total loss function constructed is: wherein and represent the weights of different module losses.

9. The method of claim 3, wherein, In step S202, the joint loss includes a generator loss , a discriminator loss , and a cycle loss , and the constraint formula is as follows: wherein, represents non-text-to-text distribution translation, represents text-to-non-text translation, wherein , and represents whether the input is a textual representation, represents the weight of the cycle loss.

Images