Method and system for multi-modal sentiment analysis based on credibility driving

Abstract

The application discloses a kind of based on credibility drive's multimodal sentiment analysis method and system, belong to artificial intelligence technical field.The system of the present application includes: modal feature decoupling reconstruction module, through the shared routing hybrid expert model of construction separates modal shared representation and modal private representation, and reconstructs modal feature, realizes semantic consistency;Dynamic modal perception calibration module, through modal contribution analysis component assesses each modal actual contribution degree, through consistency supervision constraint component based on the ranking supervision mechanism of pearson correlation coefficient Quantification model self-learning weight and the consistency of modal contribution value, enhance model reliability;Gating multimodal fusion module, through gating fusion framework actively activates the potential information of low contribution mode, improve the integrity of fusion.The method of the present application strengthens weak modal contribution, alleviates the problem of semantic ambiguity and unreliable fusion weight, can effectively improve the reliability and accuracy of model.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal information processing technology, specifically relating to a credibility-driven multimodal sentiment analysis method and system. Background Technology

[0002] With the development of artificial intelligence and human-computer interaction, multimodal sentiment analysis technology has gradually become an important research direction. Multimodal refers to the comprehensive use of multiple information sources such as audio, text, and vision to identify human emotional states. Compared with single-modal methods, multimodal technology can capture emotional cues more comprehensively, such as the tone of audio, the semantics of text, and changes in facial expressions, thereby improving the accuracy and robustness of sentiment recognition.

[0003] Existing multimodal sentiment analysis methods mainly include: attention-based fusion methods, which use cross-modal attention or multi-head attention networks to weight and fuse different modalities to highlight important modal features, such as using Transformer structures to achieve alignment and fusion between modalities; feature decoupling-based fusion methods, which separate shared and private information in modal features to reduce redundancy and semantic conflicts, and combine orthogonal constraints or contrastive learning to achieve decoupling between modalities; and dynamic routing or gating methods, which use gating networks or dynamic routing to adaptively allocate modal weights to achieve dynamic fusion of multimodal features, which can improve robustness to some extent, but still suffer from bias towards strong modalities.

[0004] However, existing multimodal sentiment analysis methods still have some shortcomings: 1) Multimodal information is mixed together, and semantics are easily confused: Many methods do not distinguish well between modal shared information and modal private information during fusion, and features of different modalities are prone to mutual interference, resulting in unclear final sentiment representation. 2) Modal weights are not realistic enough and have poor credibility: Many methods rely on attention or gating mechanisms to allocate modal weights, but these weights are often biased towards strong modalities (such as text) rather than reflecting the true role of each modality. This may result in situations where audio or visual modalities are crucial in certain circumstances, but the model ignores them, resulting in unreliable fusion. 3) Weak modalities are easily ignored, and information is underutilized: In real-world scenarios, audio and visual modal signals are often noisy, incomplete, or difficult to capture. Models tend to reject these modalities during fusion, rarely using them. However, these modalities often contain subtle but important supplementary cues, and ignoring them will result in information loss. Summary of the Invention

[0005] To address the problems of semantic confusion, unreliable weights, and insufficient utilization of weak modalities in existing multimodal sentiment analysis methods, this invention proposes a credibility-driven multimodal sentiment analysis method and system. Through decoupling and reconstruction, dynamic calibration, and a credibility-driven fusion mechanism, the robustness and credibility of the fusion results are effectively improved.

[0006] This invention adopts the following technical solution: a credibility-driven multimodal sentiment analysis method, comprising the following steps:

[0007] Step 1, Multimodal Feature Extraction: Extract features from the original inputs of text modality, audio modality, and visual modality, and perform normalization and dimension alignment to obtain an initial modal feature vector set;

[0008] Step 2, Modal Feature Decoupling and Reconstruction: Construct a shared routing hybrid expert model to decouple modal shared features from private features; introduce a load balancing loss function based on KL divergence to promote balanced utilization of expert resources; use similarity constraints and difference constraints to ensure consistency among shared features and distinguishability among private features; enhance cross-modal semantic consistency by leveraging a private-guided shared reconstruction mechanism.

[0009] Step 3, Dynamic Modality Awareness Calibration: Construct a dynamic modality awareness calibration model, evaluate the actual contribution of each modality through the modality contribution analysis component, and quantify the consistency between the model's self-learning weights and modality contribution values ​​through the consistency supervision constraint component and the ranking supervision mechanism based on the Pearson correlation coefficient.

[0010] Step 4, Credibility-Driven Fusion: In gated multimodal fusion, a weak modality enhancement mechanism is introduced. Through the gated fusion framework, the potential emotional information of weak modalities is actively activated, weak modality enhancement constraints are constructed, supplementary information of weak modalities is mined, all modality weights are limited to a reasonable range, and the final overall optimization goal is determined to ensure information integrity.

[0011] As a preferred embodiment, the multimodal feature extraction method in step 1 is as follows:

[0012] For text modal data, a pre-trained BERT model is used to encode the text to obtain multi-dimensional semantic features.

[0013] For audio modal data, the speech signal is converted into a Mel spectrogram, which is then input into a stacked sLSTM model to extract time series features.

[0014] For visual modal data, facial expression features are extracted using the OpenFace tool, and dynamic changes are captured using an sLSTM model.

[0015] Raw input for three modes Features are extracted, normalized, and aligned in dimensions to obtain the initial modality feature vector set. ; where subscript These represent the text modality, visual modality, and audio modality, respectively. These represent the input text modality, visual modality, and audio modality data, respectively.

[0016] As a preferred embodiment, the shared routing hybrid expert model described in step 2 includes one shared expert and eight routing experts with different structures, which are used to extract the shared features and private features of each modality, respectively.

[0017] Among them, shared features Extracted by shared experts, for the first A routing expert The input is the multimodal feature representation generated by the upstream encoder. This constructs and outputs the corresponding expert-specific feature vector. Then, by dynamically selecting some experts through Top-K routing, the input features are processed. Embedded with modal indicator tags Concatenate and input the two-layer neural network with non-linear activation to generate the first... The probability of a routing expert's choice After obtaining the selection probabilities of all routing experts, the dynamically selected routing experts are weighted and aggregated to obtain modality-specific features. .

[0018] Furthermore, the privately guided shared reconstruction mechanism employs a two-layer fully connected network to fuse features element by element for each modality, resulting in the reconstructed features of the three modalities. , , .

[0019] As a preferred embodiment, in step 3, the modal contribution analysis component compares the prediction results of the three complete modes with those of the mode after masking a specific mode to evaluate the actual contribution of each mode. The method is as follows:

[0020] For modal features Given all three modalities, the prediction result is expressed as follows: , It is a prediction function;

[0021] After removing a certain modality, the prediction result is expressed as: ;

[0022] The contribution of the excluded mode is: This is used to reflect the degree of impact of the removed mode on the prediction results. The larger the modality, the higher its contribution.

[0023] Contribution vectors for each mode [ , , Perform softmax normalization to obtain the modal contribution weight vector contrib.

[0024] Furthermore, the ranking supervision mechanism based on Pearson correlation coefficient has the following process:

[0025] Gated multimodal fusion is performed to obtain the modal weights. Modal contribution After normalization, the probability distribution is obtained. and ;

[0026] Standardize the probability distribution: ;in, Represents a probability distribution. This represents the average probability distribution within a batch.

[0027] Based on the standardized vectors, the Pearson correlation coefficient is calculated along the sample dimension. And through the consistency supervision and constraint component, based on the Pearson correlation coefficient Define the loss function.

[0028] As a preferred embodiment, in step 4, the credibility-driven fusion is performed using the following method:

[0029] Construct a gated multimodal fusion module to reconstruct modal features. , , spliced ​​into a unified representation After being processed by a multilayer perceptron (MLP), the data is input into a gating mechanism along with noise, and the fusion weights are obtained through softmax. .

[0030] Furthermore, a bidirectional threshold constraint loss was introduced. The goal is to limit all modal weights to a reasonable range. Within this framework, the overall optimization objective is determined.

[0031] The present invention also provides: a credibility-driven multimodal sentiment analysis system for implementing the above-mentioned multimodal sentiment analysis method, comprising:

[0032] The multimodal feature extraction module is used to extract features from the original inputs of text modality, audio modality, and visual modality, and to perform normalization and dimension alignment to obtain an initial modal feature vector set;

[0033] The modal feature decoupling and reconstruction module is used to build a shared routing hybrid expert model, distinguish and reconstruct shared features and private features between modalities; promote the balanced utilization of expert resources through a load balancing loss function based on KL divergence; ensure the consistency between shared features and the distinguishability between private features through similarity constraints and difference constraints; and enhance cross-modal semantic consistency through a private-guided shared reconstruction mechanism to prevent semantic confusion and modal misalignment.

[0034] The dynamic modality-aware calibration module includes a modality contribution analysis component and a consistency supervision and constraint component, which are used to construct a dynamic modality-aware calibration model, introduce a modality contribution evaluation method, and calibrate the fusion weights to ensure that the weights are consistent with the actual effects of the modes.

[0035] The credibility-driven fusion module is used to introduce a weak modality enhancement mechanism in gated multimodal fusion. It actively activates the potential emotional information of weak modalities through the gated fusion framework, constructs weak modality enhancement constraints, mines supplementary information of weak modalities, limits all modality weights to a reasonable range, and determines the final overall optimization goal to ensure information integrity.

[0036] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0037] 1. The method of this invention achieves feature decoupling through the shared routing hybrid expert mechanism SR-MoE of the modal decoupling and reconstruction (MDR) module, clearly separating shared sentiment information from private details, and then reconstructing and optimizing feature representation through the private guided shared reconstruction mechanism PSR, eliminating semantic interference between different modalities, making features clearer, providing a high-quality foundation for subsequent fusion, and improving the accuracy of sentiment analysis.

[0038] 2. The method of this invention quantifies the actual contribution of a modality through modality contribution analysis (MCA) of the Dynamic Modality Perceived Calibration (DMC) module, and combines consistency supervision constraints to calibrate weights, ensuring that the weight allocation is consistent with the real role of the modality in sentiment analysis, avoiding unreliable analysis results due to weight bias, and enhancing the credibility of the model.

[0039] 3. The method of this invention introduces a weak modality enhancement mechanism in the gated multimodal fusion (GMF) module. By actively activating the potential emotional information of weak modalities through the gated fusion framework, it avoids over-reliance on strong modalities such as text, ensures the integrity of multimodal information, and improves the model's adaptability and robustness to complex emotional scenarios. Attached Figure Description

[0040] Figure 1 This is the overall network structure diagram of the credibility-driven multimodal sentiment analysis system of this invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0042] Example 1

[0043] A credibility-driven multimodal sentiment analysis method is provided, such as Figure 1 As shown, it includes the following steps:

[0044] (I) Multimodal Feature Extraction

[0045] In this embodiment, multimodal features are extracted, including:

[0046] Text modality: The text is encoded using a pre-trained BERT model to obtain 768-dimensional semantic features.

[0047] Audio modality: The speech signal is converted into a Mel spectrogram and input into a stacked LSTM (sLSTM) to extract time series features.

[0048] Visual modality: Facial expression features are extracted using tools such as OpenFace, and dynamic changes are captured using sLSTM.

[0049] Further analysis of the original inputs for the three modalities The features are extracted and normalized and aligned in dimensions, as shown in the following formula:

[0050] ;

[0051] Obtain the initial modal feature vector set .

[0052] in, These represent the text modality, visual modality, and audio modality, respectively. These represent the input text modality, visual modality, and audio modality data, respectively. These represent the linear mapping parameters for dimension alignment of text, visual, and audio data, respectively. The function represents a pre-trained language model specifically designed for processing text data. The function represents a stacked long short-term memory network. The function represents the sigmoid activation function. Function representation layer normalization.

[0053] (II) Modal Feature Decoupling and Reconstruction

[0054] To address the semantic confusion and modal misalignment issues in the early stages of fusion, this embodiment designs a modal feature decoupling and reconstruction module. This module decouples shared modal features from private features using Shared Routing Hybrid Expert (SR-MoE) technology and enhances cross-modal semantic consistency through Private Guided Shared Reconstruction (PSR) mechanism, thereby laying a solid foundation for reliable multimodal fusion.

[0055] Specifically, the shared routing hybrid expert consists of one shared expert and eight routing experts with different structures, which are used to extract shared features and private features of each modality, respectively.

[0056] Shared features Extracted by shared experts in the following manner:

[0057] ;

[0058] in, The function represents the processing logic of the shared expert. This represents multimodal feature representation. ; The function represents a fully connected layer. The function represents the linear rectifier unit as the activation function.

[0059] And for the first A routing expert The input is the multimodal feature representation generated by the upstream encoder. The structure and output are the corresponding expert-specific feature vectors. .

[0060] The overall calculation formula is as follows:

[0061] ;

[0062] in, The function represents batch normalization. The function represents the dimension flattened into a 1-dimensional vector. The function represents the hyperbolic tangent. The function represents a 1-dimensional convolutional unit. The function represents the sigmoid linear unit as the activation function.

[0063] Subsequently, SR-MoE selects some experts through dynamic Top-K routing to process the input features. Embedded with modal indicator tags The concatenation and input into a two-layer neural network with non-linear activation generates the first... The probability of a routing expert's choice :

[0064] ;

[0065] in, It is a learnable representation of modality indicator label embedding. The function represents vector concatenation. The function represents a multilayer perceptron, including fully connected layers and activation functions. The function represents the normalized exponential function.

[0066] After obtaining the selection probabilities of all routing experts, the modality-private features are obtained by weighted aggregation of the dynamically selected routing experts. :

[0067] ;

[0068] in, This represents the top K experts in route selection. This indicates a modality indicator label, used to identify which modality a feature belongs to.

[0069] To promote the balanced utilization of expert resources, this embodiment introduces a load balancing loss function based on KL divergence:

[0070] ;

[0071] in, This represents the probability distribution of the actual routing expert being selected. Let represent an ideal uniform distribution, where the probability of each expert being selected is... ,and This indicates divergence calculation.

[0072] In addition, after obtaining modality-sharing features and private characteristics Subsequently, this embodiment employs similarity constraints. and difference constraints Ensure consistency among shared features and distinctiveness among private features:

[0073] ;

[0074] in, The similarity constraint is represented and calculated using the CMD function; Indicates difference constraints; Indicates two shared features, superscript Indicates transpose. Denotes the squared Frobenius norm. It can represent any two of the three modalities: text, visual, and audio.

[0075] This design constructs a semantically coordinated shared space that preserves modality specificity while improving fusion robustness.

[0076] Furthermore, in order to improve the representation accuracy of modal private information in shared features and enhance semantic consistency, a Private Guided Shared Reconstruction (PSR) mechanism is proposed, which integrates private and shared features to perform refined reconstruction of modal features.

[0077] Specifically, PSR employs a two-layer fully connected network to fuse features element-wise for each modality:

[0078] ;

[0079] in, The function represents a linear mapping. Represents the modal features of reconstruction. The three modal features of reconstruction , , It will be used as input for subsequent modules.

[0080] (III) Dynamic Modal Sensing Calibration

[0081] To address the issue of inaccurate mode weights in multimodal fusion, this embodiment proposes a Dynamic Modality Aware Calibration (DMC) module. This module utilizes two key components—Modal Contribution Analysis (MCA) and Consistency Supervision Constraints. This ensures that the allocated fusion weights accurately reflect the true contributions of each modality, significantly improving the reliability of the fusion process.

[0082] In modal contribution analysis, this embodiment evaluates the actual contribution of each mode by comparing the prediction results of the three complete modes with those of a specific mode being masked.

[0083] Specifically, for modal features , If it is a prediction function, then the prediction result for all three modalities can be expressed as: The prediction result after removing a certain mode can be expressed as .

[0084] The contribution of the removed mode is defined as This value reflects the degree of influence of the removed mode on the prediction result. The larger the value, the higher the contribution of that mode. Contribution vectors for each mode [ , , After performing softmax normalization, the modal contribution weight vector contrib is obtained.

[0085] Furthermore, to quantify the consistency between the model's self-learning weights and modal contribution values, this embodiment designs a ranking supervision mechanism based on the Pearson correlation coefficient.

[0086] The specific process is as follows: First, the modal weights obtained in the gated multimodal fusion are... Modal contribution Normalization is performed to obtain the probability distribution. and .

[0087] Next, the probability distribution is standardized: .

[0088] Subsequently, based on the standardized vectors, the Pearson correlation coefficient is calculated along the sample dimension:

[0089] ;

[0090] in, Indicates the number of modes. For numerically stable terms, , These represent the standardized modal weights and modal contributions, respectively.

[0091] The final loss function is defined as:

[0092] ;

[0093] in, Indicates batch size. This represents the Pearson correlation coefficient calculated for each sample in the batch.

[0094] This loss, while maintaining ranking consistency, allows for greater weight flexibility, thereby further improving the generalization ability of supervised learning and enhancing the interpretability of model fusion.

[0095] (iv) Credibility-driven integration

[0096] To address the issue that models may over-rely on dominant modes or neglect weak modes during multimodal fusion, this embodiment designs a gated multimodal fusion (GMF) module.

[0097] First, the reconstructed modal features , , spliced ​​into a unified representation After processing by a multilayer perceptron (MLP), the output and noise are input together into a gating mechanism, and the fusion weights are obtained through softmax. These weights, through adaptive fusion of multimodal features, enable the model to utilize complementary information across modalities.

[0098] Furthermore, to further promote the utilization of multimodal information, we introduce a bidirectional threshold constraint loss. Its goal is to limit all modal weights to a reasonable range. This allows the model to fully explore multimodal information and improve fusion robustness. The specific definitions are as follows:

[0099] ;

[0100] in, It is a hyperparameter that controls the degree of weight balance, ensuring that weak mode information is not completely ignored; This represents the weight of each modality for each sample in the batch.

[0101] The ultimate overall optimization goal is:

[0102] ;

[0103] in, This represents the regression loss in sentiment analysis. This indicates load balancing losses; and The weights for similarity loss and difference loss are respectively (set to 1 in the experiment). =0.6 and =0.1); The load balancing weights in the SR-MOE module (set to [value] in the experiment) =0.15); and The weights corresponding to the balance contribution ranking constraint and the weak modal excitation mechanism are respectively set as follows (in the experiment, they are set as follows). =0.05 and =0.15).

[0104] Example 2

[0105] A credibility-driven multimodal sentiment analysis system is provided for implementing the multimodal sentiment analysis method of Embodiment 1, including: a multimodal feature extraction module, a modal feature decoupling and reconstruction module, a dynamic modal perception calibration module, and a credibility-driven fusion module.

[0106] The multimodal feature extraction module is used to extract features from the original inputs of text modality, audio modality, and visual modality, and to perform normalization and dimension alignment to obtain an initial modal feature vector set.

[0107] The modal feature decoupling and reconstruction module is used to build a shared routing hybrid expert model, distinguish and reconstruct shared features and private features between modalities; promote the balanced utilization of expert resources through a load balancing loss function based on KL divergence; ensure the consistency between shared features and the distinguishability between private features through similarity constraints and difference constraints; and enhance cross-modal semantic consistency through a private-guided shared reconstruction mechanism to prevent semantic confusion and modal misalignment.

[0108] The dynamic modality-aware calibration module includes a modality contribution analysis component and a consistency supervision and constraint component, which are used to construct a dynamic modality-aware calibration model, introduce a modality contribution evaluation method, and calibrate the fusion weights to ensure that the weights are consistent with the actual effects of the modes.

[0109] The credibility-driven fusion module is used to introduce a weak modality enhancement mechanism in gated multimodal fusion. It actively activates the potential emotional information of weak modalities through the gated fusion framework, constructs weak modality enhancement constraints, mines supplementary information from audio and visual modalities, limits all modal weights to a reasonable range, and determines the final overall optimization goal to ensure information integrity.

[0110] In this embodiment, the system first collects and preprocesses data from different modalities through a multimodal feature extraction module, and encodes it into feature representations. Subsequently, features are obtained through the Modal Decoupling and Reconstruction (MDR) module. Next, the modality fusion process is supervised, corrected, and refined sequentially through the Dynamic Modality Awareness Calibration (DMC) module and the Gated Multimodal Fusion (GMF) module; finally, the fused features are... It is used to complete the prediction and obtain the final prediction result. .

[0111] Furthermore, the credibility-driven multimodal sentiment analysis method of the present invention was experimentally verified: the method of the present invention was compared with existing mainstream methods on the public datasets CMU-MOSI and CMU-MOSEI, and the results are shown in Table 1.

[0112] Table 1: Performance Comparison of the Trustworthiness-Driven Multimodal Sentiment Analysis System of this Invention with Existing Mainstream Methods on the MOSI / MOSEI Dataset.

[0113]

[0114] As can be seen from Table 1, the credibility-driven multimodal sentiment analysis system of this invention exhibits stable and competitive performance on both the MOSI and MOSEI datasets.

[0115] On the MOSI dataset, the credibility-driven multimodal sentiment analysis system achieved the highest Acc7 (44.9), Acc2 (84.2), and Corr (0.801), and the lowest MAE (0.740).

[0116] On the MOSEI dataset, the credibility-driven multimodal sentiment analysis system also achieved the best Acc7 (53.0), F1 (87.8) and Corr (0.786) scores, and maintained the lowest MAE (0.536), further validating its generalization ability in large-scale, semantically complex sentiment analysis tasks.

[0117] While some existing methods perform well on a single metric, the credibility-driven multimodal sentiment analysis system of this invention achieves a better balance among multiple evaluation dimensions.

[0118] Overall, the fusion mechanism proposed in this invention effectively alleviates common problems in multimodal fusion, such as inconsistency in modal contributions and semantic conflicts, and provides a more reliable and robust solution for multimodal sentiment analysis.

[0119] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method of credibility-driven multi-modal sentiment analysis, the method comprising: Comprising the following steps: Step 1, multi-modal feature extraction: extracting features from the original input of the text, audio and visual modalities, and performing normalization and dimension alignment to obtain an initial set of modal feature vectors; Step 2, modal feature decoupling and reconstruction: constructing a shared routing hybrid expert model to decouple modal shared features and private features; introducing a load balancing loss function based on KL divergence to promote balanced utilization of expert resources; using similarity constraints and difference constraints to ensure consistency between shared features and distinctiveness between private features; using a private guided shared reconstruction mechanism to enhance cross-modal semantic consistency; Step 3, dynamic modal perception calibration: constructing a dynamic modal perception calibration model, evaluating the actual contribution of each modality through a modal contribution analysis component; quantifying the consistency of model self-learning weights and modal contribution values based on a ranking supervision mechanism of Pearson correlation coefficient through a consistency supervision constraint component; Step 4, credibility-driven fusion: introducing a weak modality enhancement mechanism in the gated multi-modal fusion, actively activating the potential emotional information of the weak modality through the gated fusion framework, constructing a weak modality enhancement constraint, mining the supplementary information of the weak modality, limiting all modality weights within a reasonable range, and determining the final overall optimization target to ensure information integrity, the specific method is as follows: Construct a gated multimodal fusion module to reconstruct modal features. , , spliced ​​into a unified representation After being processed by a multilayer perceptron (MLP), the data is input into a gating mechanism along with noise, and the fusion weights are obtained through softmax. ; Introducing bidirectional threshold constraint loss The goal is to limit all modal weights within a reasonable range The formula is as follows: ； wherein, is a hyper-parameter controlling the degree of weight balancing, used to ensure that weak modal information is not ignored; denotes the weight of each modality for each sample in the batch; denotes the batch size; Determine the overall optimization target, represented as: ； wherein, represents a regression loss of sentiment analysis, represents a load balancing loss; represents a similarity constraint, calculated by a CMD function; represents a difference constraint; represents a loss function of the consistency supervision constraint component; and are weights of the similarity loss and the difference loss, respectively, is a weight of load balancing in the SR-MOE module, and correspond to weights of the balance contribution ranking constraint and the weak modality excitation mechanism, respectively.

2. The trust driven multi-modal sentiment analysis method of claim 1, wherein, The multi-modal feature extraction of step 1 is as follows: For text modality data, use a pre-trained BERT model to encode the text to obtain multi-dimensional semantic features; For audio modality data, convert the speech signal to a Mel spectrogram and input it into a stacked sLSTM model to extract time series features; For visual modality data, use the OpenFace tool to extract facial expression features and capture dynamic changes through an sLSTM model; Raw inputs of three types of modalities Extract features and normalize and dimension align to get initial modal feature vector set , which is expressed by the formula: ； wherein, respectively denote text, visual, and audio modalities, respectively denote input text, visual, audio modal data, respectively denote linear mapping parameters for dimension alignment for text, visual, and audio data, the function denotes a pre-trained language model processing text data, the function denotes a stacked long short-term memory network, the function denotes a sigmoid activation function, the function denotes layer normalization.

3. The trust driven multi-modal sentiment analysis method of claim 2, wherein, The shared routing hybrid expert model of step 2 includes 1 shared expert and 8 routing experts with different structures, which are used to extract shared features and private features of each modality, the specific processing is as follows: Shared features Extracted by the sharing expert in the following way: ； wherein, represents a processing logic function of a shared expert, represents a multi-modal feature representation, ; the function represents a fully connected layer, the function represents a linear rectifier unit as an activation function; For the A routing expert The input is the multimodal feature representation generated by the upstream encoder. The system constructs and outputs the corresponding expert-specific feature vectors. The formula is as follows: ； wherein, the function represents batch normalization, the function represents dimension expansion into a 1-dimensional vector, the function represents hyperbolic tangent, the function represents a 1-dimensional convolution unit, the function represents sigmoid linear unit as an activation function, and N is the total number of routing experts. The input features are processed by a dynamic Top-K routing portion expert to generate a selection probability for each of the plurality of experts with a modality indicator label The selection probability is input to a non-linear activation function to generate a selection probability for each of the plurality of experts :​ ； wherein, is a learnable representation of a modal indicator label embedding, the function represents vector concatenation, the function represents a multi-layer perceptron, including fully connected layers and activation functions, the function represents a normalized exponential function; After obtaining the selection probability of all routing experts, the dynamically selected routing experts are weighted and aggregated to obtain the modal private feature : ； in, Indicates the previous route selection One expert, This indicates a modal indicator label.

4. The trust driven multi-modal sentiment analysis method of claim 3, wherein, In step 2, the load balancing loss function based on KL divergence is as follows: ； where, represents the probability distribution of the actual routing experts being chosen, represents the ideal uniform distribution, where each expert has a probability of , represents the divergence calculation.

5. The trust driven multi-modal sentiment analysis method of claim 3, wherein, In step 2, the similarity constraints and difference constraints are as follows: ； where denotes two shared features, denotes two private features, superscript denotes transpose, denotes squared Frobenius norm, denotes any two of the three modalities of text, vision, and audio.

6. The trust driven multi-modal sentiment analysis method of claim 3, wherein, In step 2, the private guided shared reconstruction mechanism adopts a double-layer fully connected network to element-wise fuse the features of each modality, obtaining the reconstructed three-modality features , , : ； wherein, The function represents a linear mapping, represents the reconstructed modal feature, .

7. The trust driven multi-modal sentiment analysis method of claim 5, wherein, In step 3, the modal contribution analysis component compares the prediction results of complete three modalities and the prediction results after shielding a specific modality to evaluate the actual contribution of each modality, the method is as follows: For modal features , input complete three modal, the prediction result is expressed as: , is the prediction function; After removing a certain modality, the prediction results are represented as: ; The contribution degree of the removed modal is: , , for reflecting the influence degree of the removed modal on the prediction result, The higher the modal contribution degree is, the higher the modal contribution degree is. Contribution vectors for each mode [ , , Perform softmax normalization to obtain the modal contribution weight vector contrib.

8. The trust-driven multi-modal sentiment analysis method of claim 7, wherein, In step 3, the ranking supervision mechanism based on Pearson correlation coefficient is as follows: Carrying out gated multimodal fusion, obtaining the weight of each modality and the contribution of the modality , carrying out normalization processing, obtaining a probability distribution and ; Standardizing the probability distribution: ; where, represents the probability distribution, represents the average probability distribution within a batch, Based on the standardized vector, calculate the Pearson correlation coefficient in the sample dimension: ； wherein, denotes the number of modalities, is a numerical stability term, , denote the normalized modality weight and modality contribution, respectively; The consistency supervision constraint component, the loss function is defined as: ； wherein, represents the Pearson correlation coefficient calculated for each sample in the batch. 9.A credibility-driven multi-modal sentiment analysis system for implementing the multi-modal sentiment analysis method of any one of claims 1 to 8. Comprising: A multi-modal feature extraction module for extracting features from the original input of the text, audio and visual modalities, and performing normalization and dimension alignment to obtain an initial set of modal feature vectors; The modal feature decoupling and reconstruction module is used to construct a shared routing hybrid expert model, distinguish and reconstruct shared features and private features between modes; a load balancing loss function based on KL divergence is used to promote balanced use of expert resources; similarity constraints and difference constraints are used to ensure consistency between shared features and distinguishability between private features; a private guided shared reconstruction mechanism is used to enhance cross-modal semantic consistency and prevent semantic confusion and modal misplacement; The dynamic modal perception calibration module includes a modal contribution analysis component and a consistency supervision constraint component, which is used to construct a dynamic modal perception calibration model, introduce a modal contribution evaluation method, and calibrate the fusion weight so that the weight is consistent with the true effect of the mode. The credibility driving fusion module is used for introducing a weak mode enhancement mechanism in gated multi-modal fusion, actively activating potential emotional information of the weak mode through a gated fusion framework, constructing a weak mode enhancement constraint, mining supplementary information of the weak mode, and limiting weights of all modes in a reasonable range The internal and external information is integrated, and the final overall optimization target is determined to ensure information integrity.

Images