A multi-modal sentiment analysis method and system based on channel re-labeling and polarity membership consistency gating

Abstract

The application discloses a kind of multi-modal sentiment analysis method and system based on channel re-labeling and polarity membership consistency gate, it is related to sentiment analysis technical field, including: by channel re-labeling, the enhancement or inhibition of different modal characteristic channel is realized, based on polarity membership consistency gate mechanism, modal reliability estimation is carried out in fusion stage, text counterfactual correction is carried out to full modal prediction result, it is helpful to enhance the processing capacity of sentiment analysis model to noise channel, low reliable mode and boundary sample, and then improve sentiment analysis reliability.

Description

Technical Field

[0001] This invention relates to the field of sentiment analysis technology, and in particular to a multimodal sentiment analysis method and system based on channel recalibration and polarity membership consistency gating. Background Technology

[0002] Multimodal sentiment analysis can obtain more comprehensive information on emotional expression and has been widely applied in fields such as entertainment media, smart education, and mental health.

[0003] In existing multimodal sentiment analysis methods, different modal representations are divided into shared semantic representations and modality-specific semantic representations. Constraints are used to enhance the decoupling relationship between shared and specific information. While this semantic decoupling approach alleviates the problem of semantic confusion between modalities to some extent, the original features of different modalities are mapped to a unified dimension and then indiscriminately input into the shared encoder and modality-specific encoder. This can easily lead to irrelevant or noisy channels interfering with the subsequent semantic decoupling process. In addition, since the modal quality of samples often varies significantly (e.g., some samples have clear text sentiment but weak audio or visual signals, some samples have obvious audio intonation but short text content, and some samples have discrepancies between visual expressions and text semantics), when fusing relationships through direct concatenation and implicit learning by multilayer perceptrons, the lack of explicit judgment on which modality in the current sample is more reliable can easily lead to low-quality or conflicting modalities interfering with the final prediction. Furthermore, multimodal sentiment analysis typically requires further determination of positive and negative sentiment polarity based on continuous sentiment scores. When the predicted sentiment score approaches the zero-point boundary, even minor perturbations in audio or visual noise can cause the positive and negative classification results to flip. It also makes it difficult to explain which modality the final predicted sentiment score primarily depends on, whether that modality has high sentiment confidence, and whether that modality maintains consistency with other modalities in the shared semantic space, thus limiting interpretability. Therefore, improving the reliability of sentiment analysis is urgently needed. Summary of the Invention

[0004] This invention provides a multimodal sentiment analysis method and system based on channel recalibration and polarity membership consistency gating, which is used to improve the reliability of sentiment analysis in multimodal sentiment analysis.

[0005] The first aspect of this invention provides a multimodal sentiment analysis method based on channel recalibration and polarity membership consistency gating, comprising:

[0006] The training multimodal sentiment input is fed into the sentiment analysis model to be trained;

[0007] The trained multimodal emotion input is subjected to feature alignment in a channel-first format to determine text alignment features, audio alignment features, and visual alignment features, and channel recalibration is performed to determine the corresponding modality recalibration features;

[0008] The modal shared features corresponding to each modal recalibration feature are determined by a shared encoder, and the modal shared features are enhanced by a shared self-attention encoder. The hidden state at the last time point is then concatenated to construct a shared fusion feature.

[0009] After extracting the modality-specific features corresponding to each modality recalibration feature using a modality-specific encoder, cross-modal interaction is carried out through text guidance, and the corresponding high-level modality features are extracted through a memory encoder.

[0010] Based on the modal high-level predicted sentiment scores of each modality high-level feature, an associated modal sentiment polarity confidence score is constructed. Then, combined with the cross-modal shared semantic consistency score of the modal shared alignment features of each modality shared feature alignment projection, a corresponding modal scaling factor is constructed to scale each modality high-level feature, and the corresponding modal scaling high-level feature is determined.

[0011] After linear projection and Sigmoid activation function mapping are performed on the shared fusion features, the transformed shared fusion features are concatenated with the modality-scaled high-level features to construct the final fusion features, and then input into the full-modality prediction head to predict and output the full-modality prediction sentiment score.

[0012] The model parameters are iteratively optimized based on the total training loss value corresponding to the predicted sentiment score of the full modality until the trained sentiment analysis model is determined.

[0013] By using a trained sentiment analysis model, full-modal prediction and text counterfactual inference correction are performed on the counterfactual multimodal sentiment inputs of the training text, the verification multimodal sentiment inputs, and the verification text counterfactual multimodal sentiment inputs, respectively, to determine the mean counterfactual prediction score of the training text and the optimal combination of correction parameters;

[0014] The target full-modal predicted sentiment score and the target text counterfactual full-modal predicted sentiment score of the multimodal sentiment input to be analyzed are determined by the trained sentiment analysis model. The target full-modal predicted sentiment score is then corrected by text counterfactual reasoning by combining the mean of the training text counterfactual predicted scores and the optimal correction parameter combination, and the final target sentiment predicted sentiment score is determined.

[0015] Among them, text counterfactual multimodal emotional input refers to multimodal emotional input that keeps the text emotional input unchanged while setting the audio emotional input and visual emotional input to zero.

[0016] Further, the step of performing feature alignment on the trained multimodal emotion input in a channel-first format to determine text alignment features, audio alignment features, and visual alignment features, and performing channel recalibration to determine the corresponding modality recalibration features, includes:

[0017] After converting text sentiment input, audio sentiment input, and visual sentiment input into channel-first format through a transpose layer, they are input into a one-dimensional convolutional projection layer for feature alignment to determine text alignment features, audio alignment features, and visual alignment features.

[0018] A residual one-dimensional channel recalibration module is used to perform channel recalibration on the text alignment features, the audio alignment features, and the visual alignment features respectively, to determine the text recalibration features, audio recalibration features, and visual recalibration features;

[0019] The processing procedure of the residual one-dimensional channel recalibration module includes:

[0020] ;

[0021] In the formula, Modal index , Representing modes The recalibration features, Representing modes Alignment features, Representing modes Learnable residual coefficients This indicates multiplication by channel. This represents the Sigmoid activation function. Representing modes The first calibration can be learned projection matrix. Representing modes The second calibration learnable projection matrix, Represents the ReLU activation function. This represents a one-dimensional global average pooling layer.

[0022] Furthermore, the modal recalibration features include text recalibration features, audio recalibration features, and visual recalibration features; after extracting the modal-specific features corresponding to each of the modal recalibration features using a modal-specific encoder, cross-modal interaction is conducted through text guidance, and the corresponding high-level modal features are extracted through a memory encoder, including:

[0023] The text recalibration features, audio recalibration features, and visual recalibration features are respectively input into the corresponding modality-specific encoder to determine the text-specific features, audio-specific features, and visual-specific features;

[0024] The text-specific features are used to guide the audio-specific features and the visual-specific features to perform cross-modal attention operations, and the corresponding audio interaction features and visual interaction features are output.

[0025] The text-specific features, audio interaction features, and visual interaction features are respectively input into their respective memory encoders, and the hidden state at the last time point is taken as the text high-level features, audio high-level features, and visual high-level features.

[0026] Furthermore, the high-level modal features include text high-level features, audio high-level features, and visual high-level features; the construction of the associated modal sentiment polarity confidence based on the modal high-level predicted sentiment scores of each of the high-level modal features includes:

[0027] The high-level features of the text, the high-level features of the audio, and the high-level features of the visual are respectively input into the associated high-level branch prediction head to perform sentiment prediction, and generate high-level predicted sentiment scores of the text, audio, and visual.

[0028] Positive and negative sentiment membership degrees are constructed for the high-level predicted sentiment scores of the text, audio, and visual respectively.

[0029] For each modality, the positive and negative sentiment membership degrees of the association are used to perform difference calculations, followed by absolute value calculations to determine the corresponding text sentiment polarity confidence, audio sentiment polarity confidence, and visual sentiment polarity confidence.

[0030] Further, the modality-shared features include text-shared features, audio-shared features, and visual-shared features; the modality sentiment polarity confidence score is constructed based on the modality high-level predicted sentiment scores of each modality high-level feature, and the cross-modality shared semantic consistency score of the modality-shared aligned features to which each modality-shared feature belongs is combined to construct a corresponding modality scaling factor to scale each modality high-level feature, determining the corresponding modality-scaled high-level feature, including:

[0031] Based on the modal high-level predicted sentiment scores of each modality high-level feature, an associated modal sentiment polarity confidence score is constructed.

[0032] A shared semantic alignment projection layer is used to align and project the text-shared features, the audio-shared features, and the visual-shared features respectively, to determine the text-shared alignment features, audio-shared alignment features, and visual-shared alignment features;

[0033] After calculating and linearly normalizing pairwise cosine similarities based on the text-shared alignment features, audio-shared alignment features, and visual-shared alignment features, the similarity is limited by the clamp function. The scope is defined to determine text-audio alignment similarity, text-visual alignment similarity, and audio-visual alignment similarity.

[0034] Based on the text-audio alignment similarity, the text-visual alignment similarity, and the audio-visual alignment similarity, cross-modal mean calculation is performed modally to determine the corresponding text cross-modal shared semantic consistency score, audio cross-modal shared semantic consistency score, and visual cross-modal shared semantic consistency score.

[0035] The corresponding modal reliability score is determined by multiplying the text cross-modal shared semantic consistency score, the audio cross-modal shared semantic consistency score, and the visual cross-modal shared semantic consistency score with the associated modal sentiment polarity confidence.

[0036] After concatenating the modal reliability scores, the corresponding modal reliability weights are determined by mapping using a Softmax activation function with a temperature parameter.

[0037] Determine the modal scaling factor corresponding to each modal reliability weight;

[0038] Linear projection is performed on each modal high-level feature, and after mapping through the Sigmoid activation function, it is multiplied by the corresponding modal scaling factor to determine the corresponding modal scaling high-level feature.

[0039] Furthermore, the process of determining the modal scaling factor includes:

[0040] ;

[0041] In the formula, Modal index , Representing modes scaling factor, Indicates the scaling intensity factor. Representing modes Reliability weight.

[0042] Furthermore, the trained sentiment analysis model performs full-modal predictions on the counterfactual multimodal sentiment inputs of the training text, the verification multimodal sentiment inputs, and the verification text, respectively, to determine the mean counterfactual prediction score of the training text and the optimal combination of correction parameters, including:

[0043] Construct training text counterfactual multimodal sentiment input for training multimodal sentiment input and verification text counterfactual multimodal sentiment input for verifying multimodal sentiment input;

[0044] The trained sentiment analysis model is used to perform full-modal predictions on the counterfactual multimodal sentiment inputs of the training text, the validation multimodal sentiment inputs, and the validation text counterfactual multimodal sentiment inputs, and the corresponding full-modal prediction sentiment scores of the training text, the validation text, and the validation text counterfactual multimodal prediction sentiment scores are determined.

[0045] Calculate the mean of the counterfactual prediction scores of the training texts in the full modality of counterfactual prediction sentiment scores.

[0046] The difference between the full modal prediction sentiment score of the counterfactual text and the mean counterfactual prediction score of the training text is calculated to determine the corresponding counterfactual bias of the verification text.

[0047] The correction gate corresponding to the verified full-modal predicted sentiment score is calculated using multiple preset gate modes and preset boundary gate scale parameters respectively;

[0048] By combining the counterfactual bias of the verification text, the correction gating, the full-modal prediction sentiment score of the verification text, and the counterfactual correction strength, the corresponding final prediction sentiment score of the verification text is determined.

[0049] Based on the total verification loss value and prediction performance evaluation index of the final sentiment prediction score, the optimal combination of correction parameters is selected, consisting of the gating mode, boundary gating scale parameter, and text counterfactual correction intensity.

[0050] Furthermore, the process of determining the total loss value includes:

[0051] ;

[0052] In the formula, Indicates the total loss. This indicates the main task regression loss. The weights represent the boundary-weighted binary classification loss. Indicates the boundary-weighted classification loss. This represents the loss of semantic consistency. Indicates the reconstruction loss. Represents the loss of shared semantic similarity. Representation sharing—specific orthogonal loss;

[0053] The regression loss for the main task includes:

[0054] ;

[0055] In the formula, This represents the loss of the full-modal prediction task. Indicates the shared prediction task loss. This indicates the loss of the high-level text prediction task. This indicates that the audio management team predicted the task loss. This indicates the loss of vision high-level prediction tasks. This represents the full-modal predicted sentiment score. This represents the shared predicted sentiment score, which is the input of the shared fusion feature and the output of the shared branch prediction head. This indicates the high-level predicted sentiment score of the text. This indicates the audio high-level prediction sentiment score. This indicates the visual high-level prediction sentiment score. Indicates the true sentiment score, This represents the mean absolute error loss;

[0056] The boundary-weighted classification loss includes:

[0057] ;

[0058] In the formula, This indicates taking the average. Indicates the weights of boundary samples. Represents the original boundary classification loss. Represents the cross-entropy loss in binary classification. Indicates a binary sentiment tag. Indicates the enhancement intensity of boundary samples. This represents the natural exponential function. Indicates the boundary attenuation scale. Indicates an indicator function;

[0059] The reconstruction loss includes:

[0060] ;

[0061] In the formula, Modal index , Represents text modality, Indicates audio modality, Representing visual modality, Representing modes The reconstructed recalibration features Representing modes The recalibration features, Representing modes Its unique characteristics Representing modes Shared features Representing modes The reconstructed decoder;

[0062] Among them, the unique semantic consistency loss includes:

[0063] ;

[0064] In the formula, This represents the mean square error. Representing modes The unique characteristics of reconstruction Representing modes A unique encoder;

[0065] The shared semantic similarity loss includes:

[0066] ;

[0067] In the formula, Indicates hinge loss. Represents a set of shared alignment features. Represents the set of true sentiment scores. Indicates different sample indices, This indicates the number of samples in the current batch. Indicates the first The text of each sample shares alignment features. Indicates the first Audio shared alignment features of individual samples Indicates the first Visually shared alignment features of individual samples Indicates the first The true sentiment score of each sample This indicates taking the maximum value. Indicates the first The true sentiment score of each sample This represents the shared semantic similarity between different modalities of the same sample. This represents the shared semantic similarity between different samples. Indicates the first Sample mode Shared alignment features Indicates the first Sample mode Shared alignment features Indicates the first Sample mode Shared alignment features Modal index , Modal index ;

[0068] Among them, the shared-specific orthogonal loss includes:

[0069] ;

[0070] In the formula, This represents the cosine embedding loss.

[0071] Furthermore, the prediction head is a residual fully connected prediction head, which includes a residual multilayer perceptron and a predictive linear output layer.

[0072] A second aspect of this invention provides a multimodal sentiment analysis system based on channel recalibration and polarity membership consistency gating, comprising:

[0073] The model input module is used to feed the training multimodal sentiment input into the sentiment analysis model to be trained;

[0074] The alignment and recalibration module is used to perform feature alignment to determine text alignment features, audio alignment features, and visual alignment features of the trained multimodal emotion input in a channel-first format, and to perform channel recalibration to determine the corresponding modality recalibration features.

[0075] The shared encoding module is used to determine the modal shared features corresponding to each modal recalibration feature through a shared encoder, and to enhance each modal shared feature through a shared self-attention encoder, and then to construct the shared fusion feature by taking the hidden state at the last time point and splicing them together.

[0076] A unique encoding module is used to extract the modality-specific features corresponding to each modality recalibration feature using a modality-specific encoder, conduct cross-modal interaction through text guidance, and extract the corresponding high-level modality features through a memory encoder.

[0077] The polarity membership consistency gating module is used to construct the associated modal sentiment polarity confidence based on the modal high-level predicted sentiment scores of each modal high-level feature, and to construct the corresponding modal scaling factor to scale each modal high-level feature by combining the cross-modal shared semantic consistency score of the modal shared alignment features of each modal shared feature alignment projection, and to determine the corresponding modal scaling high-level feature.

[0078] The fusion prediction module is used to perform linear projection and Sigmoid activation function mapping on the shared fusion features, transform the shared fusion features, and then concatenate them with the modality-scaled high-level features to construct the final fusion features. The final fusion features are then input into the full-modality prediction head to predict and output the training full-modality prediction sentiment score.

[0079] The model optimization module is used to iteratively optimize the model parameters based on the total training loss value corresponding to the training full-modality predicted sentiment score until the trained sentiment analysis model is determined.

[0080] The correction parameter determination module is used to perform full-modal prediction and text counterfactual inference correction on the training text counterfactual multimodal sentiment input, the verification multimodal sentiment input, and the verification text counterfactual multimodal sentiment input using the trained sentiment analysis model, and to determine the mean counterfactual prediction score of the training text and the optimal combination of correction parameters;

[0081] The sentiment prediction module is used to determine the target full-modal predicted sentiment score corresponding to the multimodal sentiment input to be analyzed and the target text counterfactual full-modal predicted sentiment score of the counterfactual multimodal sentiment input to be analyzed by using a trained sentiment analysis model. The module combines the mean of the training text counterfactual predicted scores and the optimal correction parameter combination to perform text counterfactual reasoning correction on the target full-modal predicted sentiment score to determine the final target sentiment prediction score.

[0082] Among them, text counterfactual multimodal emotional input refers to multimodal emotional input that keeps the text emotional input unchanged while setting the audio emotional input and visual emotional input to zero.

[0083] As can be seen from the above technical solutions, the present invention has the following advantages:

[0084] The above-mentioned scheme of the present invention provides a multimodal sentiment analysis method based on channel recalibration and polarity membership consistency gating, comprising: feeding training multimodal sentiment input into a sentiment analysis model to be trained; performing feature alignment on the training multimodal sentiment input in a channel-first format to determine text alignment features, audio alignment features, and visual alignment features, and performing channel recalibration to determine the corresponding modality recalibration features; determining the modality shared features corresponding to each modality recalibration feature through a shared encoder, and enhancing each modality shared feature through a shared self-attention encoder, and then concatenating the hidden states at the last time point to construct shared fusion features; extracting the modality-specific features corresponding to each modality recalibration feature using a modality-specific encoder, conducting cross-modal interaction through text guidance, and extracting the corresponding modality high-level features through a memory encoder; constructing associated modality sentiment polarity confidence based on the modality high-level predicted sentiment scores of each modality high-level feature, and constructing a corresponding modality scaling factor to scale each modality high-level feature by combining the cross-modality shared semantic consistency score of the modality shared alignment feature to which the modality shared alignment feature belongs in the alignment projection of each modality shared feature. The process involves: identifying the corresponding modality-scaled high-level features; transforming the shared fusion features through linear projection and Sigmoid activation function mapping, concatenating them with the modality-scaled high-level features to construct the final fusion feature, and inputting it into the full-modality prediction head to predict and output the training full-modality predicted sentiment score; iteratively optimizing the model parameters based on the total training loss value corresponding to the training full-modality predicted sentiment score until a well-trained sentiment analysis model is determined; using the trained sentiment analysis model, performing full-modality prediction and text counterfactual inference correction on the training text counterfactual multimodal sentiment input, the validation multimodal sentiment input, and the validation text counterfactual multimodal sentiment input respectively, determining the mean of the training text counterfactual prediction score and the optimal combination of correction parameters; using the trained sentiment analysis model, determining the target full-modality predicted sentiment score corresponding to the multimodal sentiment input to be analyzed and the target text counterfactual full-modality predicted sentiment score of the text counterfactual multimodal sentiment input to be analyzed, and combining the mean of the training text counterfactual prediction score and the optimal combination of correction parameters to perform text counterfactual inference correction on the target full-modality predicted sentiment score to determine the final target sentiment prediction score. Based on the above scheme, channel recalibration is used to enhance or suppress different modal feature channels. Based on the polarity membership consistency gating mechanism, modal reliability estimation is performed in the fusion stage. Text counterfactual correction is performed on the full modality prediction results, which helps to enhance the sentiment analysis model's ability to handle noisy channels, low-reliability modalities and boundary samples, thereby improving the reliability of sentiment analysis. Attached Figure Description

[0085] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 This is a flowchart illustrating the steps of a multimodal sentiment analysis method based on channel recalibration and polarity membership consistency gating, as provided in Embodiment 1 of the present invention.

[0087] Figure 2 This is a schematic diagram of the framework of the sentiment analysis model provided in Embodiment 1 of the present invention;

[0088] Figure 3 This is a structural block diagram of a multimodal sentiment analysis system based on channel recalibration and polarity membership consistency gating, provided in Embodiment 2 of the present invention. Detailed Implementation

[0089] This invention provides a multimodal sentiment analysis method and system based on channel recalibration and polarity membership consistency gating, which improves the reliability of sentiment analysis during multimodal sentiment analysis.

[0090] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0091] Please see Figure 1 The present invention provides a multimodal sentiment analysis method based on channel recalibration and polarity membership consistency gating, comprising:

[0092] Step 101: Input the training multimodal sentiment data into the sentiment analysis model to be trained.

[0093] Multimodal sentiment input refers to the input features when a sentiment analysis model performs multimodal sentiment analysis. In this embodiment, the input features at different stages are explicitly distinguished by terms such as "training", "validation" and "to be analyzed".

[0094] It should be noted that this embodiment establishes a platform for multimodal sentiment analysis, such as... Figure 2The sentiment analysis model shown is trained using multimodal sentiment inputs after its construction to optimize model parameters; it is understood that multimodal includes text modality. Audio modality and visual modality Correspondingly, the multimodal sentiment input for each sample includes text sentiment input. Audio emotional input and visual emotional input , specifically:

[0095] Optionally, you can configure the loading of a RoBERTa pre-trained language model to extract text features from text sentiment data and determine the text sentiment input. In order to obtain semantic features that can characterize the text context dependencies, and provide linguistic discriminative information for subsequent sentiment analysis tasks;

[0096] Audio Emotional Input Visual and emotional input The audio and video temporal features, after preprocessing and time alignment of audio and visual sentiment data, are both feature tensors that can be directly processed by the model. For example, in one implementation of this embodiment, audio features can be obtained by extracting acoustic descriptive features from the original audio signal after frame-by-frame processing, such as fundamental frequency, energy, spectral features, or COVAREP features. Visual features can be obtained by extracting facial visual features from the original video after frame sampling, face detection, and face alignment, such as facial keypoints, facial action units, head pose, or OpenFace features. These frame-level features can be aligned according to timestamps and unified into a fixed-length temporal feature tensor through truncation, padding, or average pooling. In another implementation of this embodiment, neural networks such as LSTM, bidirectional LSTM, one-dimensional temporal convolution, or temporal Transformer can be used to further encode and extract the frame-level features. Alternatively, preprocessed and aligned audio and visual temporal features from a publicly available multimodal sentiment analysis dataset can be directly used as model input; this embodiment does not impose any limitations on this approach.

[0097] Step 102: Perform feature alignment on the training multimodal emotion input in a channel-first format to determine text alignment features, audio alignment features, and visual alignment features, and perform channel recalibration to determine the corresponding modality recalibration features.

[0098] It should be noted that after converting multimodal sentiment input into a channel-first representation and mapping it to a unified dimensional space, text alignment features can be obtained. Audio alignment features and visual alignment features To provide a consistent representation basis, channel recalibration operations are then performed to enhance sentiment-related channels in text, audio, and visual modalities and suppress redundant or low-quality channels, resulting in recalibrated features for each modality, including text recalibration features. Audio recalibration features and visual recalibration features This improves the input quality for subsequent semantic modeling.

[0099] In one specific embodiment of this example, step 102 includes the following sub-steps:

[0100] S1021. After converting the text sentiment input, audio sentiment input, and visual sentiment input into a channel-first format through a transpose layer, the input is then fed into a one-dimensional convolutional projection layer for feature alignment to determine the text alignment features, audio alignment features, and visual alignment features.

[0101] It should be noted that in this embodiment, the emotional inputs of the three modalities are converted into a channel-first form through a transpose layer to adapt to the subsequent one-dimensional convolutional projection operation. This conversion process can be represented as follows: , , , This indicates a text channel priority feature. Indicates audio channel priority characteristics. Indicates visual channel priority features, This represents the transpose layer; subsequently, a one-dimensional convolutional projection layer is used to map the original feature dimensions of the three modalities to a unified hidden dimension to obtain aligned features. This feature alignment process can be represented as: , , In the formula, One-dimensional convolutional projection layers represent text, audio, and visual modalities. It is understandable that if the original feature dimension of a certain modality is equal to the unified hidden dimension, the channel-priority features of the original transpose can be directly used as the projection result.

[0102] S1022. The residual one-dimensional channel recalibration module is used to perform channel recalibration on the text alignment features, audio alignment features and visual alignment features respectively, and to determine the text recalibration features, audio recalibration features and visual recalibration features.

[0103] It should be noted that, to improve the feature quality before entering the semantic decoupling module, this embodiment introduces a residual one-dimensional channel recalibration module SE-1D (Squeeze-and-Excitation 1D) to recalibrate the alignment features of each modality projection. This module adopts a one-dimensional squeeze excitation structure. First, it obtains the global response of each channel through global average pooling. Then, it generates channel weights through two layers of linear transformation and activation functions, and uses learnable residual coefficients to control the channel recalibration intensity. For the alignment features of any modality... (in ), specifically:

[0104] First, global statistics for each channel are obtained through one-dimensional global average pooling: In the formula, Modal index , Representing modes Channel statistics, This represents a one-dimensional global average pooling layer. Representing modes Alignment features;

[0105] Subsequently, the channel statistics were... By inputting the first linear layer and the ReLU activation function, we obtain the compressed channel representation: In the formula, Represents the ReLU activation function. Representing modes The first calibration can be learned projection matrix. Representing modes Characterization of the compressed channel;

[0106] Next, the second linear layer and the sigmoid function are input to obtain the channel weights: In the formula, This represents the Sigmoid activation function. Representing modes The second calibration learnable projection matrix, Representing modes The channel weights; preferably, the compression ratio of the module is... Hidden dimension ( express (feature dimension), thereby avoiding excessively small hidden layers when the number of channels is small;

[0107] Furthermore, channel weights are used to perform channel-level scaling on the original alignment features: In the formula, , Representing modes Scaling alignment features, This indicates multiplication by channel;

[0108] Finally, feature processing is performed using a residual channel recalibration method: In the formula, Representing modes The recalibration features, Representing modes Learnable residual coefficients;

[0109] In summary, the processing steps of the residual one-dimensional channel recalibration module include: .

[0110] Step 103: Determine the modal shared features corresponding to each modal recalibration feature through a shared encoder, and enhance the features of each modal shared feature through a shared self-attention encoder. Then, take the hidden state at the last time point and splice it to construct the shared fusion feature.

[0111] It should be noted that, through the shared encoder Relabel the text features respectively Audio recalibration features and visual recalibration features Feature processing is performed to generate corresponding modality-shared features, including text-shared features. Audio sharing features Shared features with vision After applying self-attention enhancement to text-shared features, audio-shared features, and visual-shared features respectively using a shared self-attention encoder, the representation of the last position of each output sequence is taken, and the three are concatenated to construct a shared fusion feature. : , , , In the formula, A shared self-attention encoder representing text, audio, and visual modalities. This represents text-sharing enhancement features, audio-sharing enhancement features, and visual-sharing enhancement features. This represents the hidden state at the last time point of the text-sharing augmented features, audio-sharing augmented features, and visual-sharing augmented features.

[0112] Step 104: After extracting the modality-specific features corresponding to the modality recalibration features using a modality-specific encoder, cross-modal interaction is conducted through text guidance, and the corresponding high-level modality features are extracted through a memory encoder.

[0113] It should be noted that after extracting the unique information of each modality through the modality-specific encoder, a high-level cross-modal interaction method dominated by text is adopted in the modality-specific semantic space, and a high-level unique representation of text, audio and visual modalities that is more suitable for sentiment prediction is obtained through the memory encoder.

[0114] In one specific embodiment of this example, step 104 includes the following sub-steps:

[0115] S1041. Input the text recalibration features, audio recalibration features, and visual recalibration features into the corresponding modality-specific encoders to determine the text-specific features, audio-specific features, and visual-specific features.

[0116] It should be noted that the modality-specific encoder extracts the corresponding unique features based on the recalibration features of each modality: , , In the formula, This refers to text-specific encoders, audio-specific encoders, and visual-specific encoders. It represents text-specific features, audio-specific features, and visual-specific features.

[0117] S1042. Use text-specific features to guide audio-specific features and visual-specific features to perform cross-modal attention operations, and output the corresponding audio interaction features and visual interaction features.

[0118] S1043. Input the text-specific features, audio interaction features, and visual interaction features into their respective memory encoders, and take the hidden state at the last time point as the text high-level features, audio high-level features, and visual high-level features.

[0119] It should be noted that for the text modality, the text-specific features are directly input into the memory encoder for memorization encoding; for the audio and visual modalities, cross-modal high-level modeling is first achieved through text-guided cross-modal attention operations, and then the interaction representation is input into the audio and visual memory encoders for feature processing; the representation of the last position of the output sequence obtained by the memory encoder for each modality is used as the high-level feature of the corresponding modality, including the high-level text features. High-level audio characteristics and visual high-level features .

[0120] Understandably, shared encoders and modality-specific encoders can divide each modality into two relatively independent information domains: a modality-specific information domain and a modality-shared information domain. The modality-specific information domain is used to retain the unique emotional expression information of each modality, while the modality-shared information domain is used to model the emotional semantics that exist across modalities. In this way, the model can utilize both modal commonalities and modal differences, avoiding information mixing caused by simple splicing or direct fusion.

[0121] Step 105: Construct the associated modal sentiment polarity confidence based on the modal high-level predicted sentiment scores of each modality's high-level features, and combine the cross-modal shared semantic consistency scores of the modal shared alignment features to which each modality's shared features are aligned and projected to construct the corresponding modal scaling factor to scale each modality's high-level features, and determine the corresponding modal scaling high-level features.

[0122] Modal high-level prediction sentiment score refers to the sentiment score predicted based on the high-level features of that modality.

[0123] Modal sentiment polarity confidence refers to the degree of credibility in determining whether a modality's features belong to a certain type of sentiment polarity.

[0124] Cross-modal shared semantic consistency score refers to the index value that measures the degree of consistency between the shared alignment semantics of a certain modality and the shared alignment semantics of other modalities.

[0125] It should be noted that, in order to address the lack of explicit modality reliability estimation in the multimodal fusion stage, this embodiment comprehensively considers two factors: first, the sentiment polarity confidence of the high-level prediction results of each modality; and second, the consistency between the shared semantic alignment vectors of each modality. In this way, only when a modality simultaneously meets the two conditions of "its own sentiment prediction is relatively clear" and "its shared alignment semantics are relatively consistent with other modalities" will it obtain a modality scaling factor with a higher reliability weight for feature scaling to be applied to subsequent feature fusion. This design has a more explicit interpretation of reliability.

[0126] In one specific embodiment of this example, step 105 includes the following sub-steps:

[0127] S1051. Construct the associated modal sentiment polarity confidence based on the modal high-level predicted sentiment scores of each modality's high-level features.

[0128] In a more specific embodiment of this example, sub-step S1051 includes:

[0129] S11. Input the high-level features of text, audio and visual respectively into the associated high-level branch prediction head to perform sentiment prediction, and generate the high-level prediction sentiment score of text, the high-level prediction sentiment score of audio and the high-level prediction sentiment score of visual.

[0130] S12. Construct positive and negative sentiment memberships for high-level predicted sentiment scores for text, audio, and visual content, respectively.

[0131] S13. For each modality, the positive and negative sentiment membership degrees of the association are used to perform difference calculations, followed by absolute value calculations to determine the corresponding text sentiment polarity confidence, audio sentiment polarity confidence, and visual sentiment polarity confidence.

[0132] It should be noted that this embodiment determines the modal high-level prediction sentiment score based on the high-level features of each modality using the high-level branch prediction head, including the text high-level prediction sentiment score. Audio high-level predictive sentiment score Visual high-level predictive sentiment score To provide a basis for calculating the confidence level of modal affective polarity;

[0133] For any modality, the high-level predicted sentiment score (in The definition of positive sentiment membership includes The definition of negative emotional membership includes The definition of sentiment polarity confidence includes In the formula, Representing modes Positive emotional affiliation This represents the Sigmoid activation function. This represents the membership slope coefficient. Representing modes High-level predicted sentiment score, Representing modes Negative emotional membership degree Representing modes The confidence score of sentiment polarity; in the confidence score of sentiment polarity, when the predicted score of a certain modality is far from the sentiment boundary, the difference between positive sentiment membership and negative sentiment membership is large, indicating that the sentiment polarity judgment of the modality is relatively clear. When the predicted score is close to the zero boundary, the positive sentiment membership and negative sentiment membership are close, indicating that the sentiment judgment of the modality is highly uncertain.

[0134] S1052. Use a shared semantic alignment projection layer to align and project the text-shared features, audio-shared features, and visual-shared features respectively, and determine the text-shared alignment features, audio-shared alignment features, and visual-shared alignment features.

[0135] S1053. Based on the text shared alignment features, audio shared alignment features, and visual shared alignment features, pairwise cosine similarity is calculated and linearly normalized, then the similarity is limited by the clamp function. The scope is defined to determine text-audio alignment similarity, text-visual alignment similarity, and audio-visual alignment similarity.

[0136] It should be noted that pairwise cosine similarity is calculated for the shared alignment features of each modality, and then linear normalization mapping is performed:

[0137] ;

[0138] In the formula, Represents the normalized text-to-audio cosine similarity. This indicates that text shares alignment features. This indicates audio sharing alignment features. Represents the normalized visual cosine similarity of text. Indicates visually shared alignment features. Represents the normalized audio-visual cosine similarity. Represents the cosine function;

[0139] The normalized cosine similarity is truncated and restricted to a range using the clamp function. Within the range, the corresponding alignment similarity is obtained.

[0140] S1054. Based on the text-audio alignment similarity, text-visual alignment similarity, and audio-visual alignment similarity, perform cross-modal mean calculation for each modality to determine the corresponding text cross-modal shared semantic consistency score, audio cross-modal shared semantic consistency score, and visual cross-modal shared semantic consistency score.

[0141] It should be noted that the average consistency of each mode with the other two modes is calculated:

[0142] ;

[0143] In the formula, This represents the semantic consistency score shared across text modalities. This represents the semantic consistency score for audio cross-modal sharing. This represents the semantic consistency score for visual cross-modal sharing.

[0144] S1055. The corresponding modal reliability scores are determined by multiplying the text cross-modal shared semantic consistency score, the audio cross-modal shared semantic consistency score, and the visual cross-modal shared semantic consistency score with the associated modal sentiment polarity confidence.

[0145] It should be noted that the reliability score for each modality is obtained by multiplying the modal sentiment polarity confidence score and the cross-modal shared semantic consistency score:

[0146] ;

[0147] In the formula, Indicates the confidence level of the sentiment polarity of the text. Indicates the confidence level of audio sentiment polarity. Indicates the confidence level of visual emotional polarity. Indicates the text reliability score. This indicates the audio reliability score. This indicates the visual reliability score.

[0148] S1056. After concatenating the reliability scores of each mode, the corresponding modal reliability weights are determined by mapping using a Softmax activation function with temperature parameters.

[0149] It should be noted that the process of determining the modal reliability weights can be expressed as:

[0150] ;

[0151] In the formula, Indicates the text reliability weight. Indicates audio reliability weight. Indicates visual reliability weight. This represents the Softmax activation function. Indicates temperature parameter, Indicates splicing.

[0152] S1057. Determine the modal scaling factor corresponding to the reliability weight of each modality.

[0153] It should be noted that after obtaining the modal reliability weights, this embodiment converts them into modal scaling factors for adaptive scaling of text, audio, and visual high-level specific representations.

[0154] In a more specific embodiment of this example, the process of determining the modal scaling factor includes:

[0155] ;

[0156] In the formula, Modal index , Representing modes scaling factor, Indicates the scaling intensity factor. Representing modes The reliability weight; understandably, to ensure that the modal scaling factor is always positive and conforms to the description of "mild scaling", the following is limited. Preferably, it can be taken ;

[0157] When the reliability weight of a certain mode is equal to 1 / 3 of the uniform weight of the three modes, the scaling factor is 1; when the weight of a certain mode is greater than 1 / 3, the representation of that mode is enhanced; when the weight of a certain mode is less than 1 / 3, the representation of that mode is suppressed. This design adopts a mild scaling method instead of directly discarding a certain mode, thereby avoiding information loss when there is uncertainty in the estimation of mode weights.

[0158] S1058. Perform linear projection on the high-level features of each modality, and after mapping through the Sigmoid activation function, multiply them by the corresponding modality scaling factor to determine the corresponding modality-scaled high-level features.

[0159] It should be noted that the above process can be represented as: , , , In the formula, Projection matrices representing text, audio, and visual modalities. This represents the Sigmoid activation function. Represents high-level features of text transformation. Represents high-level features of audio transformation. Represents high-level features of visual transformation. Representing modes The transformation of high-level features, Representing modes High-level scaling features (including high-level text scaling features) Audio scaling high-level features and visual scaling of high-level features ).

[0160] Step 106: After transforming the shared fusion features by linear projection and Sigmoid activation function mapping, the transformed shared fusion features are concatenated with the high-level features of each modality scaling to construct the final fusion features, and input into the full-modality prediction head to predict the output and train the full-modality prediction sentiment score.

[0161] It should be noted that the full modality predicts the sentiment score. This refers to the full-modal fact prediction result when text, audio, and visual modalities are involved in the prediction. This result can be directly used for sentiment intensity regression and sentiment polarity judgment, or it can be used as input for subsequent text counterfactual reasoning correction.

[0162] Step 107: Iteratively optimize the model parameters based on the total training loss value corresponding to the predicted sentiment scores of the full modality, until the trained sentiment analysis model is determined.

[0163] It should be noted that the total training loss value corresponding to the predicted sentiment scores of the whole modality is determined to provide feedback on the model training status. When the training stopping condition is not met, the model parameters of the sentiment analysis model to be trained are optimized and iterated according to the total training loss value until the total training loss value converges or the training stopping condition such as reaching the maximum number of iterations is met. Then, the trained sentiment analysis model is determined.

[0164] In one specific embodiment of this example, the process of determining the total loss value includes:

[0165] ;

[0166] In the formula, Indicates the total loss. This indicates the main task regression loss. The weights represent the boundary-weighted binary classification loss. Indicates the boundary-weighted classification loss. This represents the loss of semantic consistency. Indicates the reconstruction loss. Represents the loss of shared semantic similarity. This represents a shared-specific orthogonal loss. Understandably, the weight design can be adaptively adjusted based on the model's predictive performance, but it must satisfy the principle that the primary task regression loss is dominant and the auxiliary loss plays a constraining role, with the primary task regression loss having the largest weight.

[0167] The regression loss for the main task includes:

[0168] ;

[0169] In the formula, This represents the loss of the full-modal prediction task. Indicates the shared prediction task loss. This indicates the loss of the high-level text prediction task. This indicates that the audio management team predicted the task loss. This indicates the loss of vision high-level prediction tasks. This represents the full-modal predicted sentiment score. This represents the shared predicted sentiment score, which is the input of shared fusion features to the output of the shared branch prediction head. (Understandably, this shared branch prediction result can exert task supervision on the shared semantic representation, enabling the shared semantic space to not only have cross-modal consistency but also sentiment discrimination capabilities.) This indicates the high-level predicted sentiment score of the text. This indicates the audio high-level prediction sentiment score. This indicates the visual high-level prediction sentiment score. Indicates the true sentiment score, This represents the mean absolute error loss;

[0170] The boundary-weighted classification loss includes:

[0171] ;

[0172] In the formula, This indicates taking the average. Indicates the weights of boundary samples. Represents the original boundary classification loss. Represents the cross-entropy loss in binary classification. Indicates binary sentiment labels (when) hour Expressing positive emotions, when hour (Indicates non-positive or negative emotions) Indicates the enhancement intensity of boundary samples. This represents the natural exponential function. Indicates the boundary attenuation scale. This represents the indicator function; understandably, this loss gives higher training weights to samples with label absolute values ​​close to zero, thereby improving the model's ability to discriminate and classify near the positive and negative sentiment boundaries.

[0173] The reconstruction loss includes:

[0174] ;

[0175] In the formula, Modal index , Represents text modality, Indicates audio modality, Representing visual modality, Representing modes The reconstructed recalibration features Representing modes The recalibration features, Representing modes Its unique characteristics Representing modes Shared features Representing modes The reconstruction decoder; it is understood that, in order to ensure that the shared semantic representation and the modality-specific semantic representation can jointly retain the effective information in the original modality, this embodiment sets constraints on modality reconstruction;

[0176] Among them, the unique semantic consistency loss includes:

[0177] ;

[0178] In the formula, This represents the mean square error. Representing modes The unique characteristics of reconstruction Representing modes The specific encoder; it is understood that, in order to ensure that the modality-specific semantic representation remains consistent before and after reconstruction, this embodiment re-inputs the reconstructed and recalibrated features into the corresponding modality-specific encoder to obtain the reconstructed specific semantic representation. This enables the modality-specific semantic representation to retain recoverable modality discrimination information and avoids the specific information being weakened or lost in the sharing-specific decoupling process.

[0179] The shared semantic similarity loss includes:

[0180] ;

[0181] In the formula, Indicates hinge loss. Represents a set of shared alignment features. Represents the set of true sentiment scores. Indicates different sample indices, This indicates the number of samples in the current batch. Indicates the first The text of each sample shares alignment features. Indicates the first Audio shared alignment features of individual samples Indicates the first Visually shared alignment features of individual samples Indicates the first The true sentiment score of each sample This indicates taking the maximum value. Indicates the first The true sentiment score of each sample This represents the shared semantic similarity between different modalities of the same sample. This represents the shared semantic similarity between different samples (which can be understood as either a comparison within the same modality between different samples or a comparison across different modalities). Indicates the first Sample mode Shared alignment features Indicates the first Sample mode Shared alignment features Indicates the first Sample mode Shared alignment features Modal index , Modal index Understandably, this similarity constraint can encourage cross-modal shared representations of similar sentiment labels to become more similar, while maintaining the distinction between shared representations of different sentiment labels.

[0182] Among them, the shared-specific orthogonal loss includes:

[0183] ;

[0184] In the formula, The cosine embedding loss is used to represent the loss. It is understood that, to further improve the separation between shared semantics and modality-specific semantics, this embodiment uses cosine embedding loss to constrain shared and specific representations within the same modality to maintain low similarity in the representation space. The tag " "This indicates the desire for the two sets of representations to be as dissimilar as possible in direction."

[0185] In one specific embodiment of this example, the prediction heads such as the shared branch prediction head, the high-level branch prediction head, and the full-modal prediction head are residual fully connected prediction heads, including a residual multilayer perceptron and a predictive linear output layer. Specifically, the process of the prediction head outputting the predicted sentiment score can be represented as follows: , In the formula, Indicates the predicted input features, This represents different multilayer perceptron projection matrices. Represents the ReLU activation function. This indicates the Dropout layer. This represents the residual sensing output feature. Represents the linear output projection matrix. This indicates the predicted sentiment score.

[0186] Step 108: Using the trained sentiment analysis model, perform full-modal prediction and text counterfactual inference correction on the training text counterfactual multimodal sentiment input, the verification multimodal sentiment input, and the verification text counterfactual multimodal sentiment input, respectively, and determine the mean counterfactual prediction score of the training text and the optimal combination of correction parameters.

[0187] It should be noted that after the model training is completed, this embodiment also considers using text counterfactual information to perform posterior correction on the full modality prediction. During the correction process, this embodiment uses the average score of the training text counterfactual prediction as a reference benchmark for the inference stage. At the same time, it searches for correction parameter combinations through the validation set and selects the optimal combination for prediction in the inference stage. It can be understood that text counterfactual inference correction (TCIC) does not participate in the training backpropagation and does not change the model parameters of the sentiment analysis model. Instead, it performs additional inference and post-processing correction on the basis of the trained model.

[0188] In one specific embodiment of this example, step 108 includes the following sub-steps:

[0189] S1081. Construct training text counterfactual multimodal sentiment input for training multimodal sentiment input and verification text counterfactual multimodal sentiment input for verifying multimodal sentiment input.

[0190] Textual counterfactual multimodal sentiment input refers to multimodal sentiment input that keeps the textual sentiment input unchanged while setting the audio and visual sentiment inputs to zero. This embodiment explicitly distinguishes different stages of textual counterfactual multimodal sentiment input through "training," "validation," and "to be analyzed."

[0191] It should be noted that corresponding textual counterfactual multimodal sentiment inputs are constructed from the training multimodal sentiment inputs in the training set and the validation multimodal sentiment inputs in the validation set.

[0192] S1082. Using the trained sentiment analysis model, perform full-modal predictions on the counterfactual multimodal sentiment input of the training text, the verification multimodal sentiment input, and the verification text counterfactual multimodal sentiment input, respectively, and determine the corresponding training text counterfactual full-modal prediction sentiment score, verification full-modal prediction sentiment score, and verification text counterfactual full-modal prediction sentiment score.

[0193] It should be noted that the text counterfactual multimodal sentiment input prediction uses the same trained sentiment analysis model, rather than retraining a text unimodal model. This approach can estimate the sentiment shift when the text modality acts alone on the final prediction while keeping the model parameters consistent.

[0194] S1083. Calculate the mean of the counterfactual prediction scores of the training text in the full modality of the counterfactual prediction sentiment score.

[0195] It should be noted that, in order to avoid the overall distribution shift caused by directly using text counterfactual multimodal sentiment input prediction, this embodiment uses the mean of text counterfactual predictions in the training set as a reference benchmark:

[0196] ;

[0197] In the formula, This represents the mean counterfactual prediction score of the training text. Indicates sample index , This represents the number of samples in the training set. Indicates the first The counterfactual full-modal prediction sentiment score of the training text of each training sample.

[0198] S1084. The difference between the full modal prediction sentiment score of the counterfactual text and the mean of the counterfactual prediction score of the training text is calculated to determine the corresponding counterfactual bias of the verification text.

[0199] It should be noted that text counterfactual bias represents the deviation of the current sample's text counterfactual prediction from the average level of the training set's text counterfactual predictions.

[0200] S1085. The correction gate corresponding to the full modality predicted sentiment score is calculated and verified by using multiple preset gating modes and preset boundary gating scale parameters.

[0201] It should be noted that this embodiment has multiple preset gating modes and boundary gating scale parameters. The corresponding correction gating is generated based on the boundary gating scale parameters and the full modality predicted sentiment score according to the gating mode.

[0202] In a more specific implementation of this embodiment, the preset gating modes include:

[0203] Global mode: ;

[0204] Soft boundary mode:

[0205] ;

[0206] Hard boundary mode:

[0207] ;

[0208] Linear boundary mode:

[0209] ;

[0210] In the formula, Indicates calibration gating. Indicates the boundary gating scale parameter. This represents the truncation function; it is understandable that when When the value is close to 0, the sample is near the boundary between positive and negative sentiment; soft boundary patterns and linear boundary patterns will give larger correction weights. When the value is far from 0, it indicates that the model has a clearer judgment on positive and negative polarity, and the correction weight decreases.

[0211] S1086. Comprehensively verify text counterfactual bias, correct gating, verify full-modal predicted sentiment score and text counterfactual correction strength, and determine the corresponding final verified sentiment prediction score.

[0212] It should be noted that the final sentiment prediction sentiment score The process of determining this includes: In the formula, Indicates the strength of counterfactual correction in the text. This indicates a counterfactual bias in the text.

[0213] S1087. Based on the total verification loss value and prediction performance evaluation index of the final sentiment prediction sentiment score of each verification, the optimal combination of correction parameters is selected, consisting of the gating mode, boundary gating scale parameter and text counterfactual correction intensity.

[0214] It should be noted that during the parameter selection process, several candidate correction parameter combinations are pre-set. Each correction parameter combination includes a gating mode, a boundary gating scale parameter, and a text counterfactual correction intensity. For any candidate correction parameter combination, the corresponding final sentiment prediction score is determined through the validation set. The total validation loss value (the process of determining the total loss value can be found in the training loss value section) and prediction performance evaluation indicators (such as binary classification accuracy, F1 score, etc.) are calculated based on the final sentiment prediction score. The optimal correction parameter combination is selected based on the overall performance. For example, the candidate correction parameter combination with the highest binary classification accuracy can be determined first. If it is not unique, the final parameter combination used in the inference stage is determined by comparing the F1 score and the loss value.

[0215] Step 109: Determine the target full-modal predicted sentiment score corresponding to the multimodal sentiment input to be analyzed and the target text counterfactual full-modal predicted sentiment score of the counterfactual multimodal sentiment input to be analyzed using the trained sentiment analysis model. Combine the mean of the training text counterfactual predicted scores and the optimal correction parameter combination to perform text counterfactual reasoning correction on the target full-modal predicted sentiment score, and determine the final target sentiment predicted sentiment score.

[0216] It should be noted that, in the inference phase, the trained sentiment analysis model is first used with the multimodal sentiment input to be analyzed and the counterfactual multimodal sentiment input of the text to be analyzed as model inputs. The counterfactual multimodal sentiment input of the text to be analyzed refers to the text counterfactual multimodal sentiment input obtained by keeping the text sentiment input unchanged and setting the audio and visual sentiment inputs to zero. Then, following the similarity process in steps 102-106, full-modal sentiment prediction is performed, outputting the corresponding target full-modal predicted sentiment score and the target text of the counterfactual multimodal sentiment input of the text to be analyzed. The counterfactual full-modal prediction sentiment score is determined by the difference between the target text's counterfactual full-modal prediction sentiment score and the mean of the training text's counterfactual prediction scores. A target correction gating is then generated based on the target full-modal prediction sentiment score according to the optimal correction parameter combination. Finally, the target full-modal prediction sentiment score, the target correction gating, the target text's counterfactual bias, and the text counterfactual correction intensity in the optimal correction parameter combination are combined to determine the final target sentiment prediction sentiment score. It can be understood that the final sentiment prediction sentiment score can be used as a sentiment intensity regression result, or it can be converted into a positive or negative sentiment polarity classification result based on whether it is greater than zero.

[0217] This method of using textual counterfactual information for posterior correction of full-modal prediction results does not require retraining the main model. It is particularly suitable for samples where the full-modal prediction results are close to the positive and negative sentiment boundaries, or where there is noise interference in the audio and visual modalities. It can improve the prediction stability of boundary samples and samples affected by audio and visual noise.

[0218] To better illustrate the effectiveness of the method in this embodiment, experiments were conducted on the CMU-MOSI and CMU-MOSEI datasets. As shown in Table 1, the experimental results demonstrate that the method in this embodiment shows certain improvements over existing models.

[0219] Table 1 Experimental Results

[0220]

[0221] In this embodiment of the invention, channel recalibration adaptively enhances sentiment-related channels and suppresses low-contribution channels to improve the quality of input features and the stability of subsequent semantic modeling. Based on a residual one-dimensional channel recalibration module, learnable residual coefficients are used to control the channel recalibration intensity, allowing for the gradual introduction of channel adjustment capabilities while preserving the original projected feature information, thus reducing the disturbance to the baseline model caused by the new module. Through a polarity membership consistency gating mechanism, the sentiment polarity confidence predicted by the higher modality level is combined with cross-modal shared semantic consistency, enabling sample-level modality reliability estimation and avoiding susceptibility to interference from low-confidence modalities. The system addresses several issues: First, it transforms modal reliability into a mild scaling factor, adaptively enhancing or suppressing high-level text, audio, and visual features. This reduces the impact of noisy modalities while preserving complementary information, helping to avoid the problem of hard-selected modalities easily losing multimodal complementary information. Second, it constructs and predicts multimodal sentiment inputs based on text counterfactual prediction scores, using the average training text counterfactual prediction score as a benchmark to calculate text counterfactual bias, and combines this with correction gating for prediction correction, which helps improve the prediction stability for boundary samples and noisy interference samples. Overall, this contributes to improving the accuracy, robustness, and interpretability of multimodal sentiment analysis.

[0222] Please see Figure 3 Embodiment 2 of the present invention provides a multimodal sentiment analysis system based on channel recalibration and polarity membership consistency gating, comprising:

[0223] The model input module 301 is used to feed the training multimodal sentiment input into the sentiment analysis model to be trained;

[0224] The alignment and recalibration module 302 is used to perform feature alignment on the training multimodal sentiment input in a channel-first format to determine text alignment features, audio alignment features and visual alignment features, and to perform channel recalibration to determine the corresponding modality recalibration features.

[0225] The shared encoding module 303 is used to determine the modal shared features corresponding to each modal recalibration feature through the shared encoder, and after enhancing the features of each modal shared feature through the shared self-attention encoder, the hidden state at the last time point is taken and spliced ​​to construct the shared fusion feature.

[0226] The unique encoding module 304 is used to extract the modality-specific features corresponding to the modality recalibration features of each modality using a modality-specific encoder, conduct cross-modality interaction through text guidance, and extract the corresponding high-level modality features through a memory encoder.

[0227] The polarity membership consistency gating module 305 is used to construct the associated modal sentiment polarity confidence based on the modal high-level predicted sentiment scores of each modal high-level feature, and to construct the corresponding modal scaling factor to scale each modal high-level feature by combining the cross-modal shared semantic consistency score of the modal shared alignment feature of each modal shared feature alignment projection, and to determine the corresponding modal scaling high-level feature.

[0228] The fusion prediction module 306 is used to perform linear projection and Sigmoid activation function mapping on the shared fusion features, transform the output of the shared fusion features, and then concatenate them with the high-level features of each modality scaling to construct the final fusion features. The final fusion features are then input into the full-modality prediction head to predict the output and train the full-modality prediction sentiment score.

[0229] The model optimization module 307 is used to iteratively optimize the model parameters based on the total training loss value corresponding to the predicted sentiment score of the full modality training, until the trained sentiment analysis model is determined.

[0230] The correction parameter determination module 308 is used to perform full-modal prediction and text counterfactual inference correction on the training text counterfactual multimodal sentiment input, the verification multimodal sentiment input and the verification text counterfactual multimodal sentiment input respectively through the trained sentiment analysis model, and to determine the mean of the training text counterfactual prediction score and the optimal combination of correction parameters;

[0231] The sentiment prediction module 309 is used to determine the target full-modal predicted sentiment score corresponding to the multimodal sentiment input to be analyzed and the target text counterfactual full-modal predicted sentiment score of the counterfactual multimodal sentiment input to be analyzed by using a trained sentiment analysis model. It combines the mean of the training text counterfactual predicted scores and the optimal correction parameter combination to perform text counterfactual reasoning correction on the target full-modal predicted sentiment score to determine the final target sentiment prediction score.

[0232] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0233] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0234] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0235] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0236] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal sentiment analysis method based on channel recalibration and polarity membership consistency gating, characterized in that, include: The training multimodal sentiment input is fed into the sentiment analysis model to be trained; The trained multimodal emotion input is subjected to feature alignment in a channel-first format to determine text alignment features, audio alignment features, and visual alignment features, and channel recalibration is performed to determine the corresponding modality recalibration features; The modal shared features corresponding to each modal recalibration feature are determined by a shared encoder, and the modal shared features are enhanced by a shared self-attention encoder. The hidden state at the last time point is then concatenated to construct a shared fusion feature. After extracting the modality-specific features corresponding to each modality recalibration feature using a modality-specific encoder, cross-modal interaction is carried out through text guidance, and the corresponding high-level modality features are extracted through a memory encoder. Based on the modal high-level predicted sentiment scores of each modality high-level feature, an associated modal sentiment polarity confidence score is constructed. Then, combined with the cross-modal shared semantic consistency score of the modal shared alignment features of each modality shared feature alignment projection, a corresponding modal scaling factor is constructed to scale each modality high-level feature, and the corresponding modal scaling high-level feature is determined. After linear projection and Sigmoid activation function mapping are performed on the shared fusion features, the transformed shared fusion features are concatenated with the modality-scaled high-level features to construct the final fusion features, and then input into the full-modality prediction head to predict and output the full-modality prediction sentiment score. The model parameters are iteratively optimized based on the total training loss value corresponding to the predicted sentiment score of the full modality until the trained sentiment analysis model is determined. By using a trained sentiment analysis model, full-modal prediction and text counterfactual inference correction are performed on the counterfactual multimodal sentiment inputs of the training text, the verification multimodal sentiment inputs, and the verification text counterfactual multimodal sentiment inputs, respectively, to determine the mean counterfactual prediction score of the training text and the optimal combination of correction parameters; include: Construct training text counterfactual multimodal sentiment input for training multimodal sentiment input and verification text counterfactual multimodal sentiment input for verifying multimodal sentiment input; The trained sentiment analysis model is used to perform full-modal predictions on the counterfactual multimodal sentiment inputs of the training text, the validation multimodal sentiment inputs, and the validation text counterfactual multimodal sentiment inputs, and the corresponding full-modal prediction sentiment scores of the training text, the validation text, and the validation text counterfactual multimodal prediction sentiment scores are determined. Calculate the mean of the counterfactual prediction scores of the training texts in the full modality of counterfactual prediction sentiment scores. The difference between the full modal prediction sentiment score of the counterfactual text and the mean counterfactual prediction score of the training text is calculated to determine the corresponding counterfactual bias of the verification text. The correction gate corresponding to the verified full-modal predicted sentiment score is calculated using multiple preset gate modes and preset boundary gate scale parameters respectively; By combining the counterfactual bias of the verification text, the correction gating, the full-modal prediction sentiment score of the verification text, and the counterfactual correction strength, the corresponding final prediction sentiment score of the verification text is determined. Based on the total verification loss value and prediction performance evaluation index of the final sentiment prediction sentiment score, the optimal combination of correction parameters is selected by prioritizing the gating mode, boundary gating scale parameter and text counterfactual correction intensity. The target full-modal predicted sentiment score and the target text counterfactual full-modal predicted sentiment score of the multimodal sentiment input to be analyzed are determined by the trained sentiment analysis model. The target full-modal predicted sentiment score is then corrected by text counterfactual reasoning by combining the mean of the training text counterfactual predicted scores and the optimal correction parameter combination, and the final target sentiment predicted sentiment score is determined. Among them, text counterfactual multimodal emotional input refers to multimodal emotional input that keeps the text emotional input unchanged while setting the audio emotional input and visual emotional input to zero.

2. The multimodal sentiment analysis method based on channel recalibration and polarity membership consistency gating according to claim 1, characterized in that, The step of performing feature alignment on the trained multimodal emotion input in a channel-first format to determine text alignment features, audio alignment features, and visual alignment features, and performing channel recalibration to determine the corresponding modality recalibration features, includes: After converting text sentiment input, audio sentiment input, and visual sentiment input into channel-first format through a transpose layer, they are input into a one-dimensional convolutional projection layer for feature alignment to determine text alignment features, audio alignment features, and visual alignment features. A residual one-dimensional channel recalibration module is used to perform channel recalibration on the text alignment features, the audio alignment features, and the visual alignment features respectively, to determine the text recalibration features, audio recalibration features, and visual recalibration features; The processing procedure of the residual one-dimensional channel recalibration module includes: ； In the formula, Modal index , Representing modes The recalibration features, Representing modes Alignment features, Representing modes Learnable residual coefficients This indicates multiplication by channel. This represents the Sigmoid activation function. Representing modes The first calibration can be learned projection matrix. Representing modes The second calibration learnable projection matrix, Represents the ReLU activation function. This represents a one-dimensional global average pooling layer.

3. The multimodal sentiment analysis method based on channel recalibration and polarity membership consistency gating according to claim 1, characterized in that, The modal recalibration features include text recalibration features, audio recalibration features, and visual recalibration features; After extracting the modality-specific features corresponding to each modality recalibration feature using a modality-specific encoder, cross-modal interaction is conducted via text guidance, and the corresponding high-level modality features are extracted using a memory encoder, including: The text recalibration features, audio recalibration features, and visual recalibration features are respectively input into the corresponding modality-specific encoder to determine the text-specific features, audio-specific features, and visual-specific features; The text-specific features are used to guide the audio-specific features and the visual-specific features to perform cross-modal attention operations, and the corresponding audio interaction features and visual interaction features are output. The text-specific features, audio interaction features, and visual interaction features are respectively input into their respective memory encoders, and the hidden state at the last time point is taken as the text high-level features, audio high-level features, and visual high-level features.

4. The multimodal sentiment analysis method based on channel recalibration and polarity membership consistency gating according to claim 1, characterized in that, The modal high-level features include text high-level features, audio high-level features, and visual high-level features; The construction of the associated modal sentiment polarity confidence based on the modal high-level predicted sentiment scores of each modality high-level feature includes: The high-level features of the text, the high-level features of the audio, and the high-level features of the visual are respectively input into the associated high-level branch prediction head to perform sentiment prediction, and generate high-level predicted sentiment scores of the text, audio, and visual. Positive and negative sentiment membership degrees are constructed for the high-level predicted sentiment score of the text, the high-level predicted sentiment score of the audio, and the high-level predicted sentiment score of the visual. For each modality, the positive and negative sentiment membership degrees of the association are used to perform difference calculations, followed by absolute value calculations to determine the corresponding text sentiment polarity confidence, audio sentiment polarity confidence, and visual sentiment polarity confidence.

5. The multimodal sentiment analysis method based on channel recalibration and polarity membership consistency gating according to claim 1, characterized in that, The modality-shared features include text-shared features, audio-shared features, and visual-shared features; the modality sentiment polarity confidence score is constructed based on the modality high-level predicted sentiment score of each modality high-level feature, and the cross-modality shared semantic consistency score of the modality-shared alignment feature to which each modality-shared feature belongs is combined to construct a corresponding modality scaling factor to scale each modality high-level feature, and the corresponding modality-scaled high-level feature is determined, including: Based on the modal high-level predicted sentiment scores of each modality high-level feature, an associated modal sentiment polarity confidence score is constructed. A shared semantic alignment projection layer is used to align and project the text-shared features, the audio-shared features, and the visual-shared features respectively, to determine the text-shared alignment features, audio-shared alignment features, and visual-shared alignment features; After calculating and linearly normalizing pairwise cosine similarities based on the text-shared alignment features, audio-shared alignment features, and visual-shared alignment features, the similarity is limited by the clamp function. The scope is defined to determine text-audio alignment similarity, text-visual alignment similarity, and audio-visual alignment similarity. Based on the text-audio alignment similarity, the text-visual alignment similarity, and the audio-visual alignment similarity, cross-modal mean calculation is performed modally to determine the corresponding text cross-modal shared semantic consistency score, audio cross-modal shared semantic consistency score, and visual cross-modal shared semantic consistency score. The corresponding modal reliability score is determined by multiplying the text cross-modal shared semantic consistency score, the audio cross-modal shared semantic consistency score, and the visual cross-modal shared semantic consistency score with the associated modal sentiment polarity confidence. After concatenating the modal reliability scores, the corresponding modal reliability weights are determined by mapping using a Softmax activation function with a temperature parameter. Determine the modal scaling factor corresponding to each modal reliability weight; Linear projection is performed on each modal high-level feature, and after mapping through the Sigmoid activation function, it is multiplied by the corresponding modal scaling factor to determine the corresponding modal scaling high-level feature.

6. The multimodal sentiment analysis method based on channel recalibration and polarity membership consistency gating according to claim 5, characterized in that, The process of determining the modal scaling factor includes: ； In the formula, Modal index , Representing modes scaling factor, Indicates the scaling intensity factor. Representing modes Reliability weight.

7. The multimodal sentiment analysis method based on channel recalibration and polarity membership consistency gating according to claim 1, characterized in that, The process of determining the total loss value includes: ； In the formula, Indicates the total loss. This indicates the main task regression loss. The weights represent the boundary-weighted binary classification loss. Indicates the boundary-weighted classification loss. This represents the loss of semantic consistency. Indicates the reconstruction loss. Represents the loss of shared semantic similarity. Representation sharing—specific orthogonal loss; The regression loss for the main task includes: ； In the formula, This represents the loss of the full-modal prediction task. Indicates the shared prediction task loss. This indicates the loss of the high-level text prediction task. This indicates that the audio management team predicted the task loss. This indicates the loss of vision high-level prediction tasks. This represents the full-modal predicted sentiment score. This represents the shared predicted sentiment score, which is the input of the shared fusion feature and the output of the shared branch prediction head. This indicates the high-level predicted sentiment score of the text. This indicates the audio high-level prediction sentiment score. This indicates the visual high-level prediction sentiment score. Indicates the true sentiment score, This represents the mean absolute error loss; The boundary-weighted classification loss includes: ； In the formula, This indicates taking the average. Indicates the weights of boundary samples. Represents the original boundary classification loss. Represents the cross-entropy loss in binary classification. Indicates a binary sentiment tag. Indicates the enhancement intensity of boundary samples. This represents the natural exponential function. Indicates the boundary attenuation scale. Indicates an indicator function; The reconstruction loss includes: ； In the formula, Modal index , Represents text modality, Indicates audio modality, Representing visual modality, Representing modes The reconstructed recalibration features, Representing modes The recalibration features, Representing modes Its unique characteristics Representing modes Shared features Representing modes The reconstructed decoder; Among them, the unique semantic consistency loss includes: ； In the formula, This represents the mean square error. Representing modes The unique characteristics of reconstruction Representing modes A unique encoder; The shared semantic similarity loss includes: ； In the formula, Indicates hinge loss. Represents a set of shared alignment features. Represents the set of true sentiment scores. Indicates different sample indices, This indicates the number of samples in the current batch. Indicates the first The text of each sample shares alignment features. Indicates the first Audio shared alignment features of individual samples Indicates the first Visually shared alignment features of individual samples Indicates the first The true sentiment score of each sample This indicates taking the maximum value. Indicates the first The true sentiment score of each sample This represents the shared semantic similarity between different modalities of the same sample. This represents the shared semantic similarity between different samples. Indicates the first Sample mode Shared alignment features Indicates the first Sample mode Shared alignment features Indicates the first Sample mode Shared alignment features Modal index , Modal index ; Among them, the shared-specific orthogonal loss includes: ； In the formula, This represents the cosine embedding loss.

8. The multimodal sentiment analysis method based on channel recalibration and polarity membership consistency gating according to claim 1, characterized in that, The prediction head is a residual fully connected prediction head, which includes a residual multilayer perceptron and a predictive linear output layer.

9. A multimodal sentiment analysis system based on channel recalibration and polarity membership consistency gating, characterized in that, The system, applied to the implementation of the multimodal sentiment analysis method based on channel recalibration and polarity membership consistency gating as described in any one of claims 1-8, comprises: The model input module is used to feed the training multimodal sentiment input into the sentiment analysis model to be trained; The alignment and recalibration module is used to perform feature alignment to determine text alignment features, audio alignment features, and visual alignment features of the trained multimodal emotion input in a channel-first format, and to perform channel recalibration to determine the corresponding modality recalibration features. The shared encoding module is used to determine the modal shared features corresponding to each modal recalibration feature through a shared encoder, and to enhance each modal shared feature through a shared self-attention encoder, and then to construct the shared fusion feature by taking the hidden state at the last time point and splicing them together. A unique encoding module is used to extract the modality-specific features corresponding to each modality recalibration feature using a modality-specific encoder, conduct cross-modal interaction through text guidance, and extract the corresponding high-level modality features through a memory encoder. The polarity membership consistency gating module is used to construct the associated modal sentiment polarity confidence based on the modal high-level predicted sentiment scores of each modal high-level feature, and to construct the corresponding modal scaling factor to scale each modal high-level feature by combining the cross-modal shared semantic consistency score of the modal shared alignment features of each modal shared feature alignment projection, and to determine the corresponding modal scaling high-level feature. The fusion prediction module is used to perform linear projection and Sigmoid activation function mapping on the shared fusion features, transform the shared fusion features, and then concatenate them with the modality scaling high-level features to construct the final fusion features. The final fusion features are then input into the full-modality prediction head to predict and output the training full-modality prediction sentiment score. The model optimization module is used to iteratively optimize the model parameters based on the total training loss value corresponding to the training full-modality predicted sentiment score until the trained sentiment analysis model is determined. The correction parameter determination module is used to perform full-modal prediction and text counterfactual inference correction on the training text counterfactual multimodal sentiment input, the verification multimodal sentiment input, and the verification text counterfactual multimodal sentiment input using the trained sentiment analysis model, and to determine the mean counterfactual prediction score of the training text and the optimal combination of correction parameters; The sentiment prediction module is used to determine the target full-modal predicted sentiment score corresponding to the multimodal sentiment input to be analyzed and the target text counterfactual full-modal predicted sentiment score of the counterfactual multimodal sentiment input to be analyzed by using a trained sentiment analysis model. The module combines the mean of the training text counterfactual predicted scores and the optimal correction parameter combination to perform text counterfactual reasoning correction on the target full-modal predicted sentiment score to determine the final target sentiment prediction score. Among them, text counterfactual multimodal emotional input refers to multimodal emotional input that keeps the text emotional input unchanged while setting the audio emotional input and visual emotional input to zero.

Images