A multi-modal counterfactual sentiment recognition method and device based on emotional prototype driving

Abstract

This invention discloses a multimodal counterfactual emotion recognition method and apparatus based on emotion prototype-driven approach, relating to the field of emotion analysis technology. The method includes: separating cross-modal shared semantics and modality-specific emotional information through shared private decoupling coding; using source modality reliability to dynamically reflect the consistency between modalities and the clarity of emotion intensity attribution; modulating homogeneous graph distillation edge weights and suppressing low-reliability source modalities in the shared semantic space using source modality reliability; achieving heterogeneous enhancement graph distillation through cross-modal attention in the private information domain; and finally introducing controlled audio-visual gated residual calibration based on the predicted emotion intensity value of reliability-aware multimodal fusion prediction using shared features of text, audio, and video, as well as private enhancement features. This reduces the risk of unreliable modal propagation of erroneous knowledge, decreases noise interference and semantic shift in multimodal fusion, thereby improving the accuracy of multimodal emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of sentiment analysis technology, and in particular to a method and apparatus for multimodal counterfactual sentiment recognition based on sentiment prototype-driven approaches. Background Technology

[0002] Emotion is an important component of human psychological activity, and it has a significant impact on individual cognitive expression, behavioral decision-making, and social interaction. Multimodal sentiment analysis can identify an individual's emotional state by simultaneously utilizing multi-source information such as language content, tone of voice, facial expressions, and changes in movement, thus having broad application value.

[0003] Existing multimodal sentiment analysis methods typically extract features from text, audio, and video modalities separately, then obtain a fused representation through methods such as cross-modal attention, and finally output sentiment prediction results using a classifier or regressor. Although this type of method improves sentiment recognition performance to some extent, the information distribution of different modalities varies greatly. Directly interacting with the features of the three modalities can easily lead to cross-modal consistent semantics, modality-specific sentiment cues, and noise information being mixed in the same representation space, resulting in insufficient discriminative power of the fused features and consequently, low accuracy in multimodal sentiment recognition. Summary of the Invention

[0004] This invention provides a method and apparatus for multimodal counterfactual emotion recognition based on emotion prototypes, which can improve the accuracy of multimodal emotion recognition.

[0005] The first aspect of this invention provides a multimodal counterfactual sentiment recognition method based on sentiment prototype-driven approach, comprising:

[0006] The training multimodal sentiment data is input into the multimodal sentiment recognition model to be trained, and the dimensional unified features of each modality are extracted based on the training multimodal sentiment data;

[0007] Private and shared encodings are performed on the unified features of each dimension respectively to determine the private and shared features of each modality;

[0008] The shared features are used to construct the corresponding homogeneous graph distillation node features based on the shared semantic residuals and determine the associated homogeneous sentiment intensity prediction values. The target homogeneous edge weights of the cross-modal directed edges of each homogeneous graph from the source modality to the target modality are determined by homogeneous graph distillation.

[0009] Cross-modal relative reliability is determined based on the shared features described above. Prototype reliability is determined by using the emotional intensity prototype vector and the emotional prototype probability distribution between the features of each homogeneous graph distillation node. Reliability weighted fusion is performed according to modal association to determine the source modality reliability of each modality.

[0010] Based on the reliability of each source mode, the target homogeneous edge weight associated with the source mode is gated and weighted, and the source mode is pruned to determine the corresponding pruned homogeneous edge weight. Then, the pruned homogeneous edge weights pointing to the same target mode are normalized and scaled or set to zero within the group according to whether they have all been pruned, and the target reliability-aware homogeneous edge weights corresponding to each pruned homogeneous edge weight are determined.

[0011] Using the private features obtained by cross-modal attention enhancement based on the private features, construct the corresponding heterogeneous graph distillation node features by performing private semantic residuals, determine the associated heterogeneous sentiment intensity prediction value, and determine the target heterogeneous edge weight of each heterogeneous graph cross-modal directed edge from the source modality to the target modality based on heterogeneous graph distillation.

[0012] By combining the shared features, source modality reliability, private enhancement features, audio heterogeneous sentiment intensity prediction values, and video heterogeneous sentiment intensity prediction values, a basic sentiment intensity prediction value is output for sentiment prediction. Based on the determined text anchoring audio-video residual value and audio-video deviation gate value, the audio-video corrected basic sentiment intensity prediction value is determined.

[0013] The training multimodal emotion recognition model determines the counterfactual text corresponding to the training multimodal emotion data. Then, it combines the homogeneous emotion intensity prediction, the target reliability-perceived homogeneous edge weight, the heterogeneous graph distillation node features, the heterogeneous emotion intensity prediction, the target heterogeneous edge weight, the text anchoring audio-visual residual value, and the audio-visual deviation gate value to determine the total loss corresponding to the audio-visual corrected basic emotion intensity prediction value. The model parameters are then iteratively optimized until the trained multimodal emotion recognition model is determined.

[0014] The trained multimodal emotion recognition model determines the target audio and video correction base emotion intensity prediction value corresponding to the multimodal emotion data to be recognized.

[0015] The process involves constructing corresponding homogeneous graph distillation node features based on shared semantic residuals using the shared features and determining the associated homogeneous sentiment intensity prediction values. Homogeneous graph distillation is used to determine the target homogeneous edge weights of cross-modal directed edges from the source modality to the target modality, including:

[0016] Each of the shared features is flattened into flattened homogeneous features, and after passing through fully connected mapping, ReLU nonlinear activation, Dropout regularization, and fully connected mapping, it is connected with the residual of the associated flattened homogeneous features to determine the corresponding homogeneous graph distillation node features.

[0017] Determine the homogeneous sentiment intensity prediction value of each homogeneous graph distillation node feature as the output of sentiment prediction;

[0018] Based on the distillation node features of the homogeneous graph associated with each modality and the predicted value of homogeneous sentiment intensity, the basic homogeneous edge weights are calculated according to the cross-modal directed edges of the homogeneous graph constructed from the source modality to the target modality, and then Softmax normalization is performed to determine the corresponding target homogeneous edge weights.

[0019] Preferably, determining the cross-modal relative reliability based on each of the shared features includes:

[0020] Temporal attention pooling is performed on the shared features of each modality to determine the corresponding pooling features;

[0021] Channel recalibration is performed on each of the pooling features to output the corresponding channel enhancement features;

[0022] For any channel enhancement feature of an evaluation modality, a difference feature is constructed between it and the channel enhancement features of other modalities by feature subtraction. The channel enhancement features of each modality are then concatenated with the difference features of the evaluation modality and input into a multilayer perceptron to output the modal reliability score of the evaluation modality.

[0023] The reliability scores of each mode are normalized by using the Softmax activation function to determine the cross-modal relative reliability of each mode.

[0024] Preferably, determining prototype reliability by using the emotional intensity prototype vector and the emotional prototype probability distribution between the features of each homogeneous graph distillation node includes:

[0025] The emotional intensity prototype vector is normalized into a normalized emotional intensity prototype vector.

[0026] After mapping the features of the associated homogeneous graph distillation nodes to the prototype space through the query mapping function of each modality, the corresponding prototype query vector is normalized and constructed, and the prototype similarity between the vector and the normalized sentiment intensity prototype vector is determined respectively.

[0027] The Softmax activation function is used to normalize the prototype similarity of each modality to determine the probability distribution of the emotional prototype of each modality.

[0028] The distribution entropy of each of the aforementioned emotional prototype probability distributions is calculated to determine the corresponding prototype distribution entropy, and the inverse value of the corresponding normalized prototype distribution entropy is determined as the associated prototype reliability.

[0029] Preferably, after determining the corresponding pruned homogeneous edge weights based on the source modality pruning and gating the target homogeneous edge weights associated with each source modality based on the reliability of each source modality, the pruned homogeneous edge weights pointing to the same target modality are normalized and scaled within the group or set to zero according to whether they have all been pruned, to determine the target reliability-aware homogeneous edge weights corresponding to each pruned homogeneous edge weight, including:

[0030] The mean reliability of each source mode is calculated by averaging the reliability of each source mode, and the gating coefficient of each mode is constructed by comparing the reliability of each source mode with the mean reliability of the source modes.

[0031] The corresponding initial reliability-aware homogeneous edge weights are determined by weighting the source modes corresponding to the homogeneous edge weights of each target using associated gating coefficients.

[0032] The source mode retention mask for each mode is determined based on the comparison results between the source mode reliability and the mode reliability pruning threshold.

[0033] Based on the source modes corresponding to the initial reliability-aware homogeneous edge weights, the associated source mode retention mask is used for weighting to construct the corresponding pruned homogeneous edge weights;

[0034] The summation of the homogeneous edge weights associated with the same target mode is performed to determine the pre-modulation summation of the homogeneous edge weights for each target mode.

[0035] The weights of the pruned homogeneous edges associated with the same target mode are summed to determine the sum of the pruned homogeneous edge weights for each target mode.

[0036] When the sum of the pruned homogeneous edge weights of any of the target modes is greater than a preset numerical stability constant, the pruned homogeneous edge weights associated with the sum of the pruned homogeneous edge weights are normalized, and the sum of the pre-modulation homogeneous edge weights pointing to the same target mode is scaled to determine the target reliability perception homogeneous edge weights corresponding to the pruned homogeneous edge weights.

[0037] When the sum of the pruned homogeneous edge weights of any of the target modes is less than or equal to a preset numerical stability constant, the pruned homogeneous edge weights associated with the sum of the pruned homogeneous edge weights are set to zero as the corresponding target reliability-aware homogeneous edge weights.

[0038] Preferably, the step of constructing corresponding heterogeneous graph distillation node features using private enhanced features obtained from cross-modal attention enhancement based on each of the private features, determining the associated heterogeneous sentiment intensity prediction value, and determining the target heterogeneous edge weights of each heterogeneous graph cross-modal directed edge from the source modality to the target modality based on heterogeneous graph distillation includes:

[0039] A multimodal converter is used to perform cross-modal attention enhancement on each of the private features to determine the private enhanced features;

[0040] Each of the private enhancement features is flattened into flattened heterogeneous features, and after passing through fully connected mapping, ReLU nonlinear activation, Dropout regularization and fully connected mapping, it is connected with the residual of the associated flattened heterogeneous features to determine the corresponding heterogeneous graph distillation node features.

[0041] The heterogeneous sentiment intensity prediction value is output by determining the features of each heterogeneous graph distillation node;

[0042] Based on the features of distilled nodes in the heterogeneous graph associated with each modality and the predicted value of heterogeneous sentiment intensity, the basic heterogeneous edge weights are calculated according to the cross-modal directed edges of the heterogeneous graph constructed from the source modality to the target modality, and then Softmax normalization is performed to determine the corresponding target heterogeneous edge weights.

[0043] Preferably, the step of combining the shared features, the source modality reliability, the private enhancement features, and the audio heterogeneous sentiment intensity prediction values ​​and video heterogeneous sentiment intensity prediction values ​​to perform sentiment prediction and output a basic sentiment intensity prediction value, and then determining an audio-video corrected basic sentiment intensity prediction value based on the determined text-anchored audio-video residual value and audio-video deviation gate value, includes:

[0044] Shared semantic fusion features are constructed by fusing the shared features described above, and the output of the shared semantic fusion features is aligned by linear mapping and ReLU nonlinear activation.

[0045] The maximum value of the reliability of each source modality is taken as the modality fusion reliability of the aligned shared semantic fusion feature, and the fusion scaling factor is determined based on the reliability of each source modality and the modality fusion reliability;

[0046] Each of the private enhanced features is linearly mapped and ReLU nonlinearly activated, then concatenated with the aligned shared semantic fusion feature, and multiplied element-wise with the fusion scaling coefficient to output the scaled fusion feature;

[0047] The scaled fused features are used to predict sentiment through a two-layer perceptron with residual connections, and the basic sentiment intensity prediction value is output.

[0048] The text scaling private enhancement features in the scaling fusion features are combined with the audio scaling private enhancement features and / or video scaling private enhancement features in the scaling fusion features according to the preset expert modality to construct text anchored audio and video residual expert features.

[0049] The text-anchored audio and video residual expert features are sequentially subjected to linear mapping, ReLU nonlinear activation, Dropout regularization, linear mapping, and tanh nonlinear activation, and then limited according to the upper limit of the residual amplitude to determine the text-anchored audio and video residual value.

[0050] The absolute value of the basic emotional intensity prediction value, and the absolute deviations of the audio heterogeneous emotional intensity prediction value and the video heterogeneous emotional intensity prediction value relative to the basic emotional intensity prediction value are used to construct audio-video deviation information.

[0051] After concatenating the text-anchored audio-visual residual expert features, the audio-visual deviation information, and the reliability of each source mode, linear mapping, ReLU nonlinear activation, Dropout regularization, linear mapping, and Sigmoid nonlinear activation are performed sequentially to output the audio-visual deviation gating value.

[0052] Based on the text anchoring audio-visual residual value and the audio-visual deviation gate value, the basic sentiment intensity prediction value is calibrated with bounded audio-visual information to determine the audio-visual corrected basic sentiment intensity prediction value.

[0053] The process of determining the fusion scaling factor includes:

[0054] ;

[0055] In the formula, Indicates the blending scaling factor. This indicates taking the minimum value. This indicates taking the maximum value. Indicates the reliability scaling strength. Represents the reliability fusion vector. This represents the mean of the reliability fusion vector. This indicates the lower limit of the scaling factor. This indicates the upper limit of the scaling factor. Indicates the reliability of the text source modality. Indicates the modal reliability of the audio source. Indicates the modal reliability of the video source. Indicates splicing.

[0056] Preferably, the process of determining the total loss includes:

[0057] ;

[0058] In the formula, Indicates the total loss. This indicates the predicted loss for the main task. Indicates distillation loss in the isotropic diagram. Indicates the distillation loss in the heterogeneous diagram. Indicates cyclical loss. Indicates the reconstruction loss. This represents the loss due to shared feature similarity constraints. Indicates orthogonal loss. This indicates the loss caused by audio and video counterfactual gating surveillance. This indicates the text's counterfactual consistency loss. This indicates an auxiliary loss related to the emotional archetype;

[0059] The reconstruction loss includes:

[0060] ;

[0061] In the formula, Indicates modal index, Represents text modality, Indicates video modality, Indicates audio modality, Representing modes The unified features of the reconstruction dimension Representing modes The dimensional uniformity features Denotes the square of the L2 norm. Representing modes Shared features Representing modes Private characteristics, Indicates splicing, Representing modes Feature reconstruction decoder;

[0062] Cyclic losses include:

[0063] ;

[0064] In the formula, Representing modes Recoded private features, Representing modes Private encoder;

[0065] Orthogonal losses include:

[0066] ;

[0067] In the formula, This indicates taking the maximum value. Represents cosine similarity. Indicates the interval threshold;

[0068] The shared feature similarity constraint loss includes:

[0069] ;

[0070] In the formula, Represents the set of valid anchor point feature objects. Indicates the number of valid anchor point feature objects. Indicates the anchor feature object index. Represents a set of objects with positive characteristics. Indicates the number of objects with positive features. Represents a set of objects with negative characteristics. Indicates the number of objects with negative features. Indicates the index of the positive feature object. Indicates the index of the negative feature object. Represents anchor point feature object The corresponding true emotional intensity label value, Represents a negative feature object The corresponding true emotional intensity label value, Represents anchor point feature object Shared similarity features are obtained by flattening and linear mapping of shared features. Represents positive feature objects Shared similarity features are obtained by flattening and linear mapping of shared features. Represents a negative feature object Shared similarity features are obtained by flattening and linear mapping of shared features. Represents the complete set of feature objects. Indicates the sample index. Indicates removal. Represents positive feature objects The corresponding true emotional intensity label value, Represents anchor point feature object Corresponding sample index, Represents anchor point feature object The corresponding modal index, Indicates the first The true emotional intensity labeled value of each sample;

[0071] Among them, the emotional prototype auxiliary loss includes:

[0072] ;

[0073] In the formula, Represents discrete sentiment category-constrained loss. This indicates the loss of continuous emotional intensity constraint. Indicates the regression constraint weights;

[0074] The discrete emotion category constraint loss includes:

[0075] ;

[0076] In the formula, Represents the natural logarithm. Representing modes The probability distribution of emotional prototypes Represents a minimal constant;

[0077] The continuous emotion intensity constraint loss includes:

[0078] ;

[0079] In the formula, This indicates taking the minimum value. This represents the predictive value of continuous sentiment intensity. This represents the labeled value indicating the true intensity of emotion. Represents the transpose of the emotional intensity anchor vector;

[0080] Among them, the isomorphic distillation loss includes:

[0081] ;

[0082] In the formula, This represents the weighting coefficient for distillation loss in the homogeneous graph. This indicates the weighted prediction of distillation loss. Indicates the edge weight regularization loss;

[0083] The weighted predicted distillation loss includes:

[0084] ;

[0085] In the formula, Indicates the source modality index. Indicates the target modality index. This represents the set of directed edges across modalities in a homogeneous graph. Represents the source mode To target mode The target reliability perception homogeneous edge weight Represents the distance metric function. Represents the target mode The predicted value of homogeneous sentiment intensity Represents the source mode The predicted value of homogeneous sentiment intensity;

[0086] Among them, edge weight regularization loss includes:

[0087] ;

[0088] In the formula, This represents the edge weight regularization weight coefficient. This represents the average target reliability perception edge weight. Represents the source mode To target mode Homogeneous fixed prior edge weight vectors Indicates batch size, Indicates the first Source mode in each sample To target mode The target reliability perception homogeneous edge weight;

[0089] The heterogeneous distillation losses include:

[0090] ;

[0091] In the formula, This represents the weighting coefficient for distillation loss in the heterogeneous plot. Indicates heterogeneous prediction of distillation loss, Indicates the characteristic distillation loss of heterogeneous materials. Indicates the heterogeneous edge weight regularization loss;

[0092] Among them, the predicted distillation losses by heterogeneity include:

[0093] ;

[0094] In the formula, This represents the set of directed edges across modalities in a heterogeneous graph. Represents the source mode To target mode Target heterogeneous edge weights Represents the target mode The predicted value of heterogeneous sentiment intensity Represents the source mode The predicted value of heterogeneous sentiment intensity;

[0095] Among them, heterogeneous characteristic distillation losses include:

[0096] ;

[0097] In the formula, The characteristics of the distillation nodes in the heterogeneous graph are represented. The characteristics of distillation nodes in the heterogeneous graph are represented.

[0098] Among them, heterogeneous edge weight regularization loss includes:

[0099] ;

[0100] In the formula, This represents the edge weight regularization weight coefficient. This represents the batch average target heterogeneous edge weight. Represents the source mode To target mode Heterogeneous fixed prior edge weight vectors Indicates the first Source mode in each sample To target mode The target heterogeneous edge weight;

[0101] Among them, the losses from audio and video counterfactual gating supervision include:

[0102] ;

[0103] In the formula, Indicates the first The counterfactual supervision weights for each sample's audio and video recordings. Indicates the first The audio / video deviation gating value for each sample. Indicates the first Audio and video gating supervision targets for each sample. Indicates the target of audio and video gating supervision. This represents the Sigmoid activation function. This represents the contribution value of the audio and video residuals. Indicates the boundary parameters of audio and video contribution. Indicates the audio / video temperature coefficient. Indicates the main prediction error. This indicates the counterfactual prediction error. This represents the predicted value of basic sentiment intensity. This indicates taking the absolute value. This indicates the predicted base sentiment intensity value after counterfactual correction for audio and video recordings. Represents the counterfactual residual coefficient. This represents the text-anchored audio / video residual value;

[0104] Among them, text counterfactual consistency loss includes:

[0105] ;

[0106] In the formula, Indicates the first The textual counterfactual sample weights of each sample. Indicates an indicator function, Indicates the first The absolute value of the true sentiment intensity labeled for each sample This represents the threshold for distinguishing between samples with strong emotional bias and neutral samples. Indicates the polarity consistency loss weight. Indicates the first Polarity consistency loss for each sample Indicates the weight of intensity consistency loss. Indicates the first The intensity consistency loss of each sample Indicates the weight of neutral consistency loss. Indicates the first Neutral consistency loss for each sample;

[0107] Among them, the polarity consistency loss includes:

[0108] ;

[0109] In the formula, Indicates the polarity descent interval, Indicates the directional decrease. This represents the predicted counterfactual sentiment intensity of the text.

[0110] Among them, the strength consistency loss includes:

[0111] ;

[0112] In the formula, Indicates the interval of intensity decrease. Indicates the degree of decrease in emotional intensity;

[0113] The neutral consistency loss includes:

[0114] ;

[0115] In the formula, This represents smoothed L1 loss.

[0116] Preferably, the text counterfactual sample weights include:

[0117] ;

[0118] In the formula, This represents the minimum weight of the counterfactual sample in the text. This indicates the validity gate for text perturbation. Indicates the boundary mixing factor. This represents the absolute value of the text perturbation effect. Indicates the boundary of the text perturbation effect. Indicates the temperature coefficient of the text. This indicates the text perturbation effect. Indicates boundary weights, This represents the natural exponential function. This represents the boundary smoothing coefficient.

[0119] A second aspect of the present invention provides a multimodal counterfactual emotion recognition device based on emotion prototype-driven methods, comprising:

[0120] The multimodal feature extraction module is used to input training multimodal sentiment data into the multimodal sentiment recognition model to be trained, and extract dimensional unified features of each modality based on the training multimodal sentiment data;

[0121] The multimodal feature decoupling coding module is used to perform private coding and shared coding on the unified features of each dimension respectively, and to determine the private features and shared features of each modality;

[0122] The homogeneous graph distillation module is used to construct corresponding homogeneous graph distillation node features based on shared semantic residuals using the shared features and determine the associated homogeneous sentiment intensity prediction value. The homogeneous graph distillation module determines the target homogeneous edge weights of the cross-modal directed edges from the source modality to the target modality.

[0123] The modal reliability assessment module is used to determine the cross-modal relative reliability based on the shared features, determine the prototype reliability by using the emotional intensity prototype vector and the emotional prototype probability distribution between the features of each homogeneous graph distillation node, and perform reliability weighted fusion according to modal association to determine the source modal reliability of each modality.

[0124] The reliability homogeneity graph distillation module is used to perform gated weighting and source mode pruning on the target homogeneous edge weights associated with the source modes based on the reliability of each source mode. After determining the corresponding pruned homogeneous edge weights, the pruned homogeneous edge weights pointing to the same target mode are normalized and scaled or set to zero within the group according to whether they have all been pruned, and the target reliability-aware homogeneous edge weights corresponding to each pruned homogeneous edge weight are determined.

[0125] The heterogeneous augmented graph distillation module is used to construct corresponding heterogeneous graph distillation node features by using the private features obtained based on cross-modal attention augmentation of each private feature, and to determine the associated heterogeneous sentiment intensity prediction value. Based on the heterogeneous graph distillation, the target heterogeneous edge weights of each heterogeneous graph cross-modal directed edge from the source modality to the target modality are determined.

[0126] The audio and video gated residual prediction module is used to combine the shared features, source modality reliability, private enhancement features, audio heterogeneous sentiment intensity prediction value, and video heterogeneous sentiment intensity prediction value to perform sentiment prediction and output a basic sentiment intensity prediction value. Based on the determined text anchored audio and video residual value and audio and video deviation gate value, the module corrects and determines the audio and video corrected basic sentiment intensity prediction value.

[0127] The model training optimization module is used to determine the counterfactual base sentiment intensity prediction value of the text corresponding to the counterfactual text of the training multimodal sentiment data through the multimodal sentiment recognition model to be trained, and to determine the total loss corresponding to the audio-video corrected base sentiment intensity prediction value by combining the homogeneous sentiment intensity prediction value, the target reliability-aware homogeneous edge weight, the heterogeneous graph distillation node features, the heterogeneous sentiment intensity prediction value, the target heterogeneous edge weight, the text anchoring audio-video residual value, and the audio-video deviation gate value, and iteratively optimize the model parameters until the trained multimodal sentiment recognition model is determined;

[0128] The emotion recognition module is used to determine the target audio and video corrected basic emotion intensity prediction value corresponding to the multimodal emotion data to be recognized by using a trained multimodal emotion recognition model.

[0129] As can be seen from the above technical solutions, the present invention has the following advantages:

[0130] The above-described scheme of the present invention provides a multimodal counterfactual emotion recognition method based on emotion prototype-driven approach. It separates cross-modal shared semantics and modality-specific emotional information through shared private decoupling coding. It employs source modality reliability to dynamically reflect both intermodal consistency and the clarity of emotion intensity attribution. In the shared semantic space, it modulates homogeneous graph distillation edge weights using source modality reliability and performs low-reliability source modality suppression. In the private information domain, it achieves heterogeneous enhancement graph distillation through cross-modal attention. Finally, based on the fundamental emotion intensity prediction value obtained through reliability-aware multimodal fusion prediction using shared features from text, audio, and video, as well as private enhancement features, it introduces audio-visual gated residuals for bounded gating calibration. This reduces the risk of unreliable modal propagation of erroneous knowledge, decreases noise interference and semantic shift in multimodal fusion, thereby improving the accuracy of multimodal emotion recognition. Attached Figure Description

[0131] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0132] Figure 1 This is a flowchart illustrating the steps of a multimodal counterfactual sentiment recognition method based on emotion prototype-driven approach provided in Embodiment 1 of the present invention.

[0133] Figure 2 This is a block diagram of the overall structure of the multimodal emotion recognition model provided in Embodiment 1 of the present invention;

[0134] Figure 3 This is a partial structural block diagram of the multimodal emotion recognition model provided in Embodiment 1 of the present invention;

[0135] Figure 4 This is a structural block diagram of a multimodal counterfactual emotion recognition device based on emotion prototype driving, provided in Embodiment 2 of the present invention. Detailed Implementation

[0136] This invention provides a method and apparatus for multimodal counterfactual emotion recognition based on emotion prototypes, which can improve the accuracy of multimodal emotion recognition.

[0137] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0138] Please see Figure 1 The present invention provides a multimodal counterfactual sentiment recognition method based on sentiment prototype-driven approach, comprising:

[0139] Step 101: Input the training multimodal sentiment data into the multimodal sentiment recognition model to be trained, and extract the dimensional unified features of each modality based on the training multimodal sentiment data.

[0140] It should be noted that, as Figure 2 and Figure 3 As shown, this embodiment performs emotion recognition based on the constructed multimodal emotion recognition model, and optimizes the model parameters by training the multimodal emotion recognition model with training multimodal emotion data; wherein, the training multimodal emotion data refers to the multimodal emotion data in the training phase, and the modality index Including text modality Audio modality and video modality Multimodal sentiment data includes text sentiment data. Audio sentiment data and video sentiment data It is understandable that text sentiment data can be text sentiment input that has already been extracted by the RoBERTa model, audio sentiment data can be continuous audio temporal features that have already been extracted, and video sentiment data can be video frame-level features or video temporal features such as facial expressions, actions, and postures. All of these are feature input tensors that can be directly processed by the model.

[0141] Upon receiving the training multimodal sentiment data, dimensional alignment is performed based on the training multimodal sentiment data in a channel-first format to determine the dimensional uniform features of each modality, including the text dimensional uniform features. Unified features in audio dimension Unified features with video dimensions In one specific implementation of this embodiment, the above process includes: exchanging the time dimension and feature dimension of text sentiment data, audio sentiment data, and video sentiment data to generate text channel-priority features, audio channel-priority features, and video channel-priority features with the modal input feature dimension as the channel dimension and the modal sequence length as the convolution length dimension; when the modal input feature dimension is inconsistent with the model target dimension, inputting a one-dimensional convolutional projection layer for dimension alignment to unify the channel dimension of each modality to the target dimension, thereby obtaining dimension-unified features for each modality.

[0142] By swapping the temporal and feature dimensions of text, audio, and video temporal features, we obtain text channel-priority features, audio channel-priority features, and video channel-priority features, respectively, with the modal input feature dimension as the channel dimension and the modal sequence length as the convolution length dimension. These features are then used for feature processing by the corresponding one-dimensional convolutional projection layers. Subsequently, when the input feature dimension differs from the model's target dimension, the one-dimensional convolutional projection layer unifies the channel dimensions of each modality to the target dimension, enabling local temporal information aggregation.

[0143] Step 102: Perform private and shared encoding on the unified features of each dimension to determine the private and shared features of each modality.

[0144] It should be noted that, considering the differences in sentiment expression and information distribution among text, audio, and video modalities, directly concatenating the features of the three modalities or performing cross-modal distillation can easily lead to unstable information transmission between strong and weak modalities. Therefore, this embodiment uses shared private decoupling coding to divide the information in the dimensionally unified features of the three modalities into a shared information domain and a private information domain. The shared information domain is used to represent the consistent sentiment semantics across different modalities, while the private information domain is used to retain the unique expressive features of each modality. Specifically:

[0145] Private encoders are used to encode unified features in text, audio, and video dimensions respectively, thereby determining the corresponding private text features. Audio private features and video private features Among them, the private encoders corresponding to each modality include text private encoders. proprietary audio encoder and proprietary video encoders It is understandable that in the expression of information in the three modalities, text mainly carries word meaning and semantic logic, audio mainly carries intonation, rhythm and energy changes, and video mainly carries facial expressions, actions and facial changes. Therefore, the private encoder uses modality-specific parameters to preserve the unique emotional expression patterns of each modality.

[0146] A shared encoder is used to encode the unified features of the text dimension, audio dimension, and video dimension separately, thereby generating the corresponding text shared features. Audio sharing features Features shared with video Unlike proprietary encoders, shared encoders use the same set of parameters to process information from three modalities. Their purpose is to extract consistent semantic information across different modalities, enabling text, audio, and video to be mapped into a more closely spaced shared semantic space.

[0147] Step 103: Construct corresponding homogeneous graph distillation node features based on shared semantic residuals using each shared feature and determine the associated homogeneous sentiment intensity prediction value. Determine the target homogeneous edge weights of the cross-modal directed edges of each homogeneous graph from the source modality to the target modality through homogeneous graph distillation.

[0148] Homogeneous sentiment intensity prediction refers to the prediction result obtained by performing sentiment prediction based on the features of distilled nodes in a homogeneous graph.

[0149] It should be noted that homogeneous graph distillation adaptively learns the inter-modal edge weights based on the homogeneous graph distillation node features of the modality and the homogeneous sentiment intensity prediction value, thereby modeling the knowledge transfer relationship between different modalities in the shared semantic space. At the same time, this embodiment enhances the representation performance of the homogeneous graph distillation node features by adding residual structure design.

[0150] In one specific embodiment of this example, step 103 includes the following sub-steps:

[0151] S11. Flatten each shared feature into flattened homogeneous features, and after fully connected mapping, ReLU nonlinear activation, Dropout regularization and fully connected mapping, connect them with the residuals of the associated flattened homogeneous features to determine the corresponding homogeneous graph distillation node features.

[0152] S12. Determine the homogeneous sentiment intensity prediction value of each homogeneous graph distillation node feature as the output of sentiment prediction;

[0153] S13. Based on the distillation node features of the homogeneous graph associated with each modality and the predicted value of homogeneous sentiment intensity, calculate the basic homogeneous edge weights of the cross-modal directed edges of the homogeneous graph constructed from the source modality to the target modality, and perform Softmax normalization to determine the corresponding target homogeneous edge weights.

[0154] It should be noted that, for modal... First, share semantic information with them. Flattening is a homogeneous feature of a fixed length. Subsequently, the initial homogeneous graph node features of this mode in the homogeneous graph distillation space are obtained through fully connected (FC) mapping. To enhance the nonlinear expressive power of node representations and reduce the risk of overfitting during node mapping, [the following is done]: ReLU nonlinear activation and Dropout regularization are applied to obtain the features of intermediate homogeneous graph nodes. Then, through a second fully connected layer (FC) Map back to Same dimensions, and with Perform residual joins to obtain the final homogeneous graph distillation node features. Subsequently, based on Sentiment prediction yields the predicted value of the homogeneous sentiment intensity in a homogeneous shared space. By sharing semantic residual structures, the original shared semantic information can be preserved on the one hand, and the node mapping process can avoid excessive changes to the shared representation on the other hand, thereby improving the stability of homogeneous graph distillation training. It is understandable that during the training phase, Dropout suppresses the response of some neurons through random masks to improve the robustness of the model, while Dropout is usually turned off during the inference phase.

[0155] In the edge weight construction stage, in order to simultaneously utilize node prediction information and node representation information, modalities are constructed. The homogeneous node edge features used for edge weight calculation are determined by linearly mapping the associated homogeneous graph distillation node features to the homogeneous sentiment intensity prediction values ​​according to modality and then concatenating them: , Representing modes Homogeneous node edge characteristics, This represents a learnable linear transformation that maps the prediction results. This represents a learnable linear transformation that maps node representations. Indicates splicing; for the source mode To target mode The set of homogeneous graphs consisting of cross-modal directed edges. First, calculate the basic homogeneous edge weights. ,For example , Represents the source mode To target mode The basic homogeneous edge weights, Representing different weight mapping matrices, Indicates different biases. Indicates nonlinear activation. This indicates transpose; subsequently, the dynamic edge weights of the six directed edges are uniformly normalized along the edge dimension using Softmax to determine the corresponding target homogeneous edge weights.

[0156] Step 104: Determine the cross-modal relative reliability based on the corresponding shared features, determine the prototype reliability by using the emotional intensity prototype vector and the emotional prototype probability distribution between the features of each homogeneous graph distillation node, and perform reliability weighted fusion according to modal association to determine the source modal reliability of each modality.

[0157] Cross-modal relative reliability refers to the degree of confidence of a mode relative to the other modes within the current sample, which is estimated by combining the channel enhancement features of a certain mode, the channel enhancement features of the other modes, and the differences between them. The sum of the cross-modal relative reliability of each mode after Softmax normalization is 1.

[0158] The probability distribution of sentiment prototypes refers to the probability distribution of each sentiment intensity prototype obtained by mapping the features of homogeneous graph distillation nodes of a certain modality to the sentiment prototype space, and then normalizing it with Softmax based on the prototype similarity between the features and the sentiment intensity prototype vector.

[0159] Prototype reliability refers to an index determined by the inverse value of the normalized distribution entropy of the emotional prototype probability distribution, used to characterize the degree of concentration of the probability distribution and the certainty of the emotional prototype attribution of the modality.

[0160] Source modality reliability refers to the relative credibility of a modality as a cross-modal knowledge source in the current sample, obtained by weighted fusion of the cross-modal relative reliability of a modality and the prototype reliability, and normalization of the modality dimension.

[0161] It should be noted that, in this embodiment, inter-modal reliability estimation is performed based on each shared feature to determine the cross-modal relative reliability of each mode. Based on the prototype vector of emotional intensity and the features of distillation nodes in each homogeneous graph, the prototype reliability of emotional intensity is evaluated to determine the prototype reliability of each modality. The corresponding source modal reliability is determined by weighted fusion of the cross-modal relative reliability and the prototype reliability of the modal pair. ; In the formula, Represents the fusion coefficient ( (used to control the relative contributions of cross-modal consistency reliability and prototype deterministic reliability). Including text source modality reliability Audio source modal reliability and video source modal reliability Cross-modal relative reliability characterizes whether a modality is consistent with other modalities, while prototype reliability characterizes whether the modality itself can clearly determine the emotional intensity. That is, the former evaluates reliability from the perspective of inter-modal relationships, while the latter evaluates reliability from the perspective of emotional label space. Since the reliability of different modalities is not consistent on different samples, for example, video modalities may be affected by occlusion, pose changes or low-quality images, audio modalities may be affected by noise, unclear tone, etc., and text modalities may also have semantic ambiguity. The source modal reliability obtained by fusing the two types of reliability considers both cross-modal consistency and the certainty of emotional intensity attribution.

[0162] In one specific implementation of this embodiment, determining cross-modal relative reliability based on the corresponding shared features includes:

[0163] S21. Perform temporal attention pooling on the shared features of each modality and determine the corresponding pooling features;

[0164] S22. Perform channel recalibration on each pooling feature and output the corresponding channel enhancement feature;

[0165] S23. For any channel enhancement feature of an evaluation modality, construct the difference feature between it and the channel enhancement features of other modalities by feature subtraction, and then concatenate the channel enhancement features of each modality with the difference features of the evaluation modality and input them into a multilayer perceptron to output the modal reliability score of the evaluation modality.

[0166] S24. Normalize each reliability score using the Softmax activation function to determine the cross-modal relative reliability of each mode.

[0167] It should be noted that in multimodal emotion recognition, the reliability of text, audio, and video modalities is not fixed. For example, text is reliable when it contains explicit emotion words, but its reliability decreases when the words are ambiguous or ironic. Audio is unreliable when it may have noise, pauses, or poor speech quality. Video is reliable when it shows obvious facial expressions, but unreliable when it may have facial occlusion, weak movements, or unclear expressions. To explicitly suppress noisy modalities and explain why the model depends on a particular modality in a certain sample, this embodiment considers both the modality's own expression and its consistency with other modalities, based on the shared features of the three modalities. Estimate the relative reliability of each mode in the current sample to obtain the sample-level cross-modal reliability weights, specifically:

[0168] Let the mode The shared features are represented as: , Indicates time step index , Representing modes At time step The shared features of any modality are determined by using a fully connected layer to calculate the temporal attention scores at each time step. These scores are then normalized to temporal attention weights using the Softmax activation function and subjected to temporal pooling to determine the corresponding pooling features. This process can be represented as follows:

[0169] ;

[0170] In the formula, Representing modes At time step Time attention score Represents the time weight mapping matrix. Indicates time offset. Representing modes At time step Time attention weights Indicates time step index , Representing modes Time step length, Representing modes At time step Time attention score This represents the natural exponential function. Representing modes Pooling characteristics;

[0171] The result after time pooling This means the model performs a weighted summation across all time steps, allowing it to automatically focus on segments with more emotional information. Different channels or different feature dimensions are not necessarily equally important; for example, some dimensions may better reflect emotional information, while others may contain noise. To further distinguish the contribution of different feature channels to the judgment of emotional reliability, this embodiment... Channel recalibration is performed, taking the introduction of an efficient channel attention mechanism as an example. A cascaded one-dimensional convolution and a sigmoid activation function are used to weight the associated pooling features based on the channel weights determined by the pooling features, resulting in the recalibrated channel enhancement features. This process can enhance the channel response related to sentiment judgment and suppress the impact of noisy channels on reliability estimation;

[0172] Determine the channel enhancement features of the three modes Therefore, this embodiment will not rely solely on a particular mode itself. Instead of judging reliability, we construct consistency features of its own expression, other modal contexts, and other modal differences to determine cross-modal relative reliability:

[0173] For any evaluation mode to be evaluated for modal reliability Let the two remaining modal indices be... and Based on the modality's own expression, other modality expressions, and modality Cross-modal consistency features are constructed using difference information between different modalities, and modalities are obtained through a multilayer perceptron. The modal reliability score can be expressed as: , Representing modes Modal reliability score, This represents a multilayer perceptron. Indicates splicing, Representing modes Channel enhancement features, Representing modes Channel enhancement features, Representing modes Channel enhancement features;

[0174] The modal reliability scores of the three modalities are normalized using the Softmax activation function to obtain the cross-modal relative reliability of each modality in the current sample:

[0175] ;

[0176] In the formula, Representing modes Cross-modal relative reliability (including text cross-modal relative reliability) Audio cross-modal relative reliability Relative reliability of video across modalities ), This represents the temperature coefficient (used to adjust the smoothness of the reliability distribution of the three modes). Modal index , Representing modes The modal reliability score meets the requirements. .

[0177] In one specific implementation of this embodiment, the prototype reliability is determined by using the prototype probability distribution of sentiment intensity between the prototype vector of sentiment intensity and the features of each homogeneous graph distillation node, including:

[0178] S31. Normalize the emotional intensity prototype vector into a normalized emotional intensity prototype vector.

[0179] S32. After mapping the features of the associated homogeneous graph distillation nodes to the prototype space through the query mapping function of each modality, normalize and construct the corresponding prototype query vector, and determine the prototype similarity between the normalized sentiment intensity prototype vector and the prototype query vector.

[0180] S33. The Softmax activation function is used to normalize the prototype similarity of each modality to determine the probability distribution of the emotional prototype of each modality.

[0181] S34. Calculate the distribution entropy for each emotion prototype probability distribution, determine the corresponding prototype distribution entropy, and determine the inverse value of the corresponding normalized prototype distribution entropy as the associated prototype reliability.

[0182] It should be noted that cross-modal relative reliability mainly focuses on the consistency between text, audio, and video. However, "intermodal consistency" does not necessarily mean that the modality is sufficiently clear in judging the true emotional intensity. For example, a modality may be consistent with other modalities, but its distribution in the emotional intensity space is still ambiguous, making it impossible to clearly determine whether the current sample belongs to strong negative, weak negative, neutral, weak positive, or strong positive. Therefore, relying solely on cross-modal consistency will cause the reliability estimation to lack the constraints of the emotional task itself. To address this, this embodiment introduces an emotional prototype memory bank, matching the homogeneous representation of each modality with a set of learnable emotional intensity prototypes, and judging whether the modality is clear in its emotional intensity attribution based on the concentration of the matching distribution. This further evaluates the reliability of each modality from the perspective of emotional polarity and emotional intensity certainty. Specifically:

[0183] Let the emotional prototype memory bank be ,in Indicates the first For example, seven emotional archetypes can be used, with the emotional intensity anchor vectors set from negative to positive as follows: This emotional intensity anchor point corresponds to the continuous emotional intensity range in the emotion recognition task and is used to represent the emotional change from strong negative to strong positive.

[0184] For any mode The features of distilled nodes in a homogeneous graph are obtained through the query mapping function corresponding to the modality. Mapping to the prototype matching space and normalizing it yields the prototype query vector. Simultaneously, to mitigate the impact of vector magnitude differences on similarity calculation and to make matching focus more on the directional similarity between modal representations and emotional prototypes, the emotional intensity prototype vectors in the emotional prototype memory are normalized to obtain normalized emotional intensity prototype vectors. Subsequently, the similarity between the modality's prototype query vector and all sentiment prototypes is calculated:

[0185] ;

[0186] In the formula, Representing modes Prototype similarity, Representing modes The prototype query vector, This represents the prototype matching temperature coefficient (used to adjust the smoothness of the similarity distribution; when it is small, the similarity difference between prototypes is amplified, and the probability distribution is more concentrated on the most similar emotional prototypes; when it is large, the similarity difference between prototypes is weakened, and the probability distribution is smoother).

[0187] Considering that the model does not select only a single discrete prototype, but rather performs a weighted summation of all sentiment anchors to obtain a continuous sentiment intensity estimate, if the distribution of a certain modality is mainly concentrated near positive or negative prototypes, its predicted intensity is more explicit. If the distribution is evenly dispersed across multiple prototypes, it indicates that the modality's judgment of sentiment intensity is less certain. Therefore, Softmax normalization is applied to the prototype similarity to determine the modality. Emotional prototype probability distribution ,Right now It is modal Homogeneous graph distillation node characteristics The probability of attribution to prototypes with different emotional intensities; to further quantify this certainty, the prototype distribution entropy is calculated:

[0188] ;

[0189] In the formula, Representing modes The prototype distribution entropy, Indicates the number of emotional archetypes. Representing modes Homogeneous graph distillation node characteristics and the first The probability of emotional prototypes among emotional intensity prototypes This represents a very small constant used to prevent numerical overflow (generally taken as...). ), Represents the natural logarithm (i.e., base to the natural constant); when The more concentrated the concentration, the higher the entropy. The smaller the value, the more explicitly the modality matches a particular emotional intensity prototype. The closer to a uniform distribution, the higher the entropy. The larger the value, the less able the modality is to determine the emotional intensity of the current sample. Therefore, the inverse vector of the normalized entropy can be used as the prototype reliability of the modality.

[0190] ;

[0191] In the formula, Representing modes Prototype reliability, express The maximum entropy of the distribution of emotional prototypes (used to normalize the entropy to) Within the range); therefore, it can be seen that the sharper the prototype distribution, Approaching 0, A value close to 1 indicates that the modality makes relatively clear judgments about emotional intensity and has high reliability. The more uniform the prototype distribution, the better. Approaching , A value close to 0 indicates that the modality's attribution to emotional intensity is unclear and its reliability is low.

[0192] Step 105: Based on the reliability of each source mode, gating weights are applied to the target homogeneous edge weights associated with the source modes, and source mode pruning is used to determine the corresponding pruned homogeneous edge weights. Then, the pruned homogeneous edge weights pointing to the same target mode are normalized and scaled or set to zero within the group according to whether they have all been pruned, and the target reliability-aware homogeneous edge weights corresponding to each pruned homogeneous edge weight are determined.

[0193] It should be noted that the reliability of different modalities varies across different samples. For example, video modalities may be affected by occlusion, pose changes, or low-quality images; audio modalities may be affected by noise, unclear intonation, and other factors; and text modalities may also have semantic ambiguity. If edge weights are learned solely based on node representations and prediction results without considering the reliability of the source modal itself, unreliable modalities may still act as knowledge sources, propagating erroneous information to other modalities and increasing the risk of error distillation. Therefore, this embodiment introduces source modality reliability. By gating and modulating the target homogeneous edge weights pointing from the source modality to the target modality, homogeneous graph distillation acquires source modality filtering capabilities. When the source modality is reliable, its knowledge propagation weight is appropriately enhanced; when the source modality is unreliable, its contribution as a knowledge source is suppressed and pruned when necessary. When all incoming edges pointing to the same target modality in the cross-modal directed edges of the homogeneous graph are pruned, the target reliability-aware homogeneous edge weight corresponding to that target modality is determined to be zero. Otherwise, for each target modality, the pruned homogeneous edge weights pointing to that target modality are normalized within the group to determine the corresponding target reliability-aware homogeneous edge weights. This reduces the pollution of the shared semantic space by noisy modalities and makes homogeneous distillation more stable.

[0194] In one specific embodiment of this example, step 105 includes the following sub-steps:

[0195] S41. Calculate the mean of the reliability of each source mode by performing mean calculation, and construct the gating coefficient of each mode by comparing the reliability of each source mode with the mean of the reliability of the source modes.

[0196] S42. According to the source modes corresponding to the homogeneous edge weights of each target, the associated gating coefficients are used to weight them to determine the corresponding initial reliability perception homogeneous edge weights.

[0197] S43. Determine the source mode retention mask for each mode based on the comparison results between the source mode reliability and the mode reliability pruning threshold;

[0198] S44. Based on the source modes corresponding to each initial reliability-aware homogeneous edge weight, the associated source mode retention mask is used for weighting to construct the corresponding pruned homogeneous edge weights.

[0199] S45. Sum the homogeneous edge weights associated with the same target mode to determine the pre-modulation homogeneous edge weight sum value of each target mode;

[0200] S46. Sum the pruned homogeneous edge weights associated with the same target mode to determine the sum of the pruned homogeneous edge weights for each target mode.

[0201] S47. When the sum of the pruned homogeneous edge weights of any target mode is greater than the preset numerical stability constant, the pruned homogeneous edge weights associated with the sum of the pruned homogeneous edge weights are normalized, and the sum of the pre-modulation homogeneous edge weights pointing to the same target mode is scaled to determine the target reliability perception homogeneous edge weights corresponding to the pruned homogeneous edge weights.

[0202] S48. When the sum of the pruned homogeneous edge weights of any target mode is less than or equal to the preset numerical stability constant, the pruned homogeneous edge weights associated with the sum of the pruned homogeneous edge weights are set to zero as the corresponding target reliability-aware homogeneous edge weights.

[0203] It should be noted that the source mode is used Source mode reliability For source mode Target mode The corresponding target reliability-aware homogeneous edge weights are obtained by reliability modulation of the target homogeneous edge weights:

[0204] Determine the source mode gating coefficient : , Indicates the reliability modulation strength coefficient. This represents the mean reliability of the source modes. This represents the minimum value used to limit the gating coefficient. This represents the maximum value used to limit the gating coefficients. This design uses relative reliability instead of absolute reliability, highlighting the relative credibility between different modalities in the current sample while avoiding excessive perturbation of edge weights due to overall high or low reliability. Thus, the initial reliability-aware homogeneous edge weights are further obtained: , Represents the source mode To target mode The target homogeneous edge weight, Represents the source mode To target mode The initial reliability-perceived homogeneous edge weights; according to the above process, when the source mode When the reliability of a source modality is higher than the average reliability of the current samples, the edge weight for propagating knowledge to other modalities is enhanced, when the source modality... When the reliability of knowledge is below average, its contribution to knowledge dissemination is suppressed.

[0205] With source mode pruning enabled, further pruning is performed based on the modal reliability threshold. Determine whether the source mode should be preserved: , Represents the source mode Source mode preservation mask, This indicates an indicator function (outputting 1 if the condition is met, and 0 otherwise); this process characterizes the source mode. The binary pruning function determines whether to retain the modality for homogeneous graph distillation. If the source modality reliability is below a threshold, the modality is considered unsuitable as a knowledge source for homogeneous graph distillation in the current sample, and the output is 0. If the source modality reliability is above a threshold, the modality is considered suitable as a knowledge source for homogeneous graph distillation in the current sample, and the output is 1.

[0206] The process for determining homogeneous edge weights for target reliability perception is defined as follows:

[0207] ;

[0208] In the formula, Represents the source mode To target mode Homogeneous edge weights after pruning Indicates source modality index , Represents the source mode To target mode The target homogeneous edge weight, Represents the target mode The sum of homogeneous edge weights before modulation. Represents the source mode To target mode Homogeneous edge weights after pruning Represents the target mode The sum of homogeneous edge weights after pruning. This represents the numerical stability constant (for example, it can be taken as...). ), Represents the source mode To target mode The target reliability perception homogeneous edge weight;

[0209] As can be seen from the above process, by comparing the sum of the homogeneous edge weights after pruning each target mode with the preset numerical stability constant, when it is determined that at least one incoming edge pointing to the target mode is retained among the cross-modal directed edges of the homogeneous graph, the total weight of the incoming edges corresponding to that target mode remains the same as the total weight of the incoming edges before reliability modulation (i.e., When all incoming edges are pruned, the corresponding target reliability perception homogeneous edge weights are all zero.

[0210] Step 106: Using the private enhanced features obtained from cross-modal attention enhancement based on each private feature, construct the corresponding heterogeneous graph distillation node features using private semantic residuals, determine the associated heterogeneous sentiment intensity prediction value, and determine the target heterogeneous edge weights of each heterogeneous graph cross-modal directed edge from the source modality to the target modality based on heterogeneous graph distillation.

[0211] Heterogeneous sentiment intensity prediction refers to the prediction result obtained by performing sentiment prediction based on the features of distilled nodes in a heterogeneous graph.

[0212] In one specific embodiment of this example, step 106 includes the following sub-steps:

[0213] S51. Use a multimodal converter to perform cross-modal attention enhancement on each private feature and determine the private enhancement features;

[0214] S52. Flatten each private enhancement feature into a flattened heterogeneous feature, and after fully connected mapping, ReLU nonlinear activation, Dropout regularization and fully connected mapping, connect it with the residual of the associated flattened heterogeneous feature to determine the corresponding heterogeneous graph distillation node feature.

[0215] S53. Determine the heterogeneous sentiment intensity prediction value of each heterogeneous graph distillation node feature and output the sentiment prediction.

[0216] S54. Based on the features of the distilled nodes of the heterogeneous graph associated with each modality and the predicted value of the heterogeneous sentiment intensity, calculate the basic heterogeneous edge weights according to the cross-modal directed edges of the heterogeneous graph constructed from the source modality to the target modality, and perform Softmax normalization to determine the corresponding target heterogeneous edge weights.

[0217] It should be noted that, based on the multimodal transformer, cross-modal attention operations are performed according to the cross-modal enhancement directions of audio-enhanced text, video-enhanced text, text-enhanced audio, video-enhanced audio, text-enhanced video, and audio-enhanced video. The enhancement information of the same enhancement modality is then concatenated and input into the memory Transformer encoder for feature processing, thereby obtaining the private enhancement features of each modality, including text-private enhancement features. Audio private enhancement features and video private enhancement features ;

[0218] For any modality's private enhancement feature, after being flattened into a fixed-length flattened heterogeneous feature, it undergoes fully connected mapping, nonlinear activation, random deactivation, and another fully connected mapping. This result is then residually connected to the flattened heterogeneous feature to obtain the heterogeneous graph distillation node features used for heterogeneous graph distillation. The residual connection is used to preserve the original heterogeneous enhancement information of the Transformer encoder output, avoiding excessive loss of modality-private sentiment cues caused by multi-layer mapping;

[0219] Sentiment prediction based on heterogeneous graph distillation node features to determine the predicted value of heterogeneous sentiment intensity. This includes text heterogeneous sentiment intensity prediction values, audio heterogeneous sentiment intensity prediction values, and video heterogeneous sentiment intensity prediction values. Referring to the process of determining the target homogeneous edge weights, the target heterogeneous edge weights for each modality are determined by combining the features of the distilled nodes in the heterogeneous graph with the heterogeneous sentiment intensity prediction values.

[0220] Step 107: Combine the shared features, source modality reliability, private enhancement features, audio heterogeneous sentiment intensity prediction values, and video heterogeneous sentiment intensity prediction values ​​to perform sentiment prediction and output the basic sentiment intensity prediction value. Then, based on the determined text anchoring audio-video residual value and audio-video deviation gate value, determine the audio-video corrected basic sentiment intensity prediction value.

[0221] The audio-visual correction base emotion intensity prediction value refers to the emotion prediction result finally output by the multimodal emotion recognition model in this embodiment.

[0222] It should be noted that this embodiment first combines the shared features and private enhanced features of text, audio and video to obtain the basic multimodal fusion prediction result (i.e., the basic sentiment intensity prediction value). Based on this, it further determines whether to calibrate the basic sentiment intensity prediction value and the corresponding calibration range based on the deviation of the audio and video branch predictions from the basic main prediction value and the reliability of each modality.

[0223] In one specific implementation of this embodiment, the text scaling private enhancement feature serves as a mandatory anchoring input for the audio-visual residual expert. The calibration amount generated by the audio and video information is simultaneously constrained by the bias gate value, the upper limit of the residual amplitude, and the residual calibration strength, ensuring that it can only perform bounded calibration on the basic multimodal fusion prediction results. This avoids over-correction of the basic prediction results by low-reliability audio or video information. This process may include:

[0224] S61. Construct shared semantic fusion features by fusing various shared features, and align the output of the shared semantic fusion features through linear mapping and ReLU nonlinear activation;

[0225] The above process can be represented as: , In the formula, Indicates shared semantic fusion features, This represents a shared semantic fusion function (exemplarily, it can be implemented by self-attention and a splicing network). This indicates that the alignment shares semantic fusion features. Represents the ReLU activation function. Represents the shared weight mapping matrix. Indicates shared bias;

[0226] S62. Take the maximum value of the reliability of each source modality as the modality fusion reliability of the aligned shared semantic fusion feature, and determine the fusion scaling factor based on the reliability of each source modality and the modality fusion reliability;

[0227] The above process can be represented as: , In the formula, Indicates the blending scaling factor. This indicates taking the minimum value. This indicates taking the maximum value. Indicates the reliability of modal fusion. Indicates the reliability scaling strength. Represents the reliability fusion vector. This represents the mean of the reliability fusion vector. This indicates the lower limit of the scaling factor. This indicates the upper limit of the scaling factor. Indicates the reliability of the text source modality. Indicates the modal reliability of the audio source. Indicates the modal reliability of the video source. This indicates splicing; in this process, in order to enable the final fusion process to perceive the reliability of each modality in different samples, the reliability of the aforementioned source modalities is considered to be introduced into the fusion stage.

[0228] S63. After linear mapping and ReLU nonlinear activation, each private enhancement feature is concatenated with the aligned shared semantic fusion feature, and multiplied element-wise with the fusion scaling coefficient to output the scaled fusion feature;

[0229] The above process can be represented as: , In the formula, Representing modes Alignment-private enhancement features Represents the private weight mapping matrix. Indicates private bias. Representing modes Private enhancement features, Indicates scaling and blending features, This indicates element-wise multiplication; where scaling fusion features include text scaling private enhancement features ( ), audio scaling private enhancement features ( ), video scaling private enhancement features ( ) and scaling-shared alignment-shared semantic fusion ( );

[0230] S64. Perform sentiment prediction on scaled fused features using a two-layer perceptron with residual connections, and output the basic sentiment intensity prediction value.

[0231] S65. Take the text scaling private enhancement feature from the scaling fusion feature, and select the audio scaling private enhancement feature or / and video scaling private enhancement feature from the scaling fusion feature according to the preset expert modality, and concatenate them to construct the text anchored audio and video residual expert feature.

[0232] It should be noted that the text scaling private enhancement feature is a mandatory anchoring input for the audio-visual gated residual expert. Features are selected and concatenated according to the preset expert modality range (including audio expert modality, video expert modality, or audio-visual expert modality) to construct the text-anchored audio-visual residual expert feature. Specifically, when it is the audio expert modality, the text scaling private enhancement feature is concatenated with the audio scaling private enhancement feature; when it is the video expert modality, the text scaling private enhancement feature is concatenated with the video scaling private enhancement feature; and when it is the audio-visual expert modality, the text scaling private enhancement feature, the audio scaling private enhancement feature, and the video scaling private enhancement feature are concatenated.

[0233] S66. Perform linear mapping, ReLU nonlinear activation, Dropout regularization, linear mapping and tanh nonlinear activation on the text anchored audio and video residual expert features in sequence, and limit the amplitude according to the upper limit of the residual amplitude to determine the text anchored audio and video residual value.

[0234] The above process can be represented as: , , Represents the residual characteristics of audio and video. This indicates the Dropout layer (used to reduce the risk of overfitting). This represents a non-linear activation function (for example, the ReLU activation function can be used). Represents different residual weight mapping matrices, These represent different residual biases. This represents the text-anchored audio / video residual value. Indicates the upper limit of the residual amplitude. This represents the tanh activation function; it is understandable that, due to The output range is Therefore, there is This constraint ensures that audio and video information can only make limited corrections to the main prediction value, preventing the audio and video branches from directly overriding the sentiment judgment dominated by the text when there is strong noise, unclear expression, or conflict with the semantics of the text.

[0235] S67. Using the absolute value of the basic emotional intensity prediction value, and the absolute deviations of the audio heterogeneous emotional intensity prediction value and the video heterogeneous emotional intensity prediction value relative to the basic emotional intensity prediction value, respectively, audio and video deviation information is constructed by splicing them together.

[0236] The above process can be represented as: In the formula, Indicates audio-visual deviation information. This indicates taking the absolute value. This represents the predicted value of basic sentiment intensity. This represents the predicted value of heterogeneous sentiment intensity in audio. This represents the predicted value of heterogeneous sentiment intensity in the video. Indicates splicing;

[0237] S68. After concatenating the text-anchored audio-visual residual expert features, audio-visual deviation information and the reliability of each source mode, perform linear mapping, ReLU nonlinear activation, Dropout regularization, linear mapping and Sigmoid nonlinear activation in sequence, and output the audio-visual deviation gating value.

[0238] The above process can be represented as: , , Indicates the audio / video deviation gating value;

[0239] S69. Based on the text anchored audio-visual residual value and audio-visual deviation gate value, the basic sentiment intensity prediction value is calibrated with bounded audio-visual information to determine the audio-visual corrected basic sentiment intensity prediction value.

[0240] The above process can be represented as: , This indicates the audio / video correction of the base sentiment intensity prediction value. Indicates the audio / video residual calibration intensity coefficient ( ).

[0241] Step 108: Determine the counterfactual text to which the training multimodal emotion recognition data belongs through the multimodal emotion recognition model to be trained. The predicted value of the basic counterfactual emotion intensity of the text corresponding to the training multimodal emotion data is then determined. Combined with the predicted value of homogeneous emotion intensity, the homogeneous edge weight of target reliability perception, the features of distilled nodes in heterogeneous graphs, the predicted value of heterogeneous emotion intensity, the heterogeneous edge weight of the target, the residual value of text anchoring audio and video, and the audio and video deviation gate value, the total loss corresponding to the predicted value of audio and video correction is determined. The model parameters are then iteratively optimized until the trained multimodal emotion recognition model is determined.

[0242] Counterfactual text training multimodal sentiment data refers to training multimodal sentiment data in which text sentiment data is masked and replaced while audio and video sentiment data remain unchanged. In one specific embodiment of this example, the process of constructing counterfactual text training multimodal sentiment data includes: determining the positions of maskable lexical units based on attention masks and special lexical identifiers for the text sentiment data in the training multimodal sentiment data; selecting target maskable lexical units from these positions and replacing them with masked lexical units; and simultaneously maintaining the attention masks in the audio, video, and text sentiment data of the training multimodal sentiment data unchanged, thus determining the counterfactual text training multimodal sentiment data.

[0243] The text counterfactual base sentiment intensity prediction value refers to the base sentiment intensity prediction value output by a multimodal sentiment recognition model trained on counterfactual text as input, following the base sentiment prediction path. The same feature processing procedure.

[0244] In one specific embodiment of this example, the process of determining the total loss includes:

[0245] ;

[0246] In the formula, Indicates the total loss. This indicates the predicted loss for the main task. Indicates distillation loss in the isotropic diagram. Indicates the distillation loss in the heterogeneous diagram. Indicates cyclical loss. Indicates the reconstruction loss. This represents the loss due to shared feature similarity constraints. Indicates orthogonal loss. This indicates the loss caused by audio and video counterfactual gating monitoring. This indicates the text's counterfactual consistency loss. This indicates an auxiliary loss related to the emotional archetype.

[0247] It should be noted that by jointly optimizing the model parameters using multi-objective loss, the model can simultaneously learn multi-modal feature decoupling, reliable source modality screening, shared space knowledge propagation, private space complementary enhancement, and counterfactual calibration mechanisms, thereby obtaining a well-trained multi-modal emotion recognition model.

[0248] In a more specific implementation of this embodiment, the main task prediction loss includes:

[0249] ;

[0250] In the formula, Indicates batch size, Indicates the sample index. This indicates the audio / video correction of the base sentiment intensity prediction value. This represents the labeled value indicating the true intensity of emotion.

[0251] In a more specific implementation of this embodiment, the reconstruction loss includes:

[0252] ;

[0253] In the formula, Indicates modal index, Represents text modality, Indicates video modality, Indicates audio modality, Representing modes Unified features of the reconstructed dimensions ( Including unified features of text reconstruction dimension Unified features in audio reconstruction dimension Unified features with video reconstruction dimension ), Representing modes The dimensional uniformity features Denotes the square of the L2 norm. Representing modes Shared features Representing modes Private characteristics, Indicates splicing, Representing modes Feature reconstruction decoders (including text feature reconstruction decoders) Audio Feature Reconstruction Decoder and video feature reconstruction decoder ).

[0254] In a more specific implementation of this embodiment, the cyclic loss includes:

[0255] ;

[0256] In the formula, Representing modes Recoding private features, Representing modes The proprietary encoder.

[0257] In a more specific implementation of this embodiment, the orthogonal loss includes:

[0258] ;

[0259] In the formula, This indicates taking the maximum value. Represents cosine similarity. This represents the interval threshold (used to control the upper limit of similarity between private and shared features).

[0260] In a more specific implementation of this embodiment, the shared feature similarity constraint loss includes:

[0261] ;

[0262] In the formula, Represents the set of valid anchor point feature objects. Indicates the number of valid anchor point feature objects. Indicates the anchor feature object index. Represents a set of objects with positive characteristics. Indicates the number of objects with positive features. Represents a set of objects with negative characteristics. Indicates the number of objects with negative features. Indicates the index of the positive feature object. Indicates the index of the negative feature object. Represents anchor point feature object The corresponding true emotional intensity label value, Represents a negative feature object The corresponding true emotional intensity label value, Represents anchor point feature object Shared similarity features are obtained by flattening and linear mapping of shared features. Represents positive feature objects Shared similarity features are obtained by flattening and linear mapping of shared features. Represents a negative feature object Shared similarity features are obtained by flattening and linear mapping of shared features. Represents the complete set of feature objects (including) (one characteristic object) Indicates the sample index. Indicates removal. Represents positive feature objects The corresponding true emotional intensity label value, Represents anchor point feature object Corresponding sample index, Represents anchor point feature object The corresponding modal index, Indicates the first The true sentiment intensity labeled for each sample This represents the dynamic interval determined based on the difference in the actual emotional intensity labeled values ​​between the anchor feature object and the negative feature object. "0.15" indicates the dynamic interval scaling factor.

[0263] It should be noted that this embodiment uses reconstruction loss, cyclic loss, orthogonality loss, and shared feature similarity constraint loss to jointly constrain and achieve different decoupling objectives, making shared information more suitable for reliability estimation and homogeneous graph distillation, and private information more suitable for heterogeneous augmented graph distillation. Specifically:

[0264] After modal information separation, the intra-modal feature reconstruction decoder receives the concatenated vectors of private and shared features of the same modality and reconstructs the original dimensional unified features of that modality, obtaining the corresponding reconstructed dimensional unified features. Through reconstruction loss constraints, the model is required to retain sufficiently complete modal information after decoupling, ensuring that the original modal information is preserved after concatenation, and avoiding the degradation of decoupled features into invalid representations. That is, if the reconstruction error is small, it indicates that the original modal features can be well restored after concatenation; if the reconstruction error is large, it indicates that important modal information may have been lost during feature decoupling. Simultaneously, because the feature reconstruction decoder needs to utilize... and This can enhance the complementarity between private and shared features;

[0265] Considering that the reconstruction loss only guarantees that the information after each modality reconstruction is approximately the original modality information, and cannot guarantee that the reconstructed features still retain the modality-specific information in the private encoder, this embodiment re-inputs the unified features of the reconstruction dimension into the corresponding private encoder to obtain recoded private features, and constrains them to be consistent with the original private features through cyclic loss, which helps to improve the stability of private features and avoid the loss of modality-specific information during the reconstruction process;

[0266] Reconstruction loss and recurrence loss alone cannot guarantee complete separation of private and shared information. To avoid information leakage in the model, such as the fact that most private and shared information is repetitive and becomes invalid or degraded when it enters the private encoder and shared encoder, this embodiment uses orthogonal loss based on cosine embedding loss. By calculating and minimizing the cosine similarity between the private features and shared features of each modality, the similarity between the two is suppressed. This allows the private encoder to retain more modality-specific information and the shared encoder to retain more cross-modal common information. In this way, orthogonal loss reduces the information redundancy between private and shared information in the same modality, making the private encoder representation more modality-specific and the shared encoder representation more cross-modal shared information.

[0267] To ensure that samples with similar emotional intensities are closer together in the shared feature space, and samples with significantly different emotional intensities are further apart, this embodiment constructs a shared feature similarity constraint. First, to ensure that shared features from different modalities have the same feature dimension for cross-modal cosine similarity calculation, the shared features of the text, audio, and video modalities are flattened. A linear mapping layer corresponding to each modality is then used to map the flattened shared features to a shared similarity feature space with consistent dimensions, thus determining the shared similarity mapping features for each modality. Simultaneously, a positive feature object is defined as a feature object whose true emotional intensity label is the same as the anchor feature object (excluding the anchor feature object itself), and a negative feature object is defined as a feature object whose true emotional intensity label is different from the anchor feature object. Anchor feature objects, positive feature objects, and negative feature objects are all determined by the original sample index and the modality index. Specifically, the shared similarity mapping features of the text, audio, and video corresponding to each original sample in a training batch are each taken as a feature object, and any feature object is determined by the original sample index and the modality index, and can be represented as... The method uses any feature object as an anchor feature object, identifies other feature objects with the same true sentiment intensity label as the anchor as positive feature objects, and identifies feature objects with different true sentiment intensity labels as negative feature objects. Since each modal feature object inherits the true sentiment intensity label of its original sample, different modal feature objects of the same original sample have the same label and form a positive feature relationship. Modal feature objects corresponding to other original samples with the same label also form a positive feature relationship, while feature objects with different labels form a negative feature relationship. This constrains positive features to have a higher cosine similarity than negative features. It can be understood that this loss helps to form an ordered structure related to sentiment intensity labels in the joint shared representation space, but it is not equivalent to forcing all cross-modal shared features of the same sample to be pairwise equal.

[0268] In a more specific implementation of this embodiment, the emotion prototype-assisted loss includes:

[0269] ;

[0270] In the formula, Represents discrete sentiment category-constrained loss. This indicates the loss of continuous emotional intensity constraint. Indicates the regression constraint weights;

[0271] The discrete emotion category constraint loss includes:

[0272] ;

[0273] In the formula, Represents the natural logarithm. Representing modes The probability distribution of emotional prototypes Represents a minimal constant;

[0274] The continuous emotion intensity constraint loss includes:

[0275] ;

[0276] In the formula, This indicates taking the minimum value. This represents the predictive value of continuous sentiment intensity. This represents the labeled value indicating the true intensity of emotion. This represents the transpose of the emotional intensity anchor vector.

[0277] It should be noted that, in order to enhance the semantic recognizability of the emotional prototype, this embodiment applies both discrete emotional category constraints and continuous emotional intensity constraints to the probability distribution of the emotional prototype. Specifically, cross-entropy loss is used to constrain the consistency between the probability distribution of the emotional prototype and the emotional intensity category. This loss makes the emotional prototype corresponding to the real emotional label obtain a higher probability. The emotional prototype probability is used to perform a weighted summation of the emotional intensity anchor points to obtain the continuous emotional intensity prediction value. Then, the smoothing absolute error is used to supervise it. Thus, the continuous emotional intensity constraint can be used to constrain the continuous emotional intensity prediction value obtained by the prototype weighting to be close to the real emotional label.

[0278] In a more specific embodiment of this example, the isomorphic distillation loss includes:

[0279] ;

[0280] In the formula, This represents the weighting coefficient for distillation loss in the homogeneous graph. This indicates the weighted prediction of distillation loss. Indicates the edge weight regularization loss;

[0281] The weighted predicted distillation loss includes:

[0282] ;

[0283] In the formula, Indicates the source modality index. Indicates the target modality index. This represents the set of directed edges across modalities in a homogeneous graph. Represents the source mode To target mode The target reliability perception homogeneous edge weight The distance metric function can be either L1 distance or the square of L2 distance, but this embodiment does not limit this. Represents the target mode The predicted value of homogeneous sentiment intensity Represents the source mode The predicted value of homogeneous sentiment intensity;

[0284] Among them, edge weight regularization loss includes:

[0285] ;

[0286] In the formula, This represents the edge weight regularization weight coefficient. This represents the average target reliability perception edge weight. Represents the source mode To target mode Homogeneous fixed prior edge weight vectors Indicates batch size, Indicates the sample index. Indicates the first Source mode in each sample To target mode The target reliability perception homogeneous edge weight.

[0287] It should be noted that, to constrain the homogeneous graph distillation process, this embodiment further constructs a homogeneous graph distillation loss, including a weighted prediction distillation loss and an edge weight regularization loss. The prediction distillation term is used to constrain the transfer of knowledge from the source mode prediction result to the target mode (as can be seen from the expression, when the source mode...). When the target reliability perception homogeneous edge weight is large, its prediction result is related to the target mode. The distillation constraint is stronger when the source mode When the target reliability perception homogeneous edge weight is small, its relation to the target mode The impact is weakened), and the edge weight regularization term is used to prevent the edge weights of the dynamic graph from deviating excessively from the preset edge weight prior, so that different modalities form stable consistency constraints in the shared semantic space, ensuring the stability of the edge weights of the dynamic graph, and combined with the target reliability-aware homogeneous edge weights, suppressing the propagation of erroneous knowledge by low-reliability source modalities.

[0288] Specifically, in a preferred implementation, the process of determining homogeneous fixed prior edge weight vectors may include: setting homogeneous original edge bias vectors, such as... Understandably, this original vector can be adaptively set according to the pre-defined direction of knowledge dissemination to be emphasized. For example, if the desired direction is the shared knowledge dissemination from text to video or from text to audio, it can be set according to the set of cross-modal directed edges in a homogeneous graph. Then in and The position is set to 1; then, the Softmax function with a temperature parameter is used to normalize the homogeneous original edge bias vector to obtain a homogeneous fixed prior edge weight vector, such as... .

[0289] In a more specific embodiment of this example, the heterogeneity map distillation loss includes:

[0290] ;

[0291] In the formula, This represents the weighting coefficient for distillation loss in the heterogeneous plot. Indicates heterogeneous prediction of distillation loss, Indicates the characteristic distillation loss of heterogeneous materials. Indicates the heterogeneous edge weight regularization loss;

[0292] Among them, the predicted distillation losses by heterogeneity include:

[0293] ;

[0294] In the formula, This represents the set of directed edges across modalities in a heterogeneous graph. Represents the source mode To target mode Target heterogeneous edge weights Represents the target mode The predicted value of heterogeneous sentiment intensity Represents the source mode The predicted value of heterogeneous sentiment intensity;

[0295] Among them, heterogeneous characteristic distillation losses include:

[0296] ;

[0297] In the formula, The characteristics of the distillation nodes in the heterogeneous graph are represented. The characteristics of distillation nodes in the heterogeneous graph are represented;

[0298] Among them, heterogeneous edge weight regularization loss includes:

[0299] ;

[0300] In the formula, This represents the edge weight regularization weight coefficient. This represents the batch average target heterogeneous edge weight. Represents the source mode To target mode Heterogeneous fixed prior edge weight vectors Indicates the first Source mode in each sample To target mode The target is heterogeneous edge weights.

[0301] It should be noted that, unlike homogeneous graph distillation which focuses on reliable knowledge dissemination in a shared semantic space, heterogeneous graph distillation emphasizes the complementary enhancement between modal private information. The two work together to enable the model to learn cross-modal shared semantics while retaining and utilizing the emotional cues unique to each modality, thereby improving the integrity and robustness of multimodal emotional representation. To constrain the private modal representation after heterogeneous enhancement, a heterogeneous graph distillation loss is constructed to model the complementary relationships between private information of different modalities while preserving the differences between text, audio, and video modalities.

[0302] Among them, the heterogeneous prediction distillation loss is used to constrain the transmission of source mode prediction information to the target mode, so that the heterogeneous enhancement branches form complementary constraints at the prediction level. The heterogeneous feature distillation loss can enable the private representation after heterogeneous enhancement to learn cross-modal complementary relationships while maintaining modal differences.

[0303] Among them, the heterogeneous edge weight regularization loss is used to constrain the average target heterogeneous edge weight of each heterogeneous directed edge in the current training batch to be close to the preset normalized fixed prior edge weight, thereby suppressing the unconstrained drift of heterogeneous edge weights relative to the preset knowledge propagation structure at the batch statistical level, while allowing different samples to dynamically adjust the corresponding heterogeneous edge weights according to their private features. It can be understood that when the fixed prior edge weights intentionally strengthen some cross-modal propagation directions, this regularization loss will guide the corresponding propagation direction to maintain a relatively high weight in the batch average sense, rather than eliminating the edge weight concentration phenomenon.

[0304] Specifically, in a preferred implementation, the process of determining the heterogeneous fixed prior edge weight vector may include: setting the heterogeneous original edge bias vector. The original vector can be adaptively set according to the pre-defined direction of knowledge propagation to be highlighted, based on the set of cross-modal directed edges in a heterogeneous graph. ,like The implication is that, in addition to text-to-video / audio propagation, heterogeneous private information guidance from video to audio is further enhanced; subsequently, a Softmax function with a temperature parameter is used to normalize the heterogeneous original edge bias vector, resulting in a heterogeneous fixed prior edge weight vector, such as... .

[0305] In a more specific implementation of this embodiment, the audio / video counterfactual gating supervision loss includes:

[0306] ;

[0307] In the formula, Indicates the first The counterfactual supervision weights for each sample's audio and video recordings. Indicates the first The audio / video deviation gating value for each sample. Indicates the first Audio and video gating supervision targets for each sample. Indicates the target of audio and video gating supervision. This represents the Sigmoid activation function. This represents the contribution value of the audio and video residuals. Indicates the boundary parameters of audio and video contribution. This represents the audio / video temperature coefficient (used to control the smoothness of the monitored target from off to on). Indicates the main prediction error. Indicates counterfactual prediction error. This represents the predicted value of basic sentiment intensity. This indicates taking the absolute value. This indicates the predicted base sentiment intensity value after counterfactual correction for audio and video recordings. Represents the counterfactual residual coefficient. This represents the text anchoring audio and video residual value.

[0308] It should be noted that this embodiment introduces audio-visual counterfactual gating supervision. By constructing a counterfactual prediction that is not controlled by the current audio-visual deviation gating value, the contribution of the audio-visual residuals is explicitly evaluated, and this contribution is converted into a gating supervision signal. Then, based on changes in the counterfactual error, it learns when to turn the audio-visual residuals on or off. It can be understood that if... This indicates that the prediction error decreases after adding the audio and video residuals, suggesting that the audio and video residuals in the current sample have a positive contribution. The statement indicates that the prediction error increases after adding audio and video residuals, suggesting that the audio and video residuals in the current sample may be harmful. Therefore, the greater the contribution of the audio and video residuals, the more... The closer to 1, the smaller or negative the contribution of the audio and video residuals. The closer it is to 0.

[0309] In a more specific implementation of this embodiment, the text counterfactual consistency loss includes:

[0310] ;

[0311] In the formula, Indicates the first The textual counterfactual sample weights of each sample. Indicates an indicator function, Indicates the first The absolute value of the true sentiment intensity labeled for each sample This represents the threshold for distinguishing between samples with strong emotional bias and neutral samples. Indicates the polarity consistency loss weight. Indicates the first Polarity consistency loss for each sample Indicates the weight of intensity consistency loss. Indicates the first The intensity consistency loss of each sample Indicates the weight of neutral consistency loss. Indicates the first Neutral consistency loss for each sample;

[0312] Among them, the polarity consistency loss includes:

[0313] ;

[0314] In the formula, Indicates the polarity descent interval, Indicates the directional decrease. This represents the predicted counterfactual sentiment intensity of the text.

[0315] Among them, the strength consistency loss includes:

[0316] ;

[0317] In the formula, Indicates the interval of intensity decrease. Indicates the degree of decrease in emotional intensity;

[0318] The neutral consistency loss includes:

[0319] ;

[0320] In the formula, This indicates smoothed L1 loss;

[0321] The text counterfactual sample weights include:

[0322] ;

[0323] In the formula, This represents the minimum value of the text counterfactual sample weight (used to prevent the text counterfactual sample weight from being completely zero). This indicates the validity gate for text perturbation. Indicates the boundary mixing factor. This represents the absolute value of the text perturbation effect. Indicates the boundary of the text perturbation effect. Indicates the temperature coefficient of the text. This indicates the text perturbation effect. Indicates boundary weights, This represents the natural exponential function. This represents the boundary smoothing coefficient.

[0324] It should be noted that, in order to further protect the dominant role of text modality in sentiment polarity judgment, this embodiment introduces text counterfactual consistency and soft routing adjustment. Counterfactual samples are constructed through token-level text masking to test whether the model prediction changes in a reasonable direction after the text sentiment evidence is weakened. At the same time, the sample-level weights of the text counterfactual loss are dynamically adjusted according to the text perturbation effect and the sample boundary position, thereby improving training stability. Specifically:

[0325] Based on the baseline sentiment intensity prediction value Used as a reference predicted value;

[0326] according to , and the true emotional intensity label value Constructing directional descent ,when If the predicted value decreases after the text is masked, then This indicates that the positive textual evidence was reasonably weakened, when If the predicted value shifts towards the neutral direction (i.e., the predicted value increases) after the text is masked, then the same applies. This indicates that the negative textual evidence has been reasonably weakened; based on and Constructing the decrease in emotional intensity ,like This indicates that the prediction strength decreases after the textual evidence is masked, suggesting that the model does indeed rely on the sentiment intensity information in the text.

[0327] In textual counterfactual consistency loss, for clearly positive or negative sentiment samples, the prediction after masking textual evidence is constrained to decrease sufficiently along the direction of the true label, and the prediction strength is constrained to decrease reasonably, as shown in polarity consistency loss. This indicates that the prediction has produced a sufficient directional decline, at which point the loss is 0. The statement indicates that the decline is insufficient and the model needs to be penalized. The strength consistency loss is used to prevent the model from maintaining an overly strong sentiment intensity judgment after the text evidence is weakened. For neutral samples with small absolute values ​​of sentiment intensity labels, compared with clear sentiment samples, there may not be clear positive or negative sentiment evidence in the text. This constrains the text counterfactual prediction to be consistent with the basic prediction in order to avoid drastic fluctuations in prediction caused by masking operations.

[0328] To avoid applying a fixed strength to all samples for text counterfactual loss, this embodiment further introduces a soft-route adjustment mechanism to scale the text counterfactual loss at the sample level. Specifically: sample-level soft-route weights are generated based on the text perturbation effect and the sample's distance from the sentiment boundary, dynamically adjusting the strength of the text counterfactual consistency loss to avoid training instability caused by fixed-strength counterfactual constraints; in the text perturbation effectiveness gating, when the prediction change is large after masking the text, it indicates that the text evidence in the sample is relatively obvious, and the text perturbation effectiveness gating is large; when the prediction change is small after masking the text, it indicates that the text perturbation effect is not obvious, and the text perturbation effectiveness gating is small; in the boundary weights, when When the sample size is smaller, the sample is closer to the positive and negative sentiment boundary. Larger, when When the sample size is large, the sample is far from the decision boundary. The boundary weights are relatively small; however, to avoid completely suppressing samples far from the boundary, the boundary weights are mapped to boundary mixing factors. , The range of values ​​is It can enhance text counterfactual supervision of samples near the sentiment boundary without completely ignoring samples far from the boundary but with obvious text perturbations. In the text counterfactual sample weights, the text counterfactual sample weights are greater when text occlusion produces significant prediction changes and the sample is close to the decision boundary, and the text counterfactual sample weights are reduced when the text perturbation effect is weak or the sample is far from the boundary.

[0329] Step 109: Determine the target audio and video correction base emotion intensity prediction value corresponding to the multimodal emotion data to be identified using the trained multimodal emotion recognition model.

[0330] It should be noted that the multimodal emotion data to be identified is input into the trained multimodal emotion recognition model, and the final emotion recognition result is determined as the target audio and video corrected basic emotion intensity prediction value according to the same feature processing process as steps 101-107.

[0331] To verify the effectiveness of the method in this embodiment, experiments were conducted on the CMU-MOSI and CMU-MOSEI datasets. According to the experimental results shown in Table 1, the method in this embodiment has achieved certain improvements compared to existing models.

[0332] Table 1 Experimental Results

[0333]

[0334] In this embodiment of the invention, shared private decoupling coding separates cross-modal shared semantics and modality-specific emotional information, reducing semantic interference caused by the mixing of information from different modalities. Combining cross-modal consistency reliability with emotional prototype deterministic reliability allows source modality reliability to simultaneously reflect the degree of consistency between modalities and the clarity of emotional intensity attribution, enabling dynamic identification of reliable source modalities in different samples and thus improving the task relevance of reliability judgment. In homogeneous graph distillation, source modality reliability is used to modulate edge weights and suppress low-reliability source modalities, inhibiting the propagation of erroneous knowledge in the shared semantic space and improving graph distillation stability. In heterogeneous graph distillation, modality-private differences are preserved and cross-modal complementary enhancement is performed, avoiding excessive pruning of audio and video private emotional cues. This approach combines shared consistency learning and private complementary learning. In the final prediction stage, based on the reliability-aware multimodal fusion prediction of the base sentiment intensity prediction using shared features and private enhanced features from text, audio, and video, bounded gating calibration is introduced using audio-video gating residuals. Furthermore, through counterfactual supervision, explicit constraints are applied to audio-video gating and text perturbation responses, ensuring that audio and video only contribute to the base prediction with limited calibration when they are relevant to the current sample. This reduces the risk of over-correction of the base multimodal fusion judgment due to low-reliability audio-video information. The proposed method provides a multimodal sentiment recognition approach guided by sentiment prototype source modality screening and counterfactual residual calibration for text, audio, and video information, improving the accuracy, robustness, and interpretability of multimodal sentiment recognition.

[0335] Please see Figure 4 Embodiment 2 of the present invention provides a multimodal counterfactual emotion recognition device based on emotion prototype-driven approach, comprising:

[0336] The multimodal feature extraction module 401 is used to input the training multimodal sentiment data into the multimodal sentiment recognition model to be trained, and extract the dimensional unified features of each modality based on the training multimodal sentiment data;

[0337] The multimodal feature decoupling coding module 402 is used to perform private coding and shared coding on the unified features of each dimension respectively, and to determine the private features and shared features of each modality;

[0338] The homogeneous graph distillation module 403 is used to construct the corresponding homogeneous graph distillation node features based on the shared semantic residuals of each shared feature and determine the associated homogeneous sentiment intensity prediction value. Through homogeneous graph distillation, the target homogeneous edge weights of each homogeneous graph cross-modal directed edge from the source modality to the target modality are determined.

[0339] Modal reliability assessment module 404 is used to determine cross-modal relative reliability based on the corresponding shared features, determine prototype reliability by using the emotional intensity prototype vector and the emotional prototype probability distribution between the features of each homogeneous graph distillation node, and perform reliability weighted fusion according to modal association to determine the source modal reliability of each modality.

[0340] The reliability homogeneous graph distillation module 405 is used to perform gated weighting and source mode pruning on the target homogeneous edge weights associated with the source modes based on the reliability of each source mode. After determining the corresponding pruned homogeneous edge weights, the pruned homogeneous edge weights pointing to the same target mode are normalized and scaled or set to zero within the group according to whether they have all been pruned, and the target reliability-aware homogeneous edge weights corresponding to each pruned homogeneous edge weight are determined.

[0341] The heterogeneous augmented graph distillation module 406 is used to construct the corresponding heterogeneous graph distillation node features by using the private augmented features obtained by cross-modal attention enhancement based on each private feature, and to determine the associated heterogeneous sentiment intensity prediction value. Based on the heterogeneous graph distillation, the target heterogeneous edge weights of each heterogeneous graph cross-modal directed edge from the source modality to the target modality are determined.

[0342] The audio and video gated residual prediction module 407 is used to combine the shared features, the reliability of each source modality, each private enhancement feature, and the audio heterogeneous sentiment intensity prediction value and the video heterogeneous sentiment intensity prediction value to perform sentiment prediction and output the basic sentiment intensity prediction value. Based on the determined text anchored audio and video residual value and the audio and video deviation gate value, the modified audio and video basic sentiment intensity prediction value is determined.

[0343] The model training and optimization module 408 is used to determine the counterfactual text to which the training multimodal emotion recognition data belongs through the multimodal emotion recognition model to be trained. The predicted value of the basic counterfactual emotion intensity of the text corresponding to the training multimodal emotion data is determined by combining the homogeneous emotion intensity prediction value, the target reliability perception homogeneous edge weight, the features of the distilled nodes of the heterogeneous graph, the predicted value of the heterogeneous emotion intensity, the target heterogeneous edge weight, the text anchoring audio and video residual value and the audio and video deviation gate value. The module determines the total loss corresponding to the audio and video corrected basic emotion intensity prediction value and iteratively optimizes the model parameters until the trained multimodal emotion recognition model is determined.

[0344] The emotion recognition module 409 is used to determine the target audio and video corrected basic emotion intensity prediction value corresponding to the multimodal emotion data to be recognized by using a trained multimodal emotion recognition model.

[0345] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0346] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0347] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0348] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0349] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0350] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal counterfactual sentiment recognition method based on sentiment prototype-driven approach, characterized in that, include: The training multimodal sentiment data is input into the multimodal sentiment recognition model to be trained, and the dimensional unified features of each modality are extracted based on the training multimodal sentiment data; Private and shared encodings are performed on the unified features of each dimension respectively to determine the private and shared features of each modality; The shared features are used to construct the corresponding homogeneous graph distillation node features based on the shared semantic residuals and determine the associated homogeneous sentiment intensity prediction values. The target homogeneous edge weights of the cross-modal directed edges of each homogeneous graph from the source modality to the target modality are determined by homogeneous graph distillation. Cross-modal relative reliability is determined based on the shared features described above. Prototype reliability is determined by using the emotional intensity prototype vector and the emotional prototype probability distribution between the features of each homogeneous graph distillation node. Reliability weighted fusion is performed according to modal association to determine the source modality reliability of each modality. Based on the reliability of each source mode, the target homogeneous edge weight associated with the source mode is gated and weighted, and the source mode is pruned to determine the corresponding pruned homogeneous edge weight. Then, the pruned homogeneous edge weights pointing to the same target mode are normalized and scaled or set to zero within the group according to whether they have all been pruned, and the target reliability-aware homogeneous edge weights corresponding to each pruned homogeneous edge weight are determined. Using the private features obtained by cross-modal attention enhancement based on the private features, construct the corresponding heterogeneous graph distillation node features by performing private semantic residuals, determine the associated heterogeneous sentiment intensity prediction value, and determine the target heterogeneous edge weight of each heterogeneous graph cross-modal directed edge from the source modality to the target modality based on heterogeneous graph distillation. By combining the shared features, source modality reliability, private enhancement features, audio heterogeneous sentiment intensity prediction values, and video heterogeneous sentiment intensity prediction values, a basic sentiment intensity prediction value is output for sentiment prediction. Based on the determined text anchoring audio-video residual value and audio-video deviation gate value, the audio-video corrected basic sentiment intensity prediction value is determined. The training multimodal emotion recognition model determines the counterfactual text corresponding to the training multimodal emotion data. Then, it combines the homogeneous emotion intensity prediction, the target reliability-perceived homogeneous edge weight, the heterogeneous graph distillation node features, the heterogeneous emotion intensity prediction, the target heterogeneous edge weight, the text anchoring audio-visual residual value, and the audio-visual deviation gate value to determine the total loss corresponding to the audio-visual corrected basic emotion intensity prediction value. The model parameters are then iteratively optimized until the trained multimodal emotion recognition model is determined. The trained multimodal emotion recognition model determines the target audio and video correction base emotion intensity prediction value corresponding to the multimodal emotion data to be recognized.

2. The multimodal counterfactual sentiment recognition method based on sentiment prototype-driven approach according to claim 1, characterized in that, The process involves constructing corresponding homogeneous graph distillation node features based on shared semantic residuals using the shared features and determining the associated homogeneous sentiment intensity prediction values. Homogeneous graph distillation is used to determine the target homogeneous edge weights of cross-modal directed edges from the source modality to the target modality, including: Each of the shared features is flattened into flattened homogeneous features, and after passing through fully connected mapping, ReLU nonlinear activation, Dropout regularization, and fully connected mapping, it is connected with the residual of the associated flattened homogeneous features to determine the corresponding homogeneous graph distillation node features. Determine the homogeneous sentiment intensity prediction value of each homogeneous graph distillation node feature as the output of sentiment prediction; Based on the distillation node features of the homogeneous graph associated with each modality and the predicted value of homogeneous sentiment intensity, the basic homogeneous edge weights are calculated according to the cross-modal directed edges of the homogeneous graph constructed from the source modality to the target modality, and then Softmax normalization is performed to determine the corresponding target homogeneous edge weights.

3. The multimodal counterfactual sentiment recognition method based on sentiment prototype-driven approach according to claim 1, characterized in that, The step of determining cross-modal relative reliability based on the corresponding shared features includes: Temporal attention pooling is performed on the shared features of each modality to determine the corresponding pooling features; Channel recalibration is performed on each of the pooling features to output the corresponding channel enhancement features; For any channel enhancement feature of an evaluation modality, a difference feature is constructed between it and the channel enhancement features of other modalities by feature subtraction. The channel enhancement features of each modality are then concatenated with the difference features of the evaluation modality and input into a multilayer perceptron to output the modal reliability score of the evaluation modality. The reliability scores of each mode are normalized by using the Softmax activation function to determine the cross-modal relative reliability of each mode.

4. The multimodal counterfactual sentiment recognition method based on sentiment prototype-driven approach according to claim 1, characterized in that, The method of determining prototype reliability by using the prototype probability distribution of sentiment intensity between the prototype vector of sentiment intensity and the features of each distilled node in the homogeneous graph includes: The emotional intensity prototype vector is normalized into a normalized emotional intensity prototype vector. After mapping the features of the associated homogeneous graph distillation nodes to the prototype space through the query mapping function of each modality, the corresponding prototype query vector is normalized and constructed, and the prototype similarity between the vector and the normalized sentiment intensity prototype vector is determined respectively. The Softmax activation function is used to normalize the prototype similarity of each modality to determine the probability distribution of the emotional prototype of each modality. The distribution entropy of each of the aforementioned emotional prototype probability distributions is calculated to determine the corresponding prototype distribution entropy, and the inverse value of the corresponding normalized prototype distribution entropy is determined as the associated prototype reliability.

5. The multimodal counterfactual sentiment recognition method based on sentiment prototype-driven approach according to claim 1, characterized in that, After determining the corresponding pruned homogeneous edge weights based on the source modality reliability by gating and weighting the target homogeneous edge weights associated with each source modality and by source modality pruning, the pruned homogeneous edge weights pointing to the same target modality are normalized and scaled within the group or set to zero based on whether they have all been pruned, thus determining the target reliability-aware homogeneous edge weights corresponding to each pruned homogeneous edge weight, including: The mean reliability of each source mode is calculated by averaging the reliability of each source mode, and the gating coefficient of each mode is constructed by comparing the reliability of each source mode with the mean reliability of the source modes. The corresponding initial reliability-aware homogeneous edge weights are determined by weighting the source modes corresponding to the homogeneous edge weights of each target using associated gating coefficients. The source mode retention mask for each mode is determined based on the comparison results between the source mode reliability and the mode reliability pruning threshold. Based on the source modes corresponding to the initial reliability-aware homogeneous edge weights, the associated source mode retention mask is used for weighting to construct the corresponding pruned homogeneous edge weights; The summation of the homogeneous edge weights associated with the same target mode is performed to determine the pre-modulation summation of the homogeneous edge weights for each target mode. The weights of the pruned homogeneous edges associated with the same target mode are summed to determine the sum of the pruned homogeneous edge weights for each target mode. When the sum of the pruned homogeneous edge weights of any of the target modes is greater than a preset numerical stability constant, the pruned homogeneous edge weights associated with the sum of the pruned homogeneous edge weights are normalized, and the sum of the pre-modulation homogeneous edge weights pointing to the same target mode is scaled to determine the target reliability perception homogeneous edge weights corresponding to the pruned homogeneous edge weights. When the sum of the pruned homogeneous edge weights of any of the target modes is less than or equal to a preset numerical stability constant, the pruned homogeneous edge weights associated with the sum of the pruned homogeneous edge weights are set to zero as the corresponding target reliability-aware homogeneous edge weights.

6. The multimodal counterfactual sentiment recognition method based on sentiment prototype-driven approach according to claim 1, characterized in that, The process involves constructing corresponding heterogeneous graph distillation node features using private enhanced features obtained through cross-modal attention enhancement based on the private features, constructing corresponding heterogeneous graph distillation node features using private semantic residuals, determining the associated heterogeneous sentiment intensity prediction values, and determining the target heterogeneous edge weights of each heterogeneous graph cross-modal directed edge from the source modality to the target modality based on heterogeneous graph distillation, including: A multimodal converter is used to perform cross-modal attention enhancement on each of the private features to determine the private enhanced features; Each of the private enhancement features is flattened into flattened heterogeneous features, and after passing through fully connected mapping, ReLU nonlinear activation, Dropout regularization and fully connected mapping, it is connected with the residual of the associated flattened heterogeneous features to determine the corresponding heterogeneous graph distillation node features. The heterogeneous sentiment intensity prediction value is output by determining the features of each heterogeneous graph distillation node; Based on the features of distilled nodes in the heterogeneous graph associated with each modality and the predicted value of heterogeneous sentiment intensity, the basic heterogeneous edge weights are calculated according to the cross-modal directed edges of the heterogeneous graph constructed from the source modality to the target modality, and then Softmax normalization is performed to determine the corresponding target heterogeneous edge weights.

7. The multimodal counterfactual sentiment recognition method based on sentiment prototype-driven approach according to claim 1, characterized in that, The method combines the shared features, source modality reliability, private enhancement features, audio heterogeneous sentiment intensity prediction values, and video heterogeneous sentiment intensity prediction values ​​to perform sentiment prediction and output a basic sentiment intensity prediction value. Then, based on the determined text-anchored audio-video residual value and audio-video deviation gate value, a corrected audio-video basic sentiment intensity prediction value is determined, including: Shared semantic fusion features are constructed by fusing the shared features described above, and the output of the shared semantic fusion features is aligned by linear mapping and ReLU nonlinear activation. The maximum value of the reliability of each source modality is taken as the modality fusion reliability of the aligned shared semantic fusion feature, and the fusion scaling factor is determined based on the reliability of each source modality and the modality fusion reliability; Each of the private enhanced features is linearly mapped and ReLU nonlinearly activated, then concatenated with the aligned shared semantic fusion feature, and multiplied element-wise with the fusion scaling coefficient to output the scaled fusion feature; The scaled fused features are used to predict sentiment through a two-layer perceptron with residual connections, and the basic sentiment intensity prediction value is output. The text scaling private enhancement features in the scaling fusion features are combined with the audio scaling private enhancement features and / or video scaling private enhancement features in the scaling fusion features according to the preset expert modality to construct text anchored audio and video residual expert features. The text-anchored audio and video residual expert features are sequentially subjected to linear mapping, ReLU nonlinear activation, Dropout regularization, linear mapping, and tanh nonlinear activation, and then limited according to the upper limit of the residual amplitude to determine the text-anchored audio and video residual value. The absolute value of the basic emotional intensity prediction value, and the absolute deviations of the audio heterogeneous emotional intensity prediction value and the video heterogeneous emotional intensity prediction value relative to the basic emotional intensity prediction value are used to construct audio-video deviation information. After concatenating the text-anchored audio-visual residual expert features, the audio-visual deviation information, and the reliability of each source mode, linear mapping, ReLU nonlinear activation, Dropout regularization, linear mapping, and Sigmoid nonlinear activation are performed sequentially to output the audio-visual deviation gating value. Based on the text anchoring audio-visual residual value and the audio-visual deviation gate value, the basic sentiment intensity prediction value is calibrated with bounded audio-visual information to determine the audio-visual corrected basic sentiment intensity prediction value. The process of determining the fusion scaling factor includes: ； In the formula, Indicates the blending scaling factor. This indicates taking the minimum value. This indicates taking the maximum value. Indicates the reliability scaling strength. Represents the reliability fusion vector. This represents the mean of the reliability fusion vector. This indicates the lower limit of the scaling factor. This indicates the upper limit of the scaling factor. Indicates the reliability of the text source modality. Indicates the modal reliability of the audio source. Indicates the modal reliability of the video source. Indicates splicing.

8. The multimodal counterfactual sentiment recognition method based on sentiment prototype-driven approach according to claim 1, characterized in that, The process of determining the total loss includes: ； In the formula, Indicates the total loss. This indicates the predicted loss for the main task. Indicates distillation loss in the isotropic diagram. Indicates the distillation loss in the heterogeneous diagram. Indicates cyclical loss. Indicates the reconstruction loss. This represents the loss due to shared feature similarity constraints. Indicates orthogonal loss. This indicates the loss caused by audio and video counterfactual gating monitoring. This indicates the text's counterfactual consistency loss. This indicates an auxiliary loss related to the emotional archetype; The reconstruction loss includes: ； In the formula, Indicates modal index, Represents text modality, Indicates video modality, Indicates audio modality, Representing modes The unified features of the reconstruction dimension Representing modes The dimensional uniformity features Denotes the square of the L2 norm. Representing modes Shared features Representing modes Private characteristics, Indicates splicing, Representing modes Feature reconstruction decoder; Cyclic losses include: ； In the formula, Representing modes Recoded private features, Representing modes Private encoder; Orthogonal losses include: ； In the formula, This indicates taking the maximum value. Represents cosine similarity. Indicates the interval threshold; The shared feature similarity constraint loss includes: ； In the formula, Represents the set of valid anchor point feature objects. Indicates the number of valid anchor point feature objects. Indicates the anchor feature object index. Represents a set of objects with positive characteristics. Indicates the number of objects with positive features. Represents a set of objects with negative characteristics. Indicates the number of objects with negative features. Indicates the index of the positive feature object. Indicates the index of the negative feature object. Represents anchor point feature object The corresponding true emotional intensity label value, Represents a negative feature object The corresponding true emotional intensity label value, Represents anchor point feature object Shared similarity features are obtained by flattening and linear mapping of shared features. Represents positive feature objects Shared similarity features are obtained by flattening and linear mapping of shared features. Represents a negative feature object Shared similarity features are obtained by flattening and linear mapping of shared features. Represents the complete set of feature objects. Indicates the sample index. Indicates removal. Represents positive feature objects The corresponding true emotional intensity label value, Represents anchor point feature object Corresponding sample index, Represents anchor point feature object The corresponding modal index, Indicates the first The true emotional intensity labeled value of each sample; Among them, the emotional prototype auxiliary loss includes: ； In the formula, Represents discrete sentiment category-constrained loss. This indicates the loss of continuous emotional intensity constraint. Indicates the regression constraint weights; The discrete emotion category constraint loss includes: ； In the formula, Represents the natural logarithm. Representing modes The probability distribution of emotional prototypes Represents a minimal constant; The continuous emotion intensity constraint loss includes: ； In the formula, This indicates taking the minimum value. This represents the predictive value of continuous sentiment intensity. This represents the labeled value indicating the true intensity of emotion. Represents the transpose of the emotional intensity anchor vector; Among them, the isomorphic distillation loss includes: ； In the formula, This represents the weighting coefficient for distillation loss in the homogeneous graph. This indicates the weighted prediction of distillation loss. Indicates the edge weight regularization loss; The weighted predicted distillation loss includes: ； In the formula, Indicates the source modality index. Indicates the target modality index. This represents the set of directed edges across modalities in a homogeneous graph. Represents the source mode To target mode The target reliability perception homogeneous edge weight Represents the distance metric function. Represents the target mode The predicted value of homogeneous sentiment intensity Represents the source mode The predicted value of homogeneous sentiment intensity; Among them, edge weight regularization loss includes: ； In the formula, This represents the edge weight regularization weight coefficient. This represents the average target reliability perception edge weight. Represents the source mode To target mode Homogeneous fixed prior edge weight vectors Indicates batch size, Indicates the first Source mode in each sample To target mode The target reliability perception homogeneous edge weight; The heterogeneous distillation losses include: ； In the formula, This represents the weighting coefficient for distillation loss in the heterogeneous plot. Indicates heterogeneous prediction of distillation loss, Indicates the characteristic distillation loss of heterogeneous materials. Indicates the heterogeneous edge weight regularization loss; Among them, the predicted distillation losses by heterogeneity include: ； In the formula, This represents the set of directed edges across modalities in a heterogeneous graph. Represents the source mode To target mode Target heterogeneous edge weights Represents the target mode The predicted value of heterogeneous sentiment intensity Represents the source mode The predicted value of heterogeneous sentiment intensity; Among them, heterogeneous characteristic distillation losses include: ； In the formula, The characteristics of the distillation nodes in the heterogeneous graph are represented. The characteristics of distillation nodes in the heterogeneous graph are represented. Among them, heterogeneous edge weight regularization loss includes: ； In the formula, This represents the edge weight regularization weight coefficient. This represents the batch average target heterogeneous edge weight. Represents the source mode To target mode Heterogeneous fixed prior edge weight vectors Indicates the first Source mode in each sample To target mode The target heterogeneous edge weight; Among them, the losses from audio and video counterfactual gating supervision include: ； In the formula, Indicates the first The counterfactual supervision weights for each sample's audio and video recordings. Indicates the first The audio / video deviation gating value for each sample. Indicates the first Audio and video gating supervision targets for each sample. Indicates the target of audio and video gating supervision. This represents the Sigmoid activation function. This represents the contribution value of the audio and video residuals. Indicates the boundary parameters of audio and video contribution. Indicates the audio / video temperature coefficient. Indicates the main prediction error. This indicates the counterfactual prediction error. This represents the predicted value of basic sentiment intensity. This indicates taking the absolute value. This indicates the predicted base sentiment intensity value after counterfactual correction for audio and video recordings. Represents the counterfactual residual coefficient. This represents the text-anchored audio / video residual value; Among them, text counterfactual consistency loss includes: ； In the formula, Indicates the first The textual counterfactual sample weights of each sample. Indicates an indicator function, Indicates the first The absolute value of the true sentiment intensity labeled for each sample This represents the threshold for distinguishing between samples with strong emotional bias and neutral samples. Indicates the polarity consistency loss weight. Indicates the first Polarity consistency loss for each sample Indicates the weight of intensity consistency loss. Indicates the first The intensity consistency loss of each sample Indicates the weight of neutral consistency loss. Indicates the first Neutral consistency loss for each sample; Among them, the polarity consistency loss includes: ； In the formula, Indicates the polarity descent interval, Indicates the directional decrease. This represents the predicted counterfactual sentiment intensity of the text. Among them, the strength consistency loss includes: ； In the formula, Indicates the interval of intensity decrease. Indicates the degree of decrease in emotional intensity; The neutral consistency loss includes: ； In the formula, This represents smoothed L1 loss.

9. The multimodal counterfactual sentiment recognition method based on sentiment prototype-driven approach according to claim 8, characterized in that, The text counterfactual sample weights include: ； In the formula, This represents the minimum weight of the counterfactual sample in the text. This indicates the validity gate for text perturbation. Indicates the boundary mixing factor. This represents the absolute value of the text perturbation effect. Indicates the boundary of the text perturbation effect. Indicates the temperature coefficient of the text. This indicates the text perturbation effect. Indicates boundary weights, This represents the natural exponential function. This represents the boundary smoothing coefficient.

10. A multimodal counterfactual emotion recognition device based on emotion prototype-driven model, characterized in that, The apparatus is used to implement the emotion prototype-driven multimodal counterfactual emotion recognition method as described in any one of claims 1-9, the apparatus comprising: The multimodal feature extraction module is used to input training multimodal sentiment data into the multimodal sentiment recognition model to be trained, and extract dimensional unified features of each modality based on the training multimodal sentiment data; The multimodal feature decoupling coding module is used to perform private coding and shared coding on the unified features of each dimension respectively, and to determine the private features and shared features of each modality; The homogeneous graph distillation module is used to construct corresponding homogeneous graph distillation node features based on shared semantic residuals using the shared features and determine the associated homogeneous sentiment intensity prediction value. The homogeneous graph distillation module determines the target homogeneous edge weights of the cross-modal directed edges from the source modality to the target modality. The modal reliability assessment module is used to determine the cross-modal relative reliability based on the shared features, determine the prototype reliability by using the emotional intensity prototype vector and the emotional prototype probability distribution between the features of each homogeneous graph distillation node, and perform reliability weighted fusion according to modal association to determine the source modal reliability of each modality. The reliability homogeneity graph distillation module is used to perform gated weighting and source mode pruning on the target homogeneous edge weights associated with the source modes based on the reliability of each source mode. After determining the corresponding pruned homogeneous edge weights, the pruned homogeneous edge weights pointing to the same target mode are normalized and scaled or set to zero within the group according to whether they have all been pruned, and the target reliability-aware homogeneous edge weights corresponding to each pruned homogeneous edge weight are determined. The heterogeneous augmented graph distillation module is used to construct corresponding heterogeneous graph distillation node features by using the private features obtained based on cross-modal attention augmentation of each private feature, and to determine the associated heterogeneous sentiment intensity prediction value. Based on the heterogeneous graph distillation, the target heterogeneous edge weights of each heterogeneous graph cross-modal directed edge from the source modality to the target modality are determined. The audio and video gated residual prediction module is used to combine the shared features, source modality reliability, private enhancement features, audio heterogeneous sentiment intensity prediction value, and video heterogeneous sentiment intensity prediction value to perform sentiment prediction and output a basic sentiment intensity prediction value. Based on the determined text anchored audio and video residual value and audio and video deviation gate value, the module corrects and determines the audio and video corrected basic sentiment intensity prediction value. The model training optimization module is used to determine the counterfactual base sentiment intensity prediction value of the text corresponding to the counterfactual text of the training multimodal sentiment data through the multimodal sentiment recognition model to be trained, and to determine the total loss corresponding to the audio-video corrected base sentiment intensity prediction value by combining the homogeneous sentiment intensity prediction value, the target reliability-aware homogeneous edge weight, the heterogeneous graph distillation node features, the heterogeneous sentiment intensity prediction value, the target heterogeneous edge weight, the text anchoring audio-video residual value, and the audio-video deviation gate value, and iteratively optimize the model parameters until the trained multimodal sentiment recognition model is determined; The emotion recognition module is used to determine the target audio and video corrected basic emotion intensity prediction value corresponding to the multimodal emotion data to be recognized by using a trained multimodal emotion recognition model.

Images