Multimodal sentiment analysis method and device based on dynamic gating and retrieval enhancement

Abstract

The application discloses a multi-modal sentiment analysis method and device based on dynamic gating and retrieval enhancement, and relates to the technical field of multi-modal technology.The method is used to solve the problem that the existing multi-modal sentiment recognition is susceptible to noise interference, and the recognition precision and robustness are poor.The method comprises the following steps: decomposing each modality feature into modality-invariant features and modality-specific features through a multi-modal feature decomposition module; then, through a hybrid dynamic gating module, the modality-invariant features are adaptively weighted and fused at the sample level by combining the modality semantic content and the prediction confidence; finally, through a modality-specific retrieval module, key clues are retrieved from the modality-specific features based on a difference attention mechanism, and the retrieved key clues are dynamically fused with the fused features to generate the final sentiment prediction.

Description

Technical Field

[0001] This invention relates to the field of multimodal technology, and more specifically to a multimodal sentiment analysis method and apparatus based on dynamic gating and retrieval enhancement. Background Technology

[0002] Multimodal Sentiment Analysis (MSA) is a core research direction in the field of artificial intelligence emotion computing, relying on the joint inference of human emotions through three heterogeneous information types: language, audio, and vision. Currently, social media businesses such as short videos, live streaming, and online consultations are rapidly expanding, resulting in a massive increase in multimodal data combining text, voice, and images. Accurately identifying user emotions and capturing hidden emotional states has crucial practical value for scenarios such as platform content risk control, human-computer interaction experience optimization, and online service quality inspection. Unlike analysis methods that rely solely on single text, voice, or images, multimodal sentiment analysis can integrate multi-dimensional emotional cues, compensating for the shortcomings of single-modal information. In actual interactions, textual expressions can be ambiguous and ironic; audio tone and facial expressions can supplement or even correct the true emotions conveyed by the text. However, language, audio, and visual data suffer from temporal mismatches, are susceptible to noise, occlusion, and lighting interference, and are difficult to utilize collaboratively, presenting a significant technical bottleneck in the industry.

[0003] Currently, the mainstream solutions for multimodal sentiment analysis in the industry are divided into two main technical routes: fusion-oriented and representation-oriented. At the same time, two types of modality weighting strategies have emerged: static and dynamic. All of these solutions have unresolved technical shortcomings.

[0004] Fusion-oriented approaches rely on tensor, low-rank, and multi-head attention structures to model cross-modal interactions. Early tensor fusion and memory fusion methods were computationally expensive and had a large number of parameters, and were sensitive to noisy modalities. Mid-term low-rank fusion only had a single fusion space and could not separate shared semantics from modal-specific features. Recent multi-modal Transformer attention models can handle temporal misalignment, but they do not filter modal-specific noise in advance, and the fusion process is easily affected by invalid cues.

[0005] Representation-oriented methods: Before fusion, feature decoupling, alignment, or contrastive learning are performed to optimize the representation. Decoupling algorithms such as MISA (Modality-Invariant and Specific Alignment) and DMD (Disentangled Multimodal Distillation) separate public and private features, but cannot filter effective sentiment details within the private space; multi-level contrastive learning only brings similar samples closer together, without adaptively utilizing the unique complementary information of each modality for each sample.

[0006] Existing technologies related to modality weighting: Traditional solutions all use static fixed weights, which cannot adapt to the quality differences of each modality within a single sample; existing dynamic weighting only relies on semantic association or modality confidence as a single dimension for modeling, which cannot simultaneously combine semantic richness and prediction uncertainty, and the underlying fusion network has insufficient expressive power, so modality competition and private noise interference problems cannot be eradicated.

[0007] In summary, existing technologies lack a sample-level dynamic weighting mechanism driven by semantics and uncertainty, and are unable to screen effective complementary emotional cues from modality-specific features with mixed noise. These two core defects together result in low accuracy and poor robustness of emotion recognition in complex multimodal scenarios. Summary of the Invention

[0008] This invention provides a multimodal sentiment analysis method and apparatus based on dynamic gating and retrieval enhancement, which addresses the technical problems of existing technologies that use static weights or single-dimensional dynamic weighting schemes, which cannot adaptively allocate modal weights by combining semantics and prediction confidence, and are difficult to screen effective sentiment cues within specific modal features, ultimately resulting in multimodal sentiment recognition being susceptible to noise interference and having poor recognition accuracy and robustness.

[0009] This invention provides a multimodal sentiment analysis method based on dynamic gating and retrieval enhancement, including:

[0010] Acquire multimodal data including language modality, audio modality, and visual modality; perform feature decomposition on the multimodal data to obtain single-modality invariant features, single-modality specific features, and single-modality reconstruction output features;

[0011] The single-modal invariant features of each modality are used to construct context-aware features for each modality through a cross-modal attention mechanism;

[0012] The context-aware features of each modality are mapped to single-modal semantic scores. Based on the energy function and the single-modal classification vector, a single-modal uncertainty energy score is obtained. Based on the single-modal semantic scores and the single-modal uncertainty energy scores, sample-level dynamic fusion weights for each modality are obtained. The sample-level dynamic fusion weights are used to perform a weighted summation of the context-aware features of each modality to obtain the fused modality-invariant representation.

[0013] The single-modal specific features are merged according to the feature concatenation operator to obtain the single-modal specific feature retrieval storage matrix; the fused modality-invariant representation is used as the query, and the retrieval complementary features are obtained through difference-aware attention and the retrieval storage matrix; the fused modality-invariant representation, the retrieval complementary features, the multilayer perceptron, and the layer normalization operator are used to obtain the dynamic gating coefficients of the retrieval information and the final enhanced multimodal representation in sequence;

[0014] Based on the final enhanced multimodal representation and fully connected mapping network, the sentiment intensity score and sentiment category prediction results are obtained.

[0015] This invention provides a multimodal sentiment analysis device based on dynamic gating and retrieval enhancement, comprising:

[0016] The construction unit is used to acquire multimodal data including language modality, audio modality and visual modality, perform feature decomposition on the multimodal data to obtain single-modality invariant features, single-modality specific features and single-modality reconstruction output features; the single-modality invariant features of each modality are used to construct context-aware features of each modality through a cross-modal attention mechanism;

[0017] The first obtaining unit is used to map the context-aware features of each modality to a single-modality semantic score, and obtain a single-modality uncertainty energy score based on the energy function and the single-modality classification vector; obtain sample-level dynamic fusion weights for each modality based on the single-modality semantic score and the single-modality uncertainty energy score; and perform a weighted summation of the context-aware features of each modality based on the sample-level dynamic fusion weights to obtain a fused modality-invariant representation.

[0018] The second obtaining unit is used to merge the single-modality specific features according to the feature concatenation operator to obtain the single-modality specific feature retrieval storage matrix; using the fused modality-invariant representation as the query, and through difference-aware attention and the retrieval storage matrix, to obtain the retrieval complementary features; and according to the fused modality-invariant representation, the retrieval complementary features, the multilayer perceptron, and the layer normalization operator, to obtain the dynamic gating coefficients of the retrieval information and the final enhanced multimodal representation in sequence.

[0019] The prediction unit is used to obtain the emotion intensity score and emotion category prediction results based on the final enhanced multimodal representation and fully connected mapping network.

[0020] This invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform any of the above-described multimodal sentiment analysis methods based on dynamic gating and retrieval enhancement.

[0021] This invention provides a computer device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs any of the above-described multimodal sentiment analysis methods based on dynamic gating and retrieval enhancement.

[0022] This invention provides a multimodal sentiment analysis method and apparatus based on dynamic gating and retrieval enhancement. Addressing the shortcomings of existing technologies that rely solely on static weights or single-dimensional dynamic weighting, failing to adaptively allocate modal weights based on both semantic and confidence dimensions, and struggling to extract effective sentiment cues from specific modal features, resulting in models susceptible to noise, poor recognition accuracy, and low robustness, this invention offers the following advantages: During the modality fusion stage, it simultaneously calculates sample-level dynamic fusion weights based on single-modal semantic scores and single-modal uncertainty energy scores, achieving joint weighting based on both semantic richness and prediction confidence dimensions. This overcomes the limitations of traditional static weighting and single-dimensional dynamic weighting, enabling the analysis of each modality within a single sample. The system automatically adjusts the contribution ratio of real quality to weaken the influence of noisy modalities and solve the problem of unbalanced modal contributions. It constructs a modality-specific feature retrieval and storage matrix, and uses a difference-aware attention mechanism to filter complementary detail features. It mines effective emotional cues from modality-specific features mixed with noise, and uses dynamic gating to adaptively control the amount of detail integration. It makes full use of the unique emotional information of each modality and makes up for the shortcomings of existing technologies that discard private effective features and cannot mine complementary information. It adds ranking regularization constraints to enhance the matching degree between weights and modality history reliability, and further optimizes the dynamic weighting logic. It reduces the interference of noise on emotion prediction in two ways, and significantly improves the overall accuracy and robustness of multimodal emotion recognition in complex scenarios. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the multimodal sentiment analysis method based on dynamic gating and retrieval enhancement provided in an embodiment of the present invention;

[0025] Figure 2A visualization diagram of the feature distribution t-SNE (t-distributed Stochastic Neighbor Embedding) on ​​CMU-MOSI (Carnegie Mellon University Multimodal Opinion Sentiment Intensity) and CMU-MOSEI (Carnegie Mellon University Multimodal Opinion Sentiment and Emotion Intensity) provided for embodiments of the present invention;

[0026] Figure 3 This is a schematic diagram of the confusion matrix on the CMU-MOSI and CMU-MOSEI datasets provided in an embodiment of the present invention;

[0027] Figure 4 A schematic diagram illustrating the sensitivity analysis of key hyperparameters on the CMU-MOSI and CMU-MOSEI datasets provided in this embodiment of the invention;

[0028] Figure 5 This is a schematic diagram of the structure of a multimodal sentiment analysis device based on dynamic gating and retrieval enhancement provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The multimodal sentiment analysis method based on dynamic gating and retrieval enhancement provided in this invention belongs to the fields of artificial intelligence, sentiment computing, and multimodal fusion technology. Specifically, it involves multimodal sentiment recognition technology that integrates natural language processing, speech signal processing, and computer vision. It is particularly suitable for sentiment quantification and classification tasks that simultaneously include three types of heterogeneous data: text, audio, and video.

[0031] It can be widely applied to various multimodal scenarios that simultaneously involve text, voice, and images, including real-time multimodal user emotion monitoring, content sentiment tagging, and live streaming operation quality inspection on short video and live streaming platforms; user emotion recognition and service quality inspection for online video customer service, government affairs, and financial remote video consultation; capture of students' classroom emotional state and analysis of learning in online smart classrooms; quantitative evaluation of emotions in film, television, and advertising short video materials; user emotion perception in human-computer interaction scenarios such as smart cockpits and virtual digital humans; and batch analysis of user emotional trends in short video materials on social platforms, integrating multi-dimensional information from text, audio, and vision to accurately identify hidden and complex emotional tendencies.

[0032] Figure 1 This is a schematic diagram of the multimodal sentiment analysis method based on dynamic gating and retrieval enhancement provided in an embodiment of the present invention; the following uses... Figure 1 Taking an example, this paper details the multimodal sentiment analysis method based on dynamic gating and retrieval enhancement provided by embodiments of the present invention. Figure 1 As shown, the method includes the following steps:

[0033] Step 101: Obtain multimodal data including language modality, audio modality and visual modality; perform feature decomposition on the multimodal data to obtain single-modality invariant features, single-modality specific features and single-modality reconstruction output features;

[0034] Step 102: The single-modal invariant features of each modality are used to construct context-aware features of each modality through a cross-modal attention mechanism;

[0035] Step 103: Map the context-aware features of each modality to a single-modality semantic score; obtain the single-modality uncertainty energy score based on the energy function and the single-modality classification vector; obtain the sample-level dynamic fusion weights for each modality based on the single-modality semantic score and the single-modality uncertainty energy score; and perform a weighted summation of the sample-level dynamic fusion weights on the context-aware features of each modality to obtain the fused modality-invariant representation.

[0036] Step 104: Merge the single-modality specific features according to the feature concatenation operator to obtain the single-modality specific feature retrieval storage matrix; use the fused modality-invariant representation as the query, and obtain the retrieval complementary features through difference-aware attention and the retrieval storage matrix; according to the fused modality-invariant representation, the retrieval complementary features, the multilayer perceptron, and the layer normalization operator, obtain the dynamic gating coefficients of the retrieval information and the final enhanced multimodal representation in sequence;

[0037] Step 105: Based on the final enhanced multimodal representation and fully connected mapping network, obtain the sentiment intensity score and sentiment category prediction results.

[0038] The multimodal sentiment analysis method based on dynamic gating and retrieval enhancement provided in this invention (hereinafter referred to as DGR-Net (Dynamic Gating Retrieval Network), with the corresponding architecture named DGR-Net architecture) is executed by a multimodal sentiment analysis system built on the DGR-Net architecture. This system consists of a feature extraction module, a multimodal feature decomposition module (MFD), a hybrid dynamic gating module (HDG), a modality-specific retrieval module (MSR), and a prediction module. The entire method is executed sequentially by the various functional modules within the system.

[0039] This method, based on the DGR-Net architecture, constructs a multimodal sentiment analysis system with a complete joint optimization training mechanism. During the model training phase, the system synchronously calculates multiple sets of constraint losses and weights them to fuse them into an overall joint optimization loss. The loss system includes five categories of sub-losses: main task regression loss, single-modal auxiliary prediction loss, multimodal feature decomposition total loss, supervised contrastive learning loss, and ranking regularization loss. Each type of loss applies hierarchical constraints to the entire link of feature decomposition, modality dynamic weighting, retrieval enhancement representation, and sentiment prediction. Through gradient backpropagation, all internal operation parameters of the feature extraction module, multimodal feature decomposition module, hybrid dynamic gating module, modality-specific retrieval module, and prediction module are updated synchronously. The process is iterated until the model's judgment index stabilizes and no longer improves, thus completing the system training and finalization.

[0040] In step 101, multimodal data, including language modality, audio modality and visual modality, are acquired. To facilitate subsequent processing, the original data of each modality are converted into feature vectors through the feature extraction module.

[0041] Preferably, for the language modality, the BERT (Bidirectional Encoder Representations from Transformers) model is used to extract word-level semantic features; for the audio modality, the COVAREP (COVAriance Representation of speech) descriptor is used to extract acoustic features; and for the visual modality, a facial expression feature extraction tool is used to extract facial action units and expression features. Then, the features of each modality are mapped to a unified multidimensional space through a linear projection layer, which solves the problem of inconsistent feature dimensions across modalities. The mapped features are denoted as:

[0042] (1)

[0043] in, This represents the original feature matrix after modality alignment. They represent the language modality, audio modality, and visual modality, respectively. This indicates the length of the corresponding modal input sequence. This represents the feature dimension after unified mapping of all modalities.

[0044] Furthermore, through a multimodal feature decomposition module, the features of each modality are decomposed into two complementary parts: unimodal invariant features and unimodal specific features. Specifically, unimodal invariant features of each modality are extracted using a globally shared modal invariant encoder, and unimodal specific features of each modality are extracted using independent modal-specific encoders. In this embodiment of the invention, the unimodal invariant features and unimodal specific features are represented by the following formulas:

[0045] (2)

[0046] (3)

[0047] in, Represents the single-mode invariant characteristics of each mode. This indicates a globally shared mode-invariant encoder. Represents the single-modal specific characteristics of each mode. This represents a specific encoder that is independent of each mode. This represents the original feature matrix after each modality is aligned.

[0048] Furthermore, to prevent decomposition degradation and ensure that both parts retain sufficient information, this embodiment of the invention introduces a modality-specific reconstruction decoder, which extracts information from the stitched features. The reconstruction yields single-modal reconstruction output features, as shown below:

[0049] (4)

[0050] in, This represents the single-mode reconstruction output features of each mode. This indicates a modality-specific reconstruction decoder. Indicates splicing, This indicates a single-mode invariant feature. This represents the single-mode specific characteristics of each mode.

[0051] In step 102, the hybrid dynamic gating module performs sample-level adaptive fusion of unimodal invariant features. First, it constructs context-aware features for each modality through a cross-modal attention mechanism, using the unimodal invariant features of each modality as the query. ), using the concatenation of all modality-invariant features as the key (Key, ) and value (Value, After multi-head attention processing, the following modal context-aware features are obtained:

[0052] (5)

[0053] (6)

[0054] in, Representing the context-aware features of each modality, This represents the projection weight matrix for cross-modal attention queries. This represents the cross-modal attention key projection weight matrix. These three projection weight matrices represent the cross-modal attention value projection weight matrices, which are used to map the input features to the space required for attention computation. This represents the concatenation of invariant features from the language modality, audio modality, and visual modality. This represents a multi-head attention network operator. This represents language modality invariant features. This represents the audio modality invariant feature. This represents visual modality-invariant features.

[0055] In step 103, the single-modal semantic score and the single-modal uncertainty energy score are calculated through the following two paths.

[0056] Specifically, pooling and nonlinear transformation are performed on the context-aware features of each modality, and the unimodal semantic scores of each modality are generated through the Sigmoid function.

[0057] For each modality's context-aware features, a global pooling operation is performed on each modality's context-aware features to compress them into a global pooling vector of context features corresponding to that modality. This global pooling vector is then input into a modality-specific linear classification layer, utilizing the weights of that modality-specific classification layer. Bias After performing the linear mapping operation, the original Logit vector for unimodal classification of the modality is obtained, which can be determined by the following formula:

[0058] (7)

[0059] Using the scores of each class within the original Logit vector for single-modal classification as input, a temperature hyperparameter specifically for energy calculation is introduced. Substituting the energy function into the equation, the single-mode uncertainty energy fraction corresponding to the mode is obtained by solving the problem. The specific calculation formula is as follows:

[0060] (8)

[0061] in, This represents the single-mode uncertainty energy fraction for each mode. This indicates the temperature hyperparameter used for energy calculations. Indicates that each mode is in the first... Logit scores for each sentiment category This represents the total number of emotion categories. This represents the original Logit vector for unimodal sentiment classification of each modality. This represents the weights of the classification layer specific to each modality. This indicates the specific classification layer bias for each modality. This represents the global pooling vector of context features for each modality.

[0062] In this embodiment of the invention, the magnitude of the single-modal uncertainty energy score inversely represents the confidence level of the modality prediction; the lower the value of the single-modal uncertainty energy score, the sharper the distribution of the modality prediction and the higher the confidence level of the sentiment prediction; the higher the value of the single-modal uncertainty energy score, the more uniform the distribution of the modality prediction results, the stronger the prediction uncertainty and the lower the confidence level.

[0063] Furthermore, the sample-level dynamic fusion weights for each modality are obtained based on the single-modal semantic score and the single-modal uncertainty energy score:

[0064] (9)

[0065] in, This represents the sample-level dynamic fusion weights for each modality. Represents the monomodal semantic score for each modality. This represents the learnable balance coefficients based on confidence level uncertainty. This represents the single-mode uncertainty energy fraction for each mode.

[0066] Sample-level dynamic fusion weights based on each modality The context-aware features of each modality are weighted and summed to obtain the fused modality-invariant representation, as shown below:

[0067] (10)

[0068] in, This represents the modality-invariant representation after fusion. Represents the context-aware features of each modality. These represent the language modality, audio modality, and visual modality, respectively.

[0069] In addition, this module introduces a sorting regularizer. This regularizer penalizes the model based on the correlation between the historical prediction error of each modality and the currently assigned weights, forcing the model to assign higher weights to modalities with more reliable historical performance, thereby improving the robustness of dynamic weighting.

[0070] In step 104, the modality-specific retrieval module retrieves key information complementary to the fused modality-invariant representation from the single-modality-specific features to enrich the final representation. First, the single-modality-specific features corresponding to the language modality, audio modality, and visual modality are concatenated and merged using a feature concatenation operator, followed by a transpose operation, to construct the single-modality-specific feature retrieval and storage matrix shown below:

[0071] (11)

[0072] in, This represents the single-modal specific feature retrieval and storage matrix for each modality. Represents specific features of language modalities. Represents specific characteristics of an audio modality. Represents specific features of a visual modality. This indicates transpose.

[0073] Then, the fused modality-invariant representations are mapped to generate query vectors constructed from the fused modality-invariant representations. Using the query vector as the retrieval benchmark, the corresponding key vector and value vector are generated by relying on the single-modality specific feature retrieval storage matrix. The difference-aware attention scaling learnable coefficient is introduced, and the difference-aware attention operation is used to filter out the content that is differentiating and complementary to the fused modality-invariant representation. The retrieval complementary features shown below are calculated. This mechanism can adaptively match the features with the lowest similarity to supplement the unique details missing in the original representation.

[0074] (12)

[0075] in, Indicates the search for complementary features, This represents the learnable coefficient for difference-aware attention scaling. Indicates the dimension of the attention key projection. This represents the query vector constructed from the fused modality-invariant representation. This represents the independent key vector of the retrieval module. This represents the independent value vector of the retrieval module. express The transpose of .

[0076] The fused modality-invariant representation is concatenated with the retrieved complementary features. The concatenation result is input into a multilayer perceptron to perform a nonlinear transformation, and then through a sigmoid activation operation to obtain the dynamic gating coefficient of the retrieved information shown below. This coefficient is used to adaptively control the fusion injection ratio of the retrieved complementary features.

[0077] (13)

[0078] in, This represents the dynamic gating coefficient for retrieving information. This indicates the search for complementary features.

[0079] The fused modality-invariant representation is added element-wise to the retrieved complementary features weighted by the dynamic gating coefficients of the retrieved information, and then standardized using a layer normalization operator. The resulting enhanced multimodal representation is shown below:

[0080] (14)

[0081] in, This represents the dynamic gating coefficient for retrieving information. This represents the final enhanced multimodal representation. Indicates the search for complementary features, This represents the modality-invariant representation after fusion.

[0082] In step 105, the final enhanced multimodal representation is processed by a multi-layer perceptron (MLP) and linear transformation to output the sentiment intensity score and sentiment category prediction results.

[0083] The fully connected mapping network is a two-layer cascaded fully connected structure, consisting of a parallel sentiment regression branch and a sentiment classification branch, which respectively output continuous sentiment intensity scores and discrete sentiment category prediction results.

[0084] Furthermore, after outputting the sentiment intensity score and sentiment category prediction results, the system performs loss calculation and parameter update operations specific to the model training phase. All losses are only involved in the calculation during system training; no loss calculation is required in online inference scenarios. The calculation timing, formula, and constraints of each sub-loss are explained in turn as follows:

[0085] 1. Main task return loss :

[0086] This is the main supervision signal of the model in this invention. After the prediction module outputs the predicted sentiment intensity values ​​for all samples in a single batch, the prediction results are compared and calculated with the pre-labeled true sentiment tags of the samples. Specifically, the Mean Absolute Error (MAE) loss function is used:

[0087] (15)

[0088] in, Indicates the total number of samples in a single batch. Indicates the first The predicted sentiment intensity value for each sample Indicates the first Each sample corresponds to a manually labeled real sentiment tag. This indicates the main task regression loss.

[0089] In this invention, the emotion intensity prediction error is used as the basic optimization target to directly constrain the deviation between the final emotion prediction result and the true label, thereby reducing the overall prediction error and ensuring the basic accuracy of the core emotion analysis task of this invention.

[0090] 2. Single-mode auxiliary prediction loss :

[0091] The purpose of supervising the single-mode predictions for each mode in the hybrid dynamic gating module is to provide a reliable confidence estimate for the energy function. After the hybrid dynamic gating module obtains the original Logit vectors for single-mode classification of each mode, it immediately applies supervisory constraints to these vectors. Specifically:

[0092] (16)

[0093] in, This represents the predicted values ​​of unimodal sentiment regression for each modality. Represents discrete real sentiment classification labels. This represents the cross-entropy classification loss balance coefficient. This represents the cross-entropy classification loss operator. This represents the auxiliary single-mode prediction loss. Indicates a global, true, and continuous sentiment label. This represents the logit vector for unimodal classification.

[0094] In this invention, independent classification supervision is used to constrain the accuracy of sentiment classification for each modality individually, avoiding serious deviations in single-modality prediction results. Only when the prediction effect of a single modality itself is reliable can the modality uncertainty energy score obtained by solving the Logit vector truly reflect the modality confidence level, preventing distortion of the sample-level dynamic fusion weight allocation logic.

[0095] 3. Total loss of multimodal feature decomposition :

[0096] This is used to train the multimodal feature decomposition module, ensuring effective separation and information preservation of modality-invariant and modality-specific features. The multimodal feature decomposition module outputs single-modality-invariant features, single-modality-specific features, and single-modality reconstructed output features, which are then solved. The total loss consists of four sub-losses: reconstruction loss, cycle consistency loss, triplet loss, and orthogonality loss.

[0097] (17-1)

[0098] (17-2)

[0099] (17-3)

[0100] (17-4)

[0101] in, This represents the total loss of multimodal feature decomposition. Indicates the feature reconstruction loss. This represents the hyperparameter for balancing cycle consistency loss. Indicates the loss of cycle consistency. This represents the loss balance hyperparameter of the triplet. Indicates the comparison loss of the triplet. This represents the hyperparameters for balancing orthogonal constraint losses. This represents the loss between mode invariance and orthogonality to specific features. This represents the set of all triplet samples within a batch. This represents the total number of triplet samples in the set. This represents the modal invariant features of the anchor sample. This represents the cross-modal positive sample features with the same sentiment label as the anchor point. This represents the negative sample features that are of the same modality as the anchor but with different sentiment labels. This represents the hyperparameter of the triplet margin. This represents the function for calculating cosine similarity. Represents the single-mode invariant characteristics of each mode. Represents the single-modal specific characteristics of each mode. This represents the single-mode reconstruction output features of each mode. Indicates to Forward operations yield the corresponding modality-specific features. Denotes the square of the Frobenius norm. express Norm.

[0102] In this invention, the encoder and decoder are constrained to ensure that the original modality's effective sentiment information is not lost during the feature decomposition process, guaranteeing that the decomposed features can restore the original input data; the output stability of the modality-specific encoder is improved, ensuring that the modality-specific features output by the encoder remain consistent regardless of whether the input is the original features or the reconstructed features, thus suppressing training oscillations; the spatial discriminativeness of modality-invariant features is optimized, narrowing the distance between common features of the same sentiment samples and widening the distance between common features of different sentiment samples, thereby enhancing the cross-modality common sentiment semantic clustering effect; modality-invariant features and modality-specific features under the same modality are forced to be orthogonal to each other, eliminating the problems of redundancy and mutual interference between the two types of feature information, and achieving complete decoupling of common sentiment semantics and modality-private detail information; the four sub-losses work together to constitute the total loss of feature decomposition, constraining the rationality and effectiveness of the decomposition of single modality-invariant features and single modality-specific features as a whole.

[0103] 4. Ranking Regularization Loss :

[0104] The modal weights in the hybrid dynamic gating module are used to constrain the modal weights, ensuring that the weights are negatively correlated with the historical prediction reliability of the modality. After generating the dynamic fusion weights at the sample level for each modality, the ranking regularization loss is expressed by the following formula:

[0105] (18)

[0106] in, This represents the ranking regularization loss. Indicates sample The sign function of the weight difference under each modality Indicates sample The historical average regression error of the sliding window across each modality. Indicates sample Sample-level dynamic fusion weights assigned to each modality. Indicates sample The historical average regression error of the sliding window across each modality.

[0107] In this invention, regular constraints are applied to the sample-level dynamic fusion weight allocation logic to avoid the network randomly switching modality weight ratios and to prevent modalities with large errors and low reliability from being assigned excessive fusion weights. Long-term stability allows modalities with more reliable prediction performance to obtain a higher fusion ratio, suppresses fluctuations in fusion results, and improves the model's generalization ability.

[0108] 5. Supervised comparative learning loss :

[0109] To enhance the discriminativeness of the final enhanced representation, the final enhanced multimodal representation is generated and then solved, specifically as follows:

[0110] (19)

[0111] in This indicates the learning loss in supervised comparison. Indicates the total number of samples in a single batch. Representation and Sample A set of positive samples with the same sentiment label. Indicates the number of positive samples. Indicates sample The normalized embedding vector is output by the projection head. This indicates the temperature hyperparameters used for comparative learning. Indicates that the batch contains samples excluding samples All external sample indexes, This represents the cosine similarity function.

[0112] In this invention, the feature space distribution of the final enhanced features is regularized, which brings the enhanced features of samples with the same emotion category closer to each other in the feature space, and pushes the enhanced features of samples with different emotion categories further apart, thereby improving the feature discrimination ability of the prediction module, indirectly reducing the emotion prediction error, and further improving the accuracy of emotion recognition.

[0113] 6. Overall joint optimization loss :

[0114] (20)

[0115] in, This represents the overall joint optimization loss. This represents the regression loss of the main task. The balancing hyperparameters representing the auxiliary loss, The equilibrium hyperparameters representing the decomposition loss are... This represents the balancing hyperparameter of the contrast loss. The balancing hyperparameters representing the regularization loss are... This represents the auxiliary single-modal prediction loss. This represents the sum of losses from the multimodal eigenvalue decomposition modules. This indicates the learning loss in supervised comparison. This represents the ranking regularization loss.

[0116] In this embodiment of the invention, the overall joint optimization total loss is used as the global optimization objective. Gradient backpropagation is performed to synchronously update all learnable operational parameters within the feature extraction module, multimodal feature decomposition module, hybrid dynamic gating module, modality-specific retrieval module, and prediction module. All steps of this embodiment are executed in a loop and iteratively optimized until the accuracy of the sentiment judgment result of the verification set no longer shows a significant improvement, or the preset maximum number of iterations is reached. At this point, the training process of the entire multimodal sentiment analysis system ends, the system parameters are finalized, and it can be put into actual inference.

[0117] To provide a more detailed description of the present invention, the method is illustrated below with specific embodiments. In this embodiment, a multimodal sentiment analysis method based on dynamic gating and retrieval enhancement is deployed in a real-time user sentiment monitoring system (DGR-Net multimodal sentiment analysis system) for a short video platform's live streaming room. The system hardware consists of a cloud server and a live streaming real-time acquisition device. The software includes the corresponding program for this method. The entire method is executed collaboratively by a feature extraction module, a multimodal feature decomposition module, a hybrid dynamic gating module, a modality-specific retrieval module, and a prediction module. It performs real-time sentiment quantification and negative sentiment warning for three types of multimodal data synchronously collected from the live streaming room: text bullet comments, anchor audio, and anchor facial images. The complete execution process includes scene input and engineering configuration parameters, inference deployment process, and offline model training process.

[0118] I. Scene Input and Project Configuration Parameters

[0119] This embodiment is designed for general short video live streaming scenarios. The input consists of synchronously collected language, audio, and visual three-modal data, and the numerical parameters for unified and fixed engineering implementation are: the total number of sentiment classification categories. The classes (highly negative, mildly negative, neutral, mildly positive, highly positive) have a unified feature dimension of 256, and the attention key-value dimension is [not specified]. Training batch size The maximum number of iterations is 100 rounds; the core hyperparameters have fixed values: energy and temperature hyperparameters. Uncertainty balance coefficient Difference-perceived scaling factor The loss balancing hyperparameter is uniformly set as follows: , , , All parameters are fixed values ​​that are optimal for live streaming scenarios, ensuring stable system convergence and direct deployment.

[0120] II. Complete Implementation Process of Online Real-Time Reasoning

[0121] Step 201: The live broadcast room collects three types of raw multimodal data in real time: the language modality is the scrolling text in the live broadcast room, the audio modality is the real-time speech waveform of the host, and the visual modality is the continuous video frames of the host's face captured by the live broadcast room camera. The feature extraction module extracts basic features from the three types of raw data respectively: BERT word vector features for text, COVAREP acoustic features for audio, and facial expression feature extraction tools for visual features to extract facial action unit features. Then, all feature dimensions are unified by linear projection to obtain the aligned raw feature matrix.

[0122] Step 202: The original features of the three modalities after unification of dimensions are sent to the multimodal feature decomposition module. The module extracts cross-modal general single-modal invariant features through a globally shared modal invariant encoder, and extracts unique single-modal specific features that belong only to a single modality through a modal independent specific encoder. Then, the two types of features of the same modality are concatenated and input into the modality-specific reconstruction decoder to generate single-modal reconstruction output features, thereby achieving feature decoupling and information fidelity, as shown in the above formulas (2), (3) and (4).

[0123] in, It is a 256-dimensional unimodal invariant feature. For 256-dimensional single-modal specific features, The output features of the single-modal reconstruction after dimension restoration are strictly unified to 256 dimensions for the three types of features, achieving complete decoupling between public sentiment semantics and modality-private details, and meeting the requirements for regularized input of live multimodal features.

[0124] Step 203: All single-modal invariant features are concatenated to obtain a global invariant feature matrix, which is then fed into the multi-head attention operator built into the hybrid dynamic gating module. Single-modal invariant features are used as queries and global concatenated invariant features are used as keys and values ​​to generate context-aware features for each modality, corresponding to the above formulas (5) and (6).

[0125] Step 204: Calculation of modal uncertainty and dynamic fusion weights. Global pooling is performed on the 256-dimensional context-aware features to obtain a pooled feature vector of dimension 256. Input modality-specific linear classification layer, classification layer weights Bias Outputs a single-modal raw sentiment score vector containing 5 sentiment categories, combined with a fixed-temperature hyperparameter. The energy function is used to calculate the modal uncertainty energy fraction, corresponding to the above formulas (7) and (8).

[0126] Combined with normalized single-modal semantic scores With energy fraction and constant uncertainty equilibrium coefficient Adaptively solve the dynamic fusion weights at the sample level for each modality. After Softmax normalization, the weights are between 0 and 1. Automatically suppress noise modalities and enhance high-confidence modalities, effectively adapting to the modal noise interference problem in live streaming scenarios. Corresponding to the above formula (9), the globally optimal modality-invariant fusion representation is obtained by weighted fusion of sample-level dynamic fusion weights, corresponding to the above formula (10).

[0127] Step 205: Collect all single-modal specific features corresponding to language, audio, and vision. Generate a single-modal specific feature retrieval and storage matrix by transposing the matrix using Concat (Concatenate). Construct a query vector using fusion-invariant representation. Generate key-value vectors based on the retrieval matrix. Introduce a difference-aware coefficient to perform difference-aware attention operations and output the retrieval complementary features, corresponding to formulas (11) and (12) above. The numerical values ​​substituted into the above formulas are fixed. , By matching the attention key dimension, modal detail features that complement global features are accurately selected.

[0128] The fusion-invariant features and the retrieval complementary features are further concatenated and fed into a multilayer perceptron and sigmoid activation to obtain dynamic gating coefficients for the retrieval information, which adaptively control the integration ratio of detail features, corresponding to the above formula (13). The gated weighted retrieval complementary features are superimposed on the fusion representation and processed by the layer normalization operator to generate the final enhanced multimodal representation, corresponding to the above formula (14).

[0129] Step 206 involves inputting the 256-dimensional final enhanced multimodal representation into a two-layer cascaded fully connected mapping network. The first layer maps the dimensions to 128, and the second layer outputs continuous emotional intensity scores (range [0,1]) and 5-dimensional discrete emotional category probabilities. The live streaming system sets a threshold of 0.4. If the emotional intensity score is lower than the negative threshold and the classification result is determined to be an angry / irritable / resistant negative category, the platform backend automatically triggers a content emotional risk warning, pushing a negative emotional alert to the operations staff, thus completing the real-time emotional monitoring business output.

[0130] III. Optimization of Offline Training Process

[0131] To ensure online recognition accuracy, a training dataset of 100,000 historical live stream comments, broadcaster voice, and facial video samples was collected in advance. After performing full-link feature calculations, a complete set of constrained losses was calculated simultaneously based on the predicted output, manually labeled real sentiment tags, and intermediate features from each module. The weighted average was then used to obtain the total loss. Used for iteratively updating the operational parameters of all modules in the system.

[0132] The main task regression loss is used to constrain the accuracy of sentiment prediction, corresponding to the above formula (15), where the batch value is fixed. By comparing the predicted scores with the average error of the actual labels in a single batch of 32 live streaming samples, the accuracy of sentiment prediction is precisely constrained.

[0133] Combining single-modality assisted prediction loss, feature decomposition loss, ranking regularization loss, and supervised contrast loss, an overall joint optimization loss is constructed, corresponding to the above formula (20), where the value of the balancing hyperparameter is fixed: , , , Reasonably allocate the optimization weights of each component loss to ensure multi-constraint collaborative optimization and stable model convergence.

[0134] The training process updates all module parameters in reverse with the total loss as the optimization objective. The maximum number of iterations is 100. When the emotion recognition accuracy on the validation set does not improve for 5 consecutive iterations, or when the maximum number of iterations of 100 is reached, training is stopped immediately and the system is finalized. The finalized model can be directly used for real-time user emotion status monitoring in live streaming rooms. The parameters are fixed and the process can be completely reproduced.

[0135] This example demonstrates precise computation through computer programs and can be deployed practically in short video and live streaming user sentiment monitoring scenarios. Compared to traditional single-modal sentiment recognition methods, this example, relying on feature decoupling, dynamic uncertainty fusion, enhanced retrieval details, and joint constraints of multiple losses, effectively addresses industry pain points such as inconsistencies in sentiment between text, audio, and video in live streaming scenarios, high environmental noise, and difficulty in recognizing implicit emotions. It boasts higher recognition accuracy, stronger robustness, and strong engineering applicability and commercial application value.

[0136] The contribution ratios of language, audio, and visual modalities differ significantly across samples. In some samples, the sentiment tendency is dominated by the language modality, while in others it is dominated by the visual modality. This underscores the necessity of implementing a sample-level dynamic modality weighting mechanism. Simultaneously, single-modality specific features contain both redundant noise and detailed clues with sentiment representation value, requiring targeted mechanisms to extract effective complementary information. Based on the above analysis, this invention proposes the DGR-Net overall method architecture. On one hand, it utilizes a hybrid dynamic gating module to achieve sample-level adaptive modality weight allocation; on the other hand, it relies on a modality-specific retrieval module to selectively mine and utilize effective sentiment information from single-modality specific features.

[0137] The DGR-Net architecture employs a step-by-step serial computation process, comprising four core computational modules: a multimodal feature decomposition module, which decomposes the input features of each modality to obtain unimodal invariant features and unimodal specific features; a hybrid dynamic gating module, which performs cross-modal adaptive fusion processing on unimodal invariant features, adaptively solving the sample-level dynamic fusion weights corresponding to each modality by combining semantic interaction results and modality confidence evaluation results; a modality-specific retrieval module, which retrieves complementary feature information from all unimodal specific features using a difference-aware attention mechanism, and completes dynamic feature fusion through learnable retrieval information dynamic gating coefficients; and a prediction module, which outputs sentiment intensity scores and sentiment category prediction results based on the final enhanced multimodal representation. Through the collaborative computation of the multimodal feature decomposition module, the hybrid dynamic gating module, and the modality-specific retrieval module, DGR-Net can generate more robust and superior multimodal feature representations, effectively improving the prediction accuracy and model generalization performance of multimodal sentiment analysis tasks.

[0138] Figure 2This is a visualization diagram of the feature distribution t-SNE on the CMU-MOSI and CMU-MOSEI datasets provided in this embodiment of the invention. In the figure, HN (Highly Negative); N (Mild Negative); WN (Weakly Negative); NT (Neutral); WP (Mild Positive); P (Positive); HP (Highly Positive). The original multimodal features input to the model exhibit entangled sample distributions and severe overlap in clusters of different sentiment categories. This phenomenon is caused by modal heterogeneity and redundant noise within the features. After the final enhanced multimodal representation is obtained through the complete DGR-Net process of this invention, the feature clustering degree of samples of the same sentiment category is significantly improved, and the distinguishability between samples of different sentiment categories is significantly improved. The above results demonstrate that the multimodal feature decomposition module, hybrid dynamic gating module, and modality-specific retrieval module in this invention work together to alleviate cross-modal information conflicts, suppress interference information unrelated to sentiment prediction, and construct a feature latent space with a more compact cluster structure and more regular semantic arrangement. In particular, the effect of bringing similar sentiment samples closer together and widening the distance between neighboring sentiment category samples verifies the technical effectiveness of this invention in extracting single-modality invariant features while effectively introducing complementary sentiment cues within single-modality-specific features. This conclusion is also consistent with the performance improvement results of the quantitative indicators in the table.

[0139] Figure 3 This diagram illustrates the confusion matrix on the CMU-MOSI and CMU-MOSEI datasets provided in this embodiment of the invention. Specifically, the highest percentage of values ​​is found on the diagonal of the matrix, indicating a high degree of matching between the model's predicted sentiment category and the actual sentiment labels of the samples. This verifies that the final enhanced multimodal representation generated by this invention possesses excellent sentiment polarity and intensity discrimination capabilities. Most of the misclassified samples off-diagonal in the confusion matrix only appear between adjacent sentiment categories; the model rarely makes serious errors such as completely reversing positive and negative sentiment polarities. This phenomenon aligns with the objective characteristics of semantic ambiguity and difficult-to-define boundaries inherent in adjacent sentiment intensities. The model's recognition performance for extreme sentiment categories slightly decreases. This is due to the smaller number of training samples corresponding to extreme sentiments and the greater subjective bias inherent in the manual annotation of extreme sentiments.

[0140] Figure 4This diagram illustrates the sensitivity analysis of key hyperparameters on the CMU-MOSI and CMU-MOSEI datasets provided in this embodiment of the invention. After adjusting the balancing hyperparameter values ​​corresponding to various losses within a relatively large range, it can be seen that the overall emotion recognition performance of the method of this invention exhibits relatively small fluctuations and a generally stable performance. This demonstrates that the overall framework, composed of the multimodal feature decomposition module, the hybrid dynamic gating module, the modality-specific retrieval module, and multi-class joint loss constraints, does not overly rely on fine-tuning of hyperparameters. The performance stability of this invention is particularly outstanding on the CMU-MOSEI dataset because the dataset has a larger sample size, allowing for better mitigation of the impact of random noise during model iteration and optimization. In summary, this parameter sensitivity experiment verifies the operational stability of this invention in engineering applications. Good generalization results can be achieved using only hyperparameter configurations within a conventional range, without requiring significant computational resources for fine-tuning hyperparameter grid traversal optimization.

[0141] Table 1 shows the partitioning information for the CMU-MOSI and CMU-MOSEI datasets. The CMU-MOSI dataset contains 2199 single-sentence segments taken from movie review videos. The sentiment scores range from -3 to +3, with negative values ​​corresponding to negative sentiment and positive values ​​to positive sentiment. The absolute value of the score represents the intensity of the sentiment. The CMU-MOSEI dataset is a large-scale expansion of the CMU-MOSI dataset, containing approximately 22856 YouTube movie review segments, about ten times the size of the CMU-MOSI dataset. It covers more types of speakers, discussion topics, and recording environments. This dataset also uses the [-3, +3] continuous sentiment scoring rule. The partitioning of the training, validation, and test sets for both datasets follows the common partitioning methods used in existing mainstream research to ensure fairness in comparisons between different algorithms.

[0142] Table 1: Partitioning information between CMU-MOSI and CMU-MOSEI datasets

[0143]

[0144] Table 2 shows the evaluation results of the comparative methods and the proposed method (DGR-Net) on the CMU-MOSI dataset. The proposed method achieves the best binary classification performance, with an Acc-2 accuracy of 86.89% and an F1 score of 86.86%, both significantly better than all the comparative algorithms. The seven-class classification accuracy (Acc-7) reaches 47.67%, also ranking first among all methods. The five-class classification accuracy (Acc-5) is 53.64%, only slightly lower than the 54.33% of the MITR-DNet algorithm, demonstrating a strong competitive advantage. In the continuous sentiment regression evaluation dimension, the proposed method achieves a Pearson correlation coefficient (Corr) of 0.803, the highest among all evaluated methods; and a mean absolute error (MAE) as low as 0.699, the lowest among all evaluated methods. The experimental results above demonstrate that the final enhanced multimodal representation obtained through the feature decomposition, dynamic modal weighting, and retrieval enhancement process of this invention can not only complete the determination of discrete sentiment categories, but also achieve the fitting and prediction of continuous sentiment intensity scores, and can play a stable and effective role under various evaluation rules.

[0145] Table 2: Evaluation results of the comparative methods and the method of the present invention (DGR-Net) on the CMU-MOSI dataset.

[0146]

[0147] Table 3 shows the evaluation results of the various comparison methods and the present invention on the CMU-MOSEI dataset. The present invention also demonstrates strong overall competitiveness. Its binary classification accuracy (Accc-2) is 86.34%, and its F1 score is 86.37%, both outperforming all comparison algorithms. Its five-class classification accuracy (Accc-5) is 56.02%, and its seven-class classification accuracy (Accc-7) is 54.00%, both the best among all participating methods. The Pearson correlation coefficient (Corr) is 0.772, essentially on par with the optimal algorithm SAFCN (Self-enhancing Attention Fusion Contrastive Network) at 0.773, and its mean absolute error (MAE) is 0.533, indicating excellent overall performance. These results demonstrate that the present invention exhibits superior model generalization performance when dealing with multimodal input data characterized by complex speakers, diverse topics, and stronger noise interference.

[0148] Among them are TFN (Tensor Fusion Network), LMF (Low-rank Multimodal Fusion), MFN (Memory Fusion Network), MulT (Multimodal Transformer), DLF (Dynamic Language Fusion), MITR-Dnet (Modality-Invariant Bidirectional Temporal Representation Distillation Network), MPC (Multi-Pair Contrastive Learning), SFTTR (Sequential Fusion of Text-close and Text-far Representations), and CMHFM (Cross Modal Hierarchical Fusion Multimodal).

[0149] Table 3: Evaluation results of various comparison methods and the present invention on the CMU-MOSEI dataset.

[0150]

[0151] All experimental results from the two datasets above demonstrate that, compared to traditional tensor fusion methods such as TFN and LMF, this invention shows significant improvements in all evaluation metrics on the CMU-MOSI and CMU-MOSEI datasets. Simply improving the expressive power of the fusion layer cannot solve the core challenge of multimodal emotion recognition. The core improvement idea is to learn structured multimodal features and adaptively adjust the contribution of various modalities to the prediction results through sample-level dynamic fusion weights.

[0152] Compared to existing methods such as MFN and MulT that only model temporal series and cross-modal interactions, the present invention has significantly improved performance. This is because the present invention fully realizes the explicit decoupling of single-modal invariant features and single-modal specific features, and at the same time, with the sample-level adaptive weighting mechanism, it can generate more reliable multimodal representations.

[0153] Compared to existing frameworks such as MISA, DMD, DLF, and SAFCN that only perform feature decoupling, this invention integrates modality-specific retrieval modules for collaborative computation, further optimizing the model's recognition performance. It verifies that selectively retrieving complementary sentiment cues from single-modality-specific features can effectively supplement the fused modality-invariant representation.

[0154] Table 4: Hyperparameter settings of this invention on CMU-MOSI and CMU-MOSEI datasets

[0155]

[0156] Table 4 shows the hyperparameter settings of this invention on the CMU-MOSI and CMU-MOSEI datasets. This invention uses the AdamW (Adam with Weight Decay) optimizer to complete parameter iterative updates; an early stopping mechanism is set during training, terminating training when the validation set metric shows no improvement for 10 consecutive rounds, and the maximum number of training iterations is set to 60. In the feature extraction stage, the BERT-base-uncased model is used to extract basic features for the language modality, facial expression feature extraction tools are used for the visual modality, and acoustic features are extracted using the COVAREP descriptor for the audio modality. Model training and inference deployment are completed on a Linux system and an NVIDIA A6000 graphics card.

[0157] Table 5: Ablation experimental results of the method of the present invention on the CMU-MOSI dataset.

[0158]

[0159] Table 5 presents the ablation experiment results of the method of this invention on the CMU-MOSI dataset. The first part compares the ablation results of single-modal and multi-modal combinations: the baseline model using only the language modality already achieved good recognition results, with a binary classification accuracy of 83.99%, indicating that the emotional information in this dataset mainly relies on the language modality; when using the audio and visual modalities alone, the binary classification accuracy is only about 54.42%, indicating that the discriminative ability of the two modalities is weak when used alone. However, after adding the audio and visual modalities to the language modality, the model performance steadily improved, with the Acc-2 of language + audio and language + video combinations both reaching 84.15%; the complete method of this invention achieves the best performance after fusing the three modalities, with an Acc-2 of 86.89%. These results show that although the audio and visual modalities carry a lot of noise, the language modality, which includes prosody and facial micro-expressions, does not have unique emotional cues. Effectively mining this kind of complementary information and simultaneously suppressing modal noise are the keys to improving the accuracy of emotional intensity scores and emotional category prediction.

[0160] The second part is the ablation verification of the core modules: After removing the multimodal feature decomposition module, the model's Acc-2 dropped to 83.84% and the mean absolute error (MA) was 0.722, proving that decoupling single-modal invariant features from single-modal specific features can reduce cross-modal information conflict and filter redundant noise; after removing the hybrid dynamic gating module, the Acc-2 was 83.84% and the MAE was 0.734, which is sufficient to show that the sample-level dynamic fusion weights can alleviate the imbalance of modal contributions between samples and avoid low-confidence noise modal interference with the overall fusion representation; after removing the modality-specific retrieval module, the model performance dropped the most, with an Acc-2 of only 83.54 and an MAE of 0.711, proving that the retrieval complementary features obtained based on difference-aware attention retrieval can provide incremental effective information beyond the decoupling and dynamic weighting links.

[0161] Based on the comprehensive ablation experiment results, it can be concluded that the multimodal feature decomposition module, the hybrid dynamic gating module, and the modality-specific retrieval module work together to achieve the superior multimodal emotion recognition effect of this invention.

[0162] Based on the same inventive concept, this invention provides a multimodal sentiment analysis device based on dynamic gating and retrieval enhancement. Since the principle of this device in solving the technical problem is similar to that of the multimodal sentiment analysis method based on dynamic gating and retrieval enhancement, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be described again.

[0163] like Figure 5 As shown, the device includes a construction unit 501, a first obtaining unit 502, a second obtaining unit 503, and a prediction unit 504.

[0164] The construction unit 501 is used to acquire multimodal data including language modality, audio modality and visual modality, perform feature decomposition on the multimodal data to obtain single-modality invariant features, single-modality specific features and single-modality reconstruction output features; the single-modality invariant features of each modality are used to construct context-aware features of each modality through a cross-modal attention mechanism.

[0165] The first obtaining unit 502 is used to map the context-aware features of each modality to a single-modality semantic score, and obtain a single-modality uncertainty energy score based on the energy function and the single-modality classification vector; obtain sample-level dynamic fusion weights for each modality based on the single-modality semantic score and the single-modality uncertainty energy score; and perform a weighted summation of the context-aware features of each modality based on the sample-level dynamic fusion weights to obtain a fused modality-invariant representation.

[0166] The second obtaining unit 503 is used to merge the single-modality specific features according to the feature splicing operator to obtain the single-modality specific feature retrieval storage matrix; using the fused modality-invariant representation as the query, and through difference-aware attention and the retrieval storage matrix, to obtain retrieval complementary features; and according to the fused modality-invariant representation, the retrieval complementary features, the multilayer perceptron, and the layer normalization operator, to sequentially obtain the dynamic gating coefficients of the retrieval information and the final enhanced multimodal representation;

[0167] The prediction unit 504 is used to obtain the emotion intensity score and emotion category prediction result based on the final enhanced multimodal representation and fully connected mapping network.

[0168] It should be understood that the units included in the above-described multimodal sentiment analysis device based on dynamic gating and retrieval enhancement are merely a logical division based on the functions implemented by the device. In practical applications, the above units can be superimposed or split. Furthermore, the functions implemented by the multimodal sentiment analysis device based on dynamic gating and retrieval enhancement provided in this embodiment correspond one-to-one with the multimodal sentiment analysis method based on dynamic gating and retrieval enhancement provided in the above embodiments. The more detailed processing flow implemented by this device has been described in detail in the first embodiment of the method above, and will not be described in detail here.

[0169] Another embodiment of the present invention provides a computer device, the computer device including: a processor and a scene database; the scene database is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device executes the various steps of the multimodal sentiment analysis method based on dynamic gating and retrieval enhancement shown in the above method embodiment.

[0170] Another embodiment of the present invention provides a computer-readable storage medium storing computer instructions that, when executed on a computer device, cause the computer device to perform the various steps of the multimodal sentiment analysis method based on dynamic gating and retrieval enhancement as described in the above-described method embodiments.

[0171] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multimodal sentiment analysis method based on dynamic gating and retrieval enhancement, characterized in that, include: Acquire multimodal data including language modality, audio modality, and visual modality; perform feature decomposition on the multimodal data to obtain single-modality invariant features, single-modality specific features, and single-modality reconstruction output features; The single-modal invariant features of each modality are used to construct context-aware features for each modality through a cross-modal attention mechanism; The context-aware features of each modality are mapped to single-modal semantic scores. Based on the energy function and the single-modal classification vector, a single-modal uncertainty energy score is obtained. Based on the single-modal semantic scores and the single-modal uncertainty energy scores, sample-level dynamic fusion weights for each modality are obtained. The sample-level dynamic fusion weights are used to perform a weighted summation of the context-aware features of each modality to obtain the fused modality-invariant representation. The single-modal specific features are merged according to the feature concatenation operator to obtain the single-modal specific feature retrieval storage matrix; the fused modality-invariant representation is used as the query, and the retrieval complementary features are obtained through difference-aware attention and the retrieval storage matrix; the fused modality-invariant representation, the retrieval complementary features, the multilayer perceptron, and the layer normalization operator are used to obtain the dynamic gating coefficients of the retrieval information and the final enhanced multimodal representation in sequence; Based on the final enhanced multimodal representation and fully connected mapping network, the sentiment intensity score and sentiment category prediction results are obtained.

2. The method as described in claim 1, characterized in that, The feature decomposition of the multimodal data to obtain single-modality invariant features, single-modality specific features, and single-modality reconstruction output features specifically includes: The language modality, audio modality, and visual modality included in the multimodal data are mapped to a unified dimension; The unimodal invariant features of each modality are extracted by a globally shared modal invariant encoder, and the unimodal specific features of each modality are extracted by a modal-specific encoder. The unimodal invariant features and unimodal specific features of each modality are concatenated by a modality-specific reconstruction decoder to obtain the unimodal reconstruction output features of each modality. The single-mode invariant feature, the single-mode specific feature, and the single-mode reconstructed output feature are respectively represented by the following formulas: in, Represents the single-mode invariant characteristics of each mode. This indicates a globally shared mode-invariant encoder. Represents the single-modal specific characteristics of each mode. This represents a specific encoder that is independent of each mode. This represents the original feature matrix after modality alignment. This represents the single-mode reconstruction output features of each mode. This indicates a modality-specific reconstruction decoder. These represent the language modality, audio modality, and visual modality, respectively. Indicates splicing.

3. The method as described in claim 1, characterized in that, The unimodal invariant features of each modality are used to construct modal context-aware features through a cross-modal attention mechanism, specifically including: Using the unimodal invariant features of each modality as queries, and the concatenation of all modal invariant features as keys and values, the following modal context-aware features are obtained through a cross-modal attention mechanism: in, Representing the context-aware features of each modality, This represents the projection weight matrix for cross-modal attention queries. This represents the cross-modal attention key projection weight matrix. This represents the projection weight matrix of cross-modal attention values. This represents the concatenation of invariant features from the language modality, audio modality, and visual modality. This represents a multi-head attention network operator. This represents language modality invariant features. This represents the audio modality invariant feature. This represents visual modality-invariant features.

4. The method as described in claim 1, characterized in that, The process of mapping each modal context-aware feature to a single-modal semantic score, and obtaining the single-modal uncertainty energy score based on the energy function and the single-modal classification vector, specifically includes: Pooling and nonlinear transformation are performed on the context-aware features of each modality, and the unimodal semantic score of each modality is generated by the Sigmoid function; Global pooling is performed on the context-aware features of each modality to obtain a global pooling vector of context features; the global pooling vector of context features is input into the modality-specific linear classification layer to obtain a single-modality classification Logit vector containing the original prediction scores of all sentiment categories and a single-modality uncertainty energy score; The single-mode uncertainty energy fraction is determined by the following formula: in, This represents the single-mode uncertainty energy fraction for each mode. This indicates the temperature hyperparameter used for energy calculations. Indicates that each mode is in the first... Logit scores for each sentiment category This represents the total number of emotion categories. This represents the original Logit vector for unimodal sentiment classification of each modality. This represents the weights of the classification layer specific to each modality. This indicates the specific classification layer bias for each modality. This represents the global pooling vector of context features for each modality.

5. The method as described in claim 1, characterized in that, The sample-level dynamic fusion weights for each modality are determined by the following formula: The modal invariance after fusion is determined by the following formula: in, This represents the sample-level dynamic fusion weights for each modality. Represents the monomodal semantic score for each modality. This represents the learnable balance coefficients based on confidence level uncertainty. This represents the single-mode uncertainty energy fraction for each mode. This represents the modality-invariant representation after fusion. Represents the context-aware features of each modality. These represent the language modality, audio modality, and visual modality, respectively.

6. The method as described in claim 1, characterized in that, The single-modal specific feature retrieval and storage matrix is ​​determined by the following formula: The complementary features for retrieval are determined by the following formula: The dynamic gating coefficient for the retrieved information is determined by the following formula: The final enhanced multimodal representation is determined by the following formula: in, This represents the single-modal specific feature retrieval and storage matrix for each modality. Indicates the search for complementary features, This represents the query vector constructed from the fused modality-invariant representation. This represents the independent key vector of the retrieval module. This represents the independent value vector of the retrieval module. This represents the learnable coefficient for difference-aware attention scaling. Indicates the dimension of the attention key projection. This represents the dynamic gating coefficient for retrieving information. This represents the final enhanced multimodal representation. Represents specific features of language modalities. Represents specific characteristics of an audio modality. Represents specific features of a visual modality. This represents the modality-invariant representation after fusion. Indicates transpose. express The transpose of .

7. The method as described in claim 1, characterized in that, After obtaining the emotion intensity score and emotion category prediction results, the process also includes: The regression loss for the main task is determined based on the sentiment category prediction results and the true labels. The overall joint optimization loss is determined using the following formula: in, This represents the regression loss of the main task. Indicates the total number of samples in a single batch. Indicates the first The predicted sentiment intensity value for each sample Indicates the first The true sentiment labels of each sample This represents the overall joint optimization loss. The balancing hyperparameters representing the auxiliary loss, The equilibrium hyperparameters representing the decomposition loss are... This represents the balancing hyperparameter of the contrast loss. The balancing hyperparameters representing the regularization loss are... This represents the auxiliary single-modal prediction loss. This represents the sum of losses from the multimodal eigenvalue decomposition modules. This indicates the learning loss in supervised comparison. This represents the ranking regularization loss.

8. A multimodal sentiment analysis device based on dynamic gating and retrieval enhancement, characterized in that, include: The construction unit is used to acquire multimodal data including language modality, audio modality and visual modality, and to perform feature decomposition on the multimodal data to obtain single-modality invariant features, single-modality specific features and single-modality reconstruction output features; The single-modal invariant features of each modality are used to construct context-aware features for each modality through a cross-modal attention mechanism; The first obtaining unit is used to map the context-aware features of each modality to a single-modality semantic score, and obtain a single-modality uncertainty energy score based on the energy function and the single-modality classification vector; obtain sample-level dynamic fusion weights for each modality based on the single-modality semantic score and the single-modality uncertainty energy score; and perform a weighted summation of the context-aware features of each modality based on the sample-level dynamic fusion weights to obtain a fused modality-invariant representation. The second obtaining unit is used to merge the single-modality specific features according to the feature concatenation operator to obtain the single-modality specific feature retrieval storage matrix; using the fused modality-invariant representation as the query, and through difference-aware attention and the retrieval storage matrix, to obtain the retrieval complementary features; and according to the fused modality-invariant representation, the retrieval complementary features, the multilayer perceptron, and the layer normalization operator, to obtain the dynamic gating coefficients of the retrieval information and the final enhanced multimodal representation in sequence. The prediction unit is used to obtain the emotion intensity score and emotion category prediction results based on the final enhanced multimodal representation and fully connected mapping network.

9. A computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, causes the processor to perform the multimodal sentiment analysis method based on dynamic gating and retrieval enhancement as described in any one of claims 1-7.

10. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the multimodal sentiment analysis method based on dynamic gating and retrieval enhancement as described in any one of claims 1-7.

Images