A multimodal sentiment fusion analysis method and system

Abstract

The application discloses a kind of multimodal sentiment fusion analysis method and system, comprising: by feature extraction module, multimodal sentiment data is modally extracted features, generates text original feature, audio original feature and visual original feature;Text original feature, audio original feature and visual original feature are interactively fused based on uniform semantic alignment module, and collaborative fusion feature is constructed;Dynamic fusion regulation module is used according to text original feature, audio original feature, visual original feature and collaborative fusion feature, and the double-layer dynamic fusion optimization of mode and channel is carried out, and unified fusion feature is determined;According to high-order semantic abstraction module, hierarchical residual semantic gate enhancement is carried out based on unified fusion feature, and semantic enhancement feature is generated;Semantics enhancement feature is input into sentiment prediction module, and the output emotion analysis result is obtained. Based on the above scheme, it is helpful to provide more stable and accurate and reliable sentiment recognition result.

Description

Technical Field

[0001] This invention relates to the field of sentiment analysis technology, and in particular to a multimodal sentiment fusion analysis method and system. Background Technology

[0002] When multimodal sentiment analysis is applied to educational scenarios, it can help teachers gain a more comprehensive understanding of students' emotional states by combining students' verbal and nonverbal modalities. Verbal modalities (such as questions, answers, and written assignments) can convey emotional information through students' written expressions and verbal responses, while nonverbal modalities (such as facial expressions, body language, and audio tone) can reflect students' true emotions and psychological states through visual and audio signals.

[0003] Traditional multimodal sentiment analysis methods mainly involve directly concatenating or statically weighting the features of each modality and then mapping them to output sentiment classification results. However, in the special context of the classroom, information from the verbal modality mainly comes from homework, while information from the non-verbal modality mainly comes from real-time classroom interactions. This makes the synchronous extraction of information between modalities more complex. At the same time, students' emotional state in the classroom is often affected by various factors such as classroom content, teacher questions, and peer interactions. Therefore, there are problems such as changes in the contribution weights between modalities and insufficient adaptability of information flow, making it difficult to cope with the complexity and diversity of emotional fluctuations, resulting in low accuracy of sentiment recognition. Summary of the Invention

[0004] This invention provides a multimodal sentiment fusion analysis method and system, which solves the technical problem that traditional multimodal sentiment analysis methods mainly involve directly splicing or statically weighting the features of each modality and then mapping the output sentiment classification results. This method is prone to problems such as changes in the contribution weights between modalities and insufficient adaptability of information flow, making it difficult to cope with the complexity and diversity of sentiment fluctuations, resulting in low accuracy of sentiment recognition.

[0005] The first aspect of this invention provides a multimodal emotion fusion analysis method, comprising:

[0006] The feature extraction module extracts features from the multimodal sentiment data modally, generating original text features, original audio features, and original visual features.

[0007] Based on the unified semantic alignment module, cross-modal alignment and interactive fusion of the original text features, the original audio features, and the original visual features are performed to construct collaborative fusion features;

[0008] A dynamic fusion control module is used to perform two-layer dynamic fusion optimization of modality and channel based on the original text features, the original audio features, the original visual features, and the collaborative fusion features to determine a unified fusion feature;

[0009] Based on the unified fusion features, the higher-order semantic abstraction module performs hierarchical residual semantic gating enhancement to generate semantic enhancement features.

[0010] The semantic enhancement features are input into the sentiment prediction module, and the sentiment analysis results are output.

[0011] Optionally, the feature extraction module includes a text feature extraction submodule, an audio feature extraction submodule, and a visual feature extraction submodule; the step of performing feature extraction on the multimodal sentiment data modally by modal through the feature extraction module to generate original text features, original audio features, and original visual features includes:

[0012] Acquire text sentiment data, audio sentiment data, and visual sentiment data;

[0013] The original text features of the text sentiment data are extracted through a text feature extraction submodule; the text feature extraction submodule includes a cascaded RoBERTa model and a projection block, and the projection block includes a cascaded fully connected layer, a ReLU activation function and a layer normalization layer;

[0014] An audio feature extraction submodule is used to extract the original audio features of the audio emotion data; the audio feature extraction submodule includes a cascaded linear layer with ReLU activation function and a two-layer BiLSTM network;

[0015] The visual feature extraction submodule extracts the original visual features of the visual emotion data; the visual feature extraction submodule includes a cascaded linear layer with ReLU activation function and a Transformer encoder.

[0016] Optionally, the unified semantic alignment module includes a shared semantic projection submodule, a positional encoding submodule, a linear layer, a cross-modal transformation submodule, a residual normalization layer, and a weighted fusion layer; the cross-modal alignment and interactive fusion of the original text features, the original audio features, and the original visual features based on the unified semantic alignment module to construct collaborative fusion features includes:

[0017] The shared semantic projection submodule is used to unify the dimensions of the original text features, the original audio features, and the original visual features to determine the text alignment features, audio alignment features, and visual alignment features.

[0018] The text alignment feature, audio alignment feature, and visual alignment feature are encoded by adding position encoding to the position encoding submodule to construct text position features, audio position features, and visual position features;

[0019] Linear mapping is performed on the text location features, audio location features, and visual location features based on a linear layer, and the input is fed into the cross-modal conversion submodule for cross-modal cross-attention fusion, outputting text interaction features, audio interaction features, and visual interaction features;

[0020] Based on the residual normalization layer, the text location features are concatenated with the text interaction features, the audio interaction features and the visual interaction features respectively, and then subjected to layer normalization to construct text enhancement features, audio enhancement features and visual enhancement features;

[0021] The enhanced fusion features are determined by weighting and fusing the text enhancement features, audio enhancement features, and visual enhancement features based on the fusion weights through a weighted fusion layer.

[0022] A residual normalization layer is used to perform residual normalization on the text location features and the enhanced fusion features, and output the collaborative fusion features.

[0023] Optionally, the dynamic fusion control module includes a private encoder, a stacking layer, a private average pooling layer, a fusion average pooling layer, a multilayer perceptron, a dimension expansion layer, a weighted fusion layer, a channel modulation submodule, a residual normalization layer, and a splicing fusion layer; the step of performing hierarchical residual semantic gating enhancement based on the unified fusion features according to the high-order semantic abstraction module to generate semantic enhancement features includes:

[0024] Based on a private encoder, the original text features, the original audio features, and the original visual features are encoded respectively to construct text private features, audio private features, and visual private features;

[0025] The text-private features, visual-private features, and audio-private features are stacked sequentially along the modal dimension by stacking layers to output four-dimensional fused features;

[0026] After the collaboratively fused features are averaged along the time dimension using a fusion average pooling layer, they are input into a multilayer perceptron to generate global weights. Then, they are expanded in the time dimension by a dimension expansion layer to output expanded global weights.

[0027] The weighted fusion layer determines the weighted fusion features by multiplying the extended global weights with the four-dimensional fusion features modally and then summing the results.

[0028] The audio private features and the visual private features are respectively input into a private average pooling layer for time average pooling to determine the audio global private features and the visual global private features.

[0029] The channel modulation submodule uses the audio global private features and the visual global private features as condition vectors respectively to perform channel modulation on the weighted fusion features, and outputs the corresponding audio modulation features and visual modulation features.

[0030] The audio modulation feature and the weighted fusion feature, and the visual modulation feature and the weighted fusion feature are respectively subjected to residual enhancement through residual normalization layer to construct audio modulation residual features and visual modulation residual features;

[0031] The weighted fusion feature, the audio enhancement modulation feature, and the visual enhancement modulation feature are input into the feature fusion layer and added element by element to output a unified fusion feature.

[0032] Optionally, the channel modulation submodule includes a scaled multilayer perceptron, an offset multilayer perceptron, and a feature modulation layer; the processing procedure of the channel modulation submodule includes:

[0033] The global private features are input into the scaled multilayer perceptron and the offset multilayer perceptron for feature processing, and the scaling factor and offset are generated accordingly.

[0034] The modulated features are output by scaling the global private features based on the scaling factor and adding them to the offset through the feature modulation layer.

[0035] Optionally, the higher-order semantic abstraction module includes a layer normalization layer, a residual semantic enhancement submodule, a multi-head self-attention submodule with a residual normalization layer, and a global semantic pooling layer; the step of performing hierarchical residual semantic gating enhancement based on the unified fusion features by the higher-order semantic abstraction module to generate semantic enhancement features includes:

[0036] After processing the unified fusion features using a layer normalization layer, the input to multiple cascaded residual semantic enhancement sub-modules is subjected to hierarchical semantic enhancement aggregation to generate high-order fusion features; the residual semantic enhancement sub-modules include a feature update layer with pre-layer normalization, a gated weight adjustment layer with pre-layer normalization, and a weighted residual layer;

[0037] A multi-head self-attention submodule with a residual normalization layer is used to integrate the higher-order fusion features to construct discriminative enhancement features.

[0038] The discriminative enhancement features are averaged along the time dimension using a global semantic pooling layer to output semantic enhancement features.

[0039] A second aspect of the present invention provides a multimodal emotion fusion analysis system, comprising:

[0040] The raw feature extraction module is used to extract features from multimodal sentiment data modally by modal through the feature extraction module, generating raw text features, raw audio features, and raw visual features;

[0041] The semantic alignment module is used to perform cross-modal alignment and fusion of the original text features, the original audio features, and the original visual features based on the unified semantic alignment module, and to construct collaborative fusion features.

[0042] The dynamic fusion module is used to perform two-layer dynamic fusion optimization of modality and channel based on the original text features, the original audio features, the original visual features, and the collaborative fusion features, and to determine the unified fusion features.

[0043] The semantic enhancement module is used to perform hierarchical residual semantic gating enhancement based on the unified fusion features by the high-order semantic abstraction module, and generate semantic enhancement features.

[0044] The sentiment analysis module is used to input the semantic enhancement features into the sentiment prediction module and output the sentiment analysis results.

[0045] A computer device provided in a third aspect of the present invention includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the multimodal sentiment analysis method as described in any of the first aspects of the present invention.

[0046] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the multimodal sentiment analysis method as described in any of the first aspects of the present invention.

[0047] The fifth aspect of the present invention provides a computer program product comprising a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the multimodal sentiment analysis method as described in any of the first aspects of the present invention.

[0048] As can be seen from the above technical solutions, the present invention has the following advantages:

[0049] The above-mentioned solution of the present invention provides a multimodal sentiment fusion analysis method, comprising: extracting features from multimodal sentiment data modally by a feature extraction module to generate original text features, original audio features, and original visual features; performing cross-modal alignment and interactive fusion of the original text features, original audio features, and original visual features based on a unified semantic alignment module to construct collaborative fusion features; employing a dynamic fusion control module to perform two-layer dynamic fusion optimization of modality and channel based on the original text features, original audio features, original visual features, and collaborative fusion features to determine unified fusion features; performing hierarchical residual semantic gating enhancement based on the unified fusion features using a high-order semantic abstraction module to generate semantic enhancement features; and inputting the semantic enhancement features into a sentiment prediction module to output sentiment analysis results. Based on the above scheme, after feature extraction from multimodal sentiment data, heterogeneous modal features are mapped to a unified semantic space to enhance cross-modal semantic consistency. The expression of weak modalities is optimized by dynamically adjusting modal weights. At the same time, semantic expression is strengthened at the channel level, and a hierarchical residual semantic enhancement mechanism is introduced to optimize higher-order semantic abstraction, forming an end-to-end collaborative closed-loop optimization path. The functions of different stages are complementary and the process is coherent, which effectively enhances the overall consistency and expressive ability of cross-modal fusion, solves problems such as semantic mismatch and temporal misalignment between modalities, and helps to provide more stable and accurate and reliable sentiment recognition results. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating the steps of a multimodal emotion fusion analysis method provided in Embodiment 1 of the present invention.

[0052] Figure 2 This is a network architecture diagram of the multimodal sentiment analysis network provided in Embodiment 1 of the present invention;

[0053] Figure 3 This is a schematic diagram of the structure of the unified semantic alignment module provided in Embodiment 1 of the present invention;

[0054] Figure 4 This is a schematic diagram of the dynamic fusion control module provided in Embodiment 1 of the present invention;

[0055] Figure 5 This is a schematic diagram of the structure of the high-order semantic abstraction module provided in Embodiment 1 of the present invention;

[0056] Figure 6This is a structural block diagram of a multimodal emotion fusion analysis system provided in Embodiment 2 of the present invention. Detailed Implementation

[0057] This invention provides a multimodal sentiment fusion analysis method and system to address the technical problem that traditional multimodal sentiment analysis methods mainly involve directly splicing or statically weighting the features of each modality and then mapping the output sentiment classification results. This method is prone to problems such as changes in the contribution weights between modalities and insufficient adaptability of information flow, making it difficult to cope with the complexity and diversity of sentiment fluctuations, resulting in low accuracy of sentiment recognition.

[0058] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0059] Figure 1 This invention provides a multimodal emotion fusion analysis method according to Embodiment 1.

[0060] It should be noted that this embodiment constructs a multimodal sentiment analysis network (MIFA) for the multimodal sentiment analysis task, and its network architecture is as follows: Figure 2 As shown, it includes a feature extraction module, a unified semantic alignment module, a dynamic fusion and control module, a high-order semantic abstraction module, and a sentiment prediction module.

[0061] This embodiment provides a multimodal emotion fusion analysis method, including:

[0062] Step 101: Extract features from the multimodal sentiment data modally using the feature extraction module to generate original text features, original audio features, and original visual features.

[0063] It should be noted that multimodal includes multiple modes; specifically, modes... Can include text modality Visual modal and audio modality Right now Correspondingly, multimodal sentiment data includes text sentiment data. Audio sentiment data and visual emotion data The feature extraction module designed in this embodiment extracts features from the emotional data of each modality, thereby obtaining the corresponding original text features, original audio features, and original visual features.

[0064] In one specific embodiment of this example, the feature extraction module includes a text feature extraction submodule, an audio feature extraction submodule, and a visual feature extraction submodule; step 101 includes the following sub-steps:

[0065] S11. Obtain text sentiment data, audio sentiment data, and visual sentiment data.

[0066] S12. Extract the original text features of the text sentiment data through the text feature extraction submodule.

[0067] It should be noted that, as Figure 2 As shown, this embodiment uses a cascaded RoBERTa model and projection blocks to build a text feature extraction submodule to extract the original text features of the text modality. In specific implementation:

[0068] First, a RoBERTa pre-trained model is used to extract contextual semantic features from text sentiment data (e.g., text sequences). This process is represented as follows: In the formula, The output of the RoBERTa model, For the RoBERTa model, For text sentiment data;

[0069] Secondly, the output of the RoBERTa model is fed into a projection block to obtain a dimensionally unified embedding representation. The projection block consists of cascaded fully connected layers, ReLU activation functions, and layer normalization layers. This process is represented as follows: In the formula, Original text features For layer normalization, It is the ReLU activation function. This is a mapping matrix for text modalities. This is the bias term for the text modality.

[0070] S13. Use the audio feature extraction submodule to extract the original audio features of the audio emotion data.

[0071] It should be noted that, as Figure 2 As shown, this embodiment uses a cascaded linear layer with ReLU activation function and a two-layer BiLSTM network to build an audio feature extraction submodule to extract the original audio features of the audio modality. In specific implementation:

[0072] First, after projecting the audio emotional data (e.g., audio sequences) using a linear layer, a non-linear mapping is performed using the ReLU activation function. This process can be represented as: In the formula, For the audio modal output of a linear layer with ReLU activation function, It is the ReLU activation function. This is the mapping matrix for audio modes. For the bias term of the audio modality, For audio emotion data;

[0073] Next, a context structure model is constructed using a two-layer BiLSTM network to model the aforementioned output. In the two-layer BiLSTM network, the first... The forward and reverse states of the BiLSTM layer are respectively and In the formula, index for time steps , For the first Layer LSTM at time steps The positive hidden state, For the first Layer LSTM at time steps The positive cell state, For the first Layer LSTM, For the first Layer LSTM at time steps The audio modal output characteristics, For the first Layer LSTM at time steps The positive hidden state, For the first Layer LSTM at time steps The positive cell state, For the first Layer LSTM at time steps The reverse hidden state, For the first Layer LSTM at time steps The reverse cell state, For the first Layer LSTM at time steps The reverse hidden state, For the first Layer LSTM at time steps The reverse cell state, the final output of the original audio features is , For the first layer of BiLSTM, This is the second layer of BiLSTM. The feature processing process of the two-layer BiLSTM network can be found in existing technologies and will not be repeated here.

[0074] Understandable This is a network layer index that can be used across different module structures in multimodal sentiment analysis networks.

[0075] S14. Extract the original visual features of the visual emotion data based on the visual feature extraction submodule. It should be noted that, for example... Figure 2 As shown, this embodiment uses a cascaded linear layer with ReLU activation function and a Transformer encoder to build an audio feature extraction submodule to extract the original visual features of the visual modality. In specific implementation:

[0076] First, visual emotion data (e.g., frame-level image sequences) is compressed and non-linearly mapped using a linear layer with a ReLU activation function. This process can be represented as: , For the visual modal output of a linear layer with ReLU activation function, It is the ReLU activation function. This is the mapping matrix for visual modalities. For the visual modality bias;

[0077] Furthermore, to further capture long-term dependencies in the frame sequence, a Transformer encoder is used to model the context, ultimately obtaining the original visual features. This process can be represented as: , It is a Transformer encoder.

[0078] Therefore, the temporal features of the three modalities in a unified feature space are obtained through the feature extraction module: original text features. Original audio features and visual primitive features These can be collectively referred to as primitive features. , For sequence length, This is a general feature dimension.

[0079] Step 102: Based on the unified semantic alignment module, perform cross-modal alignment and interactive fusion of original text features, original audio features, and original visual features to construct collaborative fusion features.

[0080] It should be noted that if the features of each modality are extracted and then directly fused without alignment, redundancy and semantic bias will often be introduced due to the heterogeneity of the expressions between modalities. Therefore, this embodiment constructs a unified semantic alignment module to perform cross-modal alignment of the original features of each modality and then perform preliminary interactive fusion to achieve structural alignment and semantic synergy, thereby obtaining synergistic fused features.

[0081] In one specific implementation of this embodiment, such as Figure 2 and Figure 3 As shown, the unified semantic alignment module includes a shared semantic projection submodule, a position encoding submodule, a linear layer, a cross-modal transformation submodule, a residual normalization layer, and a weighted fusion layer; step 102 includes the following sub-steps:

[0082] S21. The shared semantic projection submodule is used to unify the dimensions of the original text features, original audio features, and original visual features to determine the text alignment features, audio alignment features, and visual alignment features.

[0083] It should be noted that, in order to address the limitations of differences in the original representation dimensions among different modalities and to establish a consistent feature base for a unified attention path, a shared semantic projection (SSP) submodule is used to share the original features of each modality. ( Perform structural normalization processing. For modality Original feature dimensions, projected onto shared feature dimensions In the shared semantic space (i.e., the output dimension after unified semantic alignment), this process can be represented as: In the formula, For modality Alignment features, For modality The projection mapping function, For modality The mapping matrix, For modality The bias term.

[0084] S22. Add positional encoding to text alignment features, audio alignment features, and visual alignment features through the positional encoding submodule to construct text positional features, audio positional features, and visual positional features.

[0085] It should be noted that, to ensure the comparability of token-level slices across time dimensions, positional encoding is added to the forward alignment features entering the attention layer, thereby ensuring semantic consistency across time steps. The process of positional encoding can be represented as follows: , For modality Location features, For position encoding, For modality Alignment features, For sequence length, To share feature dimensions.

[0086] S23. Based on the linear layer, perform linear mapping on text position features, audio position features and visual position features, and input them into the cross-modal transformation submodule for cross-modal cross-attention fusion, and output text interaction features, audio interaction features and visual interaction features.

[0087] It should be noted that in this embodiment, three paths are constructed in parallel to decouple and independently calculate attention. On the one hand, the original expression of the main modality is preserved, and on the other hand, complementary information from other modalities is injected. At the same time, the amplification / interference between paths is suppressed, thereby achieving cross-modal collaborative alignment in a unified dimension.

[0088] The cross-modal conversion submodule is based on a multi-head cross-attention mechanism. Therefore, before entering the multi-head attention calculation, the shapes of each linear mapping matrix in the linear layer are determined as follows: , For the first Attention head modality The query mapping matrix, For the first Attention head modality The key mapping matrix, For the first Attention head modality The value mapping matrix, To share feature dimensions, For attention head index, For attention head count, It is a single-head dimension and satisfies Each head uses independent projection; on these three equivalent paths, text location features, audio location features, and visual location features are projected into query matrices through linear layers. Key matrix Sum matrix The modal combinations are input into the cross-modal conversion submodule for cross-modal attention fusion.

[0089] Specifically, taking text modality dominance as an example, the text location features are projected into a text query matrix, the audio location features into an audio key matrix and an audio value matrix, and the visual location features into a visual key matrix and a visual value matrix through a linear layer. This projection process can be represented as follows: , and , index for time steps ( In the cross-modal conversion submodule, multi-head cross-attention operations are performed using the text query matrix, audio key matrix, and audio value matrix to determine the text-audio cross features. Multi-head cross-attention operations are also performed using the text query matrix, visual key matrix, and visual value matrix to determine the text-visual cross features. The text-audio cross features and text-visual cross features are then concatenated to form the text interaction features. Similarly, the remaining paths perform cross-modal cross-attention fusion by mapping audio position features to the audio query matrix and projecting visual position features to the visual query matrix, thereby obtaining the corresponding audio interaction features and visual interaction features.

[0090] In summary, the process of determining the aforementioned text interaction features, audio interaction features, and visual interaction features includes:

[0091] , , , , , and From respectively A linear mapping, representing the first Modal of attention head input Query matrix / key matrix / value matrix For transpose, Attention scaling factor (used to suppress randomness) Increased attention distribution may lead to overly sharp or oscillating gradients. For softmax operation;

[0092] Slice together and project all the attention points: , For modality At time step The cross features represent the modalities. At time step The linear transformation result after splicing the multi-head attention outputs. For splicing operations, For modality The attention output mapping matrix is ​​obtained through Concat. dimensional vector, through Return to This dimension ensures that the outputs of each path are in the same shared dimension.

[0093] Path set , Indicates except For the two modalities outside, the three equivalent paths are uniformly written as: This formula takes the "dominant mode" as His modal action The three-way structure converges to the same operator instance, formally representing "decoupling but equivalence";

[0094] Finally, the two intersecting features of each modal path are concatenated to obtain the corresponding modality. Interactive features This includes text interaction features, audio interaction features, and visual interaction features.

[0095] S24. Based on the residual normalization layer, the text location features are concatenated with the text interaction features, audio interaction features and visual interaction features respectively, and then normalized layer by layer to construct text enhancement features, audio enhancement features and visual enhancement features.

[0096] It should be noted that a residual normalization layer (Residual+LayerNorm) is introduced into the path output that guides the attention operation for each modality. , For modality Enhanced features (representing the modality) Alignment representation mapped to the text semantic space, including text interaction features, audio interaction features, and visual interaction features. For layer normalization.

[0097] S25. The enhanced fusion features are determined by weighting and fusing text enhancement features, audio enhancement features, and visual enhancement features based on the fusion weights through a weighted fusion layer.

[0098] S26. A residual normalization layer is used to perform residual normalization on the text location features and the enhanced fusion features, and output the collaborative fusion features.

[0099] It should be noted that when integrating the outputs of multiple paths, the final representation is constructed using a collaborative structural fusion approach: ,in, For modality At time step The fusion weights are scalar coefficients and satisfy normalization constraints. and To ensure that the proportions of contributions from each mode are comparable and stable, To enhance fusion features, For time step The collaborative fusion feature vector, after passing through the unified semantic alignment module, yields the fusion feature representation for each time step as follows: , To enhance fusion features, For sequence length, For batch size, This is a general feature dimension.

[0100] In a more specific implementation of this embodiment, to avoid overlap with the dynamic weight estimation function of the next-stage dynamic fusion control module, static equilibrium can be considered here:

[0101]

[0102] It is understandable that the sequence length is recorded. The main complexity of the Unified Semantic Alignment Module (UMA) consists of the three-path MHA and linear projection: ,in , For modality The original feature dimension, without introducing secondary attention or cross-layer coupling, is comparable in magnitude to MulT.

[0103] Step 103: Using the dynamic fusion control module, based on the original text features, original audio features, original visual features, and collaborative fusion features, perform two-layer dynamic fusion optimization of modality and channel to determine the unified fusion features.

[0104] It should be noted that, in order to address the problem of insufficient adaptability of traditional static fusion strategies, this embodiment adopts a dynamic fusion control module to construct a two-layer fusion control path of modality layer and channel layer. The goal is to dynamically adjust modality weights at the modality level to improve the contextual adaptability of fusion expression, and enhance the semantic detail discrimination capability at the channel level to optimize the expression quality of weak modalities.

[0105] In one specific implementation of this embodiment, such as Figure 2 and Figure 4 As shown, the dynamic fusion control module includes a private encoder, a stacking layer, a private average pooling layer, a fusion average pooling layer, a multilayer perceptron, a dimension expansion layer, a weighted fusion layer, a channel modulation submodule, a residual normalization layer, and a stitching fusion layer; step 103 includes the following sub-steps:

[0106] S31. Based on the private encoder, the original text features, original audio features and original visual features are encoded respectively to construct the private text features, private audio features and private visual features.

[0107] It should be noted that, in order to preserve the independent expressive power of each modality, this embodiment applies a private encoder to each modality to retain the private features in its original semantic space, denoted as text private features. Visual private features Audio private features These three sets of private features, as a supplement to the fusion features, will be dynamically weighted and fused based on modality-guided weights.

[0108] S32. By stacking text private features, visual private features and audio private features in sequence along the modal dimension through stacking layers, a four-dimensional fused feature is output.

[0109] It should be noted that the private features of the three modalities are concatenated along the modality dimension to construct a four-dimensional tensor for weighted fusion: , As a four-dimensional fusion feature, For stack operations, This is a dimension index; it can be understood that, in the specific implementation of this embodiment, the first dimension is the time dimension (time step). The second dimension is the batch dimension. The third dimension is the modality dimension, and the fourth dimension is the feature dimension. The stack operation is mathematically equivalent to combining three sets of features in sequence into a new tensor with a third dimension of length 3 in the modal dimension.

[0110] S33. After the collaboratively fused features are averaged along the time dimension using a fusion average pooling layer, they are input into a multilayer perceptron to generate global weights. Then, the dimensionality is expanded in the time dimension through a dimensionality expansion layer, and the expanded global weights are output.

[0111] It should be noted that in the average pooling-fusion layer, average pooling is performed along the time dimension on the collaboratively fused features to obtain global context features: This vector is used to guide the context gating mechanism in generating importance weights for each modality in the current sample;

[0112] according to Generate trimodal importance weights, i.e., use a multilayer perceptron (such as a two-layer perceptron) to generate global weights. To facilitate adaptive fusion: In the formula, It is the Sigmoid activation function. It is the ReLU activation function. This is the first perception mapping matrix. This is the second perceptual mapping matrix. For the first bias term, For the second bias term, For transpose;

[0113] The global weights are expanded using a dimension expansion layer (unsqueeze) at the first dimension, i.e., the time dimension: , To expand the global weights, For dimensional expansion, For dimensional indexing.

[0114] S34. Based on the weighted fusion layer, the extended global weights and four-dimensional fusion features are multiplied modally and then added together to determine the weighted fusion features.

[0115] It should be noted that multiplying the extended global weights element-wise with the four-dimensional fused features modally allows for dynamic filtering of the contextual relevance of feature enhancement. This can be understood as... Will be with Multiplication along the time dimension and feature dimensions The automatic broadcast resulted in the following: In the formula, For weighted fusion features, To extend global weights in modal The coefficients above are uniformly applied to all time steps. For four-dimensional fusion features in modality The eigenvectors on.

[0116] S35. Input the audio private features and visual private features into the private average pooling layer for time average pooling to determine the audio global private features and visual global private features.

[0117] It should be noted that, to alleviate semantic sparsity caused by insufficient information at certain time steps in a text-dominant modality, this embodiment introduces a channel modulation mechanism driven by cross-modal private features. A private average pooling layer is used to perform temporal average pooling to obtain global private features of both the audio and visual modalities, preserving dimensionality and feature dimension information. , ,in, These are audio private features and visual private features at time steps. The private feature vector obtained These are audio global private features and visual global private features, respectively, preserving dimensional and feature dimension information.

[0118] S36. The channel modulation submodule uses audio global private features and visual global private features as condition vectors respectively to perform channel modulation on the weighted fusion features and output the corresponding audio modulation features and visual modulation features.

[0119] It should be noted that this embodiment uses the Channel Modulation submodule to generate modulation coefficients by using the global private features of audio / visual as modulation conditions, and performs channel-level modulation on the weighted fusion features, which only applies to the audio / visual modal channels, thereby further enhancing the feature representation capability of the audio / visual modal.

[0120] In a more specific embodiment of this example, the channel modulation submodule includes a scaled multilayer perceptron, an offset multilayer perceptron, and a feature modulation layer; the processing procedure of the channel modulation submodule includes:

[0121] S361. Input the global private features into the scaled multilayer perceptron and the offset multilayer perceptron respectively for feature processing, and generate the scaling factor and offset accordingly.

[0122] S362. Through the feature modulation layer, the global private features are scaled based on the scaling factor and then added to the offset to output the modulation features.

[0123] It should be noted that this embodiment uses globally private features. As a conditional vector, perform channel-level scaling and offset: , In the formula, global private features They are generated from global state encodings of the audio and visual modalities, respectively. The perceptron consists of a scaled multilayer perceptron and an offset multilayer perceptron, using ReLU activation, and the output... Then based on the scaling factor and offset Perform channel modulation calculations: In the formula, This is element-wise multiplication. Modulation characteristics, In conjunction with weighted fusion features When multiplying or adding, the time dimension will be affected. Automatic broadcasting; following the channel modulation process described above, this embodiment constructs independent channel modulation paths for audio and visual modalities respectively, with the corresponding output being... ,in This represents the modulation branch of the audio or visual modality, ensuring that the features of the audio modality and the characteristics of the visual modality are finely tuned at the channel level.

[0124] S37. Residual enhancement is performed on the audio modulation features and weighted fusion features, and the visual modulation features and weighted fusion features respectively through the residual normalization layer to construct audio modulation residual features and visual modulation residual features.

[0125] It should be noted that residual enhancement is performed on the audio modulation features / visual modulation features through a residual normalization layer (Residual+LayerNorm) to enhance their semantic expressive power. , For modality The modulation residual characteristics, For layer normalization, For modality modulation characteristics, This is a weighted fusion feature.

[0126] S38. Input the weighted fusion features, audio enhancement modulation features, and visual enhancement modulation features into the feature fusion layer and add them element by element to output a unified fusion feature.

[0127] It should be noted that the weighted fusion feature Audio enhancement modulation features and visual enhancement modulation features When added together, they form a unified and integrated characteristic: .

[0128] It is understood that this embodiment integrates a context gating mechanism and a channel modulation mechanism in the dynamic fusion control module to synergistically improve dynamic adaptability and multimodal complementarity. First, the context gating mechanism is used to dynamically adjust the overall modal weights. The expression of weak modalities is optimized through adaptive weighted fusion to form context-sensitive preliminary fusion features. Then, the channel modulation mechanism is used to further optimize the feature expression intensity at the channel level, enhance the ability to express high-dimensional semantic details, and realize the dynamic adjustment of modal weights and channel-level semantic enhancement.

[0129] Step 104: Based on the unified fusion features, perform hierarchical residual semantic gating enhancement according to the high-order semantic abstraction module to generate semantic enhancement features.

[0130] It should be noted that the high-order semantic abstraction module designed in this embodiment integrates a hierarchical residual semantic enhancement mechanism. This mechanism performs hierarchical processing on the features of each layer and completes the high-order semantic abstraction optimization through a multi-layer residual structure.

[0131] In one specific implementation of this embodiment, such as Figure 2 and Figure 5 As shown, the high-order semantic abstraction module includes a layer normalization layer, a residual semantic enhancement submodule, a multi-head self-attention submodule with a residual normalization layer, and a global semantic pooling layer; step 104 includes the following sub-steps:

[0132] S41. After processing the unified fusion features using a layer normalization layer, the input multiple cascaded residual semantic enhancement sub-modules are aggregated for hierarchical semantic enhancement to generate high-order fusion features.

[0133] It should be noted that this embodiment adopts... A hierarchical residual semantic enhancement mechanism is implemented by stacking several residual semantic enhancement submodules, with the output of the previous layer serving as the input of the next layer; this mechanism unifies and fuses features. The input layer, after being processed by the normalization layer, serves as the input to the 0th layer of the first residual semantic enhancement submodule, denoted as... , The final output high-order fusion feature is , This represents the number of residual semantic enhancement submodules.

[0134] In a more specific implementation of this embodiment, the residual semantic enhancement submodule includes a feature update layer with pre-layer normalization, a gated weight adjustment layer with pre-layer normalization, and a weighted residual layer.

[0135] It should be noted that the residual semantic enhancement submodule is equipped with a feature update layer and a gated weight adjustment layer, as well as a weighted residual layer that weights the outputs of the two layers and connects them to the module input residual to obtain the module output. This achieves adjustable enhancement on the information residual path, effectively preserving the original semantics while introducing semantic extension. At the same time, in order to improve the discriminativeness of higher-order semantic abstraction, this embodiment reduces the internal covariance shift by applying layer normalization in advance, and makes residual injection controllable by gating at each position, thus suppressing noise amplification.

[0136] In practical implementation, the feature update layer can be a Transformer, so the feature update path can essentially be a position-wise feedforward layer (Position-wise FFN / MLP) paradigm of the Transformer: , ,in, For the first The semantic change features output by the layer feature update layer are used. For the first The feature inputs of each residual semantic enhancement submodule For layer normalization, The feature semantic mapping matrix, It is the ReLU activation function. This is the fourth mapping matrix;

[0137] The gated weight adjustment layer is used to learn position-wise weight adjustment to control the input strength of the feature residuals, and uses the sigmoid activation function to generate gate coefficients, thereby adjusting the strength of the information flow channel. ,at this time ,in, For the Sigmoid activation function, the first... The gating coefficient of the layer output by the layer gate weight adjustment layer , Here is the gate mapping matrix; the gate coefficients are... Control the first The update strength of the semantic change feature of the layer: when it approaches 0, it means that the original semantics are preserved, and when it approaches 1, it means that the modeling of the current semantic content is enhanced;

[0138] In the weighted residual layer, the outputs of the feature update layer and the gated weight adjustment layer are multiplied element-wise to control the update intensity position-by-position with the feature dimension over time steps. This result is then combined with the module input of the current residual semantic enhancement submodule to obtain the module output of the current residual semantic enhancement submodule. element-wise multiplication ensure and exist Perfectly aligned in three dimensions;

[0139] The output of the current residual semantic enhancement submodule is input into the next residual semantic enhancement submodule to further expand the semantic features. This allows for hierarchical semantic aggregation through layer stacking. Combined with a context-aware gating mechanism, semantic compression and discriminative feature enhancement are achieved, thereby improving the semantic consistency of the fused features.

[0140] S42. A multi-head self-attention submodule with a residual normalization layer is used to integrate high-order fusion features and construct discriminative enhancement features.

[0141] It should be noted that, in order to integrate long-range dependency information and further improve the discriminative ability of features, high-order fused features are fed into a lightweight multi-head self-attention submodule and combined with residual normalization: In the formula, To identify enhancement features, For layer normalization, This is a multi-head self-attention mechanism. This is a high-order fusion feature.

[0142] S43. The discriminative enhancement features are averaged along the time dimension through a global semantic pooling layer to output semantic enhancement features.

[0143] It should be noted that, in order to obtain sentence-level global semantic representation, a global semantic pooling layer is used to enhance the discriminative features. Perform average pooling over the time dimension to determine semantic enhancement features. :

[0144]

[0145] Step 105: Input the semantic enhancement features into the sentiment prediction module and output the sentiment analysis results.

[0146] It should be noted that the sentiment prediction module maps features to sentiment to obtain sentiment analysis results. Multimodal sentiment analysis tasks include sentiment classification tasks and sentiment regression tasks. According to the design of multimodal sentiment analysis tasks, sentiment analysis results can include sentiment classification results and sentiment intensity values, etc.

[0147] For example, in a sentiment classification task, the sentiment prediction module includes a fully connected layer, a softmax activation function, and an argmax function. Semantic enhancement features are input into the fully connected layer and then activated via the softmax activation function (by category dimension). The sentiment prediction probability distribution is obtained by normalization, and the sentiment category is extracted from the sentiment prediction probability distribution using the argmax function, which is then used as the sentiment analysis result. , In the formula, For the probability distribution of sentiment prediction, The Softmax activation function is used. It is a fully connected mapping matrix. For fully connected bias terms, For the number of emotion categories, For semantic enhancement features, For the sentiment classification results, For the argmax function;

[0148] For example, in a sentiment regression task, the sentiment prediction module includes a linear mapping layer, which outputs a sentiment intensity value (scalar) based on semantically enhanced feature mapping. In the formula, It is a linear mapping layer. The emotional intensity value. This is a semantic enhancement feature.

[0149] In one specific implementation of this embodiment, the method further includes: using a multimodal sentiment dataset to train the multimodal sentiment analysis network to be trained, iteratively determining the trained multimodal sentiment analysis network based on the total loss function; and inputting the trained multimodal sentiment data to be identified into the acquired multimodal sentiment data to be identified for sentiment prediction analysis.

[0150] It should be noted that after building the multimodal sentiment analysis network, the model is trained using a multimodal sentiment dataset combined with a preset total loss function to optimize the model parameters, thereby determining the trained multimodal sentiment analysis network. This network can then be used for sentiment prediction analysis using the trained multimodal sentiment data to be identified. Understandably, to ensure the stability and efficiency of model training, constraints such as early stopping mechanisms and Dropout strategies can be introduced during the training process. The multimodal sentiment dataset can include training, validation, and test sets, all of which can be obtained from the CMU-MOSI and CMU-MOSEI datasets, for example:

[0151] Table 1 Dataset Information

[0152]

[0153] In practical implementation, the total loss function of this embodiment A weighted combination of classification and regression losses can be used to train and optimize the entire model, ensuring simultaneous optimization for both sentiment classification and sentiment regression tasks. ,in , These are the weighting coefficients for classification and regression losses, respectively, and the classification loss is... and regression loss The design can refer to existing technologies;

[0154] More specifically, the regression loss can be the mean squared error loss: , For batch size, For sample index, For the sample The predicted sentiment intensity value, For the sample The true emotional intensity value.

[0155] To verify the effectiveness of the method in this embodiment, experiments were conducted on the CMU-MOSI and CMU-MOSEI datasets:

[0156] In experimental design, for sentiment regression tasks, the evaluation metrics can be the mean absolute error (MAE) and the Pearson correlation coefficient (Corr) to measure the degree of error and the predictive relevance, respectively. Sentiment classification tasks are used to determine the positive or negative polarity of sentiment tendencies, and can be evaluated using accuracy, precision, recall, and weighted F1 score.

[0157] A comparative experiment was conducted using representative models in the current field of multimodal sentiment analysis as reference bases. The experimental results are shown in Table 2.

[0158] Table 2 Experimental Results

[0159]

[0160] In Table 2, the superscript "a" or "b" is used only to indicate that the data comes from a specific document or original paper. Experimental results show that in the CMU-MOSI dataset, MIFA (in this embodiment) achieves 86.43% accuracy (Acc2) and 86.42% F1 score in binary classification, which is more than 6% higher than the classic tensor fusion methods (TFN, LMF). It also outperforms attention-based methods such as MulT and MAG-BERT, verifying the effectiveness of the unified semantic alignment path and dynamic fusion strategy proposed in this embodiment in semantic modeling. This is particularly evident in the seven-class classification task. In (Acc7), MIFA achieved 45.04%, approaching the performance of multi-task models without relying on label augmentation and multi-task mechanisms, demonstrating good expressive integrity and the ability to distinguish emotional boundaries. On the CMU-MOSEI dataset, MIFA's Acc2 and F1 scores were 86.03% and 85.81%, respectively, indicating that it still has good robustness and generalization ability on datasets with large topic spans and imbalanced labels. In terms of regression tasks, MIFA achieved the lowest MAE values ​​on both datasets, demonstrating its ability to sensitively model fluctuations in emotional intensity.

[0161] From the perspective of model architecture, MIFA ensures the basic semantic consistency among heterogeneous modalities through a unified semantic alignment mechanism, laying a solid alignment foundation for subsequent fusion regulation. The context gating mechanism and channel modulation mechanism in the dynamic fusion regulation stage work together to achieve dynamic adjustment of modal contributions and optimization of semantic dimensions, enhancing the discriminative ability of fused expressions. In addition, the hierarchical residual semantic enhancement mechanism in the high-order semantic abstraction stage further improves the adaptability of emotion modeling to complex expression patterns, with a particularly significant improvement in Acc7 classification performance, demonstrating its key role in enhancing the boundaries of multi-class emotions. Overall, the modules in the multi-stage fusion mechanism path form a well-defined and collaboratively enhanced fusion optimization process, significantly improving the overall performance of the model.

[0162] In summary, on the two major public datasets, CMU-MOSI and CMU-MOSEI, MIFA outperforms existing mainstream models in sentiment classification and intensity regression tasks, demonstrating its effectiveness and generalization ability in multimodal heterogeneous data modeling scenarios.

[0163] In this embodiment of the invention, after feature extraction of multimodal sentiment data, heterogeneous modal features are mapped to a unified semantic space to enhance cross-modal semantic consistency. The expression of weak modalities is optimized by dynamically adjusting modal weights. At the same time, semantic expression is strengthened at the channel level, and a hierarchical residual semantic enhancement mechanism is introduced to optimize higher-order semantic abstraction, forming an end-to-end collaborative closed-loop optimization path. The functions of different stages are complementary and the process is coherent, which effectively enhances the overall consistency and expressive ability of cross-modal fusion, solves problems such as semantic mismatch between modalities and temporal misalignment, and helps to provide more stable and accurate and reliable sentiment recognition results.

[0164] Figure 6 A multimodal emotion fusion analysis system provided in Embodiment 2 of the present invention includes:

[0165] The original feature extraction module 601 is used to extract features from multimodal sentiment data modally by modal through the feature extraction module, generating original text features, original audio features, and original visual features;

[0166] Semantic alignment module 602 is used to perform cross-modal alignment and fusion of original text features, original audio features and original visual features based on a unified semantic alignment module to construct collaborative fusion features;

[0167] The dynamic fusion module 603 is used to perform two-layer dynamic fusion optimization of modality and channel based on the original text features, original audio features, original visual features and collaborative fusion features, and determine the unified fusion features.

[0168] The semantic enhancement module 604 is used to perform hierarchical residual semantic gating enhancement based on unified fusion features from the high-order semantic abstraction module to generate semantic enhancement features.

[0169] The sentiment analysis module 606 is used to input semantic enhancement features into the sentiment prediction module and output sentiment analysis results.

[0170] In one specific embodiment of this example, the feature extraction module includes a text feature extraction submodule, an audio feature extraction submodule, and a visual feature extraction submodule; the original feature extraction module 601 is specifically used for:

[0171] Acquire text sentiment data, audio sentiment data, and visual sentiment data;

[0172] The text feature extraction submodule extracts the original text features of the text sentiment data. The text feature extraction submodule includes a cascaded RoBERTa model and a projection block. The projection block includes a cascaded fully connected layer, a ReLU activation function and a layer normalization layer.

[0173] The audio feature extraction submodule is used to extract the original audio features of the audio emotion data; the audio feature extraction submodule includes a cascaded linear layer with ReLU activation function and a two-layer BiLSTM network;

[0174] The visual feature extraction submodule extracts the original visual features of the visual emotion data; the visual feature extraction submodule includes a cascaded linear layer with ReLU activation function and a Transformer encoder.

[0175] In one specific embodiment of this example, the unified semantic alignment module includes a shared semantic projection submodule, a position encoding submodule, a linear layer, a cross-modal transformation submodule, a residual normalization layer, and a weighted fusion layer; the semantic alignment module 602 is specifically used for:

[0176] The shared semantic projection submodule is used to unify the dimensions of the original text features, original audio features, and original visual features to determine the text alignment features, audio alignment features, and visual alignment features.

[0177] The positional encoding submodule adds positional encoding to text alignment features, audio alignment features, and visual alignment features to construct text positional features, audio positional features, and visual positional features.

[0178] Linear layers are used to perform linear mapping on text location features, audio location features, and visual location features. The input is then fed into a cross-modal transformation submodule for cross-modal cross-attention fusion, and the output is text interaction features, audio interaction features, and visual interaction features.

[0179] Based on the residual normalization layer, the text location features are concatenated with the text interaction features, audio interaction features and visual interaction features respectively, and then normalized layer by layer to construct text enhancement features, audio enhancement features and visual enhancement features.

[0180] The enhanced fusion features are determined by weighting and fusing text enhancement features, audio enhancement features, and visual enhancement features based on fusion weights through a weighted fusion layer.

[0181] A residual normalization layer is used to perform residual normalization on text location features and enhanced fusion features, and output collaborative fusion features.

[0182] In one specific embodiment of this example, the dynamic fusion control module includes a private encoder, a stacking layer, a private average pooling layer, a fusion average pooling layer, a multilayer perceptron, a dimension expansion layer, a weighted fusion layer, a channel modulation submodule, a residual normalization layer, and a stitching fusion layer; the dynamic fusion module 603 is specifically used for:

[0183] Based on a private encoder, the original text features, original audio features, and original visual features are encoded separately to construct text private features, audio private features, and visual private features.

[0184] By stacking text-private features, visual-private features, and audio-private features sequentially along the modal dimension through stacking layers, a four-dimensional fused feature is output.

[0185] After the collaboratively fused features are averaged along the time dimension by using a fusion average pooling layer, they are input into a multilayer perceptron to generate global weights. Then, the dimensionality is expanded in the time dimension by a dimensionality expansion layer, and the expanded global weights are output.

[0186] The weighted fusion layer is based on the method of multiplying extended global weights and four-dimensional fusion features modally and then summing them to determine the weighted fusion features;

[0187] The audio private features and visual private features are respectively input into the private average pooling layer for temporal average pooling to determine the audio global private features and the visual global private features.

[0188] The channel modulation submodule uses audio global private features and visual global private features as condition vectors to perform channel modulation on the weighted fusion features, and outputs the corresponding audio modulation features and visual modulation features.

[0189] The residual enhancement is performed on the audio modulation features and weighted fusion features, and the visual modulation features and weighted fusion features respectively by the residual normalization layer, so as to construct the audio modulation residual features and the visual modulation residual features.

[0190] The weighted fusion features, audio enhancement modulation features, and visual enhancement modulation features are input into the feature fusion layer and added element by element to output a unified fusion feature.

[0191] In a more specific embodiment of this example, the channel modulation submodule includes a scaled multilayer perceptron, an offset multilayer perceptron, and a feature modulation layer; the processing procedure of the channel modulation submodule includes:

[0192] The global private features are input into the scaled multilayer perceptron and the offset multilayer perceptron for feature processing, and the scaling factor and offset are generated accordingly.

[0193] The modulated features are output by scaling the global private features based on the scaling factor and adding them to the offset through the feature modulation layer.

[0194] In one specific embodiment of this example, the high-order semantic abstraction module includes a layer normalization layer, a residual semantic enhancement submodule, a multi-head self-attention submodule with a residual normalization layer, and a global semantic pooling layer; the semantic enhancement module 604 is specifically used for:

[0195] After processing the unified fusion features using a layer normalization layer, the input is aggregated into multiple cascaded residual semantic enhancement sub-modules for hierarchical semantic enhancement to generate high-order fusion features. The residual semantic enhancement sub-modules include a feature update layer with pre-layer normalization, a gated weight adjustment layer with pre-layer normalization, and a weighted residual layer.

[0196] A multi-head self-attention submodule with a residual normalization layer is used to integrate high-order fusion features and construct discriminative enhancement features.

[0197] The discriminative enhancement features are averaged along the time dimension by a global semantic pooling layer, and the semantic enhancement features are output.

[0198] Embodiment 3 of the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the multimodal emotion fusion analysis method as described in Embodiment 1 of the present invention.

[0199] Embodiment 4 of the present invention also provides a computer-readable storage medium storing a computer program / instruction thereon, which, when executed by a processor, implements the steps of the multimodal emotion fusion analysis method of Embodiment 1 of the present invention described above.

[0200] Embodiment 5 of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the multimodal emotion fusion analysis method as described in Embodiment 1 of the present invention.

[0201] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0203] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0204] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0205] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0206] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal emotion fusion analysis method, characterized in that, include: The feature extraction module extracts features from the multimodal sentiment data modally, generating original text features, original audio features, and original visual features. Based on the unified semantic alignment module, cross-modal alignment and interactive fusion of the original text features, the original audio features, and the original visual features are performed to construct collaborative fusion features; A dynamic fusion control module is used to perform two-layer dynamic fusion optimization of modality and channel based on the original text features, the original audio features, the original visual features, and the collaborative fusion features to determine a unified fusion feature; Based on the unified fusion features, the higher-order semantic abstraction module performs hierarchical residual semantic gating enhancement to generate semantic enhancement features. The semantic enhancement features are input into the sentiment prediction module, which outputs the sentiment analysis results. The dynamic fusion control module includes a private encoder, a stacking layer, a private average pooling layer, a fusion average pooling layer, a multilayer perceptron, a dimension expansion layer, a weighted fusion layer, a channel modulation submodule, a residual normalization layer, and a splicing fusion layer. The step of performing hierarchical residual semantic gating enhancement based on the unified fusion features using the higher-order semantic abstraction module to generate semantically enhanced features includes: Based on a private encoder, the original text features, the original audio features, and the original visual features are encoded respectively to construct text private features, audio private features, and visual private features; The text-private features, visual-private features, and audio-private features are stacked sequentially along the modal dimension by stacking layers to output four-dimensional fused features; After the collaboratively fused features are averaged along the time dimension using a fusion average pooling layer, they are input into a multilayer perceptron to generate global weights. Then, they are expanded in the time dimension by a dimension expansion layer to output expanded global weights. The weighted fusion layer determines the weighted fusion features by multiplying the extended global weights with the four-dimensional fusion features modally and then summing the results. The audio private features and the visual private features are respectively input into a private average pooling layer for time average pooling to determine the audio global private features and the visual global private features. The channel modulation submodule uses the audio global private features and the visual global private features as condition vectors respectively to perform channel modulation on the weighted fusion features, and outputs the corresponding audio modulation features and visual modulation features. The audio modulation feature and the weighted fusion feature, and the visual modulation feature and the weighted fusion feature are respectively subjected to residual enhancement through residual normalization layer to construct audio modulation residual features and visual modulation residual features; The weighted fusion feature, the audio modulation residual feature, and the visual modulation residual feature are input into the feature fusion layer and added element by element to output a unified fusion feature.

2. The multimodal emotion fusion analysis method according to claim 1, characterized in that, The feature extraction module includes a text feature extraction submodule, an audio feature extraction submodule, and a visual feature extraction submodule; the step of performing feature extraction on multimodal sentiment data modally through the feature extraction module to generate original text features, original audio features, and original visual features includes: Acquire text sentiment data, audio sentiment data, and visual sentiment data; The original text features of the text sentiment data are extracted through a text feature extraction submodule; the text feature extraction submodule includes a cascaded RoBERTa model and a projection block, and the projection block includes a cascaded fully connected layer, a ReLU activation function and a layer normalization layer; An audio feature extraction submodule is used to extract the original audio features of the audio emotion data; the audio feature extraction submodule includes a cascaded linear layer with ReLU activation function and a two-layer BiLSTM network; The visual feature extraction submodule extracts the original visual features of the visual emotion data; the visual feature extraction submodule includes a cascaded linear layer with ReLU activation function and a Transformer encoder.

3. The multimodal emotion fusion analysis method according to claim 1, characterized in that, The unified semantic alignment module includes a shared semantic projection submodule, a positional encoding submodule, a linear layer, a cross-modal transformation submodule, a residual normalization layer, and a weighted fusion layer; the cross-modal alignment and interactive fusion of the original text features, the original audio features, and the original visual features based on the unified semantic alignment module to construct collaborative fusion features includes: The shared semantic projection submodule is used to unify the dimensions of the original text features, the original audio features, and the original visual features to determine the text alignment features, audio alignment features, and visual alignment features. The text alignment feature, audio alignment feature, and visual alignment feature are encoded by adding position encoding to the position encoding submodule to construct text position features, audio position features, and visual position features; Linear mapping is performed on the text location features, audio location features, and visual location features based on a linear layer, and the input is fed into the cross-modal conversion submodule for cross-modal cross-attention fusion, outputting text interaction features, audio interaction features, and visual interaction features; Based on the residual normalization layer, the text location features are concatenated with the text interaction features, the audio interaction features and the visual interaction features respectively, and then subjected to layer normalization to construct text enhancement features, audio enhancement features and visual enhancement features; The enhanced fusion features are determined by weighting and fusing the text enhancement features, audio enhancement features, and visual enhancement features based on the fusion weights through a weighted fusion layer. A residual normalization layer is used to perform residual normalization on the text location features and the enhanced fusion features, and output the collaborative fusion features.

4. The multimodal emotion fusion analysis method according to claim 1, characterized in that, The channel modulation submodule includes a scaled multilayer perceptron, an offset multilayer perceptron, and a feature modulation layer; The processing procedure of the channel modulation submodule includes: The global private features are input into the scaled multilayer perceptron and the offset multilayer perceptron for feature processing, and the scaling factor and offset are generated accordingly. The modulated features are output by scaling the global private features based on the scaling factor and adding them to the offset through the feature modulation layer.

5. The multimodal emotion fusion analysis method according to claim 1, characterized in that, The higher-order semantic abstraction module includes a layer normalization layer, a residual semantic enhancement submodule, a multi-head self-attention submodule with a residual normalization layer, and a global semantic pooling layer; The step of performing hierarchical residual semantic gating enhancement based on the unified fusion features using the higher-order semantic abstraction module to generate semantically enhanced features includes: After processing the unified fusion features using a layer normalization layer, the input to multiple cascaded residual semantic enhancement sub-modules is subjected to hierarchical semantic enhancement aggregation to generate high-order fusion features; the residual semantic enhancement sub-modules include a feature update layer with pre-layer normalization, a gated weight adjustment layer with pre-layer normalization, and a weighted residual layer; A multi-head self-attention submodule with a residual normalization layer is used to integrate the higher-order fusion features to construct discriminative enhancement features. The discriminative enhancement features are averaged along the time dimension using a global semantic pooling layer to output semantic enhancement features.

6. A multimodal emotion fusion analysis system, characterized in that, include: The raw feature extraction module is used to extract features from multimodal sentiment data modally by modal through the feature extraction module, generating raw text features, raw audio features, and raw visual features; The semantic alignment module is used to perform cross-modal alignment and fusion of the original text features, the original audio features, and the original visual features based on the unified semantic alignment module, and to construct collaborative fusion features. The dynamic fusion module is used to perform two-layer dynamic fusion optimization of modality and channel based on the original text features, the original audio features, the original visual features, and the collaborative fusion features, and to determine the unified fusion features. The semantic enhancement module is used to perform hierarchical residual semantic gating enhancement based on the unified fusion features by the high-order semantic abstraction module, and generate semantic enhancement features. The sentiment analysis module is used to input the semantic enhancement features into the sentiment prediction module and output the sentiment analysis results. The dynamic fusion control module includes a private encoder, a stacking layer, a private average pooling layer, a fusion average pooling layer, a multilayer perceptron, a dimension expansion layer, a weighted fusion layer, a channel modulation submodule, a residual normalization layer, and a stitching fusion layer; the dynamic fusion module is specifically used for: Based on a private encoder, the original text features, the original audio features, and the original visual features are encoded respectively to construct text private features, audio private features, and visual private features; The text-private features, visual-private features, and audio-private features are stacked sequentially along the modal dimension by stacking layers to output four-dimensional fused features; After the collaboratively fused features are averaged along the time dimension using a fusion average pooling layer, they are input into a multilayer perceptron to generate global weights. Then, they are expanded in the time dimension by a dimension expansion layer to output expanded global weights. The weighted fusion layer determines the weighted fusion features by multiplying the extended global weights with the four-dimensional fusion features modally and then summing the results. The audio private features and the visual private features are respectively input into a private average pooling layer for time average pooling to determine the audio global private features and the visual global private features. The channel modulation submodule uses the audio global private features and the visual global private features as condition vectors respectively to perform channel modulation on the weighted fusion features, and outputs the corresponding audio modulation features and visual modulation features. The audio modulation feature and the weighted fusion feature, and the visual modulation feature and the weighted fusion feature are respectively subjected to residual enhancement by residual normalization layer to construct audio modulation residual features and visual modulation residual features; The weighted fusion feature, the audio modulation residual feature, and the visual modulation residual feature are input into the feature fusion layer and added element by element to output a unified fusion feature.

7. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the multimodal emotion fusion analysis method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the multimodal emotion fusion analysis method as described in any one of claims 1-5.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the multimodal emotion fusion analysis method as described in any one of claims 1-5.

Images