A robust multi-modal emotion recognition method fusing multi-granularity semantic modeling and structure perception

Abstract

This invention discloses a robust multimodal emotion recognition method integrating multi-granularity semantic modeling and structure awareness, belonging to the field of multimodal and emotional intelligence recognition technology. The method involves acquiring multimodal input data, generating a missing mask to occlude the data using a structured modality missing simulation strategy, extracting features and encoding sequences from the occluded data to obtain a discourse-level basic representation, constructing a dual-granularity semantic modeling mechanism, and combining it with adaptive gating fusion to obtain an enhanced semantic representation, modeling the dialogue as a structural graph containing time dependencies and speaker interaction relationships, and performing structure-aware semantic correction based on a graph neural network, completing the missing modalities through a generative semantic reconstruction module to generate reconstructed semantic representations, constructing a multi-objective optimization function, training the model, and outputting emotion recognition results. This application improves the robustness and recognition stability of the model in modality missing scenarios, and is suitable for human-computer interaction and mental health analysis scenarios.

Description

Technical Field

[0001] This invention relates to the field of multimodal and emotional intelligence recognition technology, specifically to a dialogue emotion recognition method for scenarios with incomplete multimodal data. Background Technology

[0002] Multimodal dialogue emotion recognition technology, as a core support for fields such as human-computer interaction and mental health analysis, requires the combined use of text, speech, and visual information to accurately depict the speaker's emotional state. In real-world applications, frequent issues such as sensor malfunctions, environmental noise interference, and signal obstruction lead to random or continuous time-segment-level data gaps in multimodal data. This significantly degrades the performance of traditional recognition methods based on the assumption of complete modality, making it difficult to meet the robustness requirements of practical applications. How to achieve efficient emotion recognition under conditions of incomplete modality has become a key technical challenge that urgently needs to be addressed in the field of affective computing.

[0003] To address the challenge of modality loss, related technologies have gradually evolved from dependence on complete modalities to adaptive missing modalities. Existing solutions mainly optimize model performance through modality completion, attention-weighted fusion, or missing pattern modeling. However, most of these solutions are designed for single types of missing scenarios, and their semantic modeling granularity is limited, failing to fully exploit the complementary information between global long-term sentiment trends and local short-term sentiment fluctuations in dialogues. Furthermore, modality loss easily leads to semantic shifts, and existing methods lack effective structural constraint mechanisms, resulting in insufficient recognition accuracy and stability in scenarios with high missing rate.

[0004] Chinese patent (publication number: CN 116933051 A) discloses a multimodal emotion recognition method and system for modality missing scenarios. This technology encodes the missing modality, reconstructs the missing modality features using a self-attention mechanism, and employs a gating fusion mechanism to achieve cross-modal feature fusion, effectively improving the poor adaptability of traditional methods to missing modalities. However, this technology still has significant shortcomings: it only optimizes for random modality missing, without considering the common scenario of continuous time segment-level missing, and the reconstruction process relies on a single self-attention mechanism, lacking guidance from the dialogue temporal structure and speaker interaction relationship, resulting in insufficient semantic consistency of the reconstructed features and easy feature distortion under high missing rates.

[0005] Therefore, there is an urgent need to develop technologies that address the semantic shift and insufficient generalization capabilities of existing technologies in complex modality-deficient scenarios, and to meet the high-performance requirements of multimodal dialogue emotion recognition. Summary of the Invention

[0006] Based on the aforementioned technical problems, this application discloses a robust multimodal emotion recognition method that integrates multi-granularity semantic modeling and structure awareness, specifically including:

[0007] Acquire text, speech, and visual trimodal input data for multi-turn dialogues, wherein the multi-turn dialogues include utterances and corresponding speaker identities;

[0008] A missing mask is generated by simulating a structured modal missing data pattern, and the three-modal input data is occluded to obtain the occluded multimodal input.

[0009] Feature extraction and sequence encoding are performed on the occluded multimodal input to obtain the discourse-level basic representation;

[0010] A multi-granularity semantic modeling mechanism is constructed to perform dual-granularity semantic learning on the discourse-level basic representation, and an enhanced semantic representation is obtained by combining it with an adaptive gating fusion mechanism;

[0011] The multi-turn dialogue is modeled as a structural graph containing temporal dependencies and speaker interaction relationships. The enhanced semantic representation is then subjected to structure-aware semantic correction based on a graph neural network to obtain a structure-enhanced representation.

[0012] The structural enhancement representation is filled with missing modalities by a generative semantic reconstruction module to generate a reconstructed semantic representation;

[0013] A multi-objective optimization function is constructed by combining sentiment classification loss, generative reconstruction loss, and supervised contrastive learning loss. The model is then jointly trained and outputs the sentiment recognition result for each utterance.

[0014] Preferably, the structured modal loss simulation strategy includes a random modal loss mechanism and a time-segment level loss mechanism; the random modal loss mechanism randomly occludes the feature channels of any one or more modalities in text, speech, and vision at the discourse level with a preset probability to simulate sudden modal failure; the time-segment level loss mechanism completely masks all features of a certain modality within a continuous preset number of discourse intervals to simulate persistent sensor failure or long-term signal degradation.

[0015] The missing mask It is generated through the combined action of two types of missing mechanisms, and the formula is: ,in For the missing random modal mask, For time-segment level missing masks, This indicates a missing mode at the corresponding location. The modality is available at the corresponding position;

[0016] The original trimodal input data is masked based on the generated missing mask to obtain the masked multimodal input. The formula is: , where ⊙ represents element-wise multiplication. This is the original trimodal input data matrix.

[0017] Preferably, in the feature extraction process, a pre-trained text encoder, speech feature extractor, and visual feature extractor are used to perform feature transformation on the original data of each modality after occlusion, so as to obtain text modality features, speech modality features, and visual modality features, and each modality feature is mapped to a feature space of the same dimension.

[0018] By concatenating the text, speech, and visual modal features of the same discourse along the channel dimension, the multimodal fusion features of the discourse are obtained.

[0019] The multimodal fusion features of all utterances are arranged in dialogue temporal order and input into a bidirectional gated recurrent unit (Bi-GRU) for sequence encoding to capture the basic temporal context information between utterances. The output is a unified-dimensional utterance-level basic representation for each utterance, as shown in the formula: ,in For the first The basic representation of a discourse. The hidden layer dimension is adaptively configured based on the dataset complexity.

[0020] Preferably, the multi-granularity semantic modeling mechanism is constructed through global semantic flow and local semantic flow; the construction of the global semantic flow specifically involves: linearly mapping the utterance-level basic representation sequence to adjust the feature dimension to a preset global semantic modeling dimension; using an encoder based on a self-attention mechanism to perform global context modeling on the adjusted sequence; by calculating the attention weights between all utterances, aggregating long-range dependency information across utterances, focusing on characterizing the overall emotional evolution trend of the dialogue, and outputting a coarse-grained semantic representation containing global context information, as shown in the formula: ,in This is a discourse-level basic representation sequence. A function for global semantic modeling;

[0021] The construction of the local semantic stream specifically involves: using a sliding window of fixed size to locally truncate the utterance-level basic representation sequence, with the window size adaptively set according to the average length of utterances in the dialogue; for each local sequence within the window, a bidirectional recurrent neural network is used to model the short-term temporal dependencies between adjacent utterances, and then a one-dimensional convolution operation is used to extract the emotional change patterns within the local time window, highlighting local semantic differences and outputting a fine-grained semantic representation, as shown in the formula: ,in A function for modeling local semantics.

[0022] Preferably, the adaptive gating fusion mechanism obtains enhanced semantic representation, specifically by: transforming the discourse-level basic representation... Global semantic representation and local semantic representation Each is mapped to a fusion space of the same dimension through a linear transformation;

[0023] The three mapped representations are subjected to dialogue-level feature aggregation. The global feature vector of the entire dialogue is obtained through global average pooling or max pooling. The calculation formula is as follows: ,in This is a dialogue-level feature aggregation operation. This is the global average pooling function;

[0024] The aggregated global feature vectors are input into a learnable gating function. The fusion weights for the three branches are generated using the following formula: ,in The weights corresponding to the basic representation at the discourse level. The weights corresponding to the global semantic representation, The weights corresponding to the local semantic representation;

[0025] Based on the generated fusion weights, the three representations are weighted and summed to obtain the enhanced semantic representation. The formula is: The fusion weights are estimated at the entire dialogue level and shared over time.

[0026] Preferably, the acquisition of the structurally enhanced representation specifically involves: treating each utterance in a multi-turn dialogue as a node in a graph, with the node features being the enhanced semantic representation. ;

[0027] To characterize the relationships between nodes, temporal adjacency edges and speaker interaction edges are constructed, forming a structure graph containing nodes and two types of edges. ,in For a set of nodes, Let it be the set of edges;

[0028] Structure-aware semantic correction is performed by aggregating the features of each node with those of its neighboring nodes using a message passing mechanism in a graph neural network, and then calculating the structure-induced correction signal. The formula is as follows: ,in For the first Structural correction signals for each node For structure-aware mapping functions;

[0029] A lightweight channel aggregation module is introduced to optimize the channel dimension and correct the signal structure. By using a channel-by-channel weighting method, amplified noise channels during graph propagation are suppressed, while retaining semantic channels crucial for emotion discrimination. The formula is as follows: ,in This is the corrected signal after channel optimization. This represents the channel-level weight vector.

[0030] Preferably, the enhanced structure correction signal is refined using an enhanced hybrid parallel attention mechanism, including: refining the channel-optimized correction signal. Input a bidirectional gated recurrent unit to perform temporal feature enhancement, and obtain the enhanced temporal feature sequence;

[0031] Self-attention computation is performed on the enhanced temporal feature sequence to capture the dependencies within the sequence, resulting in self-attention enhanced features. These self-attention enhanced features are used as a memory base, along with the original enhanced semantic representation of each node. As the query vector, cross-attention is calculated using the following formula: ,in For cross-attention output features, These are self-attention and cross-attention functions, respectively.

[0032] Cross-attention output features Compared with the original enhanced semantic representation Residual connections are made, and feature transformation is performed through a feedforward network to obtain refined structure-enhanced features. The calculation formula is as follows: The refined structure-enhanced features are projected back into a unified semantic space via linear mapping to obtain the final structure-enhanced representation, as shown in the formula: ,in It is a learnable linear transformation parameter matrix.

[0033] Preferably, the generative semantic reconstruction module adopts a dual-path memory architecture with short-term memory and long-term memory paths, respectively modeling short-term local patterns and long-term global dependencies: the short-term memory path enhances the representation of the structure through a feedforward network. Feature transformation is performed to directly capture local contextual information without relying on external memory. The formula is as follows: ,in Output features for short-term memory paths;

[0034] Long-term memory pathways are represented by the aforementioned structural enhancements. As the query vector, the original multimodal feature sequence M is used as a memory. Global dependency information related to the current representation is extracted from the memory through a cross-attention mechanism, as shown in the formula: ,in , , These are linear transformation parameter matrices for the query, key, and value, respectively. For attention feature dimension, Output features for long-term memory paths;

[0035] Output features of short-term memory paths Output features of long-term memory pathways Element-wise addition is performed, followed by a linear transformation to generate the final reconstructed semantic representation. The calculation formula is as follows: ,in The linear transformation parameter matrix is ​​learnable. To reconstruct the semantic representation.

[0036] Preferably, the calculation of the generative reconstruction loss specifically involves generating a corresponding modality missing mask for each modality (text, speech, and vision). ,in Indicate the modality type; for each modality, extract the original features. With reconstruction features Through modal missing mask The locations of missing random modalities are identified; the error between the original features and the reconstructed features at the missing locations is calculated using the L2 norm, with the following formula: ,in To retain only the feature differences at the missing locations, It is the square of the L2 norm.

[0037] Preferably, the multi-objective optimization function includes three loss terms, each corresponding to a different training objective: sentiment classification loss. To optimize the emotion recognition accuracy of the model, cross-entropy loss is used for classification tasks, and mean squared error loss is used for regression tasks. The formula is as follows: ,in For the first Predicting sentiment from a single statement As a tag for genuine emotions, For loss calculation function, Total number of discourses; generative reconstruction loss Optimize the completion effect of missing modalities; supervised contrastive learning loss. To enhance the intra-class compactness and inter-class separability of the latent representation space, positive and negative sample pairs are constructed only within the same speaker. Discourses with the same sentiment label are represented as positive sample pairs, and discourses with different sentiment labels are represented as negative sample pairs. The formula is as follows: ,in , , These are the normalized discourse representations, For temperature coefficient, To be consistent with the sample A set of positive samples with the same label For the candidate sample set, The cosine similarity calculation function is used; the multi-objective optimization function fuses three loss terms through dynamic weights, and the formula is: in , These are dynamic weights.

[0038] Compared with the prior art, the technical solution of this application has the following technical effects:

[0039] This invention integrates random modal missing and time-segment-level missing mechanisms through a structured modal missing simulation strategy, accurately replicating diverse modal missing patterns in real-world scenarios. This allows the model to fully engage with various missing scenarios during the training phase, significantly enhancing its adaptability to sudden modal failures and persistent signal degradation. It lays the foundation for model robustness from the data input level and effectively solves the problem of insufficient adaptation to complex missing patterns in traditional methods.

[0040] The global and local dual-granularity semantic modeling mechanism of this invention, through collaborative adaptive gating fusion, achieves accurate capture of the multi-scale evolutionary characteristics of dialogue emotions. The global semantic flow ensures the stability of long-term emotional trends, while the local semantic flow captures short-term emotional fluctuations. The fusion weights dynamically adapt to different missing patterns, enabling the model to flexibly utilize reliable semantic cues and avoid the defects of unstable semantic representation or insufficient discriminative ability caused by single-granularity modeling.

[0041] The structure-aware semantic correction mechanism of this invention, combined with a generative semantic reconstruction module, constructs a dual semantic optimization system. The dialogue graph structure introduces temporal and speaker interaction constraints to alleviate semantic shifts caused by modality loss. The reconstruction module of the dual-path memory architecture effectively completes missing modalities and provides the model with additional self-supervised signals. The two work together to improve the consistency and completeness of semantic representation and enhance the model's ability to compensate for missing modalities.

[0042] The multi-objective joint optimization strategy of emotion classification, generative reconstruction and supervised contrastive learning in this invention takes into account both the accuracy of emotion recognition and the discriminativeness of representation. It not only ensures the accuracy of emotion recognition through the main task loss, but also enhances the intra-class compactness and inter-class separability of the latent space with the help of auxiliary loss. This enables the model to maintain stable recognition performance and have strong discriminative semantic representation under the condition of modality loss, and comprehensively improves the practical value in complex scenarios.

[0043] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings.

[0044] The above and other objects, advantages and features of this application will become more apparent to those skilled in the art from the following detailed description of specific embodiments in conjunction with the accompanying drawings. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0046] Based on the description of the figures and their corresponding technical content in the document, the titles of the figures are as follows:

[0047] Figure 1 A flowchart illustrating the steps of an incomplete multimodal emotion recognition method using multigranular generative graph neural networks.

[0048] Figure 2 Detailed process architecture diagram of structured modality missing simulation and feature extraction encoding;

[0049] Figure 3 : Trend chart of WAF1 of each model with modality missing rate on the IEMOCAP four-class classification dataset;

[0050] Figure 4 : Trend chart of WAF1 of each model with modality missing rate on the IEMOCAP six-class dataset;

[0051] Figure 5 Comparison of MSE changes with missing rate for various models on the IEMOCAP four-class and six-class classification datasets and the CMU-MOSI and CMU-MOSEI datasets;

[0052] Figure 6 : The effect of different mask_p settings on the F1 score of the model when the missing rate is 0.3 in the IEMOCAP four-class classification dataset;

[0053] Figure 7 : The effect of different numbers of global attention heads on the F1 score of the model when the missing rate is 0.3 in the IEMOCAP four-class classification dataset;

[0054] Figure 8 Comparison of sentiment prediction results of various models on incomplete dialogue data in the IEMOCAP four-class and six-class datasets. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. In the following description, specific details such as specific configurations and components are provided merely to help fully understand the embodiments of this application. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. In addition, for clarity and brevity, descriptions of known functions and structures are omitted in the embodiments.

[0056] It should be understood that the phrase "an embodiment" or "this embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "an embodiment" or "this embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0057] Furthermore, reference numerals and / or letters may be repeated in different examples within this application. Such repetition is for the purpose of simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or settings discussed.

[0058] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" in this article describes another type of relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the related objects before and after it are in an "or" relationship.

[0059] In this article, the term "at least one" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, "at least one of A and B" can mean: A exists alone, A and B exist simultaneously, or B exists alone.

[0060] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion.

[0061] Example 1

[0062] This embodiment mainly describes a robust multimodal emotion recognition method that integrates multi-granularity semantic modeling and structure awareness, such as... Figures 1-2 As shown, it specifically includes:

[0063] A robust multimodal emotion recognition method integrating multi-granularity semantic modeling and structure awareness, specifically including:

[0064] Acquire text, speech, and visual trimodal input data for multi-turn dialogues, wherein the multi-turn dialogues include utterances and corresponding speaker identities;

[0065] A missing mask is generated by simulating a structured modal missing data pattern, and the three-modal input data is occluded to obtain the occluded multimodal input.

[0066] Feature extraction and sequence encoding are performed on the occluded multimodal input to obtain the discourse-level basic representation;

[0067] A multi-granularity semantic modeling mechanism is constructed to perform dual-granularity semantic learning on the discourse-level basic representation, and an enhanced semantic representation is obtained by combining it with an adaptive gating fusion mechanism;

[0068] The multi-turn dialogue is modeled as a structural graph containing temporal dependencies and speaker interaction relationships. The enhanced semantic representation is then subjected to structure-aware semantic correction based on a graph neural network to obtain a structure-enhanced representation.

[0069] The structural enhancement representation is filled with missing modalities by a generative semantic reconstruction module to generate a reconstructed semantic representation;

[0070] A multi-objective optimization function is constructed by combining sentiment classification loss, generative reconstruction loss, and supervised contrastive learning loss. The model is then jointly trained and outputs the sentiment recognition result for each utterance.

[0071] Furthermore, text, speech, and visual trimodal input data for a multi-turn dialogue are acquired. The multi-turn dialogue contains T utterances and their corresponding speaker identities. The modal data for each utterance is represented as follows: ,in For speech modal data, For text modal data, Visual modal data;

[0072] A missing mask is generated using a structured modal missing simulation strategy to mask the three-modal input data, resulting in a masked multimodal input. The structured modal missing simulation strategy includes a random modal missing mechanism and a temporal segment-level missing mechanism. The random modal missing mechanism randomly masks the feature channels of any one or more modalities (text, speech, and vision) at the utterance level with a preset probability p (p∈[0,0.3]). The temporal segment-level missing mechanism masks all features of a specific modality over K (K≥2) consecutive utterance intervals. The missing mask M is generated through the combined action of the two missing mechanisms, using the following formula: ,in For the missing random modal mask, This is a time-segment-level missing mask, where M=1 indicates that the corresponding mode is missing, and M=0 indicates that the corresponding mode is available. Based on the generated missing mask, the original three-modal input data is masked to obtain the masked multimodal input. The formula is: , where ⊙ represents element-wise multiplication. This is the original three-modal input data matrix;

[0073] Feature extraction and sequence encoding are performed on the occluded multimodal input to obtain a discourse-level basic representation. During feature extraction, pre-trained text encoders, speech feature extractors, and visual feature extractors are used to transform the original data of each modality after occlusion, resulting in text modality features, speech modality features, and visual modality features, each with a dimension of D (D≥128). The text, speech, and visual modality features of the same discourse are concatenated along the channel dimension to obtain a 3D discourse multimodal fusion feature. The multimodal fusion features of all discourses are arranged in dialogue temporal order and input into a bidirectional gated recurrent unit (Bi-GRU) for sequence encoding. The hidden layer dimension of the Bi-GRU is configured to 100 or 200 depending on the dataset complexity, capturing the basic temporal context information between discourses and outputting a unified-dimensional discourse-level basic representation for each discourse, as shown in the formula: ,in This serves as the basic representation of the i-th utterance;

[0074] A multi-granularity semantic modeling mechanism is constructed, which performs dual-granularity semantic learning on the utterance-level basic representation and obtains enhanced semantic representation by combining an adaptive gating fusion mechanism. The multi-granularity semantic modeling mechanism is constructed through global semantic flow and local semantic flow. The construction process of the global semantic flow is as follows: the utterance-level basic representation sequence H is linearly mapped, and the feature dimension is adjusted to a preset global semantic modeling dimension G (G≥256). A self-attention-based encoder is used to perform global context modeling on the adjusted sequence. The encoder contains L (L≥2) layers of self-attention layers and a feedforward network. By calculating the attention weights between all utterances, long-range dependency information across utterances is aggregated, and a coarse-grained semantic representation containing global context information is output, as shown in the formula: ,in The global semantic modeling function is used; the construction process of the local semantic stream is as follows: a sliding window with a fixed window size W (W=2) is used to locally truncate the utterance-level basic representation sequence H; for the local sequence within each window, the short-term temporal dependency between adjacent utterances is modeled through a bidirectional recurrent neural network, and then the emotion change pattern within the local time window is extracted through a one-dimensional convolution operation (convolution kernel size of 3, stride of 1), and a fine-grained semantic representation is output, as shown in the formula: ,in The local semantic modeling function is used; the implementation process of the adaptive gating fusion mechanism is as follows: the discourse-level basic representation H and the global semantic representation are combined. and local semantic representation Each representation is mapped to a fusion space of the same dimension F (F≥256) through a linear transformation; the three mapped representations are then subjected to dialogue-level feature aggregation, and the global feature vector of the entire dialogue is obtained through global average pooling, calculated as follows: The aggregated global feature vector is input into a learnable gating function consisting of a fully connected layer and a softmax activation function. The fusion weights for the three branches are generated using the following formula: ,in The weights corresponding to the basic representation at the discourse level. The weights corresponding to the global semantic representation, The weights corresponding to the local semantic representation, and satisfying Based on the generated fusion weights, the three representations are weighted and summed to obtain the enhanced semantic representation. The formula is: The fusion weights are estimated at the entire dialogue level and shared over time.

[0075] The multi-turn dialogue is modeled as a structure graph containing temporal dependencies and speaker interaction relationships. The enhanced semantic representation is then subjected to structure-aware semantic correction based on a graph neural network to obtain a structure-enhanced representation. The construction process of the structure graph is as follows: each utterance in the multi-turn dialogue is treated as a node in the graph, and the node features are the enhanced semantic representation. The relationship between nodes is characterized by constructing temporal adjacency edges and speaker interaction edges. Temporal adjacency edges connect temporally adjacent speech nodes, and speaker interaction edges connect speech nodes corresponding to the same speaker or different speakers with interactive relationships, forming a structural graph containing a set of nodes ν and a set of edges ε. When performing structure-aware semantic correction, a relational graph convolutional network RGCN or a hypergraph convolutional network HypergraphConv is used as the structure-aware mapping function based on the message passing mechanism in graph neural networks. The features of each node are aggregated with the features of its neighboring nodes to calculate the structure-induced correction signal, using the following formula: ,in Let be the structural correction signal for the t-th node. A lightweight channel aggregation module (LCA) is introduced to optimize the channel dimension of the structural correction signal. LCA generates a channel-level weight vector ω based on global channel statistics, and suppresses amplified noise channels during graph propagation by weighting each channel sequentially, retaining the semantic channels crucial for emotion discrimination. The formula is: ,in The channel-optimized correction signal is used; the enhanced hybrid parallel attention mechanism (EHPA) is employed to refine the correction representation: the channel-optimized correction signal... A bidirectional gated recurrent unit is input to perform temporal feature enhancement, resulting in an enhanced temporal feature sequence. Self-attention computation is then performed on the enhanced temporal feature sequence to capture the dependencies within the sequence, yielding self-attention enhanced features. These self-attention enhanced features serve as a memory base, with the original enhanced semantic representation of each node as the basis. As the query vector, cross-attention is calculated using the following formula: ,in For cross-attention output features, and These are cross-attention and self-attention functions, respectively; the cross-attention output features are... Compared with the original enhanced semantic representation Residual connections are made, and feature transformation is performed through a feedforward network to obtain refined structure-enhanced features. The calculation formula is as follows: The refined structure-enhanced features are projected back into a unified semantic space via linear mapping to obtain the final structure-enhanced representation, as shown in the formula: ,in Let F be a learnable linear transformation parameter matrix with dimension F×F;

[0076] The structural enhancement representation is filled with missing modalities by a generative semantic reconstruction module to generate a reconstructed semantic representation. The generative semantic reconstruction module employs a dual-path memory architecture with short-term and long-term memory paths. The short-term memory path processes the structural enhancement representation through a feedforward network. Feature transformation is performed to directly capture local contextual information. The formula is as follows: ,in For the output features of the short-term memory path, the FFN contains two fully connected layers and a ReLU activation function. The first fully connected layer has an input dimension of F and an output dimension of 2F, and the second fully connected layer has an input dimension of 2F and an output dimension of F. The long-term memory path is represented by the aforementioned structure enhancement. As the query vector, the original multimodal feature sequence M is used as a memory. Global dependency information related to the current representation is extracted from the memory through a cross-attention mechanism, as shown in the formula: ,in , , These are the query, key, and value linear transformation parameter matrices of dimension F×F. For attention feature dimension, Output features for long-term memory paths; output features for short-term memory paths Output features of long-term memory pathways Element-wise addition is performed, followed by a linear transformation to generate the final reconstructed semantic representation. The calculation formula is as follows: ,in The matrix represents a learnable linear transformation parameter matrix with dimensions F×3D. To reconstruct the semantic representation, the dimension is consistent with the original multimodal feature sequence M;

[0077] A multi-objective optimization function is constructed by combining sentiment classification loss, generative reconstruction loss, and supervised contrastive learning loss. This function is used to jointly train the model and output the sentiment recognition result for each utterance. The sentiment recognition is achieved through a sentiment classifier and uses structure-enhanced representation. As input, the sentiment prediction result of the t-th utterance is output through a linear classifier. The formula is: ,in For the output mapping function, the softmax activation function is used in classification tasks, and the identity mapping is used in regression tasks; Here, C is the number of sentiment categories, and F is the dimension of the structure-enhanced representation; The bias vector is used; the multi-objective optimization function includes three loss terms: sentiment classification loss. To optimize the model's sentiment recognition accuracy, cross-entropy loss is used for classification tasks, and mean squared error loss is used for regression tasks. The formula is as follows: ,in For the predicted sentiment of the t-th utterance, As a tag for genuine emotions, Here, T is the total number of discourses; generative reconstruction loss is the loss function. To optimize the completion effect of missing modalities, a corresponding modality missing mask is generated for each modality: text, speech, and vision. Extracting original features With reconstruction features Through modal missing mask The locations of missing random modalities are identified, and the error between the original features and the reconstructed features at the missing locations is calculated using the L2 norm. The formula is as follows: Supervised contrastive learning loss To enhance the intra-class compactness and inter-class separability of the latent representation space, positive and negative sample pairs are constructed only within the same speaker. Discourses with the same sentiment label are represented as positive sample pairs, and discourses with different sentiment labels are represented as negative sample pairs. The formula is as follows: ,in , , These are the normalized speech representations, and τ is the temperature coefficient (τ=0.07). For the set of positive samples with the same label as sample i, For the candidate sample set, The cosine similarity calculation function is used; the multi-objective optimization function fuses three loss terms through dynamic weights, and the formula is: ,in For cosine dynamic weights based on training epoch e, E represents the maximum number of training rounds (E=100). , The baseline weights are set as the task weights. The model uses the AdamW optimizer with a weight decay coefficient of 1e-4, a batch size of 32, and an initial learning rate of 1e-3. The learning rate is adjusted using a cosine annealing strategy. The optimal model is evaluated and retained on the validation set in each round, and finally the sentiment recognition result of each utterance is output.

[0078] This detailed implementation simulates and adapts to diverse real-world scenarios with missing modalities through structured modality missingness simulation. It accurately captures emotional evolution characteristics using global-local dual-granularity semantic modeling and adaptive fusion, and mitigates semantic shifts and completes missing modalities through dialogue graph structure correction and dual-path generative reconstruction. Finally, multi-objective joint optimization enhances representational discriminativity. This comprehensive approach forms a complete chain from input adaptation, semantic modeling, semantic correction to optimized training, significantly improving the model's robustness and recognition stability under incomplete multimodal conditions, and comprehensively optimizing emotion recognition performance in complex scenarios.

[0079] Based on Example 1 or 2, this example aims to comprehensively evaluate the performance and robustness of the proposed model under different modal integrity conditions. The effectiveness of the method is validated on three commonly used multimodal dialogue / video sentiment datasets: IEMOCAP, CMU-MOSI, and CMU-MOSEI. All three datasets include text, audio, and vision features and provide standard partitions or reusable cross-validation partitions. Consistent with existing work, both four-class and six-class classification settings are reported on IEMOCAP. The original annotations for CMU-MOSI and CMU-MOSEI are sentiment intensity scores in the range [−3,+3][-3, +3][−3,+3].

[0080] The IEMOCAP dataset contains approximately 12 hours of two-person dialogue videos, labeled with multiple emotion categories. Following previous work, it employs two common setups: (i) four-class tasks (happy, sad, angry, neutral); and (ii) six-class tasks (adding excited and frustrated). Each dialogue averages approximately 9 sentences, totaling 7,543 samples, making it the most widely used multimodal benchmark for dialogue emotion recognition.

[0081] CMU-MOSI is a sentence-level multimodal sentiment regression dataset. The samples are from online video comments, and the sentiment labels are continuous values ​​ranging from negative to positive ([-3,3]).

[0082] CMU-MOSEI is an extended version of MOSI, larger in scale, containing approximately 23,000 comments covering more than 1,000 different speakers and various scenarios. Its annotation system is consistent with MOSI, and it is a key benchmark for verifying the cross-domain robustness of models.

[0083] MU-MOSI / MOSEI uses a predefined train / val / test partitioning of the dataset; IEMOCAP uses five-fold (segmented by session) cross-validation and averages the results of each fold.

[0084] All experiments were implemented using the PyTorch framework and conducted under a unified training and evaluation process. The models were trained and tested on three datasets: CMU-MOSI, CMU-MOSEI, and IEMOCAP. To ensure the fairness and reproducibility of the experimental results, the models were differentiated only in terms of model size and a few necessary hyperparameters across different datasets, while the rest of the training configuration remained consistent.

[0085] In the experimental setup, the AdamW optimizer was used uniformly, with a batch size of 32 and a maximum training duration of 100 epochs. Cosine AnnealingLR was used to adjust the learning rate. During training, the model was evaluated on the validation set after each epoch, and the best-performing model on the validation set was used for the final test. Regarding model structure, the hidden layer dimension of the GRU was set to 100 for the CMU-MOSI and CMU-MOSEI datasets. For IEMOCAP, which contains more complex multi-turn dialogues and longer contextual dependencies, the hidden layer dimension of the GRU was increased to 200 to enhance its ability to model long-distance semantics. The temperature coefficient τ of the contrastive learning module was fixed at 0.07 in all experiments. To simulate the potential loss of structured modalities in real-world scenarios, structured occlusion was incorporated during training. The occlusion ratio (mask_p) varied across different datasets: 0.01 for MOSI, 0.02 for MOSEI, and 0.15 for IEMOCAP. This design primarily considers the longer dialogues and more complex interactions in IEMOCAP; appropriately increasing the occlusion ratio can help the model learn more robust representations. Furthermore, in the early stages of training, a cosine warm-up strategy is designed for the generative reconstruction loss and contrastive learning loss. The aim is to allow the model to first focus on learning stable sentiment discrimination features, and then gradually introduce generative completion and alignment constraints to avoid auxiliary tasks interfering with the learning of the main task too early.

[0086] To evaluate the model's robustness to missing modalities, a portion of the modal inputs were randomly masked during both training and testing phases, with the missing rate gradually increasing from 0 (complete) to 0.7 (severely missing). To ensure the task is solvable, at least one usable modality was retained for each sample. For each missing rate, five different missing masks were randomly generated for experiments, and the mean and standard deviation were reported to comprehensively reflect the model's stability and generalization ability under different levels of missing data.

[0087] Regarding evaluation metrics, to comprehensively and fairly assess the model's performance on different tasks and datasets, evaluation metrics that are widely recognized and complementary in the field of sentiment computing were selected, taking into account the characteristics of each dataset and the task objectives. For the IEMOCAP sentiment classification task, the weighted average F1 score (WAF1) was chosen as the core evaluation metric, with the formula:

[0088]

[0089] Where E is the number of categories. For the first Number of samples per class. For the sentiment polarity classification task of CMU-MOSI and CMU-MOSEI (i.e., judging whether the overall sentiment of a sentence is positive or negative, where the original label score < 0 is considered negative and the score > 0 is considered positive), the weighted average F1 score (WAF1) and accuracy are also reported to facilitate a more comprehensive evaluation of the model's classification performance from different dimensions.

[0090] To verify the effectiveness of the MAGIC-GNN model, representative models covering various paradigms such as early fusion, graph neural networks, reconstruction compensation, contrastive learning, and cue optimization were selected. These included early classical methods as well as recent robust multimodal architectures. CCA, a linear statistical method, aims to find a set of basis vectors for two modalities to maximize the correlation between projected features. DCCA, a deep canonical correlation analysis, uses a neural network to perform a nonlinear transformation on each modality to maximize the correlation between modalities. DCCAE learns robust shared representations by jointly optimizing the autoencoder reconstruction loss and correlation loss. CPM-Net is specifically designed for handling incomplete multimodal data. AE, an unsupervised representation learning model, learns fusion representations based on a multimodal autoencoder framework; the model can utilize available modal information and attempt to reconstruct missing content to assist decision-making. The CRA cross-modal relationship attention model uses attention mechanisms to explicitly model fine-grained semantic associations and interactions between modalities; MCTN, based on a recurrent translation network, learns semantically consistent representations through a progressive translation process between modalities; MMIN multimodal interaction network focuses on modeling dynamic, bidirectional interactions between modalities, achieving information complementarity and filtering through gating mechanisms; the GCNet model uses graph convolutional networks to model the relationships between samples in multimodal data, treating samples as nodes in a graph, and aggregating neighbor information through graph convolution to enhance node (sample) representations; DiCMoR learns decoupled modal shared and private representations, reducing representation redundancy and improving generalization ability by separating shared and unique information; SDR-GNN provides a solution for viewpoint frequency domain reconstruction, capable of capturing certain global and structural features of data in the spectrum;

[0091] Tables 1 and 2 present a comparison of MAGIC-GNN's overall performance on IEMOCAP (quadratic / six-class classification) and CMU-MOSI and CMU-MOSEI (sentiment binary classification), with the missing data rate increased from 0.0 to 0.7 to simulate modal missing scenarios in real-world situations. Overall, MAGIC-GNN maintained stable and leading performance under different missing data rates, with a more significant advantage in high-missing-data scenarios, indicating that the model possesses stronger robustness and generalization ability under incomplete multimodal inputs.

[0092] Table 1 (IEMOCAP) shows that MAGIC-GNN achieves optimal or tied-optimal performance in both the four-class and six-class classification scenarios, and maintains a relatively gradual performance decline even with increasing missing information rates. On average, MAGIC-GNN achieves an average WAF1 of 80.00% on the IEMOCAP four-class classification task and 61.30% on the six-class classification task. This result indicates that multi-granular semantic modeling and generative completion mechanisms can continuously provide effective discriminative cues even with incomplete modal information, thereby mitigating performance degradation caused by missing information.

[0093] Table 2 (CMU-MOSI / CMU-MOSEI) also reports WAF1 / ACC (sentiment binary classification) under different missing rates. The overall trend shows that as the missing rate increases, all methods decline to varying degrees, but MAGIC-GNN maintains its lead under most missing rate conditions, indicating that the model also has stable adaptability to modality degradation in cross-dataset scenarios.

[0094] Table 1 compares performance at different missing rates on the IEMOCAP dataset. Report the WAF1 score (%). A higher WAF1 score indicates better performance. Best performance is highlighted in bold.

[0095]

[0096] Table 2 compares performance on the CMU-MOSI and CMU-MOSEI datasets at different missing rates. Report the WAF1 / ACC score (%). Best performance is highlighted in blue.

[0097]

[0098] Overall, the complete MAGIC-GNN achieved the best performance in both evaluation settings, with an average accuracy of 80.00% in the four-class classification task and 61.30% in the six-class classification task. Compared with baseline versions, the performance advantage of this method is more pronounced under high missing rate conditions, indicating a complementary and synergistic enhancement mechanism among the modules within the model.

[0099] Furthermore, to verify the impact of each component module of MAGIC-GNN on model performance, ablation experiments were conducted. The experiments were performed under two task settings: IEMOCAP (four-class and six-class classification), and model performance was tested at different missing rates (0.0–0.7). Continuous temporal segment missing data was not used as a supervision target for generative reconstruction, but only to apply regularization perturbations to the model encoder; the generative reconstruction module only performed supervised learning for random modal missing data. Therefore, the impact of different occlusion strategies on model performance in the ablation experiments mainly reflects the complementary role of random modal completion supervision and structured regularization perturbations under incomplete multimodal conditions. The global semantic modeling flow (global), local modeling flow (local), EHPA-based multi-level attention mechanism (w / o EHPA), channel attention mechanism (w / o Ca), and reconstruction module (w / o Gen) were removed sequentially from the model to analyze the effectiveness of each module. The experimental results are shown in Table 3.

[0100] After removing the global semantic flow (without global context), model performance showed a continuous decline (four-class: from 80.00% to 78.88%; six-class: from 61.30% to 59.13%). This difference widened as the missing data rate increased. This indicates that the global semantic flow plays a crucial role in modeling cross-turn dialogue dependencies, especially in providing key compensatory information when modality missing data is more severe.

[0101] Removing local semantic flow (without local) also leads to performance degradation, especially in six-class classification tasks with a larger number of emotion categories. This indicates that local features complement fine-grained emotion expression and, together with global flow, construct multi-level semantic modeling capabilities.

[0102] When EHPA was removed (without EHPA) and replaced with the original MatchingAttention, the model performance further decreased (four-class performance dropped to 77.97%). This indicates that EHPA can effectively enhance cross-modal semantic alignment and feature selection capabilities, helping to improve the model's sensitivity to subtle emotional differences;

[0103] Removing the channel attention module (w / o Ca) degrades model performance, especially under high missing value conditions. This indicates that the Ca module can dynamically model modal importance, thereby mitigating interference from redundant or noisy features.

[0104] The performance degradation was most significant after removing the reconstruction module (without Gen) (from 80.00% to 77.44% for four-class; from 61.30% to 59.63% for six-class). This trend was particularly pronounced in scenarios with high missing rates, indicating that generative completion plays a crucial role in improving the model's adaptability to missing modalities.

[0105] Removing supervised contrast loss leads to decreased intra-class compactness of the latent space and blurred inter-class boundaries.

[0106] Table 3 Ablation experiment results on the IEMOCAP (four-class and six-class) datasets.

[0107] Experimental results show that the performance of MAGIC-GNN stems from the synergistic effect of multiple core modules. The two-stream structure provides the model with multi-layer semantic understanding capabilities from local to global perspectives, while EHPA and channel attention further enhance information filtering and cross-modal alignment mechanisms. The reconstruction module plays a crucial role under modality missing conditions, significantly improving the model's recoverability to missing inputs. As the missing rate increases, the performance difference between the complete model and the ablation version widens, further validating the necessity and effectiveness of each module's design. The ablation experiments fully demonstrate the necessity and complementarity of the components in MAGIC-GNN.

[0108] Furthermore, to systematically evaluate the stability of different models under incomplete multimodal conditions, various modality missing ratios ranging from 0.0 to 0.7 were set on four datasets: IEMOCAPFour, IEMOCAPSix, CMUMOSI, and CMUMOSEI. The performance trends of regression and classification tasks as a function of the missing ratio were analyzed. A higher missing ratio indicates more masked modal information in the input, placing higher demands on the model's robustness.

[0109] like Figure 3 and Figure 4The figure shows a comparison of the weighted average F1 score (WAF1) of each model as the modality missing rate changes in the two classification tasks, IEMOCAPFour and IEMOCAPSix. Overall, the performance of all models decreases with increasing missing rate, but the degree of degradation varies significantly among different methods. On IEMOCAPSix, CRA shows the most significant performance degradation, declining rapidly after the missing rate exceeds 0.2, and its classification ability almost fails in high-missing-rate scenarios. The performance of GCNet and SDR-GNN also gradually decreases with increasing missing rate, but the decline is relatively gradual. MAGIC-GNN shows the most stable performance across the entire missing rate range, maintaining high recognition performance even at a missing rate of 0.7, indicating its strong tolerance for modality missing. On IEMOCAPFour, the overall trend is consistent with IEMOCAPSix, but the performance gap between models is more gradual. MAGIC-GNN continues to lead across all missing rates, further validating its robustness in handling modality missing in multi-class emotion recognition tasks. In contrast, CRA also shows a high sensitivity to missing modal information, with its performance deteriorating significantly as the proportion of missing information increases.

[0110] Therefore, under various datasets and task settings, traditional models often experience significant performance degradation as the degree of modality missing increases, especially when the missing rate is high. In contrast, MAGIC-GNN, with its multi-granular semantic modeling and structure-aware correction mechanism, demonstrates superior and more stable emotion recognition capabilities in various missing modal environments, indicating that this model can better adapt to complex and varied incomplete modal inputs in real-world scenarios.

[0111] To evaluate the performance of different methods on modality reconstruction tasks, verify the effectiveness of the model in multimodal dialogue data reconstruction tasks, and provide a reliable feature basis for downstream sentiment classification performance, the mean squared error (MSE) between the reconstructed data and the real data was calculated on four datasets and compared with existing state-of-the-art methods.

[0112] like Figure 5As shown, the MSE trends of different models across various modality missing proportions are illustrated on four datasets: IEMOCAPFour, IEMOCAPSix, CMUMOSI, and CMUMOSEI. Overall, as the missing rate gradually increases, the error of all models increases to varying degrees, but the rate and magnitude of this increase differ significantly among the models. Firstly, CRA and GCNet are most sensitive to modality missing; as the missing rate exceeds 0.3, their MSE rises rapidly, exhibiting a significant performance collapse in the high-missing region (0.6–0.7). In contrast, the error growth of SDR-GNN and MAGIC-GNN is more gradual, indicating that graph-based dialogue modeling can mitigate the interference caused by modality loss to some extent.

[0113] Experimental results show that the proposed MAGIC-GNN achieves the lowest MSE under all missing proportions, and its performance curve is consistently significantly lower than other models. Even under the extreme condition of a missing rate of 0.7, it still maintains a low error level, indicating that the model has better robustness in the face of multimodal degradation.

[0114] To investigate the model's performance stability under different hyperparameter settings, the performance trend of MAGIC-GNN was further evaluated under changes in two key hyperparameters: the proportion of structured missing data (mask_p) and the number of global attention heads (macro_head). Experiments were conducted with a fixed missing data rate of 0.3, and the F1 score was used as the evaluation metric. The results are as follows: Figure 6 and Figure 7 As shown.

[0115] Figure 6 The study demonstrates the impact of different mask_p settings on the model's F1 score when the modality missing rate is fixed at 0.3. It can be observed that the model is quite sensitive to changes in mask_p, with performance fluctuating significantly as the masking ratio increases from 0.00 to 0.30. The model achieves the highest F1 score when mask_p is approximately 0.10–0.15, indicating that moderate structured missing data simulation can effectively improve the model's robust learning ability during training. However, when mask_p is too small (e.g., 0.00), the model barely encounters the missing data, resulting in insufficient generalization ability; when mask_p is too large (e.g., 0.25–0.30), too much information is masked, affecting the learning of effective features. This suggests that mask_p needs to strike a balance between "simulating real missing data" and retaining sufficient effective information.

[0116] like Figure 7As shown, the F1 score of the model is presented under different numbers of global attention heads (macro_head). Overall, the number of macro_heads is highly correlated with the model's global discourse modeling ability, and its changes have a significant impact on performance.

[0117] The results show that the model performs best when macro_head = 4, indicating that the model can effectively capture long-range dependency patterns in dialogue. Fewer heads (e.g., 1–2) limit the model's expressive power, making it difficult to fully characterize diverse global semantics. Conversely, too many heads (e.g., 5–6) may introduce redundancy or noise, leading to unstable model learning. This suggests that the number of global attention heads needs to be set appropriately to achieve the best balance between expressive power and training stability.

[0118] The sensitivity analysis results of the two types of hyperparameters above show that mask_p needs to be set in a moderate range (0.10–0.15) to achieve a reasonable balance between simulating modal loss and preserving effective information; while macro_head is recommended to be set to 4 to achieve the best trade-off between global discourse modeling ability and training stability. Parameter configurations that are too strong or too weak will weaken the model's generalization ability under modal loss conditions. Therefore, while the model possesses good modal loss robustness based on structured loss simulation and multi-granularity attention mechanisms, reasonable hyperparameter selection is equally crucial for further improving the model's upper limit performance.

[0119] To further validate the model's sentiment reasoning ability under incomplete multimodal conditions, representative real-world dialogue segments were selected for case analysis in both the four-class and six-class classification settings of the IEMOCAP dataset, such as... Figure 8 As shown.

[0120] exist Figure 8In the dialogue shown, the target utterance in the example on the left can be considered a key statement with significant emotional shifts. As the dialogue progresses, the speaker's emotion suddenly shifts from a relatively neutral communicative state to obvious anger, exhibiting an emotional change in semantic intensity inconsistent with the surrounding utterances. However, due to the absence of the corresponding visual modality and the lack of obvious emotional trigger words in the text itself, most models perform poorly in predicting this target utterance, generally misclassifying it as Neutral. MMIN, focusing primarily on emotion modeling at the single utterance level, does not rely on contextual information for its predictions, thus correctly identifying anger in this scenario. In contrast, other models that rely on contextual feature propagation or aggregation tend to weaken the significant emotional changes contained in the utterance during the transmission of emotional signals. MAGIC-GNN, through a multi-granularity semantic modeling mechanism, adaptively fuses global semantics at the dialogue level with fine-grained semantics at the utterance level, effectively preserving the key emotional information in the target utterance and achieving accurate prediction.

[0121] Figure 8 In the example on the right, the emotion of the target utterance gradually accumulates through multiple rounds of communication disruption, ultimately manifesting as a Frustrated emotion. Because this emotion is not an instantaneous outburst but rather implicit in the contextual evolution, most models struggle to capture its potential changing trends, leading to predictions biased towards Neutral or Sad. In contrast, MAGIC-GNN, combined with a structure-aware graph neural network, constrains the propagation of contextual information, effectively modeling the emotion evolution process. It accurately identifies the true emotion even under incomplete multimodal conditions, validating its robustness in complex scenarios.

[0122] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any changes, modifications, substitutions, integrations, and parameter changes made to these embodiments within the spirit and principles of the present invention, without departing from the principles and spirit of the present invention, through conventional substitutions or to achieve the same function, fall within the scope of protection of the present invention.

Claims

1. A robust multimodal emotion recognition method integrating multi-granularity semantic modeling and structure awareness, characterized in that, include: Acquire text, speech, and visual trimodal input data for multi-turn dialogues, wherein the multi-turn dialogues include utterances and corresponding speaker identities; A missing mask is generated by simulating a structured modal missing data pattern, and the three-modal input data is occluded to obtain the occluded multimodal input. Feature extraction and sequence encoding are performed on the occluded multimodal input to obtain the discourse-level basic representation; A multi-granularity semantic modeling mechanism is constructed to perform dual-granularity semantic learning on the discourse-level basic representation, and an enhanced semantic representation is obtained by combining it with an adaptive gating fusion mechanism; The multi-turn dialogue is modeled as a structural graph containing temporal dependencies and speaker interaction relationships. The enhanced semantic representation is then subjected to structure-aware semantic correction based on a graph neural network to obtain a structure-enhanced representation. The structural enhancement representation is filled with missing modalities by a generative semantic reconstruction module to generate a reconstructed semantic representation; A multi-objective optimization function is constructed by combining sentiment classification loss, generative reconstruction loss, and supervised contrastive learning loss. The model is then jointly trained and outputs the sentiment recognition result for each utterance.

2. The method according to claim 1, characterized in that, The structured modal loss simulation strategy includes a random modal loss mechanism and a time-segment level loss mechanism. The random modal loss mechanism randomly occludes the feature channels of any one or more modalities in text, speech, and vision at the discourse level with a preset probability to simulate sudden modal failure. The time-segment level loss mechanism completely masks all features of a certain modality within a continuous preset number of discourse intervals to simulate persistent sensor failure or long-term signal degradation. The missing mask It is generated through the combined action of two types of missing mechanisms, and the formula is: ,in For the missing random modal mask, For time-segment level missing masks, This indicates a missing mode at the corresponding location. The modality is available at the corresponding position; The original trimodal input data is masked based on the generated missing mask to obtain the masked multimodal input. The formula is: , where ⊙ represents element-wise multiplication. This is the original trimodal input data matrix.

3. The method according to claim 1, characterized in that, In the feature extraction process, pre-trained text encoders, speech feature extractors, and visual feature extractors are used to perform feature transformation on the original data of each modality after occlusion, so as to obtain text modality features, speech modality features, and visual modality features. Each modality feature is mapped to a feature space of the same dimension. By concatenating the text, speech, and visual modal features of the same discourse along the channel dimension, the multimodal fusion features of the discourse are obtained. The multimodal fusion features of all utterances are arranged in dialogue temporal order and input into a bidirectional gated recurrent unit (Bi-GRU) for sequence encoding to capture the basic temporal context information between utterances. The output is a unified-dimensional utterance-level basic representation for each utterance, as shown in the formula: ,in For the first The basic representation of a discourse. The hidden layer dimension is adaptively configured based on the dataset complexity.

4. The method according to claim 1, characterized in that, The multi-granularity semantic modeling mechanism is constructed through global semantic flow and local semantic flow. The construction of the global semantic flow specifically involves: linearly mapping the utterance-level basic representation sequence to adjust the feature dimension to a preset global semantic modeling dimension; using a self-attention-based encoder to perform global context modeling on the adjusted sequence; calculating the attention weights between all utterances; aggregating long-range dependencies across utterances; focusing on characterizing the overall emotional evolution trend of the dialogue; and outputting a coarse-grained semantic representation containing global context information, as shown in the formula: ,in This is a discourse-level basic representation sequence. A function for global semantic modeling; The construction of the local semantic stream specifically involves: using a sliding window of fixed size to locally truncate the utterance-level basic representation sequence, with the window size adaptively set according to the average length of utterances in the dialogue; for each local sequence within the window, a bidirectional recurrent neural network is used to model the short-term temporal dependencies between adjacent utterances, and then a one-dimensional convolution operation is used to extract the emotional change patterns within the local time window, highlighting local semantic differences and outputting a fine-grained semantic representation, as shown in the formula: ,in A function for modeling local semantics.

5. The method according to claim 4, characterized in that, The adaptive gating fusion mechanism obtains enhanced semantic representation, specifically by: transforming the utterance-level basic representation... Global semantic representation and local semantic representation Each is mapped to a fusion space of the same dimension through a linear transformation; The three mapped representations are subjected to dialogue-level feature aggregation. The global feature vector of the entire dialogue is obtained through global average pooling or max pooling. The calculation formula is as follows: ,in This is a dialogue-level feature aggregation operation. This is the global average pooling function; The aggregated global feature vectors are input into a learnable gating function. The fusion weights for the three branches are generated using the following formula: ,in The weights corresponding to the basic representation at the discourse level. The weights corresponding to the global semantic representation, The weights corresponding to the local semantic representation; Based on the generated fusion weights, the three representations are weighted and summed to obtain the enhanced semantic representation. The formula is: The fusion weights are estimated at the entire dialogue level and shared over time.

6. The method according to claim 1, characterized in that, The acquisition of the structurally enhanced representation specifically involves treating each utterance in a multi-turn dialogue as a node in a graph, with the node features being the enhanced semantic representation. ; To characterize the relationships between nodes, temporal adjacency edges and speaker interaction edges are constructed, forming a structure graph containing nodes and two types of edges. ,in For a set of nodes, Let it be the set of edges; Structure-aware semantic correction is performed by aggregating the features of each node with those of its neighboring nodes using a message passing mechanism in a graph neural network, and then calculating the structure-induced correction signal. The formula is as follows: ,in For the first Structural correction signals for each node For structure-aware mapping functions; A lightweight channel aggregation module is introduced to optimize the channel dimension and correct the signal structure. By using a channel-by-channel weighting method, amplified noise channels during graph propagation are suppressed, while retaining semantic channels crucial for emotion discrimination. The formula is as follows: ,in This is the corrected signal after channel optimization. This represents the channel-level weight vector.

7. The method according to claim 6, characterized in that, The enhanced structure correction signal is refined using an enhanced hybrid parallel attention mechanism, including: optimizing the channel-optimized correction signal. Input a bidirectional gated recurrent unit to perform temporal feature enhancement, and obtain the enhanced temporal feature sequence; Self-attention computation is performed on the enhanced temporal feature sequence to capture the dependencies within the sequence, resulting in self-attention enhanced features. These self-attention enhanced features are used as a memory base, along with the original enhanced semantic representation of each node. As the query vector, cross-attention is calculated using the following formula: ,in For cross-attention output features, These are self-attention and cross-attention functions, respectively. Cross-attention output features Compared with the original enhanced semantic representation Residual connections are made, and feature transformation is performed through a feedforward network to obtain refined structure-enhanced features. The calculation formula is as follows: The refined structure-enhanced features are projected back into a unified semantic space via linear mapping to obtain the final structure-enhanced representation, as shown in the formula: ,in It is a learnable linear transformation parameter matrix.

8. The method according to claim 1, characterized in that, The generative semantic reconstruction module employs a dual-path memory architecture with short-term memory and long-term memory paths, modeling short-term local patterns and long-term global dependencies respectively. The short-term memory path enhances the structural representation through a feedforward network. Feature transformation is performed to directly capture local contextual information without relying on external memory. The formula is as follows: ,in Output features for short-term memory paths; Long-term memory pathways are represented by the aforementioned structural enhancements. As the query vector, the original multimodal feature sequence M is used as a memory. Global dependency information related to the current representation is extracted from the memory through a cross-attention mechanism, as shown in the formula: ,in , , These are linear transformation parameter matrices for the query, key, and value, respectively. For attention feature dimension, Output features for long-term memory paths; Output features of short-term memory paths Output features of long-term memory pathways Element-wise addition is performed, followed by a linear transformation to generate the final reconstructed semantic representation. The calculation formula is as follows: ,in The linear transformation parameter matrix is ​​learnable. To reconstruct the semantic representation.

9. The method according to claim 1, characterized in that, The calculation of the generative reconstruction loss specifically involves generating a corresponding modality missing mask for each modality, including text, speech, and vision. ,in Indicate the modality type; for each modality, extract the original features. With reconstruction features Through modal missing mask The locations of missing random modalities are identified; the error between the original features and the reconstructed features at the missing locations is calculated using the L2 norm, with the following formula: ,in To retain only the feature differences at the missing locations, It is the square of the L2 norm.

10. The method according to claim 1, characterized in that, The multi-objective optimization function includes three loss terms, each corresponding to a different training objective: sentiment classification loss. To optimize the emotion recognition accuracy of the model, cross-entropy loss is used for classification tasks, and mean squared error loss is used for regression tasks. The formula is as follows: ,in For the first Predicting sentiment from a single statement As a tag for genuine emotions, For loss calculation function, Total number of discourses; generative reconstruction loss Optimize the completion effect of missing modalities; supervised contrastive learning loss. To enhance the intra-class compactness and inter-class separability of the latent representation space, positive and negative sample pairs are constructed only within the same speaker. Discourses with the same sentiment label are represented as positive sample pairs, and discourses with different sentiment labels are represented as negative sample pairs. The formula is as follows: ,in , , These are the normalized discourse representations, For temperature coefficient, To be consistent with the sample A set of positive samples with the same label For the candidate sample set, The cosine similarity calculation function is used; the multi-objective optimization function fuses three loss terms through dynamic weights, and the formula is: in , These are dynamic weights.

Images