Double-branch hypergraph emotion recognition system fusing topology and semantic information

Abstract

The invention discloses a dual-branch hypergraph emotion recognition system fusing topology and semantic information, and belongs to the field of multi-mode dialogue emotion recognition. The invention aims to solve the problems of semantic blindness and message passing mechanism coarseness of hypergraph structure construction in the prior art. Comprising the following steps: extracting initial features by adopting a multi-modal feature extraction module, and forming unified frame-level feature representation by adopting a multi-modal feature alignment module; inputting the aligned multi-modal features into two parallel hypergraph branches for enhancement, and modeling a high-order relationship between data from different dimensions; calculating a feature centroid for each hyperedge by adopting a hypergraph convolution mechanism based on a feature center, and then dynamically calculating the weight of each hyperedge according to the distance between each vertex in the hyperedge and the centroid in a feature space; and performing adaptive weighted fusion on the enhanced features output by the topological branch and the semantic branch through a lightweight gating fusion mechanism to generate a final unified representation. According to the invention, more discriminative emotional characterization can be obtained.

Description

Technical Field

[0001] This invention relates to a bi-branch hypergraph emotion recognition system that integrates topological and semantic information, belonging to the field of multimodal dialogue emotion recognition. Background Technology

[0002] With the development of wearable sensors and affective computing technologies, emotion recognition is expanding from traditional text and speech modalities to encompass a wealth of physiological data. The fusion of external physical signals (such as facial expressions and voice tone) with internal physiological signals (such as electroencephalograms (EEGs) and electrocardiograms (ECGs)) provides unprecedented opportunities for a deeper and more reliable understanding of human emotional states. By collaboratively analyzing these heterogeneous data streams, physical-physiological emotion recognition systems can capture a holistic view of emotional responses, achieving more robust and accurate predictions than single-modal methods.

[0003] However, existing technologies still face significant challenges in effectively modeling complex high-order relationships in such heterogeneous data: (1) Limitations in model expressive power: Traditional deep learning models, such as those based on Transformer architecture, are adept at capturing pairwise relationships within sequences, but their attention mechanisms are essentially limited to modeling the interaction between two elements, making it difficult to capture group-wide high-order associations across different samples or subjects. This limits the model's ability to reveal shared patterns in emotional expression. (2) Structural dilemmas in hypergraph construction: Hypergraph neural networks have emerged to model high-order associations beyond pairwise relationships. However, most existing hypergraph construction methods are purely data-driven (e.g., based on K-nearest neighbors), discovering the inherent topological structure of data through feature similarity. This method is blind to high-level semantic information (e.g., sample category, modality source). That is, there is a lack of a framework that can synergistically integrate data-driven topological structures with knowledge-driven semantic structures. (3) Feature-agnostic aggregation mechanism: When traditional hypergraph convolutions perform message passing, they usually treat all vertices within the hyperedge equally, ignoring the differences in the contribution of different vertices to the core concept represented by the hyperedge. This coarse-grained aggregation method, characterized by "feature agnosticness," leads to inaccurate information dissemination, hindering the model from learning refined and discriminative feature representations.

[0004] To address the aforementioned challenges, there is an urgent need for a sentiment recognition method that can synergistically integrate data-driven local structures with knowledge-driven global semantics. Summary of the Invention

[0005] To address the semantic blindness and coarse-grained message passing mechanisms in existing hypergraph construction technologies, this invention provides a dual-branch hypergraph emotion recognition system that integrates topological and semantic information.

[0006] This invention provides a dual-branch hypergraph emotion recognition system that integrates topological and semantic information, comprising:

[0007] A multimodal feature extraction module is used to extract features from raw audio waveform data. Extracting initial audio features From initial visual data Extracting initial visual features ; from raw multichannel EEG signals Initial EEG features were extracted ;

[0008] A multimodal feature alignment module is used to align initial audio features. and initial EEG characteristics Extended replication along the time dimension yields the same visual features as the initial ones. Aligned audio features after frame-level alignment and aligned EEG characteristics ;

[0009] A dual-branch hypergraph enhancement module is used to fuse initial visual features. Aligned audio features and aligned EEG characteristics , obtain fusion features ; All fusion features in the current batch As vertices, a topological hypergraph is constructed based on the K-nearest neighbor algorithm, and a weighted correlation matrix is ​​calculated based on feature centrality. All fusion features With weighted correlation matrix Information is transferred and updated through hypergraph hybrid convolution, resulting in topology-aware enhanced features. Simultaneously based on initial visual features Aligned audio features and aligned EEG characteristics Generate modality-aware features ; All modality-aware features in the current batch As vertices, a hierarchical hypergraph is constructed based on semantic hyperedges and category superposition, and a weighted correlation matrix is ​​calculated based on feature centrality. All modality-aware features With weighted correlation matrix Information transfer and feature learning are performed through hypergraph hybrid convolution, resulting in semantically aware modality-level enhanced features. ;

[0010] The gated fusion module is used to integrate topology-aware augmented features. Modal-level augmentation features with semantic awareness Perform adaptive weighted fusion to obtain weighted fusion features. ;

[0011] Linear layers are used to employ two fully connected layers based on weighted fusion features. Generate sentiment category prediction results.

[0012] The bi-branch hypergraph emotion recognition system according to the present invention, which integrates topological and semantic information, further includes:

[0013] The joint optimization module is used to combine supervised classification loss and self-supervised contrastive loss during system training to achieve collaborative training of the system.

[0014] According to the dual-branch hypergraph emotion recognition system that integrates topological and semantic information of the present invention, the multimodal feature extraction module uses an AST feature extractor to extract the original audio waveform data through short-time Fourier transform, Mel filter bank application, logarithmic compression, and time padding. Convert to log-Mel spectrum Then, the audio Transformer model is used to generate a log-Mel spectrum. Initial audio features were extracted. ; raw multichannel EEG signals The data tensor is obtained by downsampling through a decimation operation. The EEG Transformer model is used to generate data tensors. Initial EEG features were extracted. .

[0015] According to the bi-branch hypergraph emotion recognition system of the present invention, which integrates topological and semantic information, the bi-branch hypergraph enhancement module obtains fused features. The method is as follows:

[0016] Initial visual features Aligned audio features and aligned EEG characteristics The features are concatenated column-wise and then cross-feature interaction is performed through a multi-head self-attention layer to obtain fused features. ;

[0017] Obtain modal sensing features The method is as follows: based on initial visual features Aligned audio features and aligned EEG characteristics By projecting onto a unified feature dimension through a linear layer, projected visual feature vectors, projected audio feature vectors, and projected EEG feature vectors are obtained. These are then added element-wise to their corresponding learnable modality embedding vectors to obtain visual modality perception features. Audio modality perception features and EEG modal perception features Then, visual modal perception features Audio modality perception features and EEG modal perception features Concatenate the data line by line to obtain modality-aware features. .

[0018] According to the bi-branch hypergraph emotion recognition system that integrates topological and semantic information of the present invention, enhanced features with topological awareness are obtained. The method is as follows:

[0019] All fusion features in the current batch For each vertex, the K-nearest neighbor algorithm is used to determine K-1 neighbor vertices, forming a hyperedge to obtain a topological hypergraph; all fused features within each hyperedge of the topological hypergraph are then processed. Calculate the mean to obtain the hyperedge centroid; then fuse the features that minimize the Euclidean distance from the current hyperedge centroid within the current hyperedge. As the center vertex of the hyperedge; then calculate all fused features within the current hyperedge. The Euclidean distance to the center vertex of the hyperedge is used to determine each fused feature based on the reciprocal of the Euclidean distance. The association weights with the current hyperedge are determined; by integrating all vertices and their association weights, a weighted association matrix is ​​obtained. ;

[0020] Weighted correlation matrix With all fusion features Hypergraph hybrid convolution is used for information transfer and updating to obtain topology-aware enhanced features. .

[0021] According to the bi-branch hypergraph emotion recognition system that integrates topological and semantic information of the present invention, semantically perceptive modality-level enhanced features are obtained. The method is as follows:

[0022] Features perceived by each modality Initial visual features Aligned audio features and aligned EEG characteristics Connections form semantic hyperedges; these are constructed during the training phase, allowing modal-aware features belonging to the same emotion category to be grouped together. Connect them to form a category hyperedge; obtain a hierarchical hypergraph; based on the hierarchical hypergraph, use the method to obtain a weighted correlation matrix. The same method is used to obtain the weighted correlation matrix. All modality-aware features With weighted correlation matrix Information transfer and feature learning are performed through hypergraph hybrid convolution, resulting in semantically aware modality-level enhanced features. .

[0023] According to the bi-branch hypergraph emotion recognition system that integrates topological and semantic information of the present invention, in the gated fusion module, modality-level enhancement features... The three modalities are average pooled to obtain sample-level semantic enhancement features. .

[0024] According to the bi-branch hypergraph emotion recognition system that integrates topological and semantic information of the present invention, in the gated fusion module, based on sample-level fusion weights... Enhanced features of topology awareness and sample-level semantic enhancement features Perform weighted fusion to obtain weighted fusion features. :

[0025] ,

[0026] In the formula Represents the Sigmoid function. For linear layer weights, This is the linear layer bias value. This indicates element-wise multiplication; This indicates column-based operation.

[0027] According to the bi-branch hypergraph emotion recognition system that integrates topological and semantic information of the present invention, the emotion category prediction result is represented as... :

[0028] ,

[0029] In the formula This is the weight matrix of the first fully connected layer in the linear layer. This is the weight matrix of the second fully connected layer in the linear layer. This is the bias vector of the first fully connected layer in the linear layer. This is the bias vector of the second fully connected layer in the linear layer.

[0030] According to the bi-branch hypergraph emotion recognition system that integrates topological and semantic information of the present invention, in the joint optimization module, the emotion category prediction result is used as the basis for the prediction. Compared to real labels that have undergone label smoothing Calculate supervised classification loss ;

[0031] Total loss function during system training for:

[0032] ,

[0033] In the formula To balance hyperparameters, For self-monitored comparison of losses.

[0034] The beneficial effects of this invention are as follows: This invention jointly models external physical signals (such as audio and vision) and internal physiological signals (such as electroencephalogram), synergistically integrates data-driven local structures and knowledge-driven global semantics, and achieves robust dialogue emotion recognition through a holistic hypergraph framework that executes feature-aware message passing.

[0035] This invention employs a dual-branch hypergraph approach (DBHA) that integrates topological and semantic information for dialogue emotion recognition based on physical-physiological signals. It utilizes a unified framework to collaboratively leverage the intrinsic geometric topology and extrinsic semantic knowledge of the data, and optimizes information flow through a novel feature-center convolution mechanism, ultimately learning a more discriminative emotion representation.

[0036] This invention solves the structural dilemma of hypergraph construction through an innovative dual-branch hypergraph architecture. By coordinating data-driven topological branches (discovering the intrinsic geometry of data) and knowledge-driven semantic branches (injecting high-level conceptual relationships), it creates a more comprehensive hypergraph structure that can both reflect data distribution and accept semantic guidance.

[0037] This invention employs a knowledge-injected hyperedge construction method. By constructing semantic hyperedges and category hyperedges, high-level knowledge such as modal identity and real labels is directly injected into the hypergraph structure, providing the model with powerful and interpretable supervision information and guiding it to learn more discriminative feature representations.

[0038] This invention designs an association matrix based on feature centers, making the message passing process feature-aware. By dynamically weighting the distance between vertices and hyperedge centroids, a finer and more accurate information flow is achieved, overcoming the coarse-grained defects of traditional hypergraph aggregation.

[0039] This invention employs an effective joint optimization strategy, combining label smoothing cross-entropy loss and multimodal contrastive loss, which not only ensures the accuracy of the classification task but also significantly enhances the discriminative power and robustness of the learned representations by forcing multimodal features to align semantically. Attached Figure Description

[0040] Figure 1 The weighted correlation matrix is ​​obtained in the dual-branch hypergraph emotion recognition system that integrates topological and semantic information as described in this invention. and Schematic diagram;

[0041] Figure 2 This is a schematic diagram of the gating fusion module and the joint optimization module;

[0042] Figure 3This is a flowchart of the construction process of the topological hypergraph;

[0043] Figure 4 This is a flowchart of the construction process of a hierarchical hypergraph. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Specific Implementation Method 1: Combination Figures 1 to 4 As shown, this invention provides a dual-branch hypergraph emotion recognition system that integrates topological and semantic information, including:

[0046] A multimodal feature extraction module is used to extract features from raw audio waveform data. Extracting initial audio features From initial visual data Extracting initial visual features ; from raw multichannel EEG signals Initial EEG features were extracted ;

[0047] A multimodal feature alignment module is used to align initial audio features. and initial EEG characteristics Extended replication along the time dimension yields the same visual features as the initial ones. Aligned audio features after frame-level alignment and aligned EEG characteristics ;

[0048] A dual-branch hypergraph enhancement module is used to fuse initial visual features. Aligned audio features and aligned EEG characteristics , obtain fusion features ; All fusion features in the current batch As vertices, a topological hypergraph is constructed based on the K-nearest neighbor algorithm, and a weighted correlation matrix is ​​calculated based on feature centrality. All fusion features With weighted correlation matrix Information is transferred and updated through hypergraph hybrid convolution, resulting in topology-aware enhanced features. Simultaneously based on initial visual features Aligned audio features and aligned EEG characteristics Generate modality-aware features ; All modality-aware features in the current batch As vertices, a hierarchical hypergraph is constructed based on semantic hyperedges and category superposition, and a weighted correlation matrix is ​​calculated based on feature centrality. All modality-aware features With weighted correlation matrix Information transfer and feature learning are performed through hypergraph hybrid convolution, resulting in semantically aware modality-level enhanced features. ;

[0049] The gated fusion module is used to integrate topology-aware augmented features. Modal-level augmentation features with semantic awareness Perform adaptive weighted fusion to obtain weighted fusion features. ;

[0050] Linear layers are used to employ two fully connected layers based on weighted fusion features. Generate sentiment category prediction results.

[0051] Multimodal feature-level synchronization: due to visual features It's frame-level, while audio features... and EEG characteristics It is sample-level; to achieve synchronous fusion, a multimodal feature alignment module is used with initial visual features at the frame level. The temporal resolution (25 frames per sample) is used as the alignment benchmark for all modalities. and The copy is extended along the time dimension to match the alignment reference, thereby generating frame-aligned audio features. and EEG characteristics .

[0052] This embodiment also includes:

[0053] The joint optimization module is used to combine supervised classification loss and self-supervised contrastive loss during system training to achieve collaborative training of the system.

[0054] This implementation proposes a bi-branch hypergraph network for emotion recognition in physical-physiological signal dialogue, comprising four core modules: modality feature extraction, bi-branch hypergraph enhancement, gated branch fusion, and joint optimization supervision. The specific emotion recognition process is as follows:

[0055] I. Modal Feature Extraction. First, for the three modalities of physical signals (audio, visual) and physiological signals (EEG), initial features are extracted using independent pre-trained models (such as Transformer-based audio and visual models and EEGTransformer). To ensure the synchronization of subsequent fusion analysis, features at different temporal resolutions are temporally aligned to form a unified frame-level feature representation.

[0056] II. Bi-branch Hypergraph Augmentation. Aligned multimodal features are input into two parallel hypergraph branches for augmentation, modeling higher-order relationships between data from different dimensions. The data-driven topological hypergraph branch aims to autonomously discover the inherent geometric topology of the data. It fuses multimodal features and dynamically constructs a hypergraph based on the K-nearest neighbor algorithm within each training batch. This approach captures the local nearest neighbor structure between samples in the feature space. The knowledge-driven semantic hypergraph branch aims to inject high-level, human-interpretable knowledge into the hypergraph structure. It achieves this by constructing two levels of hyperedges: semantic hyperedges, connecting vertices of different modalities (audio, visual, EEG) of the same data sample to model cross-modal inherent consistency; and category hyperedges, utilizing ground-truth sentiment labels during training to connect all vertices belonging to the same sentiment category, providing strong supervision signals to the model.

[0057] III. Feature-Center Hypergraph Convolution. To address the coarse-grained aggregation problem in traditional hypergraph convolution, a feature-center-based hypergraph convolution mechanism is proposed and applied to the two branches mentioned above. This mechanism first calculates a feature centroid for each hyperedge, and then dynamically calculates its weights based on the distances between each vertex within the hyperedge and the centroid in the feature space. When constructing the association matrix, binary values ​​are no longer used; instead, this feature-center-based weighting is employed. This allows the message passing process to be aware of feature distribution, enabling more representative vertices to play a greater role in information aggregation.

[0058] IV. Gated Branch Fusion and Joint Optimization. The enhanced features output from the topological and semantic branches are adaptively weighted and fused through a lightweight gated fusion mechanism to generate the final unified representation. Finally, a joint optimization strategy is used to train the entire network end-to-end, which combines label smoothing cross-entropy loss for the main classification task with multimodal contrastive loss for enhancing the discriminativeness of the representation.

[0059] The multimodal feature extraction module is designed to extract high-quality initial representations from raw physical signals (audio, visual) and physiological signals (EEG).

[0060] Furthermore, in the multimodal feature extraction module, an AST feature extractor is used to extract the original audio waveform data through short-time Fourier transform, Mel filter bank application, logarithmic compression, and time padding. Convert to log-Mel spectrum Then, the audio Transformer model is used to generate a log-Mel spectrum. Initial audio features were extracted. ; raw multichannel EEG signals The data tensor is obtained by downsampling through a decimation operation. The EEG Transformer model is used to generate data tensors. Initial EEG features were extracted. .

[0061] Log-Mel spectrum The input is fed into a pre-trained audio Transformer model, which is trained for unimodal classification on the EAV (audio) dataset. Finally, with all parameters of the model frozen, a 768-dimensional intermediate representation vector is extracted from the hidden layer before the classifier and used as the initial audio features encapsulating global acoustic semantic information. .

[0062] Initial visual feature extraction: For the original video clip, segmentation and frame sampling are first performed to obtain initial visual data. Then will The input is fed into a pre-trained visual Transformer model, which is trained on the EAV (Vision) dataset for frame-level unimodal classification. After training, all parameters of the model are frozen, and a 768-dimensional intermediate representation vector is extracted from its final encoder layer for each frame. Finally, the intermediate representation vectors of all frames are aggregated to obtain the initial visual features encapsulating the visual semantics of a single frame. .

[0063] Initial EEG Feature Extraction: For raw multichannel EEG signals First, through the extraction operation, Downsampling to 100Hz yields the data tensor used for analysis. Subsequently, The input is fed into a custom EEG Transformer model, which is trained on the EAV (electroencephalography) dataset for unimodal classification. After training, all parameters of the model are frozen, and a 60-dimensional intermediate representation vector is extracted from its global pooling layer as the initial EEG features. .

[0064] A bi-branch hypergraph enhancement module is constructed to model higher-order relationships in data from both topological and semantic dimensions.

[0065] To address the coarse-grained problem of traditional hypergraph message passing, this implementation defines a weighted hypergraph association matrix that reflects the importance of vertices in the local structure. This method abandons the traditional binary membership relationship and instead adopts a weighting mechanism dynamically determined by the distribution of data features. Specifically:

[0066] 1. Determine the center vertex of the hyperedge: For any hyperedge in the hypergraph structure First, calculate the mean of the eigenvectors corresponding to all vertices within the hyperedge to obtain the eigencentroid of the hyperedge. Second, within the hyperedge, determine the vertex whose eigenvector has the smallest Euclidean distance to the eigencentroid, and define it as the central vertex of the hyperedge. :

[0067] ,

[0068] In the formula The superscript of the node closest to the feature centroid among the nodes contained in the hyperedge is the label of the center node corresponding to the hyperedge; This is the feature matrix corresponding to the central node.

[0069] 2. Calculate the vertex-hyperedge association weight: For vertex-hyperedges... any vertex Calculate the features of this vertex. With the determined central vertex features The Euclidean distance between them is used to determine the vertices, and the reciprocal of this distance is used to define the vertices. With super-edge Association weight between :

[0070] .

[0071] This weighting mechanism ensures that vertices whose features are closer to the hyperedge core receive a higher contribution during information transmission. Finally, by integrating the association weights of all vertices and hyperedges, a weighted association matrix can be constructed. .

[0072] Construct a data-driven topological hypergraph branch that can autonomously discover the latent geometric topology of the original multimodal feature space without prior knowledge. For example... Figure 3 As shown, its contribution lies in providing a fundamental understanding of the inherent clustering structure of data. Specifically:

[0073] Feature preparation: In the dual-branch hypergraph enhancement module, fused features are obtained. The method is as follows:

[0074] Initial visual features Aligned audio features and aligned EEG characteristics The features are concatenated column-wise and then cross-feature interaction is performed through a multi-head self-attention layer to obtain fused features. ;

[0075] ;

[0076] In the formula This represents a multi-head self-attention mechanism.

[0077] As a supplement to the topological branch, a knowledge-driven semantic hypergraph branch is constructed, which innovatively injects high-level, human-interpretable knowledge directly into the hypergraph structure. For example... Figure 4 As shown, its purpose is to guide the model's learning process through explicit semantic information and supervision signals.

[0078] Modality-aware feature generation: Obtaining modality-aware features The method is as follows: to preserve modal identity information, the initial visual features will be used... Aligned audio features and aligned EEG characteristics By projecting independent linear layers onto a unified feature dimension, projected visual feature vectors, projected audio feature vectors, and projected EEG feature vectors are obtained. These are then compared with their corresponding learnable modality embedding vectors. Element-wise addition is performed so that each feature vector explicitly carries the identity information of its original modality, resulting in visual modality perception features. Audio modality perception features and EEG modal perception features ;

[0079] ,

[0080] In the formula These are the weights used to perform a linear transformation on the visual features. These are the weights for performing a linear transformation on the audio features. These are the weights used to perform a linear transformation on the EEG features. This is the bias value used to perform a linear transformation on the visual features. This is the bias value used to perform a linear transformation on the audio features. The bias value used for linear transformation of EEG features. The modal embedding vector is a visual feature. For the modal embedding vector of audio features, This represents the modal embedding vector of EEG features;

[0081] Then, visual modal perception features Audio modality perception features and EEG modal perception features Concatenate the data line by line to obtain modality-aware features. :

[0082] ,

[0083] In the formula This indicates that the data is concatenated along the row dimension.

[0084] Obtain topology-aware enhanced features The method is as follows:

[0085] All fusion features in the current batch For each vertex, the K-nearest neighbor algorithm is used to determine K-1 neighbor vertices, forming a hyperedge to obtain a topological hypergraph; all fused features within each hyperedge of the topological hypergraph are then processed. Calculate the mean to obtain the hyperedge centroid; then fuse the features that minimize the Euclidean distance from the current hyperedge centroid within the current hyperedge. As the center vertex of the hyperedge; then calculate all fused features within the current hyperedge. The Euclidean distance to the center vertex of the hyperedge is used to determine each fused feature based on the reciprocal of the Euclidean distance. The association weights with the current hyperedge are determined; by integrating all vertices and their association weights, a weighted association matrix is ​​obtained. ;

[0086] Dynamic Hypergraph Construction: Within each training batch, the samples in that batch are considered as vertices of the hypergraph. For each vertex, the K-nearest neighbor algorithm is used in the feature space. Find the K-1 nearest neighbors to it, and together they form a superedge. :

[0087] ,

[0088] In the formula Let j be the j-th hyperedge in the topological hypergraph. Let be the characteristic matrix corresponding to the center node of the j-th hyperedge of the topological hypergraph. Let be the feature matrix corresponding to the K-1 nearest neighboring nodes on this hyperedge. The number of samples included in each training batch.

[0089] Weighted correlation matrix With all fusion features Hypergraph hybrid convolution (including spectral and spatial domain convolution) is used for information transfer and updating to obtain topology-aware enhanced features. .

[0090] ,

[0091] In the formula For fusion features The node-enhanced feature matrix after spectral domain hypergraph convolution This is the node normalization matrix of the topological hypergraph. Let be the hyperedge weight matrix of the topological hypergraph. Let be the hyperedge normalization matrix of the topological hypergraph. For the learnable parameters of the hypergraph convolutional layer in the spectral domain, For fusion features The hyperedge feature matrix obtained during the convolution process of the hypergraph in the spatial domain. For activation function, For the learnable parameters of the spatial hypergraph convolutional layer, For fusion features The node-enhanced feature matrix after spatial hypergraph convolution These are the learnable parameters for the spatial hypergraph convolutional layer;

[0092] Obtain semantically aware modal-level augmented features The method is as follows:

[0093] Features perceived by each modality Initial visual features Aligned audio features and aligned EEG characteristics Connections form semantic hyperedges; these are constructed during the training phase, allowing modal-aware features belonging to the same emotion category to be grouped together. Connect them to form a category hyperedge; obtain a hierarchical hypergraph; based on the hierarchical hypergraph, use the method to obtain a weighted correlation matrix. The same method is used to obtain the weighted correlation matrix. All modality-aware features With weighted correlation matrix Information transfer and feature learning are performed through hypergraph hybrid convolution, resulting in semantically aware modality-level enhanced features. .

[0094] Hierarchical Hypergraph Construction: Within each training batch, a hypergraph structure containing two knowledge levels is constructed. One type is semantic hyperedge, which means that for each sample in the current batch... Connect the corresponding three modal vertices (visual, audio, and EEG) to form a hyperedge. This explicitly models cross-modal correlations within a single sample. Another approach is category hyperedges, used only during the training phase. For each sentiment category, a hyperedge is formed by connecting all vertices belonging to that category in the current batch. This directly injects the supervision information of the real labels into the hypergraph structure, forcing similar samples to aggregate in the representation space. Finally, the semantic hyperedge set is... With category superedge set Merge to form a complete set of hyperedges. :

[0095] ,

[0096] In the formula For all nodes labeled as neutral sentiment in the hierarchical hypergraph, form the category hyperedge. For the category hyperedges formed by all nodes labeled with the emotion of sadness in the hierarchical hypergraph, For the category hyperedge formed by all nodes labeled as angry in the hierarchical hypergraph, Let the category hyperedges in the hierarchical hypergraph consist of all nodes labeled as happy. The category hyperedge formed by all nodes labeled as calm in the hierarchical hypergraph.

[0097] Furthermore, in the gating fusion module, modal-level enhancement features... The three modalities are average pooled to obtain sample-level semantic enhancement features. .

[0098] Hypergraph Convolution Enhancement: Constructing a Weighted Correlation Matrix of Feature Centers and its characteristics A common input hypergraph hybrid convolutional network is used for information transfer and feature learning to obtain semantically aware modality-level enhanced features. This part is similar to the topology branch and will not be repeated here. Average pooling will be used to... The enhancement features from the three modalities are aggregated to obtain sample-level semantic enhancement features that contain high-level semantic context. :

[0099] ,

[0100] In the formula The number of samples included in each training batch.

[0101] An adaptive gating fusion module is constructed to dynamically integrate complementary information from two branches. Specifically, the outputs of the topology branches are... and the output of semantic branches Input a gated fusion module that learns sample-level fusion weights through a small network. Based on sample-level fusion weights Enhanced features of topology awareness and sample-level semantic enhancement features Perform weighted fusion to obtain weighted fusion features. :

[0102] ,

[0103] In the formula Represents the Sigmoid function. For linear layer weights, This is the linear layer bias value. This indicates element-wise multiplication; This indicates column-based operation. This allows the model to adaptively decide whether to rely more on the inherent topology of the data or high-level semantic knowledge, based on the characteristics of the input samples.

[0104] Furthermore, the sentiment category prediction results are expressed as... :

[0105] ,

[0106] In the formula This is the weight matrix of the first fully connected layer in the linear layer. This is the weight matrix of the second fully connected layer in the linear layer. This is the bias vector of the first fully connected layer in the linear layer. This is the bias vector of the second fully connected layer in the linear layer.

[0107] A joint optimization module is constructed to achieve collaborative training of the model by combining a supervised classification task with a self-supervised cross-modal alignment task. The details are as follows:

[0108] I. In the joint optimization module of this implementation method, based on the sentiment category prediction results Compared to real labels that have undergone label smoothing Calculate supervised classification loss ; Calculate supervised classification loss First, the final fusion features A multilayer perceptron classifier is used to generate predictions for the sentiment category. Secondly, the standard one-hot encoded ground truth labels are converted into smoothed soft-distribution labels to regularize the model and mitigate overfitting. Finally, the classification loss is calculated using the label-smoothed cross-entropy loss function.

[0109] II. Calculating Self-Supervised Comparison Loss As an auxiliary task, a multimodal contrastive loss is applied to the initial multimodal features before they enter the hypergraph branch for processing. This loss function aims to bring the different modal representations of the same data sample closer together in the embedding space, while pushing them further away from the representations of all other samples in the batch, thereby forcing the model to learn modality-invariant and semantically consistent features.

[0110] Finally, the calculated classification loss and contrastive loss are linearly weighted and summed, where the contribution of the contrastive loss is determined by an adjustable balancing hyperparameter. Control. By minimizing the overall objective function. The parameters of the entire model are jointly optimized end-to-end, thereby improving the quality of internal feature representation and cross-modal consistency while completing the main classification task.

[0111] Total loss function during system training for:

[0112] ,

[0113] In the formula To balance hyperparameters, For self-monitored comparison of losses.

[0114] In this implementation, during the modality feature extraction stage, frozen, modality-specific initial features are first extracted from the pre-trained backbone network. Then, by accessing the intermediate layers of the network, high-dimensional frozen features are extracted, which will serve as input for downstream tasks. During the bi-branch hypergraph learning stage, the extracted initial features are processed collaboratively by two parallel and complementary branches to model data relationships from different perspectives. The data-driven topology branch captures the inherent local structural relationships between samples by constructing a K-nearest neighbor-based hypergraph. Simultaneously, the knowledge-driven semantic branch constructs a hierarchical hypergraph by defining semantic and category-level hyperedges, explicitly modeling high-level conceptual relationships and directly injecting strong supervision signals into the hypergraph structure. Furthermore, to enhance message passing within the hypergraph, a feature-center-weighted hypergraph convolution is proposed, making the aggregation process more sensitive to the feature distribution within each hyperedge. Subsequently, the outputs of both branches are fed into the hypergraph convolutional layer, propagating information along their respective graph structures to learn refined, structure-aware node representations. In the gated branch fusion phase, refined representations from both the topological and semantic branches are integrated. A lightweight gating mechanism adaptively weighs the contribution of each branch to generate a unified and comprehensive representation that encapsulates both structural and semantic information. Finally, the entire model is trained end-to-end through a joint optimization objective. This objective function combines a label-smooth cross-entropy loss for accurate classification with a multimodal contrastive loss. The multimodal contrastive loss enables the model to learn more discriminative and modality-invariant representations.

[0115] The training method in this implementation is as follows:

[0116] Step T1: Acquire and preprocess multimodal initial features: For the original audio, visual, and EEG signals, initial features are first generated using their respective feature extraction networks. Specifically, the original audio waveform is processed using an AST feature extractor and a pre-trained audio Transformer model. To obtain the initial audio features Use a pre-trained visual Transformer model to process video frames. To obtain initial visual features Raw multichannel EEG signals were processed using downsampling and a custom EEG Transformer model. Initial EEG characteristics were obtained. Subsequently, to achieve frame-level synchronization, based on the temporal resolution of visual features, the sample-level audio features and EEG features were copied and extended along the time dimension to obtain aligned frame-level features. and .

[0117] Step T2: Enhance features through data-driven topological hypergraph branching: The aligned visual, audio, and EEG features obtained in Step T1 are concatenated and initially interacted through a multi-head self-attention layer to obtain fused features. During the training process, through the first... For each sample vertex in a batch, the K-nearest neighbor algorithm is used to find its neighbors in the feature space, which together form a hyperedge. In the formula This represents the total number of hyperedges formed by the vertices in this batch, thus enabling the construction of a topological hypergraph in batches. , This represents the total number of batches in the training set. Subsequently, a weighted correlation matrix is ​​calculated based on feature centrality. and its fusion features A common input hypergraph hybrid convolutional network, after information transfer and updating, yields topology-aware enhanced features. .

[0118] Step T3: Enhance features through knowledge-driven semantic hypergraph branches: To construct semantic branches, first project the initial features of each modality onto a unified dimension. =256, and superimposed with learnable modal embeddings Generate modality-aware features , =10000. Subsequently, during training, a hierarchical hypergraph is constructed in batches. This hypergraph contains two types of hyperedges, including semantic hyperedges. Multiple modal vertices used to connect the same data sample; category hyperedge This is used to connect all vertices within a batch that belong to the same sentiment category (the number of categories is 5), injecting supervisory information. Subsequently, a weighted association matrix is ​​calculated based on feature centrality. and combine it with modality-aware features A common input hypergraph hybrid convolutional network is used. Finally, the output modality-level augmented features are average-pooled and aggregated to obtain sample-level semantic augmented features. .

[0119] Step T4: Adaptively fuse bi-branch features and perform classification prediction: The topology-enhanced features obtained in step T2 are then... The semantic enhancement features obtained in step T3 The input is fed into an adaptive gated fusion module. This module learns a sample-level fusion weight. The features of the two branches are weighted and summed to obtain the final fused features. Finally, this fusion feature... The data is fed into a multilayer perceptron classifier to generate the final sentiment category prediction. :

[0120] ;

[0121] Step T5: Construct a joint objective function and perform end-to-end optimization: Introduce a multi-task learning framework to jointly optimize the entire model. The final training objective consists of two weighted parts: one is the supervised classification loss. Based on the prediction results of step T4 Compared to real labels that have undergone label smoothing Calculate the label smooth cross-entropy loss:

[0122] ,

[0123] in Indicates the first The true labels of each training sample For the five emotion categories contained in the dataset, It is the number of categories. It is a smoothing factor. This represents the number of training samples. On the other hand, it represents the self-supervised contrastive loss. As an auxiliary task, a multimodal contrastive loss is applied to the initial features of each modality extracted in step T1 before entering the hypergraph branch, aiming to force the model to learn cross-modal semantic consistency. The final optimization objective function is a weighted sum of the two, i.e. ,in This is a hyperparameter that balances the two loss terms. The goal is to minimize this joint objective function. End-to-end training is performed on all parameters of the model.

[0124] Figure 3 middle It is the feature matrix corresponding to the i-th sample in the topological hypergraph of the nth training batch; It is the feature matrix corresponding to the j-th KNN hyperedge in the topological hypergraph of the nth training batch; It is a topological hypergraph based on the nth training batch; Therefore, the first The nth sample is one of the remaining nodes in the hyperedge of the center node (the nth sample). The element value corresponding to the nth sample in the correlation matrix of the nth training batch.

[0125] Figure 4 middle It is the feature matrix corresponding to the mode m of the i-th sample in the n-th training batch; It is the feature matrix corresponding to the j-th semantic hyperedge in the n-th training batch; It is the feature matrix corresponding to the k-th category hyperedge of the nth training batch; It is the hierarchical hypergraph of the nth training batch.

[0126] Verification experiment:

[0127] To further validate the effectiveness of the two-branch hypergraph network model in multimodal emotion recognition, experiments were conducted on the publicly available EAV (EEG-Audio-Video) dataset. The EAV dataset is a multimodal benchmark dataset designed for emotion computing research, containing multimodal signals collected from 42 subjects in a controlled environment, including visual modality (facial expression videos), audio modality (voice tone), and EEG modality (30-channel EEG signals). During the experiment, subjects were guided to different emotional states through scripted dialogues to elicit various authentic emotional responses. The dataset provides discrete emotion category labels for each sample, covering five target emotions: anger, happiness, sadness, calmness, and neutrality. Furthermore, the dataset partitioning followed the standard EAV dataset partitioning protocol. To ensure the comparability and reproducibility of the results, a time-order-based partitioning scheme was adopted: the first 70% of the samples collected from each subject were used as the training set, and the last 30% as the test set. This partitioning method ensures sample independence while also conforming to the baseline experimental setup defined by the original authors of the EAV dataset, ensuring that the results can be directly compared with other studies.

[0128] The DBHA model proposed in this application is implemented using the PyTorch framework, and all experiments were performed on a single NVIDIA GPU. During training, the AdamW optimizer was used with an initial learning rate of 0.0001, a batch size of 128, and 20 training epochs. Furthermore, in the data-driven branch, the number of neighbors in the K-nearest neighbor hypergraph construction was set to K=3, and the model's joint optimization objective incorporated label smoothing cross-entropy loss (smoothing factor). (Set to 0.1) and multimodal contrast loss, the latter's contribution is determined by the hyperparameters. The control is also set to a value of 0.1 to balance the contributions of both to the overall loss.

[0129] To comprehensively evaluate the performance of the model in the five-class emotion recognition task, five-class accuracy and weighted F1 score are used as performance evaluation metrics. Specifically, accuracy... It is a key indicator for measuring classification accuracy. It is calculated as the ratio of the number of samples in the test set whose sentiment labels were correctly predicted to the total number of samples:

[0130] ,

[0131] In the formula Indicates the number of test samples. Represents a symbolic function. and They are and The corresponding category label. The weighted F1 score provides a more robust performance metric, especially in multi-class scenarios where the sample distribution may be imbalanced. It is the harmonic mean of precision and recall, calculated independently for each category:

[0132] ,

[0133] in It is a category F1 score, and Representing categories Precision and recall. Furthermore, the five-class weighted F1 score... It can be calculated using the following formula:

[0134] ,

[0135] The contribution of each category The scores are weighted by the number of real-world instances. Furthermore, consistent with the participant-dependent nature of the EAV dataset, all metrics are calculated independently for each participant on their respective test set. The final performance report of the model is the average of all participant scores to comprehensively evaluate the model's overall performance and cross-individual generalization ability.

[0136] To evaluate the effectiveness of the proposed DBHA method, a series of experiments were conducted on the EAV dataset. The classification performance of DBHA was compared with two existing dialogue emotion recognition methods based on physical and physiological signals. Table 1 presents the five-class classification accuracy of the DBHA method for all 42 subjects on both unimodal (EEG, audio, visual) and multimodal data. ) and F1 score ( The classification results of DBHA were presented. DBHA was also compared with two baseline methods, EAV-TFM and EAV-ATT. Overall, DBHA achieved robust and significant performance improvements across all modalities.

[0137] Specifically, under multimodal input conditions, DBHA achieves an average five-class classification accuracy of 75.18% and a weighted F1 score of 74.77%, a significant improvement of approximately 4.32% compared to the EAV-ATT baseline model's 70.86%, indicating that the proposed method possesses stronger emotion recognition capabilities at the multimodal fusion level. Furthermore, DBHA also achieves state-of-the-art performance on each unimodal task, with average accuracies of 74.30% (EEG), 73.36% (audio), and 77.51% (visual), respectively. The improvements are particularly significant on EEG data and audio signals. For EEG data, DBHA significantly outperforms EAV-ATT and EAV-TFM by 20.79% and 14.30%, respectively. Similar significant improvements were observed on audio data, with DBHA outperforming the same baseline by 12.36% and 11.46%, respectively. This clear advantage continues in the visual domain, with improvements of 10.29% and 6.11%, respectively.

[0138] The aforementioned performance improvements are primarily attributed to the core architectural innovations of DBHA. The dual-branch architecture coordinates data-driven topology modeling with knowledge-driven semantic information, thereby constructing a hypergraph structure with stronger semantic consistency and more accurate structural representation. The proposed feature-centric hypergraph convolution achieves more accurate and feature-aware message passing, thus creating more discriminative vertex representations in the semantic space. Therefore, these improvements in unimodal feature quality naturally lead to more effective multimodal fusion, ultimately achieving leading overall performance in multimodal tasks.

[0139] Finally, in addition to average performance, the results for each subject in Table 1 reveal the continued robustness of the DBHA method. For the vast majority of subjects, the method of this invention achieved the highest accuracy and weighted F1 score in both unimodal and multimodal settings, indicating its good adaptability and robustness to physiological and behavioral differences among individuals.

[0140] In summary, the effectiveness and advancement of the DBHA method in multimodal dialogue emotion recognition tasks have been verified. Experimental results show that the proposed method significantly outperforms existing baseline methods at both the unimodal and multimodal levels, especially achieving substantial performance improvements in EEG and audio modalities. This fully demonstrates the advantages and potential of the proposed bi-branch hypergraph mechanism in modeling the relationships between multi-source heterogeneous emotional signals.

[0141] Table 1. Classification results of DBHA and baseline methods on the EAV dataset.

[0142]

[0143] Table 1 (continued)

[0144]

[0145] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A dual-branch hypergraph emotion recognition system integrating topological and semantic information, characterized in that... include, A multimodal feature extraction module is used to extract features from raw audio waveform data. Extracting initial audio features From initial visual data Extracting initial visual features ; from raw multichannel EEG signals Initial EEG features were extracted ; A multimodal feature alignment module is used to align initial audio features. and initial EEG characteristics Extended replication along the time dimension yields the same visual features as the initial ones. Aligned audio features after frame-level alignment and aligned EEG characteristics ; A dual-branch hypergraph enhancement module is used to fuse initial visual features. Aligned audio features and aligned EEG characteristics , obtain fusion features ; All fusion features in the current batch As vertices, a topological hypergraph is constructed based on the K-nearest neighbor algorithm, and a weighted correlation matrix is ​​calculated based on feature centrality. All fusion features With weighted correlation matrix Information is transferred and updated through hypergraph hybrid convolution, resulting in topology-aware enhanced features. Simultaneously based on initial visual features Aligned audio features and aligned EEG characteristics Generate modality-aware features ; All modality-aware features in the current batch As vertices, a hierarchical hypergraph is constructed based on semantic hyperedges and category superposition, and a weighted correlation matrix is ​​calculated based on feature centrality. All modality-aware features With weighted correlation matrix Information transfer and feature learning are performed through hypergraph hybrid convolution, resulting in semantically aware modality-level enhanced features. ; The gated fusion module is used to integrate topology-aware augmented features. Modal-level augmentation features with semantic awareness Perform adaptive weighted fusion to obtain weighted fusion features. ; Linear layers are used to employ two fully connected layers based on weighted fusion features. Generate sentiment category prediction results.

2. The dual-branch hypergraph emotion recognition system integrating topological and semantic information according to claim 1, characterized in that, Also includes: The joint optimization module is used to combine supervised classification loss and self-supervised contrastive loss during system training to achieve collaborative training of the system.

3. The dual-branch hypergraph emotion recognition system integrating topological and semantic information according to claim 2, characterized in that, In the multimodal feature extraction module, an AST feature extractor is used to extract the original audio waveform data through short-time Fourier transform, Mel filter bank application, logarithmic compression, and time padding. Convert to log-Mel spectrum Then, the audio Transformer model is used to generate a log-Mel spectrum. Initial audio features were extracted. ; raw multichannel EEG signals The data tensor is obtained by downsampling through a decimation operation. The EEG Transformer model is used to generate data tensors. Initial EEG features were extracted. .

4. The dual-branch hypergraph emotion recognition system integrating topological and semantic information according to claim 3, characterized in that, In the dual-branch hypergraph enhancement module, fused features are obtained. The method is as follows: Initial visual features Aligned audio features and aligned EEG characteristics The features are concatenated column-wise and then cross-feature interaction is performed through a multi-head self-attention layer to obtain fused features. ; Obtain modal sensing features The method is as follows: based on initial visual features Aligned audio features and aligned EEG characteristics By projecting onto a unified feature dimension through a linear layer, projected visual feature vectors, projected audio feature vectors, and projected EEG feature vectors are obtained. These are then added element-wise to their corresponding learnable modality embedding vectors to obtain visual modality perception features. Audio modality perception features and EEG modal perception features ; Then, visual modal perception features Audio modality perception features and EEG modal perception features Concatenate the data line by line to obtain modality-aware features. .

5. The dual-branch hypergraph emotion recognition system integrating topological and semantic information according to claim 4, characterized in that, Obtain topology-aware enhanced features The method is as follows: All fusion features in the current batch For each vertex, the K-nearest neighbor algorithm is used to determine K-1 neighbor vertices, which form a hyperedge and obtain a topological hypergraph. For all fused features within each hyperedge in the topological hypergraph Calculate the mean to obtain the hyperedge centroid; then fuse the features that minimize the Euclidean distance from the current hyperedge centroid within the current hyperedge. As the center vertex of the hyperedge; then calculate all fused features within the current hyperedge. The Euclidean distance to the center vertex of the hyperedge is used to determine each fused feature based on the reciprocal of the Euclidean distance. The association weights with the current hyperedge are determined; by integrating all vertices and their association weights, a weighted association matrix is ​​obtained. ; Weighted correlation matrix With all fusion features Hypergraph hybrid convolution is used for information transfer and updating to obtain topology-aware enhanced features. .

6. The dual-branch hypergraph emotion recognition system integrating topological and semantic information according to claim 5, characterized in that, Obtain semantically aware modal-level augmented features The method is as follows: Features perceived by each modality Initial visual features Aligned audio features and aligned EEG characteristics Connections constitute semantic hyperedges; During the training phase, modal perception features belonging to the same emotion category are constructed. Connect them to form a category superedge; A hierarchical hypergraph is obtained; Based on a hierarchical hypergraph, a weighted correlation matrix is ​​obtained. The same method is used to obtain the weighted correlation matrix. All modality-aware features With weighted correlation matrix Information transfer and feature learning are performed through hypergraph hybrid convolution, resulting in semantically aware modality-level enhanced features. .

7. The dual-branch hypergraph emotion recognition system integrating topological and semantic information according to claim 6, characterized in that, In the gated fusion module, modal-level enhancement features The three modalities are average pooled to obtain sample-level semantic enhancement features. .

8. The dual-branch hypergraph emotion recognition system integrating topological and semantic information according to claim 7, characterized in that, In the gated fusion module, based on sample-level fusion weights Enhanced features of topology awareness and sample-level semantic enhancement features Perform weighted fusion to obtain weighted fusion features. : ， In the formula Represents the Sigmoid function. For linear layer weights, This is the linear layer bias value. This indicates element-wise multiplication; This indicates column-based operation.

9. The dual-branch hypergraph emotion recognition system integrating topological and semantic information according to claim 8, characterized in that, The sentiment category prediction result is represented as : ， In the formula This is the weight matrix of the first fully connected layer in the linear layer. This is the weight matrix of the second fully connected layer in the linear layer. This is the bias vector of the first fully connected layer in the linear layer. This is the bias vector of the second fully connected layer in the linear layer.

10. The dual-branch hypergraph emotion recognition system integrating topological and semantic information according to claim 9, characterized in that, In the joint optimization module, based on the sentiment category prediction results Compared to real labels that have undergone label smoothing Calculate supervised classification loss ; Total loss function during system training for: ， In the formula To balance hyperparameters, For self-monitored comparison of losses.

Images