Multi-modal emotion recognition method based on electroencephalogram signal and facial expression fusion

Abstract

The invention belongs to the field of biomedical engineering, particularly relates to a multi-modal emotion recognition method based on electroencephalogram signal and facial expression fusion, and aims to improve the accuracy of emotion recognition. The method comprises the steps that electroencephalogram signals and a face video of a subject are collected, frequency band extraction, time window division and standardization processing and key frame extraction and face embedding coding are conducted respectively, time synchronization is achieved through data alignment, electroencephalogram time sequence fragments are input into a preset coding network to extract electroencephalogram time sequence features, and the electroencephalogram time sequence features are obtained. Inputting the facial spatial feature matrix into a preset convolution and time sequence fusion network to extract facial spatial and temporal features, then realizing dynamic interaction between modals through a cross attention mechanism to obtain preliminary fusion features, obtaining modal confidence degrees corresponding to the modals through a preset regression network, weighting the electroencephalogram feature matrix and the facial expression feature matrix, and obtaining an electroencephalogram feature matrix and a facial expression feature matrix; and outputting a final fusion feature matrix. And inputting the final fusion feature matrix into a preset classifier for emotion category prediction, and outputting an emotion recognition result.

Description

Technical Field

[0001] This invention relates to the field of biomedical engineering, and in particular to a multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions. Background Technology

[0002] Emotion recognition, as a core technology in affective computing and advanced human-computer interaction, has significant application value in scenarios such as autonomous driving, mental health monitoring, and intelligent service robots. Human emotional expression is usually accompanied by coordinated changes in physiological and behavioral signals. Single-modal information is insufficient to comprehensively and accurately reflect true emotional states; therefore, multimodal information fusion has become a mainstream research direction for improving the performance and robustness of emotion recognition. Among these, electroencephalogram (EEG) signals and facial expressions are two highly complementary core modalities: EEG signals directly reflect internal brain neural activity, possessing advantages such as strong objectivity and high temporal resolution, avoiding interference from subjective faking; facial expressions, as the most intuitive representation of emotional behavior, are rich in information and easy to collect, quickly conveying intuitive features related to emotions. The fusion of these two modalities lays an important foundation for building high-precision emotion recognition systems.

[0003] However, existing technologies still face many bottlenecks in practical applications. At the level of single-modal feature extraction, EEG signals are non-stationary, have a low signal-to-noise ratio, and are significantly affected by individual physiological differences. Traditional feature extraction methods struggle to fully capture their complex temporal dynamic dependencies, resulting in insufficient cross-subject generalization ability. Facial expressions are easily affected by external factors such as changes in ambient lighting, facial occlusion, and the faking of subjective emotions, leading to poor feature quality stability and difficulty in effectively highlighting features in key emotion-related regions. At the level of multimodal fusion, traditional fusion strategies often employ static feature splicing or fixed-weight fusion methods, failing to dynamically adjust the contribution of each modality based on the modal quality of different samples. When the feature quality of a certain modality declines, low-quality modal information can easily interfere with the fusion result, making it difficult to achieve adaptive complementary enhancement between modalities, severely restricting the overall performance and robustness of emotion recognition systems. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions, aiming to improve the accuracy, robustness, and cross-scenario adaptability of emotion recognition, so as to meet the high-performance requirements of emotion recognition technology in practical application scenarios.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: This invention provides a multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions, comprising: S1: Simultaneously acquiring the subject's electroencephalogram (EEG) signals and facial video; performing frequency band extraction, time window division, and standardization on the EEG signals; extracting keyframes and embedding facial features into the facial video; simultaneously achieving time synchronization between the EEG signals and facial video through data alignment operations; and outputting an EEG time series segment and a facial spatial feature matrix. S2: Inputting the EEG time series segment into a preset encoding network to extract EEG temporal features; inputting the facial spatial feature matrix into a preset convolutional and temporal fusion network to extract facial spatiotemporal features; and outputting an EEG feature matrix and a facial expression feature matrix. S3: The EEG feature matrix and the facial expression feature matrix achieve dynamic interaction between modalities through a cross-attention mechanism to obtain preliminary fusion features; obtaining the modality confidence score corresponding to each modality through a preset regression network; and weighting the EEG feature matrix and the facial expression feature matrix based on the modality confidence score to output the final fusion feature matrix; wherein the modality confidence score is a parameter used to measure the reliability of the corresponding modality features in the current sample or time step. S4: Input the final fused feature matrix into the preset classifier for sentiment category prediction. During the training process, the joint loss function is used to optimize the model and output the sentiment recognition result.

[0006] In step S1, a bandpass filter bank is used to extract multiple feature frequency bands related to emotional activities, the time window is divided into a sliding window segmentation operation, and the face embedding encoding is to encode the features of key frames through a pre-trained facial expression network.

[0007] In step S2, the preset encoding network used to extract EEG time-series features is a multi-layer Transformer encoder. Each layer of the multi-layer Transformer encoder contains a multi-head self-attention mechanism and a feedforward fully connected layer. During the encoding process, position encoding is added to enhance the position information of the EEG time-series segments. At the same time, normalization and random deactivation are set to prevent overfitting.

[0008] In step S2, a preset convolutional and temporal fusion network for extracting facial spatiotemporal features includes a convolutional neural network (CNN) and an inter-frame fusion module. The CNN is used to extract spatial features of facial key frames. The inter-frame fusion module is a temporal convolutional module or a long short-term memory (LSTM) module, used to integrate inter-frame dynamic information. An attention mechanism is set in the preset convolutional and temporal fusion network to highlight key facial action unit regions. Key facial action unit regions include at least the eyes and the corners of the mouth.

[0009] In step S3, the cross-attention mechanism uses the EEG feature matrix as the query vector and the facial expression feature matrix as the key vector and value vector. By calculating attention weights, it realizes the weighted perception of EEG features on facial features and the complementary enhancement of facial features on EEG features. Furthermore, the cross-attention mechanism supports multi-head attention and combines residual connections and layer normalization to ensure gradient stability.

[0010] In step S3, a preset regression network for obtaining modal confidence is also set up with a dynamic gating mechanism. The dynamic gating mechanism is used to perform temporal smoothing or regularization on the modal confidence to avoid feature distortion caused by instantaneous fluctuations in confidence.

[0011] The joint loss function in step S4 includes cross-entropy loss, modality confidence loss, and knowledge distillation loss; where cross-entropy loss is used to drive the accuracy of sentiment classification, modality confidence loss is used to constrain the reliability of weighted fusion features, and knowledge distillation loss is used to realize knowledge transfer of teacher modality features to enhance the model's generalization ability.

[0012] The preset classifier in step S4 is a linear classifier or a multilayer perceptron (MLP). The preset classifier is equipped with a random deactivation layer and a batch normalization layer, which are used to reduce overfitting and accelerate training convergence, respectively. The preset classifier is also configured with a non-linear activation function to improve feature representation ability.

[0013] In step S4, when training and validating the model, a leave-one-out cross-validation strategy is adopted at the subject level. Each subject's sample is used as the test set, and the remaining subject samples are used as the training set to ensure the model's generalization performance among different subjects.

[0014] The convolutional neural network is a pre-trained residual network (ResNet), with batch normalization layers and ReLU activation functions set in the network to improve the model stability and nonlinear representation ability of features in the facial spatial feature extraction process.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This application provides a multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions. By simultaneously acquiring EEG signals and facial videos and employing data alignment operations to strictly ensure their temporal synchronization, the method eliminates interference from factors such as frame loss and signal drift in subsequent analysis, providing high-quality time-matched data for feature extraction. Simultaneously, frequency band extraction, time window division, and standardization of EEG signals effectively filter out core frequency band information related to emotional activity, reducing noise interference from non-stationary signals. Keyframe extraction and facial embedding encoding of facial videos achieve efficient condensation of expression information, avoiding the impact of redundant frames on feature quality. The above preprocessing steps form standardized single-modal data input, providing a reliable prerequisite for subsequent deep feature mining and modality fusion, significantly improving the underlying data quality of the entire recognition system. In terms of EEG temporal feature extraction, this invention employs a multi-layer Transformer encoder. Through the synergistic effect of a multi-head self-attention mechanism and a feedforward fully connected layer, it accurately captures the short-term fluctuations and long-term dependencies of EEG signals. Combined with positional encoding to enhance temporal positional information, and normalization and random deactivation operations to combat overfitting, it effectively overcomes the limitations of traditional methods in describing the dynamic features of non-stationary EEG signals. The output EEG feature matrix possesses richer temporal discriminative information. In terms of facial spatiotemporal feature extraction, the pre-set convolutional and temporal fusion network is based on a pre-trained residual network (ResNet). Batch normalization layers and ReLU activation functions enhance the stability and nonlinear expression of spatial feature extraction. The attention mechanism emphasizes key facial action unit regions such as the eyes and corners of the mouth, accurately focusing on core emotion-related features. The inter-frame fusion module, composed of temporal convolutional modules or long short-term memory (LSTM) modules, successfully integrates the inter-frame dynamic information of facial expressions, enabling the facial expression feature matrix to simultaneously cover spatial details and temporal evolution patterns, significantly improving the depth and discriminative power of single-modal features.

[0016] 2. The method provided in this application innovatively employs a cross-attention mechanism to achieve dynamic interaction between modalities. By designing the EEG feature matrix as the query vector and the facial expression feature matrix as the key and value vectors, a bidirectional complementary feature interaction channel is constructed. This allows EEG features to accurately perceive key facial information with weighted processing, while facial features provide intuitive behavioral supplements to EEG features, effectively tapping into the spatiotemporal complementary value of the two modalities. Based on this, a modality confidence weighting mechanism is introduced. A pre-set regression network is used to evaluate the reliability of each modality in the current sample or time step in real time. Combined with a dynamic gating mechanism for temporal smoothing and regularization of confidence, feature distortion caused by instantaneous fluctuations is avoided, achieving adaptive adjustment of modality contribution. When a modality is disturbed (e.g., facial occlusion, sudden increase in EEG signal noise), its weight is automatically reduced, thereby avoiding contamination of the fusion result by low-quality modalities. This dynamic interaction and confidence-weighted fusion strategy completely breaks through the rigid limitations of traditional static splicing or fixed-weight fusion, making the final fused feature matrix both complementary and robust, providing core support for improving the accuracy of emotion recognition.

[0017] 3. The method provided in this application improves the accuracy of sentiment classification through cross-entropy loss, constrains the reliability of weighted fusion features through modality confidence loss, and realizes knowledge transfer of teacher modality features through knowledge distillation loss. The synergistic effect of these three factors forms a multi-dimensional optimization objective, which not only ensures the core performance of the classification task but also enhances the rationality of modality fusion and the generalization ability of the model. Simultaneously, the pre-configured random deactivation layer and batch normalization layer in the classifier optimize the model training process from two dimensions: suppressing overfitting and accelerating training convergence. The introduction of a non-linear activation function further enhances the complex expressive power of features. Furthermore, the use of a leave-one-out cross-validation strategy at the subject level, by sequentially using each subject's sample as the test set and the remaining samples as the training set, effectively adapts to the individual differences in EEG signals, significantly improves the model's generalization performance among different subjects, avoids model performance fluctuations caused by individual physiological heterogeneity, and makes the method more applicable in real-world multi-user scenarios. Attached Figure Description

[0018] Figure 1 This is a flowchart of a multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions, provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] For example, refer to Figure 1 This application provides a multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions, including: S1: Simultaneously acquire the subject's electroencephalogram (EEG) signals and facial video, extract frequency bands, divide time windows and standardize the EEG signals, extract keyframes and encode facial embeddings in the facial video, and achieve time synchronization between the EEG signals and facial video through data alignment operations, outputting EEG time series segments and facial spatial feature matrices.

[0021] For example, in step S1, a bandpass filter bank is used to extract multiple feature frequency bands related to emotional activity, the time window is divided into a sliding window segmentation operation, and the facial embedding encoding is to encode the features of key frames through a pre-trained facial expression network.

[0022] More specifically, the acquired EEG signals were recorded using a 64-channel EEG cap at a sampling rate of 512 Hz, with a facial video sampling rate of 30 frames per second. The raw EEG signals were decomposed using a bandpass filter bank to extract five classic frequency bands closely related to human emotional activity: Wave (8-13Hz), Wave (13-30Hz) Wave (30-50Hz), Wave (0.5-4Hz) and Wave (4-8Hz). The above five frequency bands have clear physiological significance and can reflect the state of neural activity in the brain from different perspectives.

[0023] Subsequently, a rigorous sliding window segmentation is performed on the signal in each frequency band to ensure the temporal resolution and local stationarity of the analysis. As one possible implementation, the window length is 2 seconds and the step size is 0.5 seconds. At the same time, the signal within each window is standardized to obtain an EEG time sequence segment of length L.

[0024] Facial videos are processed using keyframe extraction algorithms, such as facial action unit detection or optical flow methods, to select keyframes. Each frame undergoes facial embedding encoding, for example, using a pre-trained facial expression network to obtain an F×d_f facial spatial feature matrix to represent the spatial features of each frame. Data alignment operations are then used to synchronize EEG signals with the facial video in time, eliminating the impact of dropped frames or signal drift on subsequent analysis. This results in high-quality single-modal feature representations, laying the foundation for feature extraction and multimodal fusion.

[0025] As one possible implementation, differential entropy (DE) features are calculated within each segmented time window. DE features, as a commonly used nonlinear feature, effectively reflect the complexity and uncertainty of EEG signals. Extracting DE features not only continues the work of traditional features but also provides a high-dimensional, sensitive foundational representation for subsequent dynamic time-varying analysis. All these preprocessing operations aim to minimize noise interference and provide high-quality, highly discriminative input for the deep learning model. S2: Input the EEG time series segment into a preset encoding network to extract EEG temporal features, input the facial spatial feature matrix into a preset convolutional and temporal fusion network to extract facial spatiotemporal features, and output the EEG feature matrix and facial expression feature matrix.

[0026] For example, in step S2, the preset encoding network used to extract EEG time-series features is a multi-layer Transformer encoder. Each layer of the multi-layer Transformer encoder includes a multi-head self-attention mechanism and a feedforward fully connected layer to capture short-term fluctuations and long-term dependencies of EEG signals and output an EEG feature matrix H_EEG. In the encoding process, position encoding is added to enhance the position information of the EEG time-series segments and enhance the model's sensitivity to dynamic changes in signals. At the same time, normalization and random deactivation are set within the EEG time-series segments after time window division to prevent overfitting.

[0027] In step S2, a pre-defined convolutional and temporal fusion network for extracting facial spatiotemporal features includes a convolutional neural network (CNN) and an inter-frame fusion module. The CNN is used to extract spatial features of key facial frames; the inter-frame fusion module is a temporal convolutional module or a long short-term memory (LSTM) module, used to integrate dynamic information between frames; and an attention mechanism is set in the pre-defined convolutional and temporal fusion network to highlight key facial action unit regions, ultimately outputting a facial expression feature matrix. The key facial motion unit regions include at least the eyes and corners of the mouth, thereby enhancing the representation of emotion-related features. Through EEG temporal feature extraction and facial spatiotemporal feature extraction, the final output... and It serves as a basic representation for single-modal functions, preserving the spatiotemporal characteristics of each modality and providing complete information for multimodal fusion.

[0028] As one possible implementation, the convolutional neural network is a pre-trained residual network (ResNet), with a batch normalization layer (BatchNorm) and a ReLU activation function set in the network to improve the model stability and nonlinear representation ability of features in the facial spatial feature extraction process.

[0029] S3: The EEG feature matrix and facial expression feature matrix achieve dynamic intermodal interaction through a cross-attention mechanism, yielding preliminary fusion features. A pre-defined regression network is used to obtain the modality confidence score for each modality; EEG features and facial features are regressed separately to obtain the modality confidence score. and The EEG feature matrix and facial expression feature matrix are weighted based on modal confidence to output the final fused feature matrix. Modal confidence is a parameter used to measure the reliability of the corresponding modal feature in the current sample or time step. This represents the final fused feature matrix after confidence weighting; This represents the confidence parameters of EEG modalities obtained through a pre-defined regression network; This represents the facial modality confidence parameters obtained through a pre-defined regression network; Represents the input EEG feature matrix; This represents the input facial expression feature matrix.

[0030] The cross-attention mechanism in step S3 uses the EEG feature matrix as the query vector and the facial expression feature matrix as the key and value vectors, respectively, and calculates the attention weight matrix. This achieves weighted perception of facial features by EEG and complementary enhancement of facial features to EEG features, generating preliminary fusion features. .

[0031] in, By combining the dynamic temporal information of EEG with the spatial features of facial expressions, adaptive information complementarity between modalities is achieved; This represents the calculated attention weight matrix, used to measure the correlation between EEG features and facial features; The input EEG feature matrix is ​​used as a query vector to perceive facial features. The input facial expression feature matrix is ​​represented as a key vector and a value vector. A matrix representing the transpose of the facial expression feature matrix; This represents the dimension of the feature vector (i.e., the scaling factor parameter in the formula). This indicates that the square root operation is performed on the dimension, which is used to scale the dot product result to ensure gradient stability; This represents the normalized exponential function, used to transform the correlation calculation results into weights in the form of a probability distribution. Furthermore, the cross-attention mechanism supports multi-head attention to capture fine-grained relationships of multimodal interactions in different subspaces, and combines residual connections and layer normalization to ensure gradient stability.

[0032] For example, in step S3, the preset regression network for obtaining modal confidence is also equipped with a dynamic gating mechanism. The dynamic gating mechanism is used to perform temporal smoothing or regularization on the modal confidence to avoid feature distortion caused by instantaneous fluctuations in confidence. At the same time, the alpha weight can be automatically adjusted through training, enabling the model to achieve modal adaptive weighting under different subjects and different situations, thereby enhancing the robustness of the overall features. As the final fused feature matrix, it integrates the time-series information of EEG and the spatial information of facial expressions, and is optimized for confidence, thereby improving the accuracy and robustness of emotion prediction.

[0033] S4: Input the final fused feature matrix into the preset classifier for sentiment category prediction, and output the softmax probability distribution. .in, This represents the probability distribution of the predicted sentiment category in the output. Represents the classification layer weight matrix; This represents the classification layer bias vector; during training, a joint loss function is used to optimize the model and output the emotion recognition result.

[0034] An example, the joint loss function: The joint loss function includes cross-entropy loss, modality confidence loss, and knowledge distillation loss; among which, cross-entropy loss... Modal confidence loss is used to drive sentiment classification accuracy. Knowledge distillation loss is used to constrain the reliability of weighted fusion features. This is used to implement knowledge transfer of teacher modal features to enhance the model's generalization ability. , This is the loss weighting coefficient.

[0035] As one possible implementation, the preset classifier in step S4 is a linear classifier or a multilayer perceptron (MLP). The preset classifier is equipped with a random deactivation layer and a batch normalization layer, which are used to reduce overfitting and accelerate training convergence, respectively. The preset classifier is also configured with a non-linear activation function ReLU or GELU to improve feature representation ability.

[0036] As one possible implementation, in step S4, when training and validating the model, a leave-one-out cross-validation strategy is adopted at the subject level. Each subject's sample is used as the test set, and the remaining subject samples are used as the training set, so as to ensure the generalization performance of the model among different subjects.

[0037] For example, the method provided in this application is based on a frequency-varying high-order graph emotion recognition system using EEG patterns. This system includes a preprocessing module, a dynamic time-varying representation extraction module, a high-order graph construction module based on MST, a GIN encoding module, a node clustering graph pooling module, and an emotion classification module. The specific correspondence is as follows: the basic data output from step S1 is input into the preprocessing module; the extraction of EEG temporal features in step S2 is specifically completed collaboratively by the dynamic time-varying representation extraction module, the MST-based high-order graph construction module, and the GIN encoding module. Specifically, the FFT transform in the dynamic time-varying representation extraction module is used to convert the time-domain segment from step S1 into a frequency-domain distribution, serving as input enhancement for the deep encoding network in step S2. The fusion logic in step S3 is embedded in the multimodal interaction layer; the classification and optimization in step S4 are implemented through the node clustering graph pooling module and the emotion classification module.

[0038] The preprocessing module executes the steps in S1, outputting EEG time-series segments and facial spatial feature matrices. The core objective of the dynamic time-varying representation extraction module is to overcome the limitations of traditional features in describing non-stationary signals, deeply explore the complex temporal variations in EEG signals, decompose them into intra- and inter-period variation patterns, and creatively map them to 2D space, i.e., frequency ripple representation, to more effectively capture the complexity and periodicity of the time series. The dynamic time-varying representation extraction module integrates a data transformation layer, a dynamic multi-scale feature extraction layer, and an adaptive feature aggregation layer. The data transformation layer first processes the signal at each time step... Fast Fourier Transform (FFT) reveals the intensity of its frequency components: Where F(k) represents the frequency index. The complex spectrum output value at time; x(n) represents the amplitude of the nth sampling point in the time-domain input signal; k is the frequency index, ranging from 0 to N-1; n is the time-domain sampling point sequence index, ranging from 0 to N-1; N is the total number of sampling points of the EEG signal within the processing window; i is the imaginary unit, and satisfies i 2 =-1.

[0039] By performing amplitude analysis on the FFT results, several of the most significant dominant frequency components were identified and selected. Based on the periods corresponding to these dominant frequency components, the original... Time series remodeling Tensor. This kind of... arrive The mapping transforms the periodic variations of time series into spatial texture, realizing a spatial representation of time-frequency information. The dynamic multi-scale feature extraction layer then utilizes multi-scale one-dimensional convolution operations on these... Feature extraction is performed on tensors (i.e., frequency ripples) to learn their rich spatiotemporal spatial information. Convolutional kernels of different scales ensure comprehensive capture of subtle periodic variations and macroscopic periodic patterns. Finally, the learned... The tensor was transformed back Space, through dimensional restoration and reshaping: in, This represents the generated two-dimensional ripple tensor; This represents the two-dimensional spatial feature matrix output by the dynamic multi-scale feature extraction layer; This represents a dimension reconstruction function, used to map extracted depth features back to a preset texture space dimension in order to capture the spatial evolution patterns of time-frequency information.

[0040] The adaptive feature aggregation layer will Different periods correspond to This indicates that weighted fusion is performed to generate the final dynamic time-frequency representation for each electrode node. Its definition is: in It can be adaptively determined by the amplitude information of the corresponding frequency components, ensuring that feature aggregation has clear physiological significance and higher discriminative power; K represents the total number of branches in multi-scale feature extraction; Indicates the first Local periodic features extracted from each branch.

[0041] The core task of the MST-based high-order graph construction module is to utilize the obtained dynamic time-frequency representation as node attributes and transform it into a brain network graph with topological structure and functional connectivity. It consists of an initial graph adjacency matrix construction layer, an initial graph edge embedding construction layer, and an MST-based high-order graph construction layer. The initial graph adjacency matrix construction layer treats each electrode of the input signal as a node in the graph and constructs an adjacency matrix of an undirected initial graph assuming global connectivity by calculating the similarity (e.g., dot product) between electrode node attributes. This adjacency matrix is ​​dynamic and instance-specific, reflecting the immediate functional connectivity strength under specific emotional states. The initial graph edge embedding construction layer creates edge embedding features by concatenating the embeddings of source and target nodes. In the most crucial MST-based higher-order graph construction layer, let... This represents the initial graph constructed based on EEG data, targeting The Minimum Spanning Tree (MST) is computed. The goal of the MST algorithm is to find a subgraph containing all nodes such that it satisfies connectivity, acyclicity, and minimizes the sum of edge weights. Through MST, the model can effectively filter out the most critical and strongest functional connections between EEG channels, automatically removing redundant or noisy connections with low weights, thereby constructing a higher-order graph structure with greater physiological significance and discriminative power. .

[0042] The GIN encoding module is used for the constructed high-order graph structure. This application introduces a Graph Isomorphic Network (GIN) to deeply learn the complex spatial relationships between individual electrode nodes and aggregate neighbor information, thereby obtaining an effective graph representation with global awareness. GIN has proven to be powerful in distinguishing graph structures. The GIN encoder iteratively updates the node representation through a message passing mechanism. For the first node in GIN... Layers, nodes Features The update formula clearly illustrates this aggregation process: in Let i represent the feature vector of node i after the l-th iteration. It is a characteristic of the node itself. Represents data from neighboring nodes. Aggregated information, It is a multilayer perceptron used for feature transformation. Parameters Indicates the first The layer's pre-defined learnable parameters are used to balance the aggregation ratio of node's own information and neighbor information. This iterative process ensures the final node representation is obtained. It not only contains its own dynamic time-frequency information, but also fully integrates local and global spatial topology information.

[0043] To selectively downsample the graph structure, reducing the number of nodes, removing redundant noise, and retaining the most representative structural information, the node clustering graph pooling module uses the HGP-SL (Hierarchical Graph Pooling with Self-Learned Scores) operator. HGP-SL, combined with the GIN encoding module, achieves soft selection of nodes by calculating an importance score for each node (which integrates multiple metrics such as degree centrality, feature importance, and PageRank score). This mechanism maximizes the preservation of the graph's core information and aggregates information from less important nodes to more important ones, thereby learning a more refined and compact graph structure. This effectively improves the model's running efficiency and noise resistance.

[0044] For the emotion classification module, this embodiment employs a leave-one-out cross-validation (LOSO) strategy at the subject level during the training and testing phases. In each phase, samples from each subject are used as the test set, while samples from the remaining subjects are merged into the training set. This rigorous validation method effectively ensures the model's individual-independent generalization performance. In step S6, all frequency bands and the final features after graph pooling are first aggregated. The data is then fed into a fully connected layer to obtain the final classification result. in It is an activation function used to increase the non-linear expressive power of the model; Used to reduce the over-reliance of neural networks on certain specific neurons and improve the generalization ability of the model; This is a fully connected layer used to obtain the final predicted probability of the sentiment category. The sentiment classification model is ultimately evaluated for performance using the average accuracy of leave-one-out cross-validation.

[0045] In the description of this specification, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0046] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions, characterized in that, include: S1: Simultaneously acquire the subject's EEG signals and facial video, perform frequency band extraction, time window division and standardization processing on the EEG signals, extract keyframes and perform facial embedding encoding on the facial video, and simultaneously achieve time synchronization between EEG signals and facial video through data alignment operation, and output EEG time sequence segments and facial spatial feature matrix. S2: Input the EEG time series segment into a preset encoding network to extract EEG temporal features, and input the facial spatial feature matrix into a preset convolutional and temporal fusion network to extract facial spatiotemporal features, outputting an EEG feature matrix and a facial expression feature matrix; S3: The EEG feature matrix and facial expression feature matrix achieve dynamic interaction between modalities through a cross-attention mechanism to obtain preliminary fusion features, obtain the modality confidence corresponding to each modality through a preset regression network, and weight the EEG feature matrix and facial expression feature matrix based on the modality confidence to output the final fusion feature matrix; Wherein, modality confidence is a parameter used to measure the reliability of the corresponding modality feature in the current sample or time step; S4: Input the final fused feature matrix into the preset classifier for sentiment category prediction, and use the joint loss function to optimize the model during training to output the sentiment recognition result.

2. The multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions according to claim 1, characterized in that, In step S1, a bandpass filter bank is used to extract multiple feature frequency bands related to emotional activity. The time window is divided into a sliding window segmentation operation. The facial embedding encoding is to encode the features of key frames through a pre-trained facial expression network.

3. The multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions according to claim 1, characterized in that, In step S2, the preset encoding network used to extract EEG time-series features is a multi-layer Transformer encoder. Each layer of the multi-layer Transformer encoder contains a multi-head self-attention mechanism and a feedforward fully connected layer. Position encoding is added during the encoding process to enhance the position information of the EEG time-series segments. At the same time, normalization and random deactivation are set to prevent overfitting.

4. The multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions according to claim 1, characterized in that, In step S2, a preset convolutional and temporal fusion network for extracting facial spatiotemporal features includes a convolutional neural network and an inter-frame fusion module. The convolutional neural network is used to extract spatial features of key facial frames. The inter-frame fusion module is a temporal convolutional module or a long short-term memory module, used to integrate inter-frame dynamic information. An attention mechanism is set in the preset convolutional and temporal fusion network to highlight key facial action unit regions. Key facial action unit regions include at least the eyes and the corners of the mouth.

5. The multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions according to claim 3, characterized in that, In step S3, the cross-attention mechanism uses the EEG feature matrix as the query vector and the facial expression feature matrix as the key vector and value vector. By calculating attention weights, it realizes the weighted perception of EEG features on facial features and the complementary enhancement of facial features on EEG features. Furthermore, the cross-attention mechanism supports multi-head attention and combines residual connections and layer normalization to ensure gradient stability.

6. The multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions according to claim 1, characterized in that, In step S3, a preset regression network for obtaining modal confidence is also provided, and a dynamic gating mechanism is set. The dynamic gating mechanism is used to perform temporal smoothing or regularization on the modal confidence to avoid feature distortion caused by instantaneous fluctuations in confidence.

7. The multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions according to claim 1, characterized in that, The joint loss function in step S4 includes cross-entropy loss, modality confidence loss, and knowledge distillation loss; where cross-entropy loss is used to drive the accuracy of sentiment classification, modality confidence loss is used to constrain the reliability of weighted fusion features, and knowledge distillation loss is used to realize knowledge transfer of teacher modality features to enhance the model's generalization ability.

8. The multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions according to claim 7, characterized in that, The preset classifier in step S4 is a linear classifier or a multilayer perceptron (MLP). The preset classifier is equipped with a random deactivation layer and a batch normalization layer, which are used to reduce overfitting and accelerate training convergence, respectively. The preset classifier is also configured with a non-linear activation function to improve feature representation ability.

9. A multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions according to claim 1, characterized in that, In step S4, when training and validating the model, a leave-one-out cross-validation strategy is adopted at the subject level. Each subject's sample is used as the test set, and the remaining subject samples are used as the training set to ensure the model's generalization performance among different subjects.

10. A multimodal emotion recognition method based on the fusion of electroencephalogram (EEG) signals and facial expressions according to claim 4, characterized in that, The convolutional neural network is a pre-trained residual network, and a batch normalization layer and a ReLU activation function are set in the network to improve the model stability and feature nonlinear expression ability in the facial spatial feature extraction process.

Images