A brain-inspired multi-expert multi-modal emotion recognition method, system, device and medium

Abstract

This invention discloses a brain-inspired multi-expert multimodal emotion recognition method, system, device, and medium, belonging to the fields of artificial intelligence and biomedical signal processing technology. The method simulates the brain's functional partitioning mechanism, dividing EEG signals into multiple brain regions according to neuroanatomical priors, and designing a dedicated expert network for each region. A global-local dual-stream encoder is used to collaboratively extract the spatiotemporal features of signals from each brain region, while a multi-scale large-kernel convolution module is used to extract peripheral physiological signal features. Finally, an adaptive routing network dynamically fuses multi-expert features to achieve emotion classification. The model incorporates a joint loss of expert load balancing and divergence regularization during training to ensure effective expert collaboration and feature diversity. This invention has achieved excellent recognition accuracy in practice, and interpretability analysis verifies that its decision-making process conforms to neuroscientific cognition, providing a high-precision and high-reliability solution for applications such as brain-computer interfaces and mental health monitoring.

Description

Technical Field

[0001] This invention relates to a brain-inspired multi-expert multimodal emotion recognition method, system, device, and medium, belonging to the field of artificial intelligence and biomedical signal processing technology, and particularly to a brain-inspired multi-expert multimodal emotion recognition method. Background Technology

[0002] Emotion is fundamental to human cognition, decision-making, and social interaction. Accurate decoding of physiological signals to identify emotional states is crucial for brain-computer interfaces, mental health monitoring, and human-centered artificial intelligence systems. Electroencephalography (EEG) signals, due to their direct reflection of brain neural activity and high temporal resolution at the millisecond level, have become an important data source for emotion recognition. Peripheral physiological signals (PPS), such as electrocardiogram (ECG), skin conductance response (GSR), respiration (RESP), and body temperature (TEMP), provide important supplementary information reflecting the state of the autonomic nervous system. Based on the complementarity and synergy between EEG signals and peripheral physiological signals in emotional representation, multimodal emotion recognition has become an important research direction for building robust emotion recognition systems.

[0003] Existing technologies mainly focus on the spatial topological structure modeling of EEG signals and their fusion with peripheral physiological signals, and primarily include the following three technical approaches:

[0004] The pseudo-image processing method based on convolutional neural networks (CNN) treats multi-channel EEG signals as a two-dimensional pseudo-image structure and uses CNN for feature extraction. Although it uses a mature neural network model to extract signal spatial features, it cannot explicitly model the functional specificity of brain regions.

[0005] Topological modeling methods based on graph neural networks (GNNs) abstract EEG signal electrodes as graph nodes, use the correlation between channels as edges, construct a functional connectivity graph, and use graph convolutional networks (GCNs) to model spatial topological relationships. However, they often neglect the deep integration of local temporal features.

[0006] EEG-dominated multimodal information fusion frameworks employ feature-level or decision-level fusion strategies, but the fusion methods are mostly static or simple splicing, making it difficult to characterize the complex, dynamic, nonlinear synergistic relationship between EEG signals and peripheral physiological signals.

[0007] The existing technology has the following defects: (i) it does not effectively incorporate prior knowledge of neuroscience and ignores the functional specificity of brain regions; (ii) it has insufficient ability to integrate global spatial topology and local temporal dynamic features; (iii) the multimodal fusion strategy is insufficient in modeling the dynamic synergy between modal complementarity and heterogeneity. Summary of the Invention

[0008] The present invention aims to overcome the above-mentioned defects of the prior art and proposes a brain-inspired multi-expert multimodal emotion recognition method, system, device and medium. By simulating the brain's functional partitioning mechanism, an emotion recognition model with neural interpretability, spatiotemporal feature synergy and dynamic multimodal fusion capability is constructed.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A brain-inspired, multi-expert, multimodal emotion recognition method, combining Figure 1 Its characteristic is that it includes the following steps:

[0011] Step S1: Multimodal physiological signal preprocessing: Acquire multi-channel EEG signals and peripheral physiological signals, and perform preprocessing.

[0012] Step S2, Brain Region Expert Division and Signal Allocation: Based on the international 10-20 system, the brain is divided into multiple functional regions, each region corresponding to an EEG expert network, and a corresponding subset of EEG signal channels is allocated; at the same time, an independent peripheral physiological expert network is allocated to peripheral physiological signals; the collection of all EEG expert networks and peripheral physiological expert networks is collectively referred to as the expert network.

[0013] Step S3, Global-Local Dual-Stream EEG Feature Extraction: Each EEG expert network is a global-local dual-stream encoder, used to extract the spatial topological features and local temporal features of the EEG signals of the corresponding brain region, respectively. Finally, the EEG expert features of the corresponding brain region are obtained by fusing them through the attention mechanism.

[0014] Step S4, Peripheral physiological signal feature extraction: The peripheral physiological expert network is a multi-scale large kernel convolution module. It uses convolution kernels of different scales to extract long-term temporal dependence features in peripheral physiological signals to obtain peripheral physiological features.

[0015] Step S5, Adaptive Multi-Expert Feature Fusion: The features output by all expert networks are concatenated and input into a lightweight routing network. The network generates dynamic weights for all expert networks. Then, the output features of all expert networks are adaptively weighted and fused to obtain the sentiment state classification result.

[0016] Step S6, Joint Optimization and Classification Output: Design a joint loss function, including classification loss, expert load balancing loss and expert divergence regularization loss, optimize the model parameters through gradient descent, and finally output the sentiment state classification result;

[0017] Step S7, Interpretability Analysis: Interpretability methods are used to visualize the expert contributions of each brain region and verify the consistency between the model's decisions and neuroscience knowledge.

[0018] The global-local dual-stream encoder has two parallel branches: one branch is a weighted phase lag index (wPLI) calculation of a concatenated graph convolutional network followed by a multi-head self-attention mechanism, corresponding to the extraction of spatial topological features; the other branch is a one-dimensional convolutional network followed by a feedforward network (FFN), corresponding to the extraction of local temporal features; the two branches are concatenated by Concat to achieve EEG expert feature output.

[0019] The multi-scale large-kernel convolution module consists of multiple one-dimensional convolutional layers of different kernel sizes connected in parallel, sequentially connected to perform Batch Normalization (BN), ReLU activation, and a multilayer perceptron (MLP).

[0020] Furthermore, the preprocessing described in step S1 is as follows:

[0021] S101: Use a third-order Butterworth bandpass filter to filter EEG signals and remove noise and artifacts;

[0022] S102: Divide the continuous signal into a non-overlapping 1-second time window;

[0023] S103: Perform Z-Score normalization on each channel to eliminate individual differences and equipment bias.

[0024] Furthermore, step S2, based on the international 10-20 system, divides the brain into the following six functional regions: prefrontal, central, parietal, occipital, temporal, and whole-brain EEG, each corresponding to a separate EEG expert network. The corresponding channel signal subset matrix is ​​extracted from each functional region and used as the input of the corresponding EEG expert network.

[0025] Furthermore, the working principle of the global-local dual-stream encoder described in any step S3 is as follows:

[0026] S301: The functional connectivity map of the EEG signal of the corresponding brain region is calculated based on the weighted phase lag index, which is a symmetric weighted adjacency matrix. Then, spatial topological features are extracted through a graph convolutional network.

[0027] S302: Extract local temporal features from the EEG signals of the corresponding brain regions using a one-dimensional convolutional network;

[0028] S303: Global attention features corresponding to spatial topological features are calculated through a multi-head self-attention mechanism, local features corresponding to local temporal features are calculated through a feedforward network, and then the global attention features and local features are concatenated to obtain the EEG expert features of the corresponding brain region.

[0029] Furthermore, step S4 specifically includes:

[0030] S401: Extracting multi-scale features of peripheral physiological signals by using multiple one-dimensional convolutional layers with different kernel sizes in parallel;

[0031] S402: After BN normalization and ReLU activation, multi-scale features are projected onto a unified dimension using a multilayer perceptron to obtain peripheral physiological features.

[0032] Furthermore, the lightweight routing network described in step S5 consists of a multilayer perceptron connected in series with a Softmax layer.

[0033] Furthermore, the joint loss function described in step S6 consists of classification loss. Expert load balancing losses and expert disagreement regularization loss It consists of three parts, namely Among them, hyperparameters , Used to balance various losses to similar orders of magnitude; classification loss function , The model represents the true category. The predicted probability, and These are the category balance weights and the focus parameters, respectively. Sample size for training batches; expert load balancing loss , , For the first The dynamic weights corresponding to each expert network , For the first Utilization rate of expert networks; expert disagreement regularization loss , For symmetric KL divergence, For the first The output of an expert network is normalized by a Softmax layer.

[0034] A brain-inspired, multi-expert, multimodal emotion recognition system, combining Figure 2 It is characterized by comprising a signal acquisition module, a preprocessing module, an expert segmentation module, a feature extraction module, a dynamic output module, a loss optimization module, and an interpretability module.

[0035] The signal acquisition module has its input end connected to a multimodal sensor and its output end connected to the input end of a preprocessing module, and is used to acquire multi-channel EEG signals and peripheral physiological signals.

[0036] The preprocessing module has its output connected to the input of the expert segmentation module, and performs filtering, segmentation, and normalization processing on the acquired signals.

[0037] The expert segmentation module has its output end connected to the input end of the feature extraction module. It divides the EEG signal channels according to the brain region function and assigns them to the corresponding experts.

[0038] The feature extraction module (4) has its output end connected to the input end of the dynamic output module (5) and the interpretability module (7), and includes multiple EEG expert networks and a peripheral physiological expert network connected in parallel for different brain regions. The EEG expert network uses a global-local dual-stream encoder to extract EEG expert features of the EEG signals of the corresponding brain region signal channels, and the peripheral physiological expert network uses a multi-scale large kernel convolution module to extract peripheral physiological features of the peripheral physiological signals.

[0039] The dynamic output module is a lightweight routing network, and its output end is connected to the input end of the loss optimization module and the interpretability module to realize dynamic weighted fusion of multi-expert features.

[0040] The loss optimization module is connected to the feature extraction module and the dynamic output module at its output end, and dynamically optimizes the weights of each expert network and the lightweight routing network based on the joint loss function.

[0041] The interpretability module enables the visualization and interpretable analysis of emotional state decisions and provides feedback to the user.

[0042] An electronic device includes at least one processor; and a memory communicatively connected to said at least one processor; wherein,

[0043] The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to perform the brain-inspired multi-expert multimodal emotion recognition method described above.

[0044] Finally, the present invention also discloses a computer-readable storage medium storing computer instructions for causing a processor to execute the aforementioned brain-inspired multi-expert multimodal emotion recognition method.

[0045] The beneficial effects of this invention are as follows: This invention proposes a brain-inspired multi-expert multimodal emotion recognition method, system, device, and medium. By introducing a brain functional partitioning mechanism to construct a multi-expert architecture, and adopting a global-local dual-stream collaborative feature extraction and adaptive multimodal fusion strategy, the accuracy, robustness, and neural interpretability of emotion recognition are significantly improved. The model incorporates expert load balancing and divergence regularization mechanisms during training, effectively preventing expert degradation and enhancing feature diversity. The model has very high prediction accuracy. At the same time, visualization analysis verifies the consistency between the decision-making process and neuroscience cognition, providing a high-precision and high-reliability multimodal emotion recognition solution for applications such as brain-computer interfaces and mental health monitoring. Attached Figure Description

[0046] To make the objectives and technical solutions of this invention clearer, the following figures are provided for illustration:

[0047] Figure 1 This is a flowchart of a brain-inspired multi-expert multimodal emotion recognition method according to the present invention;

[0048] Figure 2 This is an architecture diagram of a brain-inspired multi-expert multimodal emotion recognition system according to the present invention; wherein, the solid arrows indicate the data transmission direction, and the dashed arrows indicate the parameter tuning direction;

[0049] Figure 3 This is a diagram of the global-local dual-stream encoder network architecture in Embodiment 1 of the present invention;

[0050] Figure 4 This is a network architecture diagram of the multi-scale large kernel convolution module in Embodiment 1 of the present invention;

[0051] Figure 5 This is a diagram of the lightweight routing network architecture in Embodiment 1 of the present invention;

[0052] Figure 6 This is a SHAP visualization of the DEAP and DREAMER datasets in Embodiment 1 of the present invention; the left image corresponds to the DEAP dataset, and the right image corresponds to the DREAMER dataset.

[0053] Figure 7 This is a schematic diagram of the electronic device in Embodiment 3 of the present invention. Detailed Implementation

[0054] To make the objectives and technical solutions of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0055] Example 1: This example details the implementation process of the "Brain-Inspired Multi-Expert Multimodal Emotion Recognition Method" proposed in this invention on a computer platform. This method simulates the processing mechanism of functional brain regions, constructing a hybrid model composed of multiple specialized expert networks to achieve high-precision, highly interpretable emotion state recognition based on multimodal physiological signals (primarily EEG signals, supplemented by peripheral physiological signals). This example will fully demonstrate the entire process from data input to result output according to the steps defined in the invention description, and rigorously verify its effectiveness and advancement through experiments on standard publicly available datasets.

[0056] Combination Figure 1 It includes the following steps:

[0057] Step S1: Multimodal physiological signal preprocessing: The raw multi-channel physiological signals are standardized to provide clean, uniformly formatted data for subsequent model input. The specific implementation is as follows:

[0058] S101. Signal Filtering: Use a digital filter (such as a third-order Butterworth filter) to perform bandpass filtering on the raw EEG signal, for example, retaining the frequency components of 4-45Hz to remove noise such as DC offset, power frequency interference and high-frequency electromyography artifacts.

[0059] S102. Signal Segmentation: The continuous long-time sequence signal is segmented without overlap according to a fixed time window (e.g., 1 second) to form multiple independent signal samples. Each sample contains multi-channel data of a fixed time length.

[0060] S103. Signal normalization: Perform Z-Score standardization on the time-series signal of each channel, that is, subtract the mean of the channel over all samples and divide by its standard deviation to eliminate baseline differences and amplitude scale differences between different subjects and different recording sessions.

[0061] Output: The preprocessed multi-channel signal matrix set is obtained, denoted as... Each of them Representing the One sample, This represents the total number of channels. For time points.

[0062] Step S2, Brain Region Expert Division and Signal Allocation: Based on prior knowledge of neuroanatomy, the whole brain EEG signals are divided into different functional brain regions, and an EEG expert network is assigned to each region for processing. At the same time, independent experts are set up for peripheral physiological signals.

[0063] S201. Brain Region Division Definition: Based on the international 10-20 EEG electrode placement system and its corresponding brain functional areas, scalp electrodes are divided into multiple subsets. Typical divisions include: prefrontal cortex (related to higher cognition and emotion assessment), central cortex (related to sensorimotor function and arousal), parietal lobe, occipital lobe (related to visual processing), temporal lobe (related to hearing and memory), and whole-brain regions (used to capture global coordination patterns). each region Corresponding to a predefined set of electrode channel indices .

[0064] S202, Signal Assignment: For each input sample Based on the above division, extract the first... The EEG signal submatrix corresponding to each brain region:

[0065] ;

[0066] in It is a region The number of channels. Simultaneously, all peripheral physiological signals (such as ECG, PSS, etc.) are distributed to the 7th expert network. :

[0067] .

[0068] Output: Seven sets of signals are obtained, which serve as seven expert networks. Input.

[0069] Step S3, Global-Local Dual-Stream EEG Feature Extraction: Combining Figure 3 Each EEG expert network to All of them use the same global-local dual-stream encoder structure to collaboratively extract spatial topological features and local temporal dynamic features, and then fuse them to obtain the EEG expert features of the corresponding brain regions.

[0070] S301, Global Spatial Topological Feature Extraction:

[0071] 1. Functional connection graph construction: For input Calculate the weighted phase lag exponents among all channel pairs to form a symmetric weighted adjacency matrix. This matrix quantifies the strength of functional connectivity between channels in different brain regions.

[0072] 2. Graph Convolutional Coding: Converting the signal matrix... and adjacency matrix The input is fed into a multi-layer graph convolutional network. This network learns global feature representations containing complex spatial interaction patterns by performing message passing and aggregation on the functional connectivity graph. .

[0073] S302, Local Temporal Dynamic Feature Extraction:

[0074] At the same time, the signal Input a parallel, lightweight one-dimensional convolutional neural network. This network operates in the time dimension using small convolutional kernels, focusing on extracting the fine dynamic patterns of each electrode channel's own evolution over time, and outputting local feature representations. .

[0075] S303, Feature Fusion:

[0076] Global features are processed through a multi-head self-attention module. Further refinement is performed to enhance the weights of key channels. Then, the refined global features are combined with the local features processed by the feedforward network. Concatting is performed to form the EEG expert characteristics of this brain region. .

[0077] Output: Each EEG expert network outputs a high-dimensional feature vector. .

[0078] Step S4, Peripheral physiological signal feature extraction: combined with Figure 4 Peripheral Physiology Experts Network Peripheral signals are processed using multi-scale large-kernel convolution modules. .

[0079] S401, Multi-scale Convolution Extraction: The module uses multiple one-dimensional convolutional layers with different kernel sizes (e.g., sizes 15 and 11) in parallel to perform convolution operations on the input signal. Larger kernels can capture trends and slowly varying patterns over longer periods in the signal, while parallel processing at different scales aims to cover a variety of temporal dependencies.

[0080] S402. Feature Projection and Fusion: The outputs of each convolutional path are concatenated, then processed by batch normalization (BN) and ReLU activation functions. Finally, a multi-layer perceptron is used to project the multi-scale features into a unified low-dimensional embedding space, yielding the peripheral physiological features output by the peripheral physiological expert network. .

[0081] Output: Peripheral physiological feature vectors output by the peripheral physiological expert network. .

[0082] Step S5, Adaptive Multi-Expert Feature Fusion: Combining Figure 5 It intelligently integrates heterogeneous features from all seven experts through a learnable routing network.

[0083] S501, Feature Concatenation: Concatenate the output features of all experts into a comprehensive feature vector:

[0084] ;

[0085] S502, Dynamic Weight Generation and Fusion: [This section appears to be incomplete and requires further context.] Input a lightweight routing network. This routing network typically consists of several fully connected layers and a softmax output layer. It automatically learns and outputs a set of weights based on the characteristics of the current input sample. ,in Indicates the first The importance of individual expert features. The fusion features ultimately used for classification. The result is obtained by weighted summation of the characteristics of each expert:

[0086] ;

[0087] in For possible linear projection functions, For the first The feature vector output by the expert network.

[0088] S503, Sentiment Classification: Integrating Features The input is fed into a classifier (such as a fully connected layer + Softmax), which outputs the predicted probability distributions corresponding to different emotional dimensions (such as valence and arousal). .

[0089] Output: Sentiment state classification prediction results .

[0090] Step S6, Joint Optimization and Classification Output: To train this complex model and ensure effective collaboration among experts, a multi-objective joint loss function is adopted.

[0091] S601. Loss Function Calculation:

[0092] Classification loss ( The main loss function is Focal Loss. Based on the standard cross-entropy loss, the weight of easily classified samples is reduced by a modulation factor, so that the global-local dual-stream encoder and the multi-scale large-kernel convolution module can focus more on the difficult-to-classify samples, which helps to deal with the class imbalance problem.

[0093] Expert load balancing losses ( This function operates on the routing network to calculate the frequency (utilization) of each expert being selected by the network in a training batch, and encourages a near-uniform distribution of utilization across all experts. This prevents some experts from being "neglected" while others are "overused," promoting a more balanced division of labor among experts.

[0094] Expert disagreement regularization loss ( This function operates on the global-local dual-stream encoder and the multi-scale large-kernel convolution module to calculate the symmetric KL divergence between different expert outputs and maximize this difference. This encourages each expert to learn complementary, non-redundant feature representations, enhancing the overall representational power of the model.

[0095] The total loss is the weighted sum of the three: Among them, hyperparameters , Used to balance various losses to similar orders of magnitude; classification loss function , The model represents the true category. The predicted probability, and These are the category balance weights and the focus parameters, respectively. Sample size for training batches; expert load balancing loss , , For the first The dynamic weights corresponding to each expert network , For the first Utilization rate of expert networks; expert disagreement regularization loss , For symmetric KL divergence, For the first The output of an expert network is normalized by a Softmax layer.

[0096] S602, Model Training: Minimize the total loss using the backpropagation algorithm and the Adam optimizer. Update all network parameters (including those for each expert network and routing network).

[0097] Output: All parameters of the trained network model.

[0098] Step S7, Interpretability Analysis: After training is completed, use model interpretability tools (such as SHAP) to analyze the decision-making basis of the model.

[0099] S701. Contribution Analysis: For a given input sample, calculate the contribution (Shapley value) of each expert network feature to the final classification decision.

[0100] S702, Result Visualization: Generate visual charts (such as...) Figure 6 As shown in the diagram, it visually demonstrates which brain regions (corresponding experts) play a dominant role in different emotion recognition tasks. For example, it can verify whether the prefrontal cortex contributes significantly in valence assessment and whether the central region is more critical in arousal assessment.

[0101] Output: A contribution analysis report and visualization results of the model's decisions, used to verify whether the model conforms to neuroscience cognition.

[0102] Table 1. Electrode partitioning schemes on different datasets

[0103]

[0104] To objectively evaluate the performance of the method of this invention, this embodiment selects two widely recognized public benchmark datasets in the field of multimodal sentiment analysis for rigorous experiments: the DEAP dataset and the DREAMER dataset. The specific electrode partitioning schemes for the corresponding EEG expert networks are shown in Table 1. Classification accuracy is used as the primary evaluation metric. The validation protocol adopts Leave-One-Subject-Out (LOSO), where all data from one subject is used as the test set, and the data from the remaining subjects is used as the training set. The average accuracy is calculated after testing all subjects in turn. This protocol rigorously simulates the application scenario of new users (who have not seen the data) in real-world scenarios to evaluate the model's generalization ability. Baseline comparisons select representative methods published in recent years as baselines, including advanced models based on convolutional neural networks, graph neural networks, and multimodal fusion.

[0105] (I) Results on the DEAP dataset: The DEAP dataset includes classification tasks across four affective dimensions: valence, arousal, dominance, and liking, as shown in Table 2. Whether using only EEG signals or fusing EEG signals with peripheral physiological signals, the method of this invention significantly and consistently outperforms all baseline methods in all four dimensions. For example, in a multimodal setting, the method of this invention improves upon the suboptimal method by 0.87% to 1.40% in each of the four dimensions, achieving new state-of-the-art performance.

[0106] (II) Results on the DREAMER dataset: The DREAMER dataset includes a binary classification task with two affective dimensions: valence and arousal. The experimental results are shown in Table 3. Similarly, in both unimodal and multimodal settings, the method of this invention achieved significant improvements in accuracy across both dimensions, especially in the arousal recognition task, where the improvement exceeded 5% compared to the baseline method.

[0107] Table 2. Results (%) of independent experiments conducted on the DEAP dataset.

[0108]

[0109] Table 3. Results of subject-independent experiments on the DREAMER dataset (%)

[0110]

[0111] [1]. Fo Hu, Kailun He, Can Wang, Qinxu Zheng, Bin Zhou, Gang Li, and Yu Sun, “Strflnet: Spatio-temporal representation fusion learning network for eeg-based emotion recognition,” IEEE Transactions on Affective Computing, 2025.

[0112] [2]. Yi Ding, Neethu Robinson, Su Zhang, Qiuhao Zeng, and Cuntai Guan, “Tsception: Capturing temporal dynamics and spatial asymmetry from eeg for emotion recognition,” IEEE Transactions on Affective Computing, vol. 14, no. 3, pp. 2238–2250, 2022.

[0113] [3]. Aurélien Appriou, Andrzej Cichocki, and Fabien Lotte, “Modern machine-learning algorithms: for classifying cognitive and affective states from electroencephalography signals,” IEEE Systems, Man, and Cybernetics Magazine, vol. 6, no. 3, pp. 29–38, 2020.

[0114] [4].Xinda Li, “Tacoformer: Token-channel compounded cross attentionfor multimodal emotion recognition,” arXiv preprint arXiv:2306.13592, 2023.

[0115] [5].Jing Li, Ning Chen, Hongqing Zhu, Guangqiang Li, Zhangyong Xu,and Dingxin Chen, “Incongruity-aware multimodal physiology signals fusion foremotion recognition,” Information Fusion, p. 102220, 2024.

[0116] [6].Dongdong Li, Shengyao Huang, and Zhe Wang, “At-pmf: Progressivemulti-modal fusion with adversarial training for physiological emotionrecognition,” Pattern Recognition, p. 112690, 2025.

[0117] [7].Tianzuo Xin, Jing Wang, Xiyuan Jin, Xiaojun Ning, Zhiyang Feng,and Youfang Lin, “Mocernet: A modality-complete modeling framework foremotion recognition in physiological signals under imperfect modal matching,”in Proceedings of the 33rd ACM International Conference on Multimedia, 2025,pp. 5687–5696.

[0118] [8].Yi Wu, Yuhang Chen, Jiahao Cui, Jiaji Liu, Lin Liang, and ShuaiLi, “Cross-modal contrastive learning for emotion recognition: Aligning ecgwith eeg-derived features,” in International Conference on Medical ImageComputing and Computer-Assisted Intervention. Springer, 2025.

[0119] [9].Wei Li, Ye Tian, Bowen Hou, Jianzhang Dong, and Shitong Shao,“Bismsm: A hybrid mlp-based model of global self-attention processes for eeg-based emotion recognition,” in International Conference on Artificial NeuralNetworks. Springer, 2022, pp. 37–48.

[0120]

[10] .Zhipeng He, Yongshi Zhong, and Jiahui Pan, “An adversarialdiscriminative temporal convolutional network for eeg-based cross-domainemotion recognition,” Computers in biology and medicine, vol. 141, 2022.

[0121]

[11] . Chongxing Shi, CL Philip Chen, Shuzhen Li, and Tong Zhang, “Functional connectivity patterns learning for eeg-based emotion recognition,” IEEE Transactions on Cognitive and Developmental Systems, 2024.

[0122] (III) Ablation Experiment Analysis: In order to verify the necessity of each core component of the present invention, a systematic ablation experiment was conducted on the DEAP dataset, and the results are shown in Table 4.

[0123] Removing brain region expert divisions (i.e., using a single expert to process whole-brain signals): performance significantly decreased, demonstrating the effectiveness of targeted modeling based on brain functional regions.

[0124] Replacing wPLI with the phase hysteresis index (PLI): performance consistently declines across all dimensions, validating the superiority of wPLI in constructing robust functional connectivity graphs.

[0125] Removing the self-attention mechanism resulted in performance degradation, indicating that the attention mechanism is crucial for capturing critical channel dependencies.

[0126] Remove expert load balancing loss ( The model performance deteriorated, especially in certain dimensions, confirming the importance of this loss in maintaining balanced collaboration among experts.

[0127] Remove expert disagreement regularization loss ( The slight performance decrease indicates that encouraging expert diversity has a positive effect on improving model robustness.

[0128] These ablation experiments collectively demonstrate that the brain-inspired architecture, wPLI-GCN module, attention mechanism, and joint optimization strategy proposed in this invention are all indispensable parts for achieving high-performance emotion recognition.

[0129] Table 4. Results of ablation experiments on the DEAP dataset (%)

[0130]

[0131] (iv) Verification of interpretability: such as Figure 6As shown, the SHAP analysis results for the best participants indicate that experts from the central region and prefrontal cortex made the most significant contributions in the arousal classification task. This finding is highly consistent with prior knowledge in neuroscience that these brain regions are primarily involved in arousal regulation and emotion assessment, thus verifying the physiological rationality of the decision-making process of the model in this invention from a computational perspective and greatly enhancing the model's credibility.

[0132] In summary, this embodiment, through detailed step descriptions and rigorous experiments on standard datasets, fully demonstrates that the method of the present invention can not only achieve high-precision multimodal emotion recognition, but also that its internal working mechanism has good neural interpretability, effectively overcoming the shortcomings of the prior art.

[0133] Example 2: This example aims to illustrate how to construct a complete and operational emotion recognition system using the method described in Example 1. This system integrates signal acquisition, processing, modeling, analysis, and visualization functions, and can be deployed on servers, workstations, or embedded platforms for scientific research or practical applications, such as mental health monitoring platforms or brain-computer interface systems. Therefore, this example provides "a brain-inspired multi-expert multimodal emotion recognition system."

[0134] like Figure 2 As shown, the system consists of a signal acquisition module 1, a preprocessing module 2, an expert segmentation module 3, a feature extraction module 4, a dynamic output module 5, a loss optimization module 6, and an interpretability module 7. Each module strictly corresponds to a method step and is connected in series via data flow to form an end-to-end processing pipeline. Specifically:

[0135] 1. Signal Acquisition Module 1: Responsible for acquiring raw physiological signals from hardware sensors in real time or offline. It is typically connected to multi-channel EEG acquisition devices (such as EEG signal caps placed according to the international 10-20 system) and various peripheral physiological signal acquisition devices (such as electrocardiographs, electrodermal analyzers, etc.).

[0136] Input: Analog or digital electrical signals from physical sensors.

[0137] Output: Synchronized multi-channel digital signal stream or data file.

[0138] Connection: The output is directly transmitted to preprocessing module 2.

[0139] 2. Preprocessing module 2: Implements all operations in step S1 of the invention. Performs quality control on the raw signal, including filtering and denoising, segmentation, normalization calibration, etc.

[0140] Input: The raw signal output by signal acquisition module 1.

[0141] Output: A normalized signal matrix that can be directly used as model input.

[0142] Connection: Output is transmitted to expert segmentation module 3.

[0143] 3. Expert Division Module 3: Implements the operation of step S2 in the invention. Based on the pre-stored brain region-electrode mapping configuration table, the input EEG signal matrix is ​​split into sub-matrices corresponding to different brain regions, and peripheral physiological signals are separated.

[0144] Input: The complete signal matrix output by preprocessing module 2.

[0145] Output: Multiple signal subsets, each corresponding to an expert network.

[0146] Connections: The outputs are connected to the corresponding sub-modules in feature extraction module 4.

[0147] 4. Feature Extraction Module 4: Implements the operations in steps S3 and S4 of the invention. This is the core computational module of the system, containing 7 parallel sub-modules: 6 EEG expert network sub-modules: Each sub-module instantiates a global-local dual-stream encoder, receives EEG signals of a specific brain region from expert segmentation module 3, and outputs the EEG expert features of that brain region. 1 Peripheral Physiology Expert Network Sub-module: Instantiates a multi-scale large-kernel convolution module, processes all peripheral physiological signals, and outputs peripheral physiological features.

[0148] The global-local dual-stream encoder has two parallel branches: one branch is a weighted phase lag exponent calculated cascaded graph convolutional network followed by a multi-head self-attention mechanism, corresponding to the extraction of spatial topological features; the other branch is a one-dimensional convolutional network followed by a feedforward network, corresponding to the extraction of local temporal features; the two branches are concatenated by Concat to achieve EEG expert feature output.

[0149] The multi-scale large-kernel convolution module consists of multiple one-dimensional convolutional layers of different kernel sizes connected in parallel, sequentially connected to perform Batch Normalization, ReLU activation, and a multilayer perceptron.

[0150] Input: 7 subsets of signals from expert partitioning module 3.

[0151] Output: 7 high-dimensional feature vectors.

[0152] Connection: All feature vectors are output to dynamic output module 5.

[0153] 5. Dynamic Output Module 5: Implements the operation of step S5 in the invention description. Its core is a lightweight routing network. This module receives all expert features, generates a set of dynamic weights through its internal learnable parameters, performs weighted fusion of the expert features, and finally outputs a preliminary prediction of sentiment state through a classification layer.

[0154] Input: 7 feature vectors output by feature extraction module 4.

[0155] Output: The predicted probability distribution of sentiment categories.

[0156] Connection: During the training phase, the output is sent to the loss optimization module 6 for loss calculation; during the inference phase, the output can be used directly as the result or sent to the interpretability module 7 for analysis.

[0157] 6. Loss Optimization Module 6: Implements the operation of step S6 in the invention description. This module is only activated during the system training phase. It receives the prediction results and true labels from the dynamic output module 5. Based on the predefined joint loss function ( The loss value is calculated, and the loss gradient is backpropagated to the dynamic output module 5 and the feature extraction module 4 using optimization algorithms (such as Adam) and backpropagation to update the parameters of the entire network model.

[0158] Input: The predicted output of Dynamic Output Module 5 and the true sentiment label.

[0159] Output: loss value, gradient signal, updated model parameters.

[0160] Connection: Forms a closed-loop optimization circuit with the dynamic output module 5 and the feature extraction module 4.

[0161] 7. Interpretability Module 7: Implements the operation of step S7 in the invention description. After model training is completed or during inference, this module uses interpretable AI tools (such as integrating the SHAP library) to analyze the system's decision-making process. It inputs a sample and its corresponding expert features and routing weights, calculates the contribution of each expert feature to the final classification decision, and generates visual charts, such as feature contribution bar charts or brain topographic maps, to intuitively show which brain regions play a key role in the decision-making process.

[0162] Input: Feature output of feature extraction module 4, weights and prediction results of dynamic output module 5.

[0163] Outputs: Interpretability analysis report, visualization charts.

[0164] Connection: Receives data from feature extraction module 4 and dynamic output module 5, and provides interpretation results to the user interface.

[0165] The system's workflow is clearly divided into two phases: training and inference. During the training phase, the system administrator first imports the labeled historical multimodal physiological signal dataset. The data then flows sequentially through preprocessing module 2, expert segmentation module 3, feature extraction module 4, and dynamic output module 5, completing forward propagation. During this process, loss optimization module 6 is activated. It calculates the joint loss based on the model's predicted output and the real sentiment labels, and iteratively updates all network parameters within the system using the backpropagation algorithm until the model performance converges. Finally, the optimal model parameter file is saved.

[0166] During the reasoning and application phase, the user wears a physiological signal acquisition device, and signal acquisition module 1 begins acquiring data in real time. The acquired raw signals are batched and sent to preprocessing module 2 for real-time filtering, segmentation, and normalization. Subsequently, expert segmentation module 3 allocates the signals to preset brain regions. The allocated signals are then rapidly processed in parallel by pre-trained expert networks in feature extraction module 4 to extract deep features. These features are then input into a pre-trained routing network in dynamic output module 5 for dynamic weighted fusion and classification, outputting specific emotional state recognition results in real time (e.g., "high valence - high arousal"). Simultaneously, interpretability module 7 can run concurrently, generating an analysis chart revealing the contribution of each brain region, providing users or professional analysts with an intuitive and reliable reference.

[0167] This system boasts flexible deployment capabilities, adaptable to diverse application scenarios. Specifically, it can be deployed on cloud servers, where the trained model is packaged and provided with API services, facilitating data uploads and result reception on mobile devices, suitable for large-scale mental health screening. Alternatively, it can be deployed on edge computing devices, such as integrated into NVIDIA Jetson series hardware, to achieve low-latency, high-privacy-protection localized real-time emotion recognition, suitable for intelligent cockpit emotion monitoring or personalized interaction systems. Furthermore, it can be deployed on high-performance desktop workstations, primarily serving research scenarios, facilitating in-depth model debugging and data visualization analysis for researchers.

[0168] A typical application example is the construction of a dynamic monitoring platform for mental health status based on this system. In this scenario, users wear portable EEG headbands and wrist-worn physiological sensors in clinics or at home. The system operates as a backend service or local edge computing, continuously analyzing the collected physiological signals in real time and periodically generating emotional state reports including valence and arousal trend graphs. The platform's core functions include: visually tracking users' emotional fluctuations throughout the day; automatically triggering alerts when the system continuously detects a "low valence, high arousal" state that may indicate anxiety; combining the analysis in Interpretability Module 7 to assess the stabilization of activity patterns in specific brain regions after users receive interventions (such as mindfulness meditation), providing objective neurophysiological evidence for the intervention's effectiveness; and assisting researchers in exploring potential physiological markers of emotional disorders and achieving data-driven in-depth insights through the analysis of desensitized aggregated data.

[0169] Example 3: For the scenario in Example 1, Figure 7 A schematic diagram of an electronic device 90 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0170] Electronic devices can also refer to various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0171] like Figure 7 As shown, the electronic device 90 includes at least one processor 91 and a memory, such as a read-only memory (ROM) 92 or a random access memory (RAM) 93, communicatively connected to the at least one processor 91. The memory stores computer programs executable by the at least one processor. The processor 91 can perform various appropriate actions and processes based on the computer program stored in the ROM 92 or loaded into the RAM 93 from storage unit 98. The RAM 93 can also store various programs and data required for the operation of the electronic device 90. The processor 91, ROM 92, and RAM 93 are interconnected via a bus 94. An input / output (I / O) interface 95 is also connected to the bus 94.

[0172] Multiple components in electronic device 90 are connected to I / O interface 95, including: input unit 96, such as keyboard, mouse, etc.; output unit 97, such as various types of displays, speakers, etc.; storage unit 98, such as disk, optical disk, etc.; and communication unit 99, such as network card, modem, wireless transceiver, etc. Communication unit 99 allows electronic device 90 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0173] Processor 91 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 91 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 91 performs the various methods and processes described above, such as a brain-inspired multi-expert multimodal emotion recognition method.

[0174] In some embodiments, a brain-inspired multi-expert multimodal emotion recognition method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 98. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 90 via ROM 92 and / or communication unit 99. When the computer program is loaded into RAM 93 and executed by processor 91, one or more steps of the brain-inspired multi-expert multimodal emotion recognition method described above may be performed. Alternatively, in other embodiments, processor 91 may be configured to perform a brain-inspired multi-expert multimodal emotion recognition method by any other suitable means (e.g., by means of firmware).

[0175] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0176] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0177] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0178] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0179] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0180] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0181] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A brain-inspired multi-expert multi-modal emotion recognition method, characterized in that, Includes the following steps: Step S1: Multimodal physiological signal preprocessing: Acquire multi-channel EEG signals and peripheral physiological signals, and perform preprocessing. Step S2, Brain Region Expert Division and Signal Allocation: Based on the international 10-20 system, the brain is divided into multiple functional regions, each region corresponding to an EEG expert network, and a corresponding subset of EEG signal channels is allocated; at the same time, an independent peripheral physiological expert network is allocated to peripheral physiological signals; the collection of all EEG expert networks and peripheral physiological expert networks is collectively referred to as the expert network. Step S3, Global-Local Dual-Stream EEG Feature Extraction: Each EEG expert network is a global-local dual-stream encoder, used to extract the spatial topological features and local temporal features of the EEG signals of the corresponding brain region, respectively. Finally, the EEG expert features of the corresponding brain region are obtained by fusing them through the attention mechanism. Step S4, Peripheral physiological signal feature extraction: The peripheral physiological expert network is a multi-scale large kernel convolution module. It uses convolution kernels of different scales to extract long-term temporal dependence features in peripheral physiological signals to obtain peripheral physiological features. Step S5, Adaptive Multi-Expert Feature Fusion: The features output by all expert networks are concatenated and input into a lightweight routing network. The network generates dynamic weights for all expert networks. Then, the output features of all expert networks are adaptively weighted and fused to obtain the sentiment state classification result. Step S6, Joint Optimization and Classification Output: Design a joint loss function, including classification loss, expert load balancing loss and expert divergence regularization loss, optimize the model parameters through gradient descent, and finally output the sentiment state classification result; The global-local dual-stream encoder contains two parallel branches: one branch is a weighted phase lag exponent calculated cascaded graph convolutional network followed by a multi-head self-attention mechanism, corresponding to the extraction of spatial topological features; the other branch is a one-dimensional convolutional network followed by a feedforward network, corresponding to the extraction of local temporal features; the two branches are concatenated by Concat to achieve EEG expert feature output; The multi-scale large-kernel convolution module consists of multiple one-dimensional convolutional layers of different kernel sizes connected in parallel, sequentially connected to perform Batch Normalization, ReLU activation, and a multilayer perceptron.

2. The brain-inspired multi-expert multi-modal emotion recognition method according to claim 1, characterized in that, The preprocessing described in step S1 is as follows: S101: Use a third-order Butterworth bandpass filter to filter EEG signals and remove noise and artifacts; S102: Divide the continuous signal into a non-overlapping 1-second time window; S103: Perform Z-Score normalization on each channel to eliminate individual differences and equipment bias.

3. The brain-inspired multi-expert multimodal emotion recognition method according to claim 1, characterized in that, Step S2 describes dividing the brain into six functional regions based on the international 10-20 system: prefrontal cortex, central region, parietal lobe, occipital lobe, temporal lobe, and whole brain, each corresponding to a separate EEG expert network. The corresponding channel signal subset matrix is ​​extracted from each functional region and used as the input of the corresponding EEG expert network.

4. The brain-inspired multi-expert multimodal emotion recognition method according to claim 1, characterized in that, The working principle of the global-local dual-stream encoder described in any step S3 is as follows: S301: The functional connectivity map of the EEG signal of the corresponding brain region is calculated based on the weighted phase lag index, which is a symmetric weighted adjacency matrix. Then, spatial topological features are extracted through a graph convolutional network. S302: Extract local temporal features from the EEG signals of the corresponding brain regions using a one-dimensional convolutional network; S303: Global attention features corresponding to spatial topological features are calculated through a multi-head self-attention mechanism, local features corresponding to local temporal features are calculated through a feedforward network, and then the global attention features and local features are concatenated to obtain the EEG expert features of the corresponding brain region.

5. The brain-inspired multi-expert multimodal emotion recognition method according to claim 1, characterized in that, The specific steps of S4 are as follows: S401: Extracting multi-scale features of peripheral physiological signals by using multiple one-dimensional convolutional layers with different kernel sizes in parallel; S402: After normalization and ReLU activation, multi-scale features are projected onto a unified dimension using a multilayer perceptron to obtain peripheral physiological features.

6. The brain-inspired multi-expert multimodal emotion recognition method according to claim 1, characterized in that, The lightweight routing network described in step S5 consists of a multilayer perceptron connected in series with a Softmax layer.

7. The brain-inspired multi-expert multimodal emotion recognition method according to claim 1, characterized in that, The joint loss function described in step S6 consists of classification loss. Expert load balancing losses and expert disagreement regularization loss It consists of three parts, namely Among them, hyperparameters , Used to balance various losses to similar orders of magnitude; classification loss function , The model represents the true category. The predicted probability, and These are the category balance weights and the focus parameters, respectively. Sample size for training batches; expert load balancing loss , , For the first The dynamic weights corresponding to each expert network , For the first Utilization rate of expert networks; expert disagreement regularization loss , For symmetric KL divergence, For the first The output of an expert network is normalized by a Softmax layer.

8. A brain-inspired multi-expert multimodal emotion recognition system, characterized in that, It consists of a signal acquisition module (1), a preprocessing module (2), an expert segmentation module (3), a feature extraction module (4), a dynamic output module (5), a loss optimization module (6), and an interpretability module (7); The signal acquisition module (1) has its input end connected to the multimodal sensor and its output end connected to the input end of the preprocessing module (2), and is used to acquire multi-channel EEG signals and peripheral physiological signals. The output of the preprocessing module (2) is connected to the input of the expert segmentation module (3) to perform filtering, segmentation and normalization processing on the collected signals; The expert division module (3) has its output end connected to the input end of the feature extraction module (4), and divides the EEG channels according to the brain region function and assigns them to the corresponding experts; The feature extraction module (4) has its output end connected to the input end of the dynamic output module (5) and the interpretability module (7), and includes multiple EEG expert networks and a peripheral physiological expert network connected in parallel for different brain regions. The EEG expert network uses a global-local dual-stream encoder to extract EEG expert features of the EEG signals of the corresponding brain region signal channels, and the peripheral physiological expert network uses a multi-scale large kernel convolution module to extract peripheral physiological features of the peripheral physiological signals. The dynamic output module (5) is a lightweight routing network. Its output end is connected to the input end of the loss optimization module (6) and the interpretability module (7) to realize the dynamic weighted fusion of multi-expert features. The loss optimization module (6) is connected to the feature extraction module (4) and the dynamic output module (5) at its output end. It achieves dynamic optimization of the weights of each expert network and the lightweight routing network based on the joint loss function. The interpretability module (7) enables the visualization and interpretable analysis of emotional state decisions and provides feedback to the user.

9. An electronic device, characterized in that, The electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a brain-inspired multi-expert multimodal emotion recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the brain-inspired multi-expert multimodal emotion recognition method according to any one of claims 1 to 7.

Images