Multi-modal emotion recognition method based on group encoder and graph distillation

Abstract

The invention relates to the field of artificial intelligence and physiological signal processing, and particularly discloses a multi-modal emotion recognition method based on a group encoder and graph distillation, and the method comprises the steps: firstly, carrying out the grouping of subjects; secondly, extracting and decomposing initial features of the multi-modal physiological signals to obtain modal invariant features and modal specific features; further, by constructing an independent graph distillation unit, coarse-grained alignment is performed on modal invariant features in a frequency domain, fine-grained alignment is performed on modal specific features in a time domain, and cross-modal knowledge migration and fusion are realized; and finally, performing group perceptual coding and classification on the fused features by using the tested shared encoder and the group specific encoder. According to the method, deep semantic fusion and dynamic association modeling of cross-modal signals are realized through a collaborative mechanism of dual-granularity graph distillation and group coding, and the emotion recognition accuracy, the emotion continuous change depicting ability and the generalization performance of the model facing a new subject are effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and physiological signal processing, specifically a multimodal emotion recognition method based on group encoders and graph distillation. Background Technology

[0002] Emotion recognition is a key technology in human-computer interaction, mental health assessment, and neuroscience research. Physiological signal-based emotion recognition methods have attracted widespread attention because they can reflect internal states that are not entirely under conscious control. Commonly used physiological signals include electroencephalography (EEG), electromyography (EMG), and genomic skin conductance (GSR), which respectively reflect brain neural activity, muscle contraction intensity, and sympathetic nerve excitation levels, providing complementary evidence of emotional states from different physiological perspectives.

[0003] To achieve accurate emotion decoding, researchers typically employ multimodal fusion strategies, aiming to integrate information from different sources. Mainstream methods are broadly categorized into feature-level fusion and decision-level fusion. Feature-level fusion usually involves extracting features from each modality and then performing simple concatenation or weighted combination; decision-level fusion, on the other hand, first classifies each modality independently and then integrates the classification results. However, these methods are inherently "loosely coupled," treating different modalities as independent information streams for later merging, without explicitly modeling the inherent physiological connections between signals during the fusion process. For example, during emotion induction, there is a synergistic and causal relationship between the excitation of the central nervous system (manifested through partial rhythmic changes in EEG) and the arousal of the peripheral nervous system (manifested through changes in EMG and GSR). This deep, dynamic neurophysiological coupling is difficult to effectively capture and utilize within existing shallow fusion frameworks.

[0004] Therefore, existing technologies face a core deficiency in multimodal emotion recognition: the fusion process fails to achieve deep semantic alignment and decoupling of cross-modal features. This results in the model being unable to fully exploit the complementarity between heterogeneous physiological signals, struggling to characterize the dynamic evolution of emotions over time, and having limited generalization ability due to overfitting to specific individual or group data. Designing a fusion mechanism that can simultaneously model shared emotional semantics across modalities and the unique dynamic response patterns of each modality becomes crucial for improving system performance and practicality. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a multimodal emotion recognition method based on group encoders and graph distillation. This method solves the problems in existing technologies where modality fusion remains at a shallow level, cross-modal dynamic associations are not effectively modeled, and the model heavily relies on individual labeled data, resulting in weak generalization ability.

[0006] Reference Figure 1This invention provides a multimodal emotion recognition method based on group encoders and graph distillation, comprising the following steps:

[0007] S1. Initial Feature Extraction and Decomposition: Obtain the raw data of the subject's EEG, EMG and TE signals, and extract the initial features of each modality through a dedicated feature extraction network; input the initial features of each modality into the modality-invariant encoder and the modality-specific encoder to decompose and obtain the modality-invariant features and modality-specific features of each modality;

[0008] S2. Cross-modal coarse-grained fusion:

[0009] S2.1. Coarse-grained alignment: The modality-invariant features of each mode are transformed to the frequency domain and input into the first graph distillation unit for cross-modal knowledge distillation to achieve coarse-grained time alignment, obtain coarse-grained fusion features, and calculate coarse-grained distillation loss;

[0010] S2.2. Fine-grained alignment: The modality-specific features of each mode are input into the second graph distillation unit in the time domain to perform cross-modal knowledge distillation, thereby achieving fine-grained time alignment, obtaining fine-grained fusion features, and calculating fine-grained distillation loss;

[0011] S2.3. Feature aggregation: The coarse-grained fused features, fine-grained fused features, and modality-specific features of each modality are aggregated and fused through a multi-head self-attention mechanism to obtain the subject-level feature representation;

[0012] S3. Group-based emotion classification: The subject-level feature representation is simultaneously input into a subject-shared encoder and a group encoder corresponding to the current subject's group, and subject-shared features and group-specific features are extracted respectively; the subject-shared features and group-specific features are fused, and the emotion classification result is obtained through a classifier;

[0013] The first graph distillation unit and the second graph distillation unit are used to construct a fully connected graph with each modal feature as a node. By optimizing the product norm of the edge weight matrix and the distillation loss matrix, cross-modal knowledge transfer is achieved.

[0014] Preferably, before step S1, step S0 is included: subject grouping: a correlation coefficient matrix between subjects is calculated based on the EEG signal data of multiple subjects, and the subjects are divided into K groups according to the matrix.

[0015] Preferably, in step S1, modality alignment constraints and orthogonality constraints are applied to the modality-invariant features; the modality alignment constraints are achieved by minimizing the difference in central moments between the distributions of modality-invariant features of different modes; the orthogonality constraints are achieved by minimizing the sum of squared Frobenius inner products between the modality-invariant features and each modality-specific feature, as well as between each modality-specific feature and each other.

[0016] Preferably, in step S2.1, the mode-invariant features are converted to the frequency domain using the Fast Fourier Transform.

[0017] Preferably, in step S3, the parameters of the subject-shared encoder are shared among all subjects; the parameters of the group encoder are shared within their respective groups and are independent between different groups.

[0018] A training method for a multimodal emotion recognition model, used to train a model as described above for emotion recognition, includes the following steps:

[0019] Acquire multimodal physiological signal data and their emotional labels from multiple subjects;

[0020] The correlation between subjects was calculated based on the EEG signal data of all subjects, and the subjects were divided into multiple groups according to the correlation coefficient matrix.

[0021] Construct a model architecture for the emotion recognition method described above;

[0022] Using the multimodal physiological signal data as input and emotion labels as supervision, the model is trained end-to-end by combining the total loss function consisting of emotion classification task loss, modality alignment loss, modality differentiation loss, coarse-grained distillation loss, and fine-grained distillation loss.

[0023] Preferably, the specific steps for calculating the correlation between subjects include: segmenting the EEG data of each subject according to a preset time window and overlap rate; calculating the power spectral density characteristics of all channels in each data segment; calculating the correlation coefficient matrix between each pair of subjects based on the power spectral density characteristics of all subjects; and using a clustering algorithm to divide the subjects into K groups according to the correlation coefficient matrix.

[0024] A multimodal emotion recognition system based on group encoders and graph distillation includes:

[0025] The feature extraction and decomposition module is used to acquire the raw data of the subject's multimodal physiological signals, extract the initial features of each modality, and decompose them into modality-invariant features and modality-specific features of each modality.

[0026] The cross-modal fusion module includes:

[0027] The coarse-grained alignment submodule is used to transform the mode-invariant features of each mode to the frequency domain and to perform knowledge distillation and fusion using the first graph distillation unit;

[0028] The fine-grained alignment submodule is used to perform knowledge distillation and fusion of the modality-specific features of each modality in the time domain using the second graph distillation unit;

[0029] The feature aggregation submodule is used to fuse features after coarse and fine-grained alignment and generate subject-level feature representations through an attention mechanism.

[0030] The group-aware classification module includes a subject-shared encoder, multiple group encoders, and a classifier, used to perform shared and group-specific encoding on the subject-level feature representations, and to fuse the encoding results to complete emotion classification;

[0031] The first graph distillation unit and the second graph distillation unit are used to construct a fully connected graph with each modal feature as a node. By optimizing the product norm of the edge weight matrix and the distillation loss matrix, cross-modal knowledge transfer is achieved.

[0032] Preferably, in the feature extraction and decomposition module, the feature extraction network for EEG signals is a convolutional neural network based on EEGNet that includes temporal convolution, depthwise separable convolution, and an attention module; the feature extraction network for EMG and electrodermal signals is a lightweight convolutional network that includes multi-scale convolution and attention mechanisms.

[0033] Preferably, it also includes a model training module for performing the training method described above to optimize the parameters of the system.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] By introducing a unique architecture that combines group encoders and graph distillation, this invention achieves effective separation and coordination of modality-invariant and modality-specific information at the feature level, thereby simultaneously improving three aspects of multimodal emotion recognition: deep semantic fusion, dynamic evolution modeling, and cross-subject generalization ability. Specifically, this invention first constructs a dual-path graph distillation mechanism to align cross-modal shared and unique information in the frequency and time domains, respectively, enabling in-depth mining of the intrinsic correlation of heterogeneous physiological signals and overcoming the information loss problem of traditional shallow fusion.

[0036] By designing a dual encoder structure that combines parameter sharing and group specificity, the model naturally incorporates the ability to separately model common patterns and group-specific patterns, thereby significantly improving the model's adaptability and robustness when facing new subjects without requiring a large amount of individual labeled data. Ultimately, the entire method improves the accuracy of emotional state decoding, enhances the ability to depict the continuous process of emotional change, and provides the model with a better foundation for interpretability. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the multimodal emotion recognition method of the present invention;

[0038] Figure 2 This is a schematic diagram of the model structure of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Example 1: This example provides a complete multimodal emotion recognition method based on group encoders and graph distillation. Please refer to... Figure 2 This paper demonstrates the overall model architecture and process of the method of the present invention. The method mainly includes three core steps: initial feature extraction and decomposition, cross-modal coarse-grained fusion, and group-aware emotion classification. Optionally, a preprocessing step of subject grouping may also be included during the model training phase to construct a more accurate group encoder.

[0041] I. Optional pretreatment steps: Subject grouping

[0042] To improve the model's adaptability to individual differences, participants can be grouped based on the similarity of their physiological signals before training begins. The specific steps are as follows:

[0043] Data Acquisition and Segmentation: Resting-state or task-oriented electroencephalogram (EEG) data were collected from M subjects. For each subject's data, a sliding time window of 4 seconds with an overlap of 1 second was used to segment the data, resulting in multiple EEG data segments.

[0044] Feature Calculation: For each data segment, calculate the power spectral density (PSD) of each EEG channel. Frequency bands such as Delta (1-4Hz), Theta (4-8Hz), Alpha (8-13Hz), Beta (13-30Hz), and Gamma (30-45Hz) can be selected. The PSD features of all segments are averaged to obtain a representative feature vector for each subject.

[0045] Correlation analysis and grouping: The correlation coefficients (e.g., Pearson correlation coefficients) between each pair of feature vectors of all M subjects are calculated, forming an M×M correlation coefficient matrix. Finally, based on this matrix, clustering methods (e.g., K-means) are used to divide the subjects into K groups. This step aims to group subjects with similar physiological patterns into one category, providing a basis for the subsequent construction of the group encoder.

[0046] II. Core Identification Process

[0047] S1. Initial Feature Extraction and Decomposition

[0048] Data Acquisition and Preprocessing: Acquire raw synchronous multimodal physiological signal data from the target subjects, including electroencephalogram (EEG), electromyography (EMG), and conductance of skin surface (GSR) signals; perform standardized preprocessing on the raw signals, including: bandpass filtering (to remove noise and artifacts, EEG: 1-45 Hz, EMG: 20-200 Hz, GSR: 0.05-5 Hz), downsampling to a uniform frequency; and time window segmentation and alignment based on stimulus labels to obtain aligned multi-channel temporal data blocks. ,in These represent EEG, EMG, and GSR modes, respectively. , For the number of channels, The sequence length is given.

[0049] Dedicated Feature Extraction Network Construction and Feature Extraction:

[0050] EEG Feature Extraction Network: This network employs a convolutional neural network based on an improved version of the classic EEGNet architecture. It first uses temporal convolutional layers to extract frequency band features; then, depthwise separable convolutional layers capture spatial information; finally, a selective kernel attention module is introduced to adaptively fuse multi-scale features, thereby automatically extracting multi-dimensional deep features highly correlated with emotion across time, space, and frequency. .

[0051] EMG / GSR Feature Extraction Network: Employs a lightweight convolutional network containing multi-scale one-dimensional convolutional layers to capture patterns at different time scales, and embeds channel attention mechanisms (such as the SE module) to enhance key features, outputting features. .

[0052] Feature decomposition: Extracting initial features for each modality (For the sake of simplicity, here) Reference , , (Any one of them) Input two parallel encoders:

[0053] Modality-invariant encoder: A parameter-sharing fully connected network whose goal is to extract common information representing emotional states from features across modalities that is consistent across modalities, and output modality-invariant features. .

[0054] Modality-specific encoders: Each modality has an independent, structurally identical fully connected network used to preserve information unique to that modality and distinguish it from other modalities, outputting modality-specific features. .

[0055] Decomposition Constraints: During the model training phase, two constraints are imposed on the decomposition process to ensure its effectiveness:

[0056] Modal alignment constraint: using central moment difference ( Loss function To minimize the mode-invariant features of different modes The distribution distance between them; it is defined as:

[0057]

[0058] in The number of modes (in this example) ); The lower the value, the more aligned the common features of different modalities are.

[0059] Orthogonality constraint: using a loss function To maximize the discriminative power between modality-invariant and modality-specific features, this is achieved by minimizing the sum of the squared Frobenius norms:

[0060]

[0061] in The squared Frobenius norm is a constraint that encourages the modality-invariant subspace to be as orthogonal as possible to each modality-specific subspace, as well as to each modality-specific subspace to be as orthogonal as possible to each other, thereby reducing information redundancy.

[0062] S2. Cross-modal coarse-grained fusion

[0063] This step utilizes graph distillation technology to achieve deep information interaction and fusion between modes at both the frequency and time domains.

[0064] S2.1 Coarse-grained alignment (frequency domain):

[0065] Modality invariant features of each mode By mapping to the frequency domain using the Fast Fourier Transform (FFT), a frequency domain representation is obtained. Frequency domain features can capture the global spectral patterns of a signal, making them suitable for coarse-grained alignment.

[0066] Construct the first graph distillation unit (GD-Unit); this unit will extract the frequency domain features of each mode. Treat each edge as a node in the graph and construct a fully connected graph; It has a learnable weight. , indicating from modality To mode The strength of knowledge transferability.

[0067] Each First, it passes through an independent feature encoder. (e.g., a fully connected layer) is transformed to obtain .

[0068] Calculate a distillation loss matrix within the graph distillation cell. Its elements Measure node and Characteristic differences (e.g., using mean squared error); coarse-grained distillation loss Calculated as an edge weight matrix With loss matrix L1 norm of the Hadamard product:

[0069]

[0070] By optimizing this loss through backpropagation, the model automatically learns to transfer shared global sentiment information in the frequency domain, achieving coarse-grained temporal alignment and outputting aligned coarse-grained fused features.

[0071] S2.2 Fine-grained alignment (time domain):

[0072] Preserve the mode-specific features of each mode The original temporal structure.

[0073] Construct a second distillation unit with the same internal structure as the first distillation unit, but with completely independent parameters; this unit operates in the time domain.

[0074] Each First, it passes through an independent feature encoder. Transformation, to obtain .

[0075] Similarly, the edge weight matrix is ​​calculated in the distillation unit of the second graph. and distillation loss matrix And obtain the fine-particle distillation loss:

[0076]

[0077] This loss-driven model aligns with the modality-specific, dynamically changing emotional response details at more refined time points, outputting fine-grained fused features.

[0078] S2.3 Feature Aggregation:

[0079] The features, after coarse-grained and fine-grained alignment, are compared with the original mode-specific features. Perform aggregation (e.g., concatenate along feature dimensions).

[0080] The aggregated high-dimensional features are input into a multi-head self-attention mechanism; this mechanism can adaptively evaluate and weight the importance of different parts of the fused features, explore their intrinsic correlations, and ultimately generate a comprehensive and discriminative subject-level feature representation. .

[0081] S3. Group-based emotional classification

[0082] Encoder construction:

[0083] Construct a subject-shared encoder Its parameters are in all Shared among subjects to learn common, universal patterns of human emotion expression.

[0084] Construct K group encoders Each encoder corresponds to a subject group divided in step S0. The parameters of each group encoder are... It is shared only within its own group and is independent of each other in different groups; this allows the model to characterize the emotional response patterns unique to different groups.

[0085] Feature encoding and fusion:

[0086] The subject-level characteristics obtained in step S2 are represented The data is simultaneously input into the subject-shared encoder and the group encoder corresponding to the subject's group.

[0087] The subject-shared encoder outputs subject-shared features. .

[0088] Group encoder outputs group-specific features .

[0089] Will and For fusion, this embodiment preferably uses a feature concatenation operation to obtain the final comprehensive feature representation. .

[0090] Emotion Classification:

[0091] Will Input a classifier (Typically composed of fully connected layers and a softmax function), the output is a probability distribution corresponding to a preset emotion category (such as joy, sadness, anger, calm, etc.). The category with the highest probability is used as the final emotion recognition result.

[0092] III. Model Training Methods

[0093] This section details how to train the emotion recognition model described in Example 1.

[0094] Data preparation: Construct a training dataset containing multimodal physiological signal samples from multiple subjects and their corresponding emotion labels.

[0095] Subject grouping (training phase): Perform the aforementioned “subject grouping” step (S0) on the training set to obtain K subject group divisions, which are used to initialize K group encoders.

[0096] Loss function construction: Employing a multi-task learning strategy, the overall loss function... The weighted sum of all losses:

[0097]

[0098] in:

[0099] For the loss function in the emotion classification task, the cross-entropy loss function is used: , For real labels, To predict probabilities, This represents the number of samples in the batch.

[0100] , , , These are the aforementioned modal alignment loss, orthogonal constraint loss, coarse-grained distillation loss, and fine-grained distillation loss, respectively.

[0101] , , , The hyperparameters used to balance the weights of various losses can be adjusted using the validation set.

[0102] End-to-end training: Using backpropagation algorithms (such as the Adam optimizer) to minimize To achieve this, end-to-end joint training is performed on all model parameters, from the feature extraction network to the classifier. During training, the corresponding group encoder is automatically selected based on the subject ID of the sample.

[0103] Example 2: This example provides a multimodal emotion recognition system based on group encoders and graph distillation.

[0104] System composition: The system includes the following modules:

[0105] Feature extraction and decomposition module: Used to receive preprocessed multimodal physiological signal raw data, with built-in EEG, EMG / GSR dedicated feature extraction network and modality invariant / specific encoder to complete initial feature extraction and decomposition.

[0106] Cross-modal fusion module: includes:

[0107] Coarse-grained alignment submodule: integrates an FFT transformer and a first-graph distillation unit.

[0108] Fine-grained alignment submodule: integrates the second diagram distillation unit.

[0109] Feature aggregation submodule: integrates a multi-head self-attention mechanism.

[0110] This module is responsible for receiving the decomposed features and outputting the subject-level feature representation. .

[0111] The group-aware classification module includes a shared encoder for the subject, K group encoders, and a classifier. The system maintains a "subject-group" mapping table, calls the corresponding group encoder based on the source of the input data, and finally outputs the emotion classification result.

[0112] Model training module: This system can run offline and is used to execute the training method described in Part 3 of Example 1, updating and optimizing system parameters using new data.

[0113] Workflow: During application deployment, the system receives physiological signal data in real time or offline, processes it sequentially through the modules described above, and finally outputs the emotion recognition result. The system's modular design makes it easy to integrate into various human-computer interaction, health monitoring, or neuroscience research platforms.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal emotion recognition method based on group encoders and graph distillation, characterized in that, Includes the following steps: S1. Initial Feature Extraction and Decomposition: Obtain the raw data of the subject's EEG, EMG and TE signals, and extract the initial features of each modality through a dedicated feature extraction network; The initial features of each mode are input into the mode-invariant encoder and the mode-specific encoder to decompose and obtain the mode-invariant features and mode-specific features of each mode; S2. Cross-modal coarse-grained fusion: S2.

1. Coarse-grained alignment: The modality-invariant features of each mode are transformed to the frequency domain and input into the first graph distillation unit for cross-modal knowledge distillation to achieve coarse-grained time alignment, obtain coarse-grained fusion features, and calculate coarse-grained distillation loss; S2.

2. Fine-grained alignment: The modality-specific features of each mode are input into the second graph distillation unit in the time domain to perform cross-modal knowledge distillation, thereby achieving fine-grained time alignment, obtaining fine-grained fusion features, and calculating fine-grained distillation loss; S2.

3. Feature aggregation: The coarse-grained fused features, fine-grained fused features, and modality-specific features of each modality are aggregated and fused through a multi-head self-attention mechanism to obtain the subject-level feature representation; S3. Group-perceived emotion classification: The subject-level feature representation is simultaneously input into a subject-shared encoder and a group encoder corresponding to the current subject's group, and subject-shared features and group-specific features are extracted respectively; The shared features of the participants and the group-specific features are combined, and the emotion classification result is obtained through a classifier. The first graph distillation unit and the second graph distillation unit are used to construct a fully connected graph with each modal feature as a node. By optimizing the product norm of the edge weight matrix and the distillation loss matrix, cross-modal knowledge transfer is achieved.

2. The method according to claim 1, characterized in that, Before step S1, there is also step S0, subject grouping: calculate the correlation coefficient matrix between subjects based on the EEG signal data of multiple subjects, and divide the subjects into K groups according to the matrix.

3. The method according to claim 1, characterized in that, In step S1, modality alignment constraints and orthogonality constraints are applied to the modality-invariant features. The modality alignment constraints are achieved by minimizing the difference in central moments between the distributions of modality-invariant features of different modes. The orthogonality constraints are achieved by minimizing the sum of squared Frobenius inner products between the modality-invariant features and each modality-specific feature, as well as between each modality-specific feature and each other.

4. The method according to claim 1, characterized in that, In step S2.1, the mode-invariant features are converted to the frequency domain using the Fast Fourier Transform.

5. The method according to claim 1, characterized in that, In step S3, the parameters of the subject-shared encoder are shared among all subjects; the parameters of the group encoder are shared within their respective groups and are independent between different groups.

6. A training method for a multimodal emotion recognition model, used to train the model used in the emotion recognition method as described in any one of claims 1-5, characterized in that, Includes the following steps: Acquire multimodal physiological signal data and their emotional labels from multiple subjects; The correlation between subjects was calculated based on the EEG signal data of all subjects, and the subjects were divided into multiple groups according to the correlation coefficient matrix. Construct a model architecture for the emotion recognition method as described in claim 1; Using the multimodal physiological signal data as input and emotion labels as supervision, the model is trained end-to-end by combining the total loss function consisting of emotion classification task loss, modality alignment loss, modality differentiation loss, coarse-grained distillation loss, and fine-grained distillation loss.

7. The training method according to claim 6, characterized in that, The specific steps for calculating the correlation between subjects include: segmenting the EEG data of each subject according to a preset time window and overlap rate; calculating the power spectral density characteristics of all channels in each data segment; calculating the correlation coefficient matrix between each pair of subjects based on the power spectral density characteristics of all subjects; and using a clustering algorithm to divide the subjects into K groups based on the correlation coefficient matrix.

8. A multimodal emotion recognition system based on group encoders and graph distillation, characterized in that, include: The feature extraction and decomposition module is used to acquire the raw data of the subject's multimodal physiological signals, extract the initial features of each modality, and decompose them into modality-invariant features and modality-specific features of each modality. The cross-modal fusion module includes: The coarse-grained alignment submodule is used to transform the mode-invariant features of each mode to the frequency domain and to perform knowledge distillation and fusion using the first graph distillation unit; The fine-grained alignment submodule is used to perform knowledge distillation and fusion of the modality-specific features of each modality in the time domain using the second graph distillation unit; The feature aggregation submodule is used to fuse features after coarse and fine-grained alignment and generate subject-level feature representations through an attention mechanism. The group-aware classification module includes a subject-shared encoder, multiple group encoders, and a classifier, used to perform shared and group-specific encoding on the subject-level feature representations, and to fuse the encoding results to complete emotion classification; The first graph distillation unit and the second graph distillation unit are used to construct a fully connected graph with each modal feature as a node. By optimizing the product norm of the edge weight matrix and the distillation loss matrix, cross-modal knowledge transfer is achieved.

9. The system according to claim 8, characterized in that, In the feature extraction and decomposition module, the feature extraction network for EEG signals is a convolutional neural network based on EEGNet that includes temporal convolution, depthwise separable convolution, and an attention module; the feature extraction network for EMG and electrodermal signals is a lightweight convolutional network that includes multi-scale convolution and an attention mechanism.

10. The system according to claim 8, characterized in that, It also includes a model training module for performing the training method as described in claim 6 or 7 to optimize the parameters of the system.

Images