Electroencephalogram emotion recognition method based on cross-modal knowledge distillation

Abstract

The invention discloses an electroencephalogram emotion recognition method based on cross-modal knowledge distillation, and the method comprises the steps: firstly constructing a data set containing a synchronous face video, an audio signal and an electroencephalogram signal, constructing a visual teacher model and an audio teacher model based on an encoder architecture, and carrying out the training through the face video and the audio signal, and storing the two parameters. Secondly, proposing a student model fusing modal guidance and anti-guidance mechanisms, introducing a multi-path feature extraction framework, and constructing a panoramic knowledge distillation mechanism for a visual teacher model, an audio teacher model and the student model; and finally, in combination with a local multi-head attention mechanism, student model parameters are optimized in an end-to-end training mode, a student model is used as an inference decoder for electroencephalogram modal emotion recognition, and an electroencephalogram emotion recognition result is output. According to the method, the electroencephalogram emotion recognition performance can be remarkably improved on the premise of keeping the light weight of the model.

Description

Technical Field

[0001] This project involves an EEG emotion recognition method based on cross-modal knowledge distillation, which aims to improve the decoding performance of a single EEG modality in emotion recognition tasks by combining cross-modal knowledge transfer. Background Technology

[0002] Electroencephalography (EEG), as a non-invasive method for measuring neural activity, has been widely used in the field of emotion recognition. EEG can directly reflect the brain's physiological responses during changes in emotional state, and is therefore considered an objective, real-time, and biologically based source of emotion recognition signals. However, current EEG-based emotion recognition still faces two core challenges: first, the information density and spatial resolution of EEG signals themselves are relatively low, making it difficult to fully capture fine-grained emotional features; second, significant physiological differences exist between individuals, and the EEG patterns of different subjects exhibit high variability, greatly limiting the generalization ability of the model. These factors make it difficult for single-modality EEG emotion recognition models to achieve ideal performance in practical applications.

[0003] Traditional optimization methods mainly include expanding the data scale and designing more complex deep neural network structures. However, due to the high cost of data collection and the significant influence of environmental and individual state on the quality of EEG signals, the potential for improvement using these methods has gradually been limited. Therefore, in recent years, researchers have begun to focus on multimodal learning, especially using visual or audio modalities to assist EEG modalities, thereby improving the overall performance of emotion recognition.

[0004] In emotion recognition, visual (e.g., facial expressions) and audio (e.g., speech tone) modalities have been widely used for emotion classification tasks due to their higher expressiveness and information density, achieving remarkable results. In contrast, the performance of the EEG modality is relatively weaker. Inspired by this, transferring discriminative knowledge from the visual or audio modalities to the EEG modality has become a promising direction for improving the performance of EEG emotion recognition.

[0005] Knowledge distillation (KD), as an effective knowledge transfer technique, has been widely applied in recent years to tasks such as model compression and performance improvement. Its basic idea is to transfer knowledge learned from a well-trained, high-performing teacher model to a smaller, different student model through a "teacher-student" architecture. Traditional KD methods are mostly based on the alignment of Logits outputs or intermediate features, achieving knowledge transfer by minimizing the distributional differences between the teacher and student models. These methods perform well between homogeneous models (i.e., those with similar inputs, structures, and tasks), but between heterogeneous models, especially in cross-modal scenarios, they often fail to achieve good results directly and may even introduce invalid or harmful knowledge, leading to a decline in the performance of the student model.

[0006] In recent years, knowledge distillation (KD) technology has gradually expanded into cross-modal domains, where teacher and student models come from different data modalities, such as visual to audio, or RGB images to depth images. Although some works have attempted to apply cross-modal KD to emotion recognition, most remain at the level of homogeneous input or low heterogeneity, and have not yet systematically solved the problem of effective knowledge transfer under highly heterogeneous conditions. Therefore, how to achieve robust and efficient cross-modal knowledge distillation under the premise of significant differences in modal structure and inconsistent representation spaces remains a key technical bottleneck in current multimodal emotion recognition research. Summary of the Invention

[0007] This invention proposes an EEG emotion recognition method based on cross-modal knowledge distillation, abbreviated as AVGE (Audio-Video Guided EEG), aiming to improve the discriminative performance of EEG signals in emotion recognition tasks and solve the problems of low EEG modality information density and large individual differences that limit recognition accuracy in existing technologies. This method introduces well-trained emotion recognition models from audio and visual modalities as teacher networks, and employs a multi-branch EEG student model to achieve cross-modal knowledge transfer. Simultaneously, it incorporates an adaptive fusion mechanism to effectively achieve knowledge guidance and robust modeling between heterogeneous modalities.

[0008] The core technology of this invention lies in constructing a "teacher-student" distillation architecture, leveraging the advantages of visual and audio modalities in emotion recognition to guide EEG modalities in learning more discriminative feature representations. According to the invention, the specific steps include:

[0009] Step 1: Multimodal Synchronous Data Preparation and Preprocessing: Collect and construct a multimodal emotion dataset containing synchronized facial videos, audio signals, and EEG signals. The three modalities are strictly time-aligned and labeled using unified emotion tags. Subsequently, standardized preprocessing operations are performed on each modality, including image cropping and normalization, audio spectrum mapping, EEG signal filtering, and segmentation, to generate model inputs with a uniform format and consistent structure.

[0010] Step 2, Teacher Model Construction: Construct separate visual and audio teacher models, each trained independently under emotion label supervision based on its respective modal encoder structure. After training, save the parameters of both models for use as a knowledge source for subsequent cross-modal guidance.

[0011] Step 3, Student Model Design and Training: A student model structure integrating modal guidance and anti-guidance mechanisms is proposed. A multi-path framework is introduced to model general features and modality-specific representations respectively. Semantic separation and complementarity between paths are achieved through structural decoupling adversarial strategies.

[0012] Step 4: Design of a panoramic knowledge distillation mechanism: For the visual teacher model, audio teacher model, and student model, construct multiple distillation paths at the response, feature, and relation levels, and introduce modules such as flexible behavior alignment, structural relation alignment, and relation graph preservation to significantly improve the expressive integrity and generalization ability of knowledge transfer.

[0013] Step 5, Feature Fusion and Optimization Training: A local multi-head attention mechanism is used to fuse features from different channels, and the model parameters are optimized through end-to-end training to achieve a dynamic balance between teacher guidance and student self-directed learning.

[0014] Step 6, Model Deployment and Inference: After training, the student model is used as the inference decoder for EEG modality emotion recognition, outputting the EEG emotion recognition results. This model features lightweight structure, strong transferability, and high adaptability, making it suitable for various resource-constrained or real-time interactive scenarios.

[0015] Through the above steps, this invention effectively incorporates emotional knowledge from visual and audio modalities while ensuring the autonomous learning ability of EEG modalities, thereby achieving higher-performance emotion recognition modeling and demonstrating good generalization and practical value.

[0016] Preferably, step 3 includes:

[0017] Step 3-1: Guiding Mechanism: Modal guidance is achieved by introducing a Query decoder. The modal shared features output by the Query decoder are decoded. This modal guidance process is jointly activated by the visual teacher model with frozen parameters and the audio teacher model to guide the student model to learn a general emotional representation that is consistent with the video / audio modality, thereby strengthening cross-modal semantic consistency.

[0018] Specifically, the EEG signal is input into the student model, encoded to obtain a set of features e, which are then input into the Query decoder as the guiding path for distillation learning. The visual teacher model guides the EEG to acquire general learning features e. V Audio-guided EEG acquisition of general learning features A .

[0019] Let V be the feature output of the visual teacher model and A be the feature output of the audio teacher model. Then, the Query decoder is constrained as a guiding path to learn cross-modal general features through subsequent feature distillation, and the similarity constraint loss L is distilled. sim The usage method is as follows:

[0020] L sim =L feature (V,e V )+L feature (a,e A (1)

[0021] Step 3-2: The anti-guidance mechanism uses the Key decoder to model the unique specific features of EEG. The anti-guidance mechanism introduces an adversarial decoupling strategy, enabling the student model to learn a stable representation of the EEG modality that is orthogonal to the two general learning features, thereby enhancing the model's robustness and adaptability to individual differences.

[0022] Specifically, a set of features e encoded by the student model is input into the Key decoder to learn the modality-specific features e of the EEG through adversarial general features. i .

[0023] For the output e of a set of visual modal guidance paths (VE) and audio modal guidance paths (AE) V e A By using soft subspace orthogonality constraints to counteract and limit the output e of the mode guiding path and the mode anti-guiding path. i Maintain orthogonality to ensure no coupling between features. Constrain the two paths to maintain structural orthogonality in the high-dimensional semantic representation space, preventing feature overlap or redundant interference between different paths. Adversarial constraint loss L diff The calculation method is shown in the following formula:

[0024]

[0025] in This represents the square of the Frobenius norm, i.e., the orthogonality constraint of the soft subspace.

[0026] Through this mechanism, the student model can integrate visual / audio modal knowledge while retaining the autonomous learning characteristics of EEG modality, thereby achieving robust, decoupled, and highly generalizable multimodal emotion representation capabilities.

[0027] Preferably, step 4 includes:

[0028] Step 4-1: Response Domain Flexible Distillation: By introducing Pearson similarity instead of KL divergence, the behavioral similarity between the teacher model and the student model in class prediction ranking is measured.

[0029] This invention employs a metric d(·) that utilizes the Pearson correlation coefficient ρ(·) instead of the KL divergence:

[0030] d(x,y)=1-ρ(x,y). (3)

[0031] ρ(x,y) represents the Pearson correlation coefficient between vectors x and y.

[0032] The predicted probability distributions of all instances within a batch are collected, and distance metrics for inter-class and intra-class relationships between teachers and students are calculated. These inter-class and intra-class relationships are then transferred to students.

[0033] Knowledge distillation based on response L logits The formula can be described as follows:

[0034]

[0035] L logits =L inter +L intra (8)

[0036] Given training batch B and emotion category C, Z s ∈R B×C It is the student's prediction matrix, and Z t ∈R B×C Let Y be the teacher's prediction matrix, Y be the response output, i: represent all categories of the i-th sample, and c: represent the j-th category of all samples. τ>0 is the temperature parameter controlling the softness.

[0037] Step 4-2: Feature Domain Structure Alignment Distillation: In the deep feature space, an instance similarity map between teachers and students is constructed, and structural relationship transfer is achieved through temperature-regulated KL divergence, thereby enhancing the overall structural consistency while preserving individual features. This invention comprehensively improves the quality of knowledge transfer by minimizing the similarity between teachers and themselves, as well as between teachers and students.

[0038] A similarity matrix is ​​constructed by performing a dot product between the teacher's representation matrix (features V and A) and their corresponding transpose matrices. This matrix includes the intrinsic relationships between teacher features and can effectively serve as an optimization objective to guide students. The derived target probability distribution is as follows:

[0039]

[0040] M∈R B×B This is a similarity matrix between teacher models, where the element in the i-th row and i-th column is represented as M. i,i ;

[0041] The similarity matrix between the teacher model and the student model is constructed using the same method described above. It aims to quantify the relationship between current student features and teacher features as the target of optimization, and the similarity probability distribution is calculated as follows:

[0042]

[0043] M′∈R B×B This is the similarity matrix between the student model and the teacher model, where the element in the i-th row and i-th column is represented as M.′ ,i τ>0 is the temperature parameter that controls softness, Q i It is the probability distribution of the similarity between teachers and students.

[0044] The ultimate goal is to preserve the similarity between features, treating KL divergence as a loss L based on feature distillation. feature To constrain relation similarity:

[0045]

[0046] By integrating the triple distillation objectives of preference guidance in the response layer, structural alignment in the feature layer, and topological preservation in the relation layer, this invention achieves a panoramic knowledge transfer mechanism of "feature-response-relationship," which has stronger adaptability, anti-interference ability, and cross-modal generalization capability.

[0047] Step 5 includes:

[0048] Step 5-1: Combine the features e learned from the visual and audio learning channels respectively. V e A After splicing, the overall learned knowledge e is captured through a multi-head self-attention mechanism. AV :

[0049] e AV =Multi headAttention <e V ,e A > (12)

[0050] Step 5-2: By learning the knowledge e in the overall learning channels AV Knowledge of invariant channels e i After splicing, the final prediction result y is obtained through a fully connected layer.

[0051] Step 5-3: The training phase of the student model includes the following optimization loss:

[0052] L = L task +αL logits +(1-α)L sim +βL diff (13)

[0053] Where α and β are weighting coefficients, L task This is the classification loss for the student model.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] (1) Improve the performance of EEG modal emotion recognition: This invention introduces a well-trained teacher model in the visual and audio modal. On the basis of ensuring the autonomous learning ability of the EEG modal, it assists the student model in learning more discriminative cross-modal semantic information, effectively alleviating the problems of low EEG signal information density and weak discriminability, and significantly improving the accuracy and robustness of emotion recognition.

[0056] (2) Overcoming the challenge of heterogeneous knowledge transfer between modalities: This invention constructs a three-branch structure consisting of an audio guidance channel, a visual guidance channel, and a modality-invariant channel, and introduces an orthogonal constraint mechanism to achieve information decoupling within the student model. This structure effectively avoids mutual interference between multimodal information and overcomes the problem of difficult alignment of knowledge distillation between heterogeneous modalities.

[0057] (3) Balancing response and feature level knowledge distillation: By combining Logits-level response distillation and intermediate feature-level similarity distillation, and combining various loss functions such as Pearson correlation coefficient and KL divergence, this invention achieves in-depth extraction and multi-level transmission of the teacher model's knowledge structure, thereby enhancing the student model's ability to learn teacher preferences and semantic structures.

[0058] (4) It has good generalization and compression capabilities: After training, the student model can be deployed independently as a lightweight decoder. It is small in size and fast inference, and is suitable for resource-constrained edge devices or real-time interactive systems. It has strong practical and promotional value.

[0059] (5) Enhance the selectivity and robustness of knowledge transfer: This invention introduces a local multi-head attention mechanism in the output stage, which can dynamically adjust the knowledge learned by the self-learning channel according to the context, automatically suppress the transfer of redundant or harmful information, and effectively improve the anti-interference ability and stability of the overall system.

[0060] In summary, the AVGE method proposed in this invention is innovative in terms of structural design, knowledge transfer method, and fusion strategy. It can significantly improve the performance of EEG emotion recognition while maintaining the model's lightweight nature. It is applicable to emotion recognition tasks in various intelligent systems such as emotion perception, brain-computer interface, and human-computer interaction, and has significant engineering application prospects and promotional value. Attached Figure Description

[0061] Figure 1 This is a flowchart of the present invention;

[0062] Figure 2 This invention provides an overview of the method framework.

[0063] Figure 3 This diagram illustrates the cross-modal knowledge distillation method proposed in this invention. Detailed Implementation

[0064] The following description, in conjunction with the accompanying drawings and specific embodiments, further illustrates the EEG emotion recognition method based on cross-modal knowledge distillation proposed in this invention. The following description is for illustrative and explanatory purposes only and does not impose any limitations on the invention. This embodiment mainly focuses on model structure design, distillation strategy, feature fusion mechanism, and training process, aiming to enable those skilled in the art to implement the invention accordingly. The overall implementation process is shown in the attached figure. Figure 1 As shown, the overall framework is as follows Figure 2 As shown.

[0065] Step 1: This invention relies on a synchronously acquired multimodal emotion dataset, including three modalities: facial video, audio signals, and electroencephalogram (EEG) signals. The data from these three modalities are strictly time-aligned during acquisition and labeled with unified emotion tags. This dataset serves as the training data foundation for constructing the visual teacher model, audio teacher model, and student model.

[0066] Data processing: Preprocessing was performed on the three modalities of data: the 5-second segmented EEG signal was downsampled to 100Hz and bandpass filtered in the range of 0.5-50Hz; the face video acquired at 30FPS was divided into 5-second segments, uniformly sampled, and one face image was selected every 6 frames, corresponding to 25 face images per 5 seconds; the audio modality was processed into a usable data format using the Librosa library.

[0067] Step 2: For the visual modality, this invention uses the popular DeepFace model as the feature encoder. After training with cross-entropy classification loss, the model weights are saved. For the audio modality, the standard feature extraction method of Mel-frequency coefficients is used to extract basic audio features, including MFCC, chroma features, and MEL spectrograms. After concatenation, a sequential convolutional neural network (SCNN) with one-dimensional data processing capabilities is used as the feature encoder. After training with cross-entropy classification loss, the model weights are saved.

[0068] Step 3: To effectively alleviate the bottleneck of cross-modal knowledge transfer caused by feature heterogeneity and model architecture differences in multimodal emotion recognition tasks, this step proposes a student model construction strategy that integrates a representation-guided adversarial decoupling (RGAD) mechanism to improve the stability, generalization and feature adaptation of the transfer process.

[0069] Step 3-1: Guiding Mechanism: By introducing a Query decoder, the modal-shared features guided by the teacher model are decoded. This mechanism is jointly activated by the visual teacher model with frozen parameters and the audio teacher model to guide the student model in learning a general emotional representation that is consistent with the video / audio modality, thereby strengthening cross-modal semantic consistency.

[0070] Specifically, a set of features e encoded by the student model is input into the Query decoder as the guiding path for distillation learning, and the visual teacher guides the general learning features e of EEG. V Audio-guided EEG general learning features e A The calculation method is as follows:

[0071] e V =Query V (e), (1)

[0072] e A =Query A (e). (2)

[0073] Let V be the feature output of the visual teacher model and A be the feature output of the audio teacher model. Then, the Query decoder is constrained as a guiding path to learn cross-modal general features through subsequent feature distillation, and the similarity constraint loss L is distilled. sim The usage method is as follows:

[0074] L sim =L feature (V,e V )+L feature (A,e A (3)

[0075] Step 3-2: The anti-guidance mechanism uses the Key decoder to model the unique specific features of EEG. The anti-guidance mechanism introduces an adversarial decoupling strategy, enabling the student model to learn the unique stability representation of the EEG modality orthogonal to the two general features, thereby enhancing the model's robustness and adaptability to individual differences.

[0076] Specifically, a set of features e encoded by the student model is input into the Key decoder as an anti-guided path. The modality-specific features e of the EEG are learned by using the general learning features of the adversarial guided path. i :

[0077] e i =Key(e). (4)

[0078] For the output e of a set of visual modal guidance paths (VE) and audio modal guidance paths (AE) V e ABy using soft subspace orthogonality constraints to counteract and limit the output e of the mode guiding path and the mode anti-guiding path. i Maintain orthogonality to ensure no coupling between features. Constrain the two paths to maintain structural orthogonality in the high-dimensional semantic representation space, preventing feature overlap or redundant interference between different paths. Adversarial constraint loss L diff The calculation method is shown in the following formula:

[0079]

[0080] in This represents the square of the Frobenius norm.

[0081] Through this mechanism, the student model can integrate visual / audio modal knowledge while retaining the autonomous learning characteristics of EEG modality, thereby achieving robust, decoupled, and highly generalizable multimodal emotion representation capabilities.

[0082] Step 4: This step addresses the shortcomings of traditional knowledge distillation methods, which over-rely on distribution similarity and lack structural adaptability, by proposing a data-driven, relationship-guided panoramic distillation mechanism. In the process of feature-based distillation and response-based distillation, data-driven inter-class / intra-class relationship modeling is introduced. Through tensor matching and relationship-preserving loss, the geometric topology of feature representations is further solidified, constructing a softer and more flexible knowledge transfer method. The proposed panoramic distillation mechanism is attached. Figure 3 As shown.

[0083] Step 4-1: Response Domain Flexible Distillation: Response-based knowledge distillation aims to extract teacher preferences (relative ranking of predictions) through the correlation between teacher and student predictions, effectively extracting knowledge even in cases of extreme differences between teachers and students. Previous response-based distillation methods overemphasize the numerical similarity of response outputs, neglecting semantic and structural relational similarities. By introducing Pearson similarity instead of KL divergence, this method measures the behavioral similarity between teachers and students in class prediction ranking, thus mitigating the gradient instability problem caused by "hard alignment." This approach emphasizes modeling teacher "preferences," enabling student models to capture their predictive tendencies even with structural inconsistencies.

[0084] This invention employs a metric d(·) that utilizes the Pearson correlation coefficient ρ(·) instead of the KL divergence:

[0085] d(x,y)=1-ρ(x,y). (6)

[0086] ρ(x,y) represents the Pearson correlation coefficient between vectors x and y.

[0087] The predicted probability distributions of all instances within a batch are collected, and distance metrics for inter-class and intra-class relationships between teachers and students are calculated. These inter-class and intra-class relationships are then transferred to students.

[0088] Knowledge distillation based on response L logits The formula can be described as follows:

[0089]

[0090] L logits =L inter +L intra (11)

[0091] Given training batch B and emotion category C, Z s ∈R B×C It is the student's prediction matrix, and Z t ∈R B×C Let Y be the teacher's prediction matrix, Y be the response output, i: represent all categories of the i-th sample, and c: represent the j-th category of all samples. τ>0 is the temperature parameter controlling the softness.

[0092] Step 4-2: Feature Domain Structure Alignment Distillation: In the deep feature space, an instance similarity map between teachers and students is constructed, and structural relationship transfer is achieved through temperature-regulated KL divergence, thereby enhancing the overall structural consistency while preserving individual features. Feature-based cross-modal knowledge distillation aims to alleviate the heterogeneity between teacher and student model representations, enabling students to extract richer knowledge from teachers. Previous feature-based distillation methods overemphasize the numerical similarity of feature distributions, neglecting relational similarity. This invention comprehensively improves the quality of knowledge transfer by minimizing the relational similarity between teachers and themselves, as well as between teachers and students.

[0093] A similarity matrix is ​​constructed by taking the dot product of the teacher's representation matrix and its transpose. This matrix includes the intrinsic relationships between teacher features and can effectively serve as an optimization objective to guide students. The derived target probability distribution is as follows:

[0094]

[0095] M∈R B×B This is a similarity matrix between teacher models, where the element in the i-th row and i-th column is represented as M. i,i τ>0 is a temperature parameter that controls softness, P i It is the target probability distribution.

[0096] The similarity matrix between the teacher model and the student model is constructed using the same method described above. It aims to quantify the relationship between current student features and teacher features as the target of optimization, and the similarity probability distribution is calculated as follows:

[0097]

[0098] M′∈R B×B This is the similarity matrix between the student model and the teacher model, where the element in the i-th row and i-th column is denoted as M′. i,i τ>0 is a temperature parameter that controls softness, Q i It is the probability distribution of the similarity between teachers and students.

[0099] The ultimate goal is to preserve the similarity between features, treating KL divergence as a loss L based on feature distillation. feature To constrain relation similarity:

[0100]

[0101] By integrating the triple distillation objectives of preference guidance in the response layer, structural alignment in the feature layer, and topological preservation in the relation layer, this invention achieves a panoramic knowledge transfer mechanism of "feature-response-relationship," which has stronger adaptability, anti-interference ability, and cross-modal generalization capability.

[0102] Step 5: Introduce a local attention fusion module at the output end of the student model to automatically adjust the knowledge learned through the learning channel and achieve a dynamic balance of teacher guidance;

[0103] Features from both guided and anti-guided paths are fused. A local multi-head attention fusion module is introduced to adaptively adjust the weight distribution of features from each path, achieving a dynamic balance between teacher-guided knowledge acquisition and student-led learning. Subsequently, an end-to-end joint training strategy is employed to optimize the global parameters of the student model under the supervision of the teacher model and emotion labels.

[0104] Step 5-1: Combine the features e learned from the visual and audio learning channels respectively. V e A After splicing, the overall learned knowledge e is captured through a multi-head self-attention mechanism. AV :

[0105] e AV =Multi headAttention <e V ,e A > (15)

[0106] Step 5-2: By concatenating the knowledge learned from the overall learning channels with the knowledge from the invariant channels, the final prediction result is obtained through a fully connected layer.

[0107] y = FC <e AV ,e i > (16)

[0108] Step 5-3: The training phase of the student model includes the following optimization loss:

[0109] L = L task +αL logits +(1-α)L sim +βL diff (17)

[0110] Where α and β are weighting coefficients, L task This is the classification loss for the student model.

[0111] Step 6: After training, the trained student model is used as the decoding model for EEG emotion recognition. This not only achieves model compression for the emotion recognition task but also effectively improves the decoding performance of EEG, demonstrating good scalability and deployment value.

[0112] This invention used 42 subjects as experimental participants to conduct experiments on five types of emotion recognition tasks, based on the cross-modal knowledge distillation method proposed in this invention. The results of the accuracy comparison are shown in Table 1.

[0113] Table 1. Accuracy of each subject in recognizing the five emotion categories.

[0114]

[0115]

[0116] The experimental results (as shown in Table 1) reveal that the average recognition accuracy for video and audio modalities was 70.47% and 59.20%, respectively, while the traditional EEG single-modal model only achieved an average accuracy of 48.93%, indicating a significant performance bottleneck. In contrast, after implementing the method of this invention, the recognition performance of the EEG modality improved to an average of 53.35%, achieving a relatively stable and consistent cross-subject performance gain, with an improvement of 4.42 percentage points. This verifies the practical effectiveness of the cross-modal knowledge transfer mechanism designed in this invention in alleviating problems such as sparsity of EEG modal information and weak feature representation ability.

[0117] Furthermore, the modal guidance-anti-guidance collaborative decoupling adversarial mechanism proposed in this invention can maintain the structural stability and prediction consistency of the model under multimodal interference conditions, and has good generalization and anti-interference capabilities. It is suitable for online emotion recognition systems or edge deployment scenarios for practical applications.

[0118] Therefore, this embodiment further illustrates the technical advantages of the present invention in terms of emotion recognition accuracy, model lightweighting, and transferability, and fully demonstrates the practicality and promotional value of the method proposed in this invention.

Claims

1. A brainwave emotion recognition method based on cross-modal knowledge distillation, characterized in that, Includes the following steps: Step 1: Construct a multimodal emotion dataset containing synchronized facial videos, audio signals, and EEG signals; Step 2: Based on the encoder architecture, construct a visual teacher model and an audio teacher model respectively, train them using face videos and audio signals respectively, and save the parameters of both after training; Step 3: Propose a student model that integrates modal guidance and anti-guidance mechanisms, introduce a multi-path feature extraction framework, and achieve semantic separation and complementarity between paths through a structural decoupling adversarial strategy; Step 4: Construct a panoramic knowledge distillation mechanism for the visual teacher model, audio teacher model, and student model; Step 5: Combine local multi-head attention mechanism to optimize student model parameters through end-to-end training; Step 6: After optimizing the student model parameters, use the student model as the inference decoder for EEG modality emotion recognition and output the EEG emotion recognition results.

2. The EEG emotion recognition method based on cross-modal knowledge distillation according to claim 1, characterized in that, In step 1, the three modalities of data are kept time-aligned and labeled according to a unified emotion tag. Then, standardized preprocessing operations are performed on each modality of data, including image cropping and normalization, audio spectrum mapping, EEG signal filtering, and segmentation, to generate input with a unified format and consistent structure.

3. The EEG emotion recognition method based on cross-modal knowledge distillation according to claim 1, characterized in that, The specific implementation process of step 3 is as follows: Step 3-1: Modal guidance is achieved by introducing a Query decoder. The modal shared features output by the Query decoder are decoded. This modal guidance process is jointly activated by the visual teacher model with frozen parameters and the audio teacher model, guiding the student model to learn a general emotional representation that is consistent with the video / audio modality. The EEG signal is input into the student model, and the resulting set of features e is input into the Query decoder as the guiding path for distillation learning. The visual teacher model guides the EEG to obtain general features e. V The audio teacher model guides EEG to obtain general features e A ; Let V be the feature output of the visual teacher model and A be the feature output of the audio teacher model. Then, the Query decoder is constrained as a guiding path to learn cross-modal general features through feature distillation, and the similarity constraint loss L is distilled. sim The calculation is as follows: L sim =L feature (V,e V )+L feature (A,e A ) L feature Indicates the loss during characteristic distillation; Step 3-2: The anti-guidance mechanism uses the Key decoder to model the features of the EEG; The anti-guidance mechanism introduces an adversarial decoupling strategy, enabling the student model to learn a stable representation of the EEG modality orthogonal to the two general features; The set of features e encoded by the student model is input into the Key decoder as an anti-guidance mechanism. The modal features e of the EEG are learned through the general learning features of the adversarial guidance mechanism. i : Targeting the general feature e V e A By using soft subspace orthogonality constraints to counteract and limit the output e of the mode guiding path and the mode anti-guiding path. i Maintaining orthogonality, the two paths are constrained to maintain structural orthogonality in the semantic representation space; adversarial constraint loss L diff The calculation method is shown in the following formula: in This represents the square of the Frobenius norm, i.e., the orthogonality constraint of the soft subspace.

4. The EEG emotion recognition method based on cross-modal knowledge distillation according to claim 3, characterized in that, The specific implementation process of step 4 is as follows: Step 4-1: Response Domain Flexible Distillation: By introducing the Pearson correlation coefficient, the behavioral similarity between the two teacher models and the student models in class prediction ranking is measured; Step 4-2: Feature Domain Structure Alignment Distillation: Construct an instance feature similarity map between teachers and students, and realize structural relationship transfer through temperature-regulated KL divergence.

5. The EEG emotion recognition method based on cross-modal knowledge distillation according to claim 4, characterized in that, Step 4-1 is implemented as follows: The Pearson correlation coefficient is used as the metric d(·): d(x,y)=1-ρ(x,y), where ρ(x,y) represents the Pearson correlation coefficient between vectors x and y; By collecting the predicted probability distribution of all instances within a batch, and calculating the distance metric d(·) between and within classes of the relationship between teachers and students; Knowledge distillation based on response L logits The formula is described as follows: L logits =L inter +L intra Given training batch B and emotion category C, Z s ∈R B×C It is the student's prediction matrix, and Z t ∈R B×C Y is the teacher's prediction matrix, Y represents the response output, i: represents all categories of the i-th sample, and c: represents the j-th category of all samples; τ>0 is the temperature parameter that controls the softness.

6. The EEG emotion recognition method based on cross-modal knowledge distillation according to claim 5, characterized in that, Step 4-2 is implemented as follows: A similarity matrix is ​​constructed by performing a dot product of features V and A with their corresponding transposes. This matrix includes the intrinsic relationships between teacher features, and the probability distribution P is calculated. i as follows: M∈R B×B This is a similarity matrix between teacher models, where the element in the i-th row and i-th column is represented as M. i,i ; The similarity matrix between the teacher and student models is constructed using the same method described above. This matrix serves as the target for optimization, quantifying the relationship between current student features and teacher features, and calculating the similarity probability distribution Q. i : M′∈R B×B This is the similarity matrix between the student model and the teacher model, where the element in the i-th row and i-th column is denoted as M′. i,i ; Treating the KL divergence as a loss based on characteristic distillation L feature Constraint similarity:

7. The EEG emotion recognition method based on cross-modal knowledge distillation according to claim 6, characterized in that, The specific implementation process of step 5 is as follows: Step 5-1: Combine the features e learned from the visual and audio learning channels respectively. V e A After concatenation, the learned knowledge is captured through a multi-head attention mechanism. AV : e AV =Multi headAttention<e V ,e A > Step 5-2: By learning the knowledge e in the overall learning channels AV Knowledge of invariant channels e i After splicing, the final prediction result y is obtained through a fully connected layer; Step 5-3: The training phase of the student model includes the following optimization loss: L=L task +αL logits +(1-α)L sim +βL diff , Where α and β are weighting coefficients, L task This is the classification loss for the student model.

Images