An emotion recognition method based on online cross-modal knowledge distillation

Abstract

The application discloses an emotion recognition method based on online cross-modal knowledge distillation, comprising the following steps: acquiring electroencephalogram and electrocardiogram original signals and windowing and cutting; constructing electroencephalogram and electrocardiogram student models, extracting intermediate features from each modal data segment through an encoder, and obtaining non-normalized prediction output through a classifier; constructing a teacher probability distribution through a joint encoder fusion; introducing adaptive contrast loss to align the cross-modal intermediate features, introducing distillation loss to constrain the prediction probability distribution of each modal to align with the teacher probability distribution; synchronously optimizing new student model parameters through online collaborative training; and performing actual inference prediction based on the student model after training. The application combines double modal signals to make up for the defects of single modal information, excavates the complementarity of modes, realizes dynamic generation of teacher supervision signals and real-time collaborative learning of modes through online distillation, does not increase test calculation overhead, effectively improves the recognition accuracy, model robustness and generalization ability, and has good application prospect.

Description

Technical Field

[0001] This invention relates to the field of multimodal signal processing technology, and in particular to an emotion recognition method based on online cross-modal knowledge distillation. Background Technology

[0002] Emotion recognition is an important research direction in the field of affective computing. Its core goal is to enable computers to accurately perceive, understand and respond to human emotional states, and it has wide application value in scenarios such as human-computer interaction, mental health assessment and intelligent healthcare.

[0003] Among numerous emotion recognition modalities, electroencephalography (EEG) has become a hot topic in emotion recognition research due to its non-invasiveness, high temporal resolution, and ability to directly reflect central nervous system activity. Meanwhile, electrocardiography (ECG), as an important physiological signal reflecting the activity of the autonomic nervous system, can also effectively characterize the dynamic changes in emotional states over time. By combining multiple physiological signals, such as EEG and ECG, for multimodal emotion recognition modeling, it is helpful to capture the complementary features of different modalities in emotional expression, thereby obtaining a more comprehensive and robust emotional representation.

[0004] However, most existing multimodal sentiment regression methods focus on extracting independent features from each individual modality and obtain the final prediction result through simple fusion after feature learning. These methods ignore the deep interaction relationships between modalities, and cross-modal information is only passively utilized in the fusion stage, making it difficult to fully explore the complementarity of different modalities in sentiment expression, thus limiting the overall performance of the model. To compensate for the insufficient information interaction between modalities in the multimodal fusion process, some studies have begun to introduce knowledge distillation mechanisms, hoping to achieve cross-modal information sharing and model capability enhancement through knowledge transfer.

[0005] However, traditional knowledge distillation methods typically employ an offline teacher-student framework, where a pre-trained single or multimodal teacher model transfers knowledge to the student model. While this paradigm can improve the performance of lightweight models to some extent, the teacher model is static during the training phase, making it difficult to dynamically adjust based on the learning status of the student model. Furthermore, supervision is usually only performed at the output layer, neglecting collaborative learning at the cross-modal feature level. Summary of the Invention

[0006] To address the shortcomings of existing traditional multimodal emotion recognition methods, such as the lack of deep intermodal interactions, incomplete emotion information in a single modality, and the static and fixed teacher model in traditional knowledge distillation mechanisms, which makes it difficult to achieve dynamic cross-modal collaborative learning, this invention provides an emotion recognition method based on online cross-modal knowledge distillation. This method combines EEG and ECG multimodal physiological signals to achieve complementary utilization of emotion features from both neural activity and autonomic nervous system regulation levels. Taking into account the structural differences between EEG and ECG signals, modality-specific encoders are set up to extract corresponding signal features. During the feature learning stage, adaptive contrastive loss constrains the encoding of different modalities. The intermediate features output by the device enable the alignment of EEG and ECG features in a shared emotional semantic space. At the model output level, joint encoding and adaptive fusion are performed based on the predicted outputs of each modality, dynamically generating a high-quality teacher probability distribution by combining the complementary advantages of polar probability distribution and marginal probability distribution. Relying on an online cross-modal knowledge distillation framework, the predicted outputs of each modality are aligned with the teacher probability distribution through distillation loss constraints. During training, the parameters of the EEG model and ECG model are updated synchronously to achieve real-time knowledge interaction and collaborative learning between modalities. The model training is completed by weighted co-optimization of label loss, adaptive contrast loss and distillation loss.

[0007] This invention aims to achieve real-time collaborative learning and high-quality dynamic supervision among modalities through an overall design that integrates multimodal complementary modeling, modality-specific feature extraction, cross-modal feature alignment, polarity-edge joint teacher signal generation, and online cross-modal knowledge distillation and multi-loss collaborative optimization. This enables the model to obtain more stable, consistent, and discriminative emotion representations, thereby significantly improving the accuracy of emotion recognition, the model's generalization ability, and the integrity of physiological signal representation.

[0008] To achieve the above-mentioned technical objectives, the present invention provides the following technical solution:

[0009] A sentiment recognition method based on online cross-modal knowledge distillation specifically includes the following steps:

[0010] The raw EEG and ECG signals were acquired and then sliced ​​into windows to obtain EEG signal data fragments and ECG signal data fragments, respectively.

[0011] Emotion recognition is performed by constructing EEG-predicted student models and ECG-predicted student models, with each student model including an encoder and a classifier;

[0012] EEG signal data segments are input into an EEG encoder to extract intermediate features of the EEG signals; ECG signal data segments are input into an ECG encoder to extract intermediate features of the ECG signals; the intermediate features of the EEG signals include the frequency and time of the EEG signals, as well as the multi-channel spatial information of the EEG signals; the intermediate features of the ECG signals include the peak morphology information of the ECG signals and the temporal variation information of the ECG signals.

[0013] The intermediate features of the EEG signal are input into the EEG classifier, and the intermediate features of the ECG signal are input into the ECG classifier to obtain the unnormalized EEG prediction output and the unnormalized ECG prediction output, respectively.

[0014] By fusing the prediction outputs of different modalities using a joint encoder, a teacher probability distribution is dynamically generated through an adaptive generation mechanism of polarity and edges.

[0015] Label loss and adaptive contrast loss are introduced to align intermediate features of different modalities, and distillation loss is introduced to align the prediction probability distribution of each student model with the teacher's probability distribution. The total loss of each student model during the training phase is calculated based on label loss, adaptive contrast loss and distillation loss, and the parameters of the EEG prediction student model and the ECG prediction student model are updated synchronously through online collaborative training.

[0016] After training, the EEG or ECG signal to be identified is input into the student model of the corresponding modality. The signal is then passed through an encoder and a classifier to obtain the unnormalized prediction output. Finally, the probability distribution of each emotion category is obtained through the Softmax function, which serves as the emotion recognition result.

[0017] Furthermore, the EEG encoder comprises a frequency band decoding and filtering module, a frequency convolution, a temporal convolution, a first feature activation fusion module, a multi-scale graph convolution module, a second feature activation fusion module, and a linear layer connected in series, wherein:

[0018] The frequency band decoding and filtering module decomposes the original broadband EEG signal into signals of different frequency bands and stacks them along the frequency band dimension to obtain the multi-frequency band features of the EEG signal.

[0019] Frequency convolution extracts cross-frequency correlation features between different frequency bands from multi-band features, while temporal convolution extracts local temporal features of EEG signals from multi-band features.

[0020] The first feature activation fusion module fuses the cross-frequency correlation features and local temporal features of the EEG signal to obtain preliminary fused features;

[0021] The multi-scale graph convolution module extracts spatial features of EEG signals at different scales from the preliminary fusion features, and stacks spatial features of different scales along the scale dimension to obtain multi-scale spatial features of EEG signals.

[0022] The second feature activation fusion module activates and fuses multi-scale spatial features to obtain multi-scale fused features; both the first and second feature activation fusion modules consist of batch regularization layers, ReLU activation functions, and fusion convolutions connected in sequence.

[0023] The linear layer aligns the multi-scale fusion features to obtain intermediate features of the EEG signal.

[0024] Furthermore, the multi-scale graph convolution module consists of a linear transformation layer and a graph convolution network. The linear transformation layer projects the initially fused features into multi-scale features with different numbers of channels, and then graph convolution operations are performed on the multi-scale features through graph convolution networks of different scales to extract spatial features of EEG signals at different scales.

[0025] Furthermore, the ECG encoder is composed of a spatial convolution module, a multi-scale convolution module, a temporal feature fusion module, a peak feature enhancement module, a one-dimensional convolution module, and a linear layer connected in series, wherein:

[0026] Spatial convolution extracts spatial channel features of electrocardiogram signals in the channel dimension;

[0027] The multi-scale convolution module extracts local temporal features of ECG signals at different scales from spatial channel features through one-dimensional convolution with different kernel sizes.

[0028] The temporal feature fusion module consists of a batch regularization layer, a ReLU activation function, and a fusion convolution connected in sequence, which fuses local temporal features of ECG signals at different scales into multi-scale temporal fusion features;

[0029] The peak feature enhancement module consists of a max pooling layer, a batch regularization layer, and a ReLU activation function connected in series, which extracts the peak features of the electrocardiogram signal from the multi-scale temporal fusion features.

[0030] One-dimensional convolution is then used to reduce the dimensionality of the obtained peak features;

[0031] The linear layer aligns the peak features after dimensionality reduction to obtain the intermediate features of the electrocardiogram signal.

[0032] Furthermore, the method of using a joint encoder to fuse prediction outputs from different modalities and dynamically generating the teacher probability distribution through an adaptive generation mechanism of polarity and edges specifically involves:

[0033] Real-time acquisition of unnormalized EEG prediction output and unnormalized ECG prediction output;

[0034] The unnormalized EEG prediction output and unnormalized ECG prediction output are normalized by using the Softmax function combined with the temperature coefficient τ to obtain the EEG prediction probability distribution and ECG prediction probability distribution, respectively.

[0035] For each emotion category c, calculate the standard deviation of the EEG prediction probability distribution and the ECG prediction probability distribution within c, and obtain the EEG prediction standard deviation. and ECG prediction standard deviation ; using the Softmax function to , Normalization was performed to obtain the polarity weights of the EEG modes. Polarity weights of ECG modes The polarity probability distribution is obtained by weighting and fusing the EEG prediction probability distribution and the ECG prediction probability distribution using polarity weights. ;

[0036] For each emotion category c, and in conjunction with the true label, a marginal adjustment probability is constructed: if c is the target category corresponding to the true label, the predicted probability is set to 1; if c is not the target category, the original predicted probability is retained. The EEG marginal adjustment probability and the ECG predicted adjustment probability are calculated separately, and the minimum value between the two is retained. After Softmax normalization, the marginal probability distribution is obtained. ;

[0037] polarity probability distribution With marginal probability distribution Linear fusion is performed according to preset weights to dynamically generate teacher probability distributions.

[0038] Furthermore, the calculation process for the label loss and the adaptive contrastive loss is as follows:

[0039] Calculate the difference between the model prediction results and the true labels for each student, and use it as the label loss;

[0040] The label loss is normalized using the Softmax function to obtain the adaptive weights of the contrastive loss for each student model.

[0041] The contrastive learning loss InfoNCE between intermediate features of EEG signals and intermediate features of ECG signals is calculated, and then multiplied by the adaptive weights of the contrastive loss of each student model to obtain the adaptive contrastive loss of each student model.

[0042] Furthermore, the calculation process for the distillation loss is as follows:

[0043] The unnormalized EEG prediction output and unnormalized ECG prediction output are normalized by using the Softmax function combined with the temperature coefficient τ to obtain the EEG prediction probability distribution and ECG prediction probability distribution, respectively.

[0044] For each student model, calculate the KL divergence loss between its corresponding predicted probability distribution and the teacher's probability distribution, which serves as the distillation loss for each student model.

[0045] Furthermore, the calculation of the total loss of each student model during the training phase based on label loss, adaptive contrast loss, and distillation loss, and the synchronous updating of the parameters of the EEG prediction student model and the ECG prediction student model through online collaborative training, specifically involves:

[0046] During online collaborative training, each student model independently completes forward inference within each mini-batch of training samples, obtaining intermediate features of each modality, unnormalized prediction outputs of each modality, and dynamically generating teacher prediction distributions.

[0047] Based on the intermediate features of each modality, the unnormalized prediction output of each modality, and the teacher prediction distribution, the label loss, adaptive contrastive loss, and distillation loss of each student model are calculated separately, and then weighted to obtain the total loss, expressed by the formula:

[0048] ;

[0049] in, Let the label loss be the model for the k-th student. For the adaptive contrastive loss of the k student models, The hyperparameter used to balance the weight of distillation loss in the total loss, The distillation loss for the k-th student model is... is the total loss of the kth student model; k takes the value 1 or 2, corresponding to the EEG prediction student model and the ECG prediction student model, respectively.

[0050] Each student model is trained simultaneously, and the parameters of each student model are optimized and updated by minimizing the total loss function.

[0051] In addition, this application also discloses an electronic device comprising a memory and a processor, wherein:

[0052] Memory is used to store computer programs that can run on a processor;

[0053] A processor is configured to execute, while running the computer program, an emotion recognition method based on online cross-modal knowledge distillation as described above.

[0054] Also disclosed is a computer-readable storage medium storing computer instructions for causing a processor to execute an emotion recognition method based on online cross-modal knowledge distillation as described above.

[0055] Based on the above technical solution, the present invention has at least the following beneficial effects:

[0056] This invention introduces dynamically generated teacher supervision signals during the unified training process to achieve real-time knowledge interaction and collaborative learning between EEG and ECG modalities. This avoids the problems of static and fixed teacher models and insufficient adaptability in traditional offline distillation, and effectively improves the overall accuracy and stability of emotion classification.

[0057] This invention combines EEG and ECG signals for emotion recognition, obtaining emotion-related information from two different physiological levels: central nervous system activity and autonomic nervous system regulation, overcoming the limitations of single-modality emotion expression. Through complementary modeling of multimodal signals, the model can more comprehensively characterize emotional states and improve its ability to distinguish and generalize complex emotion categories.

[0058] This invention addresses the significant differences in signal structure and physiological characteristics between electroencephalogram (EEG) and electrocardiogram (ECG) signals by designing modality-specific encoders for each. These encoders specifically model the frequency, temporal, and multi-channel spatial information of EEG signals, as well as the multi-peak morphology and temporal variation information of ECG signals. This enhances the discriminativeness and reliability of the feature representations for each modality, providing a high-quality feature foundation for subsequent cross-modal collaborative learning.

[0059] In the feature learning stage, this invention introduces adaptive contrastive loss to constrain the intermediate features output by different modal encoders, so that EEG features and ECG features are aligned in the shared emotional semantic space, enhancing the consistency and complementarity of cross-modal features and reducing the adverse effects of intermodal distribution differences on model training.

[0060] At the model output level, this invention performs joint encoding and adaptive fusion of the predicted outputs of each modality, and comprehensively utilizes the complementary advantages of polar probability distribution and marginal probability distribution to dynamically construct a high-quality teacher probability distribution. Among them, the polar probability distribution enhances the ability to characterize the boundary of emotion category discrimination, and the marginal probability distribution improves the stability and reliability of soft supervision signals, thereby providing more effective soft supervision information for student models.

[0061] This invention employs a multi-loss collaborative optimization mechanism, which performs weighted joint optimization of label loss, adaptive contrast loss, and distillation loss. While ensuring stable convergence during model training, it guides student models of each modality to improve together at the classification and feature levels, resulting in a stronger robustness and generalization performance of the obtained emotion recognition model.

[0062] In this invention, each modal model can independently complete emotion prediction during the inference stage without relying on additional fusion or distillation operations. Therefore, no additional computational overhead is introduced during the testing stage. It balances the improvement of recognition performance with the efficiency requirements of practical application deployment, and has good engineering practical value and application prospects. Attached Figure Description

[0063] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0064] Figure 1This is a flowchart of an emotion recognition method based on online cross-modal knowledge distillation proposed in this invention;

[0065] Figure 2 This is a schematic diagram of the structure of the online cross-modal knowledge distillation network in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of the structure of the EEG encoder and ECG encoder in an embodiment of the present invention;

[0067] Figure 3 (a) in the diagram is a schematic diagram of the EEG encoder structure. Figure 3 (b) in the diagram is a schematic diagram of the electrocardiogram encoder structure;

[0068] Figure 4 This is a schematic diagram of the combined encoder in an embodiment of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0070] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0071] like Figures 1-4 As shown, this embodiment provides an emotion recognition method based on online cross-modal knowledge distillation. Following a unified online training framework, it utilizes the complementary properties of EEG and ECG signals, introduces adaptive contrast constraints during the feature learning stage, and jointly encodes and adaptively fuses the predicted outputs of each modality to dynamically generate a teacher probability distribution. By effectively utilizing the online knowledge distillation mechanism, it achieves real-time knowledge interaction and bidirectional collaborative learning between modalities, improving the accuracy, stability, and robustness of the emotion recognition model. Figure 1 As shown, the method specifically includes the following steps:

[0072] S1. Obtain the raw EEG and ECG signals, and perform window slicing to obtain EEG signal data segments and ECG signal data segments respectively;

[0073] As a preferred implementation, raw EEG and ECG signals can be acquired in real time using a data acquisition device or directly obtained from a database. In this embodiment, the raw EEG and ECG signals are derived from MAHNOB-HCI (Human-Computer Interaction Tagging), a multimodal database for emotion recognition and implicit labeling research. This multimodal database was used in experiments involving browsing stimulus videos or images and making emotional responses and label judgments. The experiments used approximately 20 carefully selected video clips, each ranging from 34 to 117 seconds in length, plus a baseline recording period of 30 seconds before and after. After viewing the stimuli, participants rated emotional dimensions such as valence, arousal, sense of control, and predictability (on a scale of 1-9) and made implicit emotional label judgments. This yielded experimental data including 32-channel EEG signals, 3-channel ECG signals, and various peripheral physiological signals (skin conductance, temperature, respiration, etc.) and eye movement signals. After removing some missing data from the participants' data, 25 complete experimental data sets were obtained from the publicly available portion. The 25 complete datasets were preprocessed, including bad channel removal, signal synchronization calibration, filtering, and baseline correction. Valence and arousal labels were binarized based on a threshold of 5. This invention uses the preprocessed complete experimental data as the raw EEG and ECG signals, primarily performing binary classification of emotion categories based on either the valence or arousal labels.

[0074] After obtaining the raw EEG and ECG signals, this embodiment performs window slicing processing using a 2-second segmentation window with no overlapping portions, dividing the continuous signal into signal data segments. Each modality signal data segment ultimately contains 512 data points. The resulting dataset is obtained by integrating the sliced ​​data. Then divide it into training sets in an 8:2 ratio. and test set The specific steps for obtaining dataset X are as follows:

[0075] The EEG signals after window segmentation are denoted as The electrocardiogram signals are denoted as follows: Each data point is in the form of channel number × time length, and the binarized valence label / awakening label is denoted as... Based on this, a dataset is constructed. , The sample size is denoted as . This represents the EEG modality sample of the i-th subject, and its shape is represented as... ; This represents the ECG modality sample of the i-th subject, and its shape is represented as... , Then it means the first Each sample has a true label, ranging from 0 to 1. That is, each sample includes corresponding EEG signals, ECG signals, and a true label. All participant samples are integrated to form the dataset. Based on this dataset, we can make continuous and rapid predictions of the current emotional state.

[0076] S2. Construct EEG-predicted student models and ECG-predicted student models for emotion recognition. Each student model includes an encoder and a classifier, such as... Figure 2 As shown.

[0077] EEG signal data segments are input into an EEG encoder to extract intermediate features of the EEG signal; ECG signal data segments are input into an ECG encoder to extract intermediate features of the ECG signal; the intermediate features of the EEG signal include the frequency and time of the EEG signal, as well as the multi-channel spatial information of the EEG signal; the intermediate features of the ECG signal include the peak morphology information of the ECG signal and the temporal variation information of the ECG signal.

[0078] As a preferred embodiment, such as Figure 3 As shown in (a), the EEG encoder consists of a frequency band decoding and filtering module, a frequency convolution, a temporal convolution, a first feature activation fusion module, a multi-scale graph convolution module, a second feature activation fusion module, and a linear layer connected in series, wherein:

[0079] The frequency band decoding and filtering module decomposes the original broadband EEG signal into signals in six different frequency bands, including the delta band (1-4Hz), theta band (4-8Hz), the alpha band (8-12Hz), the beta1 band (12-18Hz), the beta2 band (18-30Hz), and the gamma band (30-45Hz); and stacks them along the frequency band dimension to obtain the multi-frequency band features of the EEG signal;

[0080] The kernel size of the frequency convolution is set to 1×6 to extract cross-frequency correlation features between different frequency bands from the multi-band features. The kernel size of the temporal convolution is set to 1×5, the stride is set to 5, and the number of convolutions, i.e. the number of output channels, is 32, to extract local temporal features of the EEG signal from the multi-band features.

[0081] The first feature activation fusion module fuses the cross-frequency correlation features and local temporal features of the EEG signal to obtain preliminary fused features;

[0082] The multi-scale graph convolutional module extracts spatial features of EEG signals at different scales from the preliminary fusion features. Preferably, in this embodiment, the multi-scale graph convolutional module consists of a linear transformation layer and a graph convolutional network; the linear transformation layer projects the preliminary fusion features into multi-scale features with different numbers of channels (in this embodiment, the original channels are first transformed from...). The projection is shown as three channels of different sizes. , and Then, three graph convolutional networks of different scales are used to perform graph convolution operations on the multi-scale features respectively. The adjacency matrix of each graph convolutional network corresponds to the number of three channels after projection, thereby extracting the spatial features of the EEG signal at different scales. The spatial features of different scales are stacked along the scale dimension (the number of channels must be unified to the same before stacking along the scale dimension) to obtain the multi-scale spatial features of the EEG signal.

[0083] The second feature activation fusion module activates and fuses multi-scale spatial features to obtain multi-scale fused features; both the first and second feature activation fusion modules consist of batch regularization layers, ReLU activation functions, and fusion convolutions (with a kernel size of 1×1) connected in sequence.

[0084] The linear layer aligns the multi-scale fused features to obtain intermediate features of the EEG signal. In this embodiment, the intermediate feature of the EEG signal corresponding to the i-th sample is denoted as... .

[0085] As a preferred embodiment, such as Figure 3 As shown in (b) of the figure, the ECG encoder consists of a spatial convolution module, a multi-scale convolution module (the (B, N, K, T) labeled in the figure are the input / output feature dimension parameters of the multi-scale convolution module, where B is the batch size, N is the number of feature channels, K is the kernel size, and T is the time step), a temporal feature fusion module, a peak feature enhancement module, a one-dimensional convolution module, and a linear layer, which are connected in series.

[0086] Spatial convolution sets the kernel size to 1× The number is 32, which are used to extract the spatial channel features of the electrocardiogram signal in the channel dimension;

[0087] The multi-scale convolution module extracts local temporal features of ECG signals at different scales from spatial channel features through three one-dimensional convolutions with kernel sizes of 3, 5 and 7.

[0088] The temporal feature fusion module consists of a batch regularization layer, a ReLU activation function, and a fusion convolution (with the same kernel size of 1×1) connected in sequence, which fuses local temporal features of ECG signals at different scales into multi-scale temporal fusion features;

[0089] The peak feature enhancement module consists of a max pooling layer, a batch regularization layer, and a ReLU activation function connected in series, which extracts the peak features of the electrocardiogram signal from the multi-scale temporal fusion features.

[0090] One-dimensional convolution is then used to reduce the dimensionality of the obtained peak features;

[0091] The linear layer aligns the dimensionality-reduced peak features to obtain the intermediate features of the electrocardiogram (ECG) signal. In this embodiment, the intermediate feature of the ECG signal corresponding to the i-th sample is denoted as... .

[0092] Using the aforementioned EEG and ECG encoders, this method extracts discriminative intermediate features from EEG and ECG signals. Furthermore, the EEG encoder models the changes in EEG signals across different frequency bands and temporal dimensions through frequency and temporal convolution, effectively capturing emotion-related rhythmic and instantaneous dynamic features. Simultaneously, it introduces a multi-scale convolutional structure to adapt to the complex variations in EEG signals across different time scales and spatial channels, thereby enhancing the representational ability of multi-frequency, multi-channel EEG patterns. The ECG encoder models the morphological features of multi-lead ECG signals through spatial convolution, focusing on extracting multi-peak structure information closely related to emotional states. Subsequently, a max-pooling layer is used to downsample local temporal features. This operation highlights representative peak responses in the ECG signal, enhancing robust representation of heart rate changes and differences in heartbeat morphology, while suppressing noise interference, consistent with the physiological characteristics of ECG signals, which are primarily characterized by periodic waveforms and significant peaks.

[0093] Furthermore, this method incorporates multiple batch regularization layers and ReLU activation functions into both the EEG encoder and the ECG encoder to accelerate model convergence, stabilize the training process, and enhance nonlinear expressive capabilities. Through this modality-specific structural design, the EEG encoder can fully extract the unique information of EEG signals in the frequency, time, and spatial dimensions, while the ECG encoder can effectively capture the waveform structure and temporal variation features of ECG signals. This enables accurate extraction of modality-specific emotional features, providing high-quality intermediate feature representations for subsequent cross-modal collaborative learning.

[0094] S3. Input the intermediate features of the EEG signal into the EEG classifier and the intermediate features of the ECG signal into the ECG classifier to obtain the unnormalized EEG prediction output and the unnormalized ECG prediction output, respectively.

[0095] In this embodiment, the classifier structure is consistent across all modalities, consisting of a first linear layer, a ReLU activation function, a forgetting layer, and a second linear layer in sequence. This ensures consistent mapping capabilities and comparability across different modalities during the prediction phase. Specifically, the input dimension of the first linear layer of the classifier is consistent with the intermediate feature dimension, while the output dimension is set to 64 to enhance the expressive power of emotion discrimination features. A ReLU activation function and a forgetting layer are introduced between the first and second linear layers to improve the stability and generalization performance of the classification results. The second linear layer takes 64-dimensional features as input and maps them to the unnormalized prediction output logits corresponding to each emotion category, representing the discrimination score of each emotion category. This achieves a precise mapping from the feature space to the emotion category space, completing the emotion recognition and classification task for each student model. Through the structural design combining two layers of linear mapping and nonlinear enhancement, the efficient transformation of complex intermediate features into emotion category discrimination results can be effectively achieved, providing a stable and consistent prediction output foundation for subsequent cross-modal collaborative learning and knowledge distillation.

[0096] S4. Use a joint encoder to fuse the prediction outputs of different modalities, and dynamically generate the teacher probability distribution through an adaptive generation mechanism of polarity and edges.

[0097] In a preferred embodiment, step S4 specifically comprises:

[0098] Real-time acquisition of unnormalized EEG prediction output and unnormalized ECG prediction output;

[0099] The unnormalized EEG prediction output and unnormalized ECG prediction output are normalized by using the Softmax function combined with the temperature coefficient τ to obtain the EEG prediction probability distribution and ECG prediction probability distribution, respectively.

[0100] For each emotion category c, calculate the standard deviation of the EEG prediction probability distribution and the ECG prediction probability distribution within c, and obtain the EEG prediction standard deviation. and ECG prediction standard deviation The formula for calculating the predictive standard deviation is as follows:

[0101] ;

[0102] in, This represents the standard deviation of the prediction of the k-th student model under emotion category c; This represents the original prediction output (i.e., the unnormalized prediction output) of the k-th student model for the j-th sample. This indicates that under the temperature coefficient τ, The predicted probability distribution is obtained after normalization using the Softmax function; This represents the true label of the j-th sample; The standard deviation function is represented by its subscript. This means that the prediction standard deviation of the k-th student model is calculated only for samples whose true label is equal to category c; k takes the value 1 or 2, when it is 1 it means the EEG predicts the student model, when it is 2 it means the ECG predicts the student model;

[0103] Using the Softmax function , Normalization was performed to obtain the polarity weights of the EEG modes. Polarity weights of ECG modes The formula for calculating polarity weight is:

[0104] ;

[0105] in, This represents an exponential function with base e; The parameters for iterating through each student model are used to support the summation operation; Let represent the polarity weight of the k-th student model for the j-th sample under emotion category c. , That is, it is obtained by concatenating the polarity weights of each sample;

[0106] The polarity probability distribution is obtained by weighting and fusing the EEG prediction probability distribution and the ECG prediction probability distribution using polarity weights. The formula is expressed as:

[0107] ;

[0108] For each emotion category c, and combined with the true label, an edge-adjusted probability is constructed: if c is the target category corresponding to the true label, the predicted probability is set to 1; if c is not the target category, the original predicted probability is retained. In this embodiment, the formula for calculating the edge-adjusted probability is:

[0109] ;

[0110] in, This represents the original prediction output of the model for the k-th student. Let represent the original predicted probability of the k-th student model for the c-th emotion category of the j-th sample under the temperature coefficient τ (the output of the classifier is normalized by the Softmax function). This represents the marginal adjustment probability of the k-th student model for the c-th emotion category of the j-th sample;

[0111] The marginal probability distribution is obtained by calculating the EEG marginal adjustment probability and the ECG predictive adjustment probability separately, retaining the minimum value between the two, and then normalizing it using Softmax. The formula is expressed as:

[0112] ;

[0113] polarity probability distribution With marginal probability distribution Linear fusion is performed according to preset weights to dynamically generate teacher probability distributions. The formula is expressed as:

[0114] ;

[0115] in, This represents the probability distribution of teachers under a temperature coefficient τ. and The weighting coefficients are used to control the two probability distributions.

[0116] In this embodiment, the complementary roles of polar probability distribution and marginal probability distribution in teacher signal generation are comprehensively considered. The polar probability distribution focuses on characterizing the discriminative differences between different categories, helping to enhance the model's ability to model category diversity; the marginal probability distribution focuses on providing smoother and more stable probability estimates, thus generating more reliable soft supervision information. By introducing a correlation coefficient to adaptively weight and fuse these two types of probability distributions, a high-quality teacher probability distribution is dynamically constructed to guide the model training process. This adaptive generation mechanism of polarity and marginality promotes effective collaborative learning between student models of different modalities, enabling full interaction and sharing of knowledge during the training phase, and enhancing the model's robustness to input perturbations and noise.

[0117] S5. Introduce label loss and adaptive contrast loss to align intermediate features of different modalities; as a preferred embodiment, in this example, the calculation process of the label loss and the adaptive contrast loss is as follows:

[0118] Calculate the difference between each student's model prediction and the true label, and use this difference as the label loss; the formula is as follows:

[0119] ;

[0120] in, This represents the label loss of the k-th student model. Represents the cross-entropy function. For real labels, Let represent the predicted probability distribution of the model for the k-th student;

[0121] The label loss is normalized using the Softmax function, and the adaptive weights of the contrastive loss for each student model are obtained, expressed by the formula:

[0122] ;

[0123] in, This represents the adaptive weights of the contrastive loss for the k-th student model; m is the parameter for iterating through the two student models, used for the summation operation.

[0124] The contrastive learning loss InfoNCE between intermediate features of EEG signals and intermediate features of ECG signals is calculated (in this embodiment, features from the same sample across modalities are considered positive pairs, and features from different samples across modalities are considered negative pairs). This is then multiplied by the adaptive weights of the contrastive loss for each student model to obtain the adaptive contrastive loss for each student model. The formula is as follows:

[0125] ;

[0126] in, This represents the adaptive contrastive loss of the k-th student model. Indicates the InfoNCE contrast loss, through Adjust the learning level of the model to ensure that the poorly performing student model learns more from the well-performing student model.

[0127] In this embodiment, adaptive contrastive loss enables student models to learn more robust and consistent cross-modal representations in a shared sentiment semantic space, effectively mitigating the inconsistency problem caused by differences in feature distributions across different modalities. Simultaneously, the contrastive loss weights are dynamically calculated based on the label loss, adaptively adjusting the learning intensity of each modal student model according to its current prediction performance. This allows weaker models to acquire more effective knowledge from higher-performing student models, while stronger models avoid excessive interference from noisy features.

[0128] Then, distillation loss is introduced to align the predicted probability distribution of each student model with the teacher's probability distribution. As a preferred implementation, in this embodiment, the calculation process of the distillation loss is as follows:

[0129] The unnormalized EEG prediction output and unnormalized ECG prediction output are normalized by using the Softmax function combined with the temperature coefficient τ to obtain the EEG prediction probability distribution and ECG prediction probability distribution, respectively.

[0130] For each student model, calculate the KL divergence loss between its corresponding predicted probability distribution and the teacher's probability distribution, which serves as the distillation loss for each student model. The formula is as follows:

[0131] ;

[0132] in, Indicated in temperature coefficient Below, the predicted probability distribution of the k-th student model; This represents the KL divergence loss.

[0133] In this embodiment, by introducing a distillation loss function, the predicted probability distributions of each student model are guided to align with the teacher's probability distribution. This allows the student models to not only learn real label information but also fully absorb the soft supervision knowledge inherent in multimodal collaborative modeling, thereby enhancing the consistency and discriminative ability of prediction results across different modalities. This mechanism helps alleviate the problem of limited learning capacity of a single modality, promotes knowledge sharing and complementarity among modalities, and improves the overall performance and robustness of the emotion recognition model.

[0134] Then, based on label loss, adaptive contrast loss, and distillation loss, the total loss of each student model during the training phase is calculated, and the parameters of the EEG prediction student model and the ECG prediction student model are updated synchronously through online collaborative training; as a preferred implementation, such as Figure 4 As shown, specifically:

[0135] During online collaborative training, each student model independently completes forward inference within each mini-batch of training samples, obtaining intermediate features of each modality, unnormalized prediction outputs of each modality, and dynamically generating teacher prediction distributions.

[0136] Based on the intermediate features of each modality, the unnormalized prediction output of each modality, and the teacher prediction distribution, the label loss, adaptive contrastive loss, and distillation loss of each student model are calculated separately, and then weighted to obtain the total loss, expressed by the formula:

[0137] ;

[0138] in, Let the label loss be the model for the k-th student. For the adaptive contrastive loss of the k student models, The hyperparameter used to balance the weight of distillation loss in the total loss, The distillation loss for the k-th student model is... This represents the total loss of the k-th student model; k takes the value 1 or 2, corresponding to the EEG prediction student model and the ECG prediction student model, respectively; additionally, it should be noted that... Figure 2 In , , These represent the operations for calculating label loss, adaptive contrast loss, and distillation loss, respectively, without specifically distinguishing which student model it is;

[0139] In the online knowledge distillation paradigm, each student model is trained simultaneously, and they exchange knowledge and learn from each other in each mini-batch through dynamically generated teacher probability distributions. The optimization and updating of the parameters of each student model are achieved by minimizing the total loss function.

[0140] In this embodiment, the training set is used. Train the overall emotion recognition network and save the parameters of each student model. Then, let the trained student models run on the test set. The effectiveness of the model system is verified.

[0141] S6. After training, the EEG or ECG signal to be identified is input into the student model of the corresponding modality. The signal is then passed through the encoder and classifier to obtain the unnormalized prediction output. Finally, the probability distribution of each emotion category is obtained through the Softmax function, which serves as the emotion recognition result.

[0142] During the inference phase, each student model can independently complete the prediction without relying on additional fusion or distillation operations. Therefore, the performance improvement brought by this invention will not introduce additional test computation overhead, ensuring the efficiency and practicality of the model. Through cross-modal online knowledge distillation and adaptive collaborative training mechanism, the EEG and ECG models can achieve deep information interaction during the training phase while maintaining the independence of the inference phase, effectively improving the accuracy, stability and robustness of the emotion recognition model in complex scenarios.

[0143] In addition, this application also discloses an electronic device comprising a memory and a processor, wherein:

[0144] Memory is used to store computer programs that can run on a processor;

[0145] A processor is configured to execute, while running the computer program, an emotion recognition method based on online cross-modal knowledge distillation as described above.

[0146] Also disclosed is a computer-readable storage medium storing computer instructions for causing a processor to execute an emotion recognition method based on online cross-modal knowledge distillation as described above.

[0147] In summary, the method proposed in this invention, through signal complementarity, modality-specific feature extraction, cross-modal feature alignment, and a polarity-edge joint teacher signal generation mechanism, combined with online cross-modal knowledge distillation and multi-loss collaborative optimization, achieves real-time collaborative learning and high-quality dynamic supervision between modalities compared to traditional simple multimodal fusion and offline distillation methods. This guides the model to obtain more stable, consistent, and discriminative emotion representations, thereby significantly improving the accuracy of emotion recognition, the model's generalization ability, and the integrity of physiological signal representations. It has good application prospects and broad application potential.

[0148] In this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to at least one embodiment or example described in connection with a specific feature, structure, material, or characteristic. These specific features, structures, materials, or characteristics may be combined in a suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples and their features described in this specification.

[0149] The logic and / or steps shown in the flowchart or otherwise described can be viewed as a sequence of executable instructions for implementing logical functions. These instructions may be implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device. Such systems, apparatus, or devices include processor systems or other systems capable of receiving and executing instructions.

[0150] The above embodiments detail the principles and implementation methods of the present invention, and illustrate its working principle using specific examples. These examples are only used to help understand the method and core ideas of the present invention. Furthermore, based on the ideas of the present invention, actual implementation methods and application scope may vary. Therefore, the content of this specification should not be construed as limiting the present invention.

Claims

1. A sentiment recognition method based on online cross-modal knowledge distillation, characterized in that, Specifically, the following steps are included: The raw EEG and ECG signals were acquired and then sliced ​​into windows to obtain EEG signal data fragments and ECG signal data fragments, respectively. Emotion recognition is performed by constructing EEG-predicted student models and ECG-predicted student models, with each student model including an encoder and a classifier; Input the EEG signal data segments into the EEG encoder to extract intermediate features of the EEG signals; Input ECG signal data segments into an ECG encoder to extract intermediate features of the ECG signal; The intermediate features of the EEG signal include the frequency and time of the EEG signal, as well as the multi-channel spatial information of the EEG signal. The intermediate features of the ECG signal include the peak morphology information of the ECG signal and the temporal variation information of the ECG signal. The EEG encoder is composed of a frequency band decoding and filtering module, a frequency convolution, a temporal convolution, a first feature activation fusion module, a multi-scale graph convolution module, a second feature activation fusion module, and a linear layer connected in series. The ECG encoder is composed of a spatial convolution, a multi-scale convolution module, a temporal feature fusion module, a peak feature enhancement module, a one-dimensional convolution, and a linear layer connected in series. The intermediate features of the EEG signal are input into the EEG classifier, and the intermediate features of the ECG signal are input into the ECG classifier to obtain the unnormalized EEG prediction output and the unnormalized ECG prediction output, respectively. A joint encoder is used to fuse the prediction outputs of different modalities. A teacher probability distribution is dynamically generated through an adaptive generation mechanism of polarity and edges. Specifically: Real-time acquisition of unnormalized EEG prediction output and unnormalized ECG prediction output; The unnormalized EEG prediction output and unnormalized ECG prediction output are normalized by using the Softmax function combined with the temperature coefficient τ to obtain the EEG prediction probability distribution and ECG prediction probability distribution, respectively. For each emotion category c, calculate the standard deviation of the EEG prediction probability distribution and the ECG prediction probability distribution within c, and obtain the EEG prediction standard deviation. and ECG prediction standard deviation ; using the Softmax function to , Normalization was performed to obtain the polarity weights of the EEG modes. Polarity weights of ECG modes The polarity probability distribution is obtained by weighting and fusing the EEG prediction probability distribution and the ECG prediction probability distribution using polarity weights. ; For each emotion category c, and in conjunction with the true label, a marginal adjustment probability is constructed: if c is the target category corresponding to the true label, the predicted probability is set to 1; if c is not the target category, the original predicted probability is retained. The EEG marginal adjustment probability and the ECG predicted adjustment probability are calculated separately, and the minimum value between the two is retained. After Softmax normalization, the marginal probability distribution is obtained. ; polarity probability distribution With marginal probability distribution Linear fusion is performed according to preset weights to dynamically generate a teacher probability distribution; Label loss and adaptive contrastive loss are introduced to align intermediate features from different modalities, and distillation loss is introduced to align the prediction probability distribution of each student model with the teacher's probability distribution. The total loss of each student model during the training phase is calculated based on label loss, adaptive contrastive loss, and distillation loss. The parameters of the EEG prediction student model and the ECG prediction student model are synchronously updated through online collaborative training. Specifically: During online collaborative training, each student model independently completes forward inference within each mini-batch of training samples, obtaining intermediate features of each modality, unnormalized prediction outputs of each modality, and dynamically generating teacher prediction distributions. Based on the intermediate features of each modality, the unnormalized prediction output of each modality, and the teacher prediction distribution, the label loss, adaptive contrastive loss, and distillation loss of each student model are calculated separately, and then weighted to obtain the total loss, expressed by the formula: ； in, Let the label loss be the model for the k-th student. For the adaptive contrastive loss of the k student models, The hyperparameter used to balance the weight of distillation loss in the total loss, The distillation loss for the k-th student model is... is the total loss of the kth student model; k takes the value 1 or 2, corresponding to the EEG prediction student model and the ECG prediction student model, respectively. Each student model is trained simultaneously, and the parameters of each student model are optimized and updated by minimizing the total loss function; After training, the EEG or ECG signal to be identified is input into the student model of the corresponding modality. The signal is then passed through an encoder and a classifier to obtain the unnormalized prediction output. Finally, the probability distribution of each emotion category is obtained through the Softmax function, which serves as the emotion recognition result.

2. The emotion recognition method based on online cross-modal knowledge distillation according to claim 1, characterized in that, The specific modules in the EEG encoder are as follows: The frequency band decoding and filtering module decomposes the original broadband EEG signal into signals of different frequency bands and stacks them along the frequency band dimension to obtain the multi-frequency band features of the EEG signal. Frequency convolution extracts cross-frequency correlation features between different frequency bands from multi-band features, while temporal convolution extracts local temporal features of EEG signals from multi-band features. The first feature activation fusion module fuses the cross-frequency correlation features and local temporal features of the EEG signal to obtain preliminary fused features; The multi-scale graph convolution module extracts spatial features of EEG signals at different scales from the preliminary fusion features, and stacks spatial features of different scales along the scale dimension to obtain multi-scale spatial features of EEG signals. The second feature activation and fusion module activates and fuses multi-scale spatial features to obtain multi-scale fused features; Both the first feature activation fusion module and the second feature activation fusion module consist of a batch regularization layer, a ReLU activation function, and a fusion convolution connected in sequence. The linear layer aligns the multi-scale fusion features to obtain intermediate features of the EEG signal.

3. The emotion recognition method based on online cross-modal knowledge distillation according to claim 2, characterized in that, The multi-scale graph convolution module consists of a linear transformation layer and a graph convolution network. The linear transformation layer projects the initially fused features into multi-scale features with different numbers of channels. Then, graph convolution operations are performed on the multi-scale features through graph convolution networks of different scales to extract spatial features of EEG signals at different scales.

4. The emotion recognition method based on online cross-modal knowledge distillation according to claim 1, characterized in that, The specific modules in the electrocardiogram encoder are as follows: Spatial convolution extracts spatial channel features of electrocardiogram signals in the channel dimension; The multi-scale convolution module extracts local temporal features of ECG signals at different scales from spatial channel features through one-dimensional convolution with different kernel sizes. The temporal feature fusion module consists of a batch regularization layer, a ReLU activation function, and a fusion convolution connected in sequence, which fuses local temporal features of ECG signals at different scales into multi-scale temporal fusion features; The peak feature enhancement module consists of a max pooling layer, a batch regularization layer, and a ReLU activation function connected in series, which extracts the peak features of the electrocardiogram signal from the multi-scale temporal fusion features. One-dimensional convolution is then used to reduce the dimensionality of the obtained peak features; The linear layer aligns the peak features after dimensionality reduction to obtain the intermediate features of the electrocardiogram signal.

5. The emotion recognition method based on online cross-modal knowledge distillation according to claim 1, characterized in that, The calculation process for the label loss and the adaptive contrast loss is as follows: Calculate the difference between the model prediction results and the true labels for each student, and use it as the label loss; The label loss is normalized using the Softmax function to obtain the adaptive weights of the contrastive loss for each student model. The contrastive learning loss InfoNCE between intermediate features of EEG signals and intermediate features of ECG signals is calculated, and then multiplied by the adaptive weights of the contrastive loss of each student model to obtain the adaptive contrastive loss of each student model.

6. The emotion recognition method based on online cross-modal knowledge distillation according to claim 1, characterized in that, The calculation process for the distillation loss is as follows: The unnormalized EEG prediction output and unnormalized ECG prediction output are normalized by using the Softmax function combined with the temperature coefficient τ to obtain the EEG prediction probability distribution and ECG prediction probability distribution, respectively. For each student model, calculate the KL divergence loss between its corresponding predicted probability distribution and the teacher's probability distribution, which serves as the distillation loss for each student model.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein: Memory is used to store computer programs that can run on a processor; A processor, configured to, while running the computer program, execute an emotion recognition method based on online cross-modal knowledge distillation as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute an emotion recognition method based on online cross-modal knowledge distillation as described in any one of claims 1-6.

Images