EEG-EOG Multimodal Emotion Recognition Method and System Based on Contrastive Learning

Abstract

The present invention discloses an EEG-EOG multimodal emotion recognition method based on contrastive learning, comprising the following steps: S1: Design an experimental paradigm; S2: Collect EEG signals and EOG signals of subjects; S3: Data preprocessing; S4: Contrastive learning: Use a base encoder and a projection module to align mini-batch data of EEG and EOG signal segments of the same subject, and continuously optimize the parameters of the base encoder and the projection module; S5: Use the pre-trained base encoder to extract DE emotion features from the aligned EEG and EOG signal representations; S6: Classification and recognition to obtain the recognition rate of emotion recognition. Also disclosed is an EEG-EOG multimodal emotion recognition system based on contrastive learning. The present invention is based on the contrastive learning method, pre-trains the base encoder to align EEG and EOG signals, thereby realizing the fusion of multimodal signals and improving the recognition rate of emotion recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of electroencephalogram signal processing, and particularly to an EEG-EOG multimodal emotion recognition method and system based on contrastive learning. Background Art

[0002] Emotions play an important role in interpersonal communication and social environments. As an important field of current artificial intelligence applications, emotion recognition has broad application prospects in psychotherapy, disease prevention, etc. Moreover, cognitive theories and psychological experiment results show that brain activities are closely related to emotions. As physiological signals capable of recognizing emotions, electroencephalogram (EEG), electro-oculogram (EOG), electromyogram (EMG), galvanic skin response (GSR), and respiration rate (RR) are usually used to reflect emotional states. Among them, EEG is the most commonly used because it has good temporal resolution and spatial resolution. In addition, EEG has been widely applied in brain-computer interfaces (BCIs). Research on emotion recognition based on EEG may provide great value for improving the user experience and performance of BCI applications. Recently, advanced technologies have combined EEG with other physiological signals, such as EOG, and adopted multimodal brain-computer interface systems to improve the accuracy and speed of standard unimodal brain-computer interfaces.

[0003] Multimodal data usually has the characteristics of complementarity, mutual verification, fusion, and convertibility. Complementarity is an important property of multimodal data. Various data types can provide partial explanations and descriptions of a certain thing or phenomenon. The structural differences between different types of raw data are very large, but after appropriate processing, these diverse data can be fused together and even converted into each other to verify the consistency of the results generated by the same event.

[0004] Currently, research on multimodal emotion recognition based on EEG and EOG has not been deeply studied. There is little research on emotion recognition by combining the two modalities, and simply concatenating the features of the two modalities does not consider the phase difference of the data of the two modalities. Most research has only performed simple concatenation processing on multimodal signals, making the multimodal signals still stay in their respective spaces. It is difficult to fuse and align the features of multimodal data, and there are some noises unrelated to emotion analysis in the data after fusion. It is difficult for multimodal encoders to learn and establish interactions between signals. Therefore, combining multimodal signals such as EEG and EOG plays an important role in emotion recognition.

[0005] Therefore, there is an urgent need to provide a new EEG-EOG multimodal emotion recognition method and its system to solve the above problems. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an EEG-EOG multimodal emotion recognition method and its system based on contrastive learning, which can fuse multimodal signals, so as to obtain more advantageous recognition results in the process of emotion recognition.

[0007] To solve the above technical problem, a technical solution adopted by the present invention is: to provide an EEG-EOG multimodal emotion recognition method based on contrastive learning, including the following steps:

[0008] S1: Design an experimental paradigm: The subject watches a movie and gives feedback on the emotion state label, and at the same time, the EEG and EOG signals of the subject are recorded.

[0009] S2: Data collection: Collect the EEG signals and EOG signals of the subject in the audio-visual stimulation mode, and intercept the useful data of all the collected EEG and EOG data from the event occurrence calibration point to the event end calibration point.

[0010] S3: Data preprocessing: Preprocess the EEG and EOG signals intercepted in step S2 to obtain pure EEG and EOG signals after preprocessing.

[0011] S4: Contrastive learning: Process the preprocessed EEG and EOG signals to generate a small batch of data containing several pairs of EEG and EOG signal segments, and use a base encoder and a projection module to align the small batch of data of the EEG and EOG signal segments of the same subject, and continuously optimize the parameters of the base encoder and the projection module.

[0012] S5: Feature extraction: Use the pre-trained base encoder to align the preprocessed EEG and EOG signals, and then extract DE emotion features from the aligned EEG and EOG signal representations, and smooth the extracted DE emotion features using a linear dynamic system model.

[0013] S6: Classification and recognition: Divide the smoothed features into a training set and a test set respectively, put the training set and the corresponding emotion state labels into the SVM classifier for learning and classification to obtain a trained model, and use this model to predict the emotion state labels of the test set to obtain the recognition rate of emotion recognition.

[0014] In a preferred embodiment of the present invention, in step S2, under video stimulation, an electroencephalogram is recorded from a 28-channel electrode cap at a sampling rate of 250 Hz using a BP neural scanning system, and an electrooculogram of the subject is recorded from a 4-channel electrode cap at a sampling rate of 250 Hz; effective 28-lead EEG signals, 4-lead EOG signals, and actual emotional state labels are obtained:

[0015] L = {L1, L2, ..., L q}

[0016] where q represents the number of emotional state labels, and the value of each label is:

[0017] L k = {-7, -6, ..., 0, ..., 6, 7}

[0018] where k represents the k-th label, k = 1, 2, …, q; the emotional labels are classified using two emotional dimensions of Valence and Arousal. The Valence dimension is represented as positive, neutral, and negative, and the Arousal dimension is represented as 1, 2, 3, …, 7, that is, -7 to -3 are negative emotions, -2 to 2 are neutral emotions, and 3 to 7 are positive emotions.

[0019] In a preferred embodiment of the present invention, the process of preprocessing the EEG and EOG signals intercepted in step S2 includes filtering, removing the mean value, and removing artifacts.

[0020] In a preferred embodiment of the present invention, the specific steps of step S4 include:

[0021] S401: Using a data sampler, the preprocessed EEG and EOG signals are processed to generate a small batch of data containing several pairs of EEG and EOG signal segments;

[0022] S402: Using a basic encoder to process the segments, converting the EEG and EOG signals of the same subject into a modality-consistent representation between multiple modalities of the same subject;

[0023] S403: Using a non-linear projection module based on a multi-layer perceptron to map the representations of the EEG and EOG aligned by the base encoder into a latent space to calculate the similarity;

[0024] S404: By minimizing the contrast loss function to maximize the similarity of the representations of two pairs of positive EEG and EOG signals, optimizing the parameters of the base encoder and the projection module, and obtaining the trained base encoder and projection module for subsequent alignment of EEG-EOG signals to extract features.

[0025] In a preferred embodiment of the present invention, in step S5, a trained base encoder is used to align the EEG and EOG data from the same subject, and DE emotion features are extracted from the aligned EEG and EOG signal representations. The specific steps are as follows:

[0026] The preprocessed data is divided into single-trial samples using a window length of 1 s without overlap, and DE features are extracted separately:

[0027]

[0028] where σ 2 represents the variances of EEG and EOG, and the extracted DE features are smoothed using the LSD model.

[0029] In a preferred embodiment of the present invention, ten-fold cross-validation is used to divide the features extracted from the aligned EEG and EOG representations into 10 parts, where 9 parts are the training set and 1 part is the test set. Then, an SVM classifier is used to learn and classify the extracted features, with the radial basis function as the kernel function, where x is the feature vector of the training set:

[0030] K(x, x i ) = exp(-|x - x i | 2 / σ 2 )

[0031] The grid search algorithm is used to optimize the penalty coefficient c and the parameter gamma in the SVM algorithm with a step size of 0.5 in the interval [-10, 10] to obtain the optimal classification result.

[0032] To solve the above technical problems, another technical solution adopted by the present invention is: to provide an EEG-EOG multimodal emotion recognition system based on contrastive learning, including:

[0033] An experimental paradigm design module for subjects to watch movies and feedback emotion state labels, while recording the EEG and EOG signals of the subjects;

[0034] A data acquisition module for acquiring the EEG signals and EOG signals of the subjects in the audiovisual stimulation mode, and intercepting useful data from all the acquired EEG and EOG data from the event occurrence calibration point to the event end calibration point;

[0035] A data preprocessing module for preprocessing the EEG and EOG signals intercepted by the data acquisition module to obtain pure EEG and EOG signals after preprocessing;

[0036] A contrastive learning module for processing pre - processed EEG and EOG signals, generating a mini - batch data containing several pairs of EEG and EOG signal segments, using a base encoder and a projection module to align the mini - batch data of EEG and EOG signal segments of the same subject, and continuously optimizing the parameters of the base encoder and the projection module;

[0037] A feature extraction module for using the pre - trained base encoder to align the pre - processed EEG and EOG signals, then extracting DE emotion features from the aligned EEG and EOG signal representations, and smoothing the extracted DE emotion features using a linear dynamical system model;

[0038] A classification and recognition module for dividing the smoothed features into a training set and a test set respectively, putting the training set and the corresponding emotion state labels into an SVM classifier for learning and classification to obtain a trained model, and using this model to predict the emotion state labels of the test set to obtain the recognition rate of emotion recognition.

[0039] In a preferred embodiment of the present invention, the data acquisition module acquires EEG signals and EOG signals of a subject in an audiovisual stimulation mode, and the specific steps of intercepting useful data from all the acquired EEG and EOG data from the event occurrence calibration point to the event end calibration point include:

[0040] Under video stimulation, using a BP neural scanning system to record electroencephalograms at a sampling rate of 250Hz from a 28 - channel electrode cap, and recording electrooculograms of the subject at a sampling rate of 250Hz from a 4 - channel electrode cap; obtaining effective 28 - lead EEG signals, 4 - lead EOG signals and actual emotion state labels:

[0041] L = {L1, L2,..., L q}

[0042] where q represents the number of emotion state labels, and the value of each label is:

[0043] L k = {-7, -6,..., 0,..., 6, 7}

[0044] where k represents the k - th label, k = 1, 2,..., q; the emotion labels are classified using two emotion dimensions of Valence and Arousal. The Valence dimension is represented as positive, neutral, negative, and the Arousal dimension is represented as 1, 2, 3,..., 7, that is, -7 to -3 are negative emotions, -2 to 2 are neutral emotions, and 3 to 7 are positive emotions.

[0045] In a preferred embodiment of the present invention, the process of the data preprocessing module preprocessing the EEG and EOG signals intercepted by the data acquisition module includes filtering, mean removal, and artifact removal.

[0046] In a preferred embodiment of the present invention, the contrastive learning module processes the preprocessed EEG and EOG signals to generate a small batch of data containing several pairs of EEG and EOG signal segments, and uses a base encoder and a projection module to align the small batch of data of the EEG and EOG signal segments of the same subject, and continuously optimizes the parameters of the base encoder and the projection module. The specific steps include:

[0047] First, use a data sampler to process the preprocessed EEG and EOG signals to generate a small batch of data containing several pairs of EEG and EOG signal segments;

[0048] Then, use the basic encoder to process the segments, and convert the EEG and EOG signals of the same subject into a modality-consistent representation between multiple modalities of the same subject;

[0049] Next, use a non-linear projection module based on a multi-layer perceptron to map the representations of the EEG and EOG aligned by the base encoder to the latent space to calculate the similarity;

[0050] Finally, by minimizing the contrastive loss function, maximize the similarity of the representations of two positive pairs of EEG and EOG signals, optimize the parameters of the base encoder and the projection module, and obtain the trained base encoder and projection module for subsequent alignment of EEG-EOG signals to extract features.

[0051] The beneficial effects of the present invention are:

[0052] (1) The data collected by the present invention has higher accuracy:

[0053] At present, emotion recognition methods mainly rely on facial expressions and speech to achieve, but facial expressions and speech are extremely vulnerable to human subjective control, so that objective results cannot be obtained and the accuracy is not high. EEG-EOG signals can record the real activities of the brain, are not subject to subjective changes, and can objectively reflect the emotional state of the subject at that time. In terms of signal acquisition, the present invention uses 28 electrodes to collect electroencephalogram data, which more completely covers different regions of the brain. At the same time, 4 electrodes are also used to collect electrooculogram signals in a bipolar lead mode. In addition, the present invention requires the subject to feedback an emotion state label every 60s, which can obtain feedback information closer to real time, reduce the generation of singular samples, so that a more accurate model can be trained subsequently and the recognition rate can be improved;

[0054] (2) The present invention provides a direction for further exploring the fusion of EEG and EOG multimodal signals:

[0055] Currently, there is little research on combining two modalities for emotion recognition. Simply concatenating the features of two modalities does not consider the phase difference between the data of the two modalities. Most studies only perform simple concatenation processing on multi-modal signals, leaving the multi-modal signals still in their respective spaces, making it difficult for the multi-modal encoder to learn and establish interactions between signals. The present invention provides an EEG-EOG multi-modal emotion recognition method based on contrastive learning. In the contrastive learning process, a base encoder is used to perform signal transformation and alignment on EEG and EOG signals, and then projection is used to map them to the latent space for calculating similarity. Then, the contrastive loss is minimized to maximize the similarity between two pairs of positive EEG and EOG signals. At the same time, the parameters of the base encoder and projection are optimized to obtain a pre-trained base encoder. In the prediction process, the pre-trained base encoder is used to align EEG and EOG data, extract DE features, and perform emotion recognition. It provides a direction for further exploring the fusion of EEG and EOG multi-modal signals;

[0056] (3) The present invention has broad application prospects:

[0057] Currently, the vast majority of emotion recognition remains at the EEG level. Data shows that the eye functions of most disabled people are not damaged. Therefore, EOG is considered to be one of the most promising physiological signals for disabled people. The present invention collects EEG and EOG signals based on audiovisual stimuli, and realizes the fusion of EEG and EOG multi-modal signals based on the method of contrastive learning, making greater use of the physiological signals of patients to analyze the physiological conditions of disabled people. Therefore, it has broad application prospects;

[0058] In addition to having important significance in the field of psychotherapy, the present invention can also be combined with VR technology to provide personalized services in the fields of VR games, VR education, etc. For example, in VR games, according to the emotional state of players, dynamically adjust elements such as game difficulty, scenes, sound effects, etc., to enhance the immersion and fun of players; in VR education, according to the emotional state of students, give feedback, encouragement, guidance, etc. in a timely manner to improve the learning interest and effect of students. Through the present invention, two-way interaction between VR device wearers and virtual environments can be achieved, making the VR experience more rich and intelligent. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flowchart of the EEG-EOG multi-modal emotion recognition method based on contrastive learning of the present invention;

[0060] Figure 2 is a schematic structural diagram of the data acquisition system;

[0061] Figure 3 is a distribution diagram of electrodes for collecting EEG signals;

[0062] Figure 4 It is a distribution diagram of electrodes for collecting EOG signals;

[0063] Figure 5 It is a schematic diagram of the experimental paradigm;

[0064] Figure 6 It is a flowchart of the contrastive learning process;

[0065] Figure 7 It is a flowchart of the emotion prediction process. Specific implementation manner

[0066] The following elaborates on the preferred embodiments of the present invention in conjunction with the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0067] Please refer to Figure 1 , the embodiments of the present invention include:

[0068] An EEG-EOG multimodal emotion recognition method based on contrastive learning, comprising the following steps:

[0069] S1: Design an experimental paradigm: First, design an experimental paradigm. Through a three-class emotion evaluation of 40 videos, there are a total of 15 trials in one experiment. There is a 5-second prompt before each trial, a 60-second video clip, a 45-second self-assessment, and a 15-second break. For the self-assessment, three questions are used:

[0070] 1) Their true feelings when watching the movie clip;

[0071] 2) Have they seen this movie before;

[0072] 3) They understand the movie clip.

[0073] The subjects watch the movie and give feedback on the emotion state labels, and at the same time, the EEG and EOG of the subjects are recorded. The specific scheme is as Figure 5 shown.

[0074] S2: Data collection: Collect the EEG signals and EOG signals of the subjects in the audiovisual stimulation mode; Intercept the useful data of all the collected EEG and EOG data from the event occurrence calibration point to the event end calibration point; The specific steps include:

[0075] Under audiovisual stimulation, according to the international 10-20 system, EEG was recorded from a 28-channel electrode cap at a sampling rate of 250 Hz using a BP neural scanning system, and EOG of the subject was simultaneously recorded in bipolar lead mode from a 4-channel electrode cap at a sampling rate of 250 Hz. Effective 28-lead EEG signals, 4-lead EOG signals and actual emotion labels were obtained:

[0076] L = {L1, L2, ..., L q}

[0077] q represents the number of emotion state labels, and the value of each label is:

[0078] L k = {-7, -6, ..., 0, ..., 6, 7}

[0079] where k represents the k-th label, k = 1, 2, …, q; the emotion labels are classified using two emotion dimensions of Valence and Arousal. The Valence dimension is represented as positive, neutral, negative, and the Arousal dimension is represented as 1, 2, 3, …, 7, that is, -7 to -3 are negative emotions, -2 to 2 are neutral emotions, and 3 to 7 are positive emotions.

[0080] S3: Data preprocessing: Filter, remove the mean value, and remove artifacts from the EEG and EOG signals intercepted in step S2 to obtain pure EEG and EOG signals after preprocessing.

[0081] S4: Contrastive learning: In the process of contrastive learning, first, a data sampler is used to generate a small batch of data containing a pair of EEG and EOG segments for training. Then, a basic encoder is used to process these segments to convert the data from unimodal signals to an aligned representation between multimodal signals. Then, a projection module is used to transform the data into another latent space for calculating similarity. Finally, by minimizing the contrastive loss function, the similarity between the representations of two positive pairs of EEG and EOG signals is maximized, and the parameters of the basic encoder and projection are optimized. The trained basic encoder and projection are used for subsequent alignment of EEG-EOG signals to extract features.

[0082] In the process of contrastive learning, first, a data sampler is used to convert the input signal into a small batch of data. In the EEG and EOG datasets, the data of one subject consists of N trials. In each trial, the subject watches an emotional video. To obtain a small batch of data, first, a random EEG sample with a time length less than the length of one experiment is drawn from each trial of subject A, and M is the number of electrodes, and T is the number of time points of an EEG sample. Then, N EOG signal samples are extracted from subject A and represented as sample and correspond to the same time period of trial i (i = 1, 2,......, N). The set In this small batch of data, given a sample sample forms a positive pair with it, and other samples and the training set form 2(N - 1) negative pairs and one positive pair.

[0083] Then the base encoder takes the EEG and EOG signals of the same subject and as inputs and converts the unimodal data of the same subject into a modality - consistent representation between multimodalities of the same subject. Specifically, a two - dimensional convolutional module is used in the base encoder to perform convolution on the EEG, and a one - dimensional convolutional module is used to perform convolution on the EOG signal.

[0084] The convolution of the EEG signal using a two - dimensional convolutional structure is expressed as:

[0085]

[0086] where X A ∈R M*T is the size of the given EEG signal, f uv is the convolution kernel size, u and v are its dimensions, there are k1 convolution kernels in total, and the output values share the weights and biases of the first convolution kernel, while the weights and biases of the k1 convolution kernels may be different, so the k1 convolution kernels generate k1 different feature maps

[0087] Due to the characteristic of the EOG signal being a dynamic change over time. Therefore, one - dimensional temporal convolution is used to learn the EOG signal.

[0088] The transformation of the EOG signal using a one - dimensional convolutional structure is expressed as:

[0089] H B = W k ·X B

[0090] where X B ∈R M*T is the size of the given EOG signal, W k ∈R 1*M is the weight of the k - th one - dimensional temporal convolution filter, and there are k1 filters in total, is the representation extracted by temporal convolution, and each row is a latent signal identified by a linear combination of the original signal X B ∈R M*T ​

[0091] Then, a nonlinear projection module based on a multilayer perceptron (MLP) is used to map the EEG and EOG representations aligned with the base encoder into the latent space to calculate the similarity.

[0092] MLP is a neural network composed of fully connected layers with at least one hidden layer, and the output of each hidden layer is transformed by the activation function ReLU (Rectified Linear Unit). The number of MLP layers and the number of hidden units in each hidden layer are hyperparameters. The MLP here has only one hidden layer. For simplicity, H is omitted below. A and H B Corner mark.

[0093] The output of the hidden layer is:

[0094]

[0095] The output of the output layer is:

[0096]

[0097] The input is The number of hidden units is h. The output of the hidden layer is have W h ∈R d*h is the weight parameter of the hidden layer, b h ∈R 1*h is the hidden layer bias parameter, W o ∈R h*q is the weight of the output layer, b o ∈R 1*q is the output layer bias parameter, and Φ represents the activation function ReLU.

[0098] Finally, convert O into a one-dimensional vector for similarity calculation. Input sample:

[0099]

[0100] Transformed by base encoder and projector into:

[0101]

[0102] Then by the formula:

[0103]

[0104] Calculate the input sample and Similarity. The purpose of the contrastive loss is to maximize the similarity between two positive pairs of EEG signals.

[0105] The Normalized Temperature-Scaled Cross Entropy Loss (NT-Xent) is calculated as:

[0106]

[0107] where is an indicator function that is set to 1 if and only if i≠j. By minimizing the such loss function, the model will increase the and similarity between, rather than all other possible pairs of samples involved. Finally, the total loss for a mini-batch of data is:

[0108]

[0109] By minimizing the contrastive loss function, the similarity between the representations of two positive pairs of EEG and EOG signals is maximized, and the parameters of the base encoder and projection are optimized. The trained base encoder and projection are obtained for subsequent alignment of EEG-EOG signals to extract features.

[0110] S5: Feature extraction: Use the trained base encoder to align the representations of EEG and EOG from the same subject, and extract differential entropy (DE) emotion features from the aligned EEG and EOG signal representations. The specific steps are as follows: Divide the data after the trained base encoder into single-trial samples using non-overlapping window lengths of 1 s each, and extract DE features respectively:

[0111]

[0112] where σ 2 represents the variance of the time series, and the extracted DE features are smoothed using a linear dynamical system model.

[0113] S6: Classification and recognition: Use the ten-fold cross-validation algorithm to divide the features extracted from the aligned EEG and EOG representations into 10 parts, where 9 parts are the training set and 1 part is the test set. Then use the Libsvm classifier to put the extracted features into a Support Vector Machine (SVM) classifier for learning and classification, with the radial basis function as the kernel function, where x is the feature vector of the training set:

[0114] K(x, x i ) = exp(-|x - x i| 2 / σ 2 )

[0115] The grid search optimization algorithm is used to optimize the penalty coefficient c and the parameter gamma in the SVM algorithm with a step size of 0.5 in the interval [-10, 10] to obtain the optimal classification result.

[0116] The example of the present invention also provides an EEG-EOG multimodal emotion recognition system based on contrastive learning, including:

[0117] An experimental paradigm design module, which is used for the subject to watch a movie and feedback the emotion state label, and at the same time record the EEG and EOG signals of the subject;

[0118] A data acquisition module, which is used to collect the EEG signals and EOG signals of the subject in the audiovisual stimulation mode, and intercept the useful data of all the collected EEG and EOG data from the event occurrence calibration point to the event end calibration point;

[0119] A data preprocessing module, which is used to preprocess the EEG and EOG signals intercepted by the data acquisition module to obtain the pure EEG and EOG signals after preprocessing;

[0120] A contrastive learning module, which is used to process the preprocessed EEG and EOG signals, generate a small batch of data containing several pairs of EEG and EOG signal segments, and use a base encoder and a projection module to align the small batch of data of the EEG and EOG signal segments of the same subject, and continuously optimize the parameters of the base encoder and the projection module;

[0121] A feature extraction module, which is used to align the preprocessed EEG and EOG signals using the pre-trained base encoder, then extract the DE emotion features from the aligned EEG and EOG signal representations, and smooth the extracted DE emotion features using a linear dynamic system model;

[0122] A classification and recognition module, which is used to divide the smoothed features into a training set and a test set respectively, put the training set and the corresponding emotion state labels into the SVM classifier for learning and classification to obtain a trained model, and use this model to predict the emotion state labels of the test set to obtain the recognition rate of emotion recognition.

[0123] The following specifically describes the method of the present invention and its advantages in combination with the accompanying drawings, specific experiments and their experimental results:

[0124] See Figure 2, which shows the schematic diagram of the data acquisition system in this embodiment. All the acquisition devices in the experiment are a 34-lead electrode cap 1 (including a reference electrode ref and a ground electrode GND) produced by BP Company in Germany, an electroencephalogram synchronization amplifier 2, a wireless receiver 3, an interface expander 4, a connecting wire 5, a computer installed with E-prime software for presenting stimuli (stimulus computer 7), and a computer installed with Vision Recorder software (acquisition computer 6). The specific acquisition devices and connections are as Figure 2 shown. Among them, the electroencephalogram synchronization amplifier 2 is directly connected to the dedicated wearable electrode cap 1 and the interface expander 4 respectively, and then the interface expander 4 is connected to the acquisition computer 6 and the stimulus computer 7 respectively. Through this device, the stimulus time, event occurrence nodes, and stimulus type labels provided by E-prime during the experiment can be transmitted to VisionRecorder. VisionRecorder saves these data, the position information (X1, X2, X3) of the electroencephalogram synchronization amplifier 2, and the corresponding electroencephalogram signals simultaneously, and displays them on the display interface of this software. The acquired data is the electroencephalogram signals of 32 channels (excluding the reference electrode, ground electrode, and amplifier position), and the sampling frequency is 250Hz.

[0125] See Figure 3 and Figure 4 , which is an illustration of the electrode distribution map used for acquiring EEG and EOG signals in this embodiment. Under video stimuli, according to the international 10 - 20 system, the electroencephalogram is recorded from a 28-channel electrode cap at a sampling rate of 250Hz using a BP neuroscan system, and the electrooculogram of the subject is simultaneously recorded in a bipolar lead mode from a 4-channel electrode cap at a sampling rate of 250Hz. Effective 28-lead EEG signals, 4-lead EOG signals, and actual emotional state labels are obtained.

[0126] See Figure 5 , which is an illustration of the experimental paradigm used in this embodiment. There are a total of 40 trials in one experiment. There is a 5-second prompt, a 60-second video clip, a 45-second self-assessment, and a 15-second break before each trial. For the self-assessment, three questions are adopted:

[0127] 1) Their true feelings when watching the movie clip;

[0128] 2) Have they seen this movie before;

[0129] 3) They understand the movie clip.

[0130] The subject watches the movie and gives feedback on the emotional state label, and at the same time, the EEG and EOG of the subject are recorded.

[0131] See Figure 6, which is an illustration of the contrastive learning method for interdisciplinary alignment. During the contrastive learning process, first, a data sampler is used to generate a mini-batch of data containing a pair of EEG and EOG picture segments for training; then these segments are processed using a basic encoder to convert the data from unimodal signals to an aligned representation between multimodal signals; and then projection is used to map the representation to another latent space for calculating similarity. At the same time, the parameters of the basic encoder and the projector are optimized to minimize the contrastive loss, and the DE features of the aligned EEG-EOG signals are extracted by the base encoder.

[0132] See Figure 7 , which is an illustration of the emotion prediction process. During the prediction process, we use the trained base encoder to align the representations of different signals of the consent subjects (shown as flat cuboids in the figure), and then extract the DE features (shown as tall cuboids in the figure) from these representations for emotion recognition. Here, the training data is represented as {X pred}, and its labels are represented as {y}. The label y is a categorical variable. For example, if there are three emotion categories, y can take three values: 0, 1, or 2. The present invention needs to predict the emotion category y of each sample .

[0133] The above are only embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or equivalent process transformation made using the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. An EEG-EOG multimodal emotion recognition method based on contrastive learning, characterized in that: The following steps are involved: S1: Experimental paradigm design: subjects watched the film and provided feedback on their emotional state labels, while EEG and EOG signals were recorded. S2: Data acquisition: Collect the EEG and EOG signals of the subjects in the audio-visual stimulation mode, and extract useful data from all the collected EEG and EOG data from the event occurrence calibration point to the event end calibration point; S3: Data preprocessing: preprocess the EEG and EOG signals intercepted in step S2 to obtain pure EEG and EOG signals after preprocessing; S4: Contrastive Learning: Process the preprocessed EEG and EOG signals to generate a mini-batch of several pairs of EEG and EOG signal segments. Use a base encoder and projection module to align the mini-batch of EEG and EOG signal segments of the same subject, and continuously optimize the parameters of the base encoder and projection module. S5: Feature extraction: Use the pre-trained base encoder to align the pre-processed EEG and EOG signals, and then extract the DE emotion features from the aligned EEG and EOG signal representations. The specific steps are as follows: Use a non-overlapping window length of 1s to divide the pre-processed data into single experimental samples and extract DE feature: in Represents the variance of EEG and EOG, and uses the LSD model to smooth the extracted DE features; The extracted DE sentiment features are smoothed using a linear dynamical system model; S6: Classification and recognition: The smoothed features are divided into training sets and test sets respectively. The training set and the corresponding emotional state labels are put into the SVM classifier for learning and classification to obtain a trained model. The model is used to predict the emotional state labels of the test set to obtain the recognition rate of emotion recognition.

2. The EEG-EOG multimodal emotion recognition method based on contrastive learning according to claim 1, characterized in that In step S2, under video stimulation, the BP neuroscan system is used to record the EEG from a 28-channel electrode cap at a sampling rate of 250 Hz, and the subject's EOG is recorded from a 4-channel electrode cap at a sampling rate of 250 Hz; a valid 28-lead EEG signal, a 4-lead EOG signal, and the actual emotional state label are obtained: Among them, q represents the number of emotional state labels, and the value of each label is: Where k represents the kth label, k = 1, 2, ..., q; the emotional label uses two emotional dimensions, Valence and Arousal, for classification. The Valence dimension is expressed as positive, neutral and negative, and the Arousal dimension is expressed as 1, 2, 3, ..., 7, that is, -7 to -3 represents negative emotions, -2 to 2 represents neutral emotions, and 3 to 7 represents positive emotions.

3. The EEG-EOG multimodal emotion recognition method based on contrastive learning according to claim 1, characterized in that: The process of preprocessing the EEG and EOG signals intercepted in step S2 includes filtering, removing the mean, and removing artifacts.

4. The EEG-EOG multimodal emotion recognition method based on contrastive learning according to claim 1, characterized in that The specific steps of step S4 include: S401: Using a data sampler, the pre-processed EEG and EOG signals are processed to generate a small batch of data containing several pairs of EEG and EOG signal segments; S402: Processing the segment using a basic encoder to convert EEG and EOG signals of the same subject into a modality-consistent representation of the same subject across multiple modalities; S403: Using a nonlinear projection module based on a multi-layer perceptron to map the EEG and EOG representations aligned with the base encoder into a latent space to calculate similarity; S404: Maximizing the similarity between the two EEG and EOG signal representations by minimizing the contrast loss function, optimizing the parameters of the base encoder and projection module, and obtaining the trained base encoder and projection module for subsequent EEG-EOG signal alignment and feature extraction.

5. The EEG-EOG multimodal emotion recognition method based on contrastive learning according to claim 1, characterized in that: Ten-fold cross validation is used to divide the features extracted from the aligned EEG and EOG representations into 10 parts, 9 of which are training sets and 1 is a test set. The extracted features are then classified using the SVM classifier, with the radial basis function as the kernel function. x is the feature vector of the training set: The grid optimization algorithm is used to optimize the penalty coefficient c and parameter gamma in the SVM algorithm in the interval [-10, 10] with a step size of 0.5 to obtain the optimal classification result.

6. An EEG-EOG multimodal emotion recognition system based on contrastive learning, characterized in that: include: The experimental paradigm design module is used for subjects to watch the film and provide feedback on the emotional state label, while recording the subjects' EEG and EOG signals; The data acquisition module is used to collect the EEG and EOG signals of the subject in the audio-visual stimulation mode, and to intercept useful data from all the collected EEG and EOG data from the event occurrence calibration point to the event end calibration point; A data preprocessing module is used to preprocess the EEG and EOG signals intercepted by the data acquisition module to obtain pure EEG and EOG signals after preprocessing; A contrastive learning module processes the preprocessed EEG and EOG signals to generate a mini-batch of several pairs of EEG and EOG signal segments. It then uses a base encoder and projection module to align the mini-batch of EEG and EOG signal segments from the same subject, and continuously optimizes the parameters of the base encoder and projection module. The feature extraction module is used to align the preprocessed EEG and EOG signals using the pre-trained base encoder, and then extract the DE emotion features from the aligned EEG and EOG signal representations. The specific steps are as follows: Use a non-overlapping window length of 1s to divide the pre-processed data into single experimental samples and extract DE feature: in Represents the variance of EEG and EOG, and uses the LSD model to smooth the extracted DE features; The extracted DE sentiment features are smoothed using a linear dynamical system model; The classification and recognition module is used to divide the smoothed features into a training set and a test set. The training set and the corresponding emotional state labels are put into the SVM classifier for learning and classification to obtain a trained model. The model is used to predict the emotional state labels of the test set to obtain the recognition rate of emotion recognition.

7. The EEG-EOG multimodal emotion recognition system based on contrastive learning according to claim 6, characterized in that: The data acquisition module acquires EEG signals and EOG signals of the subject in the audiovisual stimulation mode, and extracts useful data from all the acquired EEG and EOG data from the event occurrence calibration point to the event end calibration point. The specific steps include: Under video stimulation, the BP neuroscan system was used to record the EEG from a 28-channel electrode cap at a sampling rate of 250 Hz, and the subject's EOG was recorded from a 4-channel electrode cap at a sampling rate of 250 Hz; valid 28-lead EEG signals, 4-lead EOG signals, and actual emotional state labels were obtained: Among them, q represents the number of emotional state labels, and the value of each label is: Where k represents the kth label, k = 1, 2, ..., q; the emotional label uses two emotional dimensions, Valence and Arousal, for classification. The Valence dimension is expressed as positive, neutral and negative, and the Arousal dimension is expressed as 1, 2, 3, ..., 7, that is, -7 to -3 represents negative emotions, -2 to 2 represents neutral emotions, and 3 to 7 represents positive emotions.

8. The EEG-EOG multimodal emotion recognition system based on contrastive learning according to claim 6, characterized in that: The data preprocessing module preprocesses the EEG and EOG signals intercepted by the data acquisition module, including filtering, removing the mean, and removing artifacts.

9. The EEG-EOG multimodal emotion recognition system based on contrastive learning according to claim 6, characterized in that: The contrastive learning module processes the preprocessed EEG and EOG signals to generate a small batch of data containing several pairs of EEG and EOG signal segments, uses a base encoder and projection module to align the small batches of EEG and EOG signal segments of the same subject, and continuously optimizes the parameters of the base encoder and projection module. The specific steps include: First, the preprocessed EEG and EOG signals are processed using a data sampler to generate a small batch of data containing several pairs of EEG and EOG signal segments; Then, the segments were processed using a basic encoder to convert the EEG and EOG signals of the same subject into a modality-consistent representation of the same subject across multiple modalities; Then, a nonlinear projection module based on multi-layer perceptron is used to map the EEG and EOG representations aligned with the base encoder into the latent space to calculate the similarity; Finally, the similarity between the two EEG and EOG signal representations is maximized by minimizing the contrast loss function, and the parameters of the base encoder and projection module are optimized to obtain the trained base encoder and projection module for subsequent EEG-EOG signal alignment and feature extraction.

Images