Continuous dynamic emotion recognition method and system fusing electroencephalogram and electro-oculogram signals

Abstract

The invention discloses a continuous dynamic emotion recognition method and system fusing electroencephalogram and electro-oculogram signals, and relates to the field of emotion recognition, and the method comprises the steps: obtaining electroencephalogram signal data, electro-oculogram signal data, a big five personality score and a dynamic emotion label of a target subject; respectively preprocessing the electroencephalogram signal data and the electro-oculogram signal data; according to the preprocessed electroencephalogram signal data, differential entropy features are extracted; the target subjects are divided into pilots and common subjects according to the differential entropy features and the large five personality scores; according to the preprocessed electroencephalogram signal data and the preprocessed electro-oculogram signal data of the pilot and the common subject, a continuous dynamic emotion recognition model is adopted for emotion recognition; the continuous dynamic emotion recognition model comprises a convolution module, a bidirectional cross attention mechanism fusion network, a Transform module and a full connection layer; according to the invention, the precision and real-time performance of continuous dynamic emotion recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of emotion recognition, and in particular to a continuous dynamic emotion recognition method and system that integrates electroencephalogram and electrooculogram signals. Background Art

[0002] Affective computing is an important branch of artificial intelligence, dedicated to improving robots' ability in emotional expression and perception, and has played an important role in many fields such as human-computer interaction, mental health, and educational science. Emotional expression involves the synchronous changes of multiple physiological signals. Single signal analysis often has data limitations, which affects the accuracy of emotion recognition. Existing emotion recognition research is usually limited to static laboratory situations and fails to fully address the problem of changes in emotional states in dynamic situations.

[0003] Therefore, based on the above problems, there is an urgent need to provide a continuous dynamic emotion recognition method that integrates EEG and EOG signals to improve the accuracy and real-time performance of continuous dynamic emotion recognition. Summary of the invention

[0004] The purpose of this application is to provide a continuous dynamic emotion recognition method and system that integrates EEG and EOG signals, which can improve the accuracy and real-time performance of continuous dynamic emotion recognition.

[0005] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides a continuous dynamic emotion recognition method integrating EEG and EOG signals, the continuous dynamic emotion recognition method integrating EEG and EOG signals comprising: Obtaining the target subject's EEG signal data, EOG signal data, Big Five personality scores and dynamic emotion labels; the Big Five personality scores are obtained through the Big Five Personality Scale; the dynamic emotion labels are scored by the target subject on the emotional state from 1 to 9 points every 5 seconds based on the two dimensions of pleasure and arousal; the scoring types are calm to happy, happy to calm, sad to calm, calm to sad, nervous to calm and calm to nervous; the target subjects include pilots and ordinary subjects; the EOG signal data include horizontal EOG signals and vertical EOG signals; Preprocessing the EEG signal data and the EOG signal data respectively to obtain preprocessed EEG signal data and preprocessed EOG signal data; Extract differential entropy features based on the preprocessed EEG signal data; and divide the target subjects into pilots and ordinary subjects based on the differential entropy features and the Big Five personality scores; According to the preprocessed EEG signal data and preprocessed EO signal data of the pilots and the preprocessed EEG signal data and preprocessed EO signal data of ordinary subjects, a continuous dynamic emotion recognition model is used for emotion recognition; the continuous dynamic emotion recognition model includes: a convolution module, a bidirectional cross attention mechanism fusion network, a Transformer module and a fully connected layer; the convolution module is used to extract EEG signal features and EO signal features from the preprocessed EEG signal data and the preprocessed EO signal data using spatiotemporal convolution technology and SE attention mechanism network module; the bidirectional cross attention mechanism fusion network is used to fuse the EEG signal features and the EO signal features to obtain multimodal features; the Transformer module is used to extract the time series features of the multimodal features during the emotion change process; the fully connected layer is used to perform continuous dynamic emotion recognition based on the time series features.

[0006] Optionally, the preprocessing of the EEG signal data and the EOG signal data to obtain the preprocessed EEG signal data and the preprocessed EOG signal data specifically includes: The EEG signal data is subjected to filtering processing, artifact removal processing and resampling processing in sequence to obtain preprocessed EEG signal data; The electrooculogram signal data is filtered and resampled in sequence to obtain preprocessed electrooculogram signal data.

[0007] Optionally, extracting differential entropy features according to the preprocessed EEG signal data specifically includes: Using the formula Determine differential entropy characteristics; in, is the differential entropy feature of the EEG signal data, is the total number of samples, is the value of the EEG signal data at the nth sampling point, is the value of the EEG signal data at the n+1th sampling point.

[0008] Optionally, the frequency bands of the differential entropy feature include: 1 Hz~3 Hz frequency band, 4 Hz~7 Hz frequency band, 8 Hz~13 Hz frequency band, 14 Hz~30 Hz frequency band and 31 Hz~50 Hz frequency band.

[0009] Optionally, the method further comprises dividing the target subjects into pilots and ordinary subjects according to the differential entropy features and the Big Five personality scores: The differential entropy traits and Big Five personality scores were standardized.

[0010] Optionally, the convolution module includes: a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer, a batch normalization layer between the first convolution layer and the second convolution layer, a batch normalization layer between the second convolution layer and the third convolution layer, a dropout layer after the third convolution layer, and a SE attention mechanism network module between the third convolution layer and the fourth convolution layer; The first convolution layer uses 1D convolution to remove noise and extract features; The activation function is excluded from the first convolution layer; the second convolution layer includes a spatial convolution layer; the third convolution layer includes an average pooling layer; the SE attention mechanism network module learns the weights between channels through adaptive learning; and the fourth convolution layer adopts an advanced feature fusion layer.

[0011] In a second aspect, the present application provides a continuous dynamic emotion recognition system integrating EEG and EOG signals, the continuous dynamic emotion recognition system integrating EEG and EOG signals comprising: A data acquisition module is used to acquire EEG signal data, electrooculogram signal data, Big Five personality scores and dynamic emotion labels of target subjects; the Big Five personality scores are acquired through the Big Five personality scale; the dynamic emotion labels are scored by the target subjects on the emotional state from 1 to 9 points every 5 seconds according to the two dimensions of pleasure and arousal; the scoring types are calm to happy, happy to calm, sad to calm, calm to sad, nervous to calm and calm to nervous; the target subjects include pilots and ordinary subjects; the electrooculogram signal data include horizontal electrooculogram signals and vertical electrooculogram signals; A preprocessing module, used to preprocess the EEG signal data and the EOG signal data respectively to obtain preprocessed EEG signal data and preprocessed EOG signal data; The subject classification module is used to extract differential entropy features based on the preprocessed EEG signal data; and classify the target subjects into pilots and ordinary subjects based on the differential entropy features and the Big Five personality scores; The emotion recognition module is used to perform emotion recognition using a continuous dynamic emotion recognition model based on the preprocessed EEG signal data and preprocessed EO signal data of the pilot and the preprocessed EEG signal data and preprocessed EO signal data of the general subjects; the continuous dynamic emotion recognition model includes: a convolution module, a bidirectional cross attention mechanism fusion network, a Transformer module and a fully connected layer; the convolution module is used to extract EEG signal features and EO signal features from the preprocessed EEG signal data and the preprocessed EO signal data using the spatiotemporal convolution technology and the SE attention mechanism network module; the bidirectional cross attention mechanism fusion network is used to perform feature fusion on the EEG signal features and the EO signal features to obtain multimodal features; the Transformer module is used to extract the time series features of the multimodal features during the emotion change process; the fully connected layer is used to perform continuous dynamic emotion recognition based on the time series features.

[0012] According to the specific embodiments provided in this application, this application has the following technical effects: The present application provides a continuous dynamic emotion recognition method and system integrating EEG and EOG signals. The method collects EEG signal data (Electroencephalogram, EEG), electrooculogram signal data (Electrooculogram, EOG), Big Five personality scores and dynamic emotion labels of target subjects, pre-processes the EEG signal data and the EOG signal data, and then uses a hierarchical model architecture for recognition. In the first layer of the hierarchical model architecture, differential entropy features (Differential Entropy, DE) are extracted from the pre-processed EEG signal data, and the target subjects are divided into pilots and ordinary subjects by combining the differential entropy features and the Big Five personality scores. Then, continuous dynamic emotion recognition is performed on the two types of subjects respectively in the second layer of the hierarchical model architecture. Specifically, a convolution module is used to extract features from the preprocessed EEG signal data and the preprocessed EOG signal data, a bidirectional cross-attention mechanism fusion network (BCAF-Net) is used to fuse the EEG features and the EOG features, a Transformer module is used to learn global temporal features from multimodal features, and finally a fully connected layer is used to determine the continuous dynamic emotion category of the target subject, thereby realizing the recognition of continuous dynamic emotions. This application improves the accuracy and real-time performance of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 This is a flow chart of a continuous dynamic emotion recognition method integrating EEG and EOG signals in one embodiment of the present application; Figure 2 A detailed flow chart of the continuous dynamic emotion recognition method of fusing EEG and EOG signals provided in this application; Figure 3 Schematic diagram of the process of obtaining EEG signal data, electrooculogram signal data, Big Five personality scores and dynamic emotion labels for this application; Figure 4 This is a schematic diagram of the process of continuous dynamic emotion recognition that integrates EEG and EOG signals in this application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0017] In an exemplary embodiment, Figure 1 and Figure 2 As shown, a continuous dynamic emotion recognition method integrating EEG and EOG signals is provided, and the method includes the following S101 to S104. Among them: S101, obtaining EEG signal data, electrooculogram signal data, Big Five personality scores and dynamic emotion labels of the target subject; the Big Five personality scores are obtained through the Big Five Inventory (BFI); the dynamic emotion labels are scored by the target subject on the emotional state from 1 to 9 points every 5 seconds based on the two dimensions of pleasure and arousal; the scoring types include calm to happy (characterizing the dynamic emotional change from calm to happy), happy to calm (characterizing the dynamic emotional change from happy to calm), sad to calm (characterizing the dynamic emotional change from sad to calm), calm to sad (characterizing the dynamic emotional change from calm to sad), nervous to calm (characterizing the dynamic emotional change from nervous to calm) and calm to nervous (characterizing the dynamic emotional change from calm to nervous); the target subjects include pilots and ordinary subjects; the electrooculogram signal data include horizontal electrooculogram signals and vertical electrooculogram signals; As important representatives of human physiological signals, EEG signals and EOG signals are unbiased and objective. EEG signals reflect the changes in electrical signals of brain activity, while EOG signals record the trajectory and frequency of eye movements. Both signals can be used to reflect a person's emotional state.

[0018] The Big Five Personality Inventory is known as the Five-Factor Model of Personality and is a commonly used personality assessment tool in psychology. It assesses individual personality traits through five dimensions (neuroticism, extraversion, openness, agreeableness, and conscientiousness). Each dimension has a corresponding score range, which can help us understand the individual's personality characteristics in different aspects.

[0019] The original EEG signal data and original EOG signal data of the target subject are collected based on the set experimental paradigm. In this embodiment, the original EEG signal data and the original EOG signal data are collected synchronously in real time and stored in a database. The original EEG signal data include: EEG signal data in normal mode and EEG signal data in target task state mode. The original EOG signal data include: EOG signal data in normal mode and EOG signal data in target task state mode.

[0020] Among them, in the training process of the continuous dynamic emotion recognition model, the experimental paradigm is: 48 movie clips are selected from the material library as stimuli used in the experiment. Each movie clip is carefully edited to produce coherent emotions. Each video should cause two expected target emotions, including six types of stimulus movie clips such as happy calm and calm happy, sad calm and calm sad, and nervous calm and calm nervous. There are eight videos of each type, and the duration is about 3 minutes. Figure 3As shown in the figure, we first create an experimental paradigm for EEG and EOG signals, and synchronously acquire the original EEG data, original EOG data, and dynamic emotion labels of the target subjects through acquisition equipment (EEG cap and surface electrodes). In actual application, the experimental paradigm is similar to the training process of the deep learning model, the only difference is that dynamic emotion labels are not collected.

[0021] Experts were organized to analyze application scenarios and possible situations, and to design a relatively complete experimental paradigm that can accurately reflect EEG and EOG signals in this cognitive state. In isolation from external interference, the discharge signals of the cerebral cortex of the subjects under different cognitive activities were recorded and collected. At the same time, for comparison, the EEG and EOG signals under normal conditions were recorded.

[0022] The quality of the collected EEG signal data and EOG signal data is evaluated based on the performance of the subjects at the scene of the collection. The raw EEG signal data and raw EOG signal data with poor performance and that cannot accurately reflect the process are eliminated, and the data in the database are updated with feedback.

[0023] like Figure 3 As shown, as a specific implementation method, the process of acquiring the EEG signal data and the EOG signal data is as follows.

[0024] 1) In a quiet room with appropriate temperature and brightness, the subjects were asked to wear EEG data and EOG data acquisition equipment, and 5 seconds of prompts and precautions were played to the subjects. Two experiments were conducted in total, each experiment was divided into 3 sections, and each section played a video that induced two emotions for the subjects. There were three types of videos that induced different types of emotional changes, namely happy to calm and calm to happy, calm to sad and sad to calm, calm to nervous and nervous to calm. Each video lasted about 180 seconds, of which each emotion lasted 90 seconds. The two experiments were separated by more than one week to ensure the independence of each emotional experiment and improve the accuracy of the experiment.

[0025] 2) The subjects were tested in a dedicated environment that could block external interference. The brightness was always set at a comfortable level so that the subjects could clearly see the movie content. In order to ensure the smooth progress of the experiment, each subject was required to fill out a mental health assessment questionnaire before the experiment officially began, and each subject was trained using additional movie clips. In addition, the subjects were also required to complete a personality measurement questionnaire to obtain the personality data of the target subject.

[0026] The purpose of the personality measurement questionnaire is to obtain the data of the five personality traits of the target subject, and use them as the personality attributes of the target subject to obtain the classification accuracy of the classification task when performing the subject classification task. The personality measurement questionnaire uses the Chinese version of the Big Five Personality Inventory, which includes a total of 44 questions.

[0027] 3) After the experiment started, the order of presentation of each movie clip was fixed. The subjects first adjusted their emotions to be in a relatively relaxed state. When the subjects' brain waves were stable, the movie clips were played. EEG and EOG signals were collected for 2 seconds in the non-task state, and 180 seconds in the cognitive activity state.

[0028] 4) Set a rest time, rest for 15 to 30 seconds after watching each movie clip, so that the subject's brain waves are in a stable state. Then repeat the above process until the subject completes all the processes of the entire experimental paradigm. After an interval of 10 seconds, the subject evaluates his or her self-state and completes the experiment.

[0029] Five experts were selected to select film clips. All of them had received professional film training and had experience in rehearsing films. Thirty physically and mentally healthy subjects were selected to participate in the experiment, including 10 pilots and 20 ordinary subjects, with an average age of (22±1.5) years old. All participants had normal hearing and vision and were right-handed. They were aware of and agreed to the experimental content.

[0030] S102, preprocessing the EEG signal data and the EOG signal data respectively to obtain preprocessed EEG signal data and preprocessed EOG signal data; S102 specifically includes: S21, performing filtering processing, artifact removal processing and resampling processing on the EEG signal data in sequence to obtain preprocessed EEG signal data; The EEG signal of the required frequency band is filtered out from the EEG signal data through a bandpass filter to obtain a filtered EEG signal. Irrelevant information outside the effective EEG frequency band (1Hz~50Hz) is removed.

[0031] Performing artifact removal processing on the filtered EEG signal data to obtain EEG signal processing data. Removing power frequency interference and artifacts of the filtered EEG signal, that is, removing disturbances such as eyelid twitching, electrocardiogram, electromyography, limb shaking, and 50 Hz power frequency interference that interfere with the EEG signal data.

[0032] The EEG signal processing data is resampled to obtain the EEG signal data of the target subject. The EEG signal processing data is downsampled, and the sampling frequency is set to 128 Hz. This can improve the computing efficiency while retaining information related to the cognitive state.

[0033] Specifically, the EEG signal data was imported into ASA, and the electrode positioning, electrode selection, re-referencing, filtering, bad segment removal, bad lead interpolation, independent component analysis, segmentation and baseline correction were performed in sequence, and then the data was saved.

[0034] S22, filter and resample the electrooculogram signal data in sequence to obtain preprocessed electrooculogram signal data. That is, the 50Hz power frequency interference and artifacts are removed from the EEG signal data, and the δ (1-3Hz) frequency band, (4-7Hz) frequency band, (8-13Hz) frequency band, (14-30Hz) frequency band and For frequency bands other than the (31-50Hz) band, the EOG signal data are also subjected to the 50Hz power frequency interference and filtering to remove the frequency bands other than the (0.3-30Hz) band. The EEG signal data and EOG signal data are then segmented and classified under different conditions. Finally, the preprocessed EEG signal data and EOG signal data are windowed, and a non-overlapping sliding time window of 1s is used to obtain the preprocessed EEG signal data and EOG signal data.

[0035] The bandpass filter is used to filter out the EO signal data in the required frequency band (0.3Hz~30Hz) to obtain the EO signal processing data. The irrelevant information outside the effective frequency band of the EO signal is removed. The notch filter is used to remove the power frequency interference (such as 50Hz or 60Hz power line interference) in the EO signal data.

[0036] The electrooculogram signal processing data is resampled to obtain the electrooculogram signal data of the target subject. The electrooculogram signal processing data is downsampled, and the sampling frequency is set to 128 Hz. The computational efficiency can be improved while retaining information related to the cognitive state.

[0037] Specifically, the electrooculogram signal data is imported into ASA, and filtering, notching, and resampling are performed in sequence, and then the data is saved.

[0038] After collecting the EEG signal data and EOG signal data, the subjects are required to score the stimulus video every five seconds in the valencearousal two-dimensional space, with a value range of [1,9], and normalize the dynamic emotion labels.

[0039] S103, extracting differential entropy features based on the preprocessed EEG signal data; and dividing the target subjects into pilots and ordinary subjects based on the differential entropy features and the Big Five personality scores; Calculate the differential entropy features of EEG signal data in the time domain: ; in, is the differential entropy feature of the EEG signal data, is the total number of samples, is the value of the EEG signal data at the nth sampling point, is the value of the EEG signal data at the n+1th sampling point.

[0040] This application adds a 1 second non-overlapping time window, and the 64-channel information for every 1 second Extract features.

[0041] The frequency bands of the differential entropy feature include: 1Hz~3Hz frequency band, 4Hz~7Hz frequency band, 8Hz~13Hz frequency band, 14Hz~30Hz frequency band and 31Hz~50Hz frequency band.

[0042] The differential entropy feature is combined with the Big Five personality score feature of the target subject and input into the multi-layer convolution to perform the pilot and ordinary subject binary classification task. Since each subject contains multiple samples, these samples may get different classification results, that is, some samples are classified as pilots, and others are classified as ordinary subjects. In order to solve this problem, the majority voting method is used to finally determine the category of each subject based on the classification results of all samples of each subject.

[0043] S104, using a continuous dynamic emotion recognition model to perform emotion recognition based on the preprocessed EEG signal data and preprocessed EO signal data of the pilot and the preprocessed EEG signal data and preprocessed EO signal data of the general subject; Figure 4 As shown, the continuous dynamic emotion recognition model includes: a convolution module, a bidirectional cross-attention mechanism fusion network, a Transformer module and a fully connected layer; the convolution module is used to extract EEG signal features and EOG signal features from the preprocessed EEG signal data and the preprocessed EOG signal data using spatiotemporal convolution technology and SE attention mechanism network module (Squeeze-and-Excitation Networks, SE); the bidirectional cross-attention mechanism fusion network is used to fuse the EEG signal features and the EOG signal features to obtain multimodal features; the Transformer module is used to extract the time series features of the multimodal features during the emotion change process; the fully connected layer is used to perform continuous dynamic emotion recognition based on the time series features.

[0044] Specifically, the convolution module is connected in series with the Transformer module; the bidirectional cross-attention mechanism fusion network performs feature fusion on the features extracted from the EEG signal and the EOG signal after passing through the convolution module and outputs the features as the input of the Transformer module; and the fully connected layer is connected to the Transformer module.

[0045] The continuous dynamic emotion recognition model is obtained by pre-training with a training sample set, wherein each training sample in the training sample set includes an EEG sample signal, an EOG sample signal and a corresponding dynamic emotion label.

[0046] During the training process, Bayesian optimization, cross-validation and other methods are used to tune the parameters of the deep learning model. The training set is used to train the model, and the validation set is used to evaluate the model performance. The optimal model parameters are selected based on the evaluation results. The model is then deployed and used in practical applications.

[0047] The continuous dynamic emotion recognition model uses a deep learning model to extract spatial and temporal features from the preprocessed EEG signals and EOG signals to improve the accuracy of the dynamic emotion label of the target subject; The convolution module includes: a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer, a batch normalization layer between the first convolution layer and the second convolution layer, a batch normalization layer between the second convolution layer and the third convolution layer, a dropout layer after the third convolution layer, and a SE attention mechanism network module between the third convolution layer and the fourth convolution layer; The first convolution layer uses 1D convolution to remove noise and extract features; Among them, a batch normalization layer is inserted between the first and second convolution layers to solve the problem of gradient disappearance. Then, in order to maintain the interpretability of spatiotemporal convolution, an activation function is excluded from the first convolution layer. The second convolution layer consists of a spatial convolution layer, which merges the valid information on all channels. The spatial convolution is separated in the depth dimension and implemented by the deep convolution in the code. While the activation function is restored, a batch normalization layer is added between the second and third convolution layers. The third convolution layer includes an average pooling layer with a pool length of 4. To avoid overfitting, a dropout layer is added after the third convolution layer. An SE attention mechanism network module is added between the third and fourth convolution layers. The SE attention mechanism network module automatically adjusts the response of each channel by adaptively learning the weights between channels, so that the network can pay more attention to those features that are helpful for emotion recognition tasks and improve the recognition ability of the model. After the SE attention mechanism network module, the fourth convolution layer adopts an advanced feature fusion layer. The fourth convolution layer is implemented using a separable convolution layer to minimize the number of parameters. Finally, the output of the convolutional module will be reduced in dimension to fit the following modules.

[0048] The SE attention mechanism network module is an SE attention mechanism network, which adds an attention mechanism in the channel dimension. The key operations are Squeeze and Excitation.

[0049] Specifically, the Squeeze method compresses the two-dimensional features of each channel into one real number; the Excitation method generates a weight value for each feature channel and then adds the normalized weight to the features of each channel of the original input.

[0050] This application adds a 1 second non-overlapping window and converts the 64-channel information of the EEG signal data into 1 second 2-channel information of electrooculogram data every 1 second Input to the convolution module to extract features.

[0051] The EEG signal data and the EOG signal data are standardized to ensure the scale consistency between different data, so as to avoid the influence of too large or too small data on the training of deep learning models. The standardization process is to use the Z-score standardization method to process DE features and Big Five personality scores.

[0052] The bidirectional cross-attention mechanism fusion network establishes an attention mechanism between different modalities, enabling the model to better focus on important information parts. The representation of the electrooculogram features is adjusted according to the information of the EEG features, and the representation of the EEG features is adjusted according to the information of the electrooculogram features, ensuring that the information interaction between the two modalities is bidirectional.

[0053] In the bidirectional cross-attention mechanism fusion network of this application, the EEG signal data and the eye signal data are firstly queried (Q), key (K) and value (V) by the multi-head cross-attention mechanism (MHCA) to obtain the EEG features. and electrooculographic characteristics , then and Perform dot product operation to get features Finally, global fusion is performed to obtain the three features , respectively apply the fully connected layer FC(·), Map to 2D space and concatenate to get the final fused feature vector This application can help the model better capture the interaction between EEG signal data and EOG signal data, and enhance the accuracy of continuous dynamic emotion recognition. The details are as follows: (1) EEG features are transformed into and electrooculographic characteristics Perform modality-level fusion. Specifically, firstly, the EEG features are used as Q, and the electrooculogram features are used as K and V, which are input into MHCA for fusion to obtain Then, the electrooculogram feature is used as Q, and the electrooculogram feature is used as K and V to input into MHCA, and we get ; For the query (Q), key (K), and value (V) in MHCA, they are first mapped by their respective linear transformations: ; ; ; in, , , is the weight matrix, is the input of the query, It is a key and value input (it is used as both key and value input).

[0054] In multi-head attention, the input is split into H heads (H is the number of heads), and each head processes a part of the features. For each head, the attention weight matrix between them is calculated : ; in, and are the vector representations of the i-th query and i-th key respectively. The above scores will be used to calculate the attention weights.

[0055] To prevent the dot product result from being too large and causing gradient instability, a scaling factor is used Scale the dot product result: ; Use the Softmax function to scale the dot product attention score Converted to attention weights: ; in, is the attention weight of the ith query on the ith key.

[0056] Finally, use the attention weights to Perform weighted summation to get the output of each head: ; in, is the output vector of the ith head.

[0057] After getting the output of each head, they are usually concatenated (in the feature dimension) and processed through an additional linear transformation (fully connected layer) to get the final output: ; in, is the weight matrix of the output transformation, It is the final output of the bidirectional cross-attention fusion network.

[0058] (2) Then and Perform a dot product operation to get ; ; (3) Finally, global fusion is performed to combine the three features obtained previously , and then apply them to the fully connected layer FC1(·), fully connected layer FC2(·), and fully connected layer FC3(·) respectively to obtain the features Z1, Z2, and Z3 mapped to the 2D space; connect the features Z1, Z2, and Z3 mapped to the 2D space to obtain the final multimodal feature .

[0059] ; .

[0060] in, (·) is the activation function, , and There are three characteristics The corresponding weight matrix is, , and There are three characteristics The corresponding bias term.

[0061] The Transformer module is used to further extract global temporal features from multimodal features, and finally the recognition results of pre-set dynamic emotions are obtained through the fully connected layer. The Transformer module includes position encoding, multi-head self-attention mechanism, feedforward neural network, and layer normalization. Position encoding is used to retain the position information of the input sequence. The multi-head self-attention mechanism captures different subspaces of the input features through multiple attention heads. The feedforward neural network performs feature transformation through two fully connected layers. Layer normalization is used to stabilize the training process and avoid gradient disappearance and explosion.

[0062] The present application can preprocess online multimodal data to obtain online multimodal information, perform fusion classification processing based on the online multimodal information, and determine the online psychological state and online analysis results of the online multimodal data.

[0063] Based on the same inventive concept, the embodiment of the present application also provides a continuous dynamic emotion recognition system that integrates EEG and EOG signals for implementing the above-mentioned continuous dynamic emotion recognition method that integrates EEG and EOG signals. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more embodiments of the continuous dynamic emotion recognition system that integrates EEG and EOG signals provided below can be found in the above-mentioned limitations on the continuous dynamic emotion recognition method that integrates EEG and EOG signals, and will not be repeated here.

[0064] In an exemplary embodiment, a continuous dynamic emotion recognition system integrating electroencephalogram and electrooculogram signals is provided, comprising: A data acquisition module is used to acquire EEG signal data, electrooculogram signal data, Big Five personality scores and dynamic emotion labels of target subjects; the Big Five personality scores are acquired through the Big Five personality scale; the dynamic emotion labels are scored by the target subjects on the emotional state from 1 to 9 points every 5 seconds according to the two dimensions of pleasure and arousal; the scoring types are calm to happy, happy to calm, sad to calm, calm to sad, nervous to calm and calm to nervous; the target subjects include pilots and ordinary subjects; the electrooculogram signal data include horizontal electrooculogram signals and vertical electrooculogram signals; A preprocessing module, used to preprocess the EEG signal data and the EOG signal data respectively to obtain preprocessed EEG signal data and preprocessed EOG signal data; The subject classification module is used to extract differential entropy features based on the preprocessed EEG signal data; and classify the target subjects into pilots and ordinary subjects based on the differential entropy features and the Big Five personality scores; The emotion recognition module is used to perform emotion recognition using a continuous dynamic emotion recognition model based on the preprocessed EEG signal data and preprocessed EO signal data of the pilot and the preprocessed EEG signal data and preprocessed EO signal data of the general subjects; the continuous dynamic emotion recognition model includes: a convolution module, a bidirectional cross attention mechanism fusion network, a Transformer module and a fully connected layer; the convolution module is used to extract EEG signal features and EO signal features from the preprocessed EEG signal data and the preprocessed EO signal data using the spatiotemporal convolution technology and the SE attention mechanism network module; the bidirectional cross attention mechanism fusion network is used to perform feature fusion on the EEG signal features and the EO signal features to obtain multimodal features; the Transformer module is used to extract the time series features of the multimodal features during the emotion change process; the fully connected layer is used to perform continuous dynamic emotion recognition based on the time series features.

[0065] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a continuous dynamic emotion recognition method that integrates electroencephalogram and electrooculogram signals is implemented.

[0066] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0067] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0068] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0069] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0070] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0071] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A continuous dynamic emotion recognition method integrating EEG and EOG signals, characterized in that: The continuous dynamic emotion recognition method integrating EEG and EOG signals comprises: Obtaining the target subject's EEG signal data, EOG signal data, Big Five personality scores and dynamic emotion labels; the Big Five personality scores are obtained through the Big Five Personality Scale; the dynamic emotion labels are scored by the target subject on the emotional state from 1 to 9 points every 5 seconds based on the two dimensions of pleasure and arousal; the scoring types are calm to happy, happy to calm, sad to calm, calm to sad, nervous to calm and calm to nervous; the target subjects include pilots and ordinary subjects; the EOG signal data include horizontal EOG signals and vertical EOG signals; Preprocessing the EEG signal data and the EOG signal data respectively to obtain preprocessed EEG signal data and preprocessed EOG signal data; Extract differential entropy features based on the preprocessed EEG signal data; and divide the target subjects into pilots and ordinary subjects based on the differential entropy features and the Big Five personality scores; According to the preprocessed EEG signal data and preprocessed EO signal data of the pilots and the preprocessed EEG signal data and preprocessed EO signal data of ordinary subjects, a continuous dynamic emotion recognition model is used for emotion recognition; the continuous dynamic emotion recognition model includes: a convolution module, a bidirectional cross attention mechanism fusion network, a Transformer module and a fully connected layer; the convolution module is used to extract EEG signal features and EO signal features from the preprocessed EEG signal data and the preprocessed EO signal data using spatiotemporal convolution technology and SE attention mechanism network module; the bidirectional cross attention mechanism fusion network is used to fuse the EEG signal features and the EO signal features to obtain multimodal features; the Transformer module is used to extract the time series features of the multimodal features during the emotion change process; the fully connected layer is used to perform continuous dynamic emotion recognition based on the time series features.

2. The continuous dynamic emotion recognition method integrating EEG and EOG signals according to claim 1 is characterized in that: The preprocessing of the EEG signal data and the EOG signal data respectively to obtain the preprocessed EEG signal data and the preprocessed EOG signal data specifically includes: The EEG signal data is subjected to filtering processing, artifact removal processing and resampling processing in sequence to obtain preprocessed EEG signal data; The electrooculogram signal data is filtered and resampled in sequence to obtain preprocessed electrooculogram signal data.

3. The continuous dynamic emotion recognition method integrating EEG and EOG signals according to claim 1, characterized in that: The step of extracting differential entropy features based on the preprocessed EEG signal data specifically includes: Using the formula Determine differential entropy characteristics; in, is the differential entropy feature of the EEG signal data, is the total number of samples, is the value of the EEG signal data at the nth sampling point, is the value of the EEG signal data at the n+1th sampling point.

4. The continuous dynamic emotion recognition method integrating EEG and EOG signals according to claim 1 or claim 3, characterized in that: The frequency bands of the differential entropy feature include: 1Hz~3Hz frequency band, 4Hz~7Hz frequency band, 8Hz~13Hz frequency band, 14Hz~30Hz frequency band and 31Hz~50Hz frequency band.

5. The continuous dynamic emotion recognition method integrating EEG and EOG signals according to claim 1, characterized in that: The target subjects were divided into pilots and ordinary subjects according to the differential entropy characteristics and the Big Five personality scores, and the previous ones also included: The differential entropy traits and Big Five personality scores were standardized.

6. The continuous dynamic emotion recognition method integrating EEG and EOG signals according to claim 1, characterized in that: The convolution module includes: a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer, a batch normalization layer between the first convolution layer and the second convolution layer, a batch normalization layer between the second convolution layer and the third convolution layer, a dropout layer after the third convolution layer, and a SE attention mechanism network module between the third convolution layer and the fourth convolution layer; The first convolution layer uses 1D convolution to remove noise and extract features; The activation function is excluded from the first convolution layer; the second convolution layer includes a spatial convolution layer; the third convolution layer includes an average pooling layer; the SE attention mechanism network module learns the weights between channels through adaptive learning; and the fourth convolution layer adopts an advanced feature fusion layer.

7. A continuous dynamic emotion recognition system integrating EEG and EOG signals, characterized in that: The continuous dynamic emotion recognition system integrating EEG and EOG signals comprises: A data acquisition module is used to acquire EEG signal data, electrooculogram signal data, Big Five personality scores and dynamic emotion labels of target subjects; the Big Five personality scores are acquired through the Big Five personality scale; the dynamic emotion labels are scored by the target subjects on the emotional state from 1 to 9 points every 5 seconds according to the two dimensions of pleasure and arousal; the scoring types are calm to happy, happy to calm, sad to calm, calm to sad, nervous to calm and calm to nervous; the target subjects include pilots and ordinary subjects; the electrooculogram signal data include horizontal electrooculogram signals and vertical electrooculogram signals; A preprocessing module, used to preprocess the EEG signal data and the EOG signal data respectively to obtain preprocessed EEG signal data and preprocessed EOG signal data; The subject classification module is used to extract differential entropy features based on the preprocessed EEG signal data; and classify the target subjects into pilots and ordinary subjects based on the differential entropy features and the Big Five personality scores; The emotion recognition module is used to perform emotion recognition using a continuous dynamic emotion recognition model based on the preprocessed EEG signal data and preprocessed EO signal data of the pilot and the preprocessed EEG signal data and preprocessed EO signal data of the general subjects; the continuous dynamic emotion recognition model includes: a convolution module, a bidirectional cross attention mechanism fusion network, a Transformer module and a fully connected layer; the convolution module is used to extract EEG signal features and EO signal features from the preprocessed EEG signal data and the preprocessed EO signal data using the spatiotemporal convolution technology and the SE attention mechanism network module; the bidirectional cross attention mechanism fusion network is used to perform feature fusion on the EEG signal features and the EO signal features to obtain multimodal features; the Transformer module is used to extract the time series features of the multimodal features during the emotion change process; the fully connected layer is used to perform continuous dynamic emotion recognition based on the time series features.

Images