Electroencephalogram dynamic continuous emotion recognition method, system and equipment based on channel selection

Abstract

The invention discloses an electroencephalogram dynamic continuous emotion recognition method, system and device based on channel selection, and relates to the field of electroencephalogram emotion recognition, and the method comprises the steps: collecting a multi-channel electroencephalogram time sequence signal of a subject in a dynamic situation; an optimal channel combination is determined by adopting EEG-Net and a non-dominated sorting genetic algorithm; according to the electroencephalogram time sequence signal corresponding to the optimal channel combination, determining a continuous dynamic emotion category by adopting an emotion recognition model; wherein the emotion recognition model comprises a short-term flow feature extraction network, a long-term flow feature extraction network, an attention mechanism layer and a classifier; the short-term flow feature extraction network extracts local features; the long-term flow feature extraction network extracts global features; the attention mechanism layer fuses the local features and the global features; and the classifier determines a continuous dynamic emotion category according to the fusion features. According to the method, the long-term time dependence and global features of the electroencephalogram time sequence signals can be captured, so that the precision of dynamic continuous emotion recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of EEG emotion recognition, and in particular to a method, system and device for dynamic and continuous EEG emotion recognition based on channel selection. Background Art

[0002] Human emotions are internal manifestations of personal behavior, thinking patterns, and feelings about objective things. In real life, human emotions are dynamic and diverse, and are affected by the external environment, social interactions, and the individual's internal state. Emotions are characterized by continuity, have a profound impact on the individual's subsequent behavior and psychological motivation, and play a vital role in the interaction between subjective feelings and the external environment. Under normal conditions, people usually maintain a stable mood, which is one of the most common states. However, when subjected to external positive stimulation, this stable mood may be enhanced, manifesting as positive emotions such as joy. On the contrary, when subjected to negative stimulation, emotions may turn negative, manifesting as negative emotions such as sadness and tension. Therefore, it is necessary to accurately monitor human dynamic and continuous emotions. Summary of the invention

[0003] The purpose of this application is to provide a method, system and device for dynamic and continuous EEG emotion recognition based on channel selection, which can capture the long-term time dependence and global characteristics of EEG timing signals, thereby improving the accuracy of dynamic and continuous emotion recognition.

[0004] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides a method for dynamic continuous emotion recognition based on EEG channel selection, comprising: Collect multi-channel EEG time series signals of subjects in dynamic situations; According to the multi-channel EEG time series signal, an optimal channel combination is determined using EEG-Net and a non-dominated sorting genetic algorithm; According to the EEG time series signals corresponding to the optimal channel combination, the pre-trained emotion recognition model is used to determine the continuous dynamic emotion categories of the subjects in dynamic situations; Among them, the emotion recognition model includes a short-term flow feature extraction network, a long-term flow feature extraction network, an attention mechanism layer and a classifier; the short-term flow feature extraction network is used to extract local features of EEG time series signals using a time series convolution module; the long-term flow feature extraction network is used to extract global features of EEG time series signals using a Transformer module; the attention mechanism layer is used to fuse local features and global features to obtain fused features; the classifier is used to determine the continuous dynamic emotion category based on the fused features.

[0005] In a second aspect, the present application provides an EEG dynamic continuous emotion recognition system based on channel selection, comprising: A signal acquisition unit, used to acquire multi-channel EEG time-series signals of the subject in a dynamic situation; A channel selection unit, used to determine the optimal channel combination according to the multi-channel EEG time series signal by using EEG-Net and non-dominated sorting genetic algorithm; The emotion recognition unit is used to determine the continuous dynamic emotion category of the subject in the dynamic scenario according to the EEG time series signal corresponding to the optimal channel combination and using the pre-trained emotion recognition model; Among them, the emotion recognition model includes a short-term flow feature extraction network, a long-term flow feature extraction network, an attention mechanism layer and a classifier; the short-term flow feature extraction network is used to extract local features of EEG time series signals using a time series convolution module; the long-term flow feature extraction network is used to extract global features of EEG time series signals using a Transformer module; the attention mechanism layer is used to fuse local features and global features to obtain fused features; the classifier is used to determine the continuous dynamic emotion category based on the fused features.

[0006] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned EEG dynamic continuous emotion recognition method based on channel selection.

[0007] According to the specific embodiments provided in this application, this application has the following technical effects: The present application provides a method, system and device for recognizing dynamic continuous EEG emotions based on channel selection. The method first uses EEG-Net and a non-dominated sorting genetic algorithm to determine the optimal channel combination. Then, based on the EEG time series signal corresponding to the optimal channel combination, a short-term stream feature extraction network is used to extract the local features of the EEG time series signal corresponding to the optimal channel combination. A long-term stream feature extraction network is used to extract the global features of the EEG time series signal corresponding to the optimal channel combination. The long-term time dependency and global features of the EEG time series signal are effectively captured. The attention mechanism layer is used to fuse the local features and the global features. Finally, the continuous dynamic emotion category is determined based on the fused features, thereby improving the recognition accuracy of dynamic continuous emotions. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0009] Figure 1 This is a diagram of the application environment of a method for EEG dynamic continuous emotion recognition based on channel selection in one embodiment of the present application.

[0010] Figure 2 A flowchart of a method for dynamic continuous emotion recognition based on EEG channel selection is provided in one embodiment of the present application.

[0011] Figure 3 A schematic diagram of a method for dynamic continuous emotion recognition based on EEG channel selection provided in one embodiment of the present application.

[0012] Figure 4 Schematic diagram of the acquisition process of EEG timing signals in one embodiment of the present application.

[0013] Figure 5 Schematic diagram of the structure of a dual-flow network in one embodiment of the present application.

[0014] Figure 6 A schematic diagram of the functional modules of an EEG dynamic continuous emotion recognition system based on channel selection provided in one embodiment of the present application.

[0015] Figure 7 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0018] The EEG dynamic continuous emotion recognition method based on channel selection provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, or it can be integrated on the server 104, or it can be placed on the cloud or other servers. The terminal 102 can send the multi-channel EEG timing signal of the subject in a dynamic situation to the server 104. After the server 104 receives the multi-channel EEG timing signal, it uses EEG-Net and non-dominated sorting genetic algorithm to determine the optimal channel combination according to the multi-channel EEG timing signal. According to the EEG timing signal corresponding to the optimal channel combination, the pre-trained emotion recognition model is used to determine the continuous dynamic emotion category of the subject in the dynamic situation. The server 104 can feedback the continuous dynamic emotion category of the subject in the dynamic situation to the terminal 102. In addition, in some embodiments, the EEG dynamic continuous emotion recognition method based on channel selection can also be implemented by the server 104 or the terminal 102 alone.

[0019] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0020] In an exemplary embodiment, Figure 2 and Figure 3 As shown, a method for EEG dynamic continuous emotion recognition based on channel selection is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used for explanation, and the steps include the following steps 201 to 204.

[0021] Step 201 , collecting multi-channel EEG time-series signals of a subject in a dynamic situation.

[0022] In an exemplary embodiment, an EEG acquisition paradigm in a dynamic situation is created. Based on the EEG acquisition paradigm, a 64-channel electrode cap is used to collect multi-channel EEG time series signals of the subject under target stimulation. When training EEG-Net, dynamic emotion labels corresponding to multi-channel EEG time series signals are collected at the same time, and the dynamic emotion labels are normalized. The dynamic emotion labels include the arousal and pleasure scores of six dynamic emotions.

[0023] Specifically, experts were organized to analyze the application scenarios and possible situations, and to design a relatively complete paradigm that can accurately reflect brain activity under this cognitive state. In isolation from the interference of the external environment, multiple dynamic emotion video clips that induce different dynamic emotions were played for the subjects. While the subjects were watching the dynamic emotion video clips, the discharge signals of the subjects' cerebral cortex under different cognitive activities were collected, and after watching a dynamic emotion video clip, the dynamic emotion video clip was scored for pleasure and arousal. After the scoring is completed, the next dynamic emotion video clip will be watched until the subject's EEG signal is in a stable state, until all dynamic emotion video clips are watched and scored. Comprehensively considering the subject's state at the collection site and the quality of the collected EEG signals, an effectiveness evaluation is conducted to eliminate EEG signals that perform poorly and cannot accurately reflect the dynamic emotional process.

[0024] As a specific implementation, the acquisition process of multi-channel EEG time series signals is as follows.

[0025] 1) Five experts were selected to select dynamic emotional video clips. All of them had received professional film training and had experience in rehearsing films. Twenty-four healthy subjects were selected to participate in the experiment, including 15 males and 9 females with an average age of (20±1.5) years old.

[0026] 2) The subjects were asked to wear EEG signal acquisition equipment in a quiet room with appropriate temperature and brightness.

[0027] 3) The subjects were placed in a dedicated environment that could block external interference. The brightness was always set at a comfortable level so that the subjects could clearly see the content of the dynamic emotional video clips. In order to ensure the smooth progress of the experiment, each subject was required to fill out a mental health assessment questionnaire before the experiment officially began, and each subject was trained using additional dynamic emotional video clips.

[0028] 4) If Figure 4 As shown in the figure, at the beginning of the experiment, prompts and precautions were played first. The experiment was divided into three sections. In each section, a dynamic emotion video that induced two emotions was played for the subjects. There were six dynamic emotion videos that induced different dynamic emotions. Each dynamic emotion video lasted about 180 seconds, of which each emotion was 90 seconds. Each dynamic emotion video was cut into segments of 5 seconds each, and each dynamic emotion video had a total of 36 cut segments. The presentation order of each dynamic emotion video was fixed. The subjects first adjusted their emotions to be in a relatively relaxed state. When the subjects' brain waves were stable, the dynamic emotion video segments began to play.

[0029] 5) Emotional evaluation: The subjects rated the arousal and pleasure of two different dynamic emotional videos in the dynamic emotional video, and rated the arousal and pleasure of 36 5-second cut clips of the dynamic emotional video, with 9 ratings. First, the arousal was rated, with numbers 1 to 9 corresponding to "very relaxed" to "very intense", and 5 representing "calm". Then the pleasure was rated, with numbers 1 to 9 corresponding to "very negative" to "very positive", and 5 representing "normal". The subjects filled in the ratings according to their true feelings.

[0030] 6) Set a rest time. After scoring each dynamic emotion video, rest for 15 to 30 seconds to allow the subject's brain waves to be in a stable state. Then continue to repeat the above process until the subject completes all the processes of the entire paradigm and the experiment ends.

[0031] This application can fully reflect the expression and changes of emotions in dynamic environments by obtaining the EEG timing signals and dynamic emotion labels of the subjects in dynamic situations, thereby providing rich emotion samples for the training of EEG-Net and emotion recognition models, greatly improving the practicality of emotion recognition, and filling the gap in the field of EEG emotion recognition in the direction of dynamic emotion recognition.

[0032] Step 202, filtering, artifact removal and resampling the multi-channel EEG time series signal in sequence. Further, differential operation is performed on the multi-channel EEG time series signal to obtain differential entropy features.

[0033] Step 203 : According to the multi-channel EEG time series signal, an optimal channel combination is determined using EEG-Net and a non-dominated sorting genetic algorithm (NSGA).

[0034] Among them, EEG-Net is pre-trained using a training sample set. The training sample set includes pre-processed multi-channel EEG time series sample signals and corresponding dynamic emotion labels. The non-dominated sorting genetic algorithm uses NSGA-II.

[0035] In an exemplary embodiment, the EEG time series signal of each channel is input into EEG-Net, and the score of each channel (including classification accuracy and computational complexity) is obtained by sorting according to the loss score, and the channels with scores higher than the average of all channels are screened out, and the non-dominated sorting genetic algorithm II is used to find the optimal channel combination. Step 203 includes the following steps 31 to 35.

[0036] Step 31, determining an initial population according to the multi-channel EEG time series signal. The individuals in the initial population represent a channel combination.

[0037] Step 32, for m In the next iteration, EEG-Net was used to m The EEG time series signals of each channel in the population at the iteration are classified to determine the classification accuracy and computational complexity of each channel; m >0; the population at the first iteration is the initial population.

[0038] The EEG-Net includes a two-dimensional convolutional layer, a deep convolutional layer, a channel attention layer and a separable convolutional layer connected in sequence.

[0039] (1) The two-dimensional convolution layer is used to extract the time-frequency features of the EEG time series signal, focusing on the feature extraction in the channel dimension and the time dimension. In the two-dimensional convolution layer, a two-dimensional convolution kernel of size (1, 64) is used for feature extraction, which means that the feature extraction range is between 2Hz and 128Hz. The formula is as follows: ; in, Indicates n The sample in The first The time-frequency characteristics at the time point, Indicates n The sample in The first The signal value at a time point, represents the convolution kernel size of the two-dimensional convolution layer, Represents the sample number. Each sample is a signal extracted from the EEG activity data in a time period or an experimental task. f Indicates the sequence number of the frequency components extracted after the convolution operation, Indicates the channel number. Channels represent different electrode positions in EEG recordings. Each channel corresponds to an electrode at a specific position. Indicates a point in time, represents the convolution kernel size in the channel dimension, represents the convolution kernel size in the time dimension, c represents the index on the channel dimension, Represents an index on the time dimension.

[0040] (2) In the deep convolution layer, the time-frequency features of each channel output by the two-dimensional convolution layer are individually subjected to deep convolution operations, thereby extracting higher-level spatial features from the relationship between different channels in order to better understand the differences between channels. The formula for the deep convolution layer is as follows: ; in, Indicates nThe sample in The first The spatial characteristics at a point in time, Represents the convolution kernel size of the depthwise convolutional layer.

[0041] (3) The channel attention layer introduces the channel attention mechanism to perform global average pooling on each channel to obtain the global features of each channel and generate a scalar value to measure the importance of each channel. Then, an attention weight is assigned to each channel to strengthen EEG-Net's attention to important channel features. The formula of the channel attention mechanism is as follows: ; in, Indicates n The sample in The global characteristics of the channels.

[0042] (4) The separable convolutional layer is used to integrate the global features of each channel output by the channel attention layer. The formula is as follows: ; in, Indicates n The sample in The first Final features at a point in time.

[0043] The EEG time series signal of each channel is further classified based on the final features output by the separable convolutional layer, and the classification accuracy and computational complexity of each channel are determined in combination with the true label.

[0044] Step 33, according to the classification accuracy and computational complexity of each channel, m The population at the iteration is non-dominated sorting, and the m The individuals in the population at the iteration are divided into multiple non-dominated fronts.

[0045] Step 34, calculate the crowding distance of each individual in the non-dominated front.

[0046] Step 35, according to the crowding distance of each individual in the non-dominated frontier, determine the m +1 iteration of the population, and then m +1 iterations, until the preset termination condition is met, the optimal channel combination is determined according to the population when the preset termination condition is met.

[0047] Specifically, a new population is selected based on non-dominated sorting and crowding distance, and then a new offspring population is generated through crossover and mutation operations.

[0048] In order to further optimize the performance of the EEG-Net model and reduce the computational complexity, this application adopts a non-dominated sorting genetic algorithm II for multi-objective channel screening, and determines a set of optimal channel combinations with good classification accuracy and computational complexity through NSGA-II.

[0049] Step 204, based on the EEG time series signal corresponding to the optimal channel combination, a pre-trained emotion recognition model is used to determine the continuous dynamic emotion category of the subject in the dynamic scene. The EEG time series signal input to the emotion recognition model is a signal after preliminary feature extraction.

[0050] The continuous dynamic emotion categories include joy to calm, calm to joy, sadness to calm, calm to sad, nervous to calm, and calm to nervous.

[0051] like Figure 5 As shown in the figure, based on the Transformer and Convolutional Neural Network (TCNN) algorithm, the emotion recognition model extracts local features and global features from the EEG time series signals corresponding to the optimal channel combination through short-term and long-term streams, and then uses the attention mechanism for feature fusion. The emotion recognition model includes a short-term stream feature extraction network, a long-term stream feature extraction network, an attention mechanism layer, and a classifier.

[0052] (1) The short-term stream feature extraction network is used to extract local features of EEG time series signals using a temporal convolution module. The main function of the short-term stream feature extraction network is to extract local features within a short time span.

[0053] In an exemplary embodiment, the temporal convolution module includes a causal convolution layer, a dilated convolution layer, a residual connection, a ReLU activation function and a normalization layer.

[0054] ① The input of the causal convolution layer is the EEG time series signal. The causal convolution is used to perform convolution operations on the time series data to ensure that the output of each time step depends only on the current time step and the previous input, and the output is a short-term emotional feature. The causal convolution layer can ensure the causality of the signal, that is, the convolution operation will not introduce future information, thereby avoiding data leakage and adapting to the characteristics of time series data. The formula of the causal convolution layer is as follows: ; in, Indicates t The output of the causal convolutional layer at time steps, Indicates t - j The input data of the causal convolutional layer of time steps, represents the convolution kernel weight of the causal convolution layer, represents the convolution kernel size of the causal convolution layer, Represents the bias term of the causal convolutional layer.

[0055] ② The input of the dilated convolution layer is the short-term emotional features output by the causal convolution layer. The receptive field of the convolution kernel is expanded by dilated convolution to capture the features of long-term dependence, and the output is the short-term emotional features after expansion. The dilated convolution layer introduces a dilation rate, adds intervals between convolution kernels, expands the receptive field, and does not increase the number of parameters. The formula is as follows: ; in, Indicates t The output of the convolutional layer is expanded by time steps, Indicates The input signal of the convolutional layer is expanded by time steps, d 1 represents the expansion rate, represents the convolution kernel weight of the dilated convolution layer, Represents the bias term of the dilated convolutional layer.

[0056] ③ The input of the residual connection is the short-term emotion features output by the causal convolution layer and the short-term emotion features after expansion. The short-term emotion features output by the causal convolution layer and the short-term emotion features after expansion are added, and the output is the features after the residual connection.

[0057] This application solves the gradient vanishing problem in deep networks and stabilizes the training process by introducing residual connections between the input and convolution output to help information flow in the network. The formula is as follows: ; in, Indicates t The output after residual connection of time steps, Indicates t The input signal is a time step, represents the feature map extracted by the convolution operation, Represents the convolution kernel parameters.

[0058] ④ The input of the ReLU activation function is the feature after residual connection. The ReLU activation function sets the values ​​less than zero in the feature after residual connection to zero, and the values ​​greater than zero remain unchanged. The output is the feature after ReLU activation. The formula of the ReLU activation function is as follows: ; in, Indicates t The output of the activation function for each time step.

[0059] ⑤ The input of the standardization layer is the features after ReLU activation. The features after ReLU activation are normalized by the standardization layer, and the output is the local features. The standardization layer ensures the stability of feature distribution and accelerates training. The formula is as follows: ; in, Indicates t The output after normalization of time steps is represents the mean, represents the variance, and represent the scaling parameters, represents the translation parameter, represents a smoothing term that prevents the denominator from reaching zero.

[0060] In an exemplary embodiment, during the training process of the short-term flow feature extraction network, L2 regularization is used in the loss function to prevent overfitting. The formula is as follows: ; in, represents the regularization loss term, H represents the number of weights in the short-term flow feature extraction network, represents the first h weights, Represents the regularization coefficient, which controls the strength of the penalty term.

[0061] In the training process of the short-term flow feature extraction network, Dropout is further used to randomly "discard" some neurons in the neural network to reduce the complexity of the short-term flow feature extraction network, thereby improving the generalization ability. The formula is as follows: ; in, Indicates t The output after Dropout for time steps, Representation and A random mask matrix of the same dimension, Represents neurons with probability reserve.

[0062] (2) The long-term stream feature extraction network is used to extract the global features of the EEG time series signal using the Transformer module. The main function of the long-term stream feature extraction network is to capture the long-term temporal dependency of the emotion data.

[0063] In an exemplary embodiment, the Transformer module includes a feature encoding layer, a multi-head self-attention layer, and a feedforward neural network connected in sequence.

[0064] The process of extracting global features of EEG time series signals using the Transformer module in the long-term stream feature extraction network includes the following steps 46 to 48.

[0065] Step 46, adding position coding to the EEG time series signal through the feature coding layer to obtain the encoded EEG time series signal. Since the Transformer module itself does not have the ability to process sequence order information, it is necessary to add sequence information to the EEG signal of each time step through position coding so that the Transformer module can identify the timing dependency in the EEG time series signal. Position coding is achieved by adding a vector representing the position of the time step to the EEG signal of each time step.

[0066] The feature coding layer uses the following formula to add position coding to the EEG time series signal: ; ; ; in, is the encoded EEG time series signal, X is the EEG timing signal, PE is the position code, is the time step, is the feature dimension, is the index of the feature dimension, For the Time step The position encoding of the feature dimension. For the Time step 2 + Positional encoding of 1 feature dimension.

[0067] Step 47, extracting the global relationship of the encoded EEG time series signal through the multi-head self-attention layer, and generating a multi-head attention feature.

[0068] Through the parallel calculation of multiple self-attention layers, it is possible to focus on multiple attention heads at the same time, thereby capturing information from different parts. The calculation formula for each attention head is as follows: ; ; ; in, Indicates The attention features output by the attention heads, Indicates The attention scores output by the attention heads are Indicates The query vector of the attention head, Indicates The key vector of the attention head, Indicates The value vector of the attention head, represents the scaling factor, represents the query weight matrix, represents the key weight matrix, Represents the value weight matrix.

[0069] Step 48, transform the multi-head attention features through the feedforward neural network to obtain global features.

[0070] The feedforward neural network consists of two linear layers and a nonlinear activation function, which is used to further transform the features of each time step. The formula is as follows: ; in, L Represents global features, represents a feed-forward neural network, represents the multi-head attention feature, and is the weight matrix, and is the bias term.

[0071] (3) The attention mechanism layer is used to fuse local features and global features to obtain fused features.

[0072] This application uses the attention mechanism layer to dynamically generate fusion weights. Specifically, the following formula is used to calculate the attention weights of local features and global features: ; ; in, represents the attention weight of local features, represents the attention weight of the global feature, is the bias of the global feature, is the bias term of local features, S Represents local features, L Represents global features, Represents the weight matrix of local features, The weight matrix representing the global features.

[0073] (4) The classifier is used to determine the continuous dynamic emotion category based on the fused features.

[0074] This application constructs a dual-stream network structure. The long-term stream feature extraction network ensures that the input sequence can correctly express the timing information through feature encoding, and then captures the global dependency through multi-head self-attention, enhancing the emotion recognition model's ability to focus on different features. Finally, a feedforward neural network is used for nonlinear transformation to generate a high-level feature representation, effectively capturing the long-term time dependency and global features of the EEG timing signal. This overcomes the shortcomings of TCNN in capturing long-term emotional changes and lacking global information integration, and has higher feature extraction accuracy and stronger emotional change capture capabilities, thereby improving the accuracy and stability of emotion recognition.

[0075] Based on the same inventive concept, the embodiment of the present application also provides an EEG dynamic continuous emotion recognition system for implementing the EEG dynamic continuous emotion recognition method involved above. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more EEG dynamic continuous emotion recognition system embodiments provided below can refer to the limitations of the EEG dynamic continuous emotion recognition method above, and will not be repeated here.

[0076] In an exemplary embodiment, Figure 6 As shown, a system for dynamic and continuous EEG emotion recognition is provided, including: a signal acquisition unit 601 , a channel selection unit 602 and an emotion recognition unit 603 .

[0077] The signal acquisition unit 601 is used to acquire multi-channel EEG time series signals of the subject in a dynamic situation.

[0078] The channel selection unit 602 is used to determine the optimal channel combination according to the multi-channel EEG time series signal by using EEG-Net and non-dominated sorting genetic algorithm.

[0079] The emotion recognition unit 603 is used to determine the continuous dynamic emotion category of the subject in a dynamic scenario according to the EEG time series signal corresponding to the optimal channel combination and using a pre-trained emotion recognition model.

[0080] The emotion recognition model includes a short-term stream feature extraction network, a long-term stream feature extraction network, an attention mechanism layer and a classifier. The short-term stream feature extraction network is used to extract local features of EEG time series signals using a time series convolution module. The long-term stream feature extraction network is used to extract global features of EEG time series signals using a Transformer module. The attention mechanism layer is used to fuse local features and global features to obtain fused features; the classifier is used to determine the continuous dynamic emotion category based on the fused features.

[0081] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store multi-channel EEG timing signals of subjects in dynamic situations. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for dynamic continuous emotion recognition of EEG based on channel selection is implemented.

[0082] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0083] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0084] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0085] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0086] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0087] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0088] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0089] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for dynamic continuous emotion recognition based on channel selection of EEG, characterized in that: The EEG dynamic continuous emotion recognition method based on channel selection includes: Collect multi-channel EEG time series signals of subjects in dynamic situations; According to the multi-channel EEG time series signal, an optimal channel combination is determined using EEG-Net and a non-dominated sorting genetic algorithm; According to the EEG time series signals corresponding to the optimal channel combination, the pre-trained emotion recognition model is used to determine the continuous dynamic emotion categories of the subjects in dynamic situations; Among them, the emotion recognition model includes a short-term flow feature extraction network, a long-term flow feature extraction network, an attention mechanism layer and a classifier; the short-term flow feature extraction network is used to extract local features of EEG time series signals using a time series convolution module; the long-term flow feature extraction network is used to extract global features of EEG time series signals using a Transformer module; the attention mechanism layer is used to fuse local features and global features to obtain fused features; the classifier is used to determine the continuous dynamic emotion category based on the fused features.

2. The method for dynamic continuous emotion recognition based on channel selection according to claim 1 is characterized in that: According to the multi-channel EEG time series signal, the optimal channel combination is determined by using EEG-Net and non-dominated sorting genetic algorithm, which specifically includes: Determine an initial population according to the multi-channel EEG time series signal; individuals in the initial population represent a channel combination; For the m In the next iteration, EEG-Net was used to m The EEG time series signals of each channel in the population at the iteration are classified to determine the classification accuracy and computational complexity of each channel; m >0; the population at the first iteration is the initial population; According to the classification accuracy and computational complexity of each channel, m The population at the iteration is non-dominated sorting, and the m The individuals in the population at the iteration are divided into multiple non-dominated frontiers; Calculate the crowding distance of individuals in each non-dominated front; According to the crowding distance of individuals in each non-dominated frontier, determine the m +1 iteration of the population, and then m +1 iterations, until the preset termination condition is met, the optimal channel combination is determined according to the population when the preset termination condition is met.

3. The method for dynamic continuous emotion recognition based on channel selection according to claim 2 is characterized in that: The EEG-Net includes a two-dimensional convolutional layer, a deep convolutional layer, a channel attention layer and a separable convolutional layer connected in sequence.

4. The method for EEG dynamic continuous emotion recognition based on channel selection according to claim 1, characterized in that: The temporal convolution module includes a causal convolution layer, a dilated convolution layer, a residual connection, a ReLU activation function and a normalization layer; The input of the causal convolution layer is the EEG time series signal. The causal convolution is used to perform a convolution operation on the time series data to ensure that the output of each time step depends only on the current time step and the previous input, and the output is a short-term emotional feature. The input of the dilated convolution layer is the short-term emotional features output by the causal convolution layer. The receptive field of the convolution kernel is expanded by dilated convolution to capture the features of long-term dependence, and the output is the dilated short-term emotional features. The input of the residual connection is the short-term emotion feature output by the causal convolution layer and the short-term emotion feature after expansion. The short-term emotion feature output by the causal convolution layer and the short-term emotion feature after expansion are added, and the output is the feature after residual connection; The input of the ReLU activation function is the feature after residual connection. The ReLU activation function sets the value less than zero in the feature after residual connection to zero, and the value greater than zero remains unchanged, and the output is the feature after ReLU activation; The input of the standardization layer is the features after ReLU activation. The features after ReLU activation are normalized by the standardization layer, and the output is the local features.

5. The method for EEG dynamic continuous emotion recognition based on channel selection according to claim 1, characterized in that: The Transformer module includes a feature encoding layer, a multi-head self-attention layer and a feedforward neural network connected in sequence; The process of extracting the global features of the EEG time series signal using the Transformer module in the long-term stream feature extraction network includes: Adding position coding to the EEG time series signal through the feature coding layer to obtain a coded EEG time series signal; Extracting the global relationship of the encoded EEG time series signal through the multi-head self-attention layer and generating multi-head attention features; The multi-head attention features are transformed through the feedforward neural network to obtain global features.

6. The method for EEG dynamic continuous emotion recognition based on channel selection according to claim 5, characterized in that: The feature coding layer uses the following formula to add position coding to the EEG time series signal: ； ； ； in, is the encoded EEG time series signal, X is the EEG timing signal, PE is the position code, is the time step, is the feature dimension, is the index of the feature dimension, For the Time step The position encoding of feature dimensions, For the Time step 2 + Positional encoding of 1 feature dimension.

7. The method for dynamic continuous emotion recognition based on channel selection according to claim 1, characterized in that: The continuous dynamic emotion categories include joy to calm, calm to joy, sadness to calm, calm to sad, nervous to calm, and calm to nervous.

8. The method for EEG dynamic continuous emotion recognition based on channel selection according to claim 1, characterized in that: The EEG dynamic continuous emotion recognition method based on channel selection also includes: The multi-channel EEG time series signal is filtered, artifact removed and resampled in sequence.

9. An EEG dynamic continuous emotion recognition system based on channel selection, applied to the EEG dynamic continuous emotion recognition method based on channel selection according to any one of claims 1 to 8, characterized in that: The EEG dynamic continuous emotion recognition system based on channel selection includes: A signal acquisition unit, used to acquire multi-channel EEG time-series signals of the subject in a dynamic situation; A channel selection unit, used to determine the optimal channel combination according to the multi-channel EEG time series signal by using EEG-Net and non-dominated sorting genetic algorithm; The emotion recognition unit is used to determine the continuous dynamic emotion category of the subject in the dynamic scenario according to the EEG time series signal corresponding to the optimal channel combination and using the pre-trained emotion recognition model; Among them, the emotion recognition model includes a short-term flow feature extraction network, a long-term flow feature extraction network, an attention mechanism layer and a classifier; the short-term flow feature extraction network is used to extract local features of EEG time series signals using a time series convolution module; the long-term flow feature extraction network is used to extract global features of EEG time series signals using a Transformer module; the attention mechanism layer is used to fuse local features and global features to obtain fused features; the classifier is used to determine the continuous dynamic emotion category based on the fused features.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the EEG dynamic continuous emotion recognition method based on channel selection as described in any one of claims 1 to 8.

Images