Emotion recognition method and system based on brain map features

Abstract

The invention relates to an emotion recognition method and system based on brain map features, and belongs to the field of signal processing. Comprising the following steps: constructing a video material database, and collecting an electroencephalogram signal when a subject watches a video in the video material database; preprocessing the electroencephalogram signal, performing time domain segmentation, frequency domain frequency division, feature extraction and spatial domain mapping on the preprocessed electroencephalogram signal to obtain a two-dimensional map, and performing frequency domain superposition and time domain superposition on the two-dimensional map to obtain a four-dimensional brain map; and inputting the four-dimensional brain map into a trained emotion recognition model to obtain a predicted emotion. According to the method, by collecting the electroencephalogram signals when the testee watches the video in the constructed video material database and converting the collected electroencephalogram signals into the four-dimensional data block containing the frequency domain information, the space domain information and the time domain information through the four-dimensional brain map, the emotion recognition accuracy is improved; the problem that fine-grained accurate recognition of corresponding emotions is achieved for materials with small emotional spans based on electroencephalogram signals is solved.

Description

Technical Field

[0001] The present invention relates to the field of signal processing technology, and in particular to an emotion recognition method and system based on brain map features. Background Art

[0002] Emotions are human responses to daily interactions. Accurately identifying the audience's emotions helps adjust the content and communication strategies of media production, and plays a vital role in improving the effectiveness of media communication. In recent years, with the development of machine learning and deep learning, emotion recognition through biometric recognition of brain waves (Electroencephalogram, EEG) and feature parameter extraction has become a research hotspot.

[0003] Currently, existing EEG-based emotion recognition methods usually select materials from videos with a large emotional span for subjects to watch, such as harvest, earthquake, etc., so that emotions are easier to recognize. However, in many scenarios, especially in the process of media communication, the emotional span of videos is small, and how to achieve fine-grained recognition of corresponding emotions is still the main challenge that needs to be faced. Secondly, existing EEG-based emotion recognition methods usually only consider one or two of the frequency domain, spatial domain, and time domain features of EEG signals, which may result in the loss of key emotional information and reduce the accuracy of emotion recognition. Finally, existing EEG-based emotion recognition methods usually only classify the predicted emotions into positive, negative, and neutral, and the recognition granularity is low.

[0004] It can be seen that how to achieve fine-grained and accurate recognition of corresponding emotions for materials with a smaller emotional span based on EEG signals is an urgent problem to be solved. Summary of the invention

[0005] In view of the above analysis, the present invention aims to provide an emotion recognition method and system based on brain map features to solve the problem that current emotion recognition methods cannot accurately recognize emotions for materials with a small emotional span and have low recognition granularity.

[0006] The present invention provides an emotion recognition method and system based on brain map features, the method comprising the following steps:

[0007] Building a video material database, and collecting EEG signals of subjects when they watch videos in the video material database;

[0008] Preprocessing the EEG signal, performing time domain segmentation, frequency domain division, feature extraction, and spatial domain mapping on the preprocessed EEG signal to obtain a two-dimensional map, and performing frequency domain superposition and time domain superposition on the two-dimensional map to obtain a four-dimensional brain map;

[0009] The four-dimensional brain map is input into a trained emotion recognition model to obtain predicted emotions.

[0010] Furthermore, the preprocessed EEG signal is subjected to time domain segmentation, frequency domain division, feature extraction, and spatial domain mapping to obtain a two-dimensional spectrum, including: segmenting the preprocessed EEG signal into a series of non-overlapping time segments of length Ts in the time domain; for each time segment, dividing it into four frequency bands of alpha, beta, gamma and theta in the frequency domain; for each frequency band, extracting the differential entropy and power spectral density of the EEG signal; for each frequency band, mapping the corresponding differential entropy and power spectral density to a two-dimensional grid corresponding to the position of the EEG signal electrode for spatial domain mapping, to obtain a two-dimensional spectrum corresponding to each frequency band.

[0011] Furthermore, the two-dimensional maps are superimposed in the frequency domain and the time domain to obtain a four-dimensional brain map, including: superimposing the two-dimensional maps corresponding to each frequency band, and then superimposing each time segment to obtain a four-dimensional brain map.

[0012] Furthermore, the two-dimensional grid is an 8×9 spatial layout, wherein each grid point represents the position of an EEG signal electrode that complies with the international 10-20 standard; there are two two-dimensional spectra corresponding to each frequency band, including a differential entropy two-dimensional spectrum and a power spectral density two-dimensional spectrum; each two-dimensional spectrum represents an 8×9 data block.

[0013] Furthermore, the eight two-dimensional maps corresponding to the four frequency bands are superimposed to obtain an 8×9×8 data block; then the time segments are superimposed to obtain a four-dimensional brain map, which represents a data blocks, where T represents the total duration of the preprocessed EEG signal, Ts is the length of the time segment, Represents the number of time slices.

[0014] Furthermore, the preprocessing of the EEG signal includes:

[0015] The EEG signals corresponding to each subject watching the video are extracted through bandpass filtering and downsampled;

[0016] Extract bad electrodes and interpolate the downsampled EEG signals;

[0017] The interpolated EEG signals were de-artifacted and re-referenced by whole-brain average;

[0018] The EEG signal after re-reference is intercepted.

[0019] Furthermore, the trained emotion recognition model is a multi-layer perceptron.

[0020] Furthermore, the parameters of the emotion recognition model are determined based on a genetic algorithm to obtain the trained emotion recognition model.

[0021] Furthermore, the differential entropy is obtained by the following formula:

[0022]

[0023] Among them, h(x i ) represents x i The differential entropy of x i represents the EEG signal corresponding to the ith frequency band, σ 2 Represents x i The variance of , e represents the base of the natural logarithm;

[0024] The energy is obtained by the following formula:

[0025]

[0026] Where E represents x i The energy of ij represents the EEG signal corresponding to the jth sliding time window in the ith frequency band, and N represents the number of sliding time windows.

[0027] Furthermore, the system comprises:

[0028] Video material database, used to provide videos for subjects to watch;

[0029] An EEG signal collector is used to collect EEG signals of the subject when watching the video and send them to the data processing module;

[0030] The data processing module is used to pre-process the received EEG signals, perform time domain segmentation, frequency domain division, feature extraction, and spatial domain mapping on the pre-processed EEG signals to obtain a two-dimensional map, perform frequency domain superposition and time domain superposition on the two-dimensional map to obtain a four-dimensional brain map, and send the four-dimensional brain map to the model processing module;

[0031] The model processing module is used to run the trained emotion recognition model and input the four-dimensional brain map into the trained emotion recognition model to obtain predicted emotions.

[0032] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0033] 1. The present invention converts the EEG signal into a four-dimensional data block containing its frequency domain, spatial domain and time domain information through a four-dimensional brain map, retains the key emotional information in the EEG signal, and improves the accuracy of emotion recognition.

[0034] 2. The present invention collects videos with a small emotional span to construct a video material database, and collects EEG signals and subsequently recognizes emotions based on the video material database, thereby solving the problem of achieving fine-grained and accurate recognition of corresponding emotions for materials with a small emotional span based on EEG signals, and providing a basis for improving the effectiveness of media communication.

[0035] 3. The present invention improves the accuracy of emotion recognition by extracting differential entropy and power spectral density of EEG signals as features.

[0036] 4. The present invention divides the emotions that appear when the subjects watch the videos in the video material database, constructs a discrete emotion set, and covers complex emotions intertwined with multiple emotions, so that the subjects can express their emotional feelings more accurately, providing a basis for achieving fine-grained and accurate recognition of emotions.

[0037] 5. The present invention determines the parameters of the emotion recognition model through a genetic algorithm, thereby obtaining a trained emotion recognition model, optimizing the parameters of the model, and improving the accuracy of emotion recognition.

[0038] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0040] Figure 1 This is a flow chart of an emotion recognition method based on brain map features according to an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a mind map creation process according to an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of nine videos selected for an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of performance comparison of different classifier models according to an embodiment of the present invention;

[0044] Figure 5 is a schematic diagram of an ablation experiment for different input features according to an embodiment of the present invention;

[0045] Figure 6 4 is a block diagram of an emotion recognition system based on brain map features according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0047] Embodiment 1:

[0048] A specific embodiment of the present invention discloses an emotion recognition method based on brain map features. Figure 1 As shown, the method comprises the following steps:

[0049] Step S1, constructing a video material database, and collecting EEG signals of subjects when they watch videos in the video material database;

[0050] Step S2, preprocessing the EEG signal, performing time domain segmentation, frequency domain division, feature extraction, and spatial domain mapping on the preprocessed EEG signal to obtain a two-dimensional map, and performing frequency domain superposition and time domain superposition on the two-dimensional map to obtain a four-dimensional brain map;

[0051] Step S3: input the four-dimensional brain map into the trained emotion recognition model to obtain predicted emotions.

[0052] Specifically, in step S1, videos with a small emotional span are collected to construct a video material database, videos are randomly selected from the video material database, and the subjects watch the selected videos while collecting their EEG signals.

[0053] For example, videos about Chinese stories are collected to construct a video material database, and the emotional span of the relevant videos is relatively small. It should be noted that videos can be selected to construct a video material database according to actual needs or actual application scenarios.

[0054] Specifically, when watching the selected video, the subject first rests for a period of time, and then watches the selected video. The subject's EEG signal is collected by an EEG signal collector, which includes an EEG (Electroencephalogram) electrode cap and an EEG signal recorder; the EEG electrode cap is distributed according to the international 10-20 standard, and the ground electrode is FPz; the sampling frequency of the EEG signal recorder is 1000Hz.

[0055] It should be noted that the existing emotion recognition methods based on EEG signals usually select materials from videos with a large emotional span for the subjects (i.e., the subjects) to watch, such as harvest, earthquake, etc. These videos have obvious emotional orientation, making emotions easier to recognize. However, the present invention aims to solve the problem of realizing fine-grained and accurate recognition of corresponding emotions for materials with a small emotional span based on EEG signals, thereby providing a basis for improving the effectiveness of media communication. Therefore, the present invention collects videos with a small emotional span to construct a video material database, and collects EEG signals and subsequently recognizes emotions based on the video material database, which solves the aforementioned problems and provides a basis for improving the effectiveness of media communication.

[0056] Specifically, in step S2, the preprocessing of the EEG signal includes:

[0057] The EEG signals corresponding to each subject watching the video are extracted through bandpass filtering and downsampled;

[0058] Extract bad electrodes and interpolate the downsampled EEG signals;

[0059] The interpolated EEG signals were de-artifacted and re-referenced by whole-brain average;

[0060] The EEG signal after re-reference is intercepted.

[0061] Specifically, the EEG signal corresponding to each subject watching the video was extracted through a 0.5-49 Hz bandpass filter and downsampled to 128 Hz.

[0062] It should be noted that, since the subjects first rest for a period of time and then watch the video when watching the video, the collected EEG signals include the EEG signals corresponding to the resting state and the video watching state. Through the above-mentioned bandpass filtering, the EEG signals corresponding to the subjects when watching the video can be effectively extracted.

[0063] Furthermore, bad electrodes are extracted from the downsampled EEG signal, and the extracted bad electrodes are saved in a data list. The bad electrodes include the names of electrodes marked as bad electrodes.

[0064] It should be noted that bad electrodes refer to poor brain electrode data caused by improper operation during the acquisition process, whose amplitude exceeds the upper or lower threshold, and the bad electrodes are screened out through the threshold.

[0065] Specifically, for each bad electrode in the data list:

[0066] According to the name of its electrode, get the name of its location and the electrodes around it;

[0067] Determine whether the electrodes around it are bad electrodes according to the names of the electrodes around it;

[0068] The bad electrode is interpolated based on the amplitudes of the surrounding non-bad electrodes.

[0069] Specifically, according to the international 10-20 standard distribution, the position of the bad electrode can be obtained through the electrode name of the bad electrode, and the names of the electrodes around it can be obtained through the position of the bad electrode. The names of the electrodes around it are searched in the data list in sequence. If the names are found, the corresponding electrode is judged to be a bad electrode, otherwise it is a good electrode. The bad electrode is interpolated based on the average value of the amplitudes of the non-bad electrodes around it.

[0070] Furthermore, the interpolated EEG signal was filtered, and the artifact component was obtained and removed through ICA (Independent Component Analysis), and the whole-brain average value was re-referenced.

[0071] Specifically, the interpolated EEG signal is band-pass filtered from 8 to 35 Hz to remove low-frequency drift and high-frequency noise. The number of ICA components is set, and the artifact component is obtained and removed through ICA. The average value of the EEG signal of all electrodes in the whole brain after artifact removal is used as the reference value.

[0072] It should be noted that artifacts refer to EEG signal noise or artifacts caused by environmental factors, blinking, muscle tension, etc. These artifacts will seriously interfere with the quality of EEG signals and affect the accuracy of emotion recognition, so they need to be removed. ICA can identify artifact components by decomposing EEG signals. The present invention calls the ICA algorithm through the MNE-Python library to remove artifacts. Re-referencing the whole brain average can reduce artifacts caused by different reference electrode positions.

[0073] Furthermore, the EEG signal after re-reference is intercepted, for example, the data of the last 100 seconds is retained.

[0074] Furthermore, the preprocessed EEG signal is segmented into a series of non-overlapping time segments of length Ts in the time domain; each time segment is divided into four frequency bands of alpha, beta, gamma and theta in the frequency domain; for each frequency band, the differential entropy (DE) and power spectral density (PSD) of the EEG signal are extracted; for each frequency band, the corresponding differential entropy and power spectral density are respectively mapped to a two-dimensional grid corresponding to the position of the EEG signal electrode for spatial domain mapping, and a two-dimensional spectrum corresponding to each frequency band is obtained.

[0075] Specifically, the above-mentioned frequency bands have different frequency ranges and represent different neural activity information. The frequency range of the alpha band is 8-13Hz, representing the state of a person when awake, quiet and with eyes closed; the frequency range of the beta band is 13-30Hz, representing the state of a person when the mind is active, alert and nervous; the frequency range of the theta band is 4-8Hz, representing the state of a person when sleeping, relaxing and meditating; the frequency range of the gamma band is 30-100Hz, representing the state of a person when the brain is highly active, cognitive and perceptual. For each time segment, according to the frequency range of the frequency band, the corresponding Butterworth bandpass filter is applied, and the frequency domain is divided into four key frequency bands of alpha, beta, gamma and theta, that is, the number of frequency bands is equal to 4. For each frequency band, the differential entropy of the EEG signal is obtained, and the energy of the EEG signal is obtained as its power spectral density using a sliding time window of 0.5 seconds. Figure 2 As shown, the two-dimensional grid is an 8×9 spatial layout, wherein each grid point represents the position of an EEG signal electrode that meets the international 10-20 standard; there are two two-dimensional spectra corresponding to each frequency band, including a differential entropy two-dimensional spectrum and a power spectrum density two-dimensional spectrum; the differential entropy two-dimensional spectrum and the power spectrum density two-dimensional spectrum corresponding to each frequency band are obtained respectively by the differential entropy and power spectrum density corresponding to each frequency band, therefore, there are two two-dimensional spectra corresponding to each frequency band; each two-dimensional spectrum represents an 8×9 data block.

[0076] It should be noted that each electrode is an acquisition channel, and the corresponding differential entropy and power spectral density are extracted from the EEG signal collected by each electrode, and then the extracted differential entropy and power spectral density are respectively mapped to the grid points corresponding to the electrode.

[0077] Furthermore, the differential entropy is obtained by the following formula:

[0078]

[0079] Among them, h(x i ) represents x i The differential entropy of x i represents the EEG signal corresponding to the ith frequency band, σ 2 Represents x i The variance of , e represents the base of the natural logarithm.

[0080] Furthermore, the energy is obtained by the following formula:

[0081]

[0082] Where E represents x i The energy of ijrepresents the EEG signal corresponding to the jth sliding time window in the ith frequency band, and N represents the number of sliding time windows.

[0083] It should be noted that the power spectrum density of the EEG signal can usually be approximately represented by energy. The present invention improves the accuracy of emotion recognition by extracting the differential entropy and power spectrum density of the EEG signal as features.

[0084] Furthermore, the two-dimensional maps corresponding to each frequency band are superimposed, and then each time segment is superimposed to obtain a four-dimensional brain map.

[0085] Specifically, Figure 2 As shown, the eight two-dimensional maps corresponding to the four frequency bands are superimposed to obtain an 8×9×8 data block; then the time segments are superimposed to obtain a four-dimensional brain map, which represents a data blocks, where T represents the total duration of the preprocessed EEG signal, for example, 100 seconds, and Ts is the length of the time segment. Represents the number of time slices.

[0086] It can be understood that the present invention converts the EEG signal into a four-dimensional data block containing its frequency domain, spatial domain and time domain information through a four-dimensional brain map, retains the key emotional information in the EEG signal, and improves the accuracy of emotion recognition.

[0087] Specifically, in step S3, the trained emotion recognition model is obtained by the following method:

[0088] Step S31, constructing a discrete emotion set, collecting EEG signals of the subjects when they watch the videos in the video material database;

[0089] Step S32, preprocessing the EEG signal, obtaining a four-dimensional brain map based on the preprocessed EEG signal, annotating the four-dimensional brain map based on the discrete emotion set, and constructing a data set based on the annotated four-dimensional brain map;

[0090] Step S33: construct an emotion recognition model, and determine the parameters of the emotion recognition model based on the genetic algorithm and the data set to obtain a trained emotion recognition model.

[0091] Specifically, in step S31, the emotions that appear when the subjects watch the videos in the video material database are divided to construct a discrete emotion set; the discrete emotion set includes eight discrete emotions.

[0092] Furthermore, each discrete emotion in the discrete emotion set represents a specific emotion type, and each discrete emotion has the attributes of pleasure and arousal. The pleasure represents the positivity of the emotion, high pleasure represents positive emotions, and low pleasure represents negative emotions; the arousal represents the intensity of the emotion, high arousal represents strong emotions, such as excitement or tension, and low arousal represents calm emotions, such as relaxation or fatigue. Each discrete emotion has high or low pleasure, high or low arousal attributes. After watching the video in the video material database, the subjects scored the emotions that appeared, divided the emotions that appeared based on the scoring results, and constructed a discrete emotion set.

[0093] Exemplarily, the emotions that appear when the subjects watch videos in a video material database constructed from Chinese stories are divided to construct a discrete emotion set, which includes eight discrete emotions: admired, proud, happy, surprised, introspective, relaxed, bored, and touched.

[0094] It can be understood that in the above example, since the videos related to Chinese stories are all neutral to positive emotions, the emotions in the discrete emotion set are all neutral to positive.

[0095] It should be noted that after the subjects watched the videos in the video material database, they scored the basic discrete emotions according to their own emotions. These basic discrete emotions constitute the basis of all emotions. If the emotional type classification is unreasonable, it will lead to overlaps between different emotional types, or some emotions cannot be classified into any emotional type. Since the present invention aims to achieve fine-grained and accurate recognition of emotions, the present invention divides the emotions that appear in the videos watched by the subjects in the video material database, constructs a discrete emotion set, and subdivides the emotions into eight discrete emotions, which include complex emotions intertwined with multiple emotions, so that the subjects can express their emotional feelings more accurately, providing a basis for achieving fine-grained and accurate recognition of emotions.

[0096] Furthermore, a video is randomly selected from the video material database, and the subject watches the selected video while collecting his / her EEG signal.

[0097] It should be noted that in step S31, there are multiple subjects and selected videos, and each subject needs to watch all the selected videos.

[0098] Exemplarily, 9 videos are randomly selected from the video material database such as Figure 3As shown in the figure, each video lasts 2-2.5 minutes, and each subject completes watching the 9 videos. When a subject watches each video, the subject first rests for one minute to stabilize the EEG signal, and then watches the video; after watching the video, the subject makes an emotional evaluation, and then watches the next video.

[0099] Furthermore, after watching the currently selected video, the current subject makes an emotional evaluation on the video, that is, selects a corresponding discrete emotion from the discrete emotion set.

[0100] Specifically, in step S32, the EEG signals of all subjects are preprocessed according to the method in step S2, and the four-dimensional brain maps corresponding to all subjects are obtained based on the preprocessed EEG signals, and the four-dimensional brain maps corresponding to all subjects are labeled based on the emotional evaluation; wherein, the labeled four-dimensional brain map corresponding to a subject watching a video is a sample, and the labeled four-dimensional brain maps corresponding to all subjects constitute a data set.

[0101] It should be noted that the emotional evaluation of a video by the subject includes selecting one or more discrete emotions, because when watching a video, multiple emotions may appear at the same time. For example, when watching a video of a rocket launch, emotions such as pride, happiness, and emotion may appear at the same time. Therefore, the labels of the four-dimensional brain map corresponding to the subject watching a video can be one or more.

[0102] Specifically, in step S33, a multilayer perceptron is constructed as an emotion recognition model, wherein the number of hidden layers is 2; the parameters of the emotion recognition model include the number of neurons in each layer of the hidden layer, the type of activation function, and the type of optimizer.

[0103] It should be noted that the multilayer perceptron, as a classifier, includes an input layer, a hidden layer, and an output layer; each node in the hidden layer and the output layer undergoes a linear transformation, and an activation function is applied to map the weighted sum of the neurons in the previous layer to the output for classification.

[0104] Furthermore, determining the parameters of the emotion recognition model based on the genetic algorithm and the data set to obtain a trained emotion recognition model includes:

[0105] Step S301, dividing the data set into a training set and a test set in a ratio of 8:2;

[0106] Step S302, initializing configuration parameters, and generating genetic individuals based on the configuration parameters to construct a population;

[0107] Step S303, calculating the fitness value of each genetic individual in the population, sorting all genetic individuals from small to large according to their fitness values, and selecting the genetic individuals in the first half of the sorting to form an excellent population;

[0108] Step S304, determine whether the termination condition is met. If so, obtain the corresponding trained emotion recognition model based on the genetic individual with the smallest fitness value in the population; otherwise, crossover and mutation are performed according to the crossover probability and mutation probability to obtain new genetic individuals, and an updated population is obtained based on the genetic individuals in the excellent population and the new genetic individuals. The updated population is used as the population for the next iteration, and steps S303 to S304 are repeated.

[0109] Specifically, in step S302, the configuration parameters include the number of genetic individuals in the population, the number of neurons in the first layer of the hidden layer, the number of neurons in the second layer of the hidden layer, the type of activation function, the type of optimizer, the crossover probability and the mutation probability. Genetic individuals are generated based on the number of neurons in the first layer of the hidden layer, the number of neurons in the second layer of the hidden layer, the type of activation function, the type of optimizer, that is, the parameters of the emotion recognition model. The number of neurons in the first layer of the hidden layer is an integer between [2,100], and the number of neurons in the second layer of the hidden layer is an integer between [2,50]; the type of activation function is one of identity (identity function), logistic (logistic function), tanh (hyperbolic tangent function), relu (linear rectified unit function); the type of optimizer is one of lbfgs (L-BFGS algorithm optimizer), sgd (stochastic gradient descent optimizer), and adam (Adam optimizer). The initial population is constructed by randomly selecting corresponding data within the above value range to generate genetic individuals; each genetic individual has 4 eigenvalues, among which the first eigenvalue is the number of neurons in the first layer of the hidden layer, the second eigenvalue is the number of neurons in the second layer of the hidden layer, the third eigenvalue is the type of activation function, and the fourth eigenvalue is the type of optimizer.

[0110] It should be noted that the number of genetic individuals in the population is an even number, which is convenient for subsequent crossover and mutation operations. The set crossover probability and mutation probability should ensure that the number of new genetic individuals obtained through crossover and mutation is greater than or equal to half of the number of genetic individuals in the population.

[0111] For example, the number of genetic individuals in the population is set to 40, the crossover probability is 0.95, and the mutation probability is 0.05. A genetic individual is generated as: X = [46, 16, relu, sgd].

[0112] Specifically, in step S303, the calculation of the fitness value of each genetic individual in the population includes:

[0113] Based on the characteristic value of each genetic individual, an emotion recognition model corresponding to each genetic individual is constructed, and the learning rate, regularization parameter alpha, and maximum number of iterations are set;

[0114] Using the training set and the learning rate, the regularization parameter alpha, and the maximum number of iterations to train the emotion recognition model corresponding to each genetic individual, outputting the predicted discrete emotions, and obtaining the trained emotion recognition model corresponding to each genetic individual when the loss function reaches a threshold or the iteration reaches a maximum number of iterations;

[0115] The test set is predicted using the trained emotion recognition model corresponding to each genetic individual, the accuracy is calculated based on the prediction result, and the inverse of the accuracy is used as the fitness value of each genetic individual.

[0116] It should be noted that the accuracy is the ratio of the number of correctly predicted samples to the total number of samples.

[0117] For example, the learning rate is set to 0.5 and alpha = 1×e -5 , the maximum number of iterations is 1000.

[0118] Preferably, in order to make the emotion recognition model converge quickly, the maximum number of iterations without improvement is set to 80. That is, if the accuracy does not improve after 80 iterations, the training is stopped.

[0119] Specifically, in step S304, the termination condition includes that the fitness value of a genetic individual in the current population is less than a threshold or the number of iterations reaches a maximum value.

[0120] Furthermore, if the termination condition is not reached, a pair of parent genetic individuals is established based on the population, and the second eigenvalue and the fourth eigenvalue of the parent genetic individual pair are exchanged according to the crossover probability to generate a new genetic individual; the first eigenvalue or the second eigenvalue is randomly selected to generate a random number between [1,4], and the selected eigenvalue plus the random number is mutated as a new eigenvalue according to the mutation probability to generate a new genetic individual; the fitness value of the new genetic individual is calculated; the genetic individuals in the excellent population are combined with the new genetic individuals, and are selected according to the number of genetic individuals in the population and the fitness values ​​of the genetic individuals in the combination to obtain an updated population.

[0121] Specifically, the establishing of parental genetic individual pairs based on population includes:

[0122] The first half of the genetic individuals in the sequence are selected to constitute the first parent generation, and the second half of the genetic individuals are selected to constitute the second parent generation;

[0123] The genetic individuals in the first parent generation and the second parent generation form several parent generation genetic individual pairs according to the sequence numbers.

[0124] It should be noted that the sorting is to sort all genetic individuals in the population from small to large according to their fitness values.

[0125] For example, assume that the number of genetic individuals in the population is 6, and the 6 genetic individuals are: [62,22,relu,sgd], [30,39,logistic,adam], [20,40,tanh,lbfgs], [8,14,identity,adam], [25,37,relu,lbfgs], [35,28,tanh,sgd], and the fitness values ​​of the 6 genetic individuals are: 0.725, 0.65, 0.50, 0.375, 0.63, and 0.60. The genetic individuals in the first parent generation are arranged in order: [62,22,relu,sgd], [30,39,logistic,adam], [25,37,relu,lbfgs], among which the sequence number of [62,22,relu,sgd] is 1; the genetic individuals in the second parent generation are arranged in order: [35,28,tanh,sgd], [20,40,tanh,lbfgs], [8,14,identity,adam]. The genetic individuals in the first and second parent generations are arranged in sequence to form three pairs of parent genetic individual pairs: [[62,22,relu,sgd]-[35,28,tanh,sgd]], [[30,39,logistic,adam]-[20,40,tanh,lbfgs]], and [[25,37,relu,lbfgs]-[8,14,identity,adam]].

[0126] Specifically, for each pair of parent genetic individuals, the second eigenvalue of the first parent is exchanged with the second eigenvalue of the second parent, and the fourth eigenvalue of the first parent is exchanged with the fourth eigenvalue of the second parent according to the crossover probability to generate a new genetic individual.

[0127] Exemplarily, the crossover operation is performed on the first pair of parent genetic individuals to generate new genetic individuals: [62, 28, relu, sgd], [35, 22, tanh, sgd].

[0128] Specifically, for each pair of parent genetic individuals, the selected characteristic value plus the random number is used as a new characteristic value for mutation according to the mutation probability to generate a new genetic individual.

[0129] For example, assuming that the first eigenvalue is randomly selected and a random number 3 is generated between [1,4], the above-mentioned first pair of parent genetic individuals is mutated to generate new genetic individuals: [65,22,relu,sgd], [38,28,tanh,sgd].

[0130] It should be noted that if the mutated eigenvalue exceeds the maximum value of the eigenvalue, the mutated eigenvalue is set to the maximum value of the eigenvalue.

[0131] Specifically, all genetic individuals in the combination are sorted from small to large according to their fitness values, and the first N genetic individuals whose number is the same as that of the genetic individuals in the population are selected from the sorting to obtain an updated population.

[0132] Furthermore, the four-dimensional brain map is input into a trained emotion recognition model to obtain one or more discrete emotions in a predicted discrete emotion set, wherein each predicted discrete emotion has attributes of pleasure and arousal.

[0133] Performance evaluation:

[0134] like Figure 4 As shown in Figure 2, the performance of the emotion recognition model and other common classifier models are evaluated from three aspects: accuracy, precision, and recall. Figure 4 It can be seen that the recognition results of other common classifier models for the 8 discrete emotions in the present invention are all less than 50%, while the recognition accuracy of the emotion recognition model provided by the present invention is 82.24%, which is much better than other models. This is because the emotional span of the video selected by the present invention is relatively small, which greatly increases the difficulty of emotion recognition, and it is difficult for existing models to accurately classify it.

[0135] like Figure 5 As shown in Figure 2, ablation experiments are conducted on different input features to evaluate the accuracy of different features in emotion recognition tasks. Figure 5 It can be seen that compared with using only a single time domain feature or frequency domain feature, the four-dimensional brain map combining frequency domain, time domain and spatial domain features has a higher accuracy of emotion recognition.

[0136] Embodiment 2:

[0137] Another embodiment of the present invention discloses an emotion system based on brain map features. Figure 6 As shown, the system comprises:

[0138] Video material database, used to provide videos for subjects to watch;

[0139] An EEG signal collector is used to collect EEG signals of the subject when watching the video and send them to the data processing module;

[0140] The data processing module is used to pre-process the received EEG signals, perform time domain segmentation, frequency domain division, feature extraction, and spatial domain mapping on the pre-processed EEG signals to obtain a two-dimensional map, perform frequency domain superposition and time domain superposition on the two-dimensional map to obtain a four-dimensional brain map, and send the four-dimensional brain map to the model processing module;

[0141] The model processing module is used to run the trained emotion recognition model and input the four-dimensional brain map into the trained emotion recognition model to obtain predicted emotions.

[0142] Compared with the prior art, the emotion recognition method based on brain map features provided by the present invention has the following beneficial effects:

[0143] 1. The present invention converts the EEG signal into a four-dimensional data block containing its frequency domain, spatial domain and time domain information through a four-dimensional brain map, retains the key emotional information in the EEG signal, and improves the accuracy of emotion recognition.

[0144] 2. The present invention collects videos with a small emotional span to construct a video material database, and collects EEG signals and subsequently recognizes emotions based on the video material database, thereby solving the problem of achieving fine-grained and accurate recognition of corresponding emotions for materials with a small emotional span based on EEG signals, and providing a basis for improving the effectiveness of media communication.

[0145] 3. The present invention improves the accuracy of emotion recognition by extracting differential entropy and power spectral density of EEG signals as features.

[0146] 4. The present invention divides the emotions that appear when the subjects watch the videos in the video material database, constructs a discrete emotion set, and covers complex emotions intertwined with multiple emotions, so that the subjects can express their emotional feelings more accurately, providing a basis for achieving fine-grained and accurate recognition of emotions.

[0147] 5. The present invention determines the parameters of the emotion recognition model through a genetic algorithm, thereby obtaining a trained emotion recognition model, optimizing the parameters of the model, and improving the accuracy of emotion recognition.

[0148] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0149] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. An emotion recognition method based on brain map features, characterized in that: The method comprises the following steps: Building a video material database, and collecting EEG signals of subjects when they watch videos in the video material database; Preprocessing the EEG signal, performing time domain segmentation, frequency domain division, feature extraction, and spatial domain mapping on the preprocessed EEG signal to obtain a two-dimensional map, and performing frequency domain superposition and time domain superposition on the two-dimensional map to obtain a four-dimensional brain map; The four-dimensional brain map is input into a trained emotion recognition model to obtain predicted emotions.

2. The emotion recognition method based on brain map features according to claim 1 is characterized in that: The method of performing time domain segmentation, frequency domain division, feature extraction, and spatial domain mapping on the preprocessed EEG signal to obtain a two-dimensional spectrum includes: segmenting the preprocessed EEG signal into a series of non-overlapping time segments with a length of Ts in the time domain; dividing each time segment into four frequency bands of alpha, beta, gamma, and theta in the frequency domain; extracting the differential entropy and power spectral density of the EEG signal for each frequency band; and mapping the corresponding differential entropy and power spectral density of each frequency band to a two-dimensional grid corresponding to the position of the EEG signal electrode for spatial domain mapping to obtain a two-dimensional spectrum corresponding to each frequency band.

3. The emotion recognition method based on brain map features according to claim 2 is characterized in that: The two-dimensional maps are superimposed in the frequency domain and the time domain to obtain a four-dimensional brain map, including: superimposing the two-dimensional maps corresponding to each frequency band, and then superimposing each time segment to obtain a four-dimensional brain map.

4. The emotion recognition method based on brain map features according to claim 2 is characterized in that: The two-dimensional grid is an 8×9 spatial layout, in which each grid point represents the position of an EEG signal electrode that meets the international 10-20 standard; there are 2 two-dimensional spectra corresponding to each frequency band, including a differential entropy two-dimensional spectrum and a power spectrum density two-dimensional spectrum; each two-dimensional spectrum represents an 8×9 data block.

5. The emotion recognition method based on brain map features according to claim 4 is characterized in that: The eight two-dimensional maps corresponding to the four frequency bands are superimposed to obtain an 8×9×8 data block; then the time segments are superimposed to obtain a four-dimensional brain map, which represents a data blocks, where T represents the total duration of the preprocessed EEG signal, Ts is the length of the time segment, Represents the number of time slices.

6. The emotion recognition method based on brain map features according to claim 1, characterized in that: The preprocessing of the EEG signal comprises: The EEG signals corresponding to each subject watching the video are extracted through bandpass filtering and downsampled; Extract bad electrodes and interpolate the downsampled EEG signals; The interpolated EEG signals were de-artifacted and re-referenced by whole-brain average; The EEG signal after re-reference is intercepted.

7. The emotion recognition method based on brain map features according to claim 1, characterized in that: The trained emotion recognition model is a multi-layer perceptron.

8. The emotion recognition method based on brain map features according to claim 7, characterized in that: The parameters of the emotion recognition model are determined based on a genetic algorithm to obtain the trained emotion recognition model.

9. The emotion recognition method based on brain map features according to claim 2, characterized in that: The differential entropy is obtained by the following formula: Among them, h(x i ) represents x i The differential entropy of x i represents the EEG signal corresponding to the ith frequency band, σ 2 Represents x i The variance of , e represents the base of the natural logarithm; The energy is obtained by the following formula: Where E represents x i The energy of ij represents the EEG signal corresponding to the jth sliding time window in the ith frequency band, and N represents the number of sliding time windows.

10. An emotion recognition system based on brain map features, characterized in that: The system comprises: Video material database, used to provide videos for subjects to watch; An EEG signal collector is used to collect EEG signals of the subject when watching the video and send them to the data processing module; The data processing module is used to pre-process the received EEG signals, perform time domain segmentation, frequency domain division, feature extraction, and spatial domain mapping on the pre-processed EEG signals to obtain a two-dimensional map, perform frequency domain superposition and time domain superposition on the two-dimensional map to obtain a four-dimensional brain map, and send the four-dimensional brain map to the model processing module; The model processing module is used to run the trained emotion recognition model and input the four-dimensional brain map into the trained emotion recognition model to obtain predicted emotions.

Images