Emotion recognition method of EEG signals based on brain network and multi-scale permutation entropy

Abstract

The invention provides an electroencephalogram (EEG) signal emotion recognition method based on brain network and multi-scale permutation entropy, which removes artifacts from acquired EEG signals and performs downsampling; decomposes the downsampled EEG signals by using wavelet packet transform, and constructs a brain network by using mutual information and minimum spanning tree according to the decomposition result, and extracts brain network features; performs modal and frequency band decomposition on the downsampled EEG signals by using fast variational modal decomposition and wavelet packet transform, and optimizes parameter selection of multi-scale permutation entropy by using genetic algorithm to obtain nonlinear multi-scale permutation entropy features; performs feature fusion of the extracted brain network features and the nonlinear multi-scale permutation entropy features to obtain fused feature vectors; obtains an emotion state recognition result according to the fused feature vectors and a random forest classification model; the invention combines the fusion of brain network features and multi-scale permutation entropy to improve the speed and accuracy of EEG signal emotion recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to an electroencephalogram (EEG) signal emotion recognition method based on brain network and multi-scale permutation entropy. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] Emotions are closely related to people's lives and have always been the focus of research in disciplines such as psychology, physiology and cognitive science. The generation of emotions is a very complex process, which is affected by cognitive level and the surrounding environment. At the same time, emotions in turn affect people's cognition and decision-making. In recent years, emotion recognition methods have emerged one after another, such as facial expression recognition, voice intonation recognition, behavior recognition, text recognition and electroencephalogram (EEG) signal recognition. EEG is a technology that detects electrical signals inside the brain and can directly reflect the dynamic changes of the central nervous system. Compared with other non-physiological signals, EEG signals have high resolution and are difficult to hide, so they have gradually become a research hotspot in the field of emotion recognition.

[0004] One of the research goals of emotion recognition is to extract features that represent relevant emotions and train classifiers to obtain higher emotion recognition accuracy; the second goal is to analyze the relevant brain regions that generate emotions and the characteristics of information interaction between brain regions.

[0005] The inventors found that the prior art has the following technical problems:

[0006] (1) Some researchers have combined relevant theoretical research on different brain regions and started to construct brain networks from the synergistic relationship between multiple channels. They have also combined relevant knowledge of graph theory to extract brain network related features for emotion recognition. However, there is a lot of information redundancy in the current brain network construction process, which will increase the overall computational complexity and have a certain impact on the accuracy of emotion recognition.

[0007] (2) For the decomposition of EEG signals, Dragomiretskiy et al. proposed variational mode decomposition (VMD) which can adaptively process nonlinear signals. However, the speed of signal mode decomposition can be further improved. Multi-scale permutation entropy (MPE) is a nonlinear dynamic feature that can better reflect the changes in EEG signals. However, MPE has multiple parameters. In actual applications, the quality of parameter adjustment affects the accuracy of the final emotion recognition, and the cumbersome parameter adjustment process seriously affects the efficiency. Summary of the invention

[0008] In order to address the deficiencies of the prior art, the present invention provides an EEG signal emotion recognition method based on brain network and multi-scale permutation entropy, which combines the fusion of brain network features and multi-scale permutation entropy to improve the speed and accuracy of EEG signal emotion recognition.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] A first aspect of the present invention provides an EEG signal emotion recognition method based on brain network and multi-scale permutation entropy.

[0011] A method for emotion recognition of EEG signals based on brain network and multi-scale permutation entropy includes the following processes:

[0012] Remove artifacts and downsample the acquired EEG signals;

[0013] The downsampled EEG signals were decomposed using wavelet packet transform. Based on the decomposition results, the brain network was constructed using mutual information and minimum spanning tree to extract brain network features.

[0014] Fast variational mode decomposition and wavelet packet transform are used to decompose the modal and frequency band of the downsampled EEG signal, and the genetic algorithm is used to optimize the parameter selection of multi-scale permutation entropy to obtain the nonlinear multi-scale permutation entropy feature.

[0015] The extracted brain network features are fused with the nonlinear multi-scale permutation entropy features to obtain the fused feature vector;

[0016] Based on the fused feature vector and random forest classification model, the emotional state recognition result is obtained.

[0017] A second aspect of the present invention provides an EEG signal emotion recognition system based on brain network and multi-scale permutation entropy.

[0018] An EEG signal emotion recognition system based on brain network and multi-scale permutation entropy, comprising:

[0019] The data acquisition module is configured to: remove artifacts and downsample the acquired EEG signal;

[0020] The brain network feature extraction module is configured to: decompose the downsampled EEG signal using wavelet packet transform, construct a brain network based on the decomposition result using mutual information and minimum spanning tree, and extract brain network features;

[0021] The nonlinear multi-scale permutation entropy extraction module is configured to: use fast variational modal decomposition and wavelet packet transform to perform modal and frequency band decomposition on the downsampled EEG signal, and use genetic algorithm to optimize the parameter selection of multi-scale permutation entropy to obtain nonlinear multi-scale permutation entropy features;

[0022] The feature fusion module is configured to: fuse the extracted brain network features with the nonlinear multi-scale permutation entropy features to obtain a fused feature vector;

[0023] The emotion recognition module is configured to obtain an emotion state recognition result based on the fused feature vector and the random forest classification model.

[0024] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the method for EEG signal emotion recognition based on brain network and multi-scale permutation entropy as described in the first aspect of the present invention.

[0025] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for EEG signal emotion recognition based on brain network and multi-scale permutation entropy as described in the first aspect of the present invention are implemented.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. The method described in the present invention includes five parts: an EEG signal preprocessing part, a brain network feature extraction part, a nonlinear MPE feature extraction part, a feature fusion part and an emotion recognition part. Through analysis, it is found that there is a large amount of information redundancy in the construction process of the brain network, which will increase the overall calculation complexity. The present invention adopts the minimum spanning method to remove redundant information and construct a brain network, and proposes a construction and feature extraction method of a brain network based on a minimum spanning tree, which automatically extracts seven emotion-related brain network features such as leaf score, intermediate coreness, tree hierarchy, feature path length, degree, global efficiency, and eccentricity; in view of the long calculation time of traditional variational mode decomposition and the optimization problem of nonlinear MPE feature parameters, a feature extraction method based on the combination of fast variational mode decomposition and genetic algorithm optimization of MPE parameters is proposed, which speeds up the mode decomposition speed, optimizes parameters more conveniently and efficiently, and extracts nonlinear MPE features.

[0028] 2. The present invention removes artifacts such as eye movements from the extracted EEG signals and downsamples them to 128Hz according to relevant standards for artifact removal; brain network feature extraction is to construct a minimum spanning tree brain network with the preprocessed signals, and extract seven brain network features, including leaf score, intermediate coreness, tree hierarchy, characteristic path length, degree, global efficiency, and eccentricity; MPE feature extraction is to decompose the EEG signals in mode and frequency band using fast variational mode decomposition and wavelet packet transform, and optimize the parameter selection process of MPE using genetic algorithm, thereby extracting emotion-related nonlinear MPE features; the feature fusion part is to fuse the seven features extracted from the minimum spanning tree brain network with the MPE features to obtain a fused feature vector; the emotion recognition part is to obtain the emotion state recognition result based on the fused feature vector and the random forest classification model, thereby improving the recognition accuracy and speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0030] Figure 1 This is a flow chart of the emotion recognition method based on brain network and multi-scale permutation entropy provided in Example 1 of the present invention.

[0031] Figure 2 Schematic diagram of a six-layer wavelet packet transform with a sampling frequency of 128 Hz provided in Embodiment 1 of the present invention.

[0032] Figure 3 A schematic diagram of a brain network based on a minimum spanning tree provided in Example 1 of the present invention.

[0033] Figure 4 A schematic diagram of a random forest provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0037] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0038] Embodiment 1:

[0039] like Figure 1 As shown, Embodiment 1 of the present invention provides an EEG signal emotion recognition method based on brain network and multi-scale permutation entropy, comprising the following process:

[0040] Remove artifacts and downsample the acquired EEG signals;

[0041] The downsampled EEG signals were decomposed using wavelet packet transform. Based on the decomposition results, the brain network was constructed using mutual information and minimum spanning tree to extract brain network features.

[0042] Fast variational mode decomposition and wavelet packet transform are used to decompose the modal and frequency band of the downsampled EEG signal, and the genetic algorithm is used to optimize the parameter selection of multi-scale permutation entropy to obtain the nonlinear multi-scale permutation entropy feature.

[0043] The extracted brain network features are fused with the nonlinear multi-scale permutation entropy features to obtain the fused feature vector;

[0044] Based on the fused feature vector and random forest classification model, the emotional state recognition result is obtained.

[0045] Specifically, they include:

[0046] S1: Remove artifacts such as eye movements from the original EEG signal and perform downsampling;

[0047] S2: The preprocessed signal is decomposed into four frequency bands: Theta, Alpha, Beta, and Gamma by wavelet packet transform. The minimum spanning tree brain network is constructed and the brain network features are extracted on each frequency band. Seven brain network features, including leaf score, intermediate coreness, tree level, characteristic path length, degree, global efficiency, and eccentricity, are obtained to form the feature vector x1.

[0048] S3: Fast variational modal decomposition and wavelet packet transform are used to decompose the EEG signal into modal and frequency bands, and the genetic algorithm is used to optimize the MPE parameter selection process, thereby extracting nonlinear MPE features to form a feature vector x2;

[0049] S4: Linearly fuse the feature vector x1 of the brain network feature with the feature vector x2 of the nonlinear MPE feature to obtain the feature vector x, that is, x = (x1, x2);

[0050] S5: Input the fused feature vector x into the random forest classifier to identify the emotional state.

[0051] In S1, this embodiment uses the public multimodal emotion dataset DEAP. The DEAP dataset contains data from 40 experiments of 32 subjects each. The acquisition time of the EEG signal for each experiment is 63 seconds, of which the first 3 seconds are resting state data and the last 60 seconds are emotion-related data. In order to obtain the data segment most relevant to emotions, the last 60 seconds of emotion data are divided into three equal parts, and the second 20 seconds of data are intercepted. The intercepted 20 seconds of EEG data are then divided into two parts, and the first 10 seconds and the last 10 seconds are used to construct a brain network based on the minimum spanning tree in step S2.

[0052] In S2, wavelet packet transform is used to decompose the preprocessed EEG data into four frequency bands: Theta, Alpha, Beta, and Gamma. The process of 6-layer wavelet packet decomposition is as follows: Figure 2 As shown. Mutual information is used to construct the connectivity matrix on the four frequency bands. The calculation formula of mutual information is as follows:

[0053]

[0054] Among them, P ij represents the joint probability density of two EEG data, P i represents the probability density.

[0055] In order to remove redundant information in the connectivity matrix, we first take the inverse of each value in the connectivity matrix, and then use the Prim minimum spanning tree algorithm to retain the maximum weight edge connecting all 32 channel nodes, that is, to construct a brain network based on the minimum spanning tree. Figure 3 As shown in Figure 2, seven brain network features are extracted from the minimum spanning tree brain network: leaf score, intermediate coreness, tree hierarchy, characteristic path length, degree, global efficiency, and eccentricity.

[0056] The brain network based on the minimum spanning tree can measure the coupling relationship between EEG channel pairs, and the features extracted from the brain network can measure the topological structure of the brain network.

[0057] In S3, fast variational modal decomposition and wavelet packet transform are used to decompose the EEG signal into modal and frequency bands. Fast variational modal decomposition is faster than variational modal decomposition. The specific steps are as follows:

[0058] S3.1: Initialize single-component AM / FM signal Center frequency {ω k}, Lagrange multiplier The number of iterations n, t0 is 1;

[0059] S3.2: Update

[0060]

[0061] S3.3: Update ω k ;

[0062]

[0063] S3.4: First update

[0064]

[0065] S3.5: Update iterative operator t:

[0066]

[0067] S3.6: Second update

[0068]

[0069] S3.7: Repeat S3.2 to S3.6 until the following termination condition is met.

[0070]

[0071] Genetic algorithm is an optimal solution adaptive search algorithm. It draws on the natural selection and inheritance process of organisms in nature and has good parameter search ability. Genetic algorithm is used to optimize the parameter selection process of MPE, and then nonlinear MPE features are extracted. The fitness function of genetic algorithm is set as follows:

[0072]

[0073] Among them, Ske represents skewness, and its calculation formula is as follows:

[0074]

[0075] Among them, the sequence H P (X) = {H P(1),H P (2),…,H P (s)} is composed of the MPE of the EEG signal sequence X = {x(i), i = 1, 2, …, N}.

[0076] In S4, the feature vector x1 of the brain network feature is linearly fused with the feature vector x2 of the nonlinear MPE feature to obtain the feature vector x, i.e., x = (x1, x2);

[0077] In S5, the fused feature vector x is input into the random forest classifier to identify the emotional state. Figure 4 As shown in the figure, in the random forest classifier, multiple training samples are extracted from the original samples by resampling, so only fewer training samples are needed to obtain a higher accuracy.

[0078] Embodiment 2:

[0079] Embodiment 2 of the present invention provides an EEG signal emotion recognition system based on brain network and multi-scale permutation entropy, comprising:

[0080] The data acquisition module is configured to: remove artifacts and downsample the acquired EEG signal;

[0081] The brain network feature extraction module is configured to: decompose the downsampled EEG signal using wavelet packet transform, construct a brain network based on the decomposition result using mutual information and minimum spanning tree, and extract brain network features;

[0082] The nonlinear multi-scale permutation entropy extraction module is configured to: use fast variational modal decomposition and wavelet packet transform to perform modal and frequency band decomposition on the downsampled EEG signal, and use genetic algorithm to optimize the parameter selection of multi-scale permutation entropy to obtain nonlinear multi-scale permutation entropy features;

[0083] The feature fusion module is configured to: fuse the extracted brain network features with the nonlinear multi-scale permutation entropy features to obtain a fused feature vector;

[0084] The emotion recognition module is configured to obtain an emotion state recognition result based on the fused feature vector and the random forest classification model.

[0085] The working method of the system is the same as the EEG signal emotion recognition method based on brain network and multi-scale permutation entropy provided in Example 1, and will not be repeated here.

[0086] Embodiment 3:

[0087] Embodiment 3 of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the method for emotion recognition of EEG signals based on brain network and multi-scale permutation entropy as described in Embodiment 1 of the present invention.

[0088] Embodiment 4:

[0089] Embodiment 4 of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for emotion recognition of electroencephalogram signals based on brain network and multi-scale permutation entropy as described in Embodiment 1 of the present invention are implemented.

[0090] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0091] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for emotion recognition of EEG signals based on brain network and multi-scale permutation entropy, characterized by: The process includes: Remove artifacts and downsample the acquired EEG signals; The downsampled EEG signal is decomposed by wavelet packet transform. According to the decomposition results, the brain network is constructed by using mutual information and minimum spanning tree to extract brain network features; including: The connectivity matrix is ​​constructed using mutual information on the four frequency bands obtained by wavelet packet transform decomposition; Take the inverse of each value in the connectivity matrix; The Prim minimum spanning tree algorithm is used to retain the maximum weight edge connecting all channel nodes to obtain a brain network based on the minimum spanning tree; Fast variational mode decomposition and wavelet packet transform are used to decompose the modal and frequency band of the downsampled EEG signal, and the genetic algorithm is used to optimize the parameter selection of multi-scale permutation entropy to obtain the nonlinear multi-scale permutation entropy feature. The extracted brain network features are fused with the nonlinear multi-scale permutation entropy features to obtain the fused feature vector; Based on the fused feature vector and random forest classification model, the emotional state recognition result is obtained.

2. The method for EEG signal emotion recognition based on brain network and multi-scale permutation entropy as claimed in claim 1, characterized in that: The decomposition results include four frequency bands: Theta, Alpha, Beta and Gamma.

3. The method for EEG signal emotion recognition based on brain network and multi-scale permutation entropy as claimed in claim 1, characterized in that: Brain network features include: leaf fraction, betweenness coreness, tree hierarchy, characteristic path length, degree, global efficiency, and eccentricity.

4. The method for EEG signal emotion recognition based on brain network and multi-scale permutation entropy as claimed in claim 1, characterized in that: The fitness function of the genetic algorithm is 1 and Ske 2 +1, where Ske is the skewness.

5. The method for EEG signal emotion recognition based on brain network and multi-scale permutation entropy as claimed in claim 1, characterized in that: The extracted brain network features are linearly fused with the nonlinear multi-scale permutation entropy features.

6. An EEG signal emotion recognition system based on brain network and multi-scale permutation entropy, characterized by: include: The data acquisition module is configured to: remove artifacts and downsample the acquired EEG signal; The brain network feature extraction module is configured to: decompose the downsampled EEG signal using wavelet packet transform, construct a brain network based on the decomposition result using mutual information and minimum spanning tree, and extract brain network features; including: The connectivity matrix is ​​constructed using mutual information on the four frequency bands obtained by wavelet packet transform decomposition; Take the inverse of each value in the connectivity matrix; The Prim minimum spanning tree algorithm is used to retain the maximum weight edge connecting all channel nodes to obtain a brain network based on the minimum spanning tree; The nonlinear multi-scale permutation entropy extraction module is configured to: use fast variational modal decomposition and wavelet packet transform to perform modal and frequency band decomposition on the downsampled EEG signal, and use genetic algorithm to optimize the parameter selection of multi-scale permutation entropy to obtain nonlinear multi-scale permutation entropy features; The feature fusion module is configured to: fuse the extracted brain network features with the nonlinear multi-scale permutation entropy features to obtain a fused feature vector; The emotion recognition module is configured to obtain an emotion state recognition result based on the fused feature vector and the random forest classification model.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method for EEG signal emotion recognition based on brain network and multi-scale permutation entropy as described in any one of claims 1 to 5 are implemented.

8. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the method for EEG signal emotion recognition based on brain network and multi-scale permutation entropy as described in any one of claims 1 to 5 are implemented.

Images