A residual feature pyramid emotion recognition method and system based on electroencephalogram signals

Abstract

The present invention discloses a residual feature pyramid emotion recognition method and system based on electroencephalogram signals, which relates to the technical field of neural networks. The method includes: obtaining electroencephalogram data of a test subject collected by electroencephalogram electrodes placed according to the international 10-20 system; constructing a feature matrix of the electroencephalogram data of the test subject, and fusing the feature matrix to obtain a fusion matrix; constructing an emotion recognition model based on a residual bidirectional feature pyramid; the emotion recognition model includes a spatial-to-depth structure, a residual structure, and a bidirectional feature pyramid structure; inputting the fusion matrix into the trained emotion recognition model to obtain the emotion category of the test subject. The present invention can capture spatial electroencephalogram representation information and electroencephalogram context information, reduce the computational amount during model training, and fuse high-level semantic information and low-level spatial information to overcome the problem of network degradation caused by the increase in network depth.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and particularly to a residual feature pyramid emotion recognition method and system based on electroencephalogram signals. Background Art

[0002] Emotion recognition aims to establish a harmonious human-computer environment by endowing computers with the ability to recognize, understand, and adapt to human emotions. With the rapid development of technologies such as non-invasive sensing technology, machine learning algorithms, and computer computing power, the progress of cognitive science has been driven, and emotion recognition, as a frontier field of cognitive science, has received the attention of many scholars. Although more and more emotion recognition methods have been proposed, there are still two problems at present. One is that time-domain, frequency-domain, or time-frequency domain features only consider the context relevance of emotions within a certain period of time and ignore the spatial representation features of electroencephalogram signals; while spatial domain features ignore the context relevance of emotional states; the other is that as the number of network layers continues to deepen, the computational amount required by the model also increases relatively. In addition, blindly reducing features will also ignore some electroencephalogram representation feature information to a certain extent. Therefore, there is an urgent need for an emotion recognition method that can capture spatial electroencephalogram representation information and electroencephalogram context information, reduce the computational amount, and fuse high-level semantic information with low-level spatial information to overcome the problem of network degradation caused by the increase in network depth. Summary of the Invention

[0003] The purpose of the present invention is to provide a residual feature pyramid emotion recognition method and system based on electroencephalogram signals, which can capture spatial electroencephalogram representation information and electroencephalogram context information, reduce the computational amount during model training, and fuse high-level semantic information with low-level spatial information to overcome the problem of network degradation caused by the increase in network depth.

[0004] To achieve the above purpose, the present invention provides the following solutions:

[0005] A residual feature pyramid emotion recognition method based on electroencephalogram signals, the method includes:

[0006] Obtain the electroencephalogram data of the test subject collected by electroencephalogram electrodes placed according to the international 10-20 system;

[0007] Construct a feature matrix of the electroencephalogram data of the test subject, and fuse the feature matrix to obtain a fusion matrix; the feature matrix includes an initial matrix, a left and right brain region symmetric difference matrix, a left and right brain region symmetric quotient matrix, and a differential entropy matrix;

[0008] Construct an emotion recognition model based on a residual bidirectional feature pyramid; the emotion recognition model includes a spatial-to-depth structure, a residual structure, and a bidirectional feature pyramid structure;

[0009] Input the fusion matrix into the trained emotion recognition model to obtain the emotion category of the person to be tested; the emotion category includes happy, angry, motivated, sad, calm, and fearful.

[0010] Optionally, the training process of the emotion recognition model includes:

[0011] Obtain the electroencephalogram (EEG) data of the subject collected by EEG electrodes placed according to the international 10 - 20 system;

[0012] Construct a feature matrix of the EEG data of the subject;

[0013] Fuse the feature matrix of the EEG data of the subject to obtain a fusion matrix of the EEG data of the subject;

[0014] Use the fusion matrix of the EEG data of the subject as the input to train the emotion recognition model to obtain a trained emotion recognition model.

[0015] Optionally, the construction of the feature matrix of the EEG data of the person to be tested specifically includes:

[0016] Construct an initial one - dimensional vector according to the number of electrode channels and sampling frequency of the international 10 - 20 system;

[0017] Construct an initial two - dimensional matrix according to the initial one - dimensional vector and the electrode positions of the international 10 - 20 system;

[0018] Normalize the initial two - dimensional matrix to obtain an initial matrix;

[0019] Exclude the elements in the initial one - dimensional vector corresponding to the electrodes located on the anterior - posterior sagittal line in the international 10 - 20 system to obtain a one - dimensional vector after exclusion;

[0020] Renumber the elements in the one - dimensional vector after exclusion to obtain a renumbered one - dimensional vector;

[0021] Determine the element numbers in the renumbered one - dimensional vector and the symmetric element numbers symmetric to the element numbers;

[0022] Take the difference between the elements corresponding to the element numbers and the elements corresponding to the symmetric element numbers to obtain a symmetric difference one - dimensional vector;

[0023] Construct a symmetric difference two - dimensional matrix according to the symmetric difference one - dimensional vector and the electrode positions of the international 10 - 20 system;

[0024] Normalize the symmetric difference two - dimensional matrix to obtain a left - right brain region symmetric difference matrix;

[0025] Divide the element corresponding to the element number by the element corresponding to the symmetric element number to obtain a one-dimensional symmetric quotient vector;

[0026] Construct a two-dimensional symmetric quotient matrix based on the one-dimensional symmetric quotient vector and the electrode positions of the international 10-20 system;

[0027] Normalize the two-dimensional symmetric quotient matrix to obtain a left-right brain region symmetric quotient matrix;

[0028] Extract the differential entropy features of a set frequency band from the initial one-dimensional vector using a set time window to obtain a differential entropy feature vector;

[0029] Construct a differential entropy matrix based on the differential entropy feature vector and the electrode positions of the international 10-20 system.

[0030] Optionally, the fusing of the feature matrix to obtain a fused matrix specifically includes:

[0031] Apply a convolution operation to the feature matrix for preprocessing to obtain an initial feature map;

[0032] Normalize the initial feature map to obtain a normalized feature map;

[0033] Apply an activation function to the normalized feature map to obtain a feature map;

[0034] Perform nearest neighbor upsampling on the feature map to obtain a feature matrix of the feature map;

[0035] Fuse the feature matrix of the feature map to obtain a fused matrix.

[0036] Optionally, the inputting of the fused matrix into a trained emotion recognition model to obtain the emotion category of the test subject specifically includes:

[0037] Use a spatial-to-depth structure to sample and reorganize the fused matrix at a fixed scale to obtain multiple initial fused matrices;

[0038] Use a convolution operation to unify the number of channels of the multiple initial fused matrices to obtain multiple matrices with unified channel numbers;

[0039] Use a bidirectional feature pyramid structure and a residual structure to extract the emotion category features of the multiple matrices with unified channel numbers;

[0040] Apply a fully connected layer to classify the emotion category features and output the emotion category.

[0041] A residual feature pyramid emotion recognition system based on electroencephalogram signals, applied to the above-mentioned residual feature pyramid emotion recognition method based on electroencephalogram signals, the system includes:

[0042] A data acquisition module, configured to acquire electroencephalogram data of a subject to be tested collected by electroencephalogram electrodes placed according to the international 10-20 system;

[0043] A feature matrix determination module, configured to construct a feature matrix of the electroencephalogram data of the subject to be tested; the feature matrix includes an initial matrix, a left-right brain region symmetry difference matrix, a left-right brain region symmetry quotient matrix, and a differential entropy matrix;

[0044] A fusion matrix determination module, configured to fuse the feature matrix to obtain a fusion matrix;

[0045] A model construction module, configured to construct an emotion recognition model based on a residual bidirectional feature pyramid; the emotion recognition model includes a spatial-to-depth structure, a residual structure, and a bidirectional feature pyramid structure;

[0046] An emotion category determination module, configured to input the fusion matrix into a trained emotion recognition model to obtain the emotion category of the subject to be tested; the emotion category includes happy, angry, excited, sad, calm, and fearful.

[0047] Optionally, the system further includes a training module;

[0048] The training module includes:

[0049] An electroencephalogram data acquisition sub-module, configured to acquire electroencephalogram data of a subject collected by electroencephalogram electrodes placed according to the international 10-20 system;

[0050] A construction sub-module, configured to construct a feature matrix of the electroencephalogram data of the subject;

[0051] A fusion sub-module, configured to fuse the feature matrix of the electroencephalogram data of the subject to obtain a fusion matrix of the electroencephalogram data of the subject;

[0052] A training sub-module, configured to use the fusion matrix of the electroencephalogram data of the subject as an input to train the emotion recognition model to obtain a trained emotion recognition model.

[0053] Optionally, the feature matrix determination module includes:

[0054] An initial one-dimensional vector construction sub-module, configured to construct an initial one-dimensional vector according to the number of electrode channels and the sampling frequency of the international 10-20 system;

[0055] An initial two-dimensional matrix construction sub-module, configured to construct an initial two-dimensional matrix according to the initial one-dimensional vector and the electrode positions of the international 10-20 system;

[0056] An initial matrix determination sub-module, configured to standardize the initial two-dimensional matrix to obtain an initial matrix;

[0057] A rejection sub-module, configured to reject the elements in the initial one-dimensional vector corresponding to the electrodes located on the anterior-posterior sagittal line in the international 10-20 system, to obtain a one-dimensional vector after rejection;

[0058] A re-numbering sub-module, configured to re-number the elements in the one-dimensional vector after rejection, to obtain a re-numbered one-dimensional vector;

[0059] An element number determination sub-module, configured to determine the element numbers in the re-numbered one-dimensional vector and the symmetric element numbers symmetric to the element numbers;

[0060] A symmetric difference one-dimensional vector determination sub-module, configured to take the difference between the elements corresponding to the element numbers and the elements corresponding to the symmetric element numbers, to obtain a symmetric difference one-dimensional vector;

[0061] A symmetric difference two-dimensional matrix determination sub-module, configured to construct a symmetric difference two-dimensional matrix according to the symmetric difference one-dimensional vector and the electrode positions in the international 10-20 system;

[0062] A symmetric difference matrix determination sub-module, configured to standardize the symmetric difference two-dimensional matrix to obtain a left-right brain region symmetric difference matrix;

[0063] A symmetric quotient one-dimensional vector determination sub-module, configured to divide the elements corresponding to the element numbers by the elements corresponding to the symmetric element numbers, to obtain a symmetric quotient one-dimensional vector;

[0064] A symmetric quotient two-dimensional matrix determination sub-module, configured to construct a symmetric quotient two-dimensional matrix according to the symmetric quotient one-dimensional vector and the electrode positions in the international 10-20 system;

[0065] A symmetric quotient matrix determination sub-module, configured to standardize the symmetric quotient two-dimensional matrix to obtain a left-right brain region symmetric quotient matrix;

[0066] A differential entropy feature vector determination sub-module, configured to extract the differential entropy features of a set frequency band from the initial one-dimensional vector by using a set time window, to obtain a differential entropy feature vector;

[0067] A differential entropy matrix determination sub-module, configured to construct a differential entropy matrix according to the differential entropy feature vector and the electrode positions in the international 10-20 system.

[0068] Optionally, the fusion matrix determination module includes:

[0069] An initial feature map determination sub-module, configured to perform preprocessing on the feature matrix of the electroencephalogram signal by applying a convolution operation, to obtain an initial feature map;

[0070] A normalized feature map determination sub-module, which is used to normalize the initial feature map to obtain a normalized feature map;

[0071] A feature map determination sub-module, which is used to apply an activation function to the normalized feature map to obtain a feature map;

[0072] A feature matrix determination sub-module, which is used to perform nearest neighbor upsampling on the feature map to obtain a feature matrix of the feature map;

[0073] A fusion matrix determination sub-module, which is used to fuse the feature matrix of the feature map to obtain a fusion matrix.

[0074] Optionally, the emotion category determination module includes:

[0075] An initial fusion matrix determination unit, which is used to utilize a space-to-depth structure to sample and reorganize the fusion matrix at a fixed scale to obtain a plurality of initial fusion matrices;

[0076] A unified channel number matrix determination unit, which is used to unify the channel numbers of the plurality of initial fusion matrices by using a convolution operation to obtain a plurality of unified channel number matrices;

[0077] An extraction unit, which is used to extract the emotion category features of the plurality of unified channel number matrices by using a bidirectional feature pyramid structure and a residual structure;

[0078] A result output unit, which is used to classify the emotion category features by applying a fully connected layer and output the emotion category.

[0079] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0080] A residual feature pyramid emotion recognition method based on electroencephalogram (EEG) signals provided by the present invention includes: acquiring EEG data of a test subject collected by EEG electrodes placed according to the international 10-20 system; constructing a feature matrix of the EEG data of the test subject and fusing the feature matrix to obtain a fusion matrix; the feature matrix includes an initial matrix, a left and right brain region symmetric difference matrix, a left and right brain region symmetric quotient matrix, and a differential entropy matrix; constructing an emotion recognition model based on a residual bidirectional feature pyramid; the emotion recognition model includes a spatial-to-depth structure, a residual structure, and a bidirectional feature pyramid structure; inputting the fusion matrix into the trained emotion recognition model to obtain the emotion category of the test subject; the emotion category includes happiness, anger, excitation, sadness, calmness, and fear. The present invention proposes an emotion recognition model based on a residual bidirectional feature pyramid. By constructing four different matrices according to the original signals of sample points and fusing the feature layers of the four matrices, it can capture spatial EEG representation information and EEG context information; by using a spatial-to-depth structure instead of the traditional convolutional network structure as the backbone network, it can reduce the computational amount during model training, and by adding a residual structure on the basis of the bidirectional feature pyramid structure, it can fuse high-level semantic information and low-level spatial information, overcoming the problem of network degradation caused by the increase in network depth. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0082] Figure 1 It is a flowchart of the residual feature pyramid emotion recognition method based on EEG signals provided by the present invention;

[0083] Figure 2 It is an overall framework diagram of the residual feature pyramid emotion recognition method based on EEG signals provided by the present invention;

[0084] Figure 3 It is a schematic diagram of electrode placement of the international 10-20 system;

[0085] Figure 4 It is a schematic diagram of the S2D structure provided by the present invention;

[0086] Figure 5 It is a schematic diagram of the cross-scale fusion part of the residual feature pyramid provided by the present invention;

[0087] Figure 6 It is a module diagram of the residual feature pyramid emotion recognition system based on EEG signals provided by the present invention.

[0088] Symbol Explanation:

[0089] 1 - Data acquisition module, 2 - Feature matrix determination module, 3 - Fusion matrix determination module, 4 - Model construction module, 5 - Emotion category determination module. Specific Embodiment

[0090] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0091] The purpose of the present invention is to provide a residual feature pyramid emotion recognition method and system based on electroencephalogram signals, which can capture spatial electroencephalogram representation information and electroencephalogram context information, reduce the computational amount during model training, and fuse high-level semantic information and low-level spatial information to overcome the problem of network degradation with the increase in network depth.

[0092] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0093] As Figure 1 and Figure 2 shown, the residual feature pyramid emotion recognition method based on electroencephalogram signals provided by the present invention includes:

[0094] Step S1: Obtain the electroencephalogram data of the test subject collected by electroencephalogram electrodes placed according to the international 10 - 20 system; specifically, the electrode placement method of the international 10 - 20 system is as Figure 3 shown.

[0095] Step S2: Construct a feature matrix of the electroencephalogram data of the test subject, and fuse the feature matrix to obtain a fusion matrix; the feature matrix includes an initial matrix, a left - right brain region symmetric difference matrix, a left - right brain region symmetric quotient matrix, and a differential entropy matrix.

[0096] S2 specifically includes:

[0097] Step S201: Construct an initial one - dimensional vector according to the number of electrode channels and sampling frequency of the international 10 - 20 system; specifically, construct a one - dimensional vector where n is the number of electrode channels, f is the sampling frequency, and the number of one - dimensional vectors for each experiment of each subject is the duration × sampling frequency size.

[0098] Step S202: constructing an initial two-dimensional matrix according to the initial one-dimensional vector and the electrode positions of the international 10-20 system.

[0099] Specifically, for the actual electrode cap, each electrode has a spatial connection, but a one-dimensional vector cannot represent the spatial information between the electrodes. Therefore, in order to represent the spatial information of the electrodes, a 9×9 matrix is constructed according to the actual electrode positions. The schematic diagram of the matrix construction is shown in the figure. The initial two-dimensional matrix As shown in formula (1) and formula (2), other positions in the matrix are set to 0 to avoid interference from other factors. The number of channels in formula (1) is 62. Therefore, when verifying the emotion recognition accuracy of the residual feature pyramid emotion recognition method based on EEG signals provided by the present invention, the SEED data set or HIED data set with the same number of channels of 62 is used; the number of channels in formula (2) is 32. Therefore, when verifying the emotion recognition accuracy of the residual feature pyramid emotion recognition method based on EEG signals provided by the present invention, the DEAP data set with the same number of channels of 32 is used.

[0100]

[0101]

[0102] When the DEAP dataset is used for verification, each subject can obtain a 40×60×128 original signal matrix (40 represents the number of trials of each subject, 60 is the duration of each trial in seconds, and 128 is the sampling frequency). When the SEED dataset is used for verification, each subject can obtain a 15×240×200 original signal matrix.

[0103] Step S203: normalize the initial two-dimensional matrix to obtain an initial matrix (OSM).

[0104] Specifically, in order to avoid the impact of large numerical differences on subsequent classification, each two-dimensional matrix is standardized by z-score standardization to obtain the initial matrix (OSM). The calculation formula is as follows:

[0105]

[0106] in and σ l They represent the non-zero numerical mean and variance of the matrix corresponding to sampling point l respectively.

[0107] Step S204: Eliminate the elements in the initial one-dimensional vector corresponding to the electrodes located on the anterior-posterior sagittal line in the international 10-20 system to obtain a one-dimensional vector after elimination.

[0108] Specifically, from the one-dimensional vector the middle electrodes are removed, where in the DEAP dataset, the middle electrodes are Fz, Cz, Pz, and Oz, and in the SEED dataset or HIED dataset, the middle electrodes are FPz, Fz, FCz ， Cz, CPz, Pz, POz, Oz

[0109] Step S205: Renumber the elements in the one-dimensional vector after the removal to obtain a renumbered one-dimensional vector.

[0110] Step S206: Determine the element numbers in the renumbered one-dimensional vector and the symmetric element numbers symmetric to the element numbers.

[0111] Specifically, n is the number of channels, n is 28 in the DEAP dataset, and n is 54 in the SEED dataset or HIED dataset. The symmetric electrode pairs are found according to the following rules:

[0112] When n is 18, and are a pair of electrode pairs, where 0 < i ≤ 14 and i is a positive integer; known are a pair of electrode pairs, where 14 < i ≤ 28 and i is a positive integer.

[0113] When n is 54, and are a pair of electrode pairs, where 0 < i ≤ 27 and i is a positive integer; known are a pair of electrode pairs, where 27 < i ≤ 54 and i is a positive integer.

[0114] Step S207: Take the difference between the element corresponding to the element number and the element corresponding to the symmetric element number to obtain a symmetric difference one-dimensional vector.

[0115] Specifically, the calculations are performed according to formulas (4) and (5):

[0116]

[0117]

[0118] where represents the difference of the electrode pair corresponding to sampling point 1, and i represents the electrode serial number after removing the middle electrodes. Taking the DEAP dataset as an example for verification, in the DEAP dataset is equal to the first channel (FP1) corresponding to the first sampling point minus the fifteenth channel (FP2). Taking the SEED dataset as an example for verification, in the SEED dataset It is equal to the difference between Channel 1 (FP1) corresponding to the first sampling point and Channel 28 (FP2). Thus, a one-dimensional vector is obtained. Furthermore, Formula (4) is the calculation method when using the DEAP dataset for verification, and Formula (5) is the calculation method when using the SEED dataset or the HIED dataset for verification.

[0119] Step S208: Construct a symmetric difference two-dimensional matrix based on the symmetric difference one-dimensional vector and the electrode positions of the international 10-20 system.

[0120] Specifically, the symmetric difference matrix D 2D_l is shown in Formulas (6) and (7).

[0121]

[0122]

[0123] Among them, Formula (6) is the symmetric difference two-dimensional matrix obtained when using the SEED dataset or the HIED dataset to verify the emotion recognition accuracy of the residual feature pyramid emotion recognition method based on EEG signals provided by the present invention; Formula (7) is the symmetric difference two-dimensional matrix obtained when using the DEAP dataset to verify the emotion recognition accuracy of the residual feature pyramid emotion recognition method based on EEG signals provided by the present invention.

[0124] Step S209: Normalize the symmetric difference two-dimensional matrix to obtain a left-right brain region symmetric difference matrix (SDM).

[0125] Specifically, to avoid the influence of excessive numerical differences on the classification model, z-score normalization is still used for normalization to obtain a left-right brain region symmetric difference matrix (SDM). The calculation formula is shown in Formula (3).

[0126] Step S210: Divide the element corresponding to the element number by the element corresponding to the symmetric element number to obtain a symmetric quotient one-dimensional vector.

[0127] Specifically, the calculations are performed according to Formulas (8) and (9):

[0128]

[0129]

[0130] Among them, represents the electrode pair difference corresponding to sampling point l, and i represents the electrode serial number after removing the middle electrode. Taking the DEAP dataset for verification as an example, in the DEAP dataset It is equal to the ratio of Channel Five (FC5) corresponding to the tenth sampling point to Channel Nineteen (FC6). Taking the verification with the SEED dataset or HIED dataset as an example, in the SEED dataset or HIED dataset, it is equal to the ratio of Channel Five (FC5) corresponding to the tenth sampling point to Channel Thirty-Two (FC6). Thus, a one-dimensional vector of the symmetric quotient is obtained. Furthermore, Formula (8) is the calculation method for verification using the DEAP dataset, and Formula (9) is the calculation method for verification using the SEED dataset or HIED dataset.

[0131] Step S211: Construct a two-dimensional symmetric quotient matrix based on the one-dimensional symmetric quotient vector and the electrode positions of the international 10-20 system.

[0132] Specifically, the two-dimensional symmetric quotient matrix Q 2D_l is shown in Formulas (10) and (11).

[0133]

[0134]

[0135] Among them, Formula (10) is the two-dimensional symmetric quotient matrix obtained when verifying the emotion recognition accuracy of the EEG-based residual feature pyramid emotion recognition method provided by the present invention using the SEED dataset or HIED dataset; Formula (11) is the two-dimensional symmetric quotient matrix obtained when verifying the emotion recognition accuracy of the EEG-based residual feature pyramid emotion recognition method provided by the present invention using the DEAP dataset.

[0136] Step S212: Standardize the two-dimensional symmetric quotient matrix to obtain a left-right brain region symmetric quotient matrix (SQM); specifically, to avoid the influence of excessive numerical differences on the classification model, z-score standardization is still used for standardization to obtain the left-right brain region symmetric quotient matrix (SQM). The calculation formula is shown in Formula (3).

[0137] Step S213: Extract the differential entropy features of the set frequency band from the initial one-dimensional vector using a set time window to obtain a differential entropy feature vector. Specifically, to ensure data volume equivalence, the differential entropy features (DE) of five frequency bands are extracted using a 1-second time window. The five frequency bands are the δ band, θ band, α band, β band, and γ band. Among them, the δ band is 1 Hz - 4 Hz, the θ band is 4 Hz - 7 Hz, the α band is 8 Hz - 12 Hz, the β band is 13 Hz - 30 Hz, and the γ band is 31 Hz - 45 Hz. Thus, 60×5×32 features for each trial of each subject in the DEAP dataset, and 240×5×62 features in the SEED dataset or 231×5×62 features in the HIED dataset are obtained.

[0138] Step S214: Construct a differential entropy matrix according to the differential entropy feature vector and the electrode positions of the international 10 - 20 system. Specifically, construct a two-dimensional differential entropy matrix according to the differential entropy feature vector and the electrode positions of the international 10 - 20 system; standardize the two-dimensional differential entropy matrix to obtain a differential entropy matrix (DEM).

[0139] O 2D_l As shown in Formula (12) and Formula (13).

[0140]

[0141]

[0142] Among them, Formula (12) is the symmetric quotient two-dimensional matrix obtained when verifying the emotion recognition accuracy of the residual feature pyramid emotion recognition method based on electroencephalogram signals provided by the present invention using the SEED dataset or HIED dataset; Formula (13) is the symmetric quotient two-dimensional matrix obtained when verifying the emotion recognition accuracy of the residual feature pyramid emotion recognition method based on electroencephalogram signals provided by the present invention using the DEAP dataset.

[0143] To avoid the influence of excessive numerical differences on the classification model, z-score standardization is still used for standardization to obtain a differential entropy matrix (DEM). The calculation formula is as shown in Formula (3).

[0144] Step S215: Apply a convolution operation to the feature matrix for preprocessing to obtain an initial feature map.

[0145] Specifically, the four feature matrices obtained in the above steps are respectively subjected to preliminary feature processing through a 1×1 convolution operation with 50 convolution kernels.

[0146] Step S216: Normalize the initial feature map to obtain a normalized feature map. Specifically, batch normalization is used to normalize the initial feature map.

[0147] Step S217: Apply an activation function to the normalized feature map to obtain a feature map. Specifically, the activation function is the SiLU activation function. After this step, a 50×9×9 feature map of each feature matrix is obtained.

[0148] Step S218: Perform nearest neighbor upsampling on the feature map to obtain the feature matrix of the feature map. Specifically, to fully capture the spatial information of the EEG signal, nearest neighbor upsampling is performed on the feature map. Further, nearest neighbor upsampling is performed on the 50×9×9 feature map. Four feature matrices obtained after upsampling all have a dimension of 50×62×62, and then the four feature matrices are fused.

[0149] Step S219: Fuse the feature matrices of the feature map to obtain a fused matrix. Specifically, the four fused matrices obtained after upsampling are fused. The fusion operation is a simple concatenation operation, which is performed on the channel dimension, that is, a 200×62×62 feature map is obtained.

[0150] Step S3: Construct an emotion recognition model based on a residual bidirectional feature pyramid. The emotion recognition model includes a spatial-to-depth structure, a residual structure, and a bidirectional feature pyramid structure.

[0151] Train the emotion recognition model. The training process of the emotion recognition model includes:

[0152] Step S301: Obtain the electroencephalogram data of the subject collected by electroencephalogram electrodes placed according to the international 10 - 20 system.

[0153] Step S302: Construct the feature matrix of the electroencephalogram data of the subject.

[0154] Step S303: Fuse the feature matrices of the electroencephalogram data of the subject to obtain the fused matrix of the electroencephalogram data of the subject.

[0155] Step S304: Use the fused matrix of the electroencephalogram data of the subject as the input to train the emotion recognition model to obtain a trained emotion recognition model.

[0156] The hyperparameters of the trained emotion recognition model are shown in Table 1.

[0157] Table 1 Hyperparameters of the trained emotion recognition model

[0158]

[0159]

[0160] Step S4: Input the fusion matrix into the trained emotion recognition model to obtain the emotion category of the person to be tested; the emotion category includes happy, angry, motivated, sad, calm, and fearful. Among them, happy and motivated belong to positive emotions; angry, sad, and fearful belong to negative emotions; calm belongs to neutral emotions.

[0161] S4 specifically includes:

[0162] Step S401: Use the space-to-depth structure to sample and reorganize the fusion matrix at a fixed scale to obtain multiple initial fusion matrices.

[0163] Specifically, for the space-to-depth (S2D) structure, since the concept of the feature pyramid was proposed, most researchers have considered the convolutional neural network as the backbone of the feature pyramid network to obtain feature maps of different scales. However, with the continuous development of the convolutional network, the convolutional network as the network backbone is getting deeper and deeper, which undoubtedly increases the computational cost of the model. For emotion computing, its ultimate goal is to achieve real-time interaction between humans and machines. Therefore, reducing the computational cost of the model and reducing the amount of model calculation are essential steps to achieve this goal. In addition, FPN pays more attention to the fusion of high-level semantic information and low-level spatial information. Therefore, the S2D structure is used to replace the traditional convolutional network backbone. Of course, this does not mean completely abandoning the convolutional network calculation, but using the S2D structure combined with the convolutional network to form an S2D layer as the basis of the backbone network. The S2D structure transfers the spatial dimension information to the depth dimension, samples and reorganizes the feature map at a fixed scale, replaces the traditional downsampling operation, and does not add additional parameters. Then, using 1×1 convolution, the SiLU activation function, and a fixed number of channels to output multiple initial fusion matrices. Details of the backbone network based on the S2D structure are as Figure 4 shown.

[0164] In practical applications, through a 3×3 convolutional network, batch normalization, and the SiLU activation function, the fusion matrix is initially calculated, and this operation is repeated twice to effectively extract the spatial information of the matrix using convolution. Specifically, after this operation is repeated twice, a matrix of size 31×31 is obtained, and the convolution operation step size of the second cycle is 2, making the matrix size become 1 / 2 of the original. Then, through four cycles of the S2D layer, four feature matrices of different scales, M3, M4, M5, and M6, are obtained, with sizes of 15×15, 7×7, 3×3, and 1×1 respectively.

[0165] Step S402: Use convolution operation to unify the number of channels of the multiple initial fusion matrices, obtaining multiple matrices with unified number of channels; specifically, use 1×1 convolution operation to unify the number of channels of the multiple initial fusion matrices. After multiple verifications in the present invention, the optimal number of channels is 256.

[0166] Step S403: Use a bidirectional feature pyramid structure and a residual structure to extract the sentiment category features of the multiple matrices with unified number of channels; the bidirectional feature pyramid structure and the residual structure can fully obtain the fusion information between layers and between nodes.

[0167] Specifically, regarding the Residual Feature Pyramid Network (RFPN), the Feature Pyramid Network (FPN) was initially proposed to better fuse semantic information at different scales and complete the extraction of deep semantic features of multi-scale information. The BiFPN (bi-directional feature pyramid network) adds a cross-node fusion method and considers the information fusion between nodes. On the basis of BiFPN, the present invention applies a residual structure to construct a Residual Feature Pyramid Network (RFPN). The RFPN (Residual Feature Pyramid Network) includes two parts: residual connection and cross-scale fusion. The residual connection can modularize the pyramid units and can perform simple stacked loops. The application of the residual structure can, on the one hand, perform information fusion between the feature map nodes through the shortcut, and on the other hand, can well avoid the problem of gradient explosion caused by the increase in the number of cycles of the feature pyramid structure, and increase the feature extraction ability of the pyramid. Cross-scale fusion considers the feature maps of the same scale and the feature maps of adjacent scales, effectively avoiding large-scale changes in feature information. As Figure 5 shown, taking the feature map M′5 as an example, its final output includes the fusion of the nearest neighbor upsampling of the previous layer M6, the previous layer M5, the max-pooling downsampling of the previous layer M4, and the downsampling of the same layer M′4.

[0168] In the present invention, the S2D structure is applied to form the backbone network of the sentiment recognition model, and the RFPN module is used for simple stacked loops to perform effective cross-scale and cross-layer feature fusion.

[0169] Step S404: Classify the emotional category features using a fully connected layer and output the emotional category. Specifically, through a classification prediction network, i.e., a fully connected layer, the prediction result of the emotional category of the final subject to be tested is obtained.

[0170] To better verify the complementarity between feature matrices for emotion recognition, fourteen experimental paradigms were constructed for experimental verification. Through the experiments of these models, not only can the influence of left and right brain region differences on emotion recognition performance be verified, but it can also be proved that fusing traditional frequency domain features can make up for the deficiencies of symmetric matrices. The experimental results of different feature comparisons are shown in Table 2.

[0171] Table 2 Experimental results of different feature comparisons

[0172]

[0173]

[0174] Among all the experimental results, when the input matrix is a single matrix, SDM performs the best, which further verifies that the left and right brain region difference matrices can effectively represent emotional information. In addition, compared with the quotient value, the difference between the left and right brain regions can better represent the response differences of the left and right brain regions to different emotional states. When multiple matrices are used as inputs, the fusion of SDM, SQM, and DEM is superior to other matrix combinations, from which it can be concluded that these three matrices have good complementarity. The fusion method of the four matrices proposed in the present invention is superior to other experimental paradigms and achieves the best performance.

[0175] In addition, to prove the superiority of the S2Dlayer, a classification model was constructed using ResNet18 as the backbone of the network. Similarly, the number of network layers was unified to increase the persuasiveness of the experimental results. Taking the DEAP dataset as an example, the experimental results of model comparisons with different backbones are shown in Table 3.

[0176] Table 3 Experimental results of model comparisons with different backbones

[0177]

[0178]

[0179] Obviously, compared with using ResNet18 as the network backbone, the S2Dlayer has better performance, and the training time of the model is significantly shortened, which provides a possible reference for realizing real-time human-computer interaction. Among them, H represents happiness, A represents anger, E represents excitation, S represents sadness, N represents calmness, and F represents fear.

[0180] In addition, four classification models were constructed based on different FPN structures and applied to three datasets for verification. The comparative experimental results of different pyramid models are shown in Table 4. To increase the persuasiveness of the experimental results, a network structure with the same depth was constructed.

[0181] Table 4 Comparative experimental results of different pyramid models

[0182]

[0183] The experimental results show that the feature pyramid structure (i.e., RFPN) proposed in the present invention has a better performance. RFPN fully integrates the feature semantic information between each layer and each node, and reduces the influence caused by gradient disappearance or gradient explosion as the number of layers increases. Among them, H represents happiness, A represents anger, E represents excitation, S represents sadness, N represents calmness, and F represents fear.

[0184] In addition, the emotion recognition models using different features and different networks in existing research were compared, and the comparative results of related work are shown in Table 5.

[0185] Table 5 Comparative results of related work

[0186]

[0187] As can be seen from Table 5, the residual feature pyramid emotion recognition method based on electroencephalogram signals proposed in the present invention has achieved satisfactory results in emotion recognition. In DEAP, the average accuracy rate of the subjects in the valence dimension reached 96.89% in the binary classification experiment, 96.82% in the arousal dimension, and the average accuracy rate of the four-classification experiment reached 93.56%. The three-classification and binary-classification accuracy rates reached 98.59% and 92.84% respectively in the SEED dataset.

[0188] The present invention constructs different feature matrices based on sampling points for fusion to represent different emotional state information. It not only considers the differences in the left and right brain regions of human emotional states, but also fuses the frequency-domain feature DE that characterizes the context correlation of electroencephalogram signals. In addition, the S2D layer replaces the traditional CNN as the backbone of the classification model, and the feature pyramid that fuses high-level semantic and low-level spatial information reduces the computational cost of the model. RFPN is constructed based on BiFPN and is used to replace the traditional FPN. RFPN consists of two parts: cross-layer connections based on residual structures and cross-scale fusion. It fully captures the information fusion between layers and nodes, reducing the impact of gradient disappearance or gradient explosion that may be caused by an increase in model depth. The residual feature pyramid emotion recognition method based on electroencephalogram signals proposed by the present invention is used for emotion recognition in the public datasets DEAP dataset and SEED dataset. In addition, the experimental results are compared with other studies based on public datasets, demonstrating that the emotion recognition model based on the residual bidirectional feature pyramid proposed in this paper has achieved satisfactory results.

[0189] As Figure 6 shown, the residual feature pyramid emotion recognition system based on electroencephalogram signals provided by the present invention is applied to the above-mentioned residual feature pyramid emotion recognition method based on electroencephalogram signals. The system includes:

[0190] A data acquisition module 1, configured to acquire electroencephalogram data of a test subject collected by electroencephalogram electrodes placed according to the international 10-20 system.

[0191] A feature matrix determination module 2, configured to construct a feature matrix of the electroencephalogram data of the test subject; the feature matrix includes an initial matrix, a left and right brain region symmetric difference matrix, a left and right brain region symmetric quotient matrix, and a differential entropy matrix.

[0192] A fusion matrix determination module 3, configured to fuse the feature matrix to obtain a fusion matrix.

[0193] A model construction module 4, configured to construct an emotion recognition model based on a residual bidirectional feature pyramid; the emotion recognition model includes a spatial-to-depth structure, a residual structure, and a bidirectional feature pyramid structure.

[0194] An emotion category determination module 5, configured to input the fusion matrix into the trained emotion recognition model to obtain the emotion category of the test subject; the emotion categories include happy, angry, excited, sad, calm, and fearful.

[0195] As a specific implementation manner, the system further includes a training module.

[0196] The training module includes:

[0197] An electroencephalogram data acquisition sub-module, configured to acquire electroencephalogram data of a subject collected by electroencephalogram electrodes placed according to the international 10-20 system.

[0198] A construction sub-module, configured to construct a feature matrix of the electroencephalogram data of the subject.

[0199] A fusion sub-module, configured to fuse the feature matrix of the electroencephalogram data of the subject to obtain a fusion matrix of the electroencephalogram data of the subject.

[0200] A training sub-module, configured to use the fusion matrix of the electroencephalogram data of the subject as an input to train the emotion recognition model to obtain a trained emotion recognition model.

[0201] As a specific implementation, the feature matrix determination module 2 includes:

[0202] An initial one-dimensional vector construction sub-module, configured to construct an initial one-dimensional vector according to the number of electrode channels and the sampling frequency of the international 10-20 system.

[0203] An initial two-dimensional matrix construction sub-module, configured to construct an initial two-dimensional matrix according to the initial one-dimensional vector and the electrode positions of the international 10-20 system.

[0204] An initial matrix determination sub-module, configured to standardize the initial two-dimensional matrix to obtain an initial matrix.

[0205] An elimination sub-module, configured to eliminate elements in the initial one-dimensional vector corresponding to electrodes located on the anterior-posterior sagittal line in the international 10-20 system to obtain an eliminated one-dimensional vector.

[0206] A re-numbering sub-module, configured to re-number elements in the eliminated one-dimensional vector to obtain a re-numbered one-dimensional vector.

[0207] A number determination sub-module, configured to determine the element numbers in the re-numbered one-dimensional vector and the symmetric element numbers symmetric to the element numbers.

[0208] A symmetric difference one-dimensional vector determination sub-module, configured to take the difference between the element corresponding to the element number and the element corresponding to the symmetric element number to obtain a symmetric difference one-dimensional vector.

[0209] A symmetric difference two-dimensional matrix determination sub-module, configured to construct a symmetric difference two-dimensional matrix according to the symmetric difference one-dimensional vector and the electrode positions of the international 10-20 system.

[0210] A symmetric difference matrix determination sub-module, configured to standardize the symmetric difference two-dimensional matrix to obtain a left-right brain region symmetric difference matrix.

[0211] A symmetric quotient one-dimensional vector determination sub-module is used to divide the element corresponding to the element number by the element corresponding to the symmetric element number to obtain a symmetric quotient one-dimensional vector.

[0212] A symmetric quotient two-dimensional matrix determination sub-module is used to construct a symmetric quotient two-dimensional matrix according to the symmetric quotient one-dimensional vector and the electrode positions of the international 10-20 system.

[0213] A symmetric quotient matrix determination sub-module is used to standardize the symmetric quotient two-dimensional matrix to obtain a symmetric quotient matrix of the left and right brain regions.

[0214] A differential entropy feature vector determination sub-module is used to extract the differential entropy features of a set frequency band from the initial one-dimensional vector using a set time window to obtain a differential entropy feature vector.

[0215] A differential entropy matrix determination sub-module is used to construct a differential entropy matrix according to the differential entropy feature vector and the electrode positions of the international 10-20 system.

[0216] As a specific implementation manner, the fusion matrix determination module 3 includes:

[0217] An initial feature map determination sub-module is used to perform preprocessing on the feature matrix of the electroencephalogram signal by applying a convolution operation to obtain an initial feature map.

[0218] A standardized feature map determination sub-module is used to standardize the initial feature map to obtain a standardized feature map.

[0219] A feature map determination sub-module is used to apply an activation function to the standardized feature map to obtain a feature map.

[0220] A feature matrix determination sub-module is used to perform nearest neighbor upsampling on the feature map to obtain a feature matrix of the feature map.

[0221] A fusion matrix determination sub-module is used to fuse the feature matrix of the feature map to obtain a fusion matrix.

[0222] Among them, the emotion category determination module 5 includes:

[0223] An initial fusion matrix determination unit is used to utilize a space-to-depth structure to sample and reorganize the fusion matrix at a fixed scale to obtain a plurality of initial fusion matrices.

[0224] A unified channel number matrix determination unit is used to unify the channel numbers of the plurality of initial fusion matrices by using a convolution operation to obtain a plurality of unified channel number matrices.

[0225] An extraction unit for extracting the sentiment category features of the multiple matrices with unified number of channels by using a bidirectional feature pyramid structure and a residual structure.

[0226] A result output unit for classifying the sentiment category features by applying a fully connected layer and outputting the sentiment category.

[0227] The residual feature pyramid sentiment recognition method and system based on electroencephalogram signals provided by the present invention are based on the feature extraction and fusion strategy of sampling points and are applied to the residual feature pyramid network structure for training and classification. First, we constructed an original signal matrix (OSM), a symmetric difference matrix (SDM) of the left and right brain regions, a symmetric quotient matrix (SQM) of the left and right brain regions, and a differential entropy matrix (DEM) based on the sampling points, and performed feature-level fusion on these four feature matrices. This feature fusion strategy combines the difference features of the left and right brain regions and the context relationship information of the electroencephalogram. Then, the fused features are put into a classification model for training. The classification model consists of a space-to-depth (S2D) structure that replaces the convolutional network as the backbone, and a residual feature pyramid network (RFPN) proposed by this method. It not only emphasizes the characteristics of the feature pyramid for fusing high-level semantic information and low-level spatial information, but also reduces the training time and the possible impacts brought by the increase in the number of layers.

[0228] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and reference can be made to the description of the method part for the relevant parts.

[0229] Specific examples are used in this article to elaborate on the principles and implementation manners of the present invention. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A residual feature pyramid emotion recognition method based on electroencephalogram signals, characterized in that, The method includes: Obtaining electroencephalogram data of a subject collected by electroencephalogram electrodes placed according to the international 10-20 system; Constructing a feature matrix of the electroencephalogram data of the subject, and fusing the feature matrix to obtain a fusion matrix; the feature matrix includes an initial matrix, a left-right brain region symmetry difference matrix, a left-right brain region symmetry quotient matrix, and a differential entropy matrix; The construction of the feature matrix of the electroencephalogram data of the subject to be tested specifically includes: Constructing an initial one-dimensional vector according to the number of electrode channels and the sampling frequency of the international 10-20 system; Constructing an initial two-dimensional matrix according to the initial one-dimensional vector and the electrode positions of the international 10-20 system; Normalizing the initial two-dimensional matrix to obtain an initial matrix; Removing the elements in the initial one-dimensional vector corresponding to the electrodes located on the anterior-posterior sagittal line in the international 10-20 system to obtain a one-dimensional vector after removal; Renumbering the elements in the one-dimensional vector after removal to obtain a renumbered one-dimensional vector; Determining the element numbers in the renumbered one-dimensional vector and the symmetric element numbers symmetric to the element numbers; Taking the difference between the elements corresponding to the element numbers and the elements corresponding to the symmetric element numbers to obtain a symmetry difference one-dimensional vector; Constructing a symmetry difference two-dimensional matrix according to the symmetry difference one-dimensional vector and the electrode positions of the international 10-20 system; Normalizing the symmetry difference two-dimensional matrix to obtain a left-right brain region symmetry difference matrix; Dividing the elements corresponding to the element numbers by the elements corresponding to the symmetric element numbers to obtain a symmetry quotient one-dimensional vector; Constructing a symmetry quotient two-dimensional matrix according to the symmetry quotient one-dimensional vector and the electrode positions of the international 10-20 system; Normalizing the symmetry quotient two-dimensional matrix to obtain a left-right brain region symmetry quotient matrix; Extracting differential entropy features of a set frequency band from the initial one-dimensional vector using a set time window to obtain a differential entropy feature vector; Constructing a differential entropy matrix according to the differential entropy feature vector and the electrode positions of the international 10-20 system; Constructing an emotion recognition model based on a residual bidirectional feature pyramid; the emotion recognition model includes a spatial-to-depth structure, a residual structure, and a bidirectional feature pyramid structure; Inputting the fusion matrix into the trained emotion recognition model to obtain the emotion category of the subject, specifically including: Using the spatial-to-depth structure to sample and reorganize the fusion matrix at a fixed scale to obtain multiple initial fusion matrices; Using convolution operations to unify the number of channels of the multiple initial fusion matrices to obtain multiple matrices with unified channel numbers; Using the bidirectional feature pyramid structure and the residual structure to extract emotion category features of the multiple matrices with unified channel numbers; Applying a fully connected layer to classify the emotion category features and output the emotion category; The emotion categories include happiness, anger, excitement, sadness, calmness, and fear.

2. The residual feature pyramid emotion recognition method based on electroencephalogram signals according to claim 1, wherein The training process of the emotion recognition model includes: Obtaining electroencephalogram data of a subject collected by electroencephalogram electrodes placed according to the international 10-20 system; Constructing a feature matrix of the electroencephalogram data of the subject; fusing the feature matrix of the electroencephalogram data of the subject to obtain a fusion matrix of the electroencephalogram data of the subject; The emotion recognition model is trained using the fusion matrix of the subject's electroencephalogram data as input to obtain a trained emotion recognition model.

3. The residual feature pyramid emotion recognition method based on electroencephalogram signals according to claim 1, characterized in that The fusing the feature matrices to obtain a fusion matrix specifically includes: Applying a convolution operation to the feature matrix for preprocessing to obtain an initial feature map; Normalizing the initial feature map to obtain a standardized feature map; Applying an activation function to the standardized feature map to obtain a feature map; Performing nearest neighbor upsampling on the feature map to obtain a feature matrix of the feature map; The feature matrices of the feature maps are fused to obtain a fused matrix.

4. A residual feature pyramid emotion recognition system based on electroencephalogram signals, characterized in that, The system comprises: A data acquisition module, used to acquire EEG data of the subject to be tested collected by EEG electrodes placed according to the international 10-20 system; A feature matrix determination module is used to construct a feature matrix of the EEG data of the subject to be tested; the feature matrix includes an initial matrix, a symmetric difference matrix of left and right brain regions, a symmetric quotient matrix of left and right brain regions, and a differential entropy matrix; the feature matrix determination module includes: The initial one-dimensional vector construction submodule is used to construct the initial one-dimensional vector according to the number of electrode channels and sampling frequency of the international 10-20 system; An initial two-dimensional matrix construction submodule, used to construct an initial two-dimensional matrix according to the initial one-dimensional vector and the electrode positions of the international 10-20 system; An initial matrix determination submodule, used for normalizing the initial two-dimensional matrix to obtain an initial matrix; A elimination submodule, used to eliminate the elements in the initial one-dimensional vector corresponding to the electrodes located on the anterior-posterior sagittal line in the international 10-20 system, to obtain a one-dimensional vector after elimination; A renumbering submodule, used for renumbering the elements in the removed one-dimensional vector to obtain a renumbered one-dimensional vector; A number determination submodule, used to determine the element number in the renumbered one-dimensional vector and the symmetric element number symmetric to the element number; A symmetric difference one-dimensional vector determination submodule, used for taking the difference between the element corresponding to the element number and the element corresponding to the symmetric element number to obtain a symmetric difference one-dimensional vector; A symmetric difference two-dimensional matrix determination submodule is used to construct a symmetric difference two-dimensional matrix according to the symmetric difference one-dimensional vector and the electrode position of the international 10-20 system; A symmetric difference matrix determination submodule is used to standardize the symmetric difference two-dimensional matrix to obtain a symmetric difference matrix of left and right brain regions; A symmetric quotient one-dimensional vector determination submodule, used for dividing the element corresponding to the element number with the element corresponding to the symmetric element number to obtain a symmetric quotient one-dimensional vector; A symmetry quotient two-dimensional matrix determination submodule, used to construct a symmetry quotient two-dimensional matrix according to the symmetry quotient one-dimensional vector and the electrode position of the international 10-20 system; A symmetric quotient matrix determination submodule is used to standardize the symmetric quotient two-dimensional matrix to obtain a symmetric quotient matrix for left and right brain regions; The differential entropy feature vector determination sub-module is used to extract the differential entropy features of a set frequency band from the initial one-dimensional vector using a set time window to obtain a differential entropy feature vector; The differential entropy matrix determination sub-module is used to construct a differential entropy matrix according to the differential entropy feature vector and the electrode positions of the international 10-20 system; The fusion matrix determination module is used to fuse the feature matrix to obtain a fusion matrix; The model construction module is used to construct an emotion recognition model based on a residual bidirectional feature pyramid; the emotion recognition model includes a spatial-to-depth structure, a residual structure, and a bidirectional feature pyramid structure; The emotion category determination module is used to input the fusion matrix into the trained emotion recognition model to obtain the emotion category of the subject to be tested. The emotion category determination module includes: The initial fusion matrix determination unit is used to use the spatial-to-depth structure to sample and reorganize the fusion matrix at a fixed scale to obtain a plurality of initial fusion matrices; The unified channel number matrix determination unit is used to unify the channel numbers of the plurality of initial fusion matrices using a convolution operation to obtain a plurality of unified channel number matrices; The extraction unit is used to extract the emotion category features of the plurality of unified channel number matrices using the bidirectional feature pyramid structure and the residual structure; The result output unit is used to classify the emotion category features using a fully connected layer and output the emotion category; The emotion categories include happy, angry, excited, sad, calm, and fearful.

5. The residual feature pyramid emotion recognition system based on EEG signals according to claim 4, characterized in that, The system further includes a training module; The training module includes: The electroencephalogram data acquisition sub-module is used to acquire the electroencephalogram data of a subject collected by electroencephalogram electrodes placed according to the international 10-20 system; The construction sub-module is used to construct a feature matrix of the electroencephalogram data of the subject; The fusion sub-module is used to fuse the feature matrix of the electroencephalogram data of the subject to obtain a fusion matrix of the electroencephalogram data of the subject; The training sub-module is used to use the fusion matrix of the electroencephalogram data of the subject as an input to train the emotion recognition model to obtain a trained emotion recognition model.

6. The residual feature pyramid emotion recognition system based on electroencephalogram signals according to claim 4, wherein The fusion matrix determination module includes: The initial feature map determination sub-module is used to preprocess the feature matrix of the electroencephalogram signal using a convolution operation to obtain an initial feature map; The standardized feature map determination sub-module is used to standardize the initial feature map to obtain a standardized feature map; The feature map determination sub-module is used to apply an activation function to the standardized feature map to obtain a feature map; The feature matrix determination sub-module is used to perform nearest neighbor upsampling on the feature map to obtain a feature matrix of the feature map; The fusion matrix determination sub-module is used to fuse the feature matrix of the feature map to obtain a fusion matrix.

Images