A method and system for EEG emotion recognition based on spatiotemporal complementary learning

Abstract

This invention belongs to the field of computer technology, specifically relating to a method and system for EEG emotion recognition based on spatiotemporal complementary learning. The EEG emotion recognition method includes the following steps: S1: Constructing spatial positional relationships in EEG signals using a capsule network to obtain spatial features containing spatial information of the EEG signals. S2: Processing the spatial features using a neural network that remembers temporal information to complete dynamic learning of emotions and obtain temporal features containing temporal information of the EEG signals. S3: Presetting the connection weight matrix for the complementary learning stage to fuse spatial and temporal features, obtaining high-level spatiotemporal complementary emotion features. S4: Feeding the high-level emotion features obtained from complementary learning into a fully connected layer to obtain the final prediction result of the emotion state. This invention solves the problem of insufficient extraction of effective information from EEG signals in existing EEG emotion recognition methods, leading to low prediction accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to a method, system, and device for EEG emotion recognition based on spatiotemporal complementary learning. Background Technology

[0002] As people gain a deeper understanding of emotions, emotion recognition technology is attracting increasing attention. EEG-based emotion recognition technology, due to its non-invasiveness and high accuracy, is gradually becoming a hot research topic in emotion recognition. This technology can be widely applied in emotion recognition, neurofeedback, and human-computer interaction. In emotion recognition, it can be applied to natural language processing, intelligent healthcare, and emotion analysis. In neurofeedback, it can be applied to the treatment of mental disorders and cognitive training. In human-computer interaction, it can be applied to brain-computer interfaces and virtual reality.

[0003] The significance of EEG-based emotion recognition technology is immense. Traditional emotion recognition technologies typically rely on text and images, both of which have limitations. EEG-based emotion recognition technology can directly extract emotional information from human brain signals, offering higher accuracy and objectivity. Furthermore, this technology can provide more precise data support for fields such as neurofeedback and human-computer interaction, significantly advancing the development of these areas.

[0004] When an individual experiences emotional stimulation due to external events, different areas of the brain exchange information, and the electroencephalogram (EEG) signals reflect this emotional change. To improve the accuracy of EEG-based emotion recognition, researchers have proposed various methods, such as extracting time-frequency domain features from preprocessed EEG signals for emotion recognition, and using deep learning to further extract spatial features for emotion recognition.

[0005] The challenge of EEG emotion recognition lies in how to extract highly representative and generalized emotional information from raw signals for identification. Existing research suffers from the following problems: First, most existing methods rely on single modules for feature extraction, which has significant limitations; second, while using convolutional neural networks, graph networks, and other methods to process time-slice information of EEG signals provides some representation of spatial relationships in EEG signals, it fails to learn temporal dynamic features of emotions; third, it ignores the spatiotemporal correlation of EEG signals, resulting in insufficient emotional representation. Summary of the Invention

[0006] To address the problem that existing brainwave emotion recognition methods are insufficient in extracting effective information from brainwave signals, leading to low prediction accuracy, this invention provides a brainwave emotion recognition method, system, and device based on spatiotemporal complementary learning.

[0007] This invention is achieved using the following technical solution:

[0008] A brainwave emotion recognition method based on spatiotemporal complementary learning includes the following steps:

[0009] S1: Using capsule networks to process raw EEG signals X, constructing spatial relationships within EEG signals X, and then obtaining spatial features V containing spatial information of the EEG signals. j Specifically, it includes the following steps:

[0010] S11: Perform convolution operations on the input raw EEG signal to obtain low-level features, and then transform the low-level features into primary capsules u of the capsule network through transformation operations. i .

[0011] S12: Preset weights W for the affine transformation ij Then, for the primary capsule u i Perform affine transformation to obtain an advanced capsule.

[0012] S13: Further map the emotional relationships of the advanced capsules through dynamic routing iteration, find advanced capsules that are highly correlated with the emotional capsules, and continuously extract spatial features with high-probability emotional associations to obtain emotional capsules s containing spatial information. j .

[0013] S14: The emotion capsule is compressed using a nonlinear activation function to obtain the final spatial feature V. j .

[0014] S2: Utilize a neural network that remembers temporal information to analyze the spatial features V from the previous step. j The process involves dynamic learning of emotions to obtain temporal features h that contain temporal information from brainwave signals. j .

[0015] S3: Preset connection weight matrix O for the complementary learning phase j For spatial features V j and time features h j By fusing these elements, we obtain advanced emotional features g that are complementary in time and space. j .

[0016] S4: Complement the high-level sentiment features g obtained in the previous step. j The data is fed into a fully connected layer to obtain the final prediction of the sentiment state.

[0017] As a further improvement of the present invention, in step S11, the primary capsule u i The generating functions are as follows:

[0018] u i =Reshape[Conv(X)]

[0019] Where i represents the number of each primary capsule; Conv represents the convolution operation; and Reshape represents the transformation operation.

[0020] As a further improvement of the present invention, in step S12, the advanced capsule... The generating functions are as follows:

[0021]

[0022] As a further improvement of the present invention, in step S13, the emotional capsule s j The generating functions are as follows:

[0023]

[0024] in, Indicates emotional capsules j Related advanced capsules; b ij Let b represent the coupling coefficient between the i-th advanced capsule and the j-th emotion capsule. ij Initialize to 0 and update based on the product of the advanced capsule and the emotion capsule; c ij Let be the coefficients of the dynamic routing algorithm, and use them as the weight coefficients of each advanced capsule in the emotion capsule; and satisfy the following: the sum of the weight coefficients of all advanced capsules is 1, that is: ∑ i c ij =1.

[0025] As a further improvement of the present invention, in step S14, the spatial feature V j The generating functions are as follows:

[0026]

[0027] In the above formula, α is a preset hyperparameter used to adjust the degree of compression of spatial features.

[0028] As a further improvement of this invention, in step S2, an RNN network is used as the neural network for recording temporal information. The RNN network receives spatial features at each time step and generates a temporal feature sequence that encodes the temporal information of the emotional state; and adaptively learns the temporal feature representation through the backpropagation algorithm; the final output of the RNN network is:

[0029] h j =RNN(h j-1 V j );

[0030] In the above formula, V j Represents the spatial characteristics at the current moment; h j-1 Indicates the hidden state at the previous moment; hj This represents the time feature extracted at the current moment.

[0031] As a further improvement of the present invention, in step S3, the advanced emotional feature g j The fusion equation is as follows:

[0032] g j =O j (h j +V j );

[0033] In step S4, the classification prediction function of the fully connected layer is as follows:

[0034] y = softmax(W·g j +b);

[0035] In the above formula, W is the weight matrix of the fully connected layer; b is the bias; and softmax is the normalized exponential function.

[0036] As a further improvement of this invention, a large number of EEG signals {X, y} with emotional labels are used as samples to form a training set; the overall network model, including a capsule network, a neural network containing memory temporal information, a fusion module that integrates spatial and temporal features, and a fully connected layer, is trained; the parameters and weights of the overall network model are updated; wherein, the objective function L during the training phase is... k as follows:

[0037] L k =T k max(0,m + -||y k ||) 2 +λ(1-T k max(0,||y) k ||-m - ) 2

[0038] Among them, T k Let m be a state flag indicating whether or not a type k emotion exists; + and m - y is a pair of hyperparameters, representing the upper and lower bounds of the probability constraint, respectively; λ is a pre-defined hyperparameter used to adjust the loss of the objective function; k This represents the probability of predicting the k-th emotion category.

[0039] This invention also includes a spatiotemporal complementary learning-based EEG emotion recognition system, which analyzes the original EEG signals of a specified object using the aforementioned spatiotemporal complementary learning-based EEG emotion recognition method, and outputs the predicted emotional state of the specified object. The spatiotemporal complementary learning-based EEG emotion recognition system provided by this invention includes: an EEG signal acquisition module, a spatial feature extraction module, a temporal feature extraction module, a feature fusion model, and a classification module.

[0040] The EEG signal acquisition module acquires the raw EEG signals of a specified object to be identified. The spatial feature extraction module includes a primary capsule generation unit, a higher-level capsule generation unit, an emotional capsule generation unit, and a feature compression unit. The primary capsule generation unit first performs a convolution operation on the input raw EEG signal to obtain low-level features, and then transforms these low-level features into primary capsules through a transformation operation. The higher-level capsule generation unit performs an affine transformation on the primary capsules according to preset weights to obtain higher-level capsules. The emotional capsule generation unit further maps the higher-level capsules to emotional relationships through dynamic routing iteration, searching for higher-level capsules highly correlated with emotional capsules, and continuously extracting spatial features with high-probability emotional associations to obtain emotional capsules containing spatial information. The feature compression unit uses a non-linear activation function to compress the emotional capsules to obtain the final spatial features.

[0041] The time feature extraction module uses an RNN network as a neural network to record time-series information. The RNN network receives spatial features at each time step and generates a time feature sequence that encodes the time information of emotional state. The module then adaptively learns the time feature representation through the backpropagation algorithm and outputs the time features implied in the spatial features.

[0042] The feature fusion model is used to pre-set the connection weight matrix in the complementary learning stage, and fuse spatial and temporal features to obtain high-level sentiment features that are complementary in time and space.

[0043] The classification module uses a fully connected layer, which is used to classify the advanced sentiment features after spatiotemporal complementary fusion and output the predicted sentiment state.

[0044] The present invention also includes a brainwave emotion recognition device based on spatiotemporal complementary learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the brainwave emotion recognition method based on spatiotemporal complementary learning as described above, and then predicts the corresponding emotional state based on the brainwave signals of a specified object.

[0045] The technical solution provided by this invention has the following beneficial effects:

[0046] This invention, building upon existing EEG emotion recognition methods, achieves the classification and extraction of spatial and temporal features. Extracting temporal features helps the model better distinguish the dynamic process of emotion changes, thereby improving the accuracy of emotion recognition. Furthermore, by creating a complementary learning framework, the spatial and temporal features extracted from the EEG signals are complementaryly fused for subsequent prediction of the emotion of a specified object. After the network model is trained and the weight parameters are iteratively updated, the emotion recognition accuracy of this invention is significantly improved. In addition, this invention has stronger representational capabilities and better generalization performance. Attached Figure Description

[0047] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0048] Figure 1 This is a flowchart of the steps of an EEG emotion recognition method based on spatiotemporal complementary learning provided in Embodiment 1 of the present invention.

[0049] Figure 2 This is a logical block diagram of the brainwave emotion recognition process in the brainwave emotion recognition method based on spatiotemporal complementary learning in Embodiment 1 of the present invention.

[0050] Figure 3 This is a structural topology diagram of an EEG emotion recognition system based on spatiotemporal complementary learning provided in Embodiment 2 of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0052] Example 1

[0053] A brainwave emotion recognition method based on spatiotemporal complementary learning, such as Figure 1 As shown, it includes the following steps:

[0054] S1: Using capsule networks to process raw EEG signals X, constructing spatial relationships within EEG signals X, and then obtaining spatial features V containing spatial information of the EEG signals. j Specifically, the steps include the following:

[0055] S11: Perform convolution operations on the input raw EEG signal to obtain low-level features, and then transform the low-level features into primary capsules u of the capsule network through transformation operations. i Primary capsules ui The generating functions are as follows:

[0056] u i =Reshape[Conv(X)]

[0057] Where i represents the number of each primary capsule; Conv represents the convolution operation; and Reshape represents the transformation operation.

[0058] S12: Preset weights W for the affine transformation ij Then, for the primary capsule u i Perform affine transformation to obtain an advanced capsule. Premium Capsules The generating functions are as follows:

[0059]

[0060] S13: Further map the emotional relationships of the advanced capsules through dynamic routing iteration, find advanced capsules that are highly correlated with the emotional capsules, and continuously extract spatial features with high-probability emotional associations to obtain emotional capsules s containing spatial information. j Emotional capsules j The generating functions are as follows:

[0061]

[0062] in, Indicates emotional capsules j Related advanced capsules; b ij Let b represent the coupling coefficient between the i-th advanced capsule and the j-th emotion capsule. ij Initialize to 0 and update based on the product of the advanced capsule and the emotion capsule; c ij Let be the coefficients of the dynamic routing algorithm, and use them as the weight coefficients of each advanced capsule in the emotion capsule; and satisfy the following: the sum of the weight coefficients of all advanced capsules is 1, that is: ∑ i c ij =1.

[0063] S14: The emotion capsule is compressed using a nonlinear activation function to obtain the final spatial feature V. j Spatial features V j The generating functions are as follows:

[0064]

[0065] In the above formula, α is a preset hyperparameter used to adjust the degree of compression of spatial features.

[0066] S2: Utilize a neural network that remembers temporal information to analyze the spatial features V from the previous step. jThe process involves dynamic learning of emotions to obtain temporal features h that contain temporal information from brainwave signals. j .

[0067] Specifically, in this embodiment, an RNN network is used as the neural network for recording temporal information. The RNN network receives spatial features at each time step, generating a temporal feature sequence that encodes the temporal information of the emotional state; and adaptively learns the temporal feature representation through backpropagation. The final output of the RNN network is:

[0068] h j =RNN(h j-1 V j );

[0069] In the above formula, V j Represents the spatial characteristics at the current moment; h j-1 Indicates the hidden state at the previous moment; h j This represents the time feature extracted at the current moment.

[0070] S3: Preset connection weight matrix O for the complementary learning phase j For spatial features V j and time features h j By fusing these elements, we obtain advanced emotional features g that are complementary in time and space. j Advanced emotional characteristics g j The fusion equation is as follows:

[0071] g j =O j (h j +V j ).

[0072] S4: Complement the high-level sentiment features g obtained in the previous step. j The data is fed into a fully connected layer to obtain the final prediction result of the sentiment state. The classification prediction function of the fully connected layer is as follows:

[0073] y = softmax(W·g j +b);

[0074] In the above formula, W is the weight matrix of the fully connected layer; b is the bias; and softmax is the normalized exponential function.

[0075] The EEG emotion recognition method based on spatiotemporal complementary learning proposed in this invention mainly employs the following strategy: Figure 2As shown, firstly, a spatial feature extraction module based on a capsule network is constructed to capture spatial feature information from the input raw EEG signal. Then, to further obtain the state information of the EEG signal changing over time, an RNN network is used as a neural network to store temporal information, further extracting features from the upper-level spatial features to obtain the temporal features implicit in the spatial features. Next, at the output of the temporal feature extraction module, a shortcut connection from the spatial module is constructed to achieve complementary learning of feature information from both modules, obtaining high-level spatiotemporal features. Finally, the obtained high-level spatiotemporal features are used for final emotion recognition.

[0076] In practical applications of this invention, the scheme relies heavily on the performance of the spatial feature extraction module, the temporal feature extraction module, and the network model in the complementary learning and classification stages. Therefore, this embodiment requires training the network model using a dataset containing emotional state labels. During the training phase, a large number of EEG signals {X, y} with emotional labels are used as samples to form the training set. The overall network model, including the capsule network, the neural network containing memory temporal information, the fusion module integrating spatial and temporal features, and the fully connected layers, is trained; and the parameters and weights of the overall network model are updated; wherein, the objective function L during the training phase... k as follows:

[0077] L k =T k max(0,m + -||y k ||) 2 +λ(1-T k )max(0,||yk||-m - ) 2

[0078] Among them, T k Let m be a state flag indicating whether or not a type k emotion exists; + and m - y is a pair of hyperparameters, representing the upper and lower bounds of the probability constraint, respectively; λ is a pre-defined hyperparameter used to adjust the loss of the objective function; k This represents the probability of predicting the k-th emotion category.

[0079] In this embodiment, a capsule network is used to extract features from the raw EEG signals to learn the spatial relationships between different leads of the EEG signals, simulating the correspondence between different brain regions and different emotions, thus realizing the representation of the spatial topological relationships of EEG signals in different brain regions. A recurrent neural network is then used to further extract the temporal information of the EEG signals, fully utilizing the characteristic of EEG signals to reflect subtle emotional changes at high resolution, and effectively learning the temporal dependencies of the EEG signals. Furthermore, by constructing a shortcut connection, this embodiment allows the feature information of the two modules to complement and learn from each other, promoting the optimization of the dual-module structure. This results in the high-level emotional features obtained by the model having stronger representational power and generalization performance, ultimately improving the accuracy of emotion recognition in this embodiment.

[0080] Example 2

[0081] Based on the implementation scheme, this embodiment further provides a brainwave emotion recognition system based on spatiotemporal complementary learning, which uses the brainwave emotion recognition method based on spatiotemporal complementary learning as in Embodiment 1 to analyze the original brainwave signal of the input specified object and output the predicted emotional state of the specified object.

[0082] like Figure 3 As shown, the EEG emotion recognition system based on spatiotemporal complementary learning provided in this embodiment includes: an EEG signal acquisition module, a spatial feature extraction module, a temporal feature extraction module, a feature fusion model, and a classification module.

[0083] The EEG signal acquisition module is used to acquire the raw EEG signals of the specified object to be identified.

[0084] The spatial feature extraction module includes a primary capsule generation unit, a high-level capsule generation unit, an emotional capsule generation unit, and a feature compression unit. The primary capsule generation unit first performs a convolution operation on the input raw EEG signal to obtain low-level features, and then transforms these low-level features into primary capsules through a transformation operation. The high-level capsule generation unit performs an affine transformation on the primary capsules according to preset weights to obtain high-level capsules. The emotional capsule generation unit further maps the high-level capsules to emotional relationships through dynamic routing iteration, searching for high-level capsules highly correlated with emotional capsules, and continuously extracting spatial features with high-probability emotional associations to obtain emotional capsules containing spatial information. The feature compression unit uses a non-linear activation function to compress the emotional capsules to obtain the final spatial features.

[0085] The time feature extraction module uses an RNN network as a neural network to record time-series information. The RNN network receives spatial features at each time step and generates a time feature sequence that encodes the time information of emotional state. The module then adaptively learns the time feature representation through the backpropagation algorithm and outputs the time features implied in the spatial features.

[0086] The feature fusion model is used to pre-set the connection weight matrix in the complementary learning stage, and fuse spatial and temporal features to obtain high-level sentiment features that are complementary in time and space.

[0087] The classification module uses a fully connected layer, which is used to classify the advanced sentiment features after spatiotemporal complementary fusion and output the predicted sentiment state.

[0088] Example 3

[0089] This embodiment provides a brainwave emotion recognition device based on spatiotemporal complementary learning, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the brainwave emotion recognition method based on spatiotemporal complementary learning as described in Embodiment 1, and then predicts the corresponding emotional state based on the brainwave signals of a specified object.

[0090] The computer device can be a smartphone, tablet, laptop, desktop computer, rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), etc., capable of executing programs. The computer device in this embodiment includes, but is not limited to, a memory and a processor that can communicate with each other via a system bus.

[0091] In this embodiment, the memory (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of a computer device, such as the hard disk or RAM of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device. Of course, the memory can also include both internal storage units and external storage devices of the computer device. In this embodiment, the memory is typically used to store the operating system and various application software installed on the computer device. In addition, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0092] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of a computer device.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A brainwave emotion recognition method based on spatiotemporal complementary learning, characterized in that, It includes the following steps: S1: Using capsule networks to process raw EEG signals X, constructing spatial relationships within EEG signals X, and then obtaining spatial features V containing spatial information of the EEG signals. j Specifically, it includes the following steps: S11: Perform convolution operations on the input raw EEG signal to obtain low-level features, and then transform the low-level features into primary capsules u of the capsule network through transformation operations. i ; S12: Preset weights W for the affine transformation ij Then, for the primary capsule u i Perform affine transformation to obtain an advanced capsule. S13: Further map the emotional relationships of the advanced capsules through dynamic routing iteration, find advanced capsules that are highly correlated with the emotional capsules, and continuously extract spatial features with high-probability emotional associations to obtain emotional capsules s containing spatial information. j ; S14: The emotion capsule is compressed using a nonlinear activation function to obtain the final spatial feature V. j : S2: Utilize a neural network that remembers temporal information to analyze the spatial features V from the previous step. j The process involves dynamic learning of emotions to obtain temporal features h that contain temporal information from brainwave signals. j ; An RNN network is used as the neural network to record temporal information. At each time step, the RNN network receives spatial features and generates a temporal feature sequence that encodes the temporal information of the emotional state. The temporal feature representation is adaptively learned through backpropagation. The final output of the RNN network is: h j =RNN(h j-1 ,V j ); In the above formula, V j Indicates the spatial characteristics at the current moment; h j-1 Indicates the hidden state at the previous moment; h j This represents the time feature extracted at the current moment; S3: Preset connection weight matrix O for the complementary learning phase j For spatial features V j and time features h j By fusing these elements, we obtain advanced emotional features g that are complementary in time and space. j ; Advanced emotional characteristics g j The fusion equation is as follows: g j =O j (h j +V j ); S4: Complement the high-level sentiment features g obtained in the previous step. j The data is fed into a fully connected layer to obtain the final prediction result of the sentiment state; the classification prediction function of the fully connected layer is as follows: y=softmax(W·g j +b); In the above formula, W is the weight matrix of the fully connected layer; b is the bias; softmax is the normalized exponential function; and y represents the predicted sentiment label.

2. The EEG emotion recognition method based on spatiotemporal complementary learning as described in claim 1, characterized in that: In step S11, the primary capsule u i The generating functions are as follows: u i =Reshape[Conv(X)] Where i represents the number of each primary capsule; Conv represents the convolution operation; and Reshape represents the transformation operation.

3. The EEG emotion recognition method based on spatiotemporal complementary learning as described in claim 2, characterized in that: In step S12, the advanced capsule The generating functions are as follows:

4. The EEG emotion recognition method based on spatiotemporal complementary learning as described in claim 3, characterized in that: In step S13, the emotional capsule s j The generating functions are as follows: in, Indicates emotional capsules j Related advanced capsules; b ij Let b represent the coupling coefficient between the i-th advanced capsule and the j-th emotion capsule. ij Initialize to 0 and update based on the product of the advanced capsule and the emotion capsule; c ij Let be the coefficient of the dynamic routing algorithm, and use it as the weight coefficient of each advanced capsule in the emotion capsule; and satisfy the following: the sum of the weight coefficients of all advanced capsules is 1, that is: Σ i c ij =1.

5. The EEG emotion recognition method based on spatiotemporal complementary learning as described in claim 4, characterized in that: In step S14, spatial feature V j The generating functions are as follows: In the above formula, α is a preset hyperparameter used to adjust the degree of compression of spatial features.

6. The EEG emotion recognition method based on spatiotemporal complementary learning as described in claim 1, characterized in that: A large number of EEG signals {X, y} with emotional labels were used as samples to form a training set. The overall network model, including a capsule network, a neural network containing temporal information, a fusion module integrating spatial and temporal features, and a fully connected layer, was trained. The parameters and weights of the overall network model were updated. The objective function L during the training phase was... k as follows: L k < T k max(0,m + -||y k ||) 2 +λ(1-T k )max(0,||y k ||-m - ) 2 Among them, T k Let m be a state flag indicating whether or not a type k emotion exists; + and m - These are the upper and lower bounds of the probability constraint, respectively, where λ is a manually preset hyperparameter used to adjust the loss of the objective function; y k This represents the probability of predicting the k-th emotion category.

7. A brainwave emotion recognition system based on spatiotemporal complementary learning, characterized in that, It employs the spatiotemporal complementary learning-based EEG emotion recognition method as described in any one of claims 1-6 to analyze the original EEG signal of a specified object and output the predicted emotional state of the specified object; the spatiotemporal complementary learning-based EEG emotion recognition system includes: The brainwave signal acquisition module is used to acquire the raw brainwave signals of a specified object to be identified. The spatial feature extraction module includes a primary capsule generation unit, a high-level capsule generation unit, an emotional capsule generation unit, and a feature compression unit. The primary capsule generation unit first performs a convolution operation on the input raw EEG signal to obtain low-level features, and then transforms these low-level features into primary capsules through a transformation operation. The high-level capsule generation unit performs an affine transformation on the primary capsules according to preset weights to obtain high-level capsules. The emotional capsule generation unit further maps the high-level capsules to emotional relationships through dynamic routing iteration, searching for high-level capsules highly correlated with emotional capsules, and continuously extracting spatial features with high-probability emotional associations to obtain emotional capsules containing spatial information. The feature compression unit uses a nonlinear activation function to compress the emotional capsules to obtain the final spatial features. The time feature extraction module uses an RNN network as a neural network to record time-series information. It receives spatial features at each time step through the RNN network, generates a time feature sequence that encodes the time information of emotional state, and adaptively learns the time feature representation through the backpropagation algorithm, and then outputs the time features hidden in the spatial features. A feature fusion model, which uses a pre-defined connection weight matrix for the complementary learning phase, fuses spatial and temporal features to obtain spatiotemporally complementary high-level sentiment features; and The classification module employs a fully connected layer, which is used to classify the advanced sentiment features after spatiotemporal complementary fusion and output the predicted sentiment state.

8. A brainwave emotion recognition device based on spatiotemporal complementary learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the brainwave emotion recognition method based on spatiotemporal complementary learning as described in any one of claims 1-6, and then predicts the corresponding emotional state based on the brainwave signal of the specified object.

Images