Cross-database electroencephalogram emotion recognition method, device and equipment and storage medium

Abstract

The invention relates to a cross-database electroencephalogram emotion recognition method and device, equipment and a storage medium, and the method comprises the steps: obtaining an unlabeled original electroencephalogram sample, and carrying out the mixed masking of the unlabeled original electroencephalogram sample, and obtaining a mask sample; performing feature embedding and soft contrast learning to generate original embedding features, mask embedding features, original mapping embedding features, mask mapping embedding features and soft contrast loss; performing weighted aggregation on the original embedding features and the mask embedding features to obtain reconstructed embedding features; generating a reconstructed electroencephalogram sample based on the reconstructed embedded features, calculating reconstruction loss based on the reconstructed electroencephalogram sample and the original electroencephalogram sample, and performing self-supervised training on the electroencephalogram recognition model to generate an initial electroencephalogram emotion recognition model; and performing fine adjustment on the initial electroencephalogram emotion recognition model through the labeled electroencephalogram sample to obtain a target electroencephalogram emotion recognition model, and performing recognition on the to-be-detected electroencephalogram sample to generate a target emotion recognition result. According to the method, the electroencephalogram emotion recognition efficiency and accuracy in a cross-database scene are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a cross-database EEG emotion recognition method, device, equipment and storage medium. Background Art

[0002] Emotion is a person's attitude experience towards objective things and the corresponding behavioral response. How to objectively evaluate an individual's emotional state is of great significance in the fields of human-computer interaction, disease diagnosis and rehabilitation. Traditional methods mainly rely on non-physiological signals such as questionnaires, facial expressions or voice for emotion recognition, which are highly subjective and have poor accuracy. Electroencephalography (EEG), as a physiological signal, can provide more direct and objective clues for understanding and estimating emotional states by recording ongoing brain electrode activities from different locations on the scalp. Therefore, in recent years, researchers have been committed to developing various EEG-based emotion recognition methods and have achieved excellent performance in cross-subject and cross-session scenarios.

[0003] Existing methods lack research on EEG emotion recognition in cross-dataset scenarios. Due to the huge distribution differences between different EEG emotion databases, the performance of existing methods will be significantly reduced when they are migrated to cross-database scenarios. In addition, although some researchers have begun to apply transfer learning and domain adaptation techniques to the field of EEG emotion recognition, these methods have the problem of data availability, that is, the model must obtain all labeled source domain data and unlabeled target data in advance for training. Considering the difficulty of EEG signal acquisition and the large amount of expert knowledge and time required to label them, it is obviously impractical to obtain labeled data in advance for model training in real applications. Therefore, how to build a universal EEG emotion decoding model so that the model can still extract robust and generalizable universal EEG representations in the absence of labeled data training, so as to alleviate the huge distribution differences between different databases and improve the model's EEG emotion recognition performance in cross-database scenarios is a huge challenge in the current field of affective computing. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a cross-database EEG emotion recognition method, apparatus, device and storage medium, so as to extract robust and generalizable universal EEG representations in the absence of labeled data training, so as to alleviate the huge distribution differences between different databases and improve the efficiency and accuracy of EEG emotion recognition in cross-database scenarios.

[0005] In order to solve the above technical problems, the present application embodiment provides a cross-database EEG emotion recognition method, including:

[0006] Obtaining unlabeled original EEG samples, and performing mixed masking on the original EEG samples to obtain masked samples;

[0007] The original EEG sample and the mask sample are encoded and mapped and embedded to obtain original embedded features, mask embedded features, original mapped embedded features and mask mapped embedded features;

[0008] Performing soft contrast learning processing based on the original EEG sample, the mask sample, the original mapping embedded features and the mask mapping embedded features to generate a soft contrast loss;

[0009] Calculating the cosine similarity between the original mapping embedding feature and the mask mapping embedding feature, and performing weighted aggregation on the original embedding feature and the mask embedding feature based on the cosine similarity to obtain a reconstructed embedding feature;

[0010] Decoding the reconstructed embedded features to generate reconstructed EEG samples, calculating and generating reconstruction losses based on the reconstructed EEG samples and the original EEG samples, and performing self-supervised training of an EEG recognition model based on the reconstruction losses and the soft contrast losses to generate an initial EEG emotion recognition model;

[0011] Acquire labeled EEG samples, and train and optimize model parameters of the emotion classifier of the initial EEG emotion recognition model based on the labeled EEG samples to obtain a target EEG emotion recognition model;

[0012] Obtain an EEG sample to be tested, and perform emotion recognition on the EEG sample to be tested based on the target EEG emotion recognition model to generate a target emotion recognition result.

[0013] In order to solve the above technical problems, the present application provides a cross-database EEG emotion recognition device, including:

[0014] A sample masking unit, used for obtaining unlabeled original EEG samples and performing mixed masking on the original EEG samples to obtain masked samples;

[0015] A sample embedding unit, used for encoding and mapping the original EEG sample and the mask sample to obtain original embedding features, mask embedding features, original mapping embedding features and mask mapping embedding features;

[0016] A soft contrast loss generating unit, configured to perform soft contrast learning processing based on the original EEG sample, the mask sample, the original mapping embedding feature and the mask mapping embedding feature to generate a soft contrast loss;

[0017] A feature reconstruction unit, configured to calculate the cosine similarity between the original mapping embedded feature and the mask mapping embedded feature, and perform weighted aggregation on the original embedded feature and the mask embedded feature based on the cosine similarity to obtain a reconstructed embedded feature;

[0018] A reconstruction loss calculation unit, used for decoding the reconstructed embedded features to generate a reconstructed EEG sample, calculating a reconstruction loss based on the reconstructed EEG sample and the original EEG sample, and performing self-supervised training of an EEG recognition model based on the reconstruction loss and the soft contrast loss to generate an initial EEG emotion recognition model;

[0019] A model fine-tuning unit, used for acquiring annotated EEG samples, and training and optimizing model parameters of the emotion classifier of the initial EEG emotion recognition model based on the annotated EEG samples, to obtain a target EEG emotion recognition model;

[0020] The emotion recognition unit is used to obtain the EEG sample to be tested, and perform emotion recognition on the EEG sample to be tested based on the target EEG emotion recognition model to generate a target emotion recognition result.

[0021] To solve the above technical problems, a technical solution adopted by the present invention is: to provide a computer device, including one or more processors; a memory for storing one or more programs, so that one or more processors can implement any one of the cross-database EEG emotion recognition methods described above.

[0022] To solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements any one of the above-mentioned cross-database EEG emotion recognition methods.

[0023] The embodiment of the present invention provides a cross-database EEG emotion recognition method, device, equipment and storage medium. The method includes: obtaining unlabeled original EEG samples, and performing mixed masking on the original EEG samples to obtain mask samples; encoding and mapping embedding processing on the original EEG samples and the mask samples to obtain original embedded features, mask embedded features, original mapped embedded features and mask mapped embedded features; performing soft contrast learning processing based on the original EEG samples, the mask samples, the original mapped embedded features and the mask mapped embedded features to generate soft contrast loss; calculating the cosine similarity of the original mapped embedded features and the mask mapped embedded features, and adding the original embedded features and the mask embedded features based on the cosine similarity. Weight aggregation to obtain reconstructed embedded features; decode the reconstructed embedded features to generate reconstructed EEG samples, calculate the reconstruction loss based on the reconstructed EEG samples and the original EEG samples, and perform self-supervised training of the EEG recognition model based on the reconstruction loss and the soft contrast loss to generate an initial EEG emotion recognition model; obtain labeled EEG samples, and train and optimize the model parameters of the emotion classifier of the initial EEG emotion recognition model based on the labeled EEG samples to obtain the target EEG emotion recognition model; obtain the EEG samples to be tested, and perform emotion recognition on the EEG samples to be tested based on the target EEG emotion recognition model to generate the target emotion recognition result. The embodiment of the present invention performs soft contrast mask pre-training on the model through unlabeled original EEG samples, so as to achieve the extraction of robust and generalizable universal EEG representations in the absence of labeled data training, so as to alleviate the huge distribution differences between different databases and improve the efficiency and accuracy of EEG emotion recognition in cross-database scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 It is a flowchart of the implementation process of the cross-database EEG emotion recognition method provided in the embodiment of the present application;

[0026] Figure 2 It is a flowchart of the implementation of the sub-process in the cross-database EEG emotion recognition method provided in the embodiment of the present application;

[0027] Figure 3 Schematic diagram of random mask, channel mask and mixed mask fusion strategy provided in the embodiment of the present application;

[0028] Figure 4 It is a flowchart of the implementation of the sub-process in the cross-database EEG emotion recognition method provided in the embodiment of the present application;

[0029] Figure 5 It is a flowchart of the implementation of the sub-process in the cross-database EEG emotion recognition method provided in the embodiment of the present application;

[0030] Figure 6 It is a flowchart of the implementation of the sub-process in the cross-database EEG emotion recognition method provided in the embodiment of the present application;

[0031] Figure 7 It is a flowchart of the implementation of the sub-process in the cross-database EEG emotion recognition method provided in the embodiment of the present application;

[0032] Figure 8 It is a flowchart of the implementation of the sub-process in the cross-database EEG emotion recognition method provided in the embodiment of the present application;

[0033] Fig. 9 This is a schematic diagram of a cross-database EEG emotion recognition device provided in an embodiment of the present application;

[0034] Fig.10 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0036] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0038] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0039] It should be noted that the cross-database EEG emotion recognition method provided in the embodiment of the present application is generally executed by a server, and accordingly, the cross-database EEG emotion recognition device is generally configured in the server.

[0040] See also Figure 1 , Figure 1 A specific implementation of the cross-database EEG emotion recognition method is shown.

[0041] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 1 The process sequence shown is limited to the following steps:

[0042] S1: Obtain unlabeled original EEG samples, and perform mixed masking on the original EEG samples to obtain masked samples.

[0043] Specifically, the present application provides a cross-database EEG emotion recognition method, which is an emotion recognition method of a soft contrast masked modeling framework (Soft Contrast ive Masked Modeling, SCMM) that combines contrast learning and mask modeling, and is used for EEG signal emotion recognition tasks in cross-database scenarios.

[0044] In the embodiment of the present application, the unlabeled original EEG sample x is obtained. i , and mix and mask the original EEG samples through a variety of different masking methods to produce a richer and more diverse mask sample In a specific embodiment, the unlabeled raw EEG sample x i Multi-channel EEG signal samples for pre-training EEG emotion database Among them, C is the number of channels, and F is the feature dimension of each channel.

[0045] See also Figure 2 and Figure 3 , Figure 2 A specific implementation of step S1 is shown. Figure 3 Schematic diagram of random mask, channel mask and mixed mask fusion strategy provided by the embodiment of the present application, which is described in detail as follows:

[0046] S11: Obtain the unlabeled original EEG samples from the pre-trained EEG emotion database.

[0047] S12: Generate a random mask matrix and a channel mask matrix according to the number of channels and feature dimensions corresponding to the original EEG sample.

[0048] S13: Modify the features of the positions corresponding to the original EEG samples according to the random mask matrix and the channel mask matrix respectively to obtain modified EEG samples.

[0049] Specifically, the existing method is to use a random masking method to randomly mask each dimension of each channel of the original EEG sample with a certain probability (the element value of the masked feature dimension is set to 0); or to use a channel masking method to randomly mask the features of all dimensions of each channel of the original EEG sample with a certain probability (the element values ​​of all feature dimensions of the masked channel are set to 0). However, for random masking, since it randomly masks the features, it is impossible to capture the relationship between different channels; for channel masking, since it randomly masks all dimensional features of certain channels, it will ignore the relationship between features of different dimensions. Therefore, both methods will lead to modeling difficulties and cannot simultaneously capture the relationship between channels and features of multi-channel EEG signals.

[0050] In view of the difficulty of modeling in existing methods and the inability to simultaneously capture the relationship between channels and features of multi-channel EEG signals, the present application embodiment provides a hybrid masking strategy to generate more diverse mask samples. In the present application embodiment, a random mask matrix MASK with a shape of C×F is constructed by using two binomial distributions. R ∈{0, 1} and a channel mask matrix MASK of shape C×1 C ∈{0, 1}. According to the random mask matrix MASK R At the position of the element with a median value of 0, the eigenvalue of the corresponding position of the original EEG sample is set to 0; then according to the channel mask matrix MASK C At the position where the element with a median value of 0 is located, the characteristic values ​​of all dimensions of the corresponding channel of the original EEG sample are set to 0, thereby modifying the characteristics of the position corresponding to the original EEG sample and obtaining the modified EEG sample.

[0051] S14: constructing a uniformly distributed probability matrix, and performing mixed masking on the modified EEG samples based on the probability matrix and a preset probability threshold to obtain the masked samples.

[0052] Specifically, a probability matrix U with a shape of C×1 is constructed using a uniform distribution method, and the modified EEG samples are mixed and masked based on the probability matrix and the preset probability threshold μ to obtain masked samples. The specific implementation method is to add a random mask matrix MASK according to the position of the element whose value is greater than (1-μ) in the probability matrix U. RIn the , select the element values ​​of all dimensions of the corresponding channel, and randomly mask the feature dimensions of the modified EEG sample; then according to the position of the element whose value is less than or equal to (1-μ) in the probability matrix U, in the channel mask matrix MASK C , select the element values ​​of all feature dimensions of the corresponding channel, perform random masking of the channel dimension on the modified EEG sample, and obtain a masked sample. The embodiment of the present application can realize an input multi-channel EEG sample, and obtain a masked sample that combines random masking and channel masking. Compared with the use of random masking or channel masking alone, the embodiment of the present application uses a hybrid masking strategy to generate richer and more diverse masked samples, and urges the model to simultaneously capture the relationship between the different dimensional features and channels of the sample.

[0053] S2: Encoding and mapping embedding the original EEG sample and the mask sample to obtain original embedding features, mask embedding features, original mapping embedding features and mask mapping embedding features.

[0054] See also Figure 4 , Figure 4 A specific implementation of step S2 is shown, which is described in detail as follows:

[0055] S21: Input the original EEG sample and the mask sample into an encoder composed of a convolutional neural network for encoding to obtain the original embedded features and the mask embedded features.

[0056] S22: Projecting the original embedding feature and the mask embedding feature to a latent space through a mapper composed of fully connected layers to obtain the original mapping embedding feature and the mask mapping embedding feature.

[0057] Specifically, the original EEG samples and the mask samples generated by the mixed mask are input into an encoder E composed of a convolutional neural network for encoding to obtain the original embedded feature h i and mask embedding features That is: h i =E(x i )and Then, the original embedding feature h is transformed through a mapper P consisting of fully connected layers. i and mask embedding features Projected to the latent space, the original mapping embedding feature and the mask mapping embedding feature are obtained, namely: z i =P(h i )and

[0058] S3: Perform soft contrast learning processing based on the original EEG sample, the mask sample, the original mapping embedded features and the mask mapping embedded features to generate a soft contrast loss.

[0059] Specifically, traditional contrastive learning defines the same sample and its mask sample as a positive sample pair, and defines different samples and different mask samples as negative sample pairs. When calculating the contrast loss, the weight of the positive sample pair is set to 1, and the weight of the negative sample pair is set to 0. During the training process, the model shortens the distance between the positive sample pairs and moves the negative sample pairs away from each other by optimizing the contrast loss. However, since human emotions and EEG signals have short-term continuity and long-term similarity, if the traditional contrastive learning method is used, the model will mistakenly distance similar but different sample pairs, thereby failing to capture similar features of EEG signals in different time dimensions. Therefore, the embodiment of the present application improves the traditional contrastive learning and provides a soft contrastive learning strategy for EEG signals to better model the relationship between different EEG samples.

[0060] See also Figure 5 , Figure 5 A specific implementation of step S3 is shown, which is described in detail as follows:

[0061] S31: constructing a sample pair based on the original EEG sample and the mask sample.

[0062] S32: Calculate the normalized distance between the sample pairs in the original data space.

[0063] S33: Perform nonlinear mapping on the normalized distance to obtain a mapping value, and use the mapping value as a soft weight allocation value between the sample pairs.

[0064] S34: Calculate the relative similarity between the mapping embedding features based on the original EEG sample, the mask sample, the original mapping embedding feature and the mask mapping embedding feature.

[0065] S35: performing loss calculation based on the soft weight allocation value and the relative similarity according to a preset soft contrast loss calculation formula to obtain the soft contrast loss.

[0066] Specifically, a sample pair (x i , x j ), calculate the sample pair (x i , x j ) is the normalized distance Dist(x) in the original data space i , x j ). Normalized distance Dist(x i , x j ) is calculated as follows:

[0067] Dist(x i , x j)=Norm(-sim(x i , x j ))∈[0,1];

[0068]

[0069] Among them, Dist(x i , x j ) represents the normalized distance between the sample pair and , and Norm represents the normalization operation. Here, the embodiment of the present application uses the cosine similarity sim(x i , x j ) is used as the distance metric between sample pairs.

[0070] The embodiment of the present application uses the σ activation function to perform nonlinear mapping on the normalized distance, and uses the mapping value as the soft weight distribution value w(x i , x j ). The soft weight assignment value w(x i , x j ) is calculated as follows:

[0071] w(x i , x j )=2α·σ(-β'Dist(x i , x j ));

[0072]

[0073] Among them, β is a hyperparameter used to adjust the sharpness. The larger the β value, the greater the weight difference between sample pairs, and vice versa. α∈[0, 1] is a hyperparameter that controls the boundary and is used to limit the upper bound of the soft weight distribution value between sample pairs.

[0074] The preset soft contrast loss calculation formula is as follows:

[0075]

[0076] Among them, x j (j≠i, j≠i+B) represents the same as sample x i Different raw EEG samples x j and its mask sample z j (j≠i, j≠i+B) represents the same as sample x i Different raw EEG samples x j The mapping embedding feature z j and its mask sample The mapping embedding features B is the training batch size, The embodiment of the present application introduces soft weight allocation for different sample pairs, so that the model can better capture the short-term continuity and long-term similarity characteristics of EEG signals, thereby better modeling the relationship between different samples. During the training process, the soft contrast loss of all samples in the training batch is summed and averaged as the final soft contrast loss for model optimization.

[0077] S4: Calculate the cosine similarity between the original mapping embedding feature and the mask mapping embedding feature, and perform weighted aggregation on the original embedding feature and the mask embedding feature based on the cosine similarity to obtain a reconstructed embedding feature.

[0078] Specifically, the traditional Masked Auto Encoder (MAE) performs a single reconstruction of the masked sample to restore it to an unmasked sample, and calculates the reconstruction loss (MSE) between the original sample and the reconstructed sample for modeling. However, due to the randomness of the masked sample in the masking process, using a lower mask ratio will make it difficult for the model to learn robust semantic features, while a higher mask ratio will lead to difficulties in model reconstruction. Both methods will seriously affect the performance and generalization ability of the model. Therefore, the embodiment of the present application goes beyond the traditional mask modeling paradigm, and improves the process of reconstructing the original sample using a single mask sample in the traditional method to reconstruct the original sample using a weighted aggregation method of all samples in the training batch.

[0079] See also Figure 6 , Figure 6 A specific implementation of step S4 is shown, which is described in detail as follows:

[0080] S41: Calculate the cosine similarity of the original mapping embedding feature and the mask mapping embedding feature in all sample pairs in the same batch.

[0081] S42: constructing a similarity matrix based on the cosine similarity.

[0082] S43: performing weighted aggregation on the original embedded features and the mask embedded features according to the similarity matrix to reconstruct the original embedded features to obtain the reconstructed embedded features.

[0083] Specifically, calculate the original mapping embedding feature z of all sample pairs in the same batch i and the mask map embedding feature z j The pairwise cosine similarity sim(z i , z j). Then, the pairwise cosine similarities of all sample pairs form a similarity matrix, and the original embedding features and mask embedding features are weightedly aggregated according to the similarity matrix to reconstruct the original embedding features to obtain the reconstructed embedding features. For an original EEG sample x i , reconstruct the embedded features The weighted aggregation process can be expressed as:

[0084]

[0085] Among them, h j (j≠i) represents the mapping embedding feature z j Corresponding embedded features. In the weighted aggregation calculation process, the embodiment of the present application excludes the embedded features of the original sample for weighted aggregation to avoid the sample itself dominating the model training process. By introducing the weighted aggregation process and under the soft contrast learning constraint, the model can fully fuse the embedded features of different EEG samples while suppressing the interference of irrelevant noise sequences, thereby accurately learning the similar representation of the masked sample and the original sample, and absorbing the similar features of different EEG samples.

[0086] S5: Decode the reconstructed embedded features to generate reconstructed EEG samples, calculate and generate reconstruction losses based on the reconstructed EEG samples and the original EEG samples, and perform self-supervised training of the EEG recognition model based on the reconstruction loss and the soft contrast loss to generate an initial EEG emotion recognition model.

[0087] See also Figure 7 , Figure 7 A specific implementation of step S5 is shown, which is described in detail as follows:

[0088] S51: Decode the reconstructed embedded features through a decoder composed of fully connected layers to obtain the reconstructed EEG samples.

[0089] S52: Calculate the mean square error between the reconstructed EEG sample and the original EEG sample to obtain the reconstruction loss.

[0090] S53: Perform self-supervised training of the EEG recognition model based on the reconstruction loss and the soft contrast loss to generate the initial EEG emotion recognition model.

[0091] Specifically, the reconstructed embedding features are processed by a decoder D composed of a fully connected layer. Decode to reconstruct the embedded features Restore to reconstruct EEG samples The mean square error (MSE) is used as the reconstruction loss for model optimization to narrow the distance between the original sample and the reconstructed sample. The calculation formula is:

[0092]

[0093] During the training process, all samples in the training batch are Sum and average the results as the reconstruction loss for the final model optimization. Finally, by jointly optimizing the soft contrast loss and reconstruction loss Perform self-supervised pre-training. Therefore, the total loss function of model pre-training is:

[0094]

[0095] Among them, λ C and λ R is a predefined trade-off parameter used to control the weights of the two loss terms. During the training process, the embodiment of the present application adopts an adaptive adjustment strategy, through the adaptive adjustment of the homoscedastic uncertainty of each loss term and λ C and λ R Finally, the present embodiment of the invention passes the total loss Conduct self-supervised training of the EEG recognition model to generate an initial EEG emotion recognition model.

[0096] S6: Obtain labeled EEG samples, and train and optimize model parameters of the emotion classifier of the initial EEG emotion recognition model based on the labeled EEG samples to obtain a target EEG emotion recognition model.

[0097] Specifically, the embodiment of the present application is a supervised fine-tuning stage of the initial EEG emotion recognition model. By obtaining a small amount of annotated EEG samples, and training the emotion classifier of the initial EEG emotion recognition model and optimizing the model parameters based on the annotated EEG samples, the target EEG emotion recognition model is obtained.

[0098] See also Figure 8 , Figure 8 A specific implementation of step S6 is shown, which is described in detail as follows:

[0099] S61: Obtain the labeled EEG samples from the fine-tuned EEG emotion dataset.

[0100] S62: Extracting the embedding features of the labeled EEG samples through the encoder in the initial EEG emotion recognition model to obtain basic embedding features.

[0101] S63: Reshape the basic embedding features into a one-dimensional feature vector by means of a Flatten tiling operation.

[0102] S64: Input the one-dimensional feature vector into the emotion classifier of the initial EEG emotion recognition model, train the emotion classifier and update the model parameters by minimizing the cross entropy loss function, and obtain the target EEG emotion recognition model.

[0103] Specifically, a small number of labeled samples are obtained from the fine-tuned EEG emotion dataset, and the labeled samples are input into the pre-trained encoder E to extract the corresponding embedding features and obtain the basic embedding features. The basic embedding features are reshaped into a one-dimensional feature vector by the Flatten operation; finally, the one-dimensional feature vector is input into an emotion classifier C composed of two fully connected layers, and the emotion classifier is trained and the model parameters are optimized by minimizing the cross entropy loss function to obtain the target EEG emotion recognition model.

[0104] Among them, the calculation formula of the cross entropy loss function is as follows:

[0105]

[0106] Among them, p m，n represents the probability that the mth EEG sample belongs to the nth category, y m，n ∈{0, 1}, M is the total number of labeled samples used for model fine-tuning, and N is the total number of emotion categories in the fine-tuning dataset. By optimizing the classification loss function The model will accurately classify emotions for each input EEG signal sample. In the supervised fine-tuning stage, in order to quickly deploy and test the EEG emotion recognition model, the embodiment of the present application only performs model fine-tuning and parameter updates by optimizing the classification loss, and no longer calculates and optimizes the soft contrast loss and reconstruction loss. Finally, the fine-tuned model is applied to the test EEG emotion database for the final EEG emotion recognition test.

[0107] S7: Acquire an EEG sample to be tested, and perform emotion recognition on the EEG sample to be tested based on the target EEG emotion recognition model to generate a target emotion recognition result.

[0108] Specifically, when emotion recognition is required, an EEG sample to be tested is obtained, and emotion recognition is performed on the EEG sample to be tested based on the target EEG emotion recognition model to generate a target emotion recognition result, wherein the target emotion recognition result is any one of positive, neutral, and negative.

[0109] In an embodiment of the present application, an unlabeled original EEG sample is obtained, and a mixed mask is performed on the original EEG sample to obtain a mask sample; the original EEG sample and the mask sample are encoded and mapped and embedded to obtain an original embedded feature, a mask embedded feature, an original mapped embedded feature, and a mask mapped embedded feature; a soft contrast learning process is performed based on the original EEG sample, the mask sample, the original mapped embedded feature, and the mask mapped embedded feature to generate a soft contrast loss; the cosine similarity of the original mapped embedded feature and the mask mapped embedded feature is calculated, and the original embedded feature and the mask embedded feature are added based on the cosine similarity. Weight aggregation to obtain reconstructed embedded features; decode the reconstructed embedded features to generate reconstructed EEG samples, calculate the reconstruction loss based on the reconstructed EEG samples and the original EEG samples, and perform self-supervised training of the EEG recognition model based on the reconstruction loss and the soft contrast loss to generate an initial EEG emotion recognition model; obtain labeled EEG samples, and train and optimize the model parameters of the emotion classifier of the initial EEG emotion recognition model based on the labeled EEG samples to obtain the target EEG emotion recognition model; obtain the EEG samples to be tested, and perform emotion recognition on the EEG samples to be tested based on the target EEG emotion recognition model to generate the target emotion recognition result. The embodiment of the present invention performs soft contrast mask pre-training on the model through unlabeled original EEG samples, so as to achieve the extraction of robust and generalizable universal EEG representations in the absence of labeled data training, so as to alleviate the huge distribution differences between different databases and improve the efficiency and accuracy of EEG emotion recognition in cross-database scenarios.

[0110] Please refer to Fig. 9 , as a response to the above Figure 1 The present application provides an embodiment of a cross-database EEG emotion recognition device, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0111] like Fig. 9 As shown, the cross-database EEG emotion recognition device of this embodiment includes: a sample mask unit 81, a sample embedding unit 82, a soft contrast loss generation unit 83, a feature reconstruction unit 84, a reconstruction loss calculation unit 85, a model fine-tuning unit 86 and an emotion recognition unit 87, wherein:

[0112] The sample masking unit 81 is used to obtain unlabeled original EEG samples and perform mixed masking on the original EEG samples to obtain masked samples;

[0113] A sample embedding unit 82 is used to encode and map the original EEG sample and the mask sample to obtain an original embedding feature, a mask embedding feature, an original mapping embedding feature and a mask mapping embedding feature;

[0114] A soft contrast loss generating unit 83, configured to perform soft contrast learning processing based on the original EEG sample, the mask sample, the original mapping embedded feature and the mask mapping embedded feature to generate a soft contrast loss;

[0115] A feature reconstruction unit 84, configured to calculate the cosine similarity between the original mapping embedded feature and the mask mapping embedded feature, and perform weighted aggregation on the original embedded feature and the mask embedded feature based on the cosine similarity to obtain a reconstructed embedded feature;

[0116] A reconstruction loss calculation unit 85 is used to decode the reconstructed embedded features to generate a reconstructed EEG sample, calculate and generate a reconstruction loss based on the reconstructed EEG sample and the original EEG sample, and perform self-supervised training of the EEG recognition model based on the reconstruction loss and the soft contrast loss to generate an initial EEG emotion recognition model;

[0117] A model fine-tuning unit 86 is used to obtain annotated EEG samples, and train and optimize model parameters of the emotion classifier of the initial EEG emotion recognition model based on the annotated EEG samples to obtain a target EEG emotion recognition model;

[0118] The emotion recognition unit 87 is used to obtain the EEG sample to be tested, and perform emotion recognition on the EEG sample to be tested based on the target EEG emotion recognition model to generate a target emotion recognition result.

[0119] Further, the sample mask unit 81 comprises:

[0120] An original EEG sample acquisition unit, used to acquire the unlabeled original EEG samples from a pre-trained EEG emotion database;

[0121] A mask matrix construction unit, used to generate a random mask matrix and a channel mask matrix according to the number of channels and feature dimensions corresponding to the original EEG sample;

[0122] A feature modification unit, used to modify the features of the positions corresponding to the original EEG samples according to the random mask matrix and the channel mask matrix respectively, to obtain modified EEG samples;

[0123] The mixed mask unit is used to construct a uniformly distributed probability matrix, and perform mixed masking on the modified EEG sample based on the probability matrix and a preset probability threshold to obtain the mask sample.

[0124] Further, the sample embedding unit 82 includes:

[0125] A feature embedding unit, used for inputting the original EEG sample and the mask sample into an encoder composed of a convolutional neural network for encoding, so as to obtain the original embedded feature and the mask embedded feature;

[0126] A feature mapping unit is used to project the original embedding feature and the mask embedding feature into a latent space through a mapper composed of a fully connected layer to obtain the original mapping embedding feature and the mask mapping embedding feature.

[0127] Further, the soft contrast loss generating unit 83 includes:

[0128] A sample pair construction unit, used for constructing a sample pair based on the original EEG sample and the mask sample;

[0129] A normalized distance calculation unit, used to calculate the normalized distance of the sample pair in the original data space;

[0130] A soft weight allocation unit, configured to perform nonlinear mapping on the normalized distance to obtain a mapping value, and use the mapping value as a soft weight allocation value between the sample pairs;

[0131] A similarity calculation unit, configured to calculate a relative similarity between mapping embedding features based on the original EEG sample, the mask sample, the original mapping embedding feature and the mask mapping embedding feature;

[0132] The first loss calculation unit is used to perform loss calculation based on the soft weight allocation value and the relative similarity according to a preset soft contrast loss calculation formula to obtain the soft contrast loss.

[0133] Furthermore, the feature reconstruction unit 84 includes:

[0134] A cosine similarity calculation unit, used to calculate the cosine similarity of the original mapping embedding feature and the mask mapping embedding feature in all sample pairs in the same batch;

[0135] A similarity matrix construction unit, used to construct a similarity matrix based on the cosine similarity;

[0136] A weighted aggregation unit is used to perform weighted aggregation on the original embedded features and the mask embedded features according to the similarity matrix to reconstruct the original embedded features to obtain the reconstructed embedded features.

[0137] Furthermore, the reconstruction loss calculation unit 85 includes:

[0138] A feature restoration unit, used for decoding the reconstructed embedded features through a decoder composed of fully connected layers to obtain the reconstructed EEG samples;

[0139] A second loss calculation unit, used for calculating the mean square error between the reconstructed EEG sample and the original EEG sample to obtain the reconstruction loss;

[0140] An initial model generating unit is used to perform self-supervised training of the electroencephalogram recognition model based on the reconstruction loss and the soft contrast loss to generate the initial electroencephalogram emotion recognition model.

[0141] Furthermore, the model fine-tuning unit 86 includes:

[0142] A labeled EEG sample acquisition unit, used to acquire the labeled EEG sample from the fine-tuned EEG emotion dataset;

[0143] A basic embedding feature generating unit, used for extracting the embedding features of the labeled EEG samples through the encoder in the initial EEG emotion recognition model to obtain basic embedding features;

[0144] A feature reshaping unit, used for reshaping the basic embedding feature into a one-dimensional feature vector by means of a Flatten tiling operation;

[0145] The target model generation unit is used to input the one-dimensional feature vector into the emotion classifier of the initial EEG emotion recognition model, train the emotion classifier and update the model parameters by minimizing the cross entropy loss function, and obtain the target EEG emotion recognition model.

[0146] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0147] The computer device 7 includes a memory 91, a processor 92, and a network interface 93 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 7 having three components: a memory 91, a processor 92, and a network interface 93, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0148] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0149] The memory 91 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 91 can be an internal storage unit of the computer device 7, such as a hard disk or memory of the computer device 7. In other embodiments, the memory 91 can also be an external storage device of the computer device 7, such as a plug-in hard disk equipped on the computer device 7, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 91 can also include both the internal storage unit of the computer device 7 and its external storage device. In this embodiment, the memory 91 is generally used to store the operating system and various application software installed on the computer device 7, such as the program code of the cross-database EEG emotion recognition method, etc. In addition, the memory 91 can also be used to temporarily store various types of data that have been output or are to be output.

[0150] The processor 92 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 92 is generally used to control the overall operation of the computer device 7. In this embodiment, the processor 92 is used to run the program code stored in the memory 91 or process data, such as running the program code of the cross-database EEG emotion recognition method to implement various embodiments of the cross-database EEG emotion recognition method.

[0151] The network interface 93 may include a wireless network interface or a wired network interface, and the network interface 93 is generally used to establish a communication connection between the computer device 7 and other electronic devices.

[0152] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to perform the steps of a cross-database EEG emotion recognition method as described above.

[0153] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0154] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A cross-database EEG emotion recognition method, characterized in that: include: Obtaining unlabeled original EEG samples, and performing mixed masking on the original EEG samples to obtain masked samples; The original EEG sample and the mask sample are encoded and mapped and embedded to obtain original embedded features, mask embedded features, original mapped embedded features and mask mapped embedded features; Performing soft contrast learning processing based on the original EEG sample, the mask sample, the original mapping embedded features and the mask mapping embedded features to generate a soft contrast loss; Calculating the cosine similarity between the original mapping embedding feature and the mask mapping embedding feature, and performing weighted aggregation on the original embedding feature and the mask embedding feature based on the cosine similarity to obtain a reconstructed embedding feature; Decoding the reconstructed embedded features to generate reconstructed EEG samples, calculating and generating reconstruction losses based on the reconstructed EEG samples and the original EEG samples, and performing self-supervised training of an EEG recognition model based on the reconstruction losses and the soft contrast losses to generate an initial EEG emotion recognition model; Acquire labeled EEG samples, and train and optimize model parameters of the emotion classifier of the initial EEG emotion recognition model based on the labeled EEG samples to obtain a target EEG emotion recognition model; Obtain an EEG sample to be tested, and perform emotion recognition on the EEG sample to be tested based on the target EEG emotion recognition model to generate a target emotion recognition result.

2. The cross-database EEG emotion recognition method according to claim 1, characterized in that: The step of obtaining an unlabeled original EEG sample and performing mixed masking on the original EEG sample to obtain a masked sample includes: Obtaining the unlabeled original EEG samples from a pre-trained EEG emotion database; Generate a random mask matrix and a channel mask matrix according to the number of channels and feature dimensions corresponding to the original EEG sample; Modifying the features of the positions corresponding to the original EEG samples according to the random mask matrix and the channel mask matrix respectively to obtain modified EEG samples; A uniformly distributed probability matrix is ​​constructed, and a mixed mask is performed on the modified EEG sample based on the probability matrix and a preset probability threshold to obtain the mask sample.

3. The cross-database EEG emotion recognition method according to claim 1, characterized in that: The encoding and mapping embedding processing of the original EEG sample and the mask sample to obtain the original embedding feature, the mask embedding feature, the original mapping embedding feature and the mask mapping embedding feature includes: Inputting the original EEG sample and the mask sample into an encoder composed of a convolutional neural network for encoding to obtain the original embedded features and the mask embedded features; The original embedding feature and the mask embedding feature are projected into a latent space through a mapper composed of fully connected layers to obtain the original mapping embedding feature and the mask mapping embedding feature.

4. The cross-database EEG emotion recognition method according to claim 1, characterized in that: The performing soft contrast learning processing based on the original EEG sample, the mask sample, the original mapping embedding feature and the mask mapping embedding feature to generate a soft contrast loss includes: Constructing a sample pair based on the original EEG sample and the mask sample; Calculating the normalized distance of the sample pair in the original data space; Performing nonlinear mapping on the normalized distance to obtain a mapping value, and using the mapping value as a soft weight distribution value between the sample pairs; Calculating the relative similarity between the mapping embedding features based on the original EEG sample, the mask sample, the original mapping embedding feature and the mask mapping embedding feature; The soft contrast loss is obtained by performing loss calculation based on the soft weight allocation value and the relative similarity according to a preset soft contrast loss calculation formula.

5. The cross-database EEG emotion recognition method according to claim 1, characterized in that: The calculating the cosine similarity of the original mapping embedding feature and the mask mapping embedding feature, and performing weighted aggregation on the original embedding feature and the mask embedding feature based on the cosine similarity to obtain the reconstructed embedding feature, comprises: Calculating the cosine similarity of the original mapping embedding feature and the mask mapping embedding feature in all sample pairs in the same batch; Constructing a similarity matrix based on the cosine similarity; The original embedded features and the mask embedded features are weightedly aggregated according to the similarity matrix to reconstruct the original embedded features to obtain the reconstructed embedded features.

6. The cross-database EEG emotion recognition method according to any one of claims 1 to 5, characterized in that: The decoding of the reconstructed embedded features to generate a reconstructed EEG sample, calculating a reconstruction loss based on the reconstructed EEG sample and the original EEG sample, and performing self-supervised training of an EEG recognition model based on the reconstruction loss and the soft contrast loss to generate an initial EEG emotion recognition model, including: Decoding the reconstructed embedded features through a decoder composed of fully connected layers to obtain the reconstructed EEG samples; Calculating the mean square error between the reconstructed EEG sample and the original EEG sample to obtain the reconstruction loss; The electroencephalogram (EEG) recognition model is trained by self-supervision based on the reconstruction loss and the soft contrast loss to generate the initial EEG emotion recognition model.

7. The cross-database EEG emotion recognition method according to any one of claims 1 to 5, characterized in that: The step of obtaining the labeled EEG samples, and training the emotion classifier of the initial EEG emotion recognition model and updating the model parameters based on the labeled EEG samples to obtain the target EEG emotion recognition model includes: Obtain the labeled EEG samples from the fine-tuned EEG emotion dataset; Extracting the embedding features of the labeled EEG samples through an encoder in the initial EEG emotion recognition model to obtain basic embedding features; Reshape the basic embedding features into a one-dimensional feature vector by means of a Flatten tiling operation; The one-dimensional feature vector is input into the emotion classifier of the initial EEG emotion recognition model, and the emotion classifier is trained and the model parameters are updated by minimizing the cross entropy loss function to obtain the target EEG emotion recognition model.

8. A cross-database EEG emotion recognition device, characterized in that: include: A sample masking unit, used for obtaining unlabeled original EEG samples and performing mixed masking on the original EEG samples to obtain masked samples; A sample embedding unit, used for encoding and mapping the original EEG sample and the mask sample to obtain original embedding features, mask embedding features, original mapping embedding features and mask mapping embedding features; A soft contrast loss generating unit, configured to perform soft contrast learning processing based on the original EEG sample, the mask sample, the original mapping embedding feature and the mask mapping embedding feature to generate a soft contrast loss; A feature reconstruction unit, configured to calculate the cosine similarity between the original mapping embedded feature and the mask mapping embedded feature, and perform weighted aggregation on the original embedded feature and the mask embedded feature based on the cosine similarity to obtain a reconstructed embedded feature; A reconstruction loss calculation unit, used for decoding the reconstructed embedded features to generate a reconstructed EEG sample, calculating a reconstruction loss based on the reconstructed EEG sample and the original EEG sample, and performing self-supervised training of an EEG recognition model based on the reconstruction loss and the soft contrast loss to generate an initial EEG emotion recognition model; A model fine-tuning unit, used for acquiring annotated EEG samples, and training and optimizing model parameters of the emotion classifier of the initial EEG emotion recognition model based on the annotated EEG samples, to obtain a target EEG emotion recognition model; The emotion recognition unit is used to obtain the EEG sample to be tested, and perform emotion recognition on the EEG sample to be tested based on the target EEG emotion recognition model to generate a target emotion recognition result.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the cross-database EEG emotion recognition method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the cross-database EEG emotion recognition method according to any one of claims 1 to 7 is implemented.

Images