Multi-regularization self-supervised training method and system for emotion recognition model

Abstract

The embodiment of the invention provides a multi-regularization self-supervised training method and system for an emotion recognition model. The method comprises the following steps: inputting an electroencephalogram sample into a first emotion recognition model based on masking channel modeling of time spectrum transformation and a second emotion recognition model based on emotion contrast learning; performing emotion recognition on the electroencephalogram sample by using a first emotion recognition model and a second emotion recognition model to obtain a predicted emotion classification result and emotion classification loss; determining a logic value of the electroencephalogram sample through an encoder of the first emotion recognition model and a classifier head, and determining an emotion feature and regularization loss of the electroencephalogram sample through an encoder of the second emotion recognition model and a projection head; and performing self-supervision training based on the emotion classification loss and the regularization loss. According to the multi-regularization self-supervised training in the embodiment of the invention, time-frequency spectrum transformation masking channel modeling and emotion comparison learning pre-training are adopted respectively, so that more accurate emotion information can be captured.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface, and in particular to a multi-regularized self-supervised training method and system for an emotion recognition model. Background Art

[0002] Understanding human emotions is a fundamental aspect of human-computer interaction, emotional computing, and psychology research. To achieve this goal, various modalities have been utilized, including non-physiological signals such as facial expressions and speech, as well as physiological signals such as EEG (electroencephalogram) and electrocardiogram. Among the many modalities, EEG has attracted much attention due to its non-invasiveness, high temporal resolution, and direct correlation with neural activity. A mainstream approach to studying human emotions is to use computational methods, such as deep neural networks, to classify pre-defined emotion categories evoked by video stimuli. Notable benchmarks include the SEED series, which contains EEG recordings across multiple sessions of multiple subjects responding to carefully selected videos. The state of the art has explored various powerful neural network architectures in these benchmarks, resulting in significant progress in the field of EEG-based emotion recognition.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art:

[0004] The existing technology ignores the complexity of such paradigms for eliciting emotions, which brings many defects. The first is emotion label noise, which is due to the assumption that emotions are stimulated and stable throughout the process of watching videos. This assumption is difficult to achieve in actual experimental settings, and often leads to differences between EEG signals and the expected induced emotional states. Secondly, it is also a difficult problem to fully capture the time-space-spectral characteristics of EEG signals. This is because a comprehensive understanding and interpretation of these multifaceted characteristics are challenging in the field of emotion recognition research. Third, low signal-to-noise ratio remains a problem. The acquisition of EEG signals is easily interfered by biological and environmental noise, which hinders the acquisition of accurate EEG recordings and hinders the ability to accurately capture EEG representations of information related to emotions. Summary of the invention

[0005] In order to at least solve the problem in the prior art that the emotion label noise causes a difference between the EEG signal and the expected emotional state, it is difficult to capture the time-space-spectrum characteristics of the EEG signal and it is difficult to accurately capture the information related to the emotion.

[0006] In a first aspect, an embodiment of the present invention provides a multi-regularized self-supervised training method for an emotion recognition model, comprising:

[0007] inputting the EEG samples into a first emotion recognition model based on masked channel modeling of time-spectral transformation and a second emotion recognition model based on emotion contrast learning;

[0008] Using the first emotion recognition model and the second emotion recognition model to identify the emotion of the electroencephalogram sample, obtain a predicted emotion classification result, and obtain an emotion classification loss for supervised learning based on the pre-prepared real emotion classification result and the predicted emotion classification result;

[0009] Determine the logic value of the electroencephalogram sample by a first encoder and a classifier head of the first emotion recognition model, determine the emotional feature of the electroencephalogram sample by a second encoder and a projection head of the second emotion recognition model, and determine a regularization loss for offsetting noise labels based on the logic value and the emotional feature;

[0010] Based on the emotion classification loss and the regularization loss, multi-regularized self-supervised training is performed on the first emotion recognition model and the second emotion recognition model until preset training requirements are met.

[0011] In a second aspect, an embodiment of the present invention provides a multi-regularized self-supervised training system for an emotion recognition model, comprising:

[0012] A data input module, for inputting EEG samples into a first emotion recognition model based on masked channel modeling of time-spectral transformation, and a second emotion recognition model based on emotion contrast learning;

[0013] A classification loss determination module, configured to use the first emotion recognition model and the second emotion recognition model to perform emotion recognition on the electroencephalogram sample to obtain a predicted emotion classification result, and to obtain an emotion classification loss for supervised learning based on a pre-prepared real emotion classification result and the predicted emotion classification result;

[0014] a regularization loss determination module, configured to determine a logic value of the electroencephalogram sample through a first encoder and a classifier head of the first emotion recognition model, determine an emotional feature of the electroencephalogram sample through a second encoder and a projection head of the second emotion recognition model, and determine a regularization loss for offsetting noise labels based on the logic value and the emotional feature;

[0015] A training module is used to perform multi-regularized self-supervised training on the first emotion recognition model and the second emotion recognition model based on the emotion classification loss and the regularization loss until preset training requirements are met.

[0016] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the multi-regularized self-supervised training method for an emotion recognition model according to any embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the multi-regularized self-supervised training method for an emotion recognition model of any embodiment of the present invention are implemented.

[0018] In a fifth aspect, an embodiment of the present invention provides a computer program product, comprising a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the multi-regularized self-supervised training method for an emotion recognition model of any embodiment of the present invention are implemented.

[0019] The beneficial effects of the embodiments of the present invention are as follows: the multi-regularized self-supervised training of the method is respectively pre-trained by masked channel modeling of time-spectral transformation and emotion contrast learning, and utilizes their ability to provide different views of the data, guides the training process through intrinsic similarity using feature dependency information, and avoids the deviation caused by noise labels, so as to capture more accurate time-space-spectral characteristics and obtain more accurate emotion-related information. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 is a flow chart of a multi-regularized self-supervised training method for an emotion recognition model provided by one embodiment of the present invention;

[0022] Figure 2 1 is a schematic diagram of pre-training of a multi-regularized self-supervised training method for an emotion recognition model provided by an embodiment of the present invention, with MCM-TST on the left and ECL on the right;

[0023] Figure 3 It is a schematic diagram of multi-regularization fine-tuning of a multi-regularization self-supervised training method for an emotion recognition model provided by one embodiment of the present invention;

[0024] Figure 4 It is a schematic diagram of a hybrid architecture of an encoder of a multi-regularized self-supervised training method for an emotion recognition model provided by an embodiment of the present invention;

[0025] Figure 5 is a performance diagram of different methods of a multi-regularized self-supervised training method for an emotion recognition model provided by an embodiment of the present invention;

[0026] Figure 6 It is a schematic diagram of the results of different methods of simulating masked EEG on a SEED dataset for a multi-regularized self-supervised training method for an emotion recognition model provided by an embodiment of the present invention;

[0027] Figure 7 is a schematic diagram of ablation results of REmoNet multi-regularization fine-tuning of a multi-regularized self-supervised training method for an emotion recognition model provided by an embodiment of the present invention, wherein DWR and AWR represent distance regularization and angle regularization, respectively;

[0028] Figure 8 It is a schematic diagram of ablation results of different variants of REmoNet of different methods of a multi-regularized self-supervised training method for an emotion recognition model provided by an embodiment of the present invention;

[0029] Fig. 9 It is a schematic diagram of the accuracy and F1 score of target features of different smoothing methods in the pre-training stage of the SEED data set of a multi-regularized self-supervised training method for an emotion recognition model provided by an embodiment of the present invention;

[0030] Fig.10 is a visualization diagram of attention weights on a SEED data set for a multi-regularized self-supervised training method for an emotion recognition model provided by an embodiment of the present invention, wherein the redder the color, the greater the weight;

[0031] Fig.11 It is a structural diagram of a multi-regularized self-supervised training system for an emotion recognition model provided by an embodiment of the present invention;

[0032] Fig.12 A schematic diagram of the structure of an embodiment of an electronic device for multi-regularized self-supervised training of an emotion recognition model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] like Figure 1 FIG. 1 is a flowchart of a multi-regularized self-supervised training method for an emotion recognition model provided by an embodiment of the present invention, comprising the following steps:

[0035] S11: inputting the EEG sample into a first emotion recognition model based on masked channel modeling of time-spectral transformation and a second emotion recognition model based on emotion contrast learning;

[0036] S12: using the first emotion recognition model and the second emotion recognition model to identify the emotion of the electroencephalogram sample, to obtain a predicted emotion classification result, and obtaining an emotion classification loss for supervised learning based on the prepared real emotion classification result and the predicted emotion classification result;

[0037] S13: determining the logic value of the electroencephalogram sample through the first encoder and the classifier head of the first emotion recognition model, determining the emotional feature of the electroencephalogram sample through the second encoder and the projection head of the second emotion recognition model, and determining a regularization loss for offsetting noise labels based on the logic value and the emotional feature;

[0038] S14: Perform multi-regularized self-supervised training on the first emotion recognition model and the second emotion recognition model based on the emotion classification loss and the regularization loss until preset training requirements are met.

[0039] In this embodiment, in order to address the deficiencies of the prior art, the method creates REMoNet (Reducing Emotional Label Noise via Multi-regularized Self-supervision), i.e., reducing emotional label noise network, through multi-regularized self-supervision. The method proposes two novel self-supervision methods to fully understand and explain the multifaceted characteristics of EEG signals and extract emotion-related representations. The method proposes modeling of masked channels (also known as EEG channels) through time-spectral transforms, which is based on the insight of capturing unique frequency bands associated with different brain activities, for example, alpha waves represent relaxation and beta waves indicate enhanced mental activity. Secondly, the method proposes emotion contrast learning, in which positive and negative pairs are selected according to the attributes of the video stimulation test to promote the extraction of emotion-related representations. In order to combat emotion label noise, the method proposes a new method called multi-regularized co-learning. The method requires two different neural networks, which are pre-trained by masked channel modeling of time-spectral transformation and contrastive learning of emotions, respectively, and exploits their ability to provide different views of the data, guides the training process with feature dependency information through intrinsic similarity, and avoids bias caused by noisy labels. The diversity of the two networks is enhanced through a multi-regularization strategy, gradually making them consistent with each other.

[0040] For step S11, the present method is conceived to express the EEG signal as an electroencephalogram sample as where C is the number of EEG channels and T is the number of time steps. Then, X is divided into non-overlapping time windows T w Cut into X = (x1, x2, ..., x N ), x i Follow the distribution and This method represents the emotion label as Y = (y1, y2, ..., y N ), where y i Follow the distribution C is the number of classes. Note that Y follows a noisy joint distribution similar to X. The goal is to find a mapping function F:X→Y that minimizes the empirical risk R of the loss function L. L (F):

[0041]

[0042] In the self-supervised pre-training process, the self-supervised pre-training stage enables the model to learn emotion-related representations from unlabeled EEG data. Figure 2As shown in Figure 2, this method uses two self-supervised learning methods, namely, temporal spectral transformation for masked channel modeling and contrastive learning of emotions. Considering their respective properties, the former will be used for subsequent fine-tuning to predict emotions, while the latter will serve as regularization to offset noisy labels. Since pre-training benefits from a large amount of data, this method uses the training data of all subjects as the training set.

[0043] As an embodiment, the method further includes:

[0044] Pre-train the first emotion recognition model. Specifically. For masked channel modeling via temporal-spectral transformation. In order to fully understand and interpret the multifaceted features of EEG signals and extract the most relevant representations of emotions, this method proposes MCM-TST (masked channel modeling via temporal-spectral transformations). The design idea is that spectral features, especially spectral features in high-frequency bands, contain more semantic and emotional information to decode human emotions, which is more conducive to model pre-training than reconstructing semantically sparse and noisy original EEG signals.

[0045] The pre-training of the first emotion recognition model includes:

[0046] Inputting the EEG sample into a first emotion recognition model based on masked channel modeling of time-spectral transformation, wherein the first emotion recognition model comprises: an encoder, a regressor and a decoder;

[0047] In the first emotion recognition model, a plurality of EEG channels of the EEG sample are randomly masked to obtain a visible channel and an invisible channel of the EEG sample;

[0048] The encoder determines a visible latent representation of the visible channel and a reference invisible latent representation of the invisible channel;

[0049] The regressor predicts the visible channel to obtain a predicted potential representation of the invisible channel;

[0050] Inputting the reference invisible latent representation into a decoder to obtain a target spectral feature, adding a learnable position embedding to the predicted latent representation and inputting it into the decoder to obtain a predicted spectral feature;

[0051] Performing an alignment constraint on the predicted latent representation, and determining an alignment loss based on the predicted latent representation and the reference unseen latent representation;

[0052] Determining a reconstruction loss based on a mean-variance of the predicted spectral feature and the target spectral feature;

[0053] The first emotion recognition model is pre-trained based on the alignment loss and the reconstruction loss until a preset training requirement is met.

[0054] In this embodiment, in order to make the model learn more effectively and efficiently, the pre-training of masked EEG modeling has three components, encoder f, regressor h and decoder g.

[0055] For each input EEG sample x, this method randomly masks a ratio of r channels, resulting in a masked channel of x m , it can be seen that the channel is x v The encoder f will see channel x v Mapped to visible latent representation (latent representation) z v It only processes visible EEG signals and aims to extract features to reconstruct the masked EEG latent representation in the latent space.

[0056] The regressor h is designed to extract the v Predict the masked EEG channel (that is, the invisible channel) x m Potential representation of z m . It consists of L with cross note h During the training process, the entire MCM-TST model network will be trained on x m The queries Q with the same number of tokens m To learn and optimize. The key and value are based on the output x of the encoder f v Calculated, where contains the information of the entire EEG sample. In order to encourage the model to reasonably encode the EEG in the latent space, the latent representation z m Alignment constraints are imposed. Masked EEG channels x m Passed to the encoder f, thus generating Then, z is transformed into m and Alignment:

[0057]

[0058] For the decoder g, it takes the latent representation of the masked EEG channels as input and generates the corresponding target spectral features o m . It is simply composed of L g Transformer block followed by a linear layer to match the target, which processes only the masked EEG channels to reduce overhead. The learnable position embedding is added to z before feeding into the decoder mIn this method, DE (differential entropy) features are used as target features. MSE (mean-square error) is used as the reconstruction loss:

[0059]

[0060] in, is the target DE feature. Specifically, the total loss of MCM-TST is:

[0061]

[0062] Here, λ is a trade-off parameter.

[0063] Using L mfp Pre-train the first emotion recognition model until it meets the preset training requirements, for example, mfp until it is less than the preset threshold.

[0064] As another embodiment, the method further includes:

[0065] The second emotion recognition model is pre-trained.

[0066] In this embodiment, the pre-training of ECL (emotion contrastive Learning) focuses on giving the model strong instance discriminability. Since it has a better ability to learn structural similarities between instances, the model is pre-trained to play a regularization role in the fine-tuning stage. Specifically, the electroencephalogram samples including positive samples and negative samples collected by the data sampler are input into the second emotion recognition model based on emotion contrastive learning, wherein the second emotion recognition model includes: an encoder, a feature predictor connected to the encoder, a momentum encoder as a second encoder, and a projection head connected to the momentum encoder;

[0067] Determining a first embedding space of masked electroencephalogram samples including positive samples and negative samples using the encoder and the feature predictor;

[0068] Determine a second embedding space of the electroencephalogram samples including positive samples and negative samples using the momentum encoder and the projection head;

[0069] An InfoNCE loss is determined based on the first embedding space and the second embedding space, and the second emotion recognition model is pre-trained based on the InfoNCE loss until a preset training requirement is met.

[0070] In this implementation, the main problem of ECL is the definition of positive and negative samples. There are two different approaches: data augmentation or data sampler. Data augmentation usually uses methods such as additive noise, time shift, amplitude scaling, etc. to enhance the original EEG signal to obtain enhanced EEG data. However, unlike image enhancement in computer vision, directly enhancing physiological signals such as EEG is not suitable for the characteristics of the signal. Therefore, this method chooses a data sampler to sample the data in the training set. Based on individual differences and the observation that samples from the same trial (video clip) are annotated with the same label, this method proposes a simple but effective definition: only samples from the same video clip of the same subject are considered as positive sample pairs, while other samples are considered as negative sample pairs. Although different clips may be used to induce the same emotion, they still differ in their effects on arousal or dominance. Based on this assumption, this method hopes that the model has a strong discriminative ability not only for emotion categories, but also for arousal or dominance, which will be beneficial for subsequent fine-tuning or regularization.

[0071] This method represents the input mini-batch data as X B . We also randomly shield some channels and obtain The masking operation not only speeds up the training process but also provides a more robust pretext task, where the model should learn global discriminative features from corrupted EEG signals. The encoder f has the same architecture as the encoder in the masked EEG feature prediction, which transforms Convert to Z m . Because Z m contains only part of the EEG signal information. This method specifies a feature predictor M that extracts the information from the masked feature Z m The feature predictor M is composed of L m Transformer blocks. Latent feature Z m They are first concatenated by a learnable vector representing the mask channels, then added by the position embedding, and finally passed to m to obtain the output M.

[0072] For the data sampler, it randomly selects a sample from each video clip of each subject in the training set, and the selected sample is denoted as X S Therefore, for a small batch X B For each sample in X S There is only one positive sample and nk-1 negative samples in the training set (assuming that there are n subjects and k video clips in each subject’s training set). SThe EEG samples in are not masked, and they are fed into the momentum encoder f′, which is a Siamese network with encoder f. The momentum encoder f′ is updated by EMA (exponential moving average) to obtain a more stable representation for contrastive learning of emotions:

[0073] θ M′ ←ηθ M′ +(1-η)θ M

[0074] Among them, θ M and θ M′ are the parameters of the encoder M and the momentum encoder M′. η is the momentum coefficient. The projection head u consists of a linear layer and a ReLU activation function, which maps the output of the momentum encoder f′ to the embedding space M′. The loss function of ECL is the InfoNCE loss:

[0075]

[0076] Among them, m′ + represents the positive sample in M', m' - represents a negative sample, and r is a temperature parameter.

[0077] Using InfoNCE loss L contra Pre-train the second emotion recognition model until it meets the preset training requirements, such as L contra until it is less than another preset threshold.

[0078] Similarly, the first emotion recognition model and the second emotion recognition model can be pre-trained in advance or pre-trained separately.

[0079] For step S12, the structure and training process of the first emotion recognition model and the second emotion recognition model of the present method have been described in detail in step S11, and the first emotion recognition model and the second emotion recognition model (which can be pre-trained separately or simultaneously) are used as universal models for each subject. At this time, in the fine-tuning of the multi-regularization joint learning stage, the universal model of each subject is independently fine-tuned to obtain a personalized model for emotion recognition. At this stage, noisy emotion labels will lead to performance degradation, which cannot be ignored. In order to overcome this problem, the present method uses the ECL pre-trained model to perform multiple regularizations through the mutual relationship of data samples. Figure 3As shown, there are two encoders f and f'. Encoder f is pre-trained by MCM-TST, while encoder f' is pre-trained by ECL. Based on the prepared real emotion classification results, the main task is supervised learning with classification loss, and the cross entropy loss is used here:

[0080]

[0081] Among them, p i is the predicted value, y i is the corresponding label. Then we get the sentiment classification loss L ce .

[0082] For step S13, since comparative learning is required, the subsequent steps of this method will use two-by-two comparisons and three-way comparisons. At this time, the EEG samples are compared with the three input samples x. i 、x j and x k For example, this method outputs logical values ​​(logits, where logits usually refers to the original unprocessed scores or scores of the output layer of the model in classification problems) through the encoder f and the classifier head v, represented as p i 、p j and p k The output emotion feature of encoder f′ and projection head u is e i 、e j and e k The encoder f′ is assisted to maximize the consistency between the two encoders in terms of logits and features. The regularization loss is determined using the logits as well as the sentiment features. To achieve this, the method introduces a structural similarity loss which consists of distance regularization and angle regularization to prevent the model from overfitting on noisy labels.

[0083] As an implementation manner, the regularization loss includes: distance regularization loss and angle regularization loss.

[0084] The distance regularization loss is determined by:

[0085] Determine the Euclidean distance between every two EEG samples using the logical value and the emotion feature, and normalize the Euclidean distance using a similarity metric;

[0086] The distance regularization loss of the normalized Euclidean distance is determined by KL divergence.

[0087] In this implementation, multi-regularization implicitly handles noisy labels and does not require explicit detection of noisy labels. Therefore, it can be adaptively applied to datasets with any noise level, even non-noisy labels, which can also improve robustness and reduce overfitting. Regarding distance regularization, distance regularization calculates the distance between every two samples to minimize the relative distance obtained by the two models. This method first calculates the Euclidean distance d(p i ,p j ) and d(e i ,e j ). It is unreasonable to directly minimize these two distances because they are not in the same feature space. Therefore, this method uses a similarity measure q to normalize the distance to [0,1]:

[0088]

[0089] Here, μ is set to 0 and σ is set to 0.5. The similarity measure q satisfies the restriction lim d→∞ q(d)=0 and lim d→0 q(d) = 1. Distance regularization is achieved through Kullback-Leible divergence:

[0090]

[0091] This gives the distance regularization loss L dis .

[0092] As an implementation manner, the angle regularization loss is determined by:

[0093] Each three EEG samples are grouped into a triplet, and the corresponding angle measurement is determined according to the logical values ​​and emotional features of the three EEG samples in the triplet;

[0094] The smooth L1 loss determined by using the angle measurement corresponding to each of the logical value and the emotional feature is used as the angle regularization loss.

[0095] In this embodiment, angle regularization is defined on triplet samples, which penalizes the angle difference between the two encoders. The angle of each triplet of samples is calculated by the angle metric a as follows:

[0096]

[0097] Where <, > denotes the inner product. The angular regularization loss is a smooth L1 loss on each triplet sample:

[0098]

[0099] The smooth L1 loss l δ for:

[0100]

[0101] This gives the angle regularization loss L agl .

[0102] For step S14, in general, the final loss is the emotion classification loss L determined by the cross entropy loss ce , distance regularization loss L dis and the angle regularization loss L agl A linear combination of:

[0103]

[0104] Where α and β are coefficients. Using L total The first emotion recognition model and the second emotion recognition model are fine-tuned with multi-regularized self-supervision. Note that the encoder f′ and the projection head u are frozen during the entire fine-tuning stage. The encoder f is partially frozen.

[0105] It should be noted that the structures of the first encoder and the second encoder include: a gated temporal convolutional network block for reducing the dimension of extracted features and a spatial Transformer block for learning spatial features across EEG channels.

[0106] In this embodiment, the structure of the encoder used in the method model is as follows Figure 4 As shown. Since the original EEG signal has a high resolution in the time domain, it is crucial to extract information features from the time series of EEG signals and reduce the dimension. Existing methods usually use one-dimensional convolution to process EEG signals in the time dimension. This method introduces GTCN (Gated Temporal Convolutional Network) because of its powerful ability to control the flow of information between layers. For an EEG sample x, two one-dimensional convolutions are applied with kernel sizes equal to their stride sizes. The gating mechanism can be expressed as:

[0107] tanh(conv-a(x))⊙σ(conv-b(x))

[0108] Where ⊙ represents the element-wise product and σ represents the sigmoid activation function. Considering the masking operation in pre-training, GTCN is only applied on the time dimension and there are no more convolutions between EEG channels. Batch normalization and GELU (Gaussian Error Linear Unit) activation are followed by gating operations. L1GTCN blocks are stacked to obtain low-dimensional features.

[0109] After the GTCN block, the position embedding is added to the output features, which are initialized with sine and cosine functions and can be learned during training. This is followed by an L2 spatial Transformer block to learn spatial features across EEG channels. The original Transformer is used here, which is implemented by multi-head self-attention and feed-forward layers (with skip connections and layer normalization). In summary, the encoder of this method can process the spatial and temporal domains of the raw EEG signal simultaneously and extract informative features for subsequent tasks.

[0110] It can be seen from this implementation that the multi-regularized self-supervised training of the present method is pre-trained by masked channel modeling of time-spectral transformation and emotion contrast learning, respectively, and utilizes their ability to provide different views of the data, guides the training process through intrinsic similarity using feature dependency information, and avoids the bias caused by noisy labels, so as to capture more accurate time-space-spectral characteristics and obtain more accurate emotion-related information.

[0111] Experimental illustration of the proposed method. Comprehensive experiments were conducted on two public datasets, SEED and SEED-IV. The most obvious difference is the number of emotions. SEED has 3 emotions (positive, neutral and negative), while SEED-IV contains 4 emotions (happy, neutral, sad and fearful). 15 subjects participated in 3 experiments in each dataset. Video clips were used as stimuli to induce emotions. Each experiment contained 15 SEED trials and 24 SEED-IV trials, respectively. 62-channel EEG signals were recorded with an international 10-20 system at a sampling rate of 1000 Hz.

[0112] For preprocessing, the EEG signal is first filtered with a 0-75Hz bandpass filter and then downsampled to 200Hz. The DE features used as prediction targets in pretraining are extracted from 5 frequency bands, namely δ(1-4Hz), θ(4-8Hz), α(8-14Hz), β(14-31Hz), and γ(31-50Hz):

[0113]

[0114] The common settings of these two datasets are followed to evaluate the framework of the model of this method. The first 9 trials are used as training data and the last 6 trials are used as test data for SEED. For SEED-IV, the first 16 trials are used as training data and the remaining 8 trials are used as test data. The time window for SEED is 1 second and that for SEED-IV is 4 seconds. Due to the natural noise of the dataset, the emotions at the beginning of the video are usually low, and the performance of the evaluation method may not be accurate. Therefore, this method removes the first 30 seconds of each trial in the test data to obtain relatively clean test data, while the training data remains unchanged. All methods strictly follow the same conditions and are fairly compared. The experiments of this method are conducted using Python 3.10.9 and PyTorch 2.0.0 under NVIDIA 2080Ti.

[0115] For the encoder, the number of GTCN blocks L1 is 4, the kernel size is [5, 5, 4, 2] for SEED, and [10, 8, 5, 2] for SEED-IV. The number of spatial transformer layers L2 is 6, with 6 multi-head attention. For the regressor, L h is set to 2, and for the decoder, L g Set to 4. The feature predictor has L m = 3 Transformer blocks, and the embedding dimension of the projection head is 16. The masking ratio r is 0.5. r for contrastive learning is set to be learnable. The batch size for both pre-training and fine-tuning is 64. For pre-training, the model is run for 300 epochs with a learning rate of 0.0001 for masked EEG feature prediction and contrastive learning, and the AdamW optimizer is applied with a weight decay of 0.05. λ is set to 2 and the momentum η is 0.996. As for fine-tuning, the learning rate is adjusted from {0.0003, 0.001, 0.003} and the weight decay is selected from {0.0001, 0.01, 0.1}. The coefficient α is 1, while β is adjusted from 0.2 or 0.1. It should be noted that the GTCN block is frozen after pre-training because our method observes that the performance degrades if it is adjusted during fine-tuning. This is because the large-scale pre-training gives the model strong temporal processing capabilities. Fine-tuning with less data may harm the learned temporal feature extractor.

[0116] To compare our REmoNet with existing methods, we considered six baselines. These include several CNN-based methods: shallow convolutional networks (SCNs), EEGNet, filter bank convolutional networks (FBCNets), which use multi-view representations followed by spatial filtering, and TSception, which is a multi-scale convolutional neural network consisting of dynamic temporal, asymmetric spatial, and high-level fusion layers. There are also self-supervised baseline methods called Bert-Inspired Neural Data Representations (BENDR), which applies wav2vec 2.0 in speech recognition to EEG through contrastive learning, and a brain model (LaBraM), which is pre-trained on 2500 hours of EEG data by predicting masked neural codes. These methods are all end-to-end. To compare the performance of non-end-to-end methods, we replicated the masked autoencoder using DE features. The performance comparison results are shown in Figure 2. Figure 5 shown.

[0117] It can be observed that our REmoNet model significantly outperforms all other baseline methods. This demonstrates the effectiveness of self-supervised pre-training and robust fine-tuning. Specifically, our REmoNet achieves an average accuracy of 83.05% on the SEED dataset with a standard deviation of 9.6%, while no other method has an accuracy of more than 80%. Similarly, our model framework achieves an accuracy of 62.91% on the SEED-IV dataset with a standard deviation of 12.47%, far outperforming other methods, which have an accuracy of around 50%. Interestingly, the performance of MAE using non-smoothed DE features is comparable to that of other end-to-end methods. However, its standard deviation is much lower.

[0118] The videos for SEED and SEED-IV were determined by selecting the highest scoring videos from the same stimulus pool. For each emotion, SEED-IV had 8 videos while SEED had only 5 videos in a session, which means that SEED-IV videos were considered less effective than SEED (due to the lower average score). This fact suggests that SEED-IV is relatively noisier than SEED. For REmoNet, the performance gain of SEED-IV is 3.58%, which is higher than SEED (1.25%), indicating that REmoNet is more robust on datasets with larger label noise. This observation also highlights the flexibility of REmoNet as it can handle different noise levels without any prior knowledge.

[0119] Since REmoNet is pre-trained by masking EEG channel modeling, it may have great potential in handling corrupted EEG signals. This method conducts experiments to simulate the situation where some EEG signal channels are randomly damaged or missing in the SEED dataset. In this case, the model is tested with different numbers of sound EEG channels (10, 20, 30, 40, and 50). The specified number of channels are randomly selected as sound channels 50 times, and the average test accuracy is calculated to make the results convincing. To this end, all methods are trained with random corruption of EEG to obtain more robust models. The experimental results are shown in Figure 6 As shown. Our method is compared with the best baselines LaBraM-Base and TSception. It can be found that our model achieves the best accuracy in all cases. When the number of acoustic EEG channels becomes smaller, our method has a greater advantage over the other methods. Even in the extreme case where only 10 channels are available, our REmoNet still has an accuracy of 61.72%, while the accuracy of other methods is significantly lower, especially TSception at 47%. In summary, these results demonstrate the robustness of REmoNet when dealing with corrupted EEG signals.

[0120] In order to verify the effectiveness of co-learning with multiple regularizations, namely distance regularization (DWR) and angle regularization (AWR) in the fine-tuning stage, this method performs an ablation study on SEED and SEED-IV. Figure 7 Experiments with no DWR and AWR, single DWR, single AWR, and DWR and AWGR are given in . The accuracy decreases when using a single regularization compared to two regularization terms. In addition, the results with a single DWR are slightly better than those with a single AWR, which indicates that DWR is more effective than AWR to some extent. Note that the performance of a single AWR is also improved compared to fine-tuning without DWR and AWR, which indicates that AWR also plays an important role in handling noisy labels.

[0121] In addition, this method studies four different variants of REmoNet:

[0122] Variant 1: Fine-tune the ECL pre-trained model on REmoNet without any other optimization.

[0123] Variant 2: Fine-tuning the MCM-TST pre-trained model on REmoNet without any other optimization.

[0124] Variant 3: Abandoning the gating mechanism in GTCN.

[0125] Variant 4: Fine-tuning two different pre-trained models jointly using co-regularization in JoCoR, a robust training scheme for denoising.

[0126] The results are as follows Figure 8 As shown. It can be inferred that the performance of masked EEG feature prediction (Variant 2) is better than contrastive learning (Variant 1). This observation suggests that predicting semantic features from unmasked EEG signals may be more effective in capturing emotion-related information. However, both fine-tuned models outperform other baselines of the state-of-the-art, which further demonstrates the superiority of self-supervised pre-training. Compared with Variant 2, REmoNet achieves an accuracy of over 1.25% on SEED and over 3.58% on SEED-IV. In addition, Variant 3 demonstrates the effectiveness of the GTCN gating mechanism. REmoNet also shows its improvement compared to Variant 4, indicating the superiority of using self-supervision to handle noisy labels.

[0127] This method further studies the effect of smoothing on the target features during the pre-training phase. Although feature smoothing is not suitable for real-time applications, it is feasible to use it in offline pre-training of the model to learn better representations. This method considers two widely used feature smoothing algorithms: moving average and linear dynamic system (LDS). For a specific feature, the moving average simply performs a weighted sum of the features within a small time window, while LDS observes the entire trial. Compare Fig. 9 As shown in Figure 2. First, features with smoothness perform better than those without smoothness. This is consistent with previous studies that directly use smooth features to recognize emotions, indicating that smooth features are also helpful for self-supervised pre-training. In addition, LDS outperforms the moving average, indicating that considering the temporal dependency of the entire trial is effective.

[0128] Since the self-attention mechanism adopted in the spatial transformer block of our method can automatically assign dynamic weights to EEG channels based on the input data, it is possible to explore key channels. The attention weights (mean and standard deviation) of neutral, positive, and negative emotions are visualized in REmoNet to study the key EEG channels for emotion recognition, such as Fig.10 As shown. The attention weights for all data and all subjects were first min-max normalized and then calculated by averaging or calculating the standard deviation of the attention weights of all heads of the last Transformer block between all channels and class labels. It can be observed that the attention weights in the lateral temporal regions have high values ​​for all emotions, indicating that the lateral temporal regions are crucial for emotion recognition. In addition, positive and negative emotions have significantly higher weights in the frontal and parietal lobes than neutral emotions. The occipital region has higher attention weights for negative emotions. The low standard deviations for all brain regions demonstrate the stability of the proposed method.

[0129] In summary, this paper introduces the training method of REmoNet, which aims to reduce emotion label noise through multi-rule self-supervision. This self-supervision method also allows to comprehensively interpret the multifaceted features of EEG signals, thus helping to identify the most relevant factors in emotion recognition. Compared with existing methods, the proposed method shows great potential in handling masked EEG signals and proves proficiency in handling different numbers of EEG channels.

[0130] like Fig.11 The figure shows a structural diagram of a multi-regularized self-supervised training system for an emotion recognition model provided by one embodiment of the present invention. The system can execute the multi-regularized self-supervised training method for an emotion recognition model described in any of the above embodiments and be configured in a terminal.

[0131] The present embodiment provides a multi-regularized self-supervised training system 10 for an emotion recognition model, comprising: a data input module 11 , a classification loss determination module 12 , a regularization loss determination module 13 and a training module 14 .

[0132] Among them, the data input module 11 is used to input the EEG samples into the first emotion recognition model based on masked channel modeling of time-spectral transformation, and the second emotion recognition model based on emotion contrast learning; the classification loss determination module 12 is used to use the first emotion recognition model and the second emotion recognition model to identify the emotions of the EEG samples, and obtain predicted emotion classification results, and obtain the emotion classification loss for supervised learning based on the pre-prepared real emotion classification results and the predicted emotion classification results; the regularization loss determination module 13 is used to determine the logical value of the EEG sample through the first encoder and classifier head of the first emotion recognition model, determine the emotional features of the EEG sample through the second encoder and projection head of the second emotion recognition model, and determine the regularization loss for offsetting the noise label based on the logical value and the emotional features; the training module 14 is used to perform multi-regularized self-supervised training on the first emotion recognition model and the second emotion recognition model based on the emotion classification loss and the regularization loss until the preset training requirements are met.

[0133] The embodiment of the present invention further provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the multi-regularized self-supervised training method for emotion recognition model in any of the above method embodiments;

[0134] As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows:

[0135] inputting the EEG samples into a first emotion recognition model based on masked channel modeling of time-spectral transformation and a second emotion recognition model based on emotion contrast learning;

[0136] Using the first emotion recognition model and the second emotion recognition model to identify the emotion of the electroencephalogram sample, obtain a predicted emotion classification result, and obtain an emotion classification loss for supervised learning based on the pre-prepared real emotion classification result and the predicted emotion classification result;

[0137] Determine the logic value of the electroencephalogram sample by a first encoder and a classifier head of the first emotion recognition model, determine the emotional feature of the electroencephalogram sample by a second encoder and a projection head of the second emotion recognition model, and determine a regularization loss for offsetting noise labels based on the logic value and the emotional feature;

[0138] Based on the emotion classification loss and the regularization loss, multi-regularized self-supervised training is performed on the first emotion recognition model and the second emotion recognition model until preset training requirements are met.

[0139] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by the processor, the multi-regularized self-supervised training method for the emotion recognition model in any of the above method embodiments is executed.

[0140] Fig.12 is a schematic diagram of the hardware structure of an electronic device for a multi-regularized self-supervised training method for an emotion recognition model provided by another embodiment of the present application, such as Fig.12 As shown, the device includes:

[0141] One or more processors 1210 and memory 1220, Fig.12 A processor 1210 is taken as an example. The device for the multi-regularized self-supervised training method for emotion recognition model may further include: an input device 1230 and an output device 1240 .

[0142] The processor 1210, the memory 1220, the input device 1230 and the output device 1240 may be connected via a bus or other means. Fig.12 The example of connecting through bus is taken in the following.

[0143] The memory 1220, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the multi-regularized self-supervised training method for the emotion recognition model in the embodiment of the present application. The processor 1210 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 1220, that is, the multi-regularized self-supervised training method for the emotion recognition model in the above method embodiment is implemented.

[0144] The memory 1220 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data, etc. In addition, the memory 1220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1220 may optionally include a memory remotely arranged relative to the processor 1210, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0145] The input device 1230 can receive input digital or character information. The output device 1240 can include a display device such as a display screen.

[0146] The one or more modules are stored in the memory 1220, and when executed by the one or more processors 1210, the multi-regularized self-supervised training method for the emotion recognition model in any of the above method embodiments is executed.

[0147] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0148] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the multi-regularized self-supervised training method for an emotion recognition model of any embodiment of the present invention.

[0150] The electronic device of the embodiment of the present application exists in various forms, including but not limited to:

[0151] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0152] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDAs, MIDs, and UMPC devices, such as tablet computers.

[0153] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0154] (4) Other electronic devices with data processing functions.

[0155] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0156] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0157] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-regularized self-supervised training method for emotion recognition models, comprising: inputting the EEG samples into a first emotion recognition model based on masked channel modeling of time-spectral transformation and a second emotion recognition model based on emotion contrast learning; Using the first emotion recognition model and the second emotion recognition model to identify the emotion of the electroencephalogram sample, obtain a predicted emotion classification result, and obtain an emotion classification loss for supervised learning based on the pre-prepared real emotion classification result and the predicted emotion classification result; Determine the logic value of the electroencephalogram sample by a first encoder and a classifier head of the first emotion recognition model, determine the emotional features of the electroencephalogram sample by a second encoder and a projection head of the second emotion recognition model, and determine a regularization loss for offsetting noise labels based on the logic value and the emotional features; Based on the emotion classification loss and the regularization loss, multi-regularized self-supervised training is performed on the first emotion recognition model and the second emotion recognition model until preset training requirements are met.

2. The method according to claim 1, wherein: The regularization loss includes: distance regularization loss and angle regularization loss.

3. The method according to claim 2, wherein: The distance regularization loss is determined by: Determine the Euclidean distance between every two EEG samples using the logical value and the emotion feature, and normalize the Euclidean distance using a similarity metric; The distance regularization loss of the normalized Euclidean distance is determined by KL divergence.

4. The method according to claim 2, wherein: The angle regularization loss is determined by: Each three EEG samples are grouped into a triplet, and the corresponding angle measurement is determined according to the logical values ​​and emotional features of the three EEG samples in the triplet; The smooth L1 loss determined by using the angle measurement corresponding to each of the logical value and the emotional feature is used as the angle regularization loss.

5. The method according to claim 1, wherein: Before inputting the EEG samples into the first emotion recognition model based on masked channel modeling of time-spectral transformation and the second emotion recognition model based on emotion contrast learning, the method further includes: Pre-training the first emotion recognition model, or pre-training the second emotion recognition model, or Both the first emotion recognition model and the second emotion recognition model are pre-trained.

6. The method according to claim 5, wherein: The pre-training of the first emotion recognition model includes: Inputting the EEG sample into a first emotion recognition model based on masked channel modeling of time-spectral transformation, wherein the first emotion recognition model comprises: an encoder, a regressor and a decoder; In the first emotion recognition model, a plurality of EEG channels of the EEG sample are randomly masked to obtain a visible channel and an invisible channel of the EEG sample; The encoder determines a visible latent representation of the visible channel and a reference invisible latent representation of the invisible channel; The regressor predicts the visible channel to obtain a predicted potential representation of the invisible channel; Inputting the reference invisible latent representation into a decoder to obtain a target spectral feature, adding a learnable position embedding to the predicted latent representation and inputting it into the decoder to obtain a predicted spectral feature; Performing an alignment constraint on the predicted latent representation, and determining an alignment loss based on the predicted latent representation and the reference unseen latent representation; Determining a reconstruction loss based on a mean-variance between the predicted spectral feature and the target spectral feature; The first emotion recognition model is pre-trained based on the alignment loss and the reconstruction loss until a preset training requirement is met.

7. The method according to claim 5, wherein: The pre-training of the second emotion recognition model includes: The electroencephalogram samples including positive samples and negative samples collected by the data sampler are input into a second emotion recognition model based on emotion contrast learning, wherein the second emotion recognition model includes: an encoder, a feature predictor connected to the encoder, a momentum encoder as a second encoder, and a projection head connected to the momentum encoder; Determining a first embedding space of masked electroencephalogram samples including positive samples and negative samples using the encoder and the feature predictor; Determine a second embedding space of the electroencephalogram samples including positive samples and negative samples using the momentum encoder and the projection head; An InfoNCE loss is determined based on the first embedding space and the second embedding space, and the second emotion recognition model is pre-trained based on the InfoNCE loss until a preset training requirement is met.

8. The method according to claim 1, wherein: The structures of the first encoder and the second encoder include: a gated temporal convolutional network block for reducing the dimension of extracted features and a spatial Transformer block for learning spatial features across EEG channels.

9. A multi-regularized self-supervised training system for an emotion recognition model, comprising: A data input module, for inputting EEG samples into a first emotion recognition model based on masked channel modeling of time-spectral transformation, and a second emotion recognition model based on emotion contrast learning; A classification loss determination module, configured to use the first emotion recognition model and the second emotion recognition model to perform emotion recognition on the electroencephalogram sample to obtain a predicted emotion classification result, and to obtain an emotion classification loss for supervised learning based on a pre-prepared real emotion classification result and the predicted emotion classification result; a regularization loss determination module, configured to determine a logic value of the electroencephalogram sample through a first encoder and a classifier head of the first emotion recognition model, determine an emotional feature of the electroencephalogram sample through a second encoder and a projection head of the second emotion recognition model, and determine a regularization loss for offsetting noise labels based on the logic value and the emotional feature; A training module is used to perform multi-regularized self-supervised training on the first emotion recognition model and the second emotion recognition model based on the emotion classification loss and the regularization loss until preset training requirements are met.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 8.

12. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

Images