EEG emotion recognition method and system based on dynamic convolution residual multi-source migration

By employing a dynamic convolutional residual multi-source transfer learning method, the problems of non-stationarity and individual differences in EEG signals are addressed, achieving high-precision cross-individual and cross-temporal EEG emotion recognition and improving the model's adaptability and training efficiency.

CN115105076BActive Publication Date: 2024-11-15INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210556070.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-20
Publication Date
2024-11-15
Estimated Expiration
2042-05-20

AI Technical Summary

Technical Problem

Current EEG emotion recognition technology cannot effectively overcome the non-stationarity and individual differences in EEG signals, resulting in insufficient prediction accuracy.

Method used

We employ a multi-source transfer learning method based on dynamic convolutional residuals. We construct a feature extractor using a static network and dynamic convolutional residual blocks. We combine edge distribution domain adaptation and conditional distribution domain adaptation methods to measure the difference between the target domain and the source domain, and perform high-dimensional feature alignment and classification.

Benefits of technology

It achieves improved accuracy in EEG signal recognition across individuals and time periods, has strong model parameter adaptability, adapts to multi-source domain transfer problems, and improves model training efficiency and generalization performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115105076B_ABST
    Figure CN115105076B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of EEG emotion recognition, and specifically relates to an EEG emotion recognition method and system based on dynamic convolution residual multi-source transfer learning, which aims to solve the problem that the existing EEG emotion recognition cannot overcome the problem of insufficient prediction accuracy caused by the non-stationarity of EEG signals and individual time differences. The present invention includes: constructing a feature extractor based on an attention mechanism based on a static network and a dynamic convolution residual block, and extracting high-dimensional features of EEG signals in the source domain and the target domain respectively; combining the marginal distribution domain adaptation method and the conditional distribution domain adaptation method to measure the difference between the target domain and the source domain, and aligning the high-dimensional features of the source domain and the target domain based on the difference; classifying the aligned high-dimensional features through a classifier to obtain the emotion recognition results of the EEG signals in the source domain and the target domain. The present invention overcomes the shortcomings caused by the non-stationarity of EEG signals and individual time differences, has high prediction accuracy and efficiency, and has higher model application value.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] Emotion recognition has enormous scientific and clinical significance. EEG signals are physiological signals with high temporal resolution. Emotion recognition is performed by analyzing the immediate brain activity caused by emotional stimuli. This is more objective and reliable than non-physiological methods such as facial expressions, gestures, and language.

[0003] Current research on emotion recognition using EEG signals has achieved high accuracy under conditions where individuals are not independent. However, EEG signals present challenges such as non-stationarity and significant individual variability. Traditional model parameters require adjustment based on individual needs, limiting the efficiency of EEG emotion recognition and the accuracy and precision of the results.

[0004] In summary, it is of practical significance to explore EEG emotion recognition algorithms based on transfer learning and further study multi-source domain EEG emotion recognition across time and individual situations. Summary of the Invention

[0005] In order to solve the above-mentioned problem in the prior art, namely, the problem that the existing EEG emotion recognition cannot overcome the problem of insufficient prediction accuracy caused by the non-stationarity and individual time differences of EEG signals, the present invention provides an EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning, the EEG emotion recognition method comprising:

[0006] A feature extractor is constructed based on a static network and a dynamic convolution residual block, and high-dimensional features of EEG signals in the source domain and the target domain are extracted respectively; the feature extractor includes a static network, a dynamic network, a first attention mechanism, and a second attention mechanism, wherein the first attention mechanism is used to adjust the weights of the dynamic network and the static network in the feature extractor according to different samples, and the second attention mechanism is used to dynamically adjust the parameters of the dynamic network;

[0007] Combining the marginal distribution domain adaptation method and the conditional distribution domain adaptation method, the difference between the target domain and the source domain is measured, and the high-dimensional features of the EEG signals of the source domain and the target domain are aligned based on the difference;

[0008] The aligned high-dimensional features are classified by a classifier to obtain emotion recognition results of the EEG signals in the source domain and the target domain.

[0009] In some preferred embodiments, the structures of the static network and the dynamic network are as follows:

[0010] The static network is a one-dimensional convolutional layer, and the parameters of the one-dimensional convolutional layer are static parameters that do not change with samples;

[0011] The dynamic network includes k one-dimensional convolutional residual blocks, and the dynamic weights of the k one-dimensional convolutional residual blocks in the dynamic network are dynamically adjusted through the second attention mechanism module according to different samples, where k is a preset hyperparameter.

[0012] In some preferred embodiments, the value of the preset hyperparameter k is:

[0013] When targeting cross-individual tasks, k = 4;

[0014] When the task is performed across time on the same individual, k=2.

[0015] In some preferred embodiments, the weight of the dynamic network in the feature extractor is:

[0016] W θ(x) =λW0+(1-λ)ΔW θ(x)

[0017]

[0018] Among them, W θ(x) Represents the main function of the feature extractor, ΔW θ(x) represents the main function of the dynamic network, W0 represents the convolution kernel of the static network, λ represents the weight of the dynamic network in the feature extractor obtained by the first attention mechanism module, and π i (x) represents the dynamic weight of the i-th one-dimensional convolution residual block in the dynamic network obtained by the second attention mechanism module as the sample x changes, Φ i Represents the i-th one-dimensional convolutional residual block.

[0019] In some preferred embodiments, the dynamic weights of the k one-dimensional convolutional residual blocks in the dynamic network are:

[0020] a=Softmax(W2(Relu(W1x+b1)+b2))

[0021] Among them, a is the dynamic weight of the one-dimensional convolutional residual block in the dynamic network, W1 and W2 represent the parameters of the two fully connected layers of the preset second attention mechanism, b1 and b2 represent the biases of the two fully connected layers of the second attention mechanism, x represents the sample, Relu represents the activation function of the first fully connected layer of the second attention mechanism, and Softmax represents the activation function of the second fully connected layer of the second attention mechanism.

[0022] In some preferred embodiments, the total loss in the training of the EEG emotion recognition method includes emotion classification loss, marginal distribution domain adaptation loss and conditional distribution domain adaptation loss;

[0023] The total loss is expressed as:

[0024] L=L cls +L MMD +L MCD

[0025] Among them, L represents the total loss function, L cls is the emotion classification loss function, L MMD is the marginal distribution domain adaptation loss function, L MCD is the conditional distribution domain adaptation loss function.

[0026] In some preferred embodiments, the emotion classification loss function is a cross entropy loss function.

[0027] In some preferred embodiments, the marginal distribution domain adapts the loss function L MMD , which is expressed as:

[0028]

[0029] Among them, X s ={x s1 , x s2 ,…,x si ,…,x sn}, X t ={x t1 , x t2 ,…,x tj ,…,x tn} respectively represent the sample data of the source domain and the target domain in training, n is the sample data volume of the source domain X, m is the sample data volume of the target domain, φ is the preset mapping function that maps the sample data to the Hilbert space, Represents the calculation of the distance between sample data in Hilbert space.

[0030] In some preferred embodiments, the conditional distribution domain adaptation loss function L MCD , each stage of the three-stage training is a different function, the feature extractor is denoted as G, and the two classifiers are denoted as F1 and F2;

[0031] In the first training phase, the feature extractor G and two classifiers F1 and F2 are trained, and the objective function is:

[0032]

[0033]

[0034] Among them, Y s is the sample data label of the source domain, L(X s , Y s) represents the cross entropy loss of source domain classification, γ is the weight that gradually increases from 0 with the number of training iterations, and r is the rth iteration training;

[0035] In the second training stage, the feature extractor G is fixed and two classifiers F1 and F2 are trained. The objective function is:

[0036]

[0037]

[0038]

[0039] in, represents the calculation of [d(p1(y1|x t ), p2(y2|x t ))], the mathematical expectation of p1(y1|x t ) represents the input of classifier F1 is x t The probability distribution of the classification as y1, p2(y2|x t ) represents the input of classifier F2 is x t The probability distribution of the classification as y2, p 1c , p 2c They represent p1(y1|x t ), p2(y2|x t ) is the probability of the cth category, where c represents the cth category and C is the number of categories of the sample data;

[0040] In the third training phase, the two classifiers F1 and F2 are fixed and the feature extractor G is trained. Its objective function is:

[0041]

[0042] Another aspect of the present invention provides an EEG emotion recognition system based on dynamic convolution residual multi-source transfer learning, the EEG emotion recognition system comprising:

[0043] A feature extraction module is configured to construct an attention-based feature extractor based on a static network and a dynamic convolutional residual block, and extract high-dimensional features of EEG signals in the source domain and the target domain respectively; the feature extractor includes a static network, a dynamic network, a first attention mechanism, and a second attention mechanism, wherein the first attention mechanism is used to adjust the weights of the dynamic network and the static network in the feature extractor according to different samples, and the second attention mechanism is used to dynamically adjust the parameters of the dynamic network;

[0044] a feature alignment module configured to combine a marginal distribution domain adaptation method and a conditional distribution domain adaptation method to measure the difference between the target domain and the source domain, and perform high-dimensional feature alignment of EEG signals of the source domain and the target domain based on the difference;

[0045] The classification module is configured to classify the aligned high-dimensional features through a classifier to obtain emotion recognition results of the EEG signals in the source domain and the target domain.

[0046] Beneficial effects of the present invention:

[0047] (1) The EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning in the present invention has only one set of model parameters. There is no need to construct a feature extractor and classifier for each individual separately. It is convenient to expand as the data set increases, and has better practical value.

[0048] (2) The present invention adopts an EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning. To solve the problem that the neural network has a large number of parameters and it is difficult for the network to learn if all parameters are dynamically adjusted, a model structure is adopted in which some parameters in the network change dynamically with the samples and other parameters remain static. This limits the model's dependence on input samples to a small number of parameters, effectively improving the efficiency of model training and model performance. In addition, the combination of dynamic parameters and static parameters effectively solves the problem of large differences in EEG signals across individuals and across time.

[0049] (2) The present invention is based on the EEG emotion recognition method of dynamic convolutional residual multi-source transfer learning. The number of convolutional residual blocks in the dynamic part is adjusted with the nature of the task. For cross-individual tasks, the number of convolutional residual blocks is large, while for cross-time tasks of the same individual, the number of convolutional residual blocks is small. The attention mechanism is used to match different combinations of convolutional residual blocks for different input samples and adaptively adjust the importance of the static part and the dynamic part, so that the model can adapt to the multi-source domain migration problem.

[0050] (3) The EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning of the present invention has low dependence on sample diversity and sample quantity. Reducing the time points and training samples can achieve the best emotion prediction results and has better stability and generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0052] Figure 1 This is a schematic diagram of the EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning of the present invention. DETAILED DESCRIPTION

[0053] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0054] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0055] The present invention provides an EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning. This method models the cross-time and cross-individual emotion recognition problem as a multi-source domain adaptation problem. Its core is mainly based on two modules: a feature extractor based on the attention mechanism and emotion classification and recognition based on transfer learning.

[0056] Multi-source domain adaptation enables the extraction of invariant features from multiple source domains. For cross-individual data where the source and target domains differ significantly, stronger transfer methods are used. For example, combining marginal distribution domain adaptation and conditional distribution domain adaptation can reduce the data distribution gap between the source and target domains while also reducing the gap between source and target domain data within each emotion category. For cross-temporal data where the source and target domains differ less, weaker transfer methods are used to align data distributions to avoid negative transfer effects.

[0057] The feature extractor integrates the attention mechanism and residual block, automatically learns the attention weight according to the sample distribution, dynamically combines the weights of the convolution residual blocks, and achieves migration between different domains through dynamic adaptation of samples, so that the model can adapt to the EEG signals of different individuals.

[0058] Emotion recognition methods based on transfer learning achieve good classification training results with multiple datasets and small training samples. Different processing strategies are adopted for different task scenarios. For cross-individual tasks, where EEG signals vary significantly between individuals, a domain adaptation method combining marginal distribution and conditional distribution domain adaptation is used. For cross-temporal tasks, where EEG signals within the same individual at different time points vary less, conditional distribution domain adaptation alone can be used.

[0059] The present invention proposes a new model for effective EEG emotion recognition in cross-individual and cross-time scenarios based on transfer learning. The recognition effect of this method on the target task is higher than other similar methods, providing an advanced and effective solution for EEG emotion recognition.

[0060] The present invention provides an EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning, the EEG emotion recognition method comprising:

[0061] A feature extractor is constructed based on a static network and a dynamic convolution residual block, and high-dimensional features of EEG signals in the source domain and the target domain are extracted respectively; the feature extractor includes a static network, a dynamic network, a first attention mechanism, and a second attention mechanism, wherein the first attention mechanism is used to adjust the weights of the dynamic network and the static network in the feature extractor according to different samples, and the second attention mechanism is used to dynamically adjust the parameters of the dynamic network;

[0062] Combining the marginal distribution domain adaptation method and the conditional distribution domain adaptation method, the difference between the target domain and the source domain is measured, and the high-dimensional features of the EEG signals of the source domain and the target domain are aligned based on the difference;

[0063] The aligned high-dimensional features are classified by a classifier to obtain emotion recognition results of the EEG signals in the source domain and the target domain.

[0064] In order to more clearly illustrate the EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning of the present invention, the following is combined with Figure 1 The embodiments of the present invention are described in detail.

[0065] The EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning in the first embodiment of the present invention is described in detail as follows:

[0066] An attention-based feature extractor is constructed based on the static network and dynamic convolutional residual blocks, and high-dimensional features of EEG signals in the source and target domains are extracted respectively.

[0067] Regarding the dynamic feature extractor, the present invention takes into account the large number of neural network parameters. If all parameters are dynamic, the model will be difficult to learn. To simplify the model design and achieve faster prediction, a model structure is adopted in which some network parameters change dynamically with the sample, while some network parameters remain static, limiting the model's dependence on the input sample to a small number of parameters. The model consists of a static network and a dynamic network (the dynamic network includes k one-dimensional convolution residual blocks, k is a preset hyperparameter: when targeting cross-individual tasks, k = 4; when targeting cross-time tasks for the same individual, k = 2). All convolutions are one-dimensional convolutions, used to model the spatial and frequency domain information of the EEG signal.

[0068] At the same time, considering that the dynamic part and the static part of each domain are not of different proportions and are not equally important, directly adding the results of the dynamic part and the static part ignores the difference in size of each domain. The present invention adopts an attention mechanism to adaptively adjust the importance of the static part and the dynamic part according to the actual data distribution, thereby improving the model migration effect.

[0069] The feature extractor includes a static network, a dynamic network, a first attention mechanism and a second attention mechanism. The first attention mechanism is used to adjust the weights of the dynamic network and the static network in the feature extractor according to different samples, and the second attention mechanism is used to dynamically adjust the parameters of the dynamic network.

[0070] The structures of the static network and the dynamic network are as follows:

[0071] The static network is a one-dimensional convolution layer, and the parameters of the one-dimensional convolution layer are static parameters that do not change with samples.

[0072] The dynamic network includes k one-dimensional convolution residual blocks, and the dynamic weights of the k one-dimensional convolution residual blocks in the dynamic network are dynamically adjusted through the second attention mechanism module according to different samples, where k is a preset hyperparameter.

[0073] The first attention mechanism obtains the weight of the dynamic network in the feature extractor, as shown in Equation (1) and Equation (2):

[0074] W θ(x) =λW0+(1-λ)ΔW θ(x) (1)

[0075]

[0076] Among them, W θ(x) Represents the main function of the feature extractor, ΔW θ(x) represents the main function of the dynamic network, W0 represents the convolution kernel of the static network, λ represents the weight of the dynamic network in the feature extractor obtained by the first attention mechanism module, and π i (x) represents the dynamic weight of the i-th one-dimensional convolution residual block in the dynamic network obtained by the second attention mechanism module as the sample x changes, Φ i Represents the i-th one-dimensional convolutional residual block.

[0077] The second attention mechanism module obtains the dynamic weights of k one-dimensional convolution residual blocks in the dynamic network, as shown in formula (3):

[0078] a=Softmax(W2(Relu(W1x+b1)+b2)) (3)

[0079] Among them, a is the dynamic weight of the one-dimensional convolutional residual block in the dynamic network, W and W2 represent the parameters of the two fully connected layers of the preset second attention mechanism, b1 and b2 represent the biases of the two fully connected layers of the second attention mechanism, x represents the sample, Relu represents the activation function of the first fully connected layer of the second attention mechanism, and Softmax represents the activation function of the second fully connected layer of the second attention mechanism.

[0080] Based on the first attention mechanism, the results of the dynamic network and the static network are scaled separately according to different samples, so that the model can adapt to the multi-source domain migration problem. Based on the second attention mechanism, the dynamic network parameters are dynamically adjusted, and multi-source domain adaptation is achieved by adapting to a single sample.

[0081] The static features extracted by the weight-adjusted static network and the dynamic features extracted by the dynamic network are added together through one-dimensional convolution to obtain the high-dimensional features of the input signal.

[0082] In one embodiment of the present invention, the specific process for feature extraction of EEG signals is as follows: input EEG signals, construct C×N×1 data according to the number of channels C and the number of frequency bands N, and simultaneously input the data into static convolution and dynamic convolution as well as two attention networks. The attention network first performs average pooling, converts the input data into one dimension, and then inputs it into the calculation of subsequent attention weights. According to the weight of the first attention mechanism (first attention mechanism module), a weight is assigned to each dynamic convolution residual block, and a weighted sum is performed. The weight a obtained by the second attention mechanism (second attention mechanism module) is used as the weight of the weighted sum of the static network and the dynamic network respectively. Finally, the result of the weighted sum of the dynamic and static parts is subjected to one-dimensional convolution to obtain the extracted high-dimensional features.

[0083] Combining the marginal distribution domain adaptation method and the conditional distribution domain adaptation method, the difference between the target domain and the source domain is measured, and the high-dimensional features of the EEG signals of the source domain and the target domain are aligned based on the difference.

[0084] The aligned high-dimensional features are classified by a classifier to obtain emotion recognition results of the EEG signals in the source domain and the target domain.

[0085] In order to ensure that the high-dimensional features extracted by the feature extractor are domain invariant, data distribution domain adaptation is required to assist in the migration from the target domain to the source domain. The marginal distribution domain adaptation algorithm adopted in the present invention is the maximum mean difference (MMD), which can measure the distance between the means of two distributions in the regenerated Hilbert space and is a kernel learning method.

[0086] For conditional distribution domain adaptation, the method used in this paper is Maximum Classifier Discrepancy (MCD). MCD focuses on samples with poor classification performance for each class, identifies the differences between the target and source domains, and constructs a feature extractor and two classifiers. In the maximization phase, the difference between the two classifiers is maximized; in the minimization phase, the difference between the two classifiers is minimized. Ultimately, after multiple rounds of iteration, the difference between the two domains is minimized.

[0087] The total loss function L in the EEG emotion recognition method training consists of three parts: one is the emotion classification loss function, which is the cross entropy loss function used in this paper; one is the marginal distribution domain adaptation MMD; and the other is the conditional distribution domain adaptation MCD. The total loss function is shown in formula (4):

[0088] L=L cls +L MMD +L MCD (4)

[0089] Among them, L cls is the emotion classification loss function, L MMD is the marginal distribution domain adaptation loss function, L MCD is the conditional distribution domain adaptation loss function.

[0090] Marginal distribution domain adaptation loss function L MMD , which is expressed as formula (5):

[0091]

[0092] Among them, X s ={x s1 , x s2 ,…,x si ,…,x sn}, X t ={x t1 , x t2 ,…,x tj ,…,x tn} respectively represent the sample data of the source domain and the target domain in training, n is the sample data volume of the source domain X, m is the sample data volume of the target domain, φ is the preset mapping function that maps the sample data to the Hilbert space, Represents the calculation of the distance between sample data in Hilbert space.

[0093] In one embodiment of the present invention, the kernel function φ is a Gaussian kernel function.

[0094] The overall computational framework is to input the data of the target domain and source domain into the network respectively. After extracting the target domain features and source domain features, the source domain features are input into the classifier to obtain the emotion classification loss. The source domain and target domain are input into the domain adaptation algorithm to obtain the MMD and MCD losses.

[0095] In the actual calculation process, because the MCD algorithm adopts a three-stage training method, the training method is slightly different from the above process: in the first stage, the source domain data is used to train the feature extractor and classifier, and then the target domain and the source domain are calculated together with the MMD loss to train the feature extractor to ensure that the feature extractor and classifier can effectively identify the source domain data; in the second stage, the feature extractor parameters are fixed and the two classifiers are trained, hoping to make the source domain classification correct and maximize the prediction difference of the two classifiers on the target domain, hoping to find samples in the target domain that are quite different from the source domain; in the third stage, the two classifiers are fixed and only the target domain samples are used to train the feature extractor. The loss function is to minimize the prediction difference between the two classifiers. In order to align the target domain with large differences with the source domain, this method will repeat the third stage four times each training.

[0096] Conditional distribution domain adaptation loss function L MCD , each stage of the three-stage training is a different function, the first training stage is the first objective function, the second training stage is the second objective function, and the third training stage is the third objective function.

[0097] The feature extractor is denoted as G, and the two classifiers are denoted as F1 and F2.

[0098] In the first training stage, the feature extractor G and two classifiers F1 and F2 are trained, and their objective function is shown in formula (6):

[0099]

[0100] Among them, Y s is the sample data label of the source domain, Y t is the sample data label of the predicted target domain, L(X s , Y s ) represents the cross entropy loss of source domain classification, and γ is the weight that gradually increases from 0 with the number of training iterations.

[0101] The calculation method of weight γ is shown in formula (7):

[0102]

[0103] Among them, r is the rth iteration training.

[0104] In the second training stage, the feature extractor G is fixed and two classifiers F1 and F2 are trained. The objective function is shown in formula (8):

[0105]

[0106] In the third training stage, the two classifiers F1 and F2 are fixed and the feature extractor G is trained. Its objective function is shown in formula (9):

[0107]

[0108] The objective functions of the second and third training stages are As shown in formula (10):

[0109]

[0110] in, represents the calculation of [d(p1(y1|x t ), p2(y2|x t ))], the mathematical expectation of p1(y1|x t ) represents the input of classifier F1 is x t The probability distribution of the classification as y1, p2(y2|x t ) represents the input of classifier F2 is x t The probability distribution of classification is y2.

[0111] d(p1(y1|x t ), p2(y2|x t ))As shown in formula (11):

[0112]

[0113] Among them, p 1c , p 2c They represent p1(y1|x t ), p2(y2|x t ) is the probability of the cth category, c represents the cth category, and C is the number of categories of the sample data.

[0114] Unlike existing multi-source domain adaptation methods that design a specific feature extractor and a specific classifier for each source domain, and single-source domain adaptation methods that assume all individuals come from a single source domain, the present invention only requires the design of a single set of feature extractors and classifiers. The network's adaptability to multiple source domains is achieved by dynamically adjusting network parameters, improving the network's expressiveness and achieving multi-source domain adaptation through adaptation to a single sample. Furthermore, the present invention achieves joint distribution domain adaptation through marginal distribution domain adaptation and conditional distribution pre-adaptation. By migrating the target domain to the source domain, the accuracy of EEG emotion recognition for the target individual is improved, achieving an accuracy of over 90% across individuals and time on the public SEED dataset.

[0115] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0116] The second embodiment of the present invention provides an EEG emotion recognition system based on dynamic convolution residual multi-source transfer learning, the EEG emotion recognition system comprising:

[0117] A feature extraction module is configured to construct an attention-based feature extractor based on a static network and a dynamic convolutional residual block, and extract high-dimensional features of EEG signals in the source domain and the target domain respectively; the feature extractor includes a static network, a dynamic network, a first attention mechanism, and a second attention mechanism, wherein the first attention mechanism is used to adjust the weights of the dynamic network and the static network in the feature extractor according to different samples, and the second attention mechanism is used to dynamically adjust the parameters of the dynamic network;

[0118] a feature alignment module configured to combine a marginal distribution domain adaptation method and a conditional distribution domain adaptation method to measure the difference between the target domain and the source domain, and perform high-dimensional feature alignment of EEG signals of the source domain and the target domain based on the difference;

[0119] The classification module is configured to classify the aligned high-dimensional features through a classifier to obtain emotion recognition results of the EEG signals in the source domain and the target domain.

[0120] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0121] It should be noted that the EEG emotion recognition system based on dynamic convolution residual multi-source transfer learning provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be decomposed or combined. For example, the modules of the above embodiment can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps and are not regarded as improper limitations on the present invention.

[0122] An electronic device according to a third embodiment of the present invention includes:

[0123] at least one processor; and

[0124] a memory communicatively connected to at least one of the processors; wherein,

[0125] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning.

[0126] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, which are used to be executed by the computer to implement the above-mentioned EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning.

[0127] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0128] Those skilled in the art should be able to appreciate that, in conjunction with the modules and method steps of each example described in the embodiments disclosed herein, it is possible to implement them with electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0129] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0130] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0131] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method for EEG emotion recognition based on dynamic convolution residual multi-source transfer learning, It is characterized in that The EEG emotion recognition method comprises: A feature extractor is constructed based on a static network and a dynamic convolution residual block, and high-dimensional features of EEG signals in a source domain and a target domain are extracted respectively; the feature extractor includes a static network, a dynamic network, a first attention mechanism and a second attention mechanism, the first attention mechanism is used to adjust the weights of the dynamic network and the static network in the feature extractor according to different samples, and the second attention mechanism is used to dynamically adjust the parameters of the dynamic network; Combining the marginal distribution domain adaptation method and the conditional distribution domain adaptation method, measuring the difference between the target domain and the source domain, and aligning the high-dimensional features of the EEG signals of the source domain and the target domain based on the difference; Classifying the aligned high-dimensional features through a classifier to obtain emotion recognition results of the EEG signals in the source domain and the target domain; The weight of the dynamic network in the feature extractor is: ; ; in, represents the main function of the feature extractor, represents the main function of the dynamic network, Represents the convolution kernel of the static network, Represents the weight of the dynamic network obtained by the first attention mechanism module in the feature extractor, Represents the dynamic network obtained by the second attention mechanism module i A one-dimensional convolution residual block follows the sample x The dynamic weight of the change, Representative i A one-dimensional convolutional residual block; k The dynamic weight of a one-dimensional convolution residual block in the dynamic network is: ; in, a is the dynamic weight of the one-dimensional convolution residual block in the dynamic network, and Represent the two fully connected layer parameters of the preset second attention mechanism, and Represent the biases of the two fully connected layers of the second attention mechanism, x Representative samples, represents the activation function of the first fully connected layer of the second attention mechanism, Activation function of the second fully connected layer representing the second attention mechanism.

2. The EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning according to claim 1, It is characterized in that The structures of the static network and the dynamic network are as follows: The static network is a one-dimensional convolutional layer, and the parameters of the one-dimensional convolutional layer are static parameters that do not change with samples; The dynamic network includes k One-dimensional convolution residual block, k The dynamic weight of the one-dimensional convolution residual block in the dynamic network is dynamically adjusted through the second attention mechanism module according to different samples, where: k are the preset hyperparameters.

3. The EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning according to claim 2, It is characterized in that The preset hyperparameters k The value of is: When it comes to cross-individual tasks, ; When dealing with tasks across time for the same individual, .

4. The EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning according to claim 1, It is characterized in that The total loss in the training of the EEG emotion recognition method includes emotion classification loss, marginal distribution domain adaptation loss and conditional distribution domain adaptation loss; The total loss is expressed as: ; in, L represents the total loss function, is the emotion classification loss function, is the marginal distribution domain adaptation loss function, is the conditional distribution domain adaptation loss function.

5. The EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning according to claim 4, It is characterized in that The emotion classification loss function is a cross entropy loss function.

6. The EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning according to claim 4, It is characterized in that The marginal distribution domain adapts the loss function , which is expressed as: ; in, Represent the sample data of the source domain and target domain in training, n Source Domain X The amount of sample data, m is the sample data volume of the target domain, is the preset mapping function that maps sample data to Hilbert space, Represents the calculation of the distance of sample data in the Hilbert space.

7. The EEG emotion recognition method based on dynamic convolution residual multi-source transfer learning according to claim 6, It is characterized in that The conditional distribution domain adaptation loss function , which is a different function in each stage of the three-stage training, and the feature extractor is recorded as G , and denote the two classifiers as ; In the first training phase, we train the feature extractor. G And two classifiers , and its objective function is: ; ; in, is the sample data label of the source domain, represents the cross entropy loss for source domain classification, is the weight that increases gradually from 0 as the number of training iterations increases, r For the r Iteration training; In the second training phase, the feature extractor is fixed G , train two classifiers , and its objective function is: ; ; ; in, Represents calculation in the target domain The mathematical expectation of Representative classifier Input is Time classification The probability distribution of Representative classifier The input is Time classification The probability distribution of Respectively represent In the c The probability of the categories, c Representative c categories, C is the number of categories of sample data; In the third training phase, two classifiers are fixed. , train the feature extractor G , and its objective function is: 。 8. An EEG emotion recognition system based on dynamic convolution residual multi-source transfer learning, It is characterized in that The EEG emotion recognition system comprises: A feature extraction module is configured to construct a feature extractor based on an attention mechanism based on a static network and a dynamic convolution residual block, and extract high-dimensional features of EEG signals in a source domain and a target domain respectively; the feature extractor includes a static network, a dynamic network, a first attention mechanism and a second attention mechanism, the first attention mechanism is used to adjust the weights of the dynamic network and the static network in the feature extractor according to different samples, and the second attention mechanism is used to dynamically adjust the parameters of the dynamic network; A feature alignment module is configured to combine the marginal distribution domain adaptation method and the conditional distribution domain adaptation method to measure the difference between the target domain and the source domain, and perform high-dimensional feature alignment of the EEG signals of the source domain and the target domain based on the difference; A classification module, configured to classify the aligned high-dimensional features through a classifier to obtain emotion recognition results of the EEG signals in the source domain and the target domain; The weight of the dynamic network in the feature extractor is: ; ; in, represents the main function of the feature extractor, represents the main function of the dynamic network, Represents the convolution kernel of the static network, Represents the weight of the dynamic network obtained by the first attention mechanism module in the feature extractor, Represents the dynamic network obtained by the second attention mechanism module i A one-dimensional convolution residual block follows the sample x The dynamic weight of the change, Representative i A one-dimensional convolutional residual block; k The dynamic weight of a one-dimensional convolution residual block in the dynamic network is: ; in, a is the dynamic weight of the one-dimensional convolution residual block in the dynamic network, and Represent the two fully connected layer parameters of the preset second attention mechanism, and Represent the biases of the two fully connected layers of the second attention mechanism, x Representative samples, represents the activation function of the first fully connected layer of the second attention mechanism, Activation function of the second fully connected layer representing the second attention mechanism.