Cross-subject emotion recognition method and system based on hybrid expert model

Abstract

The embodiment of the invention provides a cross-subject emotion recognition method and system based on a hybrid expert model. The method comprises the following steps: splitting an acquired multi-channel electroencephalogram into a plurality of local electroencephalograms according to a brain region structure so as to reduce the complexity of cross-subject identification; the local electroencephalogram is input into a hybrid expert model, and the hybrid expert model comprises a brain circuit router and a hybrid graph expert block constructed by a space-time diagram convolutional network; and in the hybrid expert model, the brain circuit distributes the local electroencephalogram to the corresponding image expert block for processing, generalization emotion recognition is carried out on the distributed local electroencephalogram of the brain circuit through the image expert block, and an emotion recognition result is output. According to the embodiment of the invention, the MoGE mixed graph expert model is provided, and the EEG signal is decomposed by using a mixed expert architecture. The complex multi-channel electroencephalogram is split into a simple partial electroencephalogram. A complex EEG signal can be decomposed into a plurality of relatively simple parts and the relatively simple parts are respectively solved, so that cross-subject emotion recognition is realized.

Description

Technical Field

[0001] The present invention relates to the field of emotional brain-computer interface technology, and in particular to a cross-subject emotion recognition method and system based on a hybrid expert model. Background Art

[0002] ER (emotion recognition) based on EEG (electroencephalography) has achieved rapid development. However, the well-known individual differences in subjects' EEG data have greatly limited the promotion of BCI (Brain Computer Interface) in real-world applications.

[0003] In order to solve the limitation of individual differences, transfer learning methods are introduced. DA (Domain Adaptation) and DG (Domain Generalization) are two basic methods in transfer learning. Compared with the DA method, the DG method does not require any data from the target domain, which matches the most common real-world medical scenario of training a model with data from multiple subjects (multiple source domains) to diagnose new subjects (target domain). The technology in this field uses the following methods to solve the limitation of individual differences:

[0004] 1. Sparse MoE (Sparse mixture-of-experts): Each token is distributed to a limited number of experts through a sparse gating mechanism for training language models.

[0005] 2. Sparse mixture of experts model combined with Transformer model - Sparse mixture-of-experts aredomain generalizable learners: By replacing the FFN (feed forward layer) in the Transformer with several expert networks, each expert is a FFN. Using a distribution route to distribute each token to a limited number of experts, the generalization of the Transformer model on visual tasks is significantly improved.

[0006] 3. Mixture of experts for EEG-based seizuresubtype classification: A mixture of experts model is used to classify different subtypes of epilepsy EEG, where each expert is used to classify a subtype of epilepsy.

[0007] 4. EEG emotion recognition via identity based multi-gate mixture-of-experts network, which distinguishes different types of emotions by integrating two experts in an emotion recognition brain-computer interface, where the first expert is used to classify valence and the second expert is used to classify arousal.

[0008] 5. Elastic graph transformer networks for EEG-based emotion recognition. A Transformer layer and a graph convolutional neural network layer are superimposed to obtain a graph Transformer block. Several graph Transformer blocks are repeatedly superimposed to obtain a graph Transformer network for emotion recognition.

[0009] 6. Plug-and-play domain adaptation for cross-subject EEG-based emotion recognition: By training an encoder and decoder for each subject, the EEG of the new subject is matched with the EEG of the original subject to perform EEG emotion recognition on the new subject.

[0010] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art:

[0011] For the above-mentioned technologies 1, 2, 3, and 4, although they all use mixed expert models, the structure of their expert networks is relatively simple, that is, each expert is a feedforward neural network, which cannot process EEG modal data well. At the same time, for the mixed expert model technologies 3 and 4 for processing EEG modalities, the purpose of using mixed experts is essentially to split the original problem into different sub-problems, such as splitting epilepsy classification into subtype epilepsy classification, and splitting emotion recognition into valence emotion recognition and arousal emotion recognition. For technology 5, it uses graph neural networks to model each channel of the EEG modality and analyze the global EEG as a whole, but fails to split complex EEG signals into small, partial EEG signals. For technology 6, its cross-subject method is extremely cumbersome, and it is necessary to train an encoder and decoder for each subject in the original domain, which is not applicable when applied to large-scale data sets. Summary of the invention

[0012] In order to at least solve the above problems in the prior art.

[0013] In a first aspect, an embodiment of the present invention provides a cross-subject emotion recognition method based on a hybrid expert model, comprising:

[0014] The acquired multi-channel EEG was split into multiple local EEGs according to the brain regional structure to reduce the complexity of cross-subject identification;

[0015] Inputting the local EEG into a hybrid expert model, wherein the hybrid expert model comprises: a hybrid graph expert block constructed by a spatiotemporal graph convolutional network and an EEG route;

[0016] In the hybrid expert model, the brain circuit is processed by assigning the local electroencephalogram to the corresponding graph expert block, and the graph expert block performs generalized emotion recognition on the local electroencephalogram assigned to the brain circuit, and outputs the emotion recognition result.

[0017] In a second aspect, an embodiment of the present invention provides a cross-subject emotion recognition system based on a hybrid expert model, comprising:

[0018] A splitting module is used to split the acquired multi-channel EEG into multiple local EEGs according to the brain regional structure to reduce the complexity of cross-subject recognition;

[0019] An input module, used to input the local EEG into a hybrid expert model, wherein the hybrid expert model includes: an EEG route, a hybrid graph expert block constructed by a spatiotemporal graph convolutional network;

[0020] The emotion recognition module is used in the hybrid expert model to assign the local electroencephalogram to the corresponding graph expert block for processing, perform generalized emotion recognition on the local electroencephalogram assigned to the brain circuit through the graph expert block, and output the emotion recognition result.

[0021] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the cross-subject emotion recognition method based on a hybrid expert model according to any embodiment of the present invention.

[0022] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the cross-subject emotion recognition method based on a hybrid expert model of any embodiment of the present invention are implemented.

[0023] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the cross-subject emotion recognition method based on a hybrid expert model of any embodiment of the present invention are implemented.

[0024] The beneficial effect of the embodiment of the present invention is that the method proposes a MoGE hybrid graph expert model, which uses a hybrid expert architecture to decompose EEG signals. The method analyzes the importance of EEG domain distribution offset and neural structure, that is, captures the properties of different brain regions, and splits complex multi-channel EEG into simple partial EEGs. And through the improved graph neural network, the EEG channel relationship in non-Euclidean space is better modeled, so that complex EEG signals can be decomposed into multiple relatively simple parts and solved separately, thereby realizing emotion recognition across subjects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 is a flow chart of a cross-subject emotion recognition method based on a hybrid expert model provided by an embodiment of the present invention;

[0027] Figure 2 It is a hybrid structure diagram of a cross-subject emotion recognition method based on a hybrid expert model provided by an embodiment of the present invention;

[0028] Figure 3 It is a schematic diagram of EEG channel routing of a cross-subject emotion recognition method based on a hybrid expert model provided by an embodiment of the present invention;

[0029] Figure 4 It is a diagram of an expert diagram of a cross-subject emotion recognition method based on a hybrid expert model provided by an embodiment of the present invention;

[0030] Figure 5 It is a schematic diagram of the overall classification accuracy of different transfer learning methods on three data sets of a cross-subject emotion recognition method based on a hybrid expert model provided by an embodiment of the present invention;

[0031] Figure 6 It is a schematic diagram of confusion matrices of a MoGE model of a cross-subject emotion recognition method based on a hybrid expert model provided by an embodiment of the present invention on three data sets;

[0032] Figure 7 is a schematic diagram of average DG performance of different ablation variants on three data sets of a cross-subject emotion recognition method based on a hybrid expert model provided by an embodiment of the present invention;

[0033] Figure 8 It is a feature visualization diagram of a SEED data set of a cross-subject emotion recognition method based on a hybrid expert model provided by an embodiment of the present invention;

[0034] Fig. 9 (left) A schematic diagram of the first block of positive emotions determined by the router of MoGE selected by experts and (right) a heat map selected by experts for a cross-subject emotion recognition method based on a hybrid expert model provided by an embodiment of the present invention, wherein the y-axis represents 10 brain attributes. The x-axis corresponds to the selected expert id;

[0035] Fig.10 It is a structural schematic diagram of a cross-subject emotion recognition system based on a hybrid expert model provided by an embodiment of the present invention;

[0036] Fig.11 A schematic structural diagram of an embodiment of an electronic device for cross-subject emotion recognition based on a hybrid expert model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] like Figure 1 FIG. 1 is a flow chart of a cross-subject emotion recognition method based on a hybrid expert model provided by an embodiment of the present invention, comprising the following steps:

[0039] S11: Split the acquired multi-channel EEG into multiple local EEGs according to the brain regional structure to reduce the complexity of cross-subject identification;

[0040] S12: inputting the local EEG into a hybrid expert model, wherein the hybrid expert model comprises: an EEG route and a hybrid graph expert block constructed by a spatiotemporal graph convolutional network;

[0041] S13: In the hybrid expert model, the brain circuit is assigned to the local EEG to be processed by the corresponding graph expert block, and the graph expert block performs generalized emotion recognition on the local EEG assigned to the brain circuit, and outputs the emotion recognition result.

[0042] In this embodiment, in order to solve the cross-subject problem of emotion recognition neural networks in the prior art, simply put, the intuitive concept of this method is to apply mixed experts to different EEG channels, aiming to split complex multi-channel EEG signals into different brain regions and regions with different functions, and use different experts to process the EEG of these different partitions. Training the network model in this way does not require novel loss functions such as adversarial loss functions, nor does it require training an encoder for each subject. Using only a unified model and a simple empirical risk minimization loss function, an emotional brain-computer interface for emotion recognition with good generalization can be trained. Furthermore, this method proposes a MoGE (Mixture of Graph Experts) model, in which the overall architecture of MoGE is as follows: Figure 2 As shown, the DE (differential entropy) features extracted from the original EEG signal are fed into L MoGE blocks (graph expert blocks). The MoGE block consists of MHA (multi-head attention) and MoE (mixture-of experts), where each "expert" is a graph network of spatiotemporal graph convolution.

[0043] For step S11, the EEG signal of the subject can be obtained through an EEG acquisition device, for example, a DSI-24 dry electrode EEG cap is used to acquire the subject's multi-channel EEG signal at a frequency of 300 Hz. Alternatively, the subject's multi-channel EEG signal is acquired elsewhere, and then this method is used as an identification terminal to receive multi-channel EEG signals from other devices. Considering that this method is to apply mixed experts to different EEG channels, it aims to split the complex multi-channel EEG signal into local EEGs of different brain regions and different functional regions, and then split the acquired multi-channel EEG into multiple local EEGs according to the brain region structure, so as to reduce the complexity of cross-subject recognition.

[0044] As an implementation manner, the process of splitting the multi-channel EEG into multiple local EEGs includes:

[0045] Differential entropy is extracted from the frequency band of the brain regional structure of each multi-channel electroencephalogram within an overlapping time window, and the multi-channel electroencephalogram is divided into small segments using overlapping sliding windows to obtain multiple local electroencephalograms with the number of differential entropies, the number of electroencephalogram channels, the size of overlapping windows, and the dimension of electroencephalogram features after splitting.

[0046] In this embodiment, in order to extract EEG frequency domain features, for EEG features, DE (differential entropy) is extracted from the five frequency bands of each sample within a non-overlapping 1 s time window to obtain: And use overlapping sliding windows to divide it into small segments, thus generating Where Ts, C, T, and D represent the number of DE samples, the number of EEG channels, the size of the overlapping window, and the dimension of the extracted features, respectively.

[0047] As an implementation mode, the multi-channel EEG is split into local EEGs of the frontal lobe region, occipital lobe region, parietal lobe region, temporal lobe region, and special functional brain regions according to the brain regional structure.

[0048] In this embodiment, in the ER (emotion recognition) task, it is important to capture the properties of different brain regions because they have stable patterns across subjects. For example, the temporal lobe regions are highly activated for positive emotions, while the prefrontal regions are less active. Figure 3 As shown, if the brain areas are divided into several key parts, such as the frontal lobe area, the occipital lobe area, and some special links that neuroscientists have not yet discovered but can be learned by the network, then the complex EEG signals can be decomposed into several relatively simple parts and solved separately.

[0049] As an implementation manner, before inputting the local EEG into the hybrid expert model, the method further includes:

[0050] A class label is appended to the beginning of each local EEG, and a learnable position embedding is added to the class label, which is used for classification by a linear layer.

[0051] In this embodiment, when X′ in Before inputting the MoGE model, the D-dimensional tensor of each channel is embedded into d dimensions in X′ in A class token is appended to the beginning of , and a learnable positional embedding with a random start is added to each token. Finally, Input to the MoGE model. After processing L MoGE blocks, only the class labels are used for classification in the linear layer.

[0052] For step S12, the local EEG determined in step S11 is input into the mixed expert model. The mixed expert model of this method includes: normalization layer, multi-head attention layer, EEG routing and multiple mixed graph expert blocks (Mixture of Graph Experts Block) composed of multiple different graph experts. The mixed graph expert block is the most important part of the MoGE mixed graph expert model of this method. This method uses MoE to replace FFN (Feed-forward Networks) in the Transformer layer.

[0053] For step S13, in the hybrid expert model, the brain circuit route assigns the local EEG to the corresponding graph expert block for processing, and the graph expert block performs generalized emotion recognition on the local EEG assigned by the brain circuit route, including using a gate function in the brain circuit route to determine the probability G(x)=TOP of the local EEG being assigned to the graph expert block k (Softmax(W g x)), where G() is the gate function, k is the number of expert blocks in the allocation graph, and TOP k is the maximum probability of k graph expert blocks, W g is the learnable parameter of the gate function, Softmax() is the linear layer processing with softmax, and x is the local EEG.

[0054] The hybrid graph expert block constructed by the spatiotemporal graph convolutional network includes: a graph convolution layer for aggregating spatial information and a causal convolution layer for capturing temporal information, wherein the graph convolution layer is used to model the local EEG as a learnable graph G = {V, E, A}, wherein V and E represent the set of vertices and edges respectively, and A is the symmetric matrix adjacency matrix of the learnable graph G. The structure of the hybrid graph expert block is as follows Figure 4 shown.

[0055] As an implementation method, specifically, each graph expert block includes a spatiotemporal GCN (Graph Convolution Network), and the output of the MoGE block is:

[0056] f MoGE (X in )=f MoE (f MHA (X in ))

[0057] Among them, f MHA is the MHA multi-head attention layer, f MoE It is a MoE expert mixing layer. MHA (X in ) is expressed as x, we can get:

[0058]

[0059] Where G() is the gate function, Ei() is the i-th graph expert block (simply speaking, a graph expert block in the mixed graph expert block can represent an expert who processes a certain EEG separately), and N is the number of experts. The gate function G() determines which expert processes which EEG channel and consists of a linear layer with softmax to calculate the probability of each expert:

[0060] G(x)=TOP k (Softmax(W g x))

[0061] in, are the learnable parameters of the gate. Each EEG channel is assigned to k experts with the largest k probabilities, where k is a hyperparameter. Specifically, the input tensor X of each expert is Ei With X in has the same shape, but only the EEG channels sent to expert i have values, while the other channels are set to zero.

[0062] Regarding spatiotemporal graph convolutional networks, since GCN can construct a graph of electrodes on the scalp and capture their spatial relationships by aggregating information based on the graph, the effectiveness of GCN has been verified by many existing methods based on EEG classification tasks. For the MoGE block, this method proposes a spatiotemporal graph convolutional network as a graph expert instead of using a simple FFN as an expert.

[0063] Graph convolutional layers are used to aggregate spatial information, and causal convolutional layers are designed to capture temporal information.

[0064] This method models EEG data as a learnable graph G = {V, E, A}, where is the symmetric matrix adjacency matrix of graph G, V and E represent the sets of vertices and edges respectively.

[0065] The graph convolutional layer is as follows:

[0066]

[0067] in, I is the identity matrix, the angle matrix satisfy represents the input tensor of the ,th layer, is the weight matrix, represents the output tensor. Therefore, H 0 =X inFor all MoGE blocks, this method uses the same shared adjacency matrix A. A causal convolutional layer is used to extract temporal information from each EEG channel, where the output at time t depends only on the input at time t or earlier. Through the above steps, the hybrid expert model outputs the emotion recognition result, which is input into the pre-prepared emotion classifier to further identify the emotion corresponding to the input multi-channel EEG.

[0068] It can be seen from this implementation that this method proposes a MoGE hybrid graph expert model, which uses a hybrid expert architecture to decompose EEG signals. This method analyzes the importance of EEG domain distribution offset and neural structure, that is, captures the properties of different brain regions, and splits complex multi-channel EEG into simple partial EEGs. And through the improved graph neural network, the EEG channel relationship in non-Euclidean space is better modeled, so that complex EEG signals can be decomposed into multiple relatively simple parts and solved separately, thereby realizing emotion recognition across subjects.

[0069] Experimental description of this method, regarding datasets and implementation details, the generalization ability of the MoGE model of this method and various transfer learning methods is verified on three publicly available datasets (SEED, SEED-IV and SEED-V) through leave-one-subject (LOSO) cross-validation experiments. The size of the overlapping window is T = 5. The batch size is 64 and the dropout rate is 0.3. The number of MoGE blocks is L = 4. The hyperparameters of the learning rate are adjusted from {1e-4, 3e-3, 1e-3}, the number of heads of MHA is adjusted from 2 to 4, the embedding dimension d is adjusted from {16, 32, 64}, and the number of experts is adjusted from 3 to 6. The MoGE model of this method is trained using the default Adam optimizer and cross entropy loss function.

[0070] Regarding the cross-subject results, in order to verify the ability of the MoGE model of this method, MoGE trained using ERM (empirical risk minimization) was compared with baseline methods and the latest SOTA (state-of-the-art) model or DG (Domain Generalization) model on three public datasets. The LOSO experimental results are shown in Figure 2. Figure 5 As shown in Figure 2, it shows that the MoGE model trained with ERM achieves the best performance among the DG models based on SEED and SEED-V datasets. Figure 5The methods in the last five rows are trained with ERM. It can be observed that the Transformer-based model and the GNN-based model already have good generalization ability, and the combination of GNN and Transformer can achieve better results. It is worth noting that EmoGT (elastic graph transformer) has better generalization ability than DResNet on the SEED and SEED-V datasets, while the latter method learns invariant properties through adversarial loss. However, these studies treat EEG signals as a unified entity without considering the differences in properties of different brain regions. Therefore, our method uses a mixture of experts to decompose complex EEG signals into simpler components and obtains 88.0%, 74.3% and 81.8% on the three datasets only through ERM training. In addition, the confusion matrix of the MoGE model of our method is shown in Figure 6 As shown in Figure 2, this indicates that MoGE has a very balanced ability to classify various emotions on the three datasets. The balanced ability confirms the generalization ability of the MoGE model of this method from another perspective, and the MoG model of this method can also be well consistent with the invariant correlation of the EEG features of each emotion.

[0071] like Figure 7 The DG performance of different ablations is shown. This method comprehensively studies two modules in MoGE: adjacency matrix and circuit routing.

[0072] (1) Learnable adjacency matrix: This method evaluates the effectiveness of learnable adjacency matrices by replacing the adjacency matrix A with other non-learnable adjacency matrices, which are defined as follows:

[0073] Fully connected adjacency matrix: A matrix whose elements are all 1.

[0074] Zero-connected adjacency matrix: A matrix whose elements are all zero.

[0075] Random adjacency matrix: A randomly generated matrix whose elements range from 0 to 1.

[0076] GMSS adjacency matrix: a matrix with the same structure as the existing GMSS.

[0077] It can be seen that MoGE with learnable adjacency matrix outperforms the constant adjacency matrix, which demonstrates the generalization ability of elastic GNNs.

[0078] (2) EEG channel routing: The most important part of the MoGE model is the EEG channel routing, which assigns normalized EEG features to corresponding experts according to the neural properties of different brain regions. The EEG channel routing in the MoGE model of this method is implemented by linear learnable gates, as defined in the G(x) equation. The effectiveness of this learnable routing is verified by replacing it with some specially designed routings based on neuroscience. These specially designed routings are defined as follows:

[0079] No routing: All EEG channels are fed into a single graph expert, i.e. a regular graph converter.

[0080] Binary routing: All EEG channels are divided into the left and right hemispheres of the brain based on the structural symmetry of the brain.

[0081] Lobar routing: Routing assigns EEG channels to experts corresponding to each lobe.

[0082] GMSS routing: The brain regions are divided in the same way as the existing GMSS.

[0083] MoGE-2-Expert: MoGE routing sends the EEG of each channel to 2 experts, that is, k=2.

[0084] It can be seen that even without routing, MoGE simplified to a GNN-Transformer already has strong generalization capabilities. Roughly assigning EEG channels to 2 experts improves the performance of DG. With lobe-based routing, MoGE significantly improves classification accuracy. However, finer-grained GMSS routing leads to diminishing returns, suggesting that too many experts are detrimental to the generalization ability of the model. In addition, it is observed that sending each channel to multiple experts increases computational complexity but does not improve performance.

[0085] Figure 8 The distribution of the EEG signal before processing and the features extracted by MoGE are shown. It can be seen that the extracted features are clearly distinguishable. To further understand the expert selection of EEG signals, this method collects the routing strategy of the first MoGE block and visualizes the recording of positive emotions. Fig. 9 The left side shows the top 1 selection for each EEG channel. Fig. 9 The heatmap of the brain attributes that each expert focused on is shown on the right. The brain attributes are the same as the brain regions described in GMSS. It can be observed that experts 0, 1, 2, and 4 focus most on the frontal, temporal, central, and occipital regions, respectively. Expert 3 processes the scalp area around the central region and pays attention to a large number of brain attributes, which helps to establish the connection of each attribute. The visualization indicates that different experts focus on different brain regions, which decompose the complex EEG signal into several simple tasks and solve them separately.

[0086] In summary, this method proposes a MoGE hybrid graph expert model that decomposes EEG signals using a hybrid expert architecture. This method analyzes the importance of domain distribution shift and neural architecture from a theoretical perspective. Experimental results show that the MoGE model trained only with the ERM method has excellent DG performance, 88.0%, 74.3% and 81.8% on the SEED, SEED-IV and SEED-V datasets, respectively. Even after only ERM training, our MoGE model still outperforms other SOTA DG methods or other SOTA deep models, which shows the effectiveness of neural architecture and multi-channel emotion EEG decomposition. This has long been overlooked in previous studies on EEG tasks. This method visualizes the expert's choice and finds that the expert is able to focus on specific brain areas.

[0087] like Fig.10 The figure shows a schematic diagram of the structure of a cross-subject emotion recognition system based on a hybrid expert model provided by an embodiment of the present invention. The system can execute the cross-subject emotion recognition method based on a hybrid expert model described in any of the above embodiments and be configured in a terminal.

[0088] The present embodiment provides a cross-subject emotion recognition system 10 based on a hybrid expert model, comprising: a splitting module 11 , an input module 12 and an emotion recognition module 13 .

[0089] Among them, the splitting module 11 is used to split the acquired multi-channel EEG into multiple local EEGs according to the brain regional structure to reduce the complexity of cross-subject recognition; the input module 12 is used to input the local EEG into a hybrid expert model, wherein the hybrid expert model includes: a brain circuit route and a hybrid graph expert block constructed by a spatiotemporal graph convolutional network; the emotion recognition module 13 is used in the hybrid expert model, where the brain circuit route assigns the local EEG to the corresponding graph expert block for processing, and the graph expert block performs generalized emotion recognition on the local EEG assigned to the brain circuit route, and outputs the emotion recognition result.

[0090] The embodiment of the present invention further provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the cross-subject emotion recognition method based on the hybrid expert model in any of the above method embodiments;

[0091] As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows:

[0092] The acquired multi-channel EEG was split into multiple local EEGs according to the brain regional structure to reduce the complexity of cross-subject identification;

[0093] Inputting the local EEG into a hybrid expert model, wherein the hybrid expert model comprises: a hybrid graph expert block constructed by a spatiotemporal graph convolutional network and an EEG route;

[0094] In the hybrid expert model, the brain circuit is processed by assigning the local electroencephalogram to the corresponding graph expert block, and the graph expert block performs generalized emotion recognition on the local electroencephalogram assigned to the brain circuit, and outputs the emotion recognition result.

[0095] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by the processor, the cross-subject emotion recognition method based on the hybrid expert model in any of the above method embodiments is executed.

[0096] Fig.11 is a schematic diagram of the hardware structure of an electronic device of a cross-subject emotion recognition method based on a hybrid expert model provided by another embodiment of the present application, such as Fig.11 As shown, the device includes:

[0097] One or more processors 1110 and memory 1120, Fig.11 A processor 1110 is taken as an example. The device of the cross-subject emotion recognition method based on the hybrid expert model may also include: an input device 1130 and an output device 1140 .

[0098] The processor 1110, the memory 1120, the input device 1130 and the output device 1140 may be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0099] The memory 1120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the cross-subject emotion recognition method based on a hybrid expert model in the embodiment of the present application. The processor 1110 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 1120, that is, the cross-subject emotion recognition method based on a hybrid expert model in the above method embodiment is implemented.

[0100] The memory 1120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data, etc. In addition, the memory 1120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1120 may optionally include a memory remotely arranged relative to the processor 1110, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0101] The input device 1130 can receive input digital or character information. The output device 1140 can include a display device such as a display screen.

[0102] The one or more modules are stored in the memory 1120 , and when executed by the one or more processors 1110 , perform the cross-subject emotion recognition method based on a hybrid expert model in any of the above method embodiments.

[0103] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0104] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the cross-subject emotion recognition method based on a hybrid expert model of any embodiment of the present invention.

[0106] The electronic device of the embodiment of the present application exists in various forms, including but not limited to:

[0107] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0108] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDAs, MIDs, and UMPC devices, such as tablet computers.

[0109] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0110] (4) Other electronic devices with data processing functions.

[0111] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-subject emotion recognition method based on a hybrid expert model, comprising: The acquired multi-channel EEG was split into multiple local EEGs according to the brain regional structure to reduce the complexity of cross-subject identification; Inputting the local EEG into a hybrid expert model, wherein the hybrid expert model comprises: a hybrid graph expert block constructed by a spatiotemporal graph convolutional network and an EEG route; In the hybrid expert model, the brain circuit is processed by assigning the local electroencephalogram to the corresponding graph expert block, and the graph expert block performs generalized emotion recognition on the local electroencephalogram assigned to the brain circuit, and outputs the emotion recognition result.

2. The method according to claim 1, wherein: The process of splitting the multi-channel EEG into multiple local EEGs includes: Differential entropy is extracted from the frequency band of the brain regional structure of each multi-channel electroencephalogram within an overlapping time window, and the multi-channel electroencephalogram is divided into small segments using overlapping sliding windows to obtain multiple local electroencephalograms with the number of differential entropies, the number of electroencephalogram channels, the size of overlapping windows, and the dimension of electroencephalogram features after splitting.

3. The method according to claim 1, wherein: The multi-channel EEG is split into local EEGs of the frontal lobe region, occipital lobe region, parietal lobe region, temporal lobe region, and special functional brain regions according to the brain regional structure.

4. The method according to claim 1, wherein: Before inputting the local electroencephalogram into the hybrid expert model, the method further includes: A class label is appended to the beginning of each local EEG, and a learnable position embedding is added to the class label, which is used for classification by a linear layer.

5. The method according to claim 1, wherein: The brain circuit is processed by assigning the local EEG to the corresponding map expert block including: The probability G(x)=TOP of assigning the local EEG to the graph expert block is determined by using a gate function in the brain circuit routing. k (Softmax(W g x)), where G() is the gate function, k is the number of expert blocks in the allocation graph, and TOP k is the maximum probability of k graph expert blocks, W g is the learnable parameter of the gate function, Softmax() is the linear layer processing with softmax, and x is the local EEG.

6. The method according to claim 1, wherein: The hybrid graph expert block constructed by the spatiotemporal graph convolutional network includes: A graph convolution layer for aggregating spatial information and a causal convolution layer for capturing temporal information, wherein the graph convolution layer is used to model the local electroencephalogram as a learnable graph G = {V, E, A}, wherein V and E represent sets of vertices and edges, respectively, and A is a symmetric matrix adjacency matrix of the learnable graph G.

7. A cross-subject emotion recognition system based on a hybrid expert model, comprising: A splitting module is used to split the acquired multi-channel EEG into multiple local EEGs according to the brain regional structure to reduce the complexity of cross-subject recognition; An input module, used to input the local EEG into a hybrid expert model, wherein the hybrid expert model includes: an EEG route, a hybrid graph expert block constructed by a spatiotemporal graph convolutional network; The emotion recognition module is used in the hybrid expert model to assign the local electroencephalogram to the corresponding graph expert block for processing, perform generalized emotion recognition on the local electroencephalogram assigned to the brain circuit through the graph expert block, and output the emotion recognition result.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 6.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images