A multimodal emotion recognition brain-computer interface instrument based on contrastive learning gating networks

Abstract

The application provides a multi-modal emotion recognition instrument based on a contrast learning gated network. First, an electroencephalogram and an eye signal are used to design a hardware acquisition device, so that efficient and accurate data acquisition is ensured. Second, a multi-modal contrast learning gated network is designed to improve the accuracy of emotion recognition.

Description

Technical Field

[0001] This invention belongs to the field of brain-computer interfaces, and in particular relates to a multimodal emotion recognition brain-computer interface instrument based on a contrastive learning gating network. Background Technology

[0002] With the development of computer and information technology, machines with emotion recognition capabilities can significantly improve the user experience of human-computer interaction, providing a smoother and more natural interface. Against this backdrop, emotion recognition has received widespread attention from academia and industry, and has been extensively applied in fields such as autonomous driving, healthcare, and intelligent robotics. Existing emotion recognition research generally uses input methods categorized into physiological signals and non-physiological signals. Physiological signals have the advantage of being unaffected by human subjective consciousness, and can more realistically and reliably reflect changes in human emotions.

[0003] Research on emotion recognition using EEG and peripheral physiological signals has a long history. However, traditional deep learning feature extractors typically focus only on specific feature scales, such as single spatial, temporal, or frequency domain features. This limitation results in an insufficiently comprehensive range of extracted features, potentially overlooking crucial multidimensional information in emotion recognition. Therefore, comprehensive multi-scale features are essential for improving the accuracy of emotion recognition. Effectively integrating these multi-scale features can help capture more subtle emotional changes, thereby improving the model's performance and robustness.

[0004] Traditional emotion recognition methods primarily rely on data from a single modality. However, human emotional expression is diverse, with different emotional information embedded in various forms of physiological signals. Therefore, it is necessary to combine multiple physiological signals to express emotions. By leveraging the differences and complementarities among different modalities, more comprehensive emotional information can be obtained.

[0005] In existing multimodal emotion recognition, a common approach is to simply concatenate or fuse features from different modalities. This simple feature fusion method may fail to fully capture the complex relationships between different modalities, thus affecting model performance. To effectively fuse features from multiple sources, some studies use contrastive learning methods to learn the similarities and differences between different modalities, thereby extracting and aligning semantic information from data from different modalities.

[0006] Furthermore, in multimodal tasks, each modality possesses unique advantages and information. A significant challenge lies in how to flexibly select the optimal modality based on specific circumstances and effectively switch between the dominant modality and multiple modalities. Existing multimodal emotion recognition methods lack flexibility in modality selection and switching, resulting in the failure to fully utilize all valuable information from different modalities in certain samples.

[0007] Therefore, in multimodal emotion recognition, we need to pay more attention to the diversity and complementarity of features and explore more efficient dynamic decision-making mechanisms to achieve a more accurate and natural human-computer interaction experience. Summary of the Invention

[0008] To overcome the inaccuracy of existing multimodal emotion recognition technologies, this invention provides a multimodal emotion recognition instrument based on a contrastive learning gated network. Through dual optimization of hardware and algorithms, the performance of emotion recognition is comprehensively improved. First, this invention combines electroencephalogram (EEG) and eye signals in its hardware acquisition design to ensure efficient and accurate data acquisition. Second, a multimodal contrastive learning gated network (MCGNet) is designed to further enhance the accuracy of emotion recognition.

[0009] This invention relates to a multimodal emotion recognition brain-computer interface instrument based on a contrastive learning gating network. The instrument comprises an EEG data acquisition device, an eye-tracking data acquisition device, and a multimodal data processing device. The multimodal data processing device includes an MCU, a neural network accelerator, a data bus, a data storage device, and an output interface. The EEG data acquisition device acquires EEG data, and the eye-tracking data acquisition device acquires eye-tracking data. The acquired EEG and eye-tracking data are stored in the data storage device under the control of the MCU. Furthermore, the MCU controls the neural network accelerator to retrieve the EEG and eye-tracking data from the data storage device for training the multimodal contrastive learning gating network.

[0010] Furthermore, the EEG data acquisition device consists of a power supply, a main control chip, and an analog-to-digital converter.

[0011] Furthermore, the eye-tracking data acquisition device consists of binocular infrared cameras, which can capture clear eye images in low-light or no-light environments, avoiding interference from visible light.

[0012] To verify the practical effect of the multimodal acquisition helmet in emotion recognition, the multimodal emotion recognition method MCGNet, which combines EEG signals and eye movement signals, was executed in the multimodal emotion recognition brain-computer interface instrument proposed in this invention. Specifically, the MCU controls the neural network accelerator to retrieve EEG data and eye movement data from the data storage for training the multimodal contrastive learning gating network. The training includes the following three modules: a single-modal feature encoder pre-training module, a multimodal contrastive learning training module, and a gating structure training module.

[0013] The unimodal feature encoder pre-training module inputs EEG primary features and eye-tracking primary features into the EEG feature encoder and eye-tracking feature encoder, respectively, and then inputs the encoding results into the classifier to predict the unimodal emotion category.

[0014] The multimodal contrastive learning training module inputs the primary EEG features and primary eye-tracking features into the feature encoder of the single-modal feature encoder pre-training module to obtain multimodal features EEG and Eye. Then, the EEG and Eye signals are decomposed using EEG similarity mappers and EEG dissimilarity mappers, and eye similarity mappers and eye dissimilarity mappers, respectively. The decomposition results are then fused and subjected to intra-sample contrastive learning and inter-sample contrastive learning.

[0015] The gating structure training module fuses primary EEG features and primary eye-tracking features and inputs them into the gating network. The gating network generates a two-dimensional vector as output to select and activate different expert network branches to complete the emotion classification task.

[0016] This invention utilizes contrastive representation learning and contrastive feature decomposition to enhance the representation of multimodal information. Furthermore, to address the issue that the fusion of multimodal information may sometimes introduce noise interference, leading to lower accuracy compared to single-modal methods, a gating structure is introduced. This structure selects either a single-modal or multimodal method and its corresponding expert network based on sample features to improve the accuracy of emotion recognition. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a diagram of a multimodal emotion recognition brain-computer interface instrument system based on a contrastive learning gating network.

[0019] Figure 2 This is a diagram of the pre-training module for a single-modal feature encoder.

[0020] Figure 3 This is a diagram of the multimodal contrastive learning training module.

[0021] Figure 4 This is a diagram of the gating structure training module.

[0022] Figure 5 The results of this invention are experimental results on the TJU-Emotion dataset.

[0023] Figure 6The results of this invention are experimental on the public dataset SEED.

[0024] Figure 7 The graph shows the performance results of this invention in ablation experiments.

[0025] Figure 8 This is a conflation diagram of the EEG mode and eye mode of the present invention.

[0026] Figure 9 The figures show the experimental results of the single-modal EEG, eye, and two multimodal fusion methods of this invention.

[0027] Figure 10 This is the two-dimensional projection result of the four decomposition features of the test sample of this invention. Detailed Implementation

[0028] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0029] like Figure 1 As shown, the instrument consists of an EEG data acquisition unit, an Eye data acquisition unit, and a multimodal data processing unit. The multimodal data processing unit includes an MCU, a neural network accelerator, a data bus, a data memory, and an output interface. The EEG data acquisition unit acquires EEG data, and the eye-tracking data acquisition unit acquires eye-tracking data. After acquisition, the EEG and eye-tracking data are stored in the data memory under the control of the MCU. Furthermore, the MCU controls the neural network accelerator to retrieve the EEG and eye-tracking data from the data memory for training the multimodal contrastive learning gated network.

[0030] Furthermore, the EEG data acquisition device consists of a power supply, a main control chip, and an analog-to-digital converter.

[0031] Furthermore, the eye-tracking data acquisition device consists of binocular infrared cameras, which can capture clear eye images in low-light or no-light environments, avoiding interference from visible light.

[0032] To verify the practical effect of the multimodal acquisition helmet in emotion recognition, the multimodal emotion recognition method MCGNet, which combines EEG signals and eye movement signals, was executed in the multimodal emotion recognition brain-computer interface instrument proposed in this invention. Specifically, the MCU controls the neural network accelerator to retrieve EEG data and eye movement data from the data storage for training the multimodal contrastive learning gating network. The training includes the following three modules: a single-modal feature encoder pre-training module, a multimodal contrastive learning training module, and a gating structure training module.

[0033] The unimodal feature encoder pre-training module inputs EEG primary features and eye-tracking primary features into the EEG feature encoder and eye-tracking feature encoder, respectively, and then inputs the encoding results into the classifier to predict the unimodal emotion category.

[0034] The multimodal contrastive learning training module inputs the primary EEG features and primary eye-tracking features into the feature encoder of the single-modal feature encoder pre-training module to obtain multimodal features EEG and Eye. Then, the EEG and Eye signals are decomposed using EEG similarity mappers and EEG dissimilarity mappers, and eye similarity mappers and eye dissimilarity mappers, respectively. The decomposition results are then fused and subjected to intra-sample contrastive learning and inter-sample contrastive learning.

[0035] The gating structure training module fuses primary EEG features and primary eye-tracking features and inputs them into the gating network. The gating network generates a two-dimensional vector as output to select and activate different expert network branches to complete the emotion classification task.

[0036] The single-modal feature encoder pre-training module constructed in this invention is as follows: Figure 2 As shown, the pre-training module includes an EEG modality pre-training module and an eye-tracking modality pre-training module. The EEG modality pre-training module consists of an EEG primary feature extraction module, an EEG feature encoder DSSTNet module, and a classifier module connected in sequence; the eye-tracking modality pre-training module consists of an eye-tracking primary feature extraction module, an eye-tracking feature encoder module, and a classifier module connected in sequence.

[0037] Specifically, in the EEG modality pre-training module, during the pre-training phase, given a sample, the EEG primary feature extraction module extracts the EEG features of the sample data and inputs these features into the EEG feature encoder DSSTNet module. The output of the EEG feature encoder DSSTNet module is then input into the classifier module to predict the category of the single-modality sentiment. The EEG feature encoder DSSTNet module includes a spatial feature extractor, a frequency band attention module, and a temporal feature extractor connected in sequence. The spatial feature extractor extracts spatial features of the EEG signal, the frequency band attention module extracts frequency domain features, and the temporal feature extractor extracts temporal features.

[0038] Specifically, the raw EEG signal is preprocessed and then input into a spatial feature extractor, which consists of a graph converter and an adaptive graph convolutional layer. In the graph converter, a 64×1 kernel with a stride of 1×1 is first used to convolve along the electrode channel dimension to obtain the spatial features of the EEG signal. The purpose of setting a larger kernel size is to obtain a wider receptive field, thereby fully extracting the spatial features. This layer uses a padding operation to maintain the input and output feature dimensions unchanged. The obtained spatial features are then represented as graph-structured data.

[0039] Since the distribution of EEG channels is irregular and discrete, an adaptive graph convolutional layer is used to extract the spatial features of the EEG signal channel dimensions. In graph theory, a graph can be defined as G = (v, e), where v and e represent the nodes and edges in the graph, respectively. The connection relationships between N distinct nodes in the graph can be represented by the adjacency matrix A ∈ R. N×N Let g be the filtering function. θ =diag(θ), for a given spatial signal x∈R N The graph convolution operation can be defined as:

[0040] g θ * G x=Udiag(θ)U T x (1)

[0041] Where θ is a variable parameter, * G This represents a graph convolution operation, where U∈R N×N Let L∈R be the Laplace matrix in graph G. N×N The orthogonal matrix formed by the eigenvectors of g. In actual calculations, g θ It is difficult to obtain directly, so a K-order Chebyshev polynomial is used to approximate g. θ This adaptive graph convolutional layer can automatically update the adjacency matrix during backpropagation without requiring manual construction. First, the adjacency matrix is ​​randomly initialized. Then, when backpropagation is applied, the optimal adjacency matrix A is adaptively learned. *When calculating the loss function for A * The partial derivatives of A * Update according to the following rules:

[0042]

[0043] Where ρ is a parameter used to control the update speed of the adjacency matrix, and in this invention, ρ is set to 0.001.

[0044] This spatial feature extractor solves the problem of effectively extracting spatial features of EEG signals and avoids interfering with the adjacency matrix, enabling adaptive updates of the adjacency matrix for specific subjects.

[0045] Frequency Band Attention Module: This module, located between the spatial and temporal feature extractors, comprises three parts: semi-global pooling, local cross-band interaction, and adaptive weighting. First, semi-global pooling is used for global average pooling in the spatial dimension while preserving temporal information. Next, frequency domain features are extracted using 7×1 convolutional kernels, with imputation operations maintaining the temporal features and electrode channel dimensions of the feature map. Compared to fully connected layers, convolutional layers reduce the number of parameters and promote local interaction. Finally, the selected features are adaptively weighted, and the frequency domain information is probabilistically converted using a softmax function. The softmax output is then multiplied by the input features using a Hadamard product.

[0046] Temporal Feature Extractor: The temporal feature extraction module is constructed by stacking multiple one-dimensional convolutional layers. The first layer consists of D convolutional kernels of size 1×2 and stride 1×1, convolving along the temporal dimension. The second convolutional layer consists of D kernels of size 32×1 and stride 1×1, aiming to reduce the spatial dimension of the features to 1 for better extraction of temporal information. Subsequently, batch normalization is used to alleviate the gradient vanishing problem and accelerate the model training process. Next, the ELU activation function is used to non-linearize the data. The third layer performs average pooling along the temporal dimension, with a kernel size of 1×1 and stride of 1×2. This pooling layer smooths temporal features while preventing overfitting and reducing computational complexity. The final layer further extracts temporal features using D kernels of size 1×2 and stride 1×1. In this invention, the hyperparameter D is set to 40.

[0047] Finally, the EEG feature map is passed through a classifier module consisting of two fully connected layers, and its output is used to calculate the predicted probability of each of the three emotions using the softmax function.

[0048] Specifically, in the eye-tracking modality pre-training module, during the pre-training phase, given a sample, the eye-tracking primary feature extraction module is used to extract the eye features of the sample data and input the features into the eye-tracking feature encoder module. The output of the eye-tracking feature encoder module is input into the classifier module to predict the category of unimodal emotion.

[0049] After preprocessing the original eye signal, primary feature differential entropy features are extracted in the frequency bands [0,0.2]Hz, [0.2,0.4]Hz, [0.4,0.6]Hz, and [0.6,1]Hz, respectively. The features extracted for the left and right eyes include the mean, standard deviation, and differential entropy features of the four frequency bands. These are used as inputs to the eye-tracking feature encoder module, denoted as F. Eye ∈R T×K The sliding window length is T, and the feature dimension is K.

[0050] The eye-tracking feature encoder module adopts the encoder structure design in the transformer architecture, including an input embedding layer, multi-head attention, layer normalization, and a feedforward layer. The advantage of the transformer is that it can simultaneously handle long-range dependencies and capture global contextual information, effectively learning the intrinsic structure and important features in the input sequence through a self-attention mechanism.

[0051] The multimodal contrastive learning training module constructed in this invention is as follows: Figure 3 As shown in the diagram, in this module, the primary EEG features and primary eye-tracking features are input into the encoder of the single-modal feature encoder pre-training module to obtain multimodal features EEG and Eye. Then, the multimodal features EEG are decomposed into similar features EEG using an EEG similarity mapper and an EEG dissimilarity mapper. s and dissimilar features EEG d The multimodal feature Eye is decomposed into similar features Eye using an eye similarity mapper and an eye dissimilarity mapper. s and dissimilar features Eye d Then, the four decomposed features are fused using different concatenation methods, and intra-sample contrastive learning and inter-sample contrastive learning are performed. This invention provides two concatenation methods. The first method concatenates features according to their dimensions and outputs the concatenation results to an MLP classifier to obtain the sentiment prediction label (i.e., Figure 3 The first method is to concatenate the data according to the batch dimension and input them into a weight-shared MLP classifier to obtain four single-modal predicted labels represented by y2.

[0052] In the first approach, the dataset is denoted as M. For a given sample i∈M, the multimodal sentiment classification loss L is calculated using the cross-entropy (CE) loss.pred The calculation method is as follows:

[0053]

[0054] Where [;] indicates concatenation according to feature dimensions, y i The true label for the emotion category.

[0055] In the second approach, the unimodal emotion classification loss L uni for:

[0056]

[0057]

[0058] Where [,] indicates concatenation according to batch dimension, y i This represents the true label predicted by a single modality, and the prediction is performed separately for each of the four decomposed features.

[0059] In intra-sample contrastive learning and inter-sample contrastive learning, the four decomposed features are used to construct positive and negative sample pairs, such as... Figure 3 As shown. First, for a given sample pair (i,j) in dataset M, features are extracted by the feature encoder and the sample [EEG] is calculated. i Eye i ] and [EEG j Eye j Cosine similarity score between ]

[0060] C i,j =sim([EEG i Eye i [EEG] j Eye j (7)

[0061] Next, select similar and dissimilar samples for sample i. This will be compared with the label y. i Identical samples are sorted from low to high according to the cosine similarity score calculated in the previous step to construct a candidate similar sample set. At the same time, label non-y i The samples are divided into candidate dissimilar sample sets. From the set Two samples with high cosine similarity scores are randomly selected from the samples and paired with sample i to form a pair of samples, denoted as Neighbor. i From set Two samples with lower cosine similarity scores are randomly selected from the samples, denoted as . Two samples with high cosine similarity scores are randomly selected and denoted as . and Together with sample i, they form a negative pair between samples, Outlier i .

[0062] First, construct the in-sample positive alignment. and negative pair

[0063]

[0064] where j∈Neighbor i ∪Outlier i Neighbor i and Outlier i Let i represent similar and dissimilar samples to sample i, respectively.

[0065] Then, construct the positive alignment between samples. and negative pair The structure is as follows:

[0066]

[0067] where j∈Neighbor i ,k∈Outlier i

[0068] The loss functions for intra-sample and inter-sample contrastive learning are represented by the joint contrastive loss, which includes two aspects: comparison between similar and dissimilar samples between samples, and comparison between similar and dissimilar features within a sample. Given sample i, the contrastive learning loss L... c for:

[0069]

[0070] The NT-Xent contrastive learning loss framework combines intra-sample and inter-sample contrastive learning for modal feature decomposition and modal representation learning. The loss for sample i is expressed as:

[0071]

[0072] Where (a,p) and (a,n) represent a pair of decomposed feature vectors within a sample, for example... Or a pair of decomposed feature vectors between samples, for example P i For a positive set, it is represented as Including in-sample positive and between samples N i For a set of negative pairs, denoted as Including in-sample negative pairs Negative pairs between samples (a,p) represents P i In the positive pair, (a,n) is N i The negative pair in the middle.

[0073] The total loss function for this module is:

[0074] L = L pred +λ uni L uni +λ c L c (14)

[0075] Where L pred For multimodal emotion classification loss, L uni For unimodal emotion classification loss, L c To compare the learning loss. λ uni and λ c This determines the degree to which each task contributes to the updating of model parameters during training.

[0076] The gating structure training module constructed in this invention is as follows: Figure 4 As shown in the diagram, this module concatenates the primary EEG feature x1 and the primary eye feature x2 and inputs them into a gated network G(x) composed of an MLP. This gated network generates a two-dimensional vector g as its output to select and activate different network branches to complete the emotion classification task. A set of expert networks was selected based on a hybrid expert model. Each expert specializes in studying a subset of all modalities. In a specific task, there can be a maximum of three expert networks for two modalities, denoted as E1(x1), E2(x1,x2), and E3(x2). While combining EEG with the eye can provide useful cues, in practical applications, sometimes single-modal EEG is better at recognizing emotions. This invention uses the single-modal EEG emotion classification network model pre-trained in the single-modal feature encoder pre-training module as expert network one E1(x1), and the multimodal contrastive learning network model trained in the multimodal contrastive learning training module as expert network two E2(x1,x2). The final output predicted label is represented by y3:

[0077] y3=g1E1(x1)+g2E2(x1,x2) (15)

[0078] This invention selects two expert network models with different complexities; generally, expert networks with higher model complexity have stronger representational capabilities. If the network is trained solely by minimizing the loss of the emotion classification task, the gating network will always choose the branch with higher model complexity. Therefore, an additional loss function β[g1C(E1)+g2C(E2)] is introduced when designing the loss function. Here, g1 and g2 represent the decision vectors output by the gating structure, C(E) represents the cost of executing the expert network (e.g., MAdds), and β is a hyperparameter used to adjust the relative importance between the two loss terms.

[0079] This invention jointly optimizes expert networks and gating networks in an end-to-end manner. For the logic vector output by the gating structure MLP, we use a softmax function with a temperature parameter to adjust the sensitivity of the decision boundary. Furthermore, this invention involves selecting different expert networks for subsequent training, allowing each network to focus on specific modalities of the data.

[0080] This invention uses a self-built dataset (TJU-Emotion) and the publicly available SEED dataset to evaluate the performance of the proposed method. Experimental results show that the proposed method outperforms a series of state-of-the-art comparative methods in emotion recognition tasks.

[0081] First, this invention constructs a three-class multimodal emotion dataset, named the TJU-Emotion dataset, which includes EEG and eye multimodal data. The experiment was conducted at Tianjin University, with 10 participants invited to participate. The EEG modality used a 32-electrode experiment, and the eye data collected included pupil diameter data from the participants' left and right eyes. The experiment used 15 film clips (which could evoke positive, neutral, and negative emotions) as stimuli, and comprised 15 trials.

[0082] The preprocessing steps for the EEG portion include filtering, reducing the sampling rate, rereference transformation, segmentation, and artifact removal. The preprocessed EEG data is then divided into samples with a sliding window length of T = 15 s and a step size ΔT = 1 s. Preprocessing of the eye portion utilizes principal component analysis to remove the first principal component, illumination reflection.

[0083] The publicly available SEED dataset contains EEG data from 15 participants while watching 15 Chinese film clips with three different emotional states: negative, positive, and neutral. The EEG electrode distribution follows the international 10-20 electrode localization scheme, with a total of 62 electrode channels.

[0084] Four types of quantitative and qualitative effect analyses were conducted on this invention: comparative experiments with state-of-the-art methods, the impact of each structure on model performance, multimodal information complementarity analysis, and comparative learning visualization analysis.

[0085] This invention conducts a participant-dependent experiment and calculates the mean classification accuracy, standard deviation, and F1-score for all participants. The experiment sets the learning rate of the Adam optimizer to 1e-5 and 1e-3 on two datasets, respectively, with parameter λ. uni , λ c Both β and β are set to 0.1.

[0086] This invention selected six baseline models to compare the performance of the present invention, including the simple decision-level fusion methods MAX and SUM, the fuzzy integral-based model fusion method Fuzzy, the DCCA-AM method with attention mechanism and deep canonical correlation analysis, the BAT method that uses bidirectional adapters to transfer complementary information between different modalities, and the CARAT method that uses contrastive learning and feature reconstruction.

[0087] First, we performed a unimodal emotion classification experiment on 10 subjects in the TJU-Emotion dataset using a unimodal feature encoder, and compared the performance of our invention with the six baseline models from the previous section. Experimental results are shown below. Figure 5 .

[0088] From the first two columns of the graph, we can see that the performance of unimodal emotion classification depends on the data type used, with the EEG modality performing best among unimodal models. Our invention, by incorporating multimodal information, significantly improves the performance of emotion classification. We then conducted comparative experiments on this dataset using six multimodal fusion methods and calculated the t-test significance p-values ​​of several models relative to our invention. The classification performance of our invention shows a significant difference compared to other methods. Experimental results demonstrate that our invention not only has a significant advantage in classification accuracy but also exhibits considerable superiority in model stability. Furthermore, our invention's average F1-score is improved compared to other methods.

[0089] We also validated the performance of our proposed method on the widely used public dataset SEED. Experimental results are shown below. Figure 6 The results show that our proposed method is competitive on public datasets as well.

[0090] To investigate the impact of various structures on the performance of this invention, we conducted several ablation experiments, and the results are as follows: Figure 7 As shown, the models include the following ablation experiments: a model without contrastive learning, a model using only intra-sample contrastive learning, a model using only inter-sample contrastive learning, and a model without a gating structure.

[0091] For the TJU-Emotion dataset, from Figure 7It can be seen that the performance of the four ablation experiment models is inferior to that of the model of this invention. Compared with the model without contrastive learning methods, the models using intra-sample contrastive learning or inter-sample contrastive learning methods all show significant performance improvements. Furthermore, the model of this invention has a more flexible structure, which can dynamically select different modalities and expert networks according to the characteristics and needs of each sample. Similar conclusions were obtained for the SEED dataset.

[0092] Figure 8 A confusion graph of the EEG and eye modalities is presented. In the eye modality, the recognition accuracy for positive states is higher than in the EEG modality. The EEG modality, however, is better at recognizing neutral and negative states. Therefore, it is hoped that these two modalities can complement each other to improve the performance in recognizing each emotional state.

[0093] Figure 9 The confusion matrices for single-modal EEG and eye modes, as well as two multimodal fusion methods: a model without gating structures and the model of this invention, are presented, demonstrating the advantages and disadvantages of different modalities and different modal fusion methods. We found that the EEG and eye modalities have important complementary features, and the multimodal fusion method significantly improves classification performance. Overall, effectively combining multiple modalities can leverage the characteristics and complementarities of each modality to improve the ability to identify three types of emotions.

[0094] A subject is randomly selected in TJU-Emotion, and the two-dimensional projections of the four decomposed features of all their test samples are as follows: Figure 10 As shown, where Figure 10 (a) No contrastive learning method was used. Figure 10 (b) A contrastive learning method was used. As can be seen from the figure, after using the contrastive learning method, the similarity features (EEG)... s and Eye s The features become increasingly similar, while dissimilar features (EEG) become increasingly similar. d and Eye d The features they are moving further and further away from their corresponding similar features. This demonstrates the effectiveness of using contrastive learning methods to learn the consistency and inconsistency between modalities.

[0095] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0097] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0098] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0099] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0100] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0101] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0102] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0103] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multi-modal emotion recognition instrument based on contrastive learning gated network, characterized in that, The instrument comprises EEG data acquisition means for acquiring EEG data, eye movement data acquisition means for acquiring eye movement data, and multi-modal data processing means; The multi-modal data processing means comprises an MCU, a neural network accelerator, a data bus, a data storage, and an output interface; the MCU controls the neural network accelerator to take out the EEG data and the eye movement data from the data storage for training the multi-modal contrast learning gated network, specifically comprising: A single-modal feature encoder pre-training module, which inputs the EEG primary features and the eye movement primary features into the EEG feature encoder and the eye movement feature encoder respectively, and inputs the encoding results into the classifiers respectively to predict the single-modal emotion categories; The EEG feature encoder comprises a spatial feature extractor, a frequency band attention module, and a time domain feature extractor connected in sequence, the spatial feature extractor is used to extract the spatial domain features of the EEG signal, the frequency band attention module is used to extract the frequency domain features, and the time domain feature extractor is used to extract the time domain features; the eye movement feature encoder module adopts the structural design of the encoder in the transformer architecture; The multi-modal contrast learning training module inputs the EEG primary features and the eye movement primary features into feature encoders of single-modal feature encoder pre-training modules respectively to obtain multi-modal features and The multi-modal features are decomposed into similar features and dissimilar features by using an EEG similarity mapper and an EEG dissimilarity mapper The multi-modal features are decomposed into similar features and dissimilar features by using an eye similarity mapper and an eye dissimilarity mapper ; and the decomposition results are fused, and intra-sample contrast learning and inter-sample contrast learning are performed. A gating structure training module, which inputs the fusion of the EEG primary features and the eye movement primary features into the gating network, and the gating network generates a two-dimensional vector as an output to select and activate different expert network branches to complete the emotion classification task.

2. The multi-modal emotion recognition instrument based on the contrast learning gated network according to claim 1, wherein the single-modal feature encoder pre-training module comprises an EEG modal pre-training module and an eye movement modal pre-training module, the EEG modal pre-training module comprises an EEG primary feature extraction module, an EEG feature encoder DSSTNet module, and a classifier module connected in sequence; and the eye movement modal pre-training module comprises an eye movement primary feature extraction module, an eye movement feature encoder module, and a classifier module connected in sequence. The EEG feature encoder DSSTNet module comprises a spatial feature extractor, a frequency band attention module, and a time domain feature extractor connected in sequence.

3. The multi-modal emotion recognition instrument based on the contrast learning gated network according to claim 1, wherein the multi-modal contrast learning training module specifically comprises: The four features after decomposition are fused in the feature dimension or the batch dimension, and the intra-sample contrast learning and the inter-sample contrast learning are performed.

4. The multi-modal emotion recognition instrument based on the contrast learning gated network of claim 1, in the gating structure training module, the single-modal EEG emotion classification network model pre-trained by the single-modal feature encoder pre-training module is taken as the expert network one the multi-modal contrast learning network model trained in the multi-modal contrast learning training module is taken as the expert network two the final output prediction label is represented as: ​ （1） wherein and denotes the decision vector of the gating structure output.

5. The multi-modal emotion recognition instrument based on the contrast learning gated network according to claim 2, wherein the eye movement feature encoder module adopts the structural design of the encoder in the transformer architecture, comprising an input embedding layer, a multi-head attention, a layer normalization, and a feedforward layer.

6. The multi-modal emotion recognition instrument based on contrastive learning gated network of claim 3, when fused in terms of feature dimensions, the data set is denoted as For a given sample The multi-modal emotion classification loss is calculated using the cross-entropy loss In the following manner: (2) （3） wherein denotes concatenation along the feature dimension, is the true label for the emotion class; MLP is a multi-layer perceptron.

7. The multi-modal emotion recognition apparatus based on contrastive learning gated network of claim 3, wherein the single-modal emotion classification loss when fused in the batch dimension is: L = 1 T åt=1 åc åi åj (y^t,c,i,j - y^t,c,i,j)2 where T is the number of batches, C is the number of emotion classes, i is the number of input samples in the batch, and j is the number of output samples in the batch.​ （4） （5） wherein denotes concatenation along the batch dimension, denotes the true label predicted by the single modality; MLP is a multi-layer perceptron.

8. The multi-modal emotion recognition instrument based on the contrast learning gated network according to claim 3, wherein in the intra-sample contrast learning and the inter-sample contrast learning, the four features after decomposition are used to construct positive and negative sample pairs, specifically comprising: For a dataset Given a pair of samples , extract features through a feature encoder and compute cosine similarity scores between samples and : （6） For the sample Select similar and dissimilar samples; match with labels The same samples are sorted from low to high according to the cosine similarity scores calculated by Equation (6) to construct a candidate similar sample set. At the same time, the label will be non- The samples are divided into candidate dissimilar sample sets. ; From the set Two samples with high cosine similarity scores are randomly selected from the samples. The samples are aligned, denoted as From set Two samples with lower cosine similarity scores are randomly selected from the samples, denoted as . Randomly select two samples with high cosine similarity scores, denoted as... ; and With sample negative pairs between samples .

9. The multi-modal emotion recognition instrument based on the contrast learning gated network according to claim 8, wherein the intra-sample positive pair construction and the inter-sample positive pair construction are as follows: First construct intra-sample positive pairs and negative pairs : （7） （8） wherein , and represent similar samples and dissimilar samples, respectively, of the sample . Constructing the sample pairs positive and negative pairs are constructed as follows: （9） （10） wherein , .

10. The multi-modal emotion recognition instrument based on contrastive learning gated network according to claim 1, wherein the EEG data acquisition device is composed of a power supply, a master control chip and an analog-to-digital converter; and the eye movement data acquisition device is composed of a binocular infrared camera.

Images