Emotion recognition method based on electroencephalogram and eye movement multi-mode signals

Abstract

The invention relates to the technical field of emotion recognition, in particular to an emotion recognition method based on electroencephalogram and eye movement multi-modal signals, which comprises the following steps: constructing an HMSB-CAF model; the extracted electroencephalogram and eye movement features are input into an HMSB-TransNet model, and mapping from high-order features to low-dimensional space is realized through hierarchical multi-scale branch residual transform, so that richer feature representation related to emotion is obtained; using canonical correlation analysis (CCA) to constrain the statistical correlation of the two types of modals in a depth feature space; dynamically calculating weight coefficients of the two types of modals through the learnable parameterized attention vector; self-adaptive fusion is realized according to the weight and the feature contribution, and optimized joint representation is obtained and input into a classifier to complete emotion discrimination. The cross-culture emotion recognition method solves the problems that in an existing cross-culture emotion recognition method, deep mining of modal features is insufficient, and second-order interference influence of modal signals is obvious.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, in particular to a method for emotion recognition based on electroencephalogram and eye movement multi-modal signals. BACKGROUND

[0002] As an important dimension of human embodied cognition, the neural generation mechanism and social expression pattern of emotion are closely coupled with specific cultural contexts. Studies have shown that individuals in different cultural backgrounds show systematic differences in emotional experience intensity, expression behavior and modality selection. For example, in collectivist cultures, social members generally tend to suppress emotions to maintain group harmony, while individualist cultures emphasize the free expression of emotions. The above cultural differences not only manifest in language communication, but also affect facial expressions, body language and social behavior norms at multiple levels.

[0003] Existing research shows that most different cultural emotion recognition literature focuses on the architecture innovation of modality fusion strategy. The field of physiological signal fusion has formed three paradigms of feature level, decision level and hybrid fusion. Early research adopted a shallow fusion strategy: Lu et al. realized bimodal fusion by handcrafted feature concatenation combined with SVM classifier; Zhang's team introduced group sparse canonical correlation analysis to construct a joint representation space, but it was still limited to linear correlation modeling. With the development of deep learning, Zhu et al. used denoising autoencoder to realize data-level feature mapping; Zheng et al. innovatively constructed a dual-channel RBM network to learn the non-linear interaction between modalities; Liu's team proposed the DCCA-AM model to optimize deep canonical correlation analysis through an attention weighting mechanism, which has become an important benchmark; Gong et al. proposed a coordinated representation decision fusion network (CoDF-Net) and designed a decision-level fusion width learning system (DF-BLS) to enhance the noise resistance of the model through a multi-subsystem voting mechanism.

[0004] Although current emotion recognition methods have made some progress in fusion mechanisms, the depth of digging into the nature of modality features is still insufficient, especially in the fine-grained expression of different cultural emotions. SUMMARY

[0005] Therefore, the emotion recognition method based on electroencephalogram and eye movement multi-modal signals of the present application proposes a hierarchical multi-scale branch residual transformation-classical correlation attention fusion network (HMSB-CAF). The network innovatively proposes a hierarchical multi-scale branch residual transformation network architecture (HMSB-TransNet). The architecture introduces multi-scale context perception (MSCP), depth separable convolution, dilated convolution, BTN normalization module and KAN network. The architecture is a novel network structure combining double-layer deep neural network and gating mechanism. The model signal is changed through the network to realize efficient fine-grained feature extraction and stable expression, and significantly enhance the modeling ability of complex semantics and spatial structure.

[0006] According to a first aspect of an embodiment of the present application, an emotion recognition method based on electroencephalogram and eye movement multi-modal signals is provided, comprising: preprocessing the electroencephalogram data and eye movement data to be tested to extract electroencephalogram features and eye movement features; inputting the electroencephalogram features and eye movement features into a hierarchical multi-scale branch residual transformation network of an HMSB-CAF model for hierarchical multi-scale branch residual transformation to obtain electroencephalogram fine-grained features and eye movement fine-grained features; performing cross-modal semantic alignment on the electroencephalogram fine-grained features and eye movement fine-grained features through a typical correlation attention fusion mechanism of the HMSB-CAF model, and using typical correlation analysis to constrain the statistical correlation of the two types of modalities of the electroencephalogram fine-grained features and eye movement fine-grained features in the depth feature space, forcing the electroencephalogram rhythm energy distribution and eye movement behavior features to establish high-order correlation in the emotion expression dimension; dynamically calculating the weight coefficients of the two types of modalities of the electroencephalogram fine-grained features and eye movement fine-grained features after high-order correlation through a learnable parameterized attention vector, and based on the weight coefficients and the contribution of the features to emotion classification, adaptively fusing the two types of modalities of the electroencephalogram fine-grained features and eye movement fine-grained features after high-order correlation to obtain an optimized joint representation, and finally inputting the optimized joint representation into a classifier of the HMSB-CAF model to complete emotion state discrimination.

[0007] Optionally, the HMSB-CAF model includes a hierarchical multi-scale branch residual transformation network and a typical correlation attention fusion network, and the architecture of the hierarchical multi-scale branch residual transformation network includes a double-layer deep neural network and a gating mechanism.

[0008] Optionally, the structure of the double-layer deep neural network includes: a first layer composed of a linear layer, a normalization layer and an activation function in sequence; and a second layer composed of a linear layer, a normalization layer, a depth separable convolution, an activation function, a linear layer and a multi-scale context perception network in sequence.

[0009] Optionally, the gating mechanism comprises: point multiplying the first layer transformed feature with the feature only transformed by the first layer linear layer; and residual connecting the second layer transformed feature with the feature only transformed by the first layer linear layer.

[0010] Optionally, the double-layer deep neural network adopts a KAN network as the linear layer.

[0011] Optionally, the double-layer deep neural network adopts an improved batch feature normalization as the normalization layer, which adaptively combines the advantages of batch normalization and layer normalization according to a specified data set or a specified task.

[0012] Optionally, the double-layer deep neural network uses a one-dimensional deep separable convolution as the convolution type, and the convolution kernel size is 3.

[0013] Optionally, the multi-scale context perception network in the double-layer deep neural network is composed of four branches: the first branch is composed of 1x1 and 3x1 ordinary convolution; the second branch is composed of 1x1 and 3x1 ordinary convolution and 3x1 hollow convolution; the third branch is composed of 1x1 and 5x1 ordinary convolution and 3x1 hollow convolution; the fourth branch is composed of 1x1 ordinary convolution; wherein the first branch, the second branch and the third branch are spliced and then subjected to 1x1 ordinary convolution, followed by residual connection with the fourth branch, and finally subjected to an activation function to obtain an output.

[0014] According to a second aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory storing a program. The program comprises instructions which, when executed by the processor, cause the processor to perform the steps of the method of the first aspect described above.

[0015] According to a third aspect of an embodiment of the present application, a computer storage medium is provided, which stores a computer program. The program is executed by a processor to implement the method of the first aspect described above.

[0016] Compared with the prior art, the present application has the following beneficial effects:

[0017] (1) For the SEED, SEED-FRA and SEED-GER public data sets, the HMSB-CAF of the present application achieves an accuracy of 97.41%, 95.37% and 90.59% respectively in the ICSD experiment, and a standard deviation of 5.56, 5.51 and 7.72; and achieves an accuracy of 91.67%, 87.21% and 86.96% respectively in the ICSI experiment, and a standard deviation of 6.08, 6.89 and 5.63.

[0018] (2) Using feature visualization discovery, the modal features of the application after HMSB-TransNet transformation, the clustering of the feature points of the class in two-dimensional space is more compact, the boundary between classes is more clear, the independence of the features and the clarity of the clustering have been significantly improved.

[0019] (3) Based on the relationship between Hofstede individualism index (China: 20, Germany: 67, France: 93) and model improvement effect, it shows that the two are significantly linearly positively correlated. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0021] Figure 1 The step flow chart of the present application.

[0022] Figure 2 The corresponding overall framework diagram. Figure 1 The corresponding overall framework diagram.

[0023] Figure 3 The HMSB-CAF model structure diagram of the present application.

[0024] Figure 4 The model structure diagram of multi-scale context perception network (MSCP).

[0025] Figure 5 The relationship diagram of different cultural individualism and effect.

[0026] Figure 6 The performance gain heat map of different cultures.

[0027] Figure 7 The feature distribution visualization comparison diagram after transformation of different algorithms and original features.

[0028] Figure 8 The feature distribution visualization comparison diagram after transformation of different algorithms and original features. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0030] The following describes in detail, with reference to the accompanying drawings, an embodiment of the present invention for emotion recognition based on EEG and eye-tracking multimodal signals.

[0031] like Figure 1 As shown in the figure, an emotion recognition method based on EEG and eye-tracking multimodal signals according to an embodiment of the present invention includes:

[0032] Step S1: Preprocess the EEG and eye movement data to be tested to extract EEG features and eye movement features;

[0033] Step S2: Input the EEG features and eye movement features into the hierarchical multi-scale branch residual transformation network of the HMSB-CAF model to perform hierarchical multi-scale branch residual transformation to obtain fine-grained EEG features and fine-grained eye movement features.

[0034] Step S3: Cross-modal semantic alignment of fine-grained EEG features and fine-grained eye movement features is performed using the canonical correlation attention fusion mechanism of the HMSB-CAF model. Canonical correlation analysis is used to constrain the statistical correlation between the two modalities of fine-grained EEG features and fine-grained eye movement features in the deep feature space, forcing the distribution of EEG rhythm energy and eye movement behavior features to establish a high-order correlation in the dimension of emotion expression.

[0035] Step S4: Dynamically calculate the weight coefficients of the two modalities, EEG fine-grained features and eye movement fine-grained features, after high-order association using learnable parameterized attention vectors. Based on the weight coefficients and the contribution of features to emotion classification, adaptively fuse the two modalities, EEG fine-grained features and eye movement fine-grained features, after high-order association to obtain an optimized joint representation. Finally, input the optimized joint representation into the classifier of the HMSB-CAF model to complete the emotion state discrimination.

[0036] Optionally, the HMSB-CAF model includes a hierarchical multi-scale branch residual transform network and a canonical correlation attention fusion network, wherein the architecture of the hierarchical multi-scale branch residual transform network includes a two-layer deep neural network and a gating mechanism.

[0037] Optionally, the structure of the two-layer deep neural network includes: the first layer consists of a linear layer, a normalization layer, and an activation function in sequence; the second layer consists of a linear layer, a normalization layer, a depthwise separable convolution, an activation function, a linear layer, and a multi-scale context-aware network in sequence.

[0038] Optionally, the gating mechanism includes: performing a dot product between the features transformed by the first layer and the features transformed by only the first linear layer; and performing a residual connection between the features transformed by the second layer and the features transformed by only the first linear layer.

[0039] It should be understood that the features after the first linear layer transformation only refer to the features after the first linear layer transformation, but without the subsequent normalization layer and activation function.

[0040] Optionally, the double-layer deep neural network adopts a KAN network as the linear layer.

[0041] Optionally, the double-layer deep neural network adopts an improved batch feature normalization as the normalization layer, which adaptively combines the advantages of batch normalization and layer normalization according to a specified data set or a specified task.

[0042] Optionally, the double-layer deep neural network uses a one-dimensional deep separable convolution as the convolution type, and the convolution kernel size is 3.

[0043] Optionally, the multi-scale context perception network in the double-layer deep neural network is composed of four branches: the first branch is composed of 1x1 and 3x1 ordinary convolution; the second branch is composed of 1x1 and 3x1 ordinary convolution and 3x1 hollow convolution; the third branch is composed of 1x1 and 5x1 ordinary convolution and 3x1 hollow convolution; the fourth branch is composed of 1x1 ordinary convolution; wherein the first branch, the second branch and the third branch are spliced and then subjected to 1x1 ordinary convolution, followed by residual connection with the fourth branch, and finally subjected to an activation function to obtain an output.

[0044] In summary, the present application innovatively proposes a hierarchical multi-scale branch residual transformation network architecture (HMSB-TransNet), which introduces multi-scale context perception (MSCP), deep separable convolution, hollow convolution, BTN normalization module and KAN network. The architecture is a new network structure combining a double-layer deep neural network with a gating mechanism. The model signal is changed through the network to realize efficient fine-grained feature extraction and stable expression, and significantly enhance the modeling ability of complex semantics and spatial structure. Combined with canonical correlation analysis and attention fusion mechanism, effective complementary and adaptive fusion of modal information is realized. The present application solves the problems of insufficient deep mining of modal features and significant influence of second-order interference of modal signals in existing cross-cultural sentiment recognition methods.

[0045] Specifically, the scheme of the present application is further described according to the following examples:

[0046] Figure 2The overall workflow of the hierarchical multi-scale branch residual transformation-classical correlation attention fusion network (HMSB-CAF) model is demonstrated. First, the EEG and eye movement data of each subject are preprocessed, and features related to emotions (such as EEG differential entropy and eye movement pupil diameter) are extracted. Subsequently, the EEG and eye movement features are respectively processed by the hierarchical multi-scale branch residual transformation network (HMSB-TransNet) to obtain fine-grained feature representation. Then, the classical correlation attention fusion mechanism (CAF) is introduced, which uses canonical correlation analysis to constrain the statistical correlation between the two modalities in the deep feature space, achieving cross-modal semantic alignment. The modal weight is dynamically adjusted by the parameterized attention vector, and the dual-modal information is adaptively fused. Finally, the optimized joint features are input into the classifier to realize emotion state discrimination.

[0047] As shown in Figure 3 , HMSB-TransNet realizes multi-scale local-global feature interaction through multi-path feature segmentation and dynamic gating; introduces depth separable convolution to improve spatial pattern capture ability; combines BTN normalization module to stabilize gradient and accelerate convergence; uses multi-scale context perception network structure (MSCP) to strengthen context-dependent modeling capability, and is supplemented by cross-layer residual connection to ensure smooth gradient propagation. The specific implementation is as follows:

[0048] Assume is a modal matrix, where B is the window number of modal features, and T is the modal feature dimension. is an EEG feature data matrix, where 310 is the EEG signal feature dimension (62 channels of differential entropy features). is an eye movement feature data matrix, and 33 is the eye movement signal feature dimension (including 33 features related to emotion recognition such as gaze point coordinates, saccade velocity, and pupil diameter). The two modalities are nonlinearly transformed by HMSB-TransNet.

[0049] First, the input features are mapped to a high-dimensional space by Kolmogorov-Arnold Networks:

[0050]

[0051] where h1 is the dimension of the first hidden layer.

[0052] is decomposed into two branches along the dimension:

[0053]

[0054] where branch 1 encodes modality-specific discriminative features. Branch 2 To preserve the original dynamic range, adaptive gating weights are generated.

[0055] Normalization and nonlinear transformation are performed on branch 1:

[0056]

[0057] The present application aims at the distribution difference of static time-frequency features in multi-modal physiological signals, and makes cross-domain improvement on the basis of the batch channel normalization framework proposed by Khaled A. By reconstructing the channel-space interaction mechanism in image processing into a feature-level dynamic coupling strategy, a BTN module suitable for electroencephalogram (EEG) and eye movement (EOG) static time-frequency feature analysis is designed. The specific calculation is as follows:

[0058] Let the input be Where B is the window number of modal features, and T is the modal feature dimension.

[0059] Calculate the cross-batch statistics independently for the feature dimension:

[0060]

[0061] Where, X b,f is the t-th feature value of the b-th sample in the input data, is the cross-batch mean of the t-th feature dimension (calculated along the batch dimension), is the cross-batch variance of the t-th feature dimension (calculated along the batch dimension), and ∈ is a numerical stability constant.

[0062] The batch normalization output is:

[0063]

[0064] Calculate the statistics of all features for each sample (normalized along the feature dimension):

[0065]

[0066] Where, is the feature mean of the b-th sample (calculated along the feature dimension), is the feature variance of the b-th sample (calculated along the feature dimension).

[0067] The layer normalization output is:

[0068]

[0069] Dynamic weight fusion is performed on the batch normalization BN and layer normalization LN outputs:

[0070]

[0071] where, is the learnable fusion weight of the t-th feature dimension, without value domain constraint. If w (t) >1, the contribution of batch normalization branch is amplified, w (t) <0, the contribution of layer normalization branch is suppressed.

[0072] Affine transformation output:

[0073]

[0074] where, γ (t) is the scaling parameter of the t-th feature dimension, β (t) is the bias parameter of the t-th feature dimension.

[0075] Element-wise multiplication of branch 2 as dynamic weight and branch 1, so that the model can adaptively focus on local features related to emotion, and the gating weight area can be reversely located to the key timing segment of the original signal (such as the emotion triggering period):

[0076]

[0077] Next, through KAN secondary mapping, further extract high-order features:

[0078]

[0079] where h2 is the second layer hidden layer dimension.

[0080] Local feature extraction is performed through depth separable convolution, which captures temporal dependence and suppresses high-frequency noise. The convolution kernel slides along the time axis to capture short-term correlation (such as event-related potentials in electroencephalogram, micro-saccades in eye movement):

[0081]

[0082] Normalize and nonlinearly transform the output of this layer to enhance the nonlinearity of the data:

[0083]

[0084] Through KAN tertiary mapping, the extraction of global features is once again strengthened:

[0085]

[0086] where h3 is the third layer hidden layer dimension.

[0087] Adaptive dimension transformation, mapping the original input to the hidden space:

[0088]

[0089] The extraction of context information features is enhanced by MSCP:

[0090]

[0091] The residual connection retains the original input information, alleviates the gradient vanishing, and retains the original input information through Retaining, avoiding over-smoothing:

[0092]

[0093] Figure 4 A multi-scale context perception network (MSCP) is provided, which can effectively extract different cultural emotion features in EEG and EOG data, enhance the expression ability of small amplitude emotion changes, and reduce the influence of background noise interference on emotion recognition.

[0094] Specifically, all branches are started with a 1×1 standard convolution, which is used to transform the input features in the channel, and the feature representation is preliminarily screened and adjusted to adapt to the feature extraction requirements of different scales. The first branch adopts a 1×1→3×1 standard convolution, which retains local features with a small receptive field and forms an equivalent mapping, and can better capture key features in short-time emotional fluctuations. The second branch adopts 1×1→3×1→3×1 (dilated convolution), wherein the second 3×1 convolution kernel adopts a dilated convolution, which is used to expand the receptive field, so that the branch can obtain more rich context information, which is helpful to distinguish subtle differences in emotional patterns. The third branch adopts 1×1→5×1→3×1 (dilated convolution), which further enhances the modeling ability of long-range dependency information by adding a dilated convolution after 5×1, so that the branch can pay attention to a larger range of emotional context information. The fourth branch only adopts a 1×1 convolution, which directly maps the input features to an equivalent mapping to retain small amplitude emotional signal features and reduce the loss of key features. The outputs of the first three branches are spliced in the channel dimension, and then fused by a 1×1 standard convolution to adjust the channel dimension of the final feature representation and enhance the complementarity of cross-scale features. The fused features are then connected in residual to retain the original feature information and reduce information loss, and the final enhanced emotional features are output after the activation layer.

[0095] Experimental results:

[0096] The model of the application is realized by using the PyTorch library, and is trained on a single NVIDIA Tesla V100-PCIE image processing unit. The training process uses an RMSprop optimizer with a learning rate of 0.0001; in the experiment, the hidden layer unit sizes of the two nonlinear transformation networks are randomly searched from the ranges [100, 200] and [20, 50] respectively, the output layer dimension is 12, and the epoch size is set to 100.

[0097] Three different cultural public emotion recognition datasets were used, namely SEED (China), SEED-FRA (France) and SEED-GER (Germany) datasets. Among them, SEED contains 12 subjects, each of which completes three experiments, with 15 trials each time; SEED-FRA contains 8 French subjects, each of which also completes three experiments, with 21 trials each time; SEED-GER contains 8 German subjects, of which 4 complete three experiments and 4 complete two experiments, with 18 trials each time. The above datasets are preprocessed, the electroencephalogram extracts differential entropy (DE) features, the eye movement extracts 33 emotion-related features, and the training set and test set are divided, with allocation ratios of 9:6, 12:9 and 12:6 respectively.

[0098] The HMSB-CAF model of the application is compared with advanced cross-cultural emotion recognition models qualitatively, including Concat, SUM, MAX, Fuzzy, BDAE, DCCA-AM, MSBLS and CoDF-Net; it is worth noting that except for the two models MSBLS and CoDF-Net which refer to the numerical values in the literature, the other models are trained from scratch.

[0099] In terms of cross-cultural emotion recognition, the performance of the application is compared with that of existing methods;

[0100] Table 1 shows the average accuracy (%) and standard deviation of emotion recognition of different methods in subject-dependent experiments.

[0101] * MSBLS and * The results of CoDF-Net are from the literature, and the other results are reproduced in this study, and the best results are in bold.

[0102] Table 1

[0103]

[0104]

[0105] Table 1 shows the emotion recognition results of different methods in subject-dependent experiments, and the quantitative results show that compared with other models, the model of the application has achieved significant improvement on the three datasets:

[0106] The accuracy of SEED-CHN is 97.41%, which is 3.11% higher than MSBLS, and the standard deviation is reduced by 32.2%;

[0107] The accuracy of SEED-GER is 95.37%, and the standard deviation is reduced by nearly 40% compared with MSBLS;

[0108] SEED-FRA accuracy 90.59%, improved by 5.91 percentage points, and the standard deviation decreased by 20.7%.

[0109] Overall, the HMSB-CAN of the present application exhibits excellent performance and more stable performance in different cultural environments.

[0110] Table 2 shows the average accuracy (%) and standard deviation of emotion recognition of different methods in the subject independent experiment.

[0111] * MSBLS and * CoDF-Net results come from the cited literature, and other results are reproduced in this study, and the best results are in bold.

[0112] Table 2

[0113]

[0114] Table 2 shows the emotion recognition results of different methods in the subject independent experiment. The quantitative results show that compared with other models, the model of the present application has achieved significant improvement on three data sets: the accuracy is 91.67%, 88.21% and 86.96% respectively, and the stability is also excellent. On SEED-CHN and SEED-GER, the standard deviation of HMSB-CAN is further reduced compared with MSBLS, showing better robustness.

[0115] It should be noted that on the SEED-FRA data set, the standard deviation of HMSB-CAN is slightly higher than that of MSBLS, but the accuracy is significantly improved, and the overall still shows the advantages of the method in precision and stability. Overall, HMSB-CAN achieves higher emotion recognition accuracy and better cross-cultural generalization ability in the subject independent experiment.

[0116] Statistical verification:

[0117] Table 3 statistical verification

[0118]

[0119] Table 3 reveals the deep connection between model performance and cultural dimensions. On the SEED, SEED-FRA and SEED-GER data sets, the comparison between the improved model and the baseline method reaches statistical significance (t value range 3.38-4.27, p<0.005), and the Bayesian factor BF10 all exceed the strong evidence threshold of 10 (SEED-FRA is as high as 105.65), confirming that the performance improvement is not accidental fluctuation.

[0120] Figure 5It is shown that the effect size of Hofstede's different cultural individualism index and the model improvement is positively correlated, indicating that the higher the degree of individualism, the stronger the need for individual-specific modeling of emotion recognition. This phenomenon Figure 6 can be further verified in the experiment.

[0121] Ablation experiment:

[0122] Table 4 Ablation experiment

[0123]

[0124]

[0125] Table 4 shows the different cultural modeling capabilities of each module of HMSB-CAF. When the baseline model is gradually integrated into the gated feature network (GUFN, which is actually the basic structure after removing the KAN network, depthwise separable convolution (DWConv), batch normalization module (BTN) and multi-scale context perception network (MSCP) in HMSB-TransNet), KAN nonlinear transformation, and depthwise separable convolution (DWConv), the recognition rate of the SEED-FRA dataset shows a steep upward trend (+9.47%), revealing the modeling sensitivity of eye movement signals in individualistic culture; the introduction of cultural perception batch normalization (BTN) makes the performance of the German dataset jump by 1.06%, confirming the adaptability of cultural feature decoupling to the Western European scene; and the multi-scale context perception network (MSCP) module finally compresses the maximum performance difference between Chinese, French and German cultures from 18.22% to 6.82%, and the complete architecture realizes unified modeling of different cultural emotion representations with an accuracy of 97.41%, 90.59% and 95.37%. This module synergistic effect reveals that the gating mechanism filters the culture-invariant primitives, the spatiotemporal convolution captures the local response pattern, the normalization layer decouples the group bias noise, and finally achieves a dynamic balance of "cultural-specific modeling-non-cultural generalization" through multi-scale context fusion.

[0126] Feature visualization analysis:

[0127] In order to more intuitively analyze the HMSB-CAF model, the differences between the DCCA-AM model and the original features in the feature space are compared. The t-SEN algorithm is used for visualization analysis of the features. The data randomly selects some experimental data of three subjects in a certain dataset for display and analysis. From Figure 7It can be seen from the feature distribution after the HMSB-CAF feature transformation that the feature points of each emotion category are more closely clustered in the two-dimensional space, and the boundaries between categories are more clear. The distinction between certain emotions is obviously better than that after the DCCA-AM transformation and the original feature, and overall, the independence of the emotion feature and the clarity of the clustering have been significantly improved. From the Figure 8 The fused feature distribution visualization can clearly show that the HMSB-CAF model is obviously superior to the DCCA-AM model in the representation performance of emotion features, and can more effectively improve the distinguishability of the features, thereby being more conducive to supporting the subsequent emotion recognition task.

[0128] The present application proposes a hierarchical multi-scale branch residual transformation-classical correlation attention fusion network (HMSB-CAF), namely HMSB-CAF model, aiming at the problems of insufficient fine-grained mining of modal features and limited robustness to environmental noise in the field of multi-modal emotion recognition. The method of the present application realizes multi-granularity deep extraction of electroencephalogram and eye movement features through a multi-scale branch residual structure (HMSB-TransNet), and effectively improves the feature expression ability and noise resistance by combining a depth separable convolution and a BTN normalization module. At the same time, the classical correlation attention fusion mechanism (CAF) can realize adaptive dynamic fusion of cross-modal information in a high-order semantic space, further enhancing the model's detailed capture of emotion features and the complementary relationship between different modalities. The subject-dependent and subject-independent experimental results on the SEED-CHN, SEED-GER and SEED-FRA three datasets show that the HMSB-CAF is significantly superior to traditional methods and existing deep models in terms of accuracy and stability, and exhibits stronger cross-cultural generalization ability and robustness to complex noise environments.

[0129] As another example, the present application also provides an electronic device, which will now be described as an electronic device that can be a server or a client of the present application, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent a variety of forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections, and relationships, and their functions, are merely examples and are not intended to limit implementations of the present application described and / or claimed herein.

[0130] The electronic device can include a processor, a communications interface, a memory, and a communications bus.

[0131] The processor, the communication interface and the memory communicate with each other through a communication bus. The communication interface is configured to communicate with other electronic devices or servers.

[0132] The processor is configured to execute a program, and specifically, can execute the related steps in the method embodiments.

[0133] Specifically, the program can include program code, and the program code includes computer operation instructions.

[0134] The processor can be a processor CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0135] The memory is configured to store the program. The memory can include a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory.

[0136] The program, when executed by the processor, is configured to cause the electronic device to perform a method for emotion recognition based on electroencephalogram and eye movement multi-modal signals.

[0137] In addition, the specific implementation of each step in the program can refer to the corresponding description of the corresponding steps and units in the method embodiments, and will not be described here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the devices and modules described above can refer to the corresponding process description in the foregoing method embodiments, and will not be described here.

[0138] The exemplary embodiments of the present application also provide a computer storage medium storing a computer program, wherein the computer program is executed by the processor to implement the method of the embodiments of the present application, and can refer to the corresponding process description in the foregoing method embodiments, and will not be described here.

[0139] The above-described methods according to embodiments of the application can be implemented in hardware, firmware, or software, or any combination thereof, and can be implemented as software storable on a recording medium which is readable from a general use computer, a special processor, or programmable or special hardware (such as an ASIC or an FPGA) using such software. It is understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (for example, RAM, ROM, flash memory, etc.) that can store or receive software or computer code when accessed and executed by the computer, the processor, or the hardware, to implement the methods described herein. Furthermore, when a general use computer accesses code for implementing the methods shown herein, the execution of the code transforms the general use computer into a special purpose computer for executing the methods shown herein.

[0140] Thus far, specific embodiments of the application have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0141] It should be understood that, although the specification is described with respect to various embodiments, not every embodiment contains only one independent technical solution, and the specification is described in this way only for clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that those skilled in the art can understand.

[0142] Finally, it should be noted that: the above implementation is only used to illustrate the embodiments of the present application, and not to limit the embodiments of the present application, and those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application, and the patent protection scope of the embodiments of the present application should be defined by the claims.

Claims

1. An emotion recognition method based on EEG and eye-tracking multimodal signals, characterized in that, include: The EEG and eye movement data to be tested are preprocessed to extract EEG and eye movement features; EEG features and eye movement features are respectively input into the hierarchical multi-scale branch residual transform network of the HMSB-CAF model to perform hierarchical multi-scale branch residual transform, thereby obtaining fine-grained EEG features and fine-grained eye movement features. Cross-modal semantic alignment of fine-grained EEG features and fine-grained eye movement features was performed using the canonical correlation attention fusion mechanism of the HMSB-CAF model. Canonical correlation analysis was used to constrain the statistical correlation between the two modalities of fine-grained EEG features and fine-grained eye movement features in the deep feature space, forcing the EEG rhythm energy distribution and eye movement behavior features to establish a high-order association in the emotion expression dimension. The weight coefficients of two modalities, EEG fine-grained features and eye movement fine-grained features after high-order association, are dynamically calculated using learnable parameterized attention vectors. Based on the weight coefficients and the contribution of features to emotion classification, the two modalities of EEG fine-grained features and eye movement fine-grained features after high-order association are adaptively fused to obtain an optimized joint representation. Finally, the optimized joint representation is input into the classifier of the HMSB-CAF model to complete the emotion state discrimination.

2. The method according to claim 1, characterized in that, The HMSB-CAF model includes a hierarchical multi-scale branch residual transformation network and a canonical correlation attention fusion network. The architecture of the hierarchical multi-scale branch residual transformation network includes a two-layer deep neural network and a gating mechanism.

3. The method according to claim 2, characterized in that, The structure of the two-layer deep neural network includes: The first layer consists of a linear layer, a normalized layer, and an activation function in sequence; The second layer consists of a linear layer, a normalization layer, a depthwise separable convolution, an activation function, another linear layer, and a multi-scale context-aware network.

4. The method according to claim 2, characterized in that, The gating mechanism includes: Perform a dot product between the features transformed by the first layer and the features transformed by only the first linear layer. Perform a residual connection between the features transformed by the second layer and the features transformed by only the first linear layer.

5. The method according to claim 3, characterized in that, The two-layer deep neural network uses the KAN network as the linear layer.

6. The method according to claim 3, characterized in that, The dual-layer deep neural network uses an improved batch feature normalization as a normalization layer, which adaptively combines the advantages of batch normalization and layer normalization according to a specified dataset or task.

7. The method according to claim 3, characterized in that, The two-layer deep neural network uses a one-dimensional depthwise separable convolution with a kernel size of 3.

8. The method according to claim 3, characterized in that, The multi-scale context-aware network in the two-layer deep neural network consists of four branches: The first branch consists of ordinary convolutions of 1×1 and 3×1; The second branch consists of 1×1 and 3×1 ordinary convolutions and 3×1 dilated convolutions; The third branch consists of 1×1 and 5×1 ordinary convolutions and 3×1 dilated convolutions; The fourth branch consists of 1×1 ordinary convolutions; The first, second, and third branches are concatenated and then subjected to a 1×1 ordinary convolution. Next, a residual connection is made with the fourth branch, and finally, the output is obtained after passing through an activation function.

9. An electronic device, characterized in that, include: processor; Memory for stored programs; The program includes instructions that, when executed by the processor, cause the processor to perform the steps of the method as described in any one of claims 1-8.

10. A computer storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1-8.

Images