Multi-mode fusion emotion recognition method and device based on electroencephalogram and eye movement signals

Abstract

The embodiment of the invention relates to the field of data processing, and provides a multimode fusion emotion recognition method and device based on electroencephalogram and eye movement signals, and the method comprises the steps: extracting electroencephalogram signals corresponding to K reference users used during model training, and obtaining a first electroencephalogram signal set, obtaining an eye movement signal set corresponding to K reference users used during model training, and obtaining a first eye movement signal set; performing feature fusion on first electroencephalogram signals in the first electroencephalogram signal set and first eye movement signals in the first eye movement signal set to obtain a first mixed feature information set; training an initial model by using the first mixed feature information in the first mixed feature information set to obtain a target model; performing emotion mode recognition on a target user by using the target model to obtain a target emotion mode recognition result; and the accuracy of the target model in performing emotion mode recognition on the target user is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a multimodal fusion emotion recognition method and device based on EEG and eye movement signals. Background Technology

[0002] Emotions are an internal, subjective experience humans have when facing unexpected events; they represent an attitude and behavioral response to objective things. Emotions are indispensable in people's daily work and life, and maintaining a good emotional state is crucial for their physical and mental health. Therefore, how to quickly acquire emotional information, and simultaneously analyze, process, and identify emotional states, has become a research hotspot in the fields of human-computer interaction and artificial intelligence.

[0003] Research has shown that both eye-tracking and electroencephalogram (EEG) features contain emotional information, which can be used in emotion pattern recognition. Furthermore, emotion pattern recognition based on physiological signals provides quantitative indicators for changes in emotional state, making the results of emotion pattern recognition more accurate and precise.

[0004] However, existing emotion pattern recognition methods do not sufficiently extract the spatiotemporal features of EEG signals, do not fully fuse multimodal features, and fail to effectively utilize modal complementarity, resulting in insufficient accuracy and robustness of emotion pattern recognition, thus leading to low accuracy in the emotion pattern recognition results for target users. Summary of the Invention

[0005] This application provides a multi-modal fusion emotion recognition method based on EEG and eye movement signals, which can improve the accuracy of the target model in recognizing the emotion patterns of the target user.

[0006] A first aspect of this application provides a multimodal fusion emotion recognition method based on electroencephalogram (EEG) and eye-tracking signals, the method comprising: Extract the EEG signals corresponding to the K reference users used in training the model to obtain the first EEG signal set, and obtain the eye movement signal set corresponding to the K reference users used in training the model to obtain the first eye movement signal set; The first EEG signal in the first EEG signal set and the first eye movement signal in the first eye movement signal set are fused to obtain a first mixed feature information set. The initial model is trained using the first mixed feature information from the first mixed feature information set to obtain the target model; The target model is used to perform emotion pattern recognition on the target user to obtain the target emotion pattern recognition result.

[0007] In this example, a first set of EEG signals is obtained by extracting the EEG signals of K reference users used in training the model, and a first set of eye-tracking signals is obtained by acquiring the eye-tracking signal set of K reference users used in training the model. Feature fusion is performed on the first EEG signals in the first set of EEG signals and the first eye-tracking signals in the first set of eye-tracking signals to obtain a first set of mixed feature information. The first mixed feature information in the first set of mixed feature information is used to train the initial model to obtain the target model. The target model is then used to perform emotion pattern recognition on the target user to obtain the target emotion pattern recognition result, thereby improving the accuracy of the target model in performing emotion pattern recognition on the target user.

[0008] In one possible implementation, a method for feature fusion of a first EEG signal in a first EEG signal set and a first eye movement signal in a first eye movement signal set to obtain a first mixed feature information set includes: Preprocessing is performed on the first EEG signal in the first EEG signal set and the first eye movement signal in the first eye movement signal set to obtain the second EEG signal set and the second eye movement signal set. The first feature information set is obtained by extracting features from the second EEG signal in the second EEG signal set. Feature extraction is performed on the second eye movement signals in the second eye movement signal set to obtain the second feature information set; The first feature information in the first feature information set and the second feature information in the second feature information set are fused to obtain the first mixed feature information set.

[0009] In one possible implementation, a method for extracting features from a second set of EEG signals in a second set of EEG signals to obtain a first set of feature information includes: Construct the adjacency matrix corresponding to the second EEG signal in the second EEG signal set to obtain the first adjacency matrix set; Based on the first adjacency matrix in the first adjacency matrix set, the second EEG signal in the second EEG signal set is subjected to GATv2 graph convolution processing to obtain the first graph feature information set; Based on the first adjacency matrix in the first adjacency matrix set, the second EEG signal in the second EEG signal set is subjected to GENConv graph convolution processing to obtain the second graph feature information set; The first image feature information in the first image feature information set and the second image feature information in the second image feature information set are fused to obtain the first feature information set to be processed. Temporal enhancement processing is performed on the first feature information in the first set of feature information to be processed to obtain the first feature information set.

[0010] In one possible implementation, a method for extracting features from second eye-movement signals in a second eye-movement signal set to obtain a second feature information set includes: The second eye movement signal in the second eye movement signal set is convolved to extract the spatiotemporal feature information corresponding to the second eye movement signal, thus obtaining the first spatiotemporal feature information set. The first spatiotemporal feature information in the first spatiotemporal feature information set is subjected to dimensionality compression processing to obtain the second spatiotemporal feature information set. The second spatiotemporal feature information in the second spatiotemporal feature information set is reorganized in dimensions to obtain the second feature information set to be processed.

[0011] In one possible implementation, a method for feature fusion of first feature information in a first feature information set and second feature information in a second feature information set to obtain a first mixed feature information set includes: The first feature information in the first feature information set and the second feature information in the second feature information set are initially fused to obtain the third feature information set. Calculate the global features corresponding to K reference users based on the third feature information in the third feature information set to obtain the fourth feature information set; Based on the fourth feature information in the fourth feature information set, calculate the attention weight information corresponding to the first feature information in the first feature information set to obtain the first weight information set; Based on the fourth feature information in the fourth feature information set, calculate the attention weight information corresponding to the second feature information in the second feature information set to obtain the second weight information set; The first feature information in the first feature information set is enhanced to obtain the fifth feature information set, and the second feature information in the second feature information set is enhanced to obtain the sixth feature information set; The fifth feature information in the fifth feature information set and the sixth feature information in the sixth feature information set are weighted and fused based on the first weight information in the first weight information set and the second weight information in the second weight information set to obtain the seventh feature information set. The seventh feature information in the seventh feature information set is subjected to residual connection processing to obtain the first mixed feature information set.

[0012] A second aspect of this application provides a multimodal fusion emotion recognition device based on electroencephalogram (EEG) and eye-tracking signals, the device comprising: The extraction unit is used to extract the EEG signals corresponding to the K reference users used in training the model to obtain the first EEG signal set, and to obtain the eye movement signal set corresponding to the K reference users used in training the model to obtain the first eye movement signal set. The fusion unit is used to perform feature fusion on the first EEG signal in the first EEG signal set and the first eye movement signal in the first eye movement signal set to obtain a first mixed feature information set. The training unit is used to train the initial model using the first mixed feature information in the first mixed feature information set to obtain the target model; The recognition unit is used to perform emotion pattern recognition on the target user using the target model, and obtain the target emotion pattern recognition result.

[0013] In one possible implementation, the fusion unit is specifically used for: Preprocessing is performed on the first EEG signal in the first EEG signal set and the first eye movement signal in the first eye movement signal set to obtain the second EEG signal set and the second eye movement signal set. The first feature information set is obtained by extracting features from the second EEG signal in the second EEG signal set. Feature extraction is performed on the second eye movement signals in the second eye movement signal set to obtain the second feature information set; The first feature information in the first feature information set and the second feature information in the second feature information set are fused to obtain the first mixed feature information set.

[0014] In one possible implementation, the fusion unit is specifically used for: Construct the adjacency matrix corresponding to the second EEG signal in the second EEG signal set to obtain the first adjacency matrix set; Based on the first adjacency matrix in the first adjacency matrix set, the second EEG signal in the second EEG signal set is subjected to GATv2 graph convolution processing to obtain the first graph feature information set; Based on the first adjacency matrix in the first adjacency matrix set, the second EEG signal in the second EEG signal set is subjected to GENConv graph convolution processing to obtain the second graph feature information set; The first image feature information in the first image feature information set and the second image feature information in the second image feature information set are fused to obtain the first feature information set to be processed. Temporal enhancement processing is performed on the first feature information in the first set of feature information to be processed to obtain the first feature information set.

[0015] In one possible implementation, the fusion unit is specifically used for: The second eye movement signal in the second eye movement signal set is convolved to extract the spatiotemporal feature information corresponding to the second eye movement signal, thus obtaining the first spatiotemporal feature information set. The first spatiotemporal feature information in the first spatiotemporal feature information set is subjected to dimensionality compression processing to obtain the second spatiotemporal feature information set. The second spatiotemporal feature information in the second spatiotemporal feature information set is reorganized in dimensions to obtain the second feature information set to be processed.

[0016] In one possible implementation, the fusion unit is specifically used for: The first feature information in the first feature information set and the second feature information in the second feature information set are initially fused to obtain the third feature information set. Calculate the global features corresponding to K reference users based on the third feature information in the third feature information set to obtain the fourth feature information set; Based on the fourth feature information in the fourth feature information set, calculate the attention weight information corresponding to the first feature information in the first feature information set to obtain the first weight information set; Based on the fourth feature information in the fourth feature information set, calculate the attention weight information corresponding to the second feature information in the second feature information set to obtain the second weight information set; The first feature information in the first feature information set is enhanced to obtain the fifth feature information set, and the second feature information in the second feature information set is enhanced to obtain the sixth feature information set; The fifth feature information in the fifth feature information set and the sixth feature information in the sixth feature information set are weighted and fused based on the first weight information in the first weight information set and the second weight information in the second weight information set to obtain the seventh feature information set. The seventh feature information in the seventh feature information set is subjected to residual connection processing to obtain the first mixed feature information set.

[0017] A third aspect of this application provides a terminal including a processor, an input device, an output device, and a memory, wherein the processor, input device, output device, and memory are interconnected, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to invoke the program instructions to execute the step instructions as described in the first aspect of this application.

[0018] A fourth aspect of this application provides a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the first aspect of this application.

[0019] A fifth aspect of this application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of this application. The computer program product may be a software installation package. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This application provides a flowchart illustrating a multimodal fusion emotion recognition method based on electroencephalogram (EEG) and eye-tracking signals. Figure 2 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application; Figure 3 This application provides a schematic diagram of the structure of a multi-modal fusion emotion recognition device based on EEG and eye movement signals. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0024] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0025] To better understand the multimodal fusion emotion recognition method based on EEG and eye movement signals provided in this application, a brief introduction to existing emotion recognition methods is given below. Existing methods typically involve simply extracting features from the target user's eye movement and EEG signals, fusing these features to obtain mixed feature information, and then inputting this mixed feature information into a recognition model to perform emotion pattern recognition on the target user, thus obtaining the recognition result. However, existing emotion pattern recognition methods suffer from insufficient spatiotemporal feature extraction from EEG signals, inadequate multimodal feature fusion, and failure to effectively utilize modal complementarity, resulting in insufficient accuracy and robustness in emotion pattern recognition, and consequently, low accuracy in identifying the target user's emotion patterns.

[0026] To address the aforementioned technical problems, this application provides a multi-modal fusion emotion recognition method based on EEG and eye-tracking signals. This method involves fusing features from EEG and eye-tracking signals corresponding to K reference users to obtain a first set of mixed feature information; training an initial model based on the first mixed feature information in the first set to obtain a target model; and using the target model to perform emotion pattern recognition on a target user to obtain the target emotion pattern recognition result. This improves the accuracy of the target model in recognizing emotion patterns on the target user.

[0027] Please see Figure 1 , Figure 1 This application provides a flowchart illustrating a multimodal fusion emotion recognition method based on electroencephalogram (EEG) and eye-tracking signals, as illustrated in this embodiment. Figure 1 As shown, the method includes: 101. Extract the EEG signals corresponding to the K reference users used in training the model to obtain the first EEG signal set, and obtain the eye movement signal set corresponding to the K reference users used in training the model to obtain the first eye movement signal set.

[0028] This can be achieved by downloading a publicly available emotion pattern recognition training dataset online, extracting EEG signals from K reference users in the dataset to obtain a first set of EEG signals, and extracting eye movement signals from the dataset to obtain a first set of eye movement signals. The emotion pattern recognition training dataset includes EEG and eye movement signals from K reference users, each with corresponding data labels. These labels represent the emotional information of the reference users corresponding to the first eye movement and first EEG signals. When processing the first eye movement and first EEG signals, the data labels are assigned to the processed signals. The emotional information from the K reference users can be categorized into four types: happiness, sadness, fear, and neutrality.

[0029] 102. Perform feature fusion on the first EEG signal in the first EEG signal set and the first eye movement signal in the first eye movement signal set to obtain a first mixed feature information set.

[0030] Specifically, this can be achieved by preprocessing the first EEG information in the first EEG information set and the first eye movement signal in the first eye movement signal set to obtain a second EEG signal set and a second eye movement signal set; performing convolution processing on the second EEG signal in the second EEG signal set and the second eye movement signal in the second eye movement signal set to extract the feature information corresponding to the second EEG signal in the second EEG signal set and the second eye movement signal in the second eye movement signal set, thus obtaining a first feature information set and a second feature information set; calculating the self-attention weight information corresponding to the first feature information in the first feature information set to obtain a first weight information set; calculating the self-attention weight information corresponding to the second feature information in the second feature information set to obtain a second weight information set; and performing feature fusion on the first feature information in the first feature information set and the second feature information in the second feature information set based on the first weight information in the first weight information set and the second weight information in the second weight information set to obtain a first hybrid feature information set.

[0031] 103. Use the first mixed feature information in the first mixed feature information set to train the initial model to obtain the target model.

[0032] Specifically, the initial model can be constructed by using a common 10-fold cross-validation method to input the first mixed feature information from the first mixed feature information set into an initial model. The initial model can classify the first mixed feature information from the input set into four categories: happiness, sadness, fear, and neutrality, resulting in a first classification result set. The classification accuracy of the initial model when classifying the first mixed feature information is calculated based on the first classification results in the first classification result set, yielding a first accuracy. The initial model is then judged to be less than a preset accuracy threshold. If the first accuracy is greater than or equal to the preset accuracy threshold, the initial model's accuracy in emotion pattern recognition meets expectations, and the initial model is designated as the target model. If the first accuracy is less than the preset accuracy threshold, the initial model's accuracy in emotion pattern recognition does not meet expectations, and the learning rate of the initial model is adjusted until the accuracy of the initial model after adjusting the learning rate when classifying the first mixed feature information from the first mixed feature information set is greater than or equal to the preset accuracy threshold. The initial model after adjusting the learning rate is then designated as the target model.

[0033] Specifically, the classification accuracy of the initial model when classifying the first mixed feature information can be calculated based on the first classification result using the method shown in the following formula, thus obtaining the first accuracy: In the formula Indicates the first accuracy rate; This indicates the number of first-class results in the first-class result set; This represents the number of correctly classified first-class results in the first-class result set. A correctly classified first-class result can be understood as a first-class result in the first-class result set that matches the data label.

[0034] 104. Use the target model to perform emotion pattern recognition on the target user and obtain the target emotion pattern recognition result.

[0035] Specifically, the process involves extracting the target user's eye movement signal using a general eye movement signal extraction method to obtain the target eye movement signal; extracting the target user's electroencephalogram (EEG) signal using a general electroencephalogram (EEG) signal extraction method to obtain the target EEG signal; fusing the target EEG signal and the target eye movement signal using a method that obtains identical first mixed feature information from a first mixed feature information set to obtain second mixed feature information; and inputting the second mixed feature information into the target model to perform emotion pattern recognition on the target user to obtain the target emotion pattern recognition result. Here, performing emotion pattern recognition on the target user can be understood as identifying the target user's current emotional state; emotional states include happiness, sadness, fear, and neutrality; and the target emotion pattern recognition result can be any one of happiness, sadness, fear, or neutrality.

[0036] In this example, a first set of EEG signals is obtained by extracting the EEG signals of K reference users used in training the model, and a first set of eye-tracking signals is obtained by acquiring the eye-tracking signal set of K reference users used in training the model. Feature fusion is performed on the first EEG signals in the first set of EEG signals and the first eye-tracking signals in the first set of eye-tracking signals to obtain a first set of mixed feature information. The first mixed feature information in the first set of mixed feature information is used to train the initial model to obtain the target model. The target model is then used to perform emotion pattern recognition on the target user to obtain the target emotion pattern recognition result, thereby improving the accuracy of the target model in performing emotion pattern recognition on the target user.

[0037] In one possible implementation, a method for feature fusion of a first EEG signal in a first EEG signal set and a first eye movement signal in a first eye movement signal set to obtain a first mixed feature information set includes: A1. Preprocess the first EEG signal in the first EEG signal set and the first eye movement signal in the first eye movement signal set to obtain the second EEG signal set and the second eye movement signal set. A2. Extract features from the second EEG signal in the second EEG signal set to obtain the first feature information set; A3. Extract features from the second eye movement signals in the second eye movement signal set to obtain the second feature information set; A4. Perform feature fusion between the first feature information in the first feature information set and the second feature information in the second feature information set to obtain a first mixed feature information set.

[0038] Specifically, the first EEG signal in the first EEG signal set can be filtered using a 1-75 Hz bandpass filter, and the first EEG signal in the filtered first EEG signal set can be baseline corrected using a general baseline correction method to obtain a first corrected EEG signal set; the first corrected EEG signal in the first corrected EEG signal set can be downsampled using a general downsampling method to generate an EEG signal with a format of 62 channels × 60 time points × 5 features to obtain a second EEG signal set; the first eye movement signal in the first eye movement signal set can be analyzed using a general PCA analysis method to eliminate the influence of illumination in the pupil diameter of the first eye movement signal, and the first eye movement signal in the first eye movement signal set can be downsampled using a general downsampling method to generate an eye movement signal with a format of 1 channel × 60 time points × 31 features to obtain a second eye movement signal set.

[0039] After obtaining the second set of EEG signals and the second set of eye-tracking signals, the first set of feature information can be obtained by performing convolution processing on the second EEG signals in the second set of EEG signals and extracting the graph features corresponding to the second EEG signals.

[0040] While obtaining the first set of feature information, the second set of feature information can be obtained by performing convolution processing on the second eye movement signal in the second set of eye movement signals to extract the spatiotemporal feature information corresponding to the second eye movement signal.

[0041] After obtaining the first feature information set and the second feature information set, the first feature information in the first feature information set and the second feature information in the second feature information set can be initially fused, and the attention weight information corresponding to the first feature information in the first feature information set and the second feature information in the second feature information set can be calculated to obtain the first weight information set and the second weight information set; based on the first weight information in the first weight information set and the second weight information in the second weight information set, the first feature information in the first feature information set and the second feature information in the second feature information set can be feature fused to obtain the first mixed feature information.

[0042] In one possible implementation, a method for extracting features from a second set of EEG signals in a second set of EEG signals to obtain a first set of feature information includes: B1. Construct the adjacency matrix corresponding to the second EEG signal in the second EEG signal set to obtain the first adjacency matrix set; B2. Based on the first adjacency matrix in the first adjacency matrix set, perform GATv2 graph convolution processing on the second EEG signal in the second EEG signal set to obtain the first graph feature information set; B3. Based on the first adjacency matrix in the first adjacency matrix set, perform GENConv graph convolution processing on the second EEG signal in the second EEG signal set to obtain the second graph feature information set; B4. Perform feature fusion on the first image feature information in the first image feature information set and the second image feature information in the second image feature information set to obtain the first feature information set to be processed. B5. Perform temporal enhancement processing on the first feature information in the first set of feature information to be processed to obtain the first feature information set.

[0043] This can be achieved by constructing the adjacency matrix corresponding to the second EEG signal in the second EEG signal set using a preset adjacency matrix generator, thereby obtaining the first adjacency matrix set.

[0044] Specifically, the adjacency matrix corresponding to the second EEG signal in the second EEG signal set can be constructed using the method shown in the following formula, thus obtaining the first adjacency matrix set: In the formula Represents the first adjacency matrix in the first adjacency matrix set; This represents the Sigmoid activation function; This represents the weight matrix corresponding to the preset adjacency matrix generator; This refers to the second EEG signal in the second set of EEG signals; The correlation coefficient matrix corresponding to the second EEG signal can be calculated using the Pierre correlation coefficient calculation method. This represents the bias vector corresponding to the preset adjacency matrix generator.

[0045] After obtaining the first adjacency matrix set, features of the second EEG signal can be extracted by inputting the first adjacency matrix from the first adjacency matrix set and the second EEG signal from the second EEG signal set into a preset first feature extraction module. The first feature extraction module includes a GATv2 convolutional neural network and a GENConv convolutional neural network. First, the first adjacency matrix from the first adjacency matrix set and the second EEG signal from the second EEG signal set are input into the GATv2 convolutional neural network for graph convolution processing. The GATv2 convolutional neural network is a self-attention graph neural network model that can convert the first adjacency matrix from the first adjacency matrix set and the second EEG signal from the second EEG signal set into a graph structure through K built-in self-attention heads, thereby improving the correlation between the first adjacency matrix from the first adjacency matrix set and the second EEG signal from the second EEG signal set. Specifically, the node feature information corresponding to the second EEG information in the second EEG signal set can be extracted using a general node feature extraction method through a GATv2 convolutional neural network to obtain a first node feature information set; the GATv2 convolutional neural network can calculate the graph feature information corresponding to the second EEG signal in the second EEG signal set based on the first adjacency matrix in the first adjacency matrix set and the first node feature information in the first node feature information set to obtain a first graph feature information set.

[0046] Specifically, the graph feature information corresponding to the second EEG signal in the second EEG signal set can be calculated based on the first adjacency matrix in the first adjacency matrix set and the first node feature information in the first node feature information set, as shown in the following formula, to obtain the first graph feature information set: In the formula This represents the feature information of the first image in the feature information set of the first image; This represents the average value corresponding to the multi-head result of the GATv2 convolutional neural network; This represents the Sigmoid activation function; This represents the k-th attention head in the GATv2 convolutional neural network; This represents the attention weight of node j to node i in the k-th attention head of the GATv2 convolutional neural network; This represents the linear transformation matrix corresponding to the k-th attention head in the GATv2 convolutional neural network; This represents the j-th first node feature in the first node feature information set; This represents the element corresponding to the i-th row and j-th column in the first adjacency matrix, when... When the j-th node in the GATv2 convolutional neural network is a neighbor of the i-th node, then node j is the neighbor set. The elements in.

[0047] While obtaining the first set of graph feature information, the first adjacency matrix in the first adjacency matrix set and the second EEG signal in the second EEG signal set can be input into the GENConv convolutional neural network in the preset first feature extraction module. The GENConv convolutional neural network is an adaptive normalized graph neural network model that can transform the first adjacency matrix in the first adjacency matrix set and the second EEG signal in the second EEG signal set into a graph structure, thereby capturing the complex spatiotemporal relationships and brain region interaction patterns in the first adjacency matrix in the first adjacency matrix set and the second EEG signal in the second EEG signal set. Specifically, the GENConv convolutional neural network can use a general node feature extraction method to extract the node feature information corresponding to the second EEG signal in the second EEG signal set, obtaining the second set of node feature information; the GENConv convolutional neural network can then calculate the graph feature information corresponding to the second EEG signal in the second EEG signal set based on the first adjacency matrix in the first adjacency matrix set and the second node feature information in the second set of node feature information, obtaining the second set of graph feature information.

[0048] Specifically, the graph feature information corresponding to the second EEG signal in the second EEG signal set can be calculated based on the first adjacency matrix in the first adjacency matrix set and the second node feature information in the second node feature information set, as shown in the following formula, to obtain the second graph feature information set: In the formula This represents the feature information of the second image in the feature information set of the second image; Represents a multilayer perceptron; This represents a learnable scalar parameter, which can be determined by user input or by system default. This represents the i-th feature information of the second node in the set of second node feature information; This represents the adaptive edge weight coefficients, which can be determined by user input or by system default. The learnable weight matrix can be determined by user input or by system default. This represents the j-th feature information of the second node in the set of second node feature information; This represents the element corresponding to the i-th row and j-th column in the first adjacency matrix, when... When , it means that the j-th node in the GENConv convolutional neural network is a neighbor of the i-th node, then node j is the neighbor set. The elements in.

[0049] After obtaining the first image feature information set and the second image feature information set, the first image feature information in the first image feature information set and the second image feature information in the second image feature information set can be spliced ​​together to obtain the first spliced ​​image feature information set; the first image feature information in the first spliced ​​image feature information set can be weighted and fused to obtain the second spliced ​​image feature information set; and the second image feature information in the second spliced ​​image feature information set can be layer normalized to obtain the first feature information set to be processed.

[0050] Specifically, the first image feature information in the first image feature information set and the second image feature information in the second image feature information set can be fused using the method shown in the following formula to obtain the first set of feature information to be processed: In the formula This represents the first feature information to be processed in the first feature information set; Presentation layer normalization processing; The fusion weight matrix can be determined by user input or by system default. This indicates splicing / joining. This represents the feature information of the first image in the feature information set of the first image; This represents the feature information of the second image in the feature information set of the second image; This indicates the fusion bias term, which can be determined by user input or by the system default.

[0051] After obtaining the first set of feature information to be processed, the first feature information to be processed in the first set of feature information to be processed can be input into a preset time series modeling enhancement module. The preset time series modeling enhancement module enhances the time series features corresponding to the first feature information to be processed to obtain the first set of feature information.

[0052] Specifically, the first feature information in the first set of feature information to be processed can be subjected to temporal enhancement processing using the method shown in the following formula to obtain the first feature information set: In the formula This represents the first feature information in the first feature information set, and ; This represents the first feature information to be processed in the first set of feature information to be processed; This indicates dimensional reshaping processing, used to... The dimension from Remodeling N represents the timing channel; Indicates to Add position encoding processing; Indicates the use of a general encoder pair Perform encoding processing to change its format from Convert to ; This indicates temporal attention pooling processing.

[0053] In this example, two different convolution processing methods are used to convolve the second EEG signals in the second EEG signal set to obtain a first image feature information set and a second image feature information set. Feature fusion is then performed on the first image feature information in the first image feature information set and the second image feature information in the second image feature information set to obtain a first feature information set to be processed. Temporal enhancement processing is then performed on the first feature information set to be processed to obtain the first feature information. This allows for the extraction of deep features from the second EEG signals in the second EEG signal set, and temporal enhancement of these deep features improves the accuracy of the first feature information in the first feature information set. Consequently, the accuracy of the target model in recognizing emotion patterns in the target user is improved.

[0054] In one possible implementation, a method for extracting features from second eye-movement signals in a second eye-movement signal set to obtain a second feature information set includes: C1. Perform convolution processing on the second eye movement signal in the second eye movement signal set, extract the spatiotemporal feature information corresponding to the second eye movement signal, and obtain the first spatiotemporal feature information set. C2. Perform dimensionality compression processing on the first spatiotemporal feature information in the first spatiotemporal feature information set to obtain the second spatiotemporal feature information set. C3. Perform dimensional reorganization on the second spatiotemporal feature information in the second spatiotemporal feature information set to obtain the second feature information set to be processed.

[0055] This can be achieved by inputting the second eye-movement signal from the second eye-movement signal set into a preset second feature extraction module, which includes a first convolutional layer and a second convolutional layer. First, the second eye-movement signal from the second eye-movement signal set is input into the first convolutional layer of the preset second feature extraction module. The first convolutional layer performs dimensionality rearrangement processing on the second eye-movement signal from the second eye-movement signal set using a common dimensionality rearrangement method to obtain a third eye-movement signal set. The third eye-movement signal from the third eye-movement signal set is then subjected to convolution processing to extract spatiotemporal features, resulting in a set of spatiotemporal feature information to be processed. Finally, the spatiotemporal feature information to be processed in the set of spatiotemporal feature information to be processed is subjected to batch normalization, activation processing, and max pooling processing sequentially to obtain a first set of spatiotemporal feature information.

[0056] Specifically, the second eye movement signal in the second eye movement signal set can be convolved using the method shown in the following formula to extract the spatiotemporal feature information corresponding to the second eye movement signal, thus obtaining the first spatiotemporal feature information set: In the formula This represents the first spatiotemporal feature information in the first spatiotemporal feature information set, and 16 represents The feature representation of the channel dimension; This indicates max pooling. This indicates that activation is being performed; This indicates batch normalization processing; Indicates convolution processing; This indicates dimensional rearrangement processing; This represents the second eye movement signal in the set of second eye movement signals, and , The second eye movement signal is represented as a 4-dimensional tensor, B represents the batch of the second eye movement signal, S represents the sequence length of the second eye movement signal, F represents the number of features of the second eye movement signal, and 1 indicates... The feature representation of the channel dimension; This represents the kernel weights corresponding to the first convolutional layer; This represents the bias term corresponding to the first convolutional layer, which can be determined by user input or by system default.

[0057] After obtaining the first set of spatiotemporal feature information, the first spatiotemporal feature information in the first set of spatiotemporal feature information can be input into the second convolutional layer in the preset second feature extraction module. The second convolutional layer performs convolution, batch normalization, activation and adaptive average pooling on the first spatiotemporal feature information in the first set of spatiotemporal feature information in sequence, compressing the spatial dimension corresponding to the first spatiotemporal feature information in the first set of spatiotemporal feature information to 1, thereby generating the global feature representation corresponding to the second eye movement signal, and obtaining the second set of spatiotemporal feature information.

[0058] Specifically, the second spatiotemporal feature information set can be obtained by performing dimensionality compression on the first spatiotemporal feature information set in the first spatiotemporal feature information set using the method shown in the following formula: In the formula This represents the second spatiotemporal feature information in the second spatiotemporal feature information set, and This indicates that the second spatiotemporal feature information is a 4-dimensional tensor; This indicates that adaptive average pooling can be used to unify the sequence lengths corresponding to the second spatiotemporal feature information in the second spatiotemporal feature information set. Indicates activation processing; This indicates batch normalization processing; Indicates convolution processing; This represents the first spatiotemporal feature information in the first spatiotemporal feature information set; This represents the kernel weights corresponding to the second convolutional layer; This represents the bias term corresponding to the second convolutional layer, which can be determined by user input or by system default.

[0059] After obtaining the second spatiotemporal feature information set, the second spatiotemporal feature information in the second spatiotemporal feature information set can be dimensionally reorganized by a preset GRU module. The gated loop mechanism in the GRU module is used to capture the temporal dependencies in the second spatiotemporal feature information set, and the second spatiotemporal feature information is converted from a 4-dimensional tensor to a 2-dimensional tensor to obtain the second feature information set.

[0060] Specifically, the second spatiotemporal feature information in the second spatiotemporal feature information set can be re-dimension-reorganized using a preset GRU module, as shown in the following formula, transforming the second spatiotemporal feature information from a 4-dimensional tensor into a 2-dimensional tensor to obtain the second feature information set: In the formula This represents the second feature information in the second feature information set, and ; Indicates to Perform dimensional restructuring; Indicates to Perform sequence modeling processing; Indicates to Perform the extrusion operation.

[0061] In this example, by performing convolution processing on the second eye movement signals in the second eye movement signal set and then performing dimensionality compression processing on the output of the convolution processing, global features corresponding to the second eye movement signals are extracted to obtain a second spatiotemporal feature information set. The second spatiotemporal features in the second spatiotemporal feature information set are then subjected to dimensionality reorganization processing to extract fixed-dimensional context-aware feature vectors of the second eye movement signals to obtain a second feature information set. This improves the accuracy of the second feature information in the second feature information set, thereby improving the accuracy of the target model in recognizing the emotion patterns of the target user.

[0062] In one possible implementation, a method for feature fusion of first feature information in a first feature information set and second feature information in a second feature information set to obtain a first mixed feature information set includes: D1. Perform preliminary feature fusion on the first feature information in the first feature information set and the second feature information in the second feature information set to obtain a third feature information set; D2. Perform global feature extraction on the third feature information in the third feature information set to obtain the fourth feature information set; D3. Calculate the attention weight information corresponding to the first feature information in the first feature information set based on the fourth feature information in the fourth feature information set, and obtain the first weight information set; D4. Calculate the attention weight information corresponding to the second feature information in the second feature information set based on the fourth feature information in the fourth feature information set, and obtain the second weight information set. D5. Enhance the first feature information in the first feature information set to obtain the fifth feature information set, and enhance the second feature information in the second feature information set to obtain the sixth feature information set. D6. Based on the first weight information in the first weight information set and the second weight information in the second weight information set, the fifth feature information in the fifth feature information set and the sixth feature information in the sixth feature information set are weighted and fused to obtain the seventh feature information set. D7. Perform residual connection processing on the seventh feature information in the seventh feature information set to obtain the first mixed feature information set.

[0063] Specifically, the first feature information in the first feature information set and the second feature information in the second feature information set can be input into a preset MS-AMFF fusion module, which will then perform preliminary fusion of the first feature information in the first feature information set and the second feature information in the second feature information set to obtain a third feature information set.

[0064] Specifically, the method shown in the following formula can be used by the preset MS-AMFF fusion module to initially fuse the first feature information in the first feature information set and the second feature information in the second feature information set: In the formula This represents the third feature information in the third feature information set, and This indicates that the third feature information is a two-dimensional tensor; This represents the first feature information in the first feature information set; This refers to the second feature information in the second feature information set.

[0065] After obtaining the third feature information set, a third linear feature information set can be obtained by performing a linear transformation on the third feature information in the third feature information set; then, a fourth feature information set can be obtained by sequentially performing batch normalization and activation processing on the third linear feature information in the third linear feature information set. In the formula This represents the fourth feature information in the fourth feature information set; This represents the linear weight matrix corresponding to the linear transformation processing of the third feature information; This represents the third feature information in the third feature information set; This represents the linear bias corresponding to the linear transformation processing of the third feature information; This indicates batch normalization processing; Indicates activation processing; express It is The second-order tensor, This indicates the number of channels corresponding to the fourth feature information in the fourth feature information set, and , Indicates the compression ratio, used to ensure , This indicates the number of channels corresponding to the third feature information in the third feature information set.

[0066] After obtaining the fourth feature information set, the original attention score corresponding to the first feature information in the first feature information set can be calculated based on the fourth feature information in the fourth feature information set using a preset EEG signal attention weight matrix, to obtain the first score information set; the first score information in the first score information set is then subjected to logical normalization processing to obtain the first weight information set.

[0067] After obtaining the first weight information set, the original attention score corresponding to the second feature information in the second feature information set can be calculated based on the fourth feature information in the fourth feature information set using a preset eye-tracking signal attention weight matrix, thus obtaining the second score information set; the second score information in the second score information set is then subjected to logical normalization processing to obtain the second weight information set.

[0068] Specifically, the first score information in the first score information set and the second score information in the second score information set can be logically normalized using the method shown in the following formula to obtain the first weight information set and the second weight information set: In the formula This represents the first score information in the first score information set; This represents a preset EEG signal attention weight matrix, which can be determined by user input or by system default. This represents the transpose matrix corresponding to the fourth feature information; This represents the preset eye-tracking signal attention weight matrix, which can be determined by user input or by system default. This represents the first weight information in the first weight information set; This represents the second weight information in the second weight information set; This indicates that the channel dimensions corresponding to the first score information in the first score information set and the second score information in the second score information set are logically normalized. express and It is a third-order tensor.

[0069] After obtaining the first weight information set and the second weight information set, the first feature information in the first feature information set and the second feature information in the second feature information set can be linearly enhanced by using a general Gaussian error linear unit to obtain the fifth feature information set and the sixth feature information set.

[0070] Specifically, the method shown in the following formula can be used to linearly enhance the first feature information in the first feature information set and the second feature information in the second feature information set, thereby obtaining the fifth feature information set and the sixth feature information set: In the formula This represents the fifth feature information in the fifth feature information set; This indicates that a linear enhancement process is performed using a general Gaussian error linear unit. The linear enhancement parameter corresponding to the first feature information can be determined by user input or by system default. This represents the first feature information in the first feature information set; This represents the linear bias corresponding to the general Gaussian error linear unit when linearly enhancing the first feature information; This represents the sixth feature information in the sixth feature information set; The linear enhancement parameter corresponding to the second feature information can be determined by user input or by system default. This represents the second feature information in the second feature information set; This represents the linear bias corresponding to the general Gaussian error linear unit when linearly enhancing the second feature information.

[0071] After obtaining the fifth feature information set and the sixth feature information set, the fifth feature information in the fifth feature information set and the sixth feature information in the sixth feature information set can be weighted and fused according to the first weight information in the first weight information set and the second weight information in the second weight information set to obtain the seventh feature information set.

[0072] Specifically, the seventh feature information set can be obtained by weighting and fusing the fifth feature information in the fifth feature information set with the sixth feature information in the sixth feature information set, based on the first weight information in the first weight information set and the second weight information in the second weight information set, using the following formula: In the formula This represents the seventh feature information in the seventh feature information set; This represents the first weight information in the first weight information set; This indicates element-wise multiplication; This represents the fifth feature information in the fifth feature information set; This represents the second weight information set within the second weight information set; This represents the sixth feature information in the set of sixth feature information.

[0073] Because the first feature information in the first feature information set and the second feature information in the second feature information set may experience gradient vanishing or gradient explosion after multiple activation processes, the seventh feature information in the resulting seventh feature information set may suffer from deep feature vanishing or chaotic feature distribution, resulting in insufficient accuracy of the seventh feature information. Therefore, it is necessary to perform residual connection processing on the seventh feature information in the seventh feature information set to avoid deep feature vanishing or chaotic feature distribution caused by gradient vanishing or gradient explosion.

[0074] After obtaining the seventh feature information set, a third linear feature information set can be obtained by performing a linear transformation on the third feature information in the third feature information set; regularization processing can be performed on the third linear feature information in the third linear feature information set to extract the residual information corresponding to the third feature information in the third feature information set, resulting in a first residual information set; and residual connection processing can be performed on the seventh feature information in the seventh feature information set based on the first residual information in the first residual information set to obtain a first mixed feature information set.

[0075] Specifically, the first mixed feature information set can be obtained by performing residual concatenation processing on the seventh feature information in the seventh feature information set based on the first residual information in the first residual information set, as shown in the following formula: In the formula This represents the first mixed feature information in the first mixed feature information set; The seventh feature information in the seventh feature information set; This indicates regularization processing; This refers to the linear parameters corresponding to the linear transformation of the third feature information in the third feature information set when performing residual extraction on the seventh feature information. These parameters can be determined by user input or by system default. This represents the third feature information in the third feature information set; This represents the linear bias corresponding to the linear transformation of the third feature information in the third feature information set when extracting the residual of the seventh feature information. It can be determined by user input or by system default.

[0076] In this example, by performing residual connection processing on the seventh feature information in the seventh feature information set, the impact of deep feature disappearance or chaotic feature distribution caused by gradient vanishing or gradient explosion on the accuracy of the seventh feature information is eliminated, thereby improving the accuracy of the seventh feature information in the seventh feature information set, and thus improving the accuracy of the target model in recognizing the emotion patterns of the target user.

[0077] For examples consistent with the above embodiments, please refer to... Figure 2 , Figure 2 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application, such as... Figure 2 As shown, it includes a processor, an input device, an output device, and a memory, which are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to call the program instructions. The program includes instructions for performing the following steps. Extract the EEG signals corresponding to the K reference users used in training the model to obtain the first EEG signal set, and obtain the eye movement signal set corresponding to the K reference users used in training the model to obtain the first eye movement signal set; The first EEG signal in the first EEG signal set and the first eye movement signal in the first eye movement signal set are fused to obtain a first mixed feature information set. The initial model is trained using the first mixed feature information from the first mixed feature information set to obtain the target model; The target model is used to perform emotion pattern recognition on the target user to obtain the target emotion pattern recognition result.

[0078] The above mainly describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, the terminal includes the corresponding hardware structure and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments provided herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0079] This application embodiment can divide the terminal into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0080] For those consistent with the above, please refer to Figure 3 , Figure 3 This application provides a schematic diagram of the structure of a multimodal fusion emotion recognition device based on electroencephalogram (EEG) and eye-tracking signals, as an embodiment of the present application. Figure 3 As shown, the device includes: Extraction unit 301 is used to extract the EEG signals corresponding to the K reference users used in training the model to obtain a first EEG signal set, and to obtain the eye movement signal set corresponding to the K reference users used in training the model to obtain a first eye movement signal set. The fusion unit 302 is used to perform feature fusion on the first EEG signal in the first EEG signal set and the first eye movement signal in the first eye movement signal set to obtain a first mixed feature information set; Training unit 303 is used to train the initial model using the first mixed feature information in the first mixed feature information set to obtain the target model; The recognition unit 304 is used to perform emotion pattern recognition on the target user using the target model to obtain the target emotion pattern recognition result.

[0081] In one possible implementation, the fusion unit 302 is specifically used for: Preprocessing is performed on the first EEG signal in the first EEG signal set and the first eye movement signal in the first eye movement signal set to obtain the second EEG signal set and the second eye movement signal set. The first feature information set is obtained by extracting features from the second EEG signal in the second EEG signal set. Feature extraction is performed on the second eye movement signals in the second eye movement signal set to obtain the second feature information set; The first feature information in the first feature information set and the second feature information in the second feature information set are fused to obtain the first mixed feature information set.

[0082] In one possible implementation, the fusion unit 302 is specifically used for: Construct the adjacency matrix corresponding to the second EEG signal in the second EEG signal set to obtain the first adjacency matrix set; Based on the first adjacency matrix in the first adjacency matrix set, the second EEG signal in the second EEG signal set is subjected to GATv2 graph convolution processing to obtain the first graph feature information set; Based on the first adjacency matrix in the first adjacency matrix set, the second EEG signal in the second EEG signal set is subjected to GENConv graph convolution processing to obtain the second graph feature information set; The first image feature information in the first image feature information set and the second image feature information in the second image feature information set are fused to obtain the first feature information set to be processed. Temporal enhancement processing is performed on the first feature information in the first set of feature information to be processed to obtain the first feature information set.

[0083] In one possible implementation, the fusion unit 302 is specifically used for: The second eye movement signal in the second eye movement signal set is convolved to extract the spatiotemporal feature information corresponding to the second eye movement signal, thus obtaining the first spatiotemporal feature information set. The first spatiotemporal feature information in the first spatiotemporal feature information set is subjected to dimensionality compression processing to obtain the second spatiotemporal feature information set. The second spatiotemporal feature information in the second spatiotemporal feature information set is reorganized in dimensions to obtain the second feature information set to be processed.

[0084] In one possible implementation, the fusion unit 302 is specifically used for: The first feature information in the first feature information set and the second feature information in the second feature information set are initially fused to obtain the third feature information set. Calculate the global features corresponding to K reference users based on the third feature information in the third feature information set to obtain the fourth feature information set; Based on the fourth feature information in the fourth feature information set, calculate the attention weight information corresponding to the first feature information in the first feature information set to obtain the first weight information set; Based on the fourth feature information in the fourth feature information set, calculate the attention weight information corresponding to the second feature information in the second feature information set to obtain the second weight information set; The first feature information in the first feature information set is enhanced to obtain the fifth feature information set, and the second feature information in the second feature information set is enhanced to obtain the sixth feature information set; The fifth feature information in the fifth feature information set and the sixth feature information in the sixth feature information set are weighted and fused based on the first weight information in the first weight information set and the second weight information in the second weight information set to obtain the seventh feature information set. The seventh feature information in the seventh feature information set is subjected to residual connection processing to obtain the first mixed feature information set.

[0085] This application also provides a computer storage medium storing a computer program for electronic data exchange, which causes a computer to perform some or all of the steps of any of the multimodal fusion emotion recognition methods based on EEG and eye movement signals described in the above method embodiments.

[0086] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program that causes a computer to perform some or all of the steps of any of the multimodal fusion emotion recognition methods based on EEG and eye movement signals described in the above method embodiments.

[0087] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0088] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0090] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] Furthermore, the functional units in the various embodiments of the application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.

[0092] If the integrated unit is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0093] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory, a random access memory, a magnetic disk, or an optical disk, etc.

[0094] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multi-modal fusion emotion recognition method based on electroencephalogram and eye movement signals, characterized in that, The method comprises: extracting electroencephalogram signals corresponding to K reference users used when training the model to obtain a first electroencephalogram signal set, and obtaining an electrooculogram signal set corresponding to the K reference users used when training the model to obtain a first electrooculogram signal set; performing feature fusion on first electroencephalogram signals in the first electroencephalogram signal set and first electrooculogram signals in the first electrooculogram signal set to obtain a first mixed feature information set; training an initial model using first mixed feature information in the first mixed feature information set to obtain a target model; performing emotion pattern recognition on a target user using the target model to obtain a target emotion pattern recognition result.

2. The multi-modal fusion emotion recognition method based on electroencephalogram and eye movement signals according to claim 1, characterized in that, The method comprises: performing preprocessing on first electroencephalogram signals in the first electroencephalogram signal set and first electrooculogram signals in the first electrooculogram signal set to obtain a second electroencephalogram signal set and a second electrooculogram signal set; performing feature extraction on second electroencephalogram signals in the second electroencephalogram signal set to obtain a first feature information set; performing feature extraction on second electrooculogram signals in the second electrooculogram signal set to obtain a second feature information set; performing feature fusion on first feature information in the first feature information set and second feature information in the second feature information set to obtain the first mixed feature information set. 3.The multi-modal fusion emotion recognition method based on electroencephalogram and eye movement signals according to claim 2, characterized in that, The method comprises: constructing an adjacency matrix corresponding to second electroencephalogram signals in the second electroencephalogram signal set to obtain a first adjacency matrix set; performing GATv2 graph convolution processing on second electroencephalogram signals in the second electroencephalogram signal set according to first adjacency matrices in the first adjacency matrix set to obtain a first graph feature information set; performing GENConv graph convolution processing on second electroencephalogram signals in the second electroencephalogram signal set according to first adjacency matrices in the first adjacency matrix set to obtain a second graph feature information set; performing feature fusion on first graph feature information in the first graph feature information set and second graph feature information in the second graph feature information set to obtain a first to-be-processed feature information set; performing time series enhancement processing on first to-be-processed feature information in the first to-be-processed feature information set to obtain the first feature information set.

4. The multi-modal fusion emotion recognition method based on electroencephalogram and eye movement signals according to claim 3, characterized in that, The method comprises: performing convolution processing on second electrooculogram signals in the second electrooculogram signal set to extract spatiotemporal feature information corresponding to the second electrooculogram signals to obtain a first spatiotemporal feature information set; performing dimension compression processing on first spatiotemporal feature information in the first spatiotemporal feature information set to obtain a second spatiotemporal feature information set; performing dimension reorganization processing on second spatiotemporal feature information in the second spatiotemporal feature information set to obtain a second to-be-processed feature information set.

5. The multi-modal fusion emotion recognition method based on electroencephalogram and eye movement signals according to claim 4, characterized in that, The first feature information in the first feature information set is preliminarily fused with the second feature information in the second feature information set to obtain a third feature information set. The first feature information in the first feature information set is preliminarily fused with the second feature information in the second feature information set to obtain a third feature information set. Global feature extraction is performed on the third feature information in the third feature information set to obtain a fourth feature information set. The first feature information in the first feature information set is preliminarily fused with the second feature information in the second feature information set to obtain a third feature information set. The first feature information in the first feature information set is preliminarily fused with the second feature information in the second feature information set to obtain a third feature information set. The first feature information in the first feature information set is preliminarily fused with the second feature information in the second feature information set to obtain a third feature information set. The first feature information in the first feature information set is preliminarily fused with the second feature information in the second feature information set to obtain a third feature information set. Residual connection processing is performed on the seventh feature information in the seventh feature information set to obtain the first mixed feature information set.

6. A multi-modal fusion emotion recognition device based on electroencephalogram and eye movement signals, characterized in that, The device comprises: The extraction unit is configured to extract K reference users' electroencephalogram signals used in training a model to obtain a first electroencephalogram signal set, and obtain K reference users' eye movement signals used in training the model to obtain a first eye movement signal set; The fusion unit is configured to perform feature fusion on the first electroencephalogram signal in the first electroencephalogram signal set and the first eye movement signal in the first eye movement signal set to obtain a first mixed feature information set; The training unit is configured to use the first mixed feature information in the first mixed feature information set to train an initial model to obtain a target model; The recognition unit is configured to use the target model to perform emotion pattern recognition on a target user to obtain a target emotion pattern recognition result.

7. The electroencephalogram and eye movement signal based multi-modal fusion emotion recognition device according to claim 6, characterized in that, In terms of performing feature fusion on the first electroencephalogram signal in the first electroencephalogram signal set and the first eye movement signal in the first eye movement signal set to obtain a first mixed feature information set, the fusion unit is specifically configured to: Perform preprocessing on the first electroencephalogram signal in the first electroencephalogram signal set and the first eye movement signal in the first eye movement signal set to obtain a second electroencephalogram signal set and a second eye movement signal set; Perform feature extraction on the second electroencephalogram signal in the second electroencephalogram signal set to obtain a first feature information set; Perform feature extraction on the second eye movement signal in the second eye movement signal set to obtain a second feature information set; Perform feature fusion on the first feature information in the first feature information set and the second feature information in the second feature information set to obtain a first mixed feature information set. 8.The multi-modal fusion emotion recognition device based on electroencephalogram and eye movement signals of claim 7, wherein, In the aspect of feature extraction on the second electroencephalogram signals in the second electroencephalogram signal set to obtain the first feature information set, the fusion unit is specifically used for: constructing an adjacency matrix corresponding to the second electroencephalogram signals in the second electroencephalogram signal set to obtain a first adjacency matrix set; performing GATv2 graph convolution processing on the second electroencephalogram signals in the second electroencephalogram signal set according to the first adjacency matrix in the first adjacency matrix set to obtain a first graph feature information set; performing GENConv graph convolution processing on the second electroencephalogram signals in the second electroencephalogram signal set according to the first adjacency matrix in the first adjacency matrix set to obtain a second graph feature information set; performing feature fusion on the first graph feature information in the first graph feature information set and the second graph feature information in the second graph feature information set to obtain a first to-be-processed feature information set; performing time series enhancement processing on the first to-be-processed feature information in the first to-be-processed feature information set to obtain the first feature information set.

9. A terminal, characterized by comprising: The computer readable storage medium stores a computer program, and the computer program includes program instructions. The program instructions, when executed by a processor, cause the processor to perform the emotion recognition method based on electroencephalogram and eye movement signals according to any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program includes program instructions. The program instructions, when executed by a processor, cause the processor to perform the emotion recognition method based on electroencephalogram and eye movement signals according to any one of claims 1-5.

Images