Emotion recognition method based on two-stream multi-scale cross-graph convolutional neural network

Abstract

This paper proposes an emotion recognition method based on a two-stream multi-scale cross-graph convolutional neural network. The implementation steps are: obtaining a training sample set and a test sample set; constructing an emotion recognition model based on the two-stream multi-scale cross-graph convolutional neural network and iteratively training it; and obtaining the emotion recognition results. This method uses a graph convolutional neural network and a cross-attention network to extract time-space-frequency fusion information and frequency-space-time fusion information from the subject's EEG and EMG data, obtaining a more comprehensive feature representation. Furthermore, a multi-scale convolutional neural network and a channel attention network are used to capture physiological parameter information at different scales, filter out the most valuable features, and conduct in-depth analysis of emotion information, effectively improving recognition accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology and relates to an emotion recognition method, specifically to an emotion recognition method based on a dual-stream multi-scale cross-graph convolutional neural network, which can be used to help the human body regulate abnormal emotions. Background Art

[0002] Emotion recognition involves analyzing the content and features of text, speech, images, or physiological parameters to identify and understand the emotional states expressed by humans. Traditional emotion recognition methods use text, speech, and other information, along with pre-built sentiment lexicons to match words within the text and determine the overall emotion based on the sentiment polarity of the words. This requires a large number of sentiment lexicons. Furthermore, these methods are unable to handle the polysemy and contextual relevance of sentiment terms, making it difficult to identify emotional states. Accurately identifying and understanding human emotions is crucial in many fields today. Multimodal emotion recognition technology based on physiological parameters has emerged as a response to this need. However, this technology faces the following challenges in practical application. First, the introduction of multimodal information also creates information redundancy. This redundant information negatively impacts model training performance by making it difficult for the model to identify valid and complementary features, thereby reducing the accuracy of emotion recognition. Secondly, a single physiological signal is often unable to fully capture subtle changes in emotions. The fusion of multimodal data can provide richer and more complementary information. However, the fusion and effective use of multimodal physiological parameters require complex technical means. How to fully explore the correlation and complementarity between different parameters is the key. Therefore, there is still room for improvement in recognition accuracy.

[0003] The application publication number is CN 118021303 A, and the name is "A multimodal sentiment analysis method and system based on pseudo-twin network". The application discloses a multimodal sentiment recognition method based on pseudo-twin network. This invention is a pyramid network with feature extraction and deep separable convolution, with three structures: bottom-up path, top-down path and lateral connection. It takes the EEG waveform data in multimodal samples as input, extracts the time domain and spatial domain data of the EEG waveform data through deep separable convolution, and extracts and fuses the time domain-spatial domain data of the EEG waveform data through the pyramid network, so as to obtain the comprehensive time domain-spatial domain intrinsic modal features. This method has limitations and shortcomings in feature extraction. First, it extracts the time domain-spatial domain data information of physiological parameter information, but only focuses on the time domain-spatial domain data information, and does not pay attention to the time domain-spatial domain-frequency domain fusion data information and the frequency domain-spatial domain-time domain fusion data information, and cannot better extract emotional parameter information from physiological parameter information; secondly, deep separable convolution is used to extract features, but deep separable convolution mainly focuses on local features, and insufficiently captures the time domain-spatial domain features of different scales, which affects the further improvement of the recognition accuracy of this method. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose an emotion recognition method based on a dual-stream multi-scale cross-graph convolutional neural network, aiming to solve the technical problem of improving the accuracy of emotion recognition in the prior art.

[0005] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0006] (1) Obtain training sample set and test sample set:

[0007] Preprocess the N1 EEG waveform data and N2 EMG waveform data of each subject in the M subjects to obtain multiple frequency domain data K and time domain data J of each EEG waveform data and each EMG waveform data of each subject, as well as multiple time domain-space domain data X h and frequency domain-spatial domain data X f , and then annotate the emotional state represented by each segment of frequency domain data and time domain data, as well as the time-space domain data and frequency-space domain data, and combine the frequency domain data and time domain data corresponding to more than half of the EEG waveform data and EMG waveform data of each subject after preprocessing, as well as the time-space domain data and frequency-space domain data and their corresponding labels into a training sample set, and combine the remaining preprocessed time-space domain data X h and frequency domain-spatial domain data X f and its corresponding label combination test sample set, where M ≥ 32, N1 ≥ 640, and N2 ≥ 640;

[0008] (2) Construct an emotion recognition model H based on a two-stream multi-scale cross-graph convolutional neural network:

[0009] Construct an emotion recognition model H including a cascaded two-stream heterogeneous time-frequency domain graph convolution cross-attention fusion network, a multi-scale convolution channel attention network and an emotion recognition module; the two-stream heterogeneous time-frequency domain graph convolution cross-attention fusion network includes a first branch consisting of a cascaded first graph convolutional neural network and a first cross-attention network, and a second branch consisting of a cascaded second graph convolutional neural network and a second cross-attention network arranged in parallel therewith; the multi-scale convolution channel attention network includes a third branch consisting of a cascaded first multi-scale convolutional neural network and a first channel attention network, and a cascaded second multi-scale convolutional neural network and a second channel attention network arranged in parallel therewith, and the first branch is cascaded with the third branch, and the second branch is cascaded with the fourth branch;

[0010] (3) Iterative training of the emotion recognition model:

[0011] The emotion recognition model H is iteratively trained through the training sample set to obtain the trained emotion recognition model H * ;

[0012] (4) Obtain emotion recognition results:

[0013] Input the test set data into the trained emotion recognition model H * Perform forward propagation to obtain the emotion recognition results of each test sample.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] First, the present invention utilizes the cross-attention mechanism to fuse time-space and frequency domain feature information as well as frequency-space and time domain feature information, which can adaptively focus on key features and obtain time-space-frequency domain fusion feature information and frequency-space-time domain fusion feature information. It avoids the defect of the existing technology that only focuses on time-space domain data information and ignores time-space-frequency domain fusion data information and frequency-space-time domain fusion data information, resulting in the inability to better extract emotional parameter information from physiological parameter information, and effectively improves the recognition accuracy.

[0016] Second, the present invention uses a multi-scale convolutional neural network to capture the rich features of time-space-frequency domain feature information from different scales, grasp the local detail features and the macroscopic overall structure; at the same time, through the channel attention mechanism, it focuses on the importance of different channels of time-space-frequency domain feature information, assigns more weights to the key channels of time-space-frequency domain features, and screens out important features in time-space-frequency domain features, solving the problem that the existing technology cannot better capture the time-space domain information of physiological parameters and deeply mine emotional information, reducing the interference of irrelevant information, improving the ability to extract key features, and effectively improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flowchart for the implementation of the present invention;

[0018] Figure 2 Schematic diagram of the structure of the emotion recognition model of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Reference Figure 1 , the present invention includes the following steps.

[0021] (1) Obtain training sample set and test sample set:

[0022] Preprocess the N1 EEG waveform data and N2 EMG waveform data of each subject in the M subjects to obtain multiple frequency domain data K and time domain data J of each EEG waveform data and each EMG waveform data of each subject, as well as multiple time domain-space domain data X h and frequency domain-spatial domain data X f , and then annotate the emotional state represented by each segment of frequency domain data and time domain data, as well as the time-space domain data and frequency-space domain data, and combine the frequency domain data and time domain data corresponding to more than half of the EEG waveform data and EMG waveform data of each subject after preprocessing, as well as the time-space domain data and frequency-space domain data and their corresponding labels into a training sample set, and combine the remaining preprocessed time-space domain data X h and frequency domain-spatial domain data X f and its corresponding label combination test sample set, where M≥32, N1≥640, and N2≥640; wherein, in this embodiment, M=32, N1=640, and N2=640;

[0023] The dataset used in the experiment is the DEAP dataset, which is a multimodal database that records the multimodal physiological parameters, behaviors and emotional report data of the subjects under different emotional and behavioral stimuli. The experimental content is that the subjects watch 40 video stimuli with emotional labels; then the subjects label according to their own emotional state after the stimulation. The data of 32 subjects were used, and each subject recorded 40 channels of physiological parameter information. The first 32 channel leads are EEG signals, and the last 8 channel leads are electrooculogram, electromyography, skin electricity, respiration, pulse and body temperature data signals, and the EEG signals and electromyography signals are selected. The 32 subjects in the dataset are divided according to the valence dimension and the arousal dimension. Each dimension has 32 subjects, and each subject is divided into 9 training sets and 1 test set;

[0024] Resting data removal was performed on each EEG waveform data and EMG waveform data of each subject, removing the baseline EEG waveform data and EMG waveform data of the subject for the first 3 seconds, reducing potential noise or interference factors and improving data accuracy. Baseline correction was then performed to calculate the average value of each EEG waveform data and EMG waveform data, eliminating baseline drift and improving data quality. Hanning window segmentation was performed on each baseline-corrected EEG waveform data and EMG waveform data, segmenting each EEG waveform data and EMG waveform data according to a window size of 1 second to reduce spectrum leakage and improve the accuracy of spectrum analysis.

[0025] (2) Construct an emotion recognition model H based on a two-stream multi-scale cross-graph convolutional neural network:

[0026] Construct an emotion recognition model H including a cascaded two-stream heterogeneous time-frequency domain graph convolution cross attention fusion network, a multi-scale convolution channel attention network and an emotion recognition module; the structure is as follows Figure 2 As shown, the dual-stream heterogeneous time-frequency domain graph convolution cross-attention fusion network includes a first branch consisting of a cascaded first graph convolutional neural network and a first cross-attention network, and a second branch consisting of a cascaded second graph convolutional neural network and a second cross-attention network arranged in parallel; the first graph convolutional neural network and the second graph convolutional neural network each include a graph convolution layer; the first cross-attention network and the second cross-attention network each include a stacked query convolution layer, a key convolution layer, and a value convolution layer;

[0027] The specific parameters are set as follows: the convolution kernel size of the first graph convolutional neural network and the second graph convolutional neural network is 1×1, and the number of convolution kernels is 1; the first cross-attention network and the second cross-attention network both contain three convolution layers, namely query convolution, key convolution and value convolution, with the convolution kernel size of 1×1 and the step size of 1 by default;

[0028] The multi-scale convolutional channel attention network includes a third branch consisting of a cascaded first multi-scale convolutional neural network and a first channel attention network, and a cascaded second multi-scale convolutional neural network and a second channel attention network arranged in parallel, wherein the first branch is cascaded with the third branch, and the second branch is cascaded with the fourth branch; the first multi-scale convolutional neural network and the second multi-scale convolutional neural network each include four parallel convolutional layers and a batch normalization layer cascaded therewith; the first channel attention network and the second channel attention network each include a stacked pooling layer, a first convolutional layer, a ReLU activation layer, a second convolutional layer and a sigmoid activation layer;

[0029] The specific parameters are set as follows: the first multi-scale convolutional neural network and the second multi-scale convolutional neural network both contain 4 convolutional layers, the first convolutional layer size is 3×3, the second convolution kernel size is 5×5, the third convolutional layer size is 7×7, and the fourth convolutional layer size is 9×9; the first convolutional layer size of the first channel attention network and the second channel attention network is 1×1, and the second convolutional layer size is 1×1;

[0030] The emotion recognition module includes a cascaded concatenation network, a first linear layer network, a ReLU activation layer, and a second linear layer network;

[0031] (3) Iteratively train the emotion recognition model H based on the two-stream multi-scale cross-graph convolutional neural network:

[0032] (3a) Initialize the number of training rounds to t, the maximum number of training rounds is T, T ≥ 100, and the emotion recognition model H obtained after the tth training t The weight parameter is ω t , and let t = 1; wherein, in this embodiment, T = 100;

[0033] (3b) The first graph convolutional neural network and the second graph convolutional neural network are respectively used to calculate X h 、X f Perform graph convolution feature extraction to obtain the time-domain-spatial domain features Y of each segment of EEG waveform data and EMG waveform data of each subject. h , frequency domain-spatial domain feature Y f ; The calculation formulas are:

[0034]

[0035] Among them, A h is the original adjacency matrix in the time domain, A f is the original adjacency matrix in the time domain, D h is the time domain degree matrix, D f is the frequency domain degree matrix, is the normalized time-domain adjacency matrix, is the normalized frequency domain adjacency matrix, W h is the time domain weight parameter, W f is the frequency domain weight parameter, ReLU is the LeakyReLU activation function;

[0036] Through the first graph convolutional neural network and the second graph convolutional neural network, graph structure features are extracted from the time-space domain data and the frequency-space domain data, and the time-space domain features and the frequency-space domain features are extracted to effectively capture the correlation between the time and space dimensions and between the spectrum and space dimensions, realize the feature learning of the data, improve the ability to integrate and comprehensively process the data, and solve the problem of insufficient correlation extraction of multi-dimensional data.

[0037] (3c) The first cross attention network outputs Y of the first graph convolutional neural network h The frequency domain data K of the corresponding EEG waveform data and EMG waveform data of each subject are fused through the attention network to obtain the time domain-space domain-frequency domain feature Z with time domain-space domain as the main and frequency domain as the auxiliary h ; The second cross attention network outputs Y of the second graph convolutional neural network f The time domain data J of the corresponding EEG waveform data and EMG waveform data of each subject are fused through the attention network to obtain the frequency domain-space domain-time domain feature Z with frequency domain-space domain as the main and time domain as the auxiliary f ;

[0038] The calculation formulas are:

[0039] Z h =(Attn1×IV1(Y h ))+K

[0040] Attn1=sigmoid(IQ1(K)-IK1(Y h ))

[0041] Z f =(Attn2×IV2(Y f ))+J

[0042] Attn2=sigmoid(IQ2(J)-IK2(Y f ))

[0043] Where: IQ1 and IQ2 are the query convolutional networks of the first and second cross-attention networks respectively, IK1 and IK2 are the key convolutional networks of the first and second cross-attention networks respectively, IV1 and IV2 are the value convolutional networks of the first and second cross-attention networks respectively, sigmoid is the sigmoid activation function, Attn1 and Attn2 are the weight parameters of the first and second cross-attention networks respectively;

[0044] The first cross-attention network and the second cross-attention network are used to perform attention-weighted splicing on the time-domain-space domain features extracted by the first graph convolutional neural network and the frequency domain data of the corresponding EEG waveform data and EMG waveform data of each subject, as well as the frequency-domain-space domain features extracted by the second graph convolutional neural network and the time domain data of the corresponding EEG waveform data and EMG waveform data of each subject through the cross-attention network weight parameters in the cross-attention network, thereby avoiding focusing only on the time-domain-space domain data information and effectively extracting the time-domain-space-frequency domain fusion data information and the frequency-domain-space-time domain fusion data information, better extracting the emotional parameter information from the physiological parameter information, and effectively improving the recognition accuracy;

[0045] (3d) Multi-scale convolutional channel attention network for Z h and Z f Perform multi-scale convolution feature extraction to obtain weighted multi-scale fusion time domain, space domain and frequency domain features V h and weighted multi-scale fusion frequency domain, space domain, and time domain features V f ; The current steps are:

[0046] The first multi-scale convolutional neural network in the multi-scale convolutional channel attention network is Z h Perform multi-scale convolution feature extraction to obtain multi-scale fusion time domain, space domain and frequency domain features U h , and cascaded with the first channel attention network, the first channel attention network for U h Focus on extracting important features and obtain weighted multi-scale fusion time domain, space domain and frequency domain features V h ; The second multi-scale convolutional neural network for Z f Perform multi-scale convolution feature extraction to obtain multi-scale fusion frequency domain, space domain and time domain features U f , and cascaded with the second channel attention network, the second channel attention network for U f Focus on extracting important features and weighted multi-scale fusion of frequency domain, space domain and time domain features V f ,in:

[0047]

[0048] V h =(IT11(ReLU(IT12(p(U h )))))×U h

[0049] V f =(IT21(ReLU(IT22(p(U f )))))×U f

[0050] Among them, ω r1 is the convolution kernel of the rth convolution of the first multi-scale convolutional neural network, ω s2 is the convolution kernel of the sth convolution of the second multi-scale convolutional neural network; p is the pooling operation, IT11 and IT12 are the first and second convolutional networks of the first channel attention network, IT21 and IT22 are the first and second convolutional networks of the second channel attention network, and ReLU is the LeakyReLU activation function;

[0051] Through the multi-scale convolution channel attention network, the time domain-spatial domain-frequency domain features Z extracted by the first cross attention network and the second cross attention network are h and frequency domain-space domain-time domain characteristics Z f First, a multi-scale convolutional neural network is used to capture the rich features of time-space-frequency domain features and frequency-space-time domain features at different scales. By comprehensively considering information at different scales, a richer and more comprehensive feature representation is obtained. This effectively comprehensively considers information from different domains, improves the analysis and processing of data, and solves the problem of insufficient capture of time-space domain features at different scales. Further, different-scale feature extraction is performed on time-space-frequency domain features and frequency-space-time domain features, more comprehensively considering the multi-scale features of time-space-frequency domain feature information and frequency-space-time domain feature information, better capturing the associations and laws between different dimensions, improving the comprehensive analysis capability of the model, and effectively improving recognition accuracy.

[0052] Then, the multi-scale fusion time-space-frequency features U output by the first cross attention network and the second cross attention network are integrated through the first channel attention network and the second channel attention network. h and multi-scale fusion time-space-frequency features U f By using attention weights for weighted splicing, we can solve the problems of insufficient fusion of features at different scales, insufficient capture of important features, and insufficient deep mining of emotional information. This allows us to more accurately capture key information of the data, optimize the fusion and integration process of features, effectively improve the expressiveness of features, and effectively improve recognition accuracy.

[0053] (3e) The emotion recognition module uses the splicing network to h and V f The spliced ​​data are then transformed through the first linear layer network, the ReLU activation layer, and the second linear layer network to obtain the predicted label y corresponding to each segment of EEG waveform data and EMG waveform data of each subject. num The calculation formula is:

[0054] y num =LIN2(ReLU(LIN1(concat(V h ,V f ))))

[0055] Among them, concat is the feature concatenation operation, ReLU is the LeakyReLU activation function, LIN1 and LIN2 are the first linear network and the second linear network respectively.

[0056] The emotion recognition module is used to fusion the weighted multi-scale time-space-frequency features of the multi-scale convolutional channel attention network output. h and weighted multi-scale fusion frequency domain, space domain, and time domain features V f The first linear network performs linear mapping, mapping the input features to a higher-dimensional feature space, thereby providing more complex feature representation capabilities. The LeakyReLU activation function then performs nonlinear mapping, helping the network learn complex features and patterns while solving the vanishing gradient problem. The second linear network then performs linear mapping, mapping the nonlinearly processed features to the final output space for the final recognition task. This effectively solves the problems of insufficient feature conversion and mapping, better adapts to the needs of emotion recognition tasks, and captures complex relationships in the data, effectively improving recognition accuracy.

[0057] (3f) Using the cross entropy loss function, through y num The true label corresponding to each segment of EEG waveform data and EMG waveform data of each subject Calculate H * The loss value L', and the gradient descent method is used to pass the L' weight parameter ω t Update and get the emotion recognition model H of this iteration t ;

[0058] Calculate the loss value L' and the weight parameter ω t The update calculation and update formula are:

[0059]

[0060] Among them, NUM represents the total number of data segments for each subject. Indicates the true label corresponding to the num-th segment of data for each subject, y num Indicates the predicted label of the model H' for the numth sample, and ln represents the logarithmic operation with base e

[0061] (3g) Determine whether t≥T is true. If so, obtain the trained target recognition model H. * , otherwise, let t = t + 1 and execute step (3b);

[0062] (4) Obtain emotion recognition results:

[0063] Input the test set data into the trained emotion recognition model H * Perform forward propagation to obtain the emotion recognition results of each test sample.

[0064] The following is a further explanation of the technical effects of the present invention in conjunction with the simulation results:

[0065] 1. Experimental conditions and content:

[0066] The simulation experiment hardware platform is: processor: 13th Gen Intel(R) Core(TM) i5-13500H, processor frequency: 2.60GHz, memory: 128GB, graphics card: NVIDIAGeForce RTX 4060; software platform: Windows 11 operating system, Python 3.10 and torch 1.5.

[0067] A comparative simulation is conducted on the recognition accuracy of the present invention and the existing emotion recognition method based on pseudo-twin network.

[0068] 2. Analysis of experimental results:

[0069] The present invention is applied to perform fusion recognition on the EEG waveform data and EMG waveform data of 32 subjects. First, the training sample set is used to train the dual-stream multi-scale cross-graph convolutional neural network to obtain a trained emotion recognition model based on the dual-stream multi-scale cross-graph convolutional neural network. Then, the test sample set is used to test the trained dual-stream multi-scale cross-graph convolutional neural network.

[0070] Among the commonly used performance indicators of sentiment models, the main ones are: accuracy, precision, recall and F value. This article uses accuracy and standard deviation as evaluation indicators:

[0071]

[0072] Among them, Accuracy is the accuracy rate, TP is predicted as a positive example, and it is actually a positive example; FP is predicted as a positive example, but it is actually a negative example; TN is predicted as a negative example, but it is actually a negative example; FN is predicted as a negative example, but it is actually a positive example;

[0073] In statistics, standard deviation is a measure of the dispersion of data distribution, indicating how much the values ​​in a data set deviate from the average value (mean). A smaller standard deviation means that the data points are more closely distributed around the mean, while a larger standard deviation means that the data points are more dispersed.

[0074]

[0075] Where std is the standard deviation; N is the total number of data points in the data set; x i is the i-th data point in the data set; μ is the mean of the data set, and the calculation formula is:

[0076]

[0077] According to the above formula, after model training and testing, the recognition accuracy reached 98.40% in the valence dimension and 98.33% in the arousal dimension; the recognition accuracy of the emotion recognition method based on the pseudo-twin network was 96.02% in the valence dimension and 96.45% in the arousal dimension; it can be seen that the emotion recognition method based on the dual-stream multi-scale cross-graph convolutional neural network proposed in the present invention can effectively improve the recognition accuracy of human emotions compared with the existing methods.

Claims

1. A sentiment recognition method based on a two-stream multi-scale cross-graph convolutional neural network, characterized in that: The steps include: (1) Obtain training sample set and test sample set: Preprocess the N1 EEG waveform data and N2 EMG waveform data of each subject in the M subjects to obtain multiple frequency domain data K and time domain data J of each EEG waveform data and each EMG waveform data of each subject, as well as multiple time domain-space domain data X h and frequency domain-spatial domain data X f , and then annotate the emotional state represented by each segment of frequency domain data and time domain data, as well as the time-space domain data and frequency-space domain data, and combine the frequency domain data and time domain data corresponding to more than half of the EEG waveform data and EMG waveform data of each subject after preprocessing, as well as the time-space domain data and frequency-space domain data and their corresponding labels into a training sample set, and combine the remaining preprocessed time-space domain data X h and frequency domain-spatial domain data X f and its corresponding label combination test sample set, where M ≥ 32, N1 ≥ 640, and N2 ≥ 640; (2) Construct an emotion recognition model H based on a two-stream multi-scale cross-graph convolutional neural network: Construct an emotion recognition model H including a cascaded two-stream heterogeneous time-frequency domain graph convolution cross-attention fusion network, a multi-scale convolution channel attention network and an emotion recognition module; the two-stream heterogeneous time-frequency domain graph convolution cross-attention fusion network includes a first branch consisting of a cascaded first graph convolutional neural network and a first cross-attention network, and a second branch consisting of a cascaded second graph convolutional neural network and a second cross-attention network arranged in parallel therewith; the multi-scale convolution channel attention network includes a third branch consisting of a cascaded first multi-scale convolutional neural network and a first channel attention network, and a cascaded second multi-scale convolutional neural network and a second channel attention network arranged in parallel therewith, and the first branch is cascaded with the third branch, and the second branch is cascaded with the fourth branch; (3) Iterative training of the emotion recognition model: The emotion recognition model H is iteratively trained through the training sample set to obtain the trained emotion recognition model H * ; (4) Obtain emotion recognition results: Input the test set data into the trained emotion recognition model H * Perform forward propagation to obtain the emotion recognition results of each test sample.

2. The method according to claim 1, characterized in that The steps of preprocessing N1 pieces of EEG waveform data and N2 pieces of EMG waveform data of each of the M subjects in step (1) are as follows: (1a) performing resting data removal processing on each EEG waveform data and EMG waveform data of each subject and then performing baseline correction, and performing Hanning window segmentation on each EEG waveform data and EMG waveform data after baseline correction to obtain multiple time domain data J corresponding to each EEG waveform data and EMG waveform data of each subject; (1b) performing Fourier transform on the time domain data corresponding to each segment of EEG waveform data and EMG waveform data of each subject to obtain multiple segments of frequency domain data K corresponding to each segment of EEG waveform data and EMG waveform data of each subject; (1c) Construct an adjacency matrix for each time domain data and frequency domain data corresponding to each EEG waveform data and EMG waveform data of each subject, and obtain the time domain-spatial domain data X containing each EEG waveform data and EMG waveform data of each subject. h and frequency domain-spatial domain data X f .

3. The method according to claim 1, characterized in that The emotion recognition model H described in step (2), wherein: The first graph convolutional neural network and the second graph convolutional neural network each include a graph convolution layer; The first cross attention network and the second cross attention network both contain stacked query convolution layers, key convolution layers, and value convolution layers; The first multi-scale convolutional neural network and the second multi-scale convolutional neural network both include four convolutional layers in parallel and a batch normalization layer cascaded therewith; Both the first-channel attention network and the second-channel attention network include stacked pooling layers, first convolutional layers, ReLU activation layers, second convolutional layers, and sigmoid activation layers; The emotion recognition module consists of a cascaded concatenation network, a first linear layer network, a ReLU activation layer, and a second linear layer network.

4. The method according to claim 1, wherein The emotion recognition model H is iteratively trained as described in step (3), and the implementation steps are as follows: (3a) Initialize the number of training rounds to t, the maximum number of training rounds is T, T ≥ 100, and the emotion recognition model H obtained after the tth training t The weight parameter is ω t , and let t = 1; (3b) The first graph convolutional neural network and the second graph convolutional neural network are respectively used to calculate X h 、X f Perform graph convolution feature extraction to obtain the time-domain-spatial domain features Y of multiple EEG waveform data and EMG waveform data of each subject h , frequency domain-spatial domain feature Y f ; (3c) The first cross attention network outputs Y of the first graph convolutional neural network h The multi-segment frequency domain data K of each EEG waveform data and each EMG waveform data of each subject are spliced ​​to obtain the time domain-space domain-frequency domain feature Z with time domain-space domain as the main and frequency domain as the auxiliary h ; The second cross attention network outputs Y of the second graph convolutional neural network f The multiple time domain data J of each EEG waveform data and each EMG waveform data of each subject are spliced ​​to obtain the frequency domain-space domain-time domain feature Z with frequency domain-space domain as the main and time domain as the auxiliary f ; (3d) Multi-scale convolutional channel attention network for Z h and Z f Perform multi-scale convolution feature extraction to obtain weighted multi-scale fusion time domain, space domain and frequency domain features V h and weighted multi-scale fusion frequency domain, space domain, and time domain features V f ; (3e) The emotion recognition module uses the splicing network to h and V f The spliced ​​data are transformed in format through the first linear layer network, ReLU activation layer and second linear layer network in turn to obtain the predicted label y of each segment of EEG waveform data and EMG waveform data of each subject. num ; (3f) Using the cross entropy loss function, through y num The true label corresponding to each segment of EEG waveform data and EMG waveform data of each subject Calculate H * The loss value L', and the gradient descent method is used to pass the L' weight parameter ω t Update and get the emotion recognition model H of this iteration t ; (3g) Determine whether t≥T is true. If so, obtain the trained emotion recognition model H * Otherwise, let t=t+1, H t =H, and execute step (3b).

5. The method according to claim 4, characterized in that The time-space domain feature Y described in step (3b) h , frequency domain-spatial domain feature Y f , the calculation formulas are: Among them, A h is the original adjacency matrix in the time domain, A f is the original adjacency matrix in the time domain, D h is the time domain degree matrix, D f is the frequency domain degree matrix, is the normalized time-domain adjacency matrix, is the normalized frequency domain adjacency matrix, W h is the time domain weight parameter, W f is the frequency domain weight parameter, and ReLU is the LeakyReLU activation function.

6. The method according to claim 4, characterized in that The time domain-space domain-frequency domain feature Z in step (3c) h , frequency domain-space domain-time domain feature Z f , the calculation formulas are: Z h =(Attn1×IV1(Y h ))+K Attn1=sigmoid(IQ1(K)-IK1(Y h )) Z f =(Attn2×IV2(Y f ))+J Attn2=sigmoid(IQ2(J)-IK2(Y f )) Among them: IQ1 and IQ2 are the query convolutional networks of the first cross attention network and the second cross attention network respectively, IK1 and IK2 are the key convolutional networks of the first cross attention network and the second cross attention network respectively, IV1 and IV2 are the value convolutional networks of the first cross attention network and the second cross attention network respectively, sigmoid is the sigmoid activation function, Attn1 and Attn2 are the weight parameters of the first cross attention network and the second cross attention network respectively.

7. The method according to claim 4, characterized in that The multi-scale convolutional channel attention network described in step (3d) is used for Z h and Z f Perform multi-scale convolution feature extraction, and the implementation steps are as follows: The first multi-scale convolutional neural network in the multi-scale convolutional channel attention network is Z h Perform multi-scale convolution feature extraction to obtain multi-scale fusion time domain, space domain and frequency domain features U h , the first channel attention network is for U h Perform feature weighted extraction to obtain weighted multi-scale fusion time domain, space domain, and frequency domain features V h ; The second multi-scale convolutional neural network for Z f Perform multi-scale convolution feature extraction to obtain multi-scale fusion frequency domain, space domain and time domain features U f , the second channel attention network is for U f Perform feature weighted extraction to obtain weighted multi-scale fusion frequency domain-space domain-time domain feature V f ,in: In h =(IT11(ReLU(IT12(p(U h )))))×U h V f =(IT21(ReLU(IT22(p(U f )))))×U f Among them, ω r1 is the convolution kernel of the rth convolution of the first multi-scale convolutional neural network, ω s2 is the convolution kernel of the sth convolution of the second multi-scale convolutional neural network; p is the pooling operation, IT11 and IT12 are the first convolutional network and the second convolutional network of the first channel attention network, IT21 and IT22 are the first convolutional network and the second convolutional network of the second channel attention network, and ReLU is the LeakyReLU activation function.

8. The method according to claim 4, characterized in that The predicted label y described in step (3e) num , the calculation formula is: y num =LIN2(ReLU(LIN1(concat(V h ,V f )))) Among them, concat is the feature concatenation operation, ReLU is the LeakyReLU activation function, LIN1 and LIN2 are the first linear network and the second linear network respectively.

9. The method according to claim 4, characterized in that The loss value L' described in step (3f), and the weight parameter ω t To update, the calculation and update formulas are: Among them, NUM represents the total number of EEG waveform data and EMG waveform data of each subject. Indicates the true label corresponding to the num-th segment of data for each subject, y num Indicates the predicted label of the model H' for the numth sample, ln represents the logarithmic operation with e as the base, Represents ω t The update result, η represents the learning rate of L', Represents a derivative operation.

Images