Emotion recognition method based on double-flow multi-scale cross graph convolutional neural network

Abstract

The invention provides an emotion recognition method based on a double-flow multi-scale cross graph convolutional neural network. The method comprises the following implementation steps: acquiring a training sample set and a test sample set; constructing an emotion recognition model based on a double-flow multi-scale cross graph convolutional neural network, and performing iterative training on the emotion recognition model; and obtaining an emotion recognition result. According to the method, time domain-space domain-frequency domain fusion information and frequency domain-space domain-time domain fusion information are extracted from electroencephalogram data and myoelectricity data of a subject through a graph convolutional neural network and a cross attention network, more comprehensive feature representation is obtained, and through a multi-scale convolutional neural network and a channel attention network, the time domain-space domain-frequency domain fusion information and the frequency domain-space domain-time domain fusion information are extracted; according to the method, physiological parameter information is captured from different scales, most valuable features are screened from the physiological parameter information, deep mining is carried out on emotion information, and the recognition accuracy is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology and relates to an emotion recognition method, specifically to an emotion recognition method based on a dual-stream multi-scale cross-graph convolutional neural network, which can be used to help the human body regulate abnormal emotions. Background Art

[0002] The emotion recognition task is to identify and understand the emotional state expressed by humans by analyzing the content and features in text, speech, images or human physiological parameter data. In the recognition of human emotional state, traditional emotion recognition methods use information such as text and speech, use pre-built emotional dictionaries to match words in the text, and determine the overall emotion based on the emotional polarity of the words, so a large number of emotional dictionaries are required. At the same time, the inability to handle the polysemy and contextual relevance of emotional words brings difficulties to the recognition of human emotional state. In today's society, accurate recognition and understanding of human emotions are of great significance in many fields. Multimodal emotion recognition technology based on physiological parameters has emerged. However, this technology faces the following challenges in practical applications. First, the introduction of multiple modal information will also cause information redundancy. Redundant information has an adverse effect on the performance of model training because it makes it difficult for the model to identify effective and complementary features, thereby reducing the accuracy of emotion recognition. Secondly, a single physiological signal is often unable to fully capture the subtle changes in emotions. The fusion of multimodal data can provide richer and complementary information. The fusion and effective use of multimodal physiological parameters require complex technical means. How to fully explore the correlation and complementarity between different parameters is the key. Therefore, there is still room for improvement in recognition accuracy.

[0003] The patent application publication number is CN 118021303 A, and the name is "A multimodal sentiment analysis method and system based on pseudo-twin network". The application discloses a multimodal sentiment recognition method based on pseudo-twin network. This invention is a pyramid network with feature extraction and deep separable convolution, which has three structures: bottom-up path, top-down path and lateral connection. The method takes the EEG waveform data in the multimodal samples as input, extracts the time domain and space domain data of the EEG waveform data through deep separable convolution, and extracts and fuses the time-space domain data of the EEG waveform data through the pyramid network, so as to obtain the comprehensive time-space domain intrinsic modal features. This method has limitations and shortcomings in feature extraction. First, the time-space domain data information of the physiological parameter information is extracted, but only the time-space domain data information is focused on, but not the time-space domain-frequency domain fusion data information and the frequency-space domain-time domain fusion data information, which cannot better extract the emotional parameter information from the physiological parameter information; secondly, deep separable convolution is used to extract features, but deep separable convolution mainly focuses on local features, and the capture of time-space domain features of different scales is insufficient, which affects the further improvement of the recognition accuracy of this method. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose a emotion recognition method based on a dual-stream multi-scale cross-graph convolutional neural network, aiming to solve the technical problem of improving the accuracy of emotion recognition in the prior art.

[0005] To achieve the above object, the technical solution adopted by the present invention comprises the following steps:

[0006] (1) Obtain training sample set and test sample set:

[0007] Preprocess the N1 pieces of EEG waveform data and N2 pieces of EMG waveform data of each subject in the M subjects to obtain multiple frequency domain data K and time domain data J of each piece of EEG waveform data and each piece of EMG waveform data of each subject, as well as multiple time domain-spatial domain data X h and frequency domain-spatial domain data X f Then, the emotional state represented by each segment of frequency domain data and time domain data, as well as the time-space domain data and frequency-space domain data are annotated, and the frequency domain data and time domain data corresponding to more than half of the EEG waveform data and EMG waveform data of each subject after preprocessing, as well as the time-space domain data and frequency-space domain data and their corresponding labels are combined into a training sample set, and the remaining preprocessed time-space domain data X h and frequency domain-spatial domain data X f and its corresponding label combination test sample set, where M≥32, N1≥640, N2≥640;

[0008] (2) Construct an emotion recognition model H based on a two-stream multi-scale cross-graph convolutional neural network:

[0009] Construct an emotion recognition model H including a cascaded dual-stream heterogeneous time-frequency domain graph convolution cross-attention fusion network, a multi-scale convolution channel attention network and an emotion recognition module; the dual-stream heterogeneous time-frequency domain graph convolution cross-attention fusion network includes a first branch consisting of a cascaded first graph convolution neural network and a first cross-attention network, and a second branch consisting of a cascaded second graph convolution neural network and a second cross-attention network arranged in parallel therewith; the multi-scale convolution channel attention network includes a third branch consisting of a cascaded first multi-scale convolution neural network and a first channel attention network, and a cascaded second multi-scale convolution neural network and a second channel attention network arranged in parallel therewith, and the first branch is cascaded with the third branch, and the second branch is cascaded with the fourth branch;

[0010] (3) Iterative training of the emotion recognition model:

[0011] The emotion recognition model H is iteratively trained through the training sample set to obtain the trained emotion recognition model H * ;

[0012] (4) Obtain emotion recognition results:

[0013] Input the test set data into the trained emotion recognition model H * Perform forward propagation to obtain the emotion recognition result of each test sample.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] First, the present invention utilizes the cross-attention mechanism to fuse time-space and frequency domain feature information as well as frequency-space and time domain feature information, and can adaptively focus on key features to obtain time-space-frequency domain fusion feature information and frequency-space-time domain fusion feature information. This avoids the defect of the prior art that it cannot better extract emotional parameter information from physiological parameter information because it only focuses on time-space domain data information and ignores time-space-frequency domain fusion data information and frequency-space-time domain fusion data information, and effectively improves the recognition accuracy.

[0016] Second, the present invention uses a multi-scale convolutional neural network to capture the rich features of time-space-frequency domain feature information from different scales, grasp the local detail features and the macroscopic overall structure; at the same time, through the channel attention mechanism, it focuses on the importance of different channels of time-space-frequency domain feature information, allocates more weights to the key channels of time-space-frequency domain features, and screens out important features in the time-space-frequency domain features, solving the problem that the existing technology cannot better capture the time-space domain information of physiological parameters and deeply mine emotional information, reducing the interference of irrelevant information, improving the ability to extract key features, and effectively improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart for realizing the present invention;

[0018] Figure 2 Schematic diagram of the structure of the emotion recognition model of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0020] Reference Figure 1 , the present invention includes the following steps.

[0021] (1) Obtain training sample set and test sample set:

[0022] Preprocess the N1 pieces of EEG waveform data and N2 pieces of EMG waveform data of each subject in the M subjects to obtain multiple frequency domain data K and time domain data J of each piece of EEG waveform data and each piece of EMG waveform data of each subject, as well as multiple time domain-spatial domain data X h and frequency domain-spatial domain data X f Then, the emotional state represented by each segment of frequency domain data and time domain data, as well as the time-space domain data and frequency-space domain data are annotated, and the frequency domain data and time domain data corresponding to more than half of the EEG waveform data and EMG waveform data of each subject after preprocessing, as well as the time-space domain data and frequency-space domain data and their corresponding labels are combined into a training sample set, and the remaining preprocessed time-space domain data X h and frequency domain-spatial domain data X f and its corresponding label combination test sample set, where M≥32, N1≥640, N2≥640; where, in this embodiment, M=32, N1=640, N2=640;

[0023] The dataset used in the experiment is the DEAP dataset, which is a multimodal database that records the multimodal physiological parameters, behaviors, and emotional report data of the subjects under different emotional and behavioral stimuli. The experimental content is that the subjects watch 40 video stimuli with emotional labels; then the subjects label according to their own emotional state after the stimulation. The data of 32 subjects were used, and each subject recorded 40 channels of physiological parameter information. The first 32 channel leads are EEG signals, and the last 8 channel leads are electrooculogram, electromyography, skin electricity, respiration, pulse and body temperature data signals, and the EEG signals and electromyography signals are selected. The 32 subjects in the dataset are divided according to the valence dimension and the arousal dimension, with 32 subjects in each dimension, and each subject is divided into 9 training sets and 1 test set;

[0024] Resting data removal processing was performed on each EEG waveform data and EMG waveform data of each subject, and the baseline EEG waveform data and EMG waveform data of the subject in the first 3 seconds were removed, reducing potential noise or interference factors and improving data accuracy; then baseline correction processing was performed to calculate the average value of each EEG waveform data and EMG waveform data, eliminate baseline drift, and improve data quality; and Hanning window segmentation cutting processing was performed on each EEG waveform data and EMG waveform data after baseline correction, and each EEG waveform data and EMG waveform data were segmented according to a window size of 1 second to reduce spectrum leakage and improve the accuracy of spectrum analysis;

[0025] (2) Construct an emotion recognition model H based on a two-stream multi-scale cross-graph convolutional neural network:

[0026] Construct an emotion recognition model H including a cascaded two-stream heterogeneous time-frequency domain graph convolution cross-attention fusion network, a multi-scale convolution channel attention network and an emotion recognition module; the structure is as follows Figure 2 As shown, the dual-stream heterogeneous time-frequency domain graph convolution cross-attention fusion network includes a first branch consisting of a cascaded first graph convolutional neural network and a first cross-attention network, and a second branch consisting of a cascaded second graph convolutional neural network and a second cross-attention network arranged in parallel; the first graph convolutional neural network and the second graph convolutional neural network each include a graph convolution layer; the first cross-attention network and the second cross-attention network each include stacked query convolution layers, key convolution layers, and value convolution layers;

[0027] The specific parameters are set as follows: the convolution kernel size of the first graph convolutional neural network and the second graph convolutional neural network is 1×1, and the number of convolution kernels is 1; the first cross-attention network and the second cross-attention network both contain three convolution layers, namely query convolution, key convolution and value convolution, the convolution kernel size is 1×1, and the step size is 1 by default;

[0028] The multi-scale convolutional channel attention network includes a third branch consisting of a cascaded first multi-scale convolutional neural network and a first channel attention network, and a cascaded second multi-scale convolutional neural network and a second channel attention network arranged in parallel, and the first branch is cascaded with the third branch, and the second branch is cascaded with the fourth branch; the first multi-scale convolutional neural network and the second multi-scale convolutional neural network both include four parallel convolutional layers and a batch normalization layer cascaded therewith; the first channel attention network and the second channel attention network both include stacked pooling layers, first convolutional layers, ReLU activation layers, second convolutional layers and sigmoid activation layers;

[0029] The specific parameters are set as follows: The first multi-scale convolutional neural network and the second multi-scale convolutional neural network both contain 4 convolutional layers, the size of the first convolutional layer is 3×3, the size of the second convolutional kernel is 5×5, the size of the third convolutional layer is 7×7, and the size of the fourth convolutional layer is 9×9; the size of the first convolutional layer of the first channel attention network and the second channel attention network is 1×1, and the size of the second convolutional layer is 1×1;

[0030] The emotion recognition module includes a cascaded concatenation network, a first linear layer network, a ReLU activation layer, and a second linear layer network;

[0031] (3) Iteratively train the emotion recognition model H based on the two-stream multi-scale cross-graph convolutional neural network:

[0032] (3a) Initialize the number of training rounds to t, the maximum number of training rounds to T, T ≥ 100, and the emotion recognition model H obtained after the tth training t The weight parameter is ω t , and let t = 1; wherein, in this embodiment, T = 100;

[0033] (3b) The first graph convolutional neural network and the second graph convolutional neural network are respectively used to h , X f Perform graph convolution feature extraction to obtain the time-domain-spatial domain features Y of each segment of EEG waveform data and EMG waveform data of each subject. h , frequency domain-spatial domain feature Y f ; The calculation formulas are:

[0034]

[0035] Among them, A h is the original adjacency matrix in the time domain, A f is the original adjacency matrix in the time domain, D h is the time domain degree matrix, D f is the frequency domain degree matrix, is the normalized time-domain adjacency matrix, is the normalized frequency domain adjacency matrix, W h is the time domain weight parameter, W f is the frequency domain weight parameter, ReLU is the LeakyReLU activation function;

[0036] Through the first graph convolutional neural network and the second graph convolutional neural network, graph structure features are extracted for time-space domain data and frequency-space domain data, and time-space domain features and frequency-space domain features are extracted to effectively capture the correlation between time and space dimensions and between spectrum and space dimensions, realize feature learning of data, improve the ability to integrate and comprehensively process data, and solve the problem of insufficient extraction of correlation of multi-dimensional data.

[0037] (3c) The first cross attention network outputs Y of the first graph convolutional neural network h The frequency domain data K of the corresponding EEG waveform data and EMG waveform data of each subject are fused through the attention network to obtain the time-space-frequency domain feature Z with time-space domain as the main domain and frequency domain as the auxiliary domain. h ; The second cross attention network outputs Y of the second graph convolutional neural network f The time domain data J of the corresponding EEG waveform data and EMG waveform data of each subject are fused through the attention network to obtain the frequency domain-space domain-time domain feature Z with frequency domain-space domain as the main and time domain as the auxiliary f ;

[0038] The calculation formulas are:

[0039] Z h =(Attn1×IV1(Y h ))+K

[0040] Attn1=sigmoid(IQ1(K)-IK1(Y h ))

[0041] Z f =(Attn2×IV2(Y f ))+J

[0042] Attn2=sigmoid(IQ2(J)-IK2(Y f ))

[0043] Among them: IQ1 and IQ2 are the query convolutional networks of the first cross attention network and the second cross attention network respectively, IK1 and IK2 are the key convolutional networks of the first cross attention network and the second cross attention network respectively, IV1 and IV2 are the value convolutional networks of the first cross attention network and the second cross attention network respectively, sigmoid is the sigmoid activation function, Attn1 and Attn2 are the weight parameters of the first cross attention network and the second cross attention network respectively;

[0044] The time-domain-space domain features extracted by the first graph convolutional neural network and the frequency domain data of the corresponding EEG waveform data and electromyography waveform data of each subject, as well as the frequency-domain-space domain features extracted by the second graph convolutional neural network and the time domain data of the corresponding EEG waveform data and electromyography waveform data of each subject are weightedly spliced ​​by the cross-attention network weight parameters in the cross-attention network through the first cross-attention network and the second cross-attention network, thereby avoiding focusing only on the time-domain-space domain data information, and effectively extracting the time-domain-space domain-frequency domain fusion data information and the frequency-domain-space domain-time domain fusion data information, so as to better extract the emotion parameter information from the physiological parameter information and effectively improve the recognition accuracy;

[0045] (3d) Multi-scale convolutional channel attention network for Z h and Z f Perform multi-scale convolution feature extraction to obtain weighted multi-scale fusion time domain-spatial domain-frequency domain feature V h and weighted multi-scale fusion frequency domain, space domain, and time domain features V f ; The current steps are:

[0046] The first multi-scale convolutional neural network in the multi-scale convolutional channel attention network is Z h Perform multi-scale convolution feature extraction to obtain multi-scale fusion time domain, space domain and frequency domain features U h , and cascaded with the first channel attention network, the first channel attention network for U h Focus on extracting important features and obtain weighted multi-scale fusion time-space-frequency features V h ; The second multi-scale convolutional neural network for Z f Perform multi-scale convolution feature extraction to obtain multi-scale fusion frequency domain-spatial domain-time domain feature U f , and cascaded with the second channel attention network, the second channel attention network for U f Focus on extracting important features and weighted multi-scale fusion of frequency domain, space domain and time domain features V f ,in:

[0047]

[0048] V h =(IT11(ReLU(IT12(p(U h ))))))×U h

[0049] V f =(IT21(ReLU(IT22(p(U f ))))))×U f

[0050] Among them, ω r1 is the convolution kernel of the rth convolution of the first multi-scale convolutional neural network, ω s2 is the convolution kernel of the sth convolution of the second multi-scale convolutional neural network; p is the pooling operation, IT11 and IT12 are the first convolutional network and the second convolutional network of the first channel attention network, IT21 and IT22 are the first convolutional network and the second convolutional network of the second channel attention network, and ReLU is the LeakyReLU activation function;

[0051] Through the multi-scale convolution channel attention network, the time-space-frequency features Z extracted by the first cross attention network and the second cross attention network are h and frequency domain-space domain-time domain characteristics Z f First, a multi-scale convolutional neural network is used to capture the rich features of time-space-frequency domain features and frequency-space-time domain features from different scales, and comprehensively consider the information of different scales to obtain a richer and more comprehensive feature representation, effectively comprehensively consider the information of different domains, improve the analysis and processing effect of data, and solve the problem of insufficient capture of time-space domain features of different scales. Further, different scale feature extraction is performed on time-space-frequency domain features and frequency-space-time domain features, and the multi-scale features of time-space-frequency domain feature information and frequency-space-time domain feature information are more comprehensively considered to better capture the associations and laws between different dimensions, improve the comprehensive analysis ability of the model, and effectively improve the recognition accuracy.

[0052] Then, the multi-scale fusion time-space-frequency features U output by the first cross attention network and the second cross attention network are integrated through the first channel attention network and the second channel attention network. h and multi-scale fusion time-space-frequency features U f By using attention weights for weighted splicing, we can solve the problems of insufficient fusion of features at different scales, insufficient capture of important features, and insufficient deep mining of emotional information. We can more accurately capture the key information of the data, optimize the fusion and integration process of features, effectively improve the expressiveness of features, and effectively improve recognition accuracy.

[0053] (3e) The emotion recognition module uses the splicing network to h and V f The spliced ​​data are then transformed in format through the first linear layer network, the ReLU activation layer, and the second linear layer network to obtain the predicted label y corresponding to each segment of EEG waveform data and EMG waveform data of each subject. num ; The calculation formula is:

[0054] y num =LIN2(ReLU(LIN1(concat(V h ,V f ))))

[0055] Among them, concat is the feature concatenation operation, ReLU is the LeakyReLU activation function, LIN1 and LIN2 are the first linear network and the second linear network respectively.

[0056] The emotion recognition module uses the weighted multi-scale fusion time-space-frequency features V output by the multi-scale convolution channel attention network. h and weighted multi-scale fusion frequency domain, space domain, and time domain features V f Through the first linear network, the input features are mapped to a higher-dimensional feature space to provide more complex feature representation capabilities. Then, the LeakyReLU activation function is used for nonlinear mapping to help the network learn complex features and patterns, while solving the gradient vanishing problem. Then, through the second linear network, the nonlinearly processed features are mapped to the final output space for the final recognition task. This effectively solves the problem of insufficient feature conversion and mapping, better adapts to the needs of emotion recognition tasks, and captures complex relationships in the data, effectively improving recognition accuracy.

[0057] (3f) Using the cross entropy loss function, through y num The true label corresponding to each EEG waveform data and EMG waveform data of each subject Calculate H * The loss value L' is obtained by using the gradient descent method through the L' weight parameter ω t Update and get the emotion recognition model H of this iteration t ;

[0058] Calculate the loss value L' and the weight parameter ω t The update calculation and update formula are:

[0059]

[0060] Among them, NUM represents the total number of data segments for each subject. Indicates the true label corresponding to the numth segment of data for each subject, y num It represents the predicted label of the model H' for the numth sample, and ln represents the logarithmic operation with base e.

[0061] (3g) Determine whether t ≥ T is true. If so, obtain the trained target recognition model H * , otherwise, let t = t + 1 and execute step (3b);

[0062] (4) Obtain emotion recognition results:

[0063] Input the test set data into the trained emotion recognition model H * Perform forward propagation to obtain the emotion recognition result of each test sample.

[0064] The following is a further description of the technical effects of the present invention in conjunction with the simulation results:

[0065] 1. Experimental conditions and contents:

[0066] The hardware platform of the simulation experiment is: the processor is 13th Gen Intel(R)Core(TM)i5-13500H, the processor frequency is 2.60GHz, the memory is 128GB, and the graphics card is NVIDIAGeForce RTX 4060; the software platform is Windows 11 operating system, Python 3.10 and torch 1.5.

[0067] A comparative simulation is conducted on the recognition accuracy of the present invention and the existing emotion recognition method based on pseudo-twin network.

[0068] 2. Experimental results analysis:

[0069] The present invention is applied to perform fusion recognition on the EEG waveform data and EMG waveform data of 32 subjects. First, the training sample set is used to train the two-stream multi-scale cross-graph convolutional neural network to obtain a trained emotion recognition model based on the two-stream multi-scale cross-graph convolutional neural network. Then, the test sample set is used to test the trained two-stream multi-scale cross-graph convolutional neural network.

[0070] Among the commonly used performance indicators of sentiment models, the main ones are: accuracy, precision, recall and F value. This article uses accuracy and standard deviation as evaluation indicators:

[0071]

[0072] Among them, Accuracy is the accuracy rate, TP is predicted as a positive example, and it is actually a positive example; FP is predicted as a positive example, but it is actually a negative example; TN is predicted as a negative example, but it is actually a negative example; FN is predicted as a negative example, but it is actually a positive example;

[0073] Standard deviation is a measure of the dispersion of data distribution in statistics, indicating the degree of deviation of the values ​​in the data set from the average value (mean). A smaller standard deviation means that the data points are more closely distributed around the mean, while a larger standard deviation means that the data points are more dispersed.

[0074]

[0075] Where std is the standard deviation; N is the total number of data points in the data set; x i is the i-th data point in the data set; μ is the mean of the data set, and the calculation formula is:

[0076]

[0077] Through the above formula, after model training and testing, the recognition accuracy reached 98.40% in the valence dimension and 98.33% in the arousal dimension; the recognition accuracy of the emotion recognition method based on the pseudo-twin network was 96.02% in the valence dimension and 96.45% in the arousal dimension; it can be seen that the emotion recognition method based on the dual-stream multi-scale cross-graph convolutional neural network proposed in the present invention can effectively improve the recognition accuracy of human emotions compared with the existing methods.

Claims

1. A sentiment recognition method based on a two-stream multi-scale cross-graph convolutional neural network, characterized in that: The steps include: (1) Obtain training sample set and test sample set: Preprocess the N1 pieces of EEG waveform data and N2 pieces of EMG waveform data of each subject in the M subjects to obtain multiple frequency domain data K and time domain data J of each piece of EEG waveform data and each piece of EMG waveform data of each subject, as well as multiple time domain-spatial domain data X h and frequency domain-spatial domain data X f Then, the emotional state represented by each segment of frequency domain data and time domain data, as well as the time-space domain data and frequency-space domain data are annotated, and the frequency domain data and time domain data corresponding to more than half of the EEG waveform data and EMG waveform data of each subject after preprocessing, as well as the time-space domain data and frequency-space domain data and their corresponding labels are combined into a training sample set, and the remaining preprocessed time-space domain data X h and frequency domain-spatial domain data X f and its corresponding label combination test sample set, where M≥32, N1≥640, N2≥640; (2) Construct an emotion recognition model H based on a two-stream multi-scale cross-graph convolutional neural network: Construct an emotion recognition model H including a cascaded dual-stream heterogeneous time-frequency domain graph convolution cross-attention fusion network, a multi-scale convolution channel attention network and an emotion recognition module; the dual-stream heterogeneous time-frequency domain graph convolution cross-attention fusion network includes a first branch consisting of a cascaded first graph convolution neural network and a first cross-attention network, and a second branch consisting of a cascaded second graph convolution neural network and a second cross-attention network arranged in parallel therewith; the multi-scale convolution channel attention network includes a third branch consisting of a cascaded first multi-scale convolution neural network and a first channel attention network, and a cascaded second multi-scale convolution neural network and a second channel attention network arranged in parallel therewith, and the first branch is cascaded with the third branch, and the second branch is cascaded with the fourth branch; (3) Iterative training of the emotion recognition model: The emotion recognition model H is iteratively trained through the training sample set to obtain the trained emotion recognition model H * ; (4) Obtain emotion recognition results: Input the test set data into the trained emotion recognition model H * Perform forward propagation to obtain the emotion recognition result of each test sample.

2. The method according to claim 1, characterized in that The steps of preprocessing N1 pieces of EEG waveform data and N2 pieces of EMG waveform data of each of the M subjects in step (1) are as follows: (1a) performing resting data removal processing on each EEG waveform data and EMG waveform data of each subject and then performing baseline correction, and performing Hanning window segmentation cutting on each EEG waveform data and EMG waveform data after baseline correction, to obtain multiple time domain data J corresponding to each EEG waveform data and EMG waveform data of each subject; (1b) performing Fourier transform on the time domain data corresponding to each segment of EEG waveform data and EMG waveform data of each subject to obtain multiple segments of frequency domain data K corresponding to each segment of EEG waveform data and EMG waveform data of each subject; (1c) For each time domain data and frequency domain data corresponding to each EEG waveform data and EMG waveform data of each subject, an adjacency matrix is ​​constructed to obtain the time domain-spatial domain data X containing each EEG waveform data and EMG waveform data of each subject. h and frequency domain-spatial domain data X f .

3. The method according to claim 1, characterized in that The emotion recognition model H described in step (2), wherein: The first graph convolutional neural network and the second graph convolutional neural network each include a graph convolutional layer; Both the first criss-cross attention network and the second criss-cross attention network contain stacked query convolution layers, key convolution layers, and value convolution layers; The first multi-scale convolutional neural network and the second multi-scale convolutional neural network both include four convolutional layers connected in parallel and a batch normalization layer cascaded therewith; Both the first channel attention network and the second channel attention network include stacked pooling layers, first convolutional layers, ReLU activation layers, second convolutional layers, and sigmoid activation layers; The emotion recognition module includes a cascaded concatenation network, a first linear layer network, a ReLU activation layer, and a second linear layer network.

4. The method according to claim 1, characterized in that The emotion recognition model H is iteratively trained as described in step (3), and the implementation steps are as follows: (3a) Initialize the number of training rounds to t, the maximum number of training rounds to T, T ≥ 100, and the emotion recognition model H obtained after the tth training t The weight parameter is ω t , and let t = 1; (3b) The first graph convolutional neural network and the second graph convolutional neural network are respectively used to h , X f Perform graph convolution feature extraction to obtain the time-domain-spatial domain features Y of multiple EEG waveform data and EMG waveform data of each subject. h , frequency domain-spatial domain feature Y f ; (3c) The first cross attention network outputs Y of the first graph convolutional neural network h The multi-segment frequency domain data K of each EEG waveform data and each EMG waveform data of each subject are spliced ​​to obtain the time domain-space domain-frequency domain feature Z with time domain-space domain as the main and frequency domain as the auxiliary h ; The second cross attention network outputs Y of the second graph convolutional neural network f The multiple time domain data J of each EEG waveform data and each EMG waveform data of each subject are spliced ​​to obtain the frequency domain-space domain-time domain feature Z with frequency domain-space domain as the main and time domain as the auxiliary f ; (3d) Multi-scale convolutional channel attention network for Z h and Z f Perform multi-scale convolution feature extraction to obtain weighted multi-scale fusion time domain-spatial domain-frequency domain feature V h and weighted multi-scale fusion frequency domain, space domain, and time domain features V f ; (3e) The emotion recognition module uses the splicing network to h and V f The spliced ​​data are then transformed in format through the first linear layer network, the ReLU activation layer, and the second linear layer network to obtain the predicted label y for each segment of EEG waveform data and EMG waveform data of each subject. num ; (3f) Using the cross entropy loss function, through y num The true label corresponding to each EEG waveform data and EMG waveform data of each subject Calculate H * The loss value L' is obtained by using the gradient descent method through the L' weight parameter ω t Update and get the emotion recognition model H of this iteration t ; (3g) Determine whether t ≥ T is true. If so, obtain the trained emotion recognition model H * Otherwise, let t = t + 1, H t =H, and execute step (3b).

5. The method according to claim 4, characterized in that The time-space feature Y described in step (3b) h , frequency domain-spatial domain feature Y f , the calculation formulas are: Among them, A h is the original adjacency matrix in the time domain, A f is the original adjacency matrix in the time domain, D h is the time domain degree matrix, D f is the frequency domain degree matrix, is the normalized time-domain adjacency matrix, is the normalized frequency domain adjacency matrix, W h is the time domain weight parameter, W f is the frequency domain weight parameter, and ReLU is the LeakyReLU activation function.

6. The method according to claim 4, characterized in that The time-space-frequency characteristics Z in step (3c) h , frequency domain-space domain-time domain characteristics Z f , the calculation formulas are: Z h =(Attn1×IV1(Y h ))+K Attn1=sigmoid(IQ1(K)-IK1(Y h )) Z f =(Attn2×IV2(Y f ))+J Attn2=sigmoid(IQ2(J)-IK2(Y f )) Among them: IQ1 and IQ2 are the query convolutional networks of the first cross-attention network and the second cross-attention network respectively, IK1 and IK2 are the key convolutional networks of the first cross-attention network and the second cross-attention network respectively, IV1 and IV2 are the value convolutional networks of the first cross-attention network and the second cross-attention network respectively, sigmoid is the sigmoid activation function, Attn1 and Attn2 are the weight parameters of the first cross-attention network and the second cross-attention network respectively.

7. The method according to claim 4, characterized in that The multi-scale convolutional channel attention network described in step (3d) is used for Z h and Z f Perform multi-scale convolution feature extraction, the implementation steps are: The first multi-scale convolutional neural network in the multi-scale convolutional channel attention network is Z h Perform multi-scale convolution feature extraction to obtain multi-scale fusion time domain, space domain and frequency domain features U h , the first channel attention network is for U h Perform weighted feature extraction to obtain weighted multi-scale fusion time domain-spatial domain-frequency domain feature V h ; The second multi-scale convolutional neural network for Z f Perform multi-scale convolution feature extraction to obtain multi-scale fusion frequency domain-spatial domain-time domain feature U f , the second channel attention network is for U f Perform feature weighted extraction to obtain weighted multi-scale fusion frequency domain-space domain-time domain feature V f ,in: In h =(IT11(ReLU(IT12(p(U h )))))×U h V f =(IT21(ReLU(IT22(p(U f )))))×U f Among them, ω r1 is the convolution kernel of the rth convolution of the first multi-scale convolutional neural network, ω s2 is the convolution kernel of the sth convolution of the second multi-scale convolutional neural network; p is the pooling operation, IT11 and IT12 are the first convolutional network and the second convolutional network of the first channel attention network, IT21 and IT22 are the first convolutional network and the second convolutional network of the second channel attention network, and ReLU is the LeakyReLU activation function.

8. The method according to claim 4, characterized in that The predicted label y described in step (3e) num , the calculation formula is: y num =LIN2(ReLU(LIN1(concat(V h ,V f )))) Among them, concat is the feature concatenation operation, ReLU is the LeakyReLU activation function, LIN1 and LIN2 are the first linear network and the second linear network respectively.

9. The method according to claim 4, characterized in that The loss value L' described in step (3f), and the weight parameter ω t To update, the calculation and update formulas are: Among them, NUM represents the total number of EEG waveform data and EMG waveform data of each subject. Indicates the true label corresponding to the numth segment of data for each subject, y num It represents the predicted label of the model H' for the numth sample, ln represents the logarithmic operation with base e, Represents ω t The update result, η represents the learning rate of L', Represents a derivative operation.

Images