Brain-computer interface emotion recognition method based on multi-scale principal component analysis and graph neural network

Abstract

The invention relates to a brain-computer interface emotion recognition method based on multi-scale principal component analysis and a graph neural network, and belongs to the technical field of data processing. The method comprises the following steps: data preprocessing: acquiring electroencephalogram signal data acquired by a brain-computer interface, and preprocessing the electroencephalogram signal data; multi-scale principal component analysis: performing multi-scale principal component analysis on the preprocessed electroencephalogram signal data, and extracting principal component features under different scales; a graph neural network model is constructed, wherein the graph neural network model TransGNN of Transform is nested on the GNN according to the extracted principal component features; model training: training a graph neural network model by using the labeled emotion data, and optimizing model parameters for accurately identifying different emotion states; and emotion recognition: inputting to-be-recognized electroencephalogram signal data into the trained graph neural network model, and outputting an emotion recognition result. According to the method, the accuracy, recall rate and other performance indexes of emotion recognition are improved.

Description

Technical Field

[0001] The present invention relates to a brain-computer interface emotion recognition method based on multi-scale principal component analysis and graph neural network, and belongs to the fields of brain-computer interface (BCI) technology, signal processing, machine learning and emotion recognition. Background Art

[0002] With the continuous advancement of science and technology, brain-computer interface technology, as an emerging human-computer interaction method, is gradually attracting widespread attention. As a frontier field of human-computer interaction, brain-computer interface (BCI) technology has made significant progress in emotion recognition in recent years. Emotion recognition aims to identify an individual's emotional state, such as happiness, sadness, fear, neutrality, etc., by analyzing brain activity (such as electroencephalogram (EEG) signals). This technology is of great significance for improving the naturalness and intelligence level of human-computer interaction. Brain-computer interface can directly read brain activity signals and realize communication between the brain and external devices, providing people with a new way of interaction. Among many application fields, emotion recognition is an important research direction of brain-computer interface technology. Accurately identifying the user's emotional state is of vital importance in virtual reality, intelligent human-computer interaction, mental health monitoring and other fields.

[0003] However, existing brain-computer interface emotion recognition methods still have some challenges and limitations. On the one hand, traditional emotion recognition methods often rely on complex feature engineering and machine learning algorithms. These methods require manual extraction of features in EEG signals, such as amplitude, frequency, and phase, and then use classifiers to classify emotions. However, manual feature extraction is not only time-consuming and labor-intensive, but also difficult to fully capture the complex emotional information in brain activity signals, may miss key information, and has poor denoising capabilities, resulting in limited accuracy in emotion recognition.

[0004] On the other hand, existing methods do not adequately consider the temporal and spatial characteristics of EEG signals. EEG signals are complex bioelectric signals with high temporal and spatial correlation. Traditional emotion recognition methods often only focus on a single dimension of the signal (such as time or space), while ignoring the characteristics of the signal at different scales. This limitation restricts the robustness and generalization ability of emotion recognition methods.

[0005] Finally, there is a lack of effective data processing methods. The brain activity signals collected by the brain-computer interface are usually high-dimensional and noisy, and effective data processing methods are needed to extract useful emotional features. However, when processing brain-computer interface data, existing data processing methods often fail to fully consider the multi-scale characteristics of the data, resulting in the extracted features being incomplete and inaccurate.

[0006] In order to solve the above problems, the present invention proposes a brain-computer interface emotion recognition method based on multi-scale principal component analysis and graph neural network. This method can make full use of the spatiotemporal characteristics of EEG signals, extract the principal component features at different scales through multi-scale principal component analysis, and combine it with the graph neural network model for emotion recognition.

[0007] In the data preprocessing stage of emotion recognition, there are many ways to eliminate noise, such as PCA, LDA, etc., among which principal component analysis (PCA) is widely used. However, since principal component analysis is a single-scale modeling, it also has some limitations. For example, PCA cannot separate certain artifact signals from brain signals with comparable amplitudes. So MPCA was later proposed, and in some current papers, MPCA also appears in the field of emotion recognition. Considering the criticality of eliminating noise and artifacts in EEG signals. The core advantage of MPCA is that it retains high-dimensional structure, has lower computational requirements, and requires fewer parameters to be estimated.

[0008] A new detection tool, MSPCA, is proposed by combining wavelet analysis and principal component analysis, introducing the concept of difference between principal component subspaces. PCA is responsible for extracting the relationship between multiple variables, while orthogonal wavelets separate random processes from deterministic processes and roughly decorrelate the autocorrelation between measurements. This new model has been applied in chemical production processes and epilepsy detection. The improved MSPCA method reduces the impact of random errors on measurement data and improves the accuracy of process performance monitoring and fault diagnosis. MSPCA combines the ability of PCA to remove linear variable correlation and the ability of wavelet transform to extract local features of variables and approximately decompose variable autocorrelation.

[0009] Multiscale principal component analysis (MSPCA) has been widely used for noise reduction of non-stationary signals such as EEG and is another useful new noise reduction technique. MSPCA technique is very important for reducing noise in various non-stationary and nonlinear signals. MSPCA first uses wavelet analysis to decorrelate the autocorrelated variables in a large number of signals, and then uses PCA to determine the linear relationship between the decorrelated variables.

[0010] Multiscale principal component analysis (MSPCA) is a method that can extract data features from different scales. By decomposing the data at multiple scales, we can effectively capture the complex information in the data, including features at different time scales, frequency scales, or spatial scales. In brain-computer interface emotion recognition, multiscale principal component analysis can help extract multi-scale emotion features and improve the accuracy of emotion recognition.

[0011] Graph neural network is a deep learning method specifically designed to process graph-structured data. In brain-computer interface emotion recognition, brain activity signals can be regarded as a kind of graph-structured data, where nodes represent brain regions and edges represent connections between brain regions. Graph neural network can effectively model this kind of graph-structured data, capture the relationship and interaction between brain regions, and thus better identify emotional states.

[0012] We propose the TransGNN model, which enhances the capabilities of Transformer and GNN layers by alternating them. TransGNN utilizes Transformer layers to enlarge the receptive field and decouples information aggregation from edges, thereby enhancing the information transfer capability of GNN.

[0013] In order to effectively capture the graph structure information, a detailed position encoding is designed and integrated into the GNN layer to encode the structural knowledge into the node attributes, thereby improving the performance of Transformer on the graph. To improve efficiency, the most relevant nodes are sampled for Transformer, and two efficient sample update strategies are proposed to reduce complexity.

[0014] This method not only simplifies the process of emotion recognition and reduces the complexity of feature extraction, but also improves the accuracy and robustness of emotion recognition. Summary of the invention

[0015] In order to solve the above problems, the present invention provides a brain-computer interface emotion recognition method based on multi-scale principal component analysis and graph neural network. This method utilizes the spatiotemporal characteristics of EEG signals, extracts principal component features at different scales through multi-scale principal component analysis, and combines the graph neural network model for emotion recognition, thereby overcoming the limitations of existing brain-computer interface emotion recognition technology and improving the accuracy and reliability of emotion recognition and the ability to recognize complex emotional states.

[0016] The technical solution of the present invention is: a brain-computer interface emotion recognition method based on multi-scale principal component analysis and graph neural network, the method comprising:

[0017] Step 1, data preprocessing: obtain the electroencephalogram (EEG) data collected by the brain-computer interface and perform preprocessing, including denoising, filtering and standardization;

[0018] Step 2, multi-scale principal component analysis: perform multi-scale principal component analysis on the pre-processed EEG signal data to extract the principal component features at different scales; divide the pre-processed EEG signal into multiple frequency bands or time windows, perform principal component analysis (PCA) on the data in each frequency band or time window, and extract the principal component features. Fusion the principal component features of different frequency bands or time windows to form a multi-scale principal component feature set;

[0019] Step 3: Graph neural network model construction: Based on the extracted principal component features, the Transformer graph neural network model TransGNN is embedded on the GNN;

[0020] Step 4: Model training: Use the labeled emotion data to train the graph neural network model and optimize the model parameters to accurately identify different emotion states.

[0021] Step 5, emotion recognition: input the EEG signal data to be recognized into the trained graph neural network model and output the emotion recognition result.

[0022] Furthermore, the Step 1 includes:

[0023] Step 1.1. Select a brain-computer interface device to ensure that it can accurately collect brain activity signals related to emotions; brain-computer interface devices include: EEG acquisition helmets and functional magnetic resonance imaging equipment;

[0024] Step 1.2, determine the experimental plan for data collection, including the selection criteria for experimental participants, emotion induction methods (such as watching specific videos, listening to music, etc.), collection time length, sampling frequency and other parameters;

[0025] Step 1.3, conduct a data collection experiment. During the experiment, use an EEG acquisition device to collect EEG signals, functional magnetic resonance imaging data or other physiological signals from the subject, and record relevant experimental environment information; use an EEG acquisition device to obtain EEG signals from the subject's brain;

[0026] Step 1.4: Preprocess the collected brain-computer interface data, including data cleaning, noise removal, filtering, standardization, and baseline correction to improve data quality; Data cleaning: exclude outliers and missing values ​​to ensure data integrity and consistency. Data standardization: convert data to the same scale to eliminate errors caused by different data dimensions;

[0027] Furthermore, in Step 2, the multi-scale principal component analysis process is to extract multi-scale features from the pre-processed EEG signal data, including time domain features, frequency domain features, etc.; multi-scale principal component analysis is performed on the extracted multi-scale features to reduce feature dimensions and extract the most representative features. The specific steps of extracting the principal component features under multi-scale are as follows:

[0028] Assume that there are m n-dimensional data, that is, m samples, each sample has n features, and the m n-dimensional data are combined into an m×n original data matrix X. The covariance matrix C of the original data matrix X is calculated, and the formula is C=(X-μ)(X-μ)T / m, where μ is the mean vector of the original data matrix X; the eigenvalues ​​and eigenvectors of the covariance matrix C are calculated, and the first k eigenvectors with large eigenvalues ​​are selected as principal components. The selected k eigenvectors form a projection matrix P; the original data matrix X is subjected to dimensionality reduction processing, and the original data matrix X is multiplied by the projection matrix P to obtain the data matrix Y after dimensionality reduction.

[0029] Wavelet analysis has the characteristics of multi-resolution analysis. It has the ability to characterize the local characteristics of signals in both time and frequency domains. It is a time-frequency localization analysis method with a fixed window size but a variable shape. It is suitable for detecting transient abnormal signals in normal signals and performing signal processing according to actual needs. The expansion and translation system of the basic wavelet Ψ(t) is generally called the wavelet function and is recorded as Where a is the scale factor and b is the displacement factor. Wavelet transform is divided into continuous wavelet transform and discrete wavelet transform. Continuous wavelet transform is Calculating the wavelet coefficients at each scale is a considerable amount of work and will generate too much data; in practice, the binary discrete wavelet transform is used, that is, taking a = 2 j , b=k2 j ,j,k∈Z, where Z is an integer set; then the binary discrete wavelet can be defined as Ψ j,k (t) = 2 -j / 2 Ψ(2 -j tk), the Mallat algorithm can quickly perform wavelet decomposition, and the mapping of the function f(t) on the scale function is the sequence a m It can be calculated as follows: a m =Ha m-1 Where H is a low-pass filter; the mapping of function f(t) on the wavelet function is the sequence d m =Gd m-1 Where G is a high resolution filter.

[0030] Furthermore, although the MSPCA method combines the ability of PCA to remove linear variable correlation and the ability of wavelet transform to extract local features of variables and approximately decompose variable autocorrelation, it overcomes the problem that the traditional PCA method is only suitable for analyzing data where faults or interference only exist at a fixed scale or frequency band, in order to more effectively improve the MSPCA method's ability to analyze and diagnose process changes so as to effectively analyze and monitor actual process variables; multi-scale principal component analysis, when establishing a principal component analysis model for each scale detail, the difference D between the principal component subspaces is used as a statistical control indicator; the multi-scale principal component analysis algorithm processing process includes: (1) collecting normal operation process data; (2) performing L-level wavelet decomposition on the data; (3) establishing a principal component analysis model for each scale detail, and using the difference D between the principal component subspaces as a statistical control indicator.

[0031] The improved MSPCA method is essentially to apply the difference D between the principal component subspaces as a statistical control index to the MSPCA method. This improved MSPCA method not only combines the ability of PCA to remove linear variable correlation and the ability of small transformation to extract local characteristics of variables and approximately decompose variable autocorrelation, but also uses the difference D between the principal component subspaces as a statistical control index. This method not only meets the need to reduce errors but also takes into account the multi-scale characteristics of data, but also extends the single-scale modeling method of PCA to multi-scale, and effectively improves the ability to analyze and diagnose process changes, so that the MSPCA method can capture both large-scale fluctuations in process variables and small fluctuations in process variables for more effective analysis.

[0032] Furthermore, the Step 2 includes:

[0033] Step 2.1, scale parameter selection: select scale parameters according to specific application requirements and data characteristics. The scales include time scale, frequency scale and space scale. For example, the time scale can choose different time window lengths, and the frequency scale can choose different frequency ranges.

[0034] Step 2.2, multi-scale decomposition: Use multi-scale decomposition method to decompose the pre-processed brain-computer interface data at different scale parameters to obtain sub-signals at multiple scale parameters; multi-scale decomposition methods include: wavelet transform, empirical mode decomposition; for example, data can be decomposed at different time scales by wavelet transform and other methods to extract emotional features at different time resolutions; calculate the principal components of the data at each scale to remove redundant information and noise and extract more representative multi-scale emotional features; through principal component analysis, the data dimension can be reduced and the efficiency and accuracy of subsequent processing can be improved;

[0035] Step 2.3, principal component analysis: perform principal component analysis on the sub-signals at each scale, calculate the principal component vector and eigenvalue; select the principal component vector with a large eigenvalue as the main feature at this scale, thereby extracting the principal component features at multiple scales and obtaining multi-scale emotional features.

[0036] Furthermore, in the Step 3, graph structure construction: construct a graph structure based on brain regions according to brain functional connection information; nodes represent brain regions, and edges represent functional connections between brain regions; graph neural network model definition: define the structure and parameters of the graph neural network model; the graph neural network model includes an input layer, a hidden layer, and an output layer; the hidden layer adopts a graph neural network layer, including a graph convolution layer and a graph attention layer; the graph convolution layer adopts an attention mechanism; the receptive field of GNN is limited because their message passing mechanism only aggregates information within the neighborhood, but deep GNN has an over-smoothing problem; in the graph, related nodes may be far away from the central node, and nodes within the neighborhood may also contain noise; this undermines the effectiveness of GNN; however, although Transformer can aggregate relevant information at a distance, it is difficult for Transformer to fully utilize structural information, and complexity is also an important issue; the graph neural network model TransGNN combines Transformer and GNN, and includes three submodules: (I) Transformer layer; (II) GNN layer; (III) samples update submodule:

[0037] (I) Transformer layer: The Transformer layer is used to improve the GNN layer and expand the receptive field to more relevant nodes that may be far away from the neighborhood. The formula for the attention mechanism of the Transformer layer is:

[0038] q=h i W q , h i =softmax(a t )V,

[0039] The multi-head attention formula is: MultiHead(h i )concat(head1...head m )W m ;

[0040] Among them, q is the query vector, K is the key vector, V is the value vector, and d out is the dimension of the vector, W q , W k , W v are three learnable weight matrices, h i , is the dimension of the embedding representation of the input sequence, a t is the attention score, head1, head m is the information channel learned from different subspaces, W m is the weight matrix;

[0041] (II) GNN layer: The GNN layer node feature representation and graph structure are used to help the Transformer layer better utilize the graph structure; the formula is h i =Conbine(h i ,h M (v i )), where Message() is the message transmission phase, Conbine() is the aggregation phase, and h k is the feature representation, v k is a node, N(v i ) is the edge set;

[0042] (III) Sample update submodule: After the Transformer layer and GNN layer, the attention samples are updated according to the new representation.

[0043] Furthermore, considering the locality of graph data, similar nodes are more likely to be included in the neighborhood. The sample update submodule is updated based on the random walk strategy, and updates the attention sample by exploring the neighborhood of the attention sample to find new related nodes; the random walk strategy is used to explore the local neighborhood of each sampled node; specifically, the transition probability of the random walk is calculated based on the similarity as:

[0044]

[0045] in, is the weight of edge (i, j), is the sum of the weights of the edges between node I and all its neighboring nodes, v j is a node;

[0046] Update based on message passing: The update strategy based on random walk has additional overhead; we propose another update strategy that uses message passing of GNN layer to update samples without additional overhead; specifically, during the message passing process of GNN layer, the attention samples of neighbor nodes of each central node are aggregated; the intuition behind this is that the attention samples of neighbors may also be relevant attention samples of the central node; the set of attention samples of neighboring nodes is represented as attention message, which is defined as: Attn_Msg(v i ) represents the attention-based message function, for node v jIn the message passing stage, the attention mechanism is used to calculate the message transmitted from the neighbor node to the node v j Message, Smp(v i ) represents the pair of nodes v j The operation or result of sampling.

[0047] Furthermore, the Step 3 includes:

[0048] Step 3.1, graph construction: brain activity signals are regarded as graph structure data, where the nodes of the graph represent the channels or regions of the EEG signals, and the weights of the edges are based on the correlation between the channels or regions; the definitions of nodes and edges are determined based on prior knowledge and functional connectivity analysis methods;

[0049] Step 3.2, network structure design: select the graph neural network structure, determine the number of layers, number of nodes, and activation function of the network; the graph neural network structure includes: graph convolutional neural network, graph attention network;

[0050] Furthermore, in the Step 4, the prediction result is calculated by forward propagation, and the model parameters are updated by back propagation; the Step 4 includes:

[0051] Step 4.1, Data preparation: Use the extracted multi-scale emotional features as the input data of the graph neural network, and mark different emotional states as different categories;

[0052] Step 4.2, model training: Use the labeled training data to train the graph neural network, use the optimization algorithm to adjust the network parameters to minimize the loss function, improve the accuracy of emotion recognition, and continuously optimize the network performance; the optimization algorithms include: stochastic gradient descent, Adam, and batch training;

[0053] Step 4.3, model evaluation and optimization: During the training process, the model performance indicators are regularly evaluated to evaluate the effectiveness of the method of the present invention; according to the evaluation results, the network structure, parameters and training strategies are adjusted to improve the performance of the model; the performance indicators include: calculating accuracy, recall rate, F1 value, and according to the evaluation results, the parameters of multi-scale principal component analysis and graph neural network are adjusted and optimized to further improve the performance of emotion recognition.

[0054] Furthermore, the Step 5 includes: before using the graph neural network model for emotion recognition, preprocessing the principal component features at multiple scales, the preprocessing including normalization, standardization or feature scaling; inputting the multi-scale emotion features obtained after preprocessing and multi-scale principal component analysis of the brain-computer interface data to be identified into the trained graph neural network model; the graph neural network model performs emotion recognition on the input multi-scale features and outputs the probability distribution of the emotion state; according to the probability distribution, the graph neural network model divides the emotion state into emotion classification results such as sad emotion, fear emotion, happy emotion or neutral emotion; setting a threshold or using voting method to make classification decisions.

[0055] Application scenarios: According to specific application requirements, the brain-computer interface emotion recognition method of the present invention is applied to the fields of virtual reality, intelligent human-computer interaction, mental health monitoring, etc. For example, in virtual reality, the presentation of virtual scenes can be adjusted according to the user's emotional state; in intelligent human-computer interaction, the interaction strategy can be adjusted according to the user's emotional state to improve the naturalness and humanization of the interaction; in mental health monitoring, the user's emotional abnormalities can be discovered in time to provide a basis for psychological intervention and treatment.

[0056] The beneficial effects of the present invention are:

[0057] 1. The present invention improves the accuracy of emotion recognition: multi-scale principal component analysis can extract multi-scale emotion features in brain-computer interface data, and graph neural network can effectively model the relationship between data. The combination of the two can capture emotion information more comprehensively, thereby improving the accuracy of emotion recognition; by fusing features of different scales, it can better adapt to the complexity and diversity of different emotional states and reduce the situation of misidentification and missed recognition;

[0058] 2. Enhance the ability to identify complex emotional states: The method of the present invention can handle mixed emotional states and subtle emotional differences, and has a stronger ability to identify complex emotional states; for example, it can accurately identify complex emotional states that contain both positive and negative emotions; the combination of multi-scale principal component analysis and graph neural network can better capture the dynamic changes and nonlinear relationships in brain activity signals, which helps to improve the recognition accuracy of complex emotional states;

[0059] 3. Provide more effective technical support: The brain-computer interface emotion recognition method of the present invention can provide more accurate and reliable emotion feedback for the fields of virtual reality, intelligent human-computer interaction, mental health monitoring, etc., and improve user experience and application effects: In terms of mental health monitoring, it can timely detect abnormal emotions of users and provide a basis for psychological intervention and treatment. In intelligent human-computer interaction, the interaction strategy can be adjusted according to the user's emotional state to achieve a more natural and humanized interaction;

[0060] 4. Good scalability and adaptability: The method of the present invention can be applied to different types of brain-computer interface data and emotion recognition tasks, and has good scalability and adaptability; it can be adjusted and optimized according to specific application requirements to meet the emotion recognition needs in different fields; with the continuous development of brain-computer interface technology and the continuous accumulation of data, the method of the present invention can continuously improve the performance of emotion recognition through continuous updating and optimization;

[0061] 5. Reduce computational complexity: The method of the present invention reduces computational complexity and improves the real-time performance of emotion recognition by optimizing the algorithm and model structure;

[0062] 6. Compared with the traditional method based on single-scale feature extraction and classification, this model has a higher emotion recognition rate and stronger generalization ability;

[0063] 7. The model also has the advantages of end-to-end feature learning and emotion recognition, avoiding a large amount of tedious manual feature selection and feature dimensionality reduction. It also includes steps to evaluate and verify the emotion recognition results, and calculates performance indicators such as accuracy and recall by comparing them with the actual emotional state. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a flow chart of the brain-computer interface emotion recognition system proposed by the present invention;

[0065] Figure 2 This is the overall architecture diagram of the brain-computer interface emotion recognition system proposed by the present invention;

[0066] Figure 3 It is a flow chart of multi-scale principal component analysis proposed by the present invention;

[0067] Figure 4 It is a schematic diagram of the structure of the graph neural network TransGNN module proposed in the present invention. DETAILED DESCRIPTION

[0068] Example 1: Figure 1-Figure 4 As shown, a brain-computer interface emotion recognition method based on multi-scale principal component analysis and graph neural network, the technical solution of the present invention is: a brain-computer interface emotion recognition method based on multi-scale principal component analysis and graph neural network, the method comprising:

[0069] Step 1, data preprocessing: obtain the electroencephalogram (EEG) data collected by the brain-computer interface and perform preprocessing, including denoising, filtering and standardization;

[0070] Furthermore, the Step 1 includes:

[0071] Step 1.1. Select a brain-computer interface device to ensure that it can accurately collect brain activity signals related to emotions; brain-computer interface devices include: EEG acquisition helmets and functional magnetic resonance imaging equipment;

[0072] Step 1.2, determine the experimental plan for data collection, including the selection criteria for experimental participants, emotion induction methods (such as watching specific videos, listening to music, etc.), collection time length, sampling frequency and other parameters;

[0073] Step 1.3, conduct a data collection experiment. During the experiment, use an EEG acquisition device to collect EEG signals, functional magnetic resonance imaging data or other physiological signals from the subject, and record relevant experimental environment information; use an EEG acquisition device to obtain EEG signals from the subject's brain;

[0074] Step 1.4: Preprocess the collected brain-computer interface data, including data cleaning, noise removal, filtering, standardization, and baseline correction to improve data quality; Data cleaning: exclude outliers and missing values ​​to ensure data integrity and consistency. Data standardization: convert data to the same scale to eliminate errors caused by different data dimensions;

[0075] Step 2, multi-scale principal component analysis: perform multi-scale principal component analysis on the pre-processed EEG signal data to extract the principal component features at different scales; divide the pre-processed EEG signal into multiple frequency bands or time windows, perform principal component analysis (PCA) on the data in each frequency band or time window, and extract the principal component features. Fusion the principal component features of different frequency bands or time windows to form a multi-scale principal component feature set;

[0076] Furthermore, in Step 2, the multi-scale principal component analysis process is to extract multi-scale features from the pre-processed EEG signal data, including time domain features, frequency domain features, etc.; multi-scale principal component analysis is performed on the extracted multi-scale features to reduce feature dimensions and extract the most representative features. The specific steps of extracting the principal component features under multi-scale are as follows:

[0077] Assume that there are m n-dimensional data, that is, m samples, each sample has n features, and the m n-dimensional data are combined into an m×n original data matrix X. The covariance matrix C of the original data matrix X is calculated, and the formula is C=(X-μ)(X-μ)T / m, where μ is the mean vector of the original data matrix X; the eigenvalues ​​and eigenvectors of the covariance matrix C are calculated, and the first k eigenvectors with large eigenvalues ​​are selected as principal components. The selected k eigenvectors form a projection matrix P; the original data matrix X is subjected to dimensionality reduction processing, and the original data matrix X is multiplied by the projection matrix P to obtain the data matrix Y after dimensionality reduction.

[0078] Wavelet analysis has the characteristics of multi-resolution analysis. It has the ability to characterize the local characteristics of signals in both time and frequency domains. It is a time-frequency localization analysis method with a fixed window size but a variable shape. It is suitable for detecting transient abnormal signals in normal signals and performing signal processing according to actual needs. The expansion and translation system of the basic wavelet Ψ(t) is generally called the wavelet function and is recorded as Where a is the scale factor and b is the displacement factor. Wavelet transform is divided into continuous wavelet transform and discrete wavelet transform. Continuous wavelet transform is Calculating the wavelet coefficients at each scale is a considerable amount of work and will generate too much data; in practice, the binary discrete wavelet transform is used, that is, taking a = 2 j , b=k2 j ,j,k∈Z, where Z is an integer set; then the binary discrete wavelet can be defined as Ψ j,k (t) = 2 -j / 2 Ψ(2 -j tk), the Mallat algorithm can quickly perform wavelet decomposition, and the mapping of the function f(t) on the scale function is the sequence a m It can be calculated as follows: a m =Ha m-1 Where H is a low-pass filter; the mapping of function f(t) on the wavelet function is the sequence d m =Gd m-1 Where G is a high resolution filter.

[0079] Furthermore, although the MSPCA method combines the ability of PCA to remove linear variable correlation and the ability of wavelet transform to extract local features of variables and approximately decompose variable autocorrelation, it overcomes the problem that the traditional PCA method is only suitable for analyzing data where faults or interference only exist at a fixed scale or frequency band. However, in order to more effectively improve the MSPCA method's ability to analyze and diagnose process changes, so as to effectively analyze and monitor actual process variables, multi-scale principal component analysis is performed. When establishing a principal component analysis model for each scale detail, the difference D between the principal component subspaces is used as a statistical control indicator. The multi-scale principal component analysis algorithm processing process includes: (1) collecting normal operation process data; (2) performing L-level wavelet decomposition on the data; (3) establishing a principal component analysis model for each scale detail, and using the difference D between the principal component subspaces as a statistical control indicator.

[0080] The improved MSPCA method is essentially to apply the difference D between the principal component subspaces as a statistical control index to the MSPCA method. This improved MSPCA method not only combines the ability of PCA to remove linear variable correlation and the ability of small transformation to extract local characteristics of variables and approximately decompose variable autocorrelation, but also uses the difference D between the principal component subspaces as a statistical control index. This method not only meets the need to reduce errors but also takes into account the multi-scale characteristics of data, but also extends the single-scale modeling method of PCA to multi-scale, and effectively improves the ability to analyze and diagnose process changes, so that the MSPCA method can capture both large-scale fluctuations in process variables and small fluctuations in process variables for more effective analysis.

[0081] Furthermore, the Step 2 includes:

[0082] Step 2.1, scale parameter selection: select scale parameters according to specific application requirements and data characteristics. The scales include time scale, frequency scale and space scale. For example, the time scale can choose different time window lengths, and the frequency scale can choose different frequency ranges.

[0083] Step 2.2, multi-scale decomposition: Use multi-scale decomposition method to decompose the pre-processed brain-computer interface data at different scale parameters to obtain sub-signals at multiple scale parameters; multi-scale decomposition methods include: wavelet transform, empirical mode decomposition; for example, data can be decomposed at different time scales by wavelet transform and other methods to extract emotional features at different time resolutions; calculate the principal components of the data at each scale to remove redundant information and noise and extract more representative multi-scale emotional features; through principal component analysis, the data dimension can be reduced and the efficiency and accuracy of subsequent processing can be improved;

[0084] Step 2.3, principal component analysis: perform principal component analysis on the sub-signals at each scale, calculate the principal component vector and eigenvalue; select the principal component vector with a large eigenvalue as the main feature at this scale, thereby extracting the principal component features at multiple scales and obtaining multi-scale emotional features.

[0085] Step 3: Graph neural network model construction: Based on the extracted principal component features, the Transformer graph neural network model TransGNN is embedded on the GNN;

[0086] Furthermore, in the Step 3, the graph neural network model includes an input layer, a hidden layer and an output layer; the hidden layer adopts a graph neural network layer, including a graph convolution layer and a graph attention layer; the graph convolution layer adopts an attention mechanism; the receptive field of GNN is limited because their message passing mechanism only aggregates information within the neighborhood, but deep GNN has an over-smoothing problem; in the graph, related nodes may be far away from the central node, and nodes within the neighborhood may also contain noise; this undermines the effectiveness of GNN; however, although Transformer can aggregate long-distance related information, it is difficult for Transformer to fully utilize structural information, and complexity is also an important issue; the graph neural network model TransGNN combines Transformer and GNN, and includes three submodules: (I) Transformer layer; (II) GNN layer; (III) samples update submodule:

[0087] (I) Transformer layer: The Transformer layer is used to improve the GNN layer and expand the receptive field to more relevant nodes that may be far away from the neighborhood. The formula for the attention mechanism of the Transformer layer is:

[0088]

[0089] The multi-head attention formula is: MultiHead(h i )concat(head1...head m )W m ;

[0090] Among them, q is the query vector, K is the key vector, V is the value vector, and d out is the dimension of the vector, W q , W k , W v are three learnable weight matrices, h i , is the dimension of the embedding representation of the input sequence, a t is the attention score, head1, head m is the information channel learned from different subspaces, W m is the weight matrix;

[0091] (II) GNN layer: The GNN layer node feature representation and graph structure are used to help the Transformer layer better utilize the graph structure; the formula is h i =Conbine(h i ,h M (v i)), where Message() is the message transmission phase, Conbine() is the aggregation phase, and h k is the feature representation, v k is a node, N(v i ) is the edge set;

[0092] (III) Sample update submodule: After the Transformer layer and GNN layer, the attention samples are updated according to the new representation.

[0093] Furthermore, considering the locality of graph data, similar nodes are more likely to be included in the neighborhood. The sample update submodule is updated based on the random walk strategy, and updates the attention sample by exploring the neighborhood of the attention sample to find new related nodes; the random walk strategy is used to explore the local neighborhood of each sampled node; specifically, the transition probability of the random walk is calculated based on the similarity as:

[0094]

[0095] in, is the weight of edge (i, j), is the sum of the weights of the edges between node I and all its neighboring nodes, v j is a node;

[0096] Update based on message passing: The update strategy based on random walk has additional overhead; we propose another update strategy that uses message passing of GNN layer to update samples without additional overhead; specifically, during the message passing process of GNN layer, the attention samples of neighbor nodes of each central node are aggregated; the intuition behind this is that the attention samples of neighbors may also be relevant attention samples of the central node; the set of attention samples of neighboring nodes is represented as attention message, which is defined as: Attn_Msg(v i ) represents the attention-based message function, for node v j In the message passing stage, the attention mechanism is used to calculate the message transmitted from the neighbor node to the node v j Message, Smp(v i ) represents the pair of nodes v j The operation or result of sampling.

[0097] The graph neural network of the present invention nests Transformer to establish a brain-computer interface emotion recognition model of multi-scale principal component analysis and graph neural network. Compared with the traditional method based on single-scale feature extraction and classification, this model has a higher emotion recognition rate and stronger generalization ability. In addition, the model also has the advantages of end-to-end feature learning and emotion recognition, avoiding a large amount of tedious manual feature selection and feature dimensionality reduction.

[0098] Furthermore, the Step 3 includes:

[0099] Step 3.1, Graph Construction: Treat brain activity signals as graph structure data, where the nodes of the graph represent the channels or brain regions of the EEG signals, the edges represent the functional connections between brain regions, and the weights of the edges are based on the correlation between channels or regions; the definitions of nodes and edges are determined based on prior knowledge and functional connectivity analysis methods;

[0100] Step 3.2, network structure design: select the graph neural network structure, determine the number of layers, number of nodes, and activation function of the network; the graph neural network structure includes: graph convolutional neural network, graph attention network;

[0101] Step 4: Model training: Use the labeled emotion data to train the graph neural network model and optimize the model parameters to accurately identify different emotion states.

[0102] Furthermore, in the Step 4, the prediction result is calculated by forward propagation, and the model parameters are updated by back propagation; the Step 4 includes:

[0103] Step 4.1, Data preparation: Use the extracted multi-scale emotional features as the input data of the graph neural network, and mark different emotional states as different categories;

[0104] Step 4.2, model training: Use the labeled training data to train the graph neural network, use the optimization algorithm to adjust the network parameters to minimize the loss function, improve the accuracy of emotion recognition, and continuously optimize the network performance; the optimization algorithms include: stochastic gradient descent, Adam, and batch training;

[0105] Step 4.3, model evaluation and optimization: During the training process, the model performance indicators are regularly evaluated to evaluate the effectiveness of the method of the present invention; according to the evaluation results, the network structure, parameters and training strategies are adjusted to improve the performance of the model; the performance indicators include: calculating accuracy, recall rate, F1 value, and according to the evaluation results, the parameters of multi-scale principal component analysis and graph neural network are adjusted and optimized to further improve the performance of emotion recognition.

[0106] Step 5, emotion recognition: input the EEG signal data to be recognized into the trained graph neural network model and output the emotion recognition result.

[0107] Furthermore, the Step 5 includes: before using the graph neural network model for emotion recognition, preprocessing the principal component features at multiple scales, the preprocessing including normalization, standardization or feature scaling; inputting the multi-scale emotion features obtained after preprocessing and multi-scale principal component analysis of the brain-computer interface data to be identified into the trained graph neural network model; the graph neural network model performs emotion recognition on the input multi-scale features and outputs the probability distribution of the emotion state; according to the probability distribution, the graph neural network model divides the emotion state into emotion classification results such as sad emotion, fear emotion, happy emotion or neutral emotion; setting a threshold or using voting method to make classification decisions.

[0108] Application scenarios: According to specific application requirements, the brain-computer interface emotion recognition method of the present invention is applied to the fields of virtual reality, intelligent human-computer interaction, mental health monitoring, etc. For example, in virtual reality, the presentation of virtual scenes can be adjusted according to the user's emotional state; in intelligent human-computer interaction, the interaction strategy can be adjusted according to the user's emotional state to improve the naturalness and humanization of the interaction; in mental health monitoring, the user's emotional abnormalities can be discovered in time to provide a basis for psychological intervention and treatment.

[0109] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A brain-computer interface emotion recognition method based on multi-scale principal component analysis and graph neural network, characterized by: The method comprises: Step 1, data preprocessing: obtain the EEG signal data collected by the brain-computer interface and perform preprocessing, including denoising, filtering and standardization; Step 2, multi-scale principal component analysis: perform multi-scale principal component analysis on the preprocessed EEG signal data to extract the principal component features at different scales; Step 3: Build the graph neural network model: Based on the extracted principal component features, we embed the Transformer graph neural network model TransGNN on the GNN. Step 4: Model training: Use the labeled emotion data to train the graph neural network model and optimize the model parameters to accurately identify different emotion states. Step 5, emotion recognition: input the EEG signal data to be recognized into the trained graph neural network model and output the emotion recognition result.

2. The method for brain-computer interface emotion recognition based on multi-scale principal component analysis and graph neural network according to claim 1, characterized in that: The Step 1 includes: Step 1.

1. Select the brain-computer interface device, which includes: EEG acquisition helmet, functional magnetic resonance imaging equipment; Step 1.2, determine the experimental plan for data collection, including the selection criteria for experimental participants, emotion induction methods, collection time length, and sampling frequency; Step 1.3, conduct data collection experiments. During the experiment, use EEG acquisition equipment to collect EEG signals, functional magnetic resonance imaging data or other physiological signals from the subjects, and record relevant experimental environment information; Step 1.4: Preprocess the collected brain-computer interface data, including data cleaning, noise removal, filtering, standardization, and baseline correction.

3. The method for brain-computer interface emotion recognition based on multi-scale principal component analysis and graph neural network according to claim 1, characterized in that: In the Step 2, the multi-scale principal component analysis process is to extract multi-scale features from the pre-processed EEG signal data, perform multi-scale principal component analysis on the extracted multi-scale features, and extract principal component features at multiple scales; The specific steps are as follows: Assume that there are m n-dimensional data, that is, m samples, each sample has n features, and the m n-dimensional data are combined into an m×n original data matrix X. The covariance matrix C of the original data matrix X is calculated, and the formula is C=(X-μ)(X-μ)T / m, where μ is the mean vector of the original data matrix X; the eigenvalues ​​and eigenvectors of the covariance matrix C are calculated, and the first k eigenvectors with large eigenvalues ​​are selected as principal components. The selected k eigenvectors form a projection matrix P; the original data matrix X is subjected to dimensionality reduction processing, and the original data matrix X is multiplied by the projection matrix P to obtain the data matrix Y after dimensionality reduction.

4. The method for brain-computer interface emotion recognition based on multi-scale principal component analysis and graph neural network according to claim 1, characterized in that: In Step 2, multi-scale principal component analysis is performed. When establishing a principal component analysis model for each scale detail, the difference D between the principal component subspaces is used as a statistical control indicator. The multi-scale principal component analysis algorithm processing process includes: (1) collecting normal operation process data; (2) performing L-level wavelet decomposition on the data; (3) establishing a principal component analysis model for each scale detail, and using the difference D between the principal component subspaces as a statistical control indicator.

5. The method for brain-computer interface emotion recognition based on multi-scale principal component analysis and graph neural network according to claim 1, characterized in that: The Step 2 includes: Step 2.1, scale parameter selection: select scale parameters according to specific application requirements and data characteristics. The scales include time scale, frequency scale and space scale; Step 2.2, multi-scale decomposition: Use a multi-scale decomposition method to decompose the pre-processed brain-computer interface data at different scale parameters to obtain sub-signals under multiple scale parameters; multi-scale decomposition methods include: wavelet transform, empirical mode decomposition; Step 2.3, principal component analysis: perform principal component analysis on the sub-signals at each scale, calculate the principal component vector and eigenvalue; select the principal component vector with a large eigenvalue as the main feature at this scale, thereby extracting the principal component features at multiple scales and obtaining multi-scale emotional features.

6. The method for brain-computer interface emotion recognition based on multi-scale principal component analysis and graph neural network according to claim 1, characterized in that: In Step 3, the graph neural network model includes an input layer, a hidden layer and an output layer; the hidden layer adopts a graph neural network layer, including a graph convolution layer and a graph attention layer; the graph convolution layer adopts an attention mechanism; the graph neural network model TransGNN combines Transformer and GNN, and includes three submodules: (I) Transformer layer; (II) GNN layer; (III) samples update submodule: (I) Transformer layer: The Transformer layer is used to improve the GNN layer and expand the receptive field to more relevant nodes that may be far away from the neighborhood. The formula for the attention mechanism of the Transformer layer is: q=h i W q , h i =softmax(a t )V, The multi-head attention formula is: MultiHead(h i )concat(head1...head m )W m ; Among them, q is the query vector, K is the key vector, V is the value vector, and d out is the dimension of the vector, W q , W k , W v are three learnable weight matrices, h i , is the dimension of the embedding representation of the input sequence, a t is the attention score, head1, head m is the information channel learned from different subspaces, W m is the weight matrix; (II) GNN layer: The GNN layer node feature representation and graph structure are used to help the Transformer layer better utilize the graph structure; the formula is h i =Conbine(h i ,h M (v i )), where Message() is the message transmission phase, Conbine() is the aggregation phase, and h k is the feature representation, v k is a node, N(v i ) is the edge set; (III) Sample update submodule: After the Transformer layer and GNN layer, the attention samples are updated according to the new representation.

7. The method for brain-computer interface emotion recognition based on multi-scale principal component analysis and graph neural network according to claim 6, characterized in that: The sample update submodule is updated based on the random walk strategy, and updates the attention sample by exploring the attention sample neighborhood to find new related nodes; the random walk strategy is used to explore the local neighborhood of each sampling node; specifically, the transition probability of the random walk is calculated based on the similarity as: in, is the weight of edge (i, j), is the sum of the weights of the edges between node I and all its neighboring nodes, v j is a node; The samples are updated using the message passing of the GNN layer. Specifically, during the message passing process of the GNN layer, the attention samples of the neighboring nodes of each central node are aggregated. The attention sample set of the neighboring nodes is represented as the attention message, which is defined as: Attn_Msg(v i ) represents the attention-based message function, for node v j In the message passing stage, the attention mechanism is used to calculate the message transmitted from the neighbor node to the node v j Message, Smp(v i ) represents the pair of nodes v j The operation or result of sampling.

8. The method for brain-computer interface emotion recognition based on multi-scale principal component analysis and graph neural network according to claim 1, characterized in that: The Step 3 includes: Step 3.1, graph construction: brain activity signals are regarded as graph structure data, where the nodes of the graph represent the channels or regions of the EEG signals, and the weights of the edges are based on the correlation between the channels or regions; the definitions of nodes and edges are determined based on prior knowledge and functional connectivity analysis methods; Step 3.2, network structure design: select the graph neural network structure, determine the number of network layers, number of nodes, and activation function; the graph neural network structure includes: graph convolutional neural network and graph attention network.

9. The method for brain-computer interface emotion recognition based on multi-scale principal component analysis and graph neural network according to claim 1, characterized in that: In the Step 4, the prediction result is calculated by forward propagation, and the model parameters are updated by back propagation; the Step 4 includes: Step 4.1, Data preparation: Use the extracted multi-scale emotional features as the input data of the graph neural network, and mark different emotional states as different categories; Step 4.2, model training: Use the labeled training data to train the graph neural network, and use the optimization algorithm to adjust the network parameters to minimize the loss function; the optimization algorithms include: stochastic gradient descent, Adam, and batch training; Step 4.3, Model evaluation and optimization: During the training process, the model performance indicators are evaluated regularly. According to the evaluation results, the network structure, parameters and training strategies are adjusted to improve the performance of the model; performance indicators include: calculation accuracy, recall rate, and F1 value.

10. The method for brain-computer interface emotion recognition based on multi-scale principal component analysis and graph neural network according to claim 1, characterized in that: The Step 5 includes: before using the graph neural network model for emotion recognition, preprocessing the principal component features at multiple scales, wherein the preprocessing includes normalization, standardization or feature scaling; inputting the multi-scale emotion features obtained after preprocessing and multi-scale principal component analysis of the brain-computer interface data to be recognized into the trained graph neural network model; the graph neural network model performs emotion recognition on the input multi-scale features and outputs the probability distribution of the emotion state; based on the probability distribution, the graph neural network model divides the emotion state into different emotion classification results; setting a threshold or using a voting method to make a classification decision.

Images