Time-frequency multi-layer brain network-based natural hand movement electroencephalogram (EEG) recognition method

Abstract

Disclosed in the present invention is a time-frequency multi-layer brain network-based natural hand movement EEG recognition method, comprising: (1) acquiring a multi-channel EEG signal of a natural hand movement; (2) pre-processing the multi-channel EEG signal to extract a δ wave, a θ wave, an α wave, a β wave, and a γ wave at each time point; (3) constructing a time domain multi-layer brain network by using a wSAR model; (4) calculating a frequency domain multi-layer brain network by using cross-frequency coupling (CFC); (5) combining and standardizing the time domain and frequency domain multi-layer brain network; and (6) calculating a multi-layer network metric of the multi-layer brain network and a decomposed super-adjacency matrix, and inputting same to a double-layer graph convolutional network (GCN) to fuse artificial, shallow, and deep features for classification. According to the present invention, a new time-frequency fusion multi-layer brain network is proposed by using the wSAR model and the CFC. In addition, feature information of the multi-layer brain network is comprehensively extracted by using two layers of GCNs, so that the neurophysiological mechanism of brain can be revealed and the decoding precision of the natural action EEG can be improved.

Description

A method for EEG recognition of natural hand movements based on time-frequency multi-layer brain network Technical Field

[0001] The present invention belongs to the technical field of electroencephalogram (EEG) signal processing, and in particular relates to an EEG recognition method for natural hand movements based on a time-frequency multi-layer brain network. Background Art

[0002] A brain-computer interface (BCI) is a method of controlling external devices through EEG signals. It collects and analyzes EEG signals, and decodes them into control instructions that can be recognized by the machine, thereby achieving control of external devices. Currently, brain-computer interfaces are widely used in the fields of medical rehabilitation and neuroengineering. In the study of EEG signals of natural hand movements, the sophistication of human hand movements and external interference lead to the complexity of natural movements. Many natural movements require the coordinated work of multiple brain regions to be realized. Traditional EEG feature extraction methods only consider the individual features of some brain regions, which makes it difficult for traditional EEG feature extraction methods to extract more effective features.

[0003] Given the growing demand for joint analysis of multiple brain regions, brain network analysis has become a hot topic in brain-computer interfaces. Brain network analysis can explore neural connections in the brain at both macroscopic and microscopic scales, and can also uncover distinct connections between individual brain regions at the structural, functional, or causal-effect levels. In recent years, brain network analysis has primarily focused on functional and causal-effect connectivity.

[0004] Although brain network analysis has shown strong vitality, the information represented by a single-layer brain network only reflects the relationship between specific times or specific frequencies, and the resulting feature information is still very limited. Therefore, multi-layer brain network analysis is necessary in current research. Due to the ambiguity of the definition of multi-layer brain networks, most current multi-layer brain network analysis studies have ignored the importance of inter-layer connections, resulting in information loss in multi-layer brain networks. In addition, many previous studies have only considered multi-layer brain networks in the frequency domain or dynamic brain networks in the time domain, while ignoring multi-layer brain networks in the time domain, which fails to fully utilize multi-layer brain network information.

[0005] Summary of the Invention

[0006] To solve the above problems, the present invention discloses a method for EEG recognition of natural hand movements based on a time-frequency multi-layer brain network. This method proposes a new possibility for inter-layer connection between time domain and frequency domain multi-layer brain networks, and also provides a new method for comprehensive analysis of natural hand movement EEG.

[0007] To achieve the above object, the technical solution of the present invention is as follows:

[0008] A method for recognizing natural hand movements using EEG based on a time-frequency multilayer brain network, comprising the following steps:

[0009] (1) Collecting multi-channel EEG signals of natural hand movements

[0010] (2) Preprocess the multi-channel EEG signals and use a sliding time window to extract the EEG signals at each time point, and extract the delta wave, theta wave, alpha wave, beta wave, and gamma wave at each time point.

[0011] (3) Data from each time point are processed, using electrodes as nodes and directional transfer functions (DTFs) as intra-layer connections to construct a time-domain multilayer brain network. Specifically, the time-domain inter-layer connections are calculated as follows: A weighted quadratic autoregressive (wSAR) model is constructed: Based on the assumption that closer temporal distances have a greater impact, a weighted term is added to the AR model, and the constant term is processed. The improved wSAR model parameters are calculated as inter-layer connections.

[0012] (4) The data of each frequency band are processed to calculate the cross-frequency inter-layer coupling. The calculation method of the frequency domain inter-layer connection is as follows: perform Hilbert transform on the EEG signal of each frequency, extract the amplitude and phase information, and calculate the cross-frequency coupling (CFC) between each frequency. Here, the average value of the phase-amplitude coupling (PAC), phase-phase coupling (PPC) and amplitude-amplitude coupling (AAC) indicators is selected as the final inter-layer connection.

[0013] (5) The time-frequency multilayer brain network constructed by (3) and (4) is combined into a 3D standard form: M = (V, E, ω, l), where V represents the point set, E represents the edge set, ω represents the size of the edge, and l represents the position mark of the point. This standard form abstracts the brain area into the z-axis, time as the x-axis, and frequency as the y-axis, that is, compressing the single-layer network into one dimension. As the number of layers increases, the dimensionality of the multilayer brain network increases.

[0014] (6) Calculate the standard form of multi-layer brain network metrics in (5), including common multi-layer degree, eigenvector centrality, and temporal flexibility unique to multi-layer brain networks, for neural analysis of natural action decoding. Finally, calculate the adjacency matrix and input this standard form into a graph convolutional network (GCN) for classification and recognition.

[0015] Furthermore, step (2) includes:

[0016] (2.1) The EEG signal was low-pass filtered at 100 Hz, outliers were removed, and linear interpolation was used to supplement the data.

[0017] (2.2) Use the ICA blind source separation algorithm to remove other bioelectric components, and then perform segmentation, baseline correction and public average reference.

[0018] (2.3) The processed EEG signal is filtered using a 4th-order Butterworth IIR filter at 0.3-3 Hz (δ wave), 4-8 Hz (θ wave), 8-13 Hz (α wave), 13-30 Hz (β wave), and 30-45 Hz (γ wave). The EEG signal of each frequency is then sampled using a time window of 1 s and a step size of 100 ms to obtain the signal value at the corresponding time point.

[0019] Furthermore, step (3) includes:

[0020] (3.1) Select the EEG signal of a certain channel in a certain frequency band and construct a wSAR model for the EEG signal:

[0021] in represents the EEG signal value of the jth channel at time i, represents the model parameter value of channel j, ω j represents the constant term, which is assumed to be the initial value of each channel, ε represents the noise, the model order is m, and k is the weighting value. In order to avoid the k value being too large, which will cause the subsequent time term to lose its meaning, we set k to arrive At the same time, in order to consider the optimal order, the initial value of m is set to 1 here.

[0022] (3.2) The wSAR model formula is rewritten as follows:

[0023] Using the least squares method, let ε 2 to a j The partial derivative of is equal to 0, which minimizes the residual sum of squares.

[0024] In order to make the formula expression simpler, the wSAR model is recorded in matrix form below.

[0025] Where N is the total number of time points, the parameter matrix A j The least squares estimate of

[0026] (3.3) The EEG signals of all channels in this frequency band are calculated by (3.2) to obtain all the parameter matrices A j According to the actual situation, when there is no action intention and action execution, each channel should have the same initial value for the action response, so it is necessary to align the initial values ​​of each channel. This method takes each parameter matrix A j The constant term ω in j The mean of ω avg ,Right now

[0027] Where M represents the total number of channels, ω j is the constant term of each channel parameter matrix. Then ω avg Substitute back the original AR model formula of all channels and repeat step (3.2) to obtain the final parameter matrix B j .

[0028] (3.4) Take the order m from 1 to 10, repeat the steps (3.2) and (3.3), and calculate the autocovariance function σ1, σ2, ..., σ of each order sample. 10 According to the final prediction error (FPE) criterion, calculate the order that makes FPE reach the minimum, and select the parameter matrix B of this order j .

[0029] (3.5) Multiply both sides of the AR model equation for all channels by a time lag term And perform z-transform to obtain the transfer function H of the system ij (f) represents the transfer relationship from channel i to channel j. Then normalize it to get the final directional transfer function (DTF)

[0030] Among them H im (f) represents the transfer function from channel i to channel m.

[0031] (3.6) All the signals of all frequency bands are calculated through steps (3.2)-(3.5), and finally a time-domain multi-layer brain network is constructed: the electrodes are used as nodes; a threshold of 60% is taken to remove false connections, and the final obtained As an intra-layer connection, that is, the connection from node i to node j in the same layer is Take a 60% threshold to remove false connections and transform the parameter matrix B j As an inter-layer connection, the connection from node i in layer k to node i in layer l in frequency band j is

[0032] Furthermore, step (4) includes:

[0033] (4.1) Perform Hilbert transform on the EEG signals of each frequency at all time points to obtain the amplitude and phase values ​​at each time point.

[0034] (4.2) Calculate the phase-amplitude coupling (PAC), phase-phase coupling (PPC), and amplitude-amplitude coupling (AAC) between any two frequencies:

[0035] Where n is the total number of time points in the EEG data, represents the amplitude of the high-frequency band H at time i, represents the phase of the low-frequency band L at time i.

[0036] in and They represent the phase values ​​of frequency bands a and b at time i respectively. The imaginary part and sgn function are taken to reduce the influence of volume conduction effect and common source.

[0037] AAC=Corr(A i ,A j )

[0038] Among them, Corr() represents the Pearson correlation coefficient, A i With A j Represent the amplitudes of frequency bands i and j respectively.

[0039] (4.3) Calculate the average value of PAC, PPC, and AAC as the cross-frequency coupling value CFC, take a threshold of 60%, remove false connections, and use the CFC value as the inter-layer coupling between corresponding frequencies to form the frequency domain part of the multi-layer brain network.

[0040] Furthermore, step (5) specifically includes:

[0041] The obtained multi-layer brain network in the time-frequency domain is written in a standardized form: M = (V, E, ω, l), with the brain area as the z-axis, time as the x-axis, and frequency as the y-axis. Where V is a point set, including the point labels, such as ("v1", ..., "v N ”); E is an edge set, including the label of the edge, such as (“e 12 ”,……,”e N(N-1) ”); ω is the connection value of the edge, such as (ω(v i , v j ), ...); l is the location label of point V, which is used to indicate the location of the point, so l is a two-dimensional matrix, such as (("v1", 1, 1, 1), ..., ("v 25 ”, 2, 5, 6), …), where since the present invention has three dimensions, the three numbers after the point number are the coordinates of the point. The reason for standardizing the multi-layer brain network is firstly to facilitate visualization and retrieval, and secondly to facilitate the unified processing of subsequent metric calculations and GCN.

[0042] Furthermore, step (6) specifically includes:

[0043] (6.1) Calculating the metrics of time-frequency multilayer brain networks: multilayer degree d M (i), eigenvector centrality θ jβ, the node flexibility F is used for neurophysiological analysis. At the same time, the manually extracted features are input into the last layer of GCN for feature weighted fusion. Q=∑ ijkr [(A ijs -γ s P ijs )δ(s,k)+ω isk ]δ(c ir ,c is )

[0044] where d M (i) is v i The total node degree of , a(i,j,k) is the element in the adjacency tensor of the time-frequency multilayer brain network; is the adjacency tensor of the multi-layer brain network, θ jβ ,θ jα is the eigenvector of the tensor corresponding to the eigenvalue λ; f i is the number of changes of a node community at each time point, N is the total number of changes, and F j For node v j The node flexibility; Q is a modular function used for community detection, where A ijs is the adjacency matrix of layer s, P ijs is the associated zero matrix, γ s is the resolution parameter, ω isk is the connection value from node i in layer s to node i in layer k, c ir with c is is to set the parameters, δ() is the Dirac function.

[0045] (6.2) The time-frequency multilayer brain network is input into GCN, and the input node signal value is the degree of freedom of each node; the input adjacency matrix is ​​the decomposed super-adjacency matrix of the time-frequency multilayer brain network. In particular, the processing method of the time-frequency multilayer brain network adjacency matrix is ​​as follows: Since the adjacency matrix of the multilayer brain network is a three-dimensional tensor, it is difficult to calculate due to its high dimension. In order to reduce the amount of calculation, inspired by the Cartesian product of the graph, the time-frequency domain multilayer brain network is decomposed into two-directional sub-networks, and the original three-dimensional adjacency matrix is ​​decomposed into two orthogonal two-dimensional super-adjacency matrices as the adjacency matrix input of GCN, and finally feature fusion is performed.

[0046] (6.3) The GCN structure consists of two GCN layers and one feature classification layer. To fully utilize the information of the time-frequency multi-layer brain network, the two GCN layers are divided into shallow and deep layers. All images are output once after passing through the shallow GCN and once again after passing through the deep GCN. Finally, the artificial, shallow, and deep features are fused to obtain the final classification result.

[0047] The beneficial effects of the present invention are:

[0048] (1) The present invention uses a time-frequency multilayer brain network method to construct a functional connectivity model of the brain. Compared with traditional methods and general brain networks, this method describes the functional connectivity of the brain during natural hand movements from a more comprehensive perspective and explains the mechanism of brain work from a new perspective.

[0049] (2) This paper emphasizes the importance of interlayer connections in multi-layer brain networks, and proposes a new interlayer connection calculation method using the wSAR model and hybrid cross-frequency coupling, which improves the interpretability of interlayer connections and increases the information validity of multi-layer brain networks, providing a new method for improving the decoding of natural hand movements.

[0050] (3) The present invention standardizes the time-frequency multi-layer brain network, which can intuitively show the connection mode of the brain functional network and is conducive to revealing the neural operation mechanism of the brain.

[0051] (4) The present invention reduces the dimensionality of the adjacency matrix of the time-frequency multi-layer brain network, reduces the amount of calculation, and uses the improved GCN to simultaneously consider the artificial, shallow, and deep features of the multi-layer brain network. While more comprehensively utilizing the multi-layer brain network information, the accuracy of natural hand movement decoding is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] FIG1 is a main flow chart of the EEG recognition method for natural hand movements based on a time-frequency multi-layer brain network according to the present invention.

[0053] FIG2 is a flow chart of constructing a time-domain multi-layer brain network in the EEG recognition method for natural hand movements based on a time-frequency multi-layer brain network according to the present invention.

[0054] FIG3 is a standardized diagram of a time-frequency multilayer brain network of the natural hand movement EEG recognition method based on the time-frequency multilayer brain network of the present invention.

[0055] Figure 4 is a GCN structure diagram of the natural hand movement EEG recognition method based on the time-frequency multi-layer brain network of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0057] As shown in FIG1 , the present invention proposes an EEG recognition method for natural hand movements based on a time-frequency multilayer brain network, which specifically includes the following steps:

[0058] (1) Collect multi-channel EEG signals of natural hand movements;

[0059] (2) Preprocess the multi-channel EEG signals and use a sliding time window to extract the EEG signals at each time point, and extract the delta wave, theta wave, alpha wave, beta wave, and gamma wave at each time point.

[0060] Specifically, step (2) includes:

[0061] (2.1) The EEG signal was low-pass filtered at 100 Hz, outliers were removed, and linear interpolation was used to supplement the data.

[0062] (2.2) Use the ICA blind source separation algorithm to remove other bioelectric components, and then perform segmentation, baseline correction and public average reference.

[0063] (2.3) The processed EEG signal was filtered using a 4th-order Butterworth IIR filter at 0.3–3 Hz (delta wave), 4–8 Hz (theta wave), 8–13 Hz (alpha wave), 13–30 Hz (beta wave), and 30–45 Hz (gamma wave). The EEG signal at each frequency was then sampled using a time window of 1 s and a step size of 100 ms to obtain the signal value at the corresponding time point.

[0064] (3) The time-point data were processed, with electrodes as nodes, directional transfer functions (DTFs) as intra-layer connections, and weighted quadratic autoregressive (wSAR) model parameters as inter-layer connections, to construct a time-domain multilayer brain network. Specifically, the time-domain inter-layer connections were calculated as follows: Constructing the wSAR model: Based on the assumption that closer temporal distances have a greater impact, weighted terms were added to the AR model, and the constant terms were processed. The improved wSAR model parameters were calculated as inter-layer connections.

[0065] Specifically, as shown in FIG2 , step (3) includes:

[0066] (3.1) Select the EEG signal of a certain channel in a certain frequency band and construct a wSAR model for the EEG signal:

[0067] in represents the EEG signal value of the jth channel at time i, represents the model parameter value of channel j, ω j represents the constant term, which is assumed to be the initial value of each channel, ε represents the noise, the model order is m, and k is the weighting value. In order to avoid the k value being too large, which will cause the subsequent time term to lose its meaning, we set k to arrive At the same time, in order to consider the optimal order, the initial value of m is set to 1 here.

[0068] (3.2) The wSAR model formula is rewritten as follows:

[0069] Using the least squares method, let ε 2 to a j The partial derivative of is equal to 0, which minimizes the residual sum of squares.

[0070] In order to make the formula expression simpler, the wSAR model is recorded in matrix form below.

[0071] Where N is the total number of time points, the parameter matrix A j The least squares estimate of

[0072] (3.3) The EEG signals of all channels in this frequency band are calculated by (3.2) to obtain all the parameter matrices A j According to the actual situation, when there is no action intention and action execution, each channel should have the same initial value for the action response, so it is necessary to align the initial values ​​of each channel. This method takes each parameter matrix A j The constant term ω in j The mean of ω avg ,Right now

[0073] Where M represents the total number of channels, ω j is the constant term of each channel parameter matrix. Then ω avg Substitute back the original AR model formula of all channels and repeat step (3.2) to obtain the final parameter matrix B j .

[0074] (3.4) Take the order m from 1 to 10, repeat the steps (3.2) and (3.3), and calculate the autocovariance function σ1, σ2, ..., σ of each order sample. 10 According to the final prediction error (FPE) criterion, calculate the order that makes FPE reach the minimum, and select the parameter matrix B of this order j .

[0075] (3.5) Multiply both sides of the AR model equation for all channels by a time lag term And perform z-transform to obtain the transfer function H of the system ij (f) represents the transfer relationship from channel i to channel j. Then normalize it to get the final directional transfer function (DTF)

[0076] Among them H im (f) represents the transfer function from channel i to channel m.

[0077] (3.6) All the signals of all frequency bands are calculated through steps (3.2)-(3.5), and finally a time-domain multi-layer brain network is constructed: the electrodes are used as nodes; a threshold of 60% is taken to remove false connections, and the final obtained As an intra-layer connection, that is, the connection from node i to node j in the same layer is Take a 60% threshold to remove false connections and transform the parameter matrix B j As an inter-layer connection, the connection from node i in layer k to node i in layer l in frequency band j is

[0078] (4) Process the frequency band data and calculate the cross-frequency inter-layer coupling. The calculation method of the frequency domain inter-layer connection is as follows: perform Hilbert transform on the EEG signal of each frequency, extract the amplitude and phase information, and calculate the cross-frequency coupling (CFC) between each frequency. Here, the average value of the phase-amplitude coupling (PAC), phase-phase coupling (PPC) and amplitude-amplitude coupling (AAC) indicators is selected as the final inter-layer connection.

[0079] Specifically, step (4) includes:

[0080] (4.1) Perform Hilbert transform on the EEG signals of each frequency at all time points to obtain the amplitude and phase values ​​at each time point.

[0081] (4.2) Calculate the phase-amplitude coupling (PAC), phase-phase coupling (PPC), and amplitude-amplitude coupling (AAC) between any two frequencies:

[0082] Where n is the total number of time points in the EEG data, represents the amplitude of the high-frequency band H at time i, represents the phase of the low-frequency band L at time i.

[0083] in and Represent the phase values ​​of frequency bands a and b at time i respectively. The imaginary part and sgn function are taken to reduce the influence of volume conduction effect and common source. i ,A j )

[0084] Among them, Corr() represents the Pearson correlation coefficient, A i With A j Represent the amplitudes of frequency bands i and j respectively.

[0085] (4.3) Calculate the average value of PAC, PPC, and AAC as the cross-frequency coupling value CFC, take a threshold of 60%, remove false connections, and use the CFC value as the inter-layer coupling between corresponding frequencies to form the frequency domain part of the multi-layer brain network.

[0086] (5) The time-frequency multilayer brain network constructed by (3) and (4) is combined into a 3D standard form: M = (V, E, ω, l), where V represents the point set, E represents the edge set, ω represents the size of the edge, and l represents the position mark of the point. This standard form abstracts the brain area into the z-axis, time as the x-axis, and frequency as the y-axis, that is, compressing the single-layer network into one dimension. As the number of layers increases, the dimensionality of the multilayer brain network increases.

[0087] Specifically, as shown in FIG3 , step (5) specifically includes:

[0088] The obtained multi-layer brain network in the time-frequency domain is written into a standardized form: M = (V, E, ω, l), with the brain area as the z-axis, time as the x-axis, and frequency as the y-axis, as shown in Figure 3. Where V is a point set, including the point labels, such as ("v1", ..., "v N ”); E is an edge set, including the label of the edge, such as (“e 12 ”,……,”e N(N-1) ”); ω is the connection value of the edge, such as (ω(v i , v j ), ...); l is the location label of point V, which is used to indicate the location of the point, so l is a two-dimensional matrix, such as (("v1", 1, 1, 1), ..., ("v 25 ”, 2, 5, 6), …), where since the present invention has three dimensions, the three numbers after the point number are the coordinates of the point. The reason for standardizing the multi-layer brain network is firstly to facilitate visualization and retrieval, and secondly to facilitate the unified processing of subsequent metric calculations and GCN.

[0089] (6) Calculate the standard form of multi-layer brain network metrics in (5), including common multi-layer degree, eigenvector centrality, and temporal flexibility unique to multi-layer brain networks, for neural analysis of natural action decoding. Finally, calculate the adjacency matrix and input this standard form into a graph convolutional network (GCN) for classification and recognition.

[0090] Specifically, as shown in FIG4 , step (6) specifically includes:

[0091] (6.1) Calculating the metrics of time-frequency multilayer brain networks: multilayer degree d M (i), eigenvector centrality θ jβ , the node flexibility F is used for neurophysiological analysis. At the same time, the manually extracted features are input into the last layer of GCN for feature weighted fusion. Q=∑ ijkr [(A ijs -γ s P ijs )δ(s,k)+ω isk ]δ(c ir ,c is )

[0092] where d M (i) is v i The total node degree of , a(i,j,k) is the element in the adjacency tensor of the time-frequency multilayer brain network; is the adjacency tensor of the multi-layer brain network, θ jβ ,θ jα is the eigenvector of the tensor corresponding to the eigenvalue λ; f i is the number of changes of a node community at each time point, N is the total number of changes, and F j For node v j The node flexibility; Q is a modular function used for community detection, where A ijs is the adjacency matrix of layer s, P ijs is the associated zero matrix, γ s is the resolution parameter, ω isk is the connection value from node i in layer s to node i in layer k, c ir with c is is to set the parameters, δ() is the Dirac function.

[0093] (6.2) The time-frequency multilayer brain network is input into GCN, and the input node signal value is the degree of freedom of each node; the input adjacency matrix is ​​the decomposed super-adjacency matrix of the time-frequency multilayer brain network. In particular, the processing method of the time-frequency multilayer brain network adjacency matrix is ​​as follows: Since the adjacency matrix of the multilayer brain network is a three-dimensional tensor, it is difficult to calculate due to its high dimension. In order to reduce the amount of calculation, inspired by the Cartesian product of the graph, the time-frequency domain multilayer brain network is decomposed into two-directional sub-networks, and the original three-dimensional adjacency matrix is ​​decomposed into two orthogonal two-dimensional super-adjacency matrices as the adjacency matrix input of GCN, and finally feature fusion is performed.

[0094] (6.3) The GCN structure consists of two GCN layers and one feature classification layer. To fully utilize the information of the time-frequency multi-layer brain network, the two GCN layers are divided into shallow and deep layers. All images are output once after passing through the shallow GCN and once again after passing through the deep GCN. Finally, the artificial, shallow, and deep features are fused to obtain the final classification result.

[0095] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A natural hand movement EEG recognition method based on a time-frequency multi-layer brain network, characterized by: The following steps are involved: (1) Collect multi-channel EEG signals of natural hand movements; (2) Preprocess the multi-channel EEG signals and use a sliding time window to extract the EEG signals at each time point, and extract the delta wave, theta wave, alpha wave, beta wave, and gamma wave at each time point; (3) The data at each time point are processed, and the electrodes are used as nodes, DTF as intra-layer connections, and wSAR model parameters as inter-layer connections to construct a time-domain multi-layer brain network. The calculation method of the time-domain inter-layer connections is as follows: Construct the wSAR model: Based on the assumption that the closer the time distance, the greater the impact, a weighted term is added to the AR model, and the constant term is processed at the same time, and the improved wSAR model parameters are calculated as the inter-layer connections; (4) Process the data of each frequency band and calculate the cross-frequency inter-layer coupling. The calculation method of frequency domain inter-layer connection is as follows: perform Hilbert transform on the EEG signal of each frequency, extract the amplitude and phase information, calculate the cross-frequency coupling CFC between each two frequencies, and select the average value of phase-amplitude coupling PAC, phase-phase coupling PPC and amplitude-amplitude coupling AAC as the final inter-layer connection; (5) The time-frequency multilayer brain network constructed by (3) and (4) is combined into a three-dimensional standard form: M = (V, E, ω, l), where V represents the point set, E represents the edge set, ω represents the size of the edge, and l represents the position mark of the point; this standard form abstracts the brain area into the z-axis, time as the x-axis, and frequency as the y-axis, that is, compressing the single-layer network into one dimension. As the dimension of the number of layers increases, the dimension of the multilayer brain network increases; (6) Calculate the standard form of multi-layer brain network metrics in (5), including the common multi-layer degree, eigenvector centrality, and temporal flexibility unique to multi-layer brain networks, which are used for neural analysis of natural action decoding; finally, calculate the adjacency matrix and input this standard form into GCN for classification and recognition.

2. According to claim 1, a natural hand movement EEG recognition method based on time-frequency multi-layer brain network is characterized in that: Step (2) comprises: (2.1) The EEG signal was low-pass filtered at 100 Hz, outliers were removed, and linear interpolation was used for data supplementation; (2.2) Use ICA blind source separation algorithm to remove other bioelectric components, then perform segmentation, baseline correction and public average reference; (2.3) The processed EEG signal was filtered using a 4th-order Butterworth IIR filter at 0.3-3 Hz, 4-8 Hz, 8-13 Hz, 13-30 Hz, and 30-45 Hz, and then the EEG signal of each frequency was sampled using a time window of 1 s in length and 100 ms in step size to obtain the signal value at the corresponding time point.

3. The method for natural hand movement EEG recognition based on time-frequency multi-layer brain network according to claim 1 is characterized in that: Step (3) includes: (3.1) Select the EEG signal of a certain channel in a certain frequency band and construct a wSAR model for the EEG signal: in represents the EEG signal value of the jth channel at time i, represents the model parameter value of channel j, ω j represents a constant term, Here we assume it as the initial value of the influence of each channel, ε represents noise, the model order is m, and k is the weighting value; in order to avoid the k value being too large, which will cause the subsequent time terms to lose their meaning, k is set to arrive The average value of; at the same time, in order to consider the optimal order, the initial value of m is set to 1 here; (3.2) The wSAR model formula is rewritten as follows: Using the least squares method, let ε 2 to a j The partial derivative is equal to 0, which minimizes the residual sum of squares; In order to simplify the formula expression, the wSAR model is recorded in matrix form as follows: Where N is the total number of time points, the parameter matrix A j The least squares estimate of (3.3) The EEG signals of all channels in this frequency band are calculated by (3.2) to obtain all the parameter matrices A j ; Take each parameter matrix A j The constant term ω in j The mean of avg ,Right now Where M represents the total number of channels, ω j is the constant term of each channel parameter matrix; then ω avg Substitute back the original AR model formula of all channels and repeat step (3.2) to obtain the final parameter matrix B j ; (3.4) Take the order m from 1 to 10, repeat the steps of (3.2) and (3.3), and calculate the autocovariance function σ1, σ2, …, σ of each order sample. 10 ; According to the final prediction error (FPE) criterion, calculate the order that minimizes FPE and select the parameter matrix B of this order j ; (3.5) Multiply both sides of the AR model equation for all channels by a time lag term And perform z-transform to obtain the transfer function H of the system ij (f) represents the transfer relationship from channel i to channel j; then normalized to obtain the final directional transfer function (DTF) Among them, H im (f) represents the transfer function from channel i to channel m; (3.6) The signals of all frequency bands are calculated through steps (3.2)-(3.5) to finally construct a time-domain multilayer brain network: the electrodes are used as nodes; a threshold of 60% is taken to remove false connections, and the final As an intra-layer connection, that is, the same The connection from node i to node j within a layer is Take a threshold of 60% to remove false connections and transform the parameter matrix B j As the inter-layer connection, the connection from node i in layer k to node i in layer l in frequency band j is 4. The method for natural hand movement EEG recognition based on time-frequency multi-layer brain network according to claim 1 is characterized in that: Step (4) comprises: (4.1) Perform Hilbert transform on all time points of the selected EEG signals of each frequency to obtain the amplitude and phase values ​​at each time point; (4.2) Calculate the phase-amplitude coupling PAC, phase-phase coupling PPC, and amplitude-amplitude coupling AAC between each frequency respectively: Where n is the total number of time points in the EEG data, represents the amplitude of the high-frequency band H at time i, represents the phase of the low-frequency band L at time i; in and They represent the phase values ​​of frequency bands a and b at time i respectively. The imaginary part and sgn function are taken to reduce the influence of volume conduction effect and common source. AAC=Corr(A i ,A j ) Among them, Corr() represents the Pearson correlation coefficient, A i With A j represent the amplitudes of frequency bands i and j respectively; (4.3) Calculate the average value of PAC, PPC, and AAC as the cross-frequency coupling value CFC, take a threshold of 60%, remove false connections, and use the CFC value as the inter-layer coupling between corresponding frequencies to form the frequency domain part of the multi-layer brain network.

5. The method for natural hand movement EEG recognition based on time-frequency multi-layer brain network according to claim 1 is characterized in that: Step (5) specifically includes: The obtained time-frequency domain multilayer brain network is written in a standardized form: M = (V, E, ω, l), the brain area is the z-axis, the time is the x-axis, and the frequency is the y-axis; wherein V is a point set containing the point number, E is an edge set containing the edge number, ω is the connection value of the edge, and l is the position label of point V, which is used to indicate the location of the point, so l is a two-dimensional matrix, wherein since the present invention has three dimensions, the three numbers after the point number are the coordinates of the point.

6. The method for natural hand movement EEG recognition based on time-frequency multi-layer brain network according to claim 1, characterized in that: Step (6) specifically includes: (6.1) Calculate the metric of the time-frequency multilayer brain network: multi-layer degree d M (i), eigenvector centrality θ jβ , the node flexibility F is used for neurophysiological analysis; at the same time, the artificially extracted features are input into the last layer of GCN for feature weighted fusion; Q=∑ ijkr [(A ijs -c s P ijs )δ(s,k)+ω isk ]δ(c ir ,c is ) where d M (i) is v i The total node degree of , a(i,j,k) is the element in the adjacency tensor of the time-frequency multilayer brain network; is the adjacency tensor of the multi-layer brain network, θ jβ ,θ jα is the eigenvector of the tensor corresponding to the eigenvalue λ; f i is the number of changes of a node community at each time point, N is the total number of changes, and F j For node v j The node flexibility; Q is the modularity function used for community detection, where A ijs is the adjacency matrix of layer s, P ijs is the associated zero matrix, γ s is the resolution parameter, ω isk is the connection value from node i in layer s to node i in layer k, c ir With c is To set the parameters, δ() is the Dirac function; (6.2) The time-frequency multilayer brain network is input into GCN, and the input node signal value is the degree of freedom of each node; the input adjacency matrix is ​​the decomposed super-adjacency matrix of the time-frequency multilayer brain network; in particular, the processing method of the time-frequency multilayer brain network adjacency matrix is ​​as follows: since the adjacency matrix of the multilayer brain network is a three-dimensional tensor, it is difficult to calculate with high dimension; in order to reduce the amount of calculation, inspired by the Cartesian product of the graph, the time-frequency domain multilayer brain network is decomposed into two-directional sub-networks, and the original three-dimensional adjacency matrix is ​​decomposed into two orthogonal two-dimensional super-adjacency matrices as the adjacency matrix input of GCN, and finally feature fusion is performed; (6.3) The structure of GCN includes two GCN layers and one feature classification layer. In order to make full use of the information of the time-frequency multi-layer brain network, the two GCN layers are divided into shallow and deep layers. All graphs output features once after passing through the shallow GCN, and output features again after passing through the deep GCN. Finally, the artificial, shallow and deep features are fused to obtain the final classification result.

Images