EEG signal decoding method, device, terminal equipment and storage medium

Abstract

This application relates to brain-computer interface technology and discloses a method, apparatus, terminal device, and storage medium for decoding EEG signals. The method comprises: constructing a paradigm-specific static brain functional connectivity matrix based on the functional connectivity relationship between each two nodes in the EEG signal; processing the EEG signal characteristics and the static brain functional connectivity matrix of each node through a functional connectivity attention mechanism to obtain a weighted brain region connectivity map; inputting the weighted brain region connectivity map into a graph convolutional neural network to obtain a decoding result; wherein, the multiple convolutional layers of the graph convolutional neural network are respectively constructed using Chebyshev polynomials with different convolution kernel orders. This application combines the functional connectivity attention mechanism with the graph convolutional network to optimize the weighted brain region connections, thereby improving the decoding accuracy.

Description

Technical Field

[0001] The present application relates to the field of brain-computer interface technology, and in particular to a method, apparatus, terminal device and storage medium for decoding electroencephalogram (EEG) signals. Background Art

[0002] Brain-computer interface (BCI) technology is a method that decodes user intent by detecting brain signals and can be used to control external devices. This technology shows broad application prospects in various fields such as rehabilitation medicine, smart home control, entertainment and gaming. Currently, BCI decoding methods mainly rely on local features extracted from electroencephalogram (EEG) signals, such as time-frequency features, and are processed using convolutional neural networks (CNN) or EEGNet. However, these methods have the following limitations: First, they ignore the importance of global collaboration and focus only on local features of the brain, failing to fully capture the global functional connectivity between brain regions; second, decoding accuracy still needs to be improved, especially when dealing with complex tasks such as bimanual coordination. The current decoding accuracy is insufficient to meet the requirements of high-precision applications. Summary of the Invention

[0003] In view of this, embodiments of the present application provide an EEG signal decoding method, apparatus, terminal device, and storage medium, which can achieve accurate decoding of EEG signals.

[0004] In a first aspect, an embodiment of the present application provides an EEG signal decoding method, comprising:

[0005] Based on the functional connectivity relationship between every two nodes in the EEG signal, a paradigm-specific static brain functional connectivity matrix is constructed;

[0006] Processing the EEG signal features of each node and the static brain functional connectivity matrix through a functional connectivity attention mechanism to obtain a weighted brain region connection map;

[0007] The weighted brain area connection map is input into a graph convolutional neural network to obtain a decoding result; wherein, the multiple convolutional layers of the graph convolutional neural network are respectively constructed by Chebyshev polynomials with different convolution kernel orders.

[0008] In some embodiments, the graph convolutional neural network includes a plurality of convolutional layers, a fully connected layer, and a classifier;

[0009] Inputting the weighted brain region connection map into a graph convolutional neural network to obtain a decoding result includes:

[0010] Processing the weighted brain area connection map step by step through multiple layers of convolutional layers to obtain spatial features of the weighted brain area connection map that are gradually extracted from local to global perspectives;

[0011] After the spatial features are processed by the fully connected layer, they are input into the classifier to obtain a decoding result for the target task.

[0012] In some embodiments, the orders of the convolution kernels of the Chebyshev polynomials of the convolution layers increase successively;

[0013] The number of convolutional layers is six;

[0014] The first convolution layer is a convolution kernel with a first-order Chebyshev polynomial;

[0015] The second, third, fourth, and fifth convolutional layers use second-order Chebyshev polynomials as convolution kernels.

[0016] The sixth convolutional layer uses a convolution kernel of the fourth-order Chebyshev polynomial.

[0017] In some embodiments, the step of processing the weighted brain area connection map through multiple layers of convolutional layers to obtain spatial features of the weighted brain area connection map extracted gradually from local to global perspectives includes:

[0018] Inputting the weighted brain region connection map into the first convolutional layer to extract initial local features between each node and its adjacent nodes;

[0019] Inputting the initial local features into the second convolutional layer, the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer in sequence to extract intermediate local features between each of the nodes and its multi-level neighboring nodes;

[0020] The intermediate local features are input into the sixth convolutional layer to extract the global features of the nodes.

[0021] In some embodiments, processing the EEG signal features of each node in the EEG signal and the static brain functional connectivity matrix through a functional connectivity attention mechanism to obtain a weighted brain region connection map includes:

[0022] Inputting the static brain functional connectivity matrix and the EEG signal features of each node into a plurality of attention heads, respectively, each of the attention heads respectively calculating an attention score between each node pair, and weighting each node pair according to the corresponding attention score;

[0023] The weighted results of the multiple attention heads are aggregated to obtain a weighted brain area connection map.

[0024] In some embodiments, if the paradigm is a bimanual motor task paradigm or a unimanual motor task paradigm, then constructing a paradigm-specific static brain functional connectivity matrix based on the functional connectivity relationship between every two nodes in the EEG signal includes:

[0025] Calculating the functional connection strength between every two nodes in the EEG signal by using the Pearson correlation coefficient algorithm;

[0026] Functional connections whose functional connection strength is less than a preset strength threshold are excluded to construct a first target static brain functional connection matrix.

[0027] In some embodiments, if the paradigm is a bimanual motor imagery task paradigm or a unimanual motor imagery task paradigm, then constructing a paradigm-specific static brain functional connectivity matrix based on the functional connectivity relationship between every two nodes in the EEG signal includes:

[0028] Calculating the functional connection strength between every two nodes in the EEG signal by a phase-locked value algorithm;

[0029] Functional connections whose functional connection strength is less than a preset strength threshold are excluded to construct a second target static brain functional connection matrix.

[0030] In a second aspect, an embodiment of the present application provides an EEG signal decoding device, comprising:

[0031] A static brain functional connectivity matrix construction module is used to construct a paradigm-specific static brain functional connectivity matrix based on the functional connectivity relationship between every two nodes in the EEG signal;

[0032] A weighted brain area connection map acquisition module is used to process the EEG signal features of each node and the static brain functional connection matrix through a functional connection attention mechanism to obtain a weighted brain area connection map;

[0033] The decoding result acquisition module is used to input the weighted brain area connection map into the graph convolutional neural network to obtain a decoding result.

[0034] In a third aspect, an embodiment of the present application provides a terminal device, which includes a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the EEG signal decoding method described in any one of the aforementioned embodiments.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed on a processor, implements the EEG signal decoding method according to any one of the aforementioned embodiments.

[0036] The embodiments of the present application have the following beneficial effects: in the process of decoding EEG signals, the present application processes the EEG signal characteristics and static brain functional connection matrix of each node through the functional connection attention mechanism, so that the functional connections between the nodes can be weighted according to the functional connection strength to highlight the key connections between brain regions. In addition, the convolution layer of the graph convolutional neural network in this embodiment introduces Chebyshev polynomials with different convolution kernel orders, and the convolution kernel order gradually increases, so that this embodiment can further optimize the spatial information extraction ability of the weighted brain area connection map, so that the model can perform feature modeling from local to global, thereby making the final decoding accuracy higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A first flow chart of the method for decoding EEG signals according to an embodiment of the present application is shown;

[0039] Figure 2 A schematic diagram of the EEG signal decoding process according to an embodiment of the present application is shown;

[0040] Figure 3 Shown Figure 2 An enlarged schematic diagram of the heat map corresponding to the movement execution part in (b);

[0041] Figure 4 Shown Figure 2 An enlarged schematic diagram of the heat map corresponding to the motor imagery part in (b);

[0042] Figure 5 A second flow chart of the method for decoding EEG signals according to an embodiment of the present application is shown;

[0043] Figure 6 A third flow chart of the method for decoding EEG signals according to an embodiment of the present application is shown;

[0044] Figure 7 A fourth flow chart of the method for decoding EEG signals according to an embodiment of the present application is shown;

[0045] Figure 8 A schematic diagram of the model decoding process in an embodiment of the present application is shown;

[0046] Figure 92 is a schematic diagram showing a fifth flow chart of the method for decoding EEG signals according to an embodiment of the present application;

[0047] Figure 10 Schematic diagram showing the sixth flow of the EEG signal decoding method according to an embodiment of the present application;

[0048] Figure 11 A schematic diagram of movement execution and movement imagery execution according to an embodiment of the present application is shown;

[0049] Figure 12 A comparison of the decoding accuracy of different algorithms on motor execution and motor imagery tasks is shown;

[0050] Figure 13 A structural diagram of an EEG signal decoding device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0052] The components of the embodiments of the present application generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but rather merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0053] Hereinafter, the terms "including", "having" and their cognates used in various embodiments of the present application are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the aforementioned items, and should not be understood as excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the aforementioned items or adding the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the aforementioned items. In addition, the terms "first", "second", "third" and the like are only used to distinguish descriptions and should not be understood as indicating or implying relative importance.

[0054] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meaning as in the context of the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning unless clearly defined in the various embodiments of the present application.

[0055] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other.

[0056] Non-invasive brain-computer interface (BCI) technology acquires electroencephalography (EEG) signals without surgical intervention and uses algorithms to decode these signals to generate control commands, which are then converted into operating signals for external devices. Current BCI decoding methods primarily rely on extracting local features from EEG signals, such as time-frequency characteristics and event-related desynchronization / synchronization (ERD / ERS). These are further processed using deep learning models such as convolutional neural networks (CNNs) or EEGNet to generate commands for controlling external devices.

[0057] However, existing methods have some limitations. First, these methods often ignore the complex collaborative relationships between different brain regions and fail to fully capture the global functional connectivity characteristics between brain regions. Second, because most existing models are designed based on Euclidean space, it is difficult to effectively explore the complex connection patterns of different brain regions in non-Euclidean space. Therefore, although models such as CNN and EEGNet perform well in extracting local time-frequency features, their decoding accuracy is limited when dealing with tasks that require coordination across brain regions. In particular, when it comes to complex motor tasks such as bimanual coordination, the decoding accuracy often cannot meet the requirements of high-precision applications.

[0058] In summary, in order to improve the performance of non-invasive BCI systems, future research should focus more on how to model and utilize the global functional connectivity between brain regions, as well as explore brain network structure analysis methods suitable for non-Euclidean space, so as to achieve higher-precision EEG signal decoding. Based on this, the present application proposes an EEG signal decoding method, apparatus, terminal and storage device. The EEG signal decoding method is an EEG signal decoding method based on a graph convolutional neural network with functional connectivity attention. By combining a graph convolutional neural network (GCN) and a functional connectivity attention (FCA) mechanism, the EEG signals collected during a bimanual movement task paradigm or a unimanual movement task paradigm are processed to obtain decoding results. This method of the present application overcomes the limitation of existing brain-computer interface decoding methods that fail to fully utilize global connectivity features, and can effectively improve decoding accuracy.

[0059] The EEG signal decoding method is described below with reference to some specific embodiments.

[0060] Figure 1 A schematic flow chart of an EEG signal decoding method according to an embodiment of the present application is shown. Exemplarily, the EEG signal decoding method includes the following steps:

[0061] Step S100: constructing a paradigm-specific static brain functional connectivity matrix based on the functional connectivity relationship between every two nodes in the EEG signal.

[0062] First, it should be noted that the paradigm applicable to this embodiment is the paradigm of a user performing a bimanual motor task or a unimanual motor task. The bimanual motor task paradigm includes bimanual motor execution (ME) and bimanual motor imagery (MI) tasks, while the unimanual motor paradigm includes unimanual motor execution and unimanual motor imagery tasks. It is understood that the EEG signals collected in this embodiment are based on the EEG signals of users performing bimanual motor tasks or unimanual motor tasks. Of course, the EEG signals collected in this embodiment can also be extended to the EEG signals of users performing bipedal motor tasks or unipedal motor tasks.

[0063] Among them, paradigm specificity refers to the brain functional connection matrix related to the task. For example, a bimanual movement task requires the construction of a static brain functional connection matrix related to bimanual movement, and a unimanual task requires the construction of a static brain functional connection matrix related to unimanual movement, rather than a universal connection matrix.

[0064] In the graph convolutional neural network of this embodiment, the EEG signal data is represented as a graph , where V represents nodes (electrodes) and E represents edges (functional connections). The adjacency matrix is a mathematical matrix used to represent the structure of a graph. In graph theory, a graph consists of vertices and edges, and the adjacency matrix is used to represent the connection relationship between nodes. Specifically, in EEG data analysis, the adjacency matrix usually represents the functional connection relationship between brain regions (or electrodes); nodes (V) represent elements in the graph, and in EEG signal analysis, nodes usually represent brain regions or electrodes; edges (E) represent the connection relationship between nodes, and are usually used to represent or quantify the functional connectivity (FC) between brain regions or electrodes, where the weight of the edge usually reflects the strength of the functional connection.

[0065] The adjacency matrix is a The matrix of is the number of nodes, that is, the number of electrodes used to collect EEG signals. In EEG signal analysis, each element of the matrix Indicates electrode and electrodes The strength of the connection between them (i.e. the functional connection strength between them). Specifically: The larger the value, the greater the electrode and electrodes There is strong functional connectivity between them (i.e., their activities are highly synchronized or correlated); A smaller value of indicates that the functional connection between the two electrodes is weak, or their activities are not significantly synchronized.

[0066] Exemplarily, the EEG signal involved in step S100 is a pre-processed EEG signal, that is, the EEG signal obtained after the original EEG data is pre-processed by filtering, downsampling, data alignment and data segmentation extraction, wherein the pre-processing process is as follows: Figure 2As shown in part (a) of the figure. Filtering mainly removes noise and interference signals to extract useful EEG signals. This can be done by low-pass filtering to remove high-frequency noise (such as electromyography (EMG) interference) or by high-pass filtering to remove low-frequency noise (such as drift and power frequency interference (such as 50Hz or 60Hz)). The purpose of downsampling is to reduce the amount of data and speed up subsequent processing while maintaining the necessary signal characteristics. When downsampling, the sampling rate can be determined first, and a suitable sampling rate can be selected, such as reducing it to 250Hz or lower to reduce storage requirements and processing time, and then resampling can be performed (such as using interpolation or averaging to downsample the data to the target sampling rate). The purpose of data alignment is to ensure that the data from different electrodes are aligned on the time axis for subsequent processing. Data segmentation and extraction is to divide continuous EEG signal data into multiple segments for subsequent analysis and decoding. It can be to first divide the collected EEG signal data into time windows of fixed length, for example, each segment contains 2 seconds or 4 seconds of EEG signal, and then for each time window, according to the task type or experimental design, give each window a corresponding label (such as movement execution, movement imagination, etc.), and finally extract useful features in each time window, such as time-frequency features, event-related potential (ERP), etc.

[0067] It can be understood that the purpose of preprocessing the raw EEG signals is to ensure the purity and availability of the EEG signal data so that subsequent decoding and analysis can accurately reflect the actual situation of brain activity. The above preprocessing process is only exemplary. The specific preprocessing process of the raw EEG signals can be set as needed and is not limited here.

[0068] Since the static brain functional connectivity matrix reflects the functional connectivity relationship between each node, in order to determine the functional connectivity between two nodes, this embodiment uses the Pearson correlation coefficient or phase locking value method to quantify the synchronization and interactivity between various brain regions, especially to capture the activity correlation and phase relationship between each acquisition electrode.

[0069] For example, Figure 5 As shown, if the two-handed motion execution task and the one-handed motion execution task of this embodiment are executed, step S100 includes:

[0070] Step S110 , calculating the functional connectivity strength between every two nodes in the EEG signal using a Pearson correlation coefficient algorithm.

[0071] Step S120 , excluding functional connections whose functional connection strength is less than a preset strength threshold, to construct a first target static brain functional connection matrix.

[0072] The Pearson correlation coefficient can calculate the linear correlation of signal strengths between different electrodes, reflecting the synchronization between electrodes. In this embodiment, the Pearson correlation coefficient algorithm is used to quantify the functional connectivity relationship in the bimanual movement execution task or the unimanual movement execution task. The calculation formula is as follows:

[0073]

[0074] Where, and Represents two acquisition channels and The EEG signal, and represent The amplitude value at the moment, and represent Mean over time; Indicates the length of the selected EEG signal time window.

[0075] The above formula of Pearson correlation coefficient can be used to obtain the functional connectivity relationship between each two nodes when the user performs a two-handed motor task or a one-handed motor task, and then each element of the initial brain functional connectivity matrix when the user performs a two-handed motor task or a one-handed motor task can be obtained. , it can be understood that the initial brain functional connection matrix is equivalent to the adjacency matrix used in the above description of the graph convolutional neural network. The initial brain functional connection matrix can be expressed as: .

[0076] In addition, in order to remove connections with weak strength or possible noise in the brain functional connection matrix and retain key network structure features, in this embodiment, a preset strength threshold is preset. Therefore, after obtaining the initial brain functional connection matrix, the functional connections with functional connection strength less than the preset strength threshold are deleted, thereby obtaining the target static brain functional connection matrix.

[0077] like Figure 6 As shown, if the two-handed motor imagery task and the one-handed motor imagery task of this embodiment are performed, step S100 includes:

[0078] Step S130: Calculate the functional connection strength between every two nodes in the EEG signal using a phase-locked value algorithm.

[0079] Step S140 , excluding functional connections whose functional connection strength is less than a preset strength threshold, to construct a second target static brain functional connection matrix.

[0080] The phase lock value algorithm can quantify the degree of phase synchronization between electrodes by calculating the stability of the signal phase, revealing more fine-grained interactions between electrodes. In this embodiment, the phase lock value algorithm is used to quantify the functional connectivity relationship in the bimanual motor imagery task or the unimanual motor imagery task. The calculation formula is as follows:

[0081]

[0082] Where, and Representing the EEG signals of two channels and exist The phase of the moment.

[0083] The formula of the phase locking value algorithm can be used to obtain the functional connection relationship between each two nodes during the user's two-handed motor imagery task or one-handed motor imagery task, and then each element of the initial brain functional connection matrix of the two-handed motor imagery task or one-handed motor imagery task can be obtained. , it can be understood that the initial brain functional connection matrix is equivalent to the adjacency matrix used in the above description of the graph convolutional neural network. The initial brain functional connection matrix can be expressed as: .

[0084] Similarly, in order to exclude weaker connections in brain functional connections, a preset intensity threshold is also set during the motor imagery task. After obtaining the initial brain functional connection matrix, the functional connections with functional connection strength less than the preset intensity threshold are deleted to obtain the target static brain functional connection matrix.

[0085] In this embodiment, the static brain functional connection matrix when the user performs a two-handed movement execution task or a one-handed movement execution task is obtained by using the Pearson correlation coefficient algorithm, and the static brain functional connection matrix when the user performs a two-handed movement imagination task or a one-handed movement imagination task is obtained by using the phase locking value algorithm. The obtained static brain functional connection matrix excludes the weaker connections in the initial brain functional connection matrix. Among them, the functional connection strength between each node is calculated by the Pearson correlation coefficient algorithm or the phase locking value algorithm to generate the following Figure 2 Part (b) of the Figure 3 and Figure 4 )and Figure 2 Part (c) (Fig. 1) shows that, Figure 2 、 Figure 3 and Figure 4The larger functional connection strength value usually corresponds to the yellow in the heat map, that is, the closer the color is to yellow, the stronger the functional connection strength is, while the smaller value corresponds to the dark blue in the heat map, that is, the closer the color is to dark blue, the weaker the functional connection strength is. Then, by deleting the association relationship of functional connection strength less than the preset strength threshold, the following is generated: Figure 2 Part (d) of Figure 8 ) in the lower right corner of the static brain functional connection matrix (graph features corresponding to EEG signals), and Figure 2 The upper right corner of (d) shows the node features (i.e., the EEG signal features of the nodes) obtained after preprocessing the initial EEG signals, which are obtained by calculating the Pearson correlation coefficient (PCC) or the phase locking value (PLV), reflecting the synchronization and phase synchronization between different nodes.

[0086] Among them, the EEG signal characteristics of the node refer to the characteristics or attributes of each node (electrode in the electroencephalogram (EEG)), which include but are not limited to the position of the electrode (the specific position of each electrode in the brain), the EEG signal strength (the strength or amplitude of the EEG signal recorded by each electrode) and other related characteristics (which may include the frequency component of the signal recorded by the electrode, the change trend of the signal, etc.); the graph characteristics are reflected in this embodiment through the functional connection matrix (i.e., the static brain functional connection matrix).

[0087] This approach can effectively reflect the brain region collaboration patterns in bimanual or unimanual motor tasks and extract task-related neural dynamic features. The brain functional connectivity matrix based on these functional connections provides specific graph structure data for EEG signal decoding, thereby improving the robustness of graph convolutional neural networks while reducing the complexity of subsequent convolution calculations.

[0088] In step S200, the EEG signal features and the static brain functional connectivity matrix of each node are processed through the functional connectivity attention mechanism to obtain a weighted brain region connection map.

[0089] The functional connectivity attention mechanism of this embodiment uses multiple attention heads to focus on the relationships between nodes in different brain regions. Each attention head assigns different weights to different node pairs (i.e., connections between different brain regions) based on the EEG signal characteristics of each input node and the static brain functional connectivity matrix. This weighting then reflects the importance of each connection in the overall network. For example, under specific task or feature conditions, connections that are more relevant to the decoding target will receive higher weights, while connections that contribute less to the task will receive lower weights.

[0090] For example, Figure 7 As shown, step S200 includes:

[0091] In step S210, the static brain functional connectivity matrix and the EEG signal features of each node are input into multiple attention heads respectively. Each attention head calculates the attention score between each node pair and weights each node pair according to the corresponding attention score.

[0092] It can be understood that the EEG signal features of each node obtained after preprocessing the initial EEG signal, as well as the static brain functional connectivity matrix reflecting the synchronization and phase synchronization between different brain regions obtained by the Pearson correlation coefficient (PCC) or phase locking value (PLV) algorithm, are input into Figure 8 The functional connectivity attention module shown is a neural network module consisting of multiple attention heads. Each attention head performs calculations based on the received static brain functional connectivity matrix and the EEG signal characteristics of each node, thereby obtaining an attention score for each node pair. When calculating the attention score, the EEG signal characteristics of the input nodes and the static brain functional connectivity matrix can first be linearly transformed; a nonlinear activation function (such as ReLU) is then applied to the result of the linear transformation to obtain an initial attention score, which is then normalized to obtain an attention score for each node pair.

[0093] In this step, when weighting the node pairs, the node pair with a higher attention score indicates that the connection of the node pair is more important, and the weight of the node pair is greater, and vice versa.

[0094] Step S220: Aggregate the weighted results of multiple attention heads to obtain a weighted brain region connection map.

[0095] In this step, the weights of the same node pair calculated by multiple attention heads can be processed by aggregation methods such as summation and averaging to obtain the final weight of each node pair, and finally a weighted brain area connection map is obtained based on the final weight of each node pair.

[0096] It can be understood that the multi-head attention mechanism of this embodiment dynamically adjusts the connection weights based on real-time input features (node EEG signal characteristics and the static brain functional connectivity matrix). This means that the connection weights between nodes will change accordingly under different tasks or situations, thereby more accurately capturing the interaction and collaborative working patterns between brain regions. Ultimately, through this dynamic adjustment, a weighted brain region connection map is obtained. This weighted brain region connection map can more accurately reflect the relative importance and interaction strength between nodes in the brain region, providing a foundation for subsequent graph convolution operations.

[0097] Step S300: Input the weighted brain region connection map into a graph convolutional neural network to obtain a decoding result.

[0098] For example, Figure 8 The graph convolutional neural network includes multiple convolutional layers, fully connected layers, and classifiers; different orders of Chebyshev polynomial convolution kernels are used in the convolutional layers, and the order is gradually increased. For example, Chebyshev polynomials of order 1, 2, and 4, or Chebyshev polynomials of order 1, 2, 4, and 8 can be used. In this embodiment, by introducing Chebyshev polynomial convolution kernels of different orders, and the order is gradually increased, the local and extended spatial dependencies in the weighted brain area connection map can be captured. Through this multi-scale convolution operation, the model can understand the synergy between different areas of the brain from a global perspective, further improving the decoding accuracy. After the weighted brain area connection map is processed by multiple convolutional layers, the final output is processed by a fully connected layer to generate a feature representation, which is then input into the softmax classifier for task-specific prediction, and the prediction result and the decoding result are obtained. It can be understood that the number of convolutional layers is different, and the Chebyshev polynomial convolution kernels in each convolutional layer can also be different, but it is necessary to satisfy that the order of the Chebyshev polynomial convolution kernel is gradually increased, and it is necessary to ensure that the first convolutional layer and the last convolutional layer correspond to the lowest and highest order polynomials. If there are six convolution layers, the first convolution layer is a first-order Chebyshev polynomial; the second, third, fourth, and fifth convolution layers are second-order Chebyshev polynomials; the sixth convolution layer is a fourth-order Chebyshev polynomial; if there are nine convolution layers, the first convolution layer is a first-order Chebyshev polynomial; the second, third, fourth, and fifth convolution layers are second-order Chebyshev polynomials; the sixth, seventh, and eighth convolution layers are fourth-order Chebyshev polynomials; the ninth convolution layer is an eighth-order Chebyshev polynomial. Among them, Figure 8 Where A is the adjacency matrix and X is the node feature.

[0099] In some preferred embodiments, Figure 9 As shown, step S300 includes:

[0100] In step S310, the weighted brain area connection map is processed step by step through multiple convolutional layers to obtain spatial features of the weighted brain area connection map that are gradually extracted from local to global.

[0101] In step S320 , the spatial features are processed by the fully connected layer and then input into the classifier to obtain a decoding result for the target task.

[0102] In a preferred implementation, the number of convolutional layers is six; the first convolutional layer is a Chebyshev polynomial with a first-order convolution kernel; the second convolutional layer, the third convolutional layer, the fourth convolutional layer and the fifth convolutional layer are Chebyshev polynomials with a second-order convolution kernel; the sixth convolutional layer is a Chebyshev polynomial with a fourth-order convolution kernel.

[0103] For example, Figure 10 As shown, step S310 includes:

[0104] In step S311, the weighted brain region connection map is input into the first convolutional layer to extract the initial local features between each node and its adjacent nodes.

[0105] In this step, the first convolutional layer uses a first-order Chebyshev polynomial kernel. First-order Chebyshev polynomials are primarily used to capture local dependencies between adjacent nodes (i.e., directly connected nodes) in the graph. This first-order convolutional kernel allows the model to extract information about nodes directly connected to the current node, thereby better understanding local connectivity patterns. For example, if node A is connected to nodes B and C, this step extracts initial local features between nodes A and B, as well as preliminary local features between nodes A and C.

[0106] In step S312, the initial local features are sequentially input into the second convolutional layer, the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer to extract the intermediate local features between each node and its multi-level neighboring nodes.

[0107] In this step, the second, third, fourth, and fifth convolutional layers all use second-order convolution kernels. A second-order convolution kernel not only considers the current node and its immediate neighbors, but also further considers the relationships between the neighbors of these neighbors, as well as the relationships between the neighbors of the neighbors. This allows for capturing a wider range of local dependencies while remaining local, helping to understand more complex local structures. For example, if node A is connected to nodes B and C, and node B is sequentially connected to nodes D, E, and so on, then this step can extract the relationships between node A and nodes B and C, respectively, while also considering the relationships between node A and nodes D and E.

[0108] In step S313, the intermediate local features are input into the sixth convolutional layer to extract the global features of the nodes.

[0109] In this step, the fourth-order convolution kernel further expands the scope of dependencies and takes into account more levels of node connections. Through the fourth-order convolution kernel, the model can capture a wider range of global dependencies, thereby better understanding the overall structure and global characteristics of the graph.

[0110] It can be understood that in the graph convolutional neural network of this embodiment, by using Chebyshev polynomial convolution kernels of different orders, the model can gradually extract features from local to global, so as to better understand the complex relationships in the graph structure. Specifically, the first-order convolution kernel can help the model capture the preliminary local features of the directly adjacent node information; the second-order convolution kernel further expands the local dependency relationship, captures a wider range of local structures, and extracts intermediate local features; the fourth-order convolution kernel can capture more global dependency relationships, so as to better understand the overall structure of the graph, that is, to obtain global features. Through this multi-scale convolution operation, the model can gradually extract features from local to global, so as to better understand the complex relationships in the weighted brain area connection map.

[0111] like Figure 11 As shown, the brain-computer interface task requires the subjects to perform motor tasks according to the visual prompts on the screen (such as Figure 11 (a) of the ) or motor imagery tasks (e.g. Figure 11 In part (b) of the diagram, both hands move in a coordinated manner, i.e., they move to the left, middle, and right target positions. Figure 11 Part (a) is a schematic diagram of the motion execution task scenario. Figure 11 The device handle in (a) is used by the subject to slide on the guide rail by dragging the handle with his hand after receiving the instruction; Figure 11 Part (b) is a schematic diagram of the motor imagery task scene; Figure 11 Part (c) is a schematic diagram of the subject's two-handed collaborative task; Figure 11 Part (d) is a schematic diagram of the single-handed task for the subjects' left and right hands.

[0112] The decoding method of this embodiment can decode the movement direction from the EEG signal. For the two-hand movement execution and movement imagination tasks, the decoding algorithm of this embodiment ( Figure 12 The classification accuracy of the two methods (medium and dark purple blocks) reached 88.95% and 67.22% respectively, which is 16% and 18% higher than the traditional lightweight convolutional neural network (EEGNet) method. Figure 12As shown. For single-hand tasks, the decoding algorithm of this embodiment achieved classification accuracy rates of 71.76% (left hand) and 74.11% (right hand), respectively, which are 11.62% and 12.88% higher than the traditional EEGNet method. This result was verified by comparing with traditional neural network algorithms (such as lightweight convolutional neural networks (EEGNet) and CNN-based deep convolutional neural networks (DeepConvNet), etc.), and it can be concluded that the decoding method of this embodiment has significant advantages in classification accuracy. The models of the existing technology, such as EEGNet and CNN-based decoding, only focus on local signal features and fail to fully explore the complex functional connection features between brain regions. This embodiment combines the multi-head attention mechanism with the graph convolutional neural network, and introduces Chebyshev polynomials of different convolution kernel orders into the convolutional layer of the graph convolutional neural network, which effectively improves the decoding accuracy and makes task decoding more accurate, especially in the EEG signal processing of motor execution or motor imagination tasks. Compared with the existing technology, the decoding effect is improved. Among them Figure 12 The medium-lightweight convolutional neural network (EEGNet) is a compact learning network designed specifically for EEG-based brain-computer interface decoding); the deep convolutional neural network (DeepConvNet) is a deep convolutional neural network designed specifically for EEG-based brain-computer interface decoding. It captures the spatiotemporal features in EEG signals through multi-layer convolution operations, thereby improving the performance of classification or regression tasks; the bidirectional long short-term memory network (BidirectionalLong Short-Term Memory, BiLSTM) is used to process time series data. BiLSTM can learn features from both directions of the sequence (from front to back and from back to front) at the same time, thereby better capturing the contextual information in the sequence; the graph convolutional neural network (GCN) is a simple graph convolutional neural network in this embodiment, and the functional connection attention graph convolutional neural network (FCA-GCN, that is, the graph convolution application network based on functional connection attention in this application) is a decoding method that combines the multi-head attention mechanism with the graph convolutional neural network in this embodiment. Among them Figure 12 * is used to indicate statistical significance between model performances, where *p<0.05, **p<0.01, ***p<0.001.

[0113] This embodiment is applied to motor execution tasks or motor imagination tasks of both hands or one hand. During the decoding process, this embodiment combines a multi-head attention mechanism with a graph convolutional neural network. The multi-head attention mechanism weights the functional connections between each node according to the functional connection strength to highlight the key connections between brain regions. That is, the multi-head attention mechanism enables the model to focus on task-related functional connections, especially in tasks involving cross-brain region collaboration. The model can significantly capture these important connection information. This not only improves the decoding accuracy, but also greatly enhances the interpretability of the model, allowing researchers to more intuitively observe the key connections that the model focuses on during the decoding process, thereby providing useful clues for further exploring the synergistic effect of various brain regions in task execution. In addition, the convolutional layer of the graph convolutional neural network in this embodiment introduces Chebyshev polynomials with different convolution kernel orders, and the convolution kernel order gradually increases. This allows this embodiment to further optimize the spatial information extraction capability of the weighted brain region connection map, enabling the model to perform feature modeling from local to global, thereby making the final decoding accuracy higher.

[0114] Figure 13 A schematic diagram of the structure of an EEG signal decoding device according to an embodiment of the present application is shown. Exemplarily, the EEG signal decoding device includes:

[0115] The static brain functional connectivity matrix construction module 100 is used to construct a paradigm-specific static brain functional connectivity matrix based on the functional connectivity relationship between every two nodes in the EEG signal.

[0116] The weighted brain area connection map acquisition module 200 is used to process the EEG signal features of each node and the static brain functional connection matrix through the functional connection attention mechanism to obtain a weighted brain area connection map.

[0117] The decoding result acquisition module 300 is used to input the weighted brain area connection map into the graph convolutional neural network to obtain a decoding result.

[0118] It can be understood that the device of this embodiment corresponds to the EEG signal decoding method of the above embodiment, and the options in the above embodiment are also applicable to this embodiment, so they will not be described again here.

[0119] The present application also provides a terminal device. Exemplarily, the terminal device includes a processor and a memory, wherein the memory stores a computer program, and the processor runs the computer program to enable the terminal device to execute the functions of each module in the above-mentioned EEG signal decoding method or the above-mentioned EEG signal decoding device.

[0120] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, etc., and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0121] The memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM). The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving an execution instruction.

[0122] This application also provides a computer-readable storage medium for storing the computer program used in the terminal device. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or flowchart, and the combination of boxes in the structure diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0124] In addition, the functional modules or units in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0125] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a smart phone, personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0126] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A method for decoding an electroencephalogram signal, characterized in that: include: Based on the functional connectivity relationship between every two nodes in the EEG signal, a paradigm-specific static brain functional connectivity matrix is constructed; Processing the EEG signal features of each node and the static brain functional connectivity matrix through a functional connectivity attention mechanism to obtain a weighted brain region connection map; The weighted brain region connection map is input into a graph convolutional neural network to obtain a decoding result; wherein, The graph convolutional neural network includes multiple convolutional layers, fully connected layers and classifiers; the multiple convolutional layers of the graph convolutional neural network are respectively constructed by Chebyshev polynomials with different convolution kernel orders; the orders of the convolution kernels of the Chebyshev polynomials of the multiple convolutional layers increase successively; the number of convolutional layers is six; the first convolutional layer is a Chebyshev polynomial with a first-order convolution kernel; the second convolutional layer, the third convolutional layer, the fourth convolutional layer and the fifth convolutional layer are Chebyshev polynomials with a second-order convolution kernel; the sixth convolutional layer is a Chebyshev polynomial with a fourth-order convolution kernel; Inputting the weighted brain area connection map into a graph convolutional neural network to obtain a decoding result includes: gradually processing the weighted brain area connection map through multiple layers of the convolutional layers to obtain spatial features of the weighted brain area connection map extracted from local to global perspectives; and inputting the spatial features into the classifier after processing them through the fully connected layer to obtain a decoding result for the target task; Among them, the EEG signal features of each node and the static brain functional connection matrix are processed through the functional connection attention mechanism to obtain a weighted brain area connection map, including: inputting the static brain functional connection matrix and the EEG signal features of each node into multiple attention heads respectively, each attention head calculates the attention score between each node pair, and weights each node pair according to the corresponding attention score; the weighted results of multiple attention heads are aggregated to obtain a weighted brain area connection map.

2. The method for decoding EEG signals according to claim 1, wherein: The step of gradually processing the weighted brain area connection map through multiple layers of convolutional layers to obtain spatial features of the weighted brain area connection map extracted gradually from local to global perspectives includes: Inputting the weighted brain region connection map into the first convolutional layer to extract initial local features between each node and its adjacent nodes; Inputting the initial local features into the second convolutional layer, the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer in sequence to extract intermediate local features between each of the nodes and its multi-level neighboring nodes; The intermediate local features are input into the sixth convolutional layer to extract the global features of the nodes.

3. The EEG signal decoding method according to claim 1, wherein: If the paradigm is a bimanual motor execution task paradigm or a unimanual motor execution task paradigm, then the paradigm-specific static brain functional connectivity matrix is constructed based on the functional connectivity relationship between every two nodes in the EEG signal, including: Calculating the functional connection strength between every two nodes in the EEG signal by using the Pearson correlation coefficient algorithm; Functional connections whose functional connection strength is less than a preset strength threshold are excluded to construct a first target static brain functional connection matrix.

4. The method for decoding EEG signals according to claim 1, wherein: If the paradigm is a bimanual motor imagery task paradigm or a unimanual motor imagery task paradigm, then the construction of a paradigm-specific static brain functional connectivity matrix based on the functional connectivity relationship between every two nodes in the EEG signal includes: Calculating the functional connection strength between every two nodes in the EEG signal by a phase-locked value algorithm; Functional connections whose functional connection strength is less than a preset strength threshold are excluded to construct a second target static brain functional connection matrix.

5. An electroencephalogram signal decoding device, characterized in that: include: A static brain functional connectivity matrix construction module is used to construct a paradigm-specific static brain functional connectivity matrix based on the functional connectivity relationship between every two nodes in the EEG signal; A weighted brain area connection map acquisition module is used to process the EEG signal features of each node and the static brain functional connection matrix through a functional connection attention mechanism to obtain a weighted brain area connection map; A decoding result acquisition module is used to input the weighted brain area connection map into a graph convolutional neural network to obtain a decoding result; wherein the graph convolutional neural network includes multiple convolutional layers, a fully connected layer and a classifier; the multiple convolutional layers of the graph convolutional neural network are respectively constructed by Chebyshev polynomials with different convolution kernel orders; the orders of the convolution kernels of the Chebyshev polynomials of the multiple convolutional layers increase successively; the number of convolutional layers is six; the first convolutional layer is a Chebyshev polynomial with a first-order convolution kernel; the second convolutional layer, the third convolutional layer, the fourth convolutional layer and the fifth convolutional layer are Chebyshev polynomials with a second-order convolution kernel; the sixth convolutional layer is a Chebyshev polynomial with a fourth-order convolution kernel; Inputting the weighted brain area connection map into a graph convolutional neural network to obtain a decoding result includes: gradually processing the weighted brain area connection map through multiple layers of the convolutional layers to obtain spatial features of the weighted brain area connection map extracted from local to global perspectives; and inputting the spatial features into the classifier after processing them through the fully connected layer to obtain a decoding result for the target task; Among them, the EEG signal features of each node and the static brain functional connection matrix are processed through the functional connection attention mechanism to obtain a weighted brain area connection map, including: inputting the static brain functional connection matrix and the EEG signal features of each node into multiple attention heads respectively, each attention head calculates the attention score between each node pair, and weights each node pair according to the corresponding attention score; the weighted results of multiple attention heads are aggregated to obtain a weighted brain area connection map.

6. A terminal device, characterized in that: The terminal device includes a processor and a memory, the memory stores a computer program, and the processor is configured to execute the computer program to implement the EEG signal decoding method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The device stores a computer program, which, when executed on a processor, implements the electroencephalogram signal decoding method according to any one of claims 1 to 4.

Images