An inner speech decoding method based on brain network constraints and graph contrast learning

Abstract

The present invention provides a method and device for inner speech decoding based on brain network constraints and graph contrastive learning, relating to the technical field of intelligent EEG signal decoding. The method comprises: obtaining an EEG signal sample dataset; constructing an initial graph contrastive learning model constrained by the spatial functions of the brain network; the model comprising a graph-based EEG representation construction module, a dual-stream graph contrastive learning module constrained by the brain network, and a spatial function fusion module; training the model based on the EEG signal sample dataset to obtain a trained graph contrastive learning model constrained by the spatial functions of the brain network; obtaining EEG signal data to be decoded; and inputting the EEG signal data to be decoded into the trained graph contrastive learning model constrained by the spatial functions of the brain network to obtain the final inner speech decoding result. The present invention can improve decoding performance.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent decoding of electroencephalogram (EEG) signals, and in particular to a method and device for decoding inner speech based on brain network constraints and graph contrast learning. Background Art

[0002] Inner speech (IS), also known as inner monologue, is the internalized process of a person's thoughts in pure meaning. It is an important and common method of expressing intentions and plays a significant role in human cognition. Brain-computer interfaces (BCIs), a promising technology field, can establish information exchange pathways between the brain and the external environment by collecting and decoding electroencephalogram (EEG) signals from brain neural activity. Compared with existing BCI paradigms, such as P300, SSVEP, or motor imagery, IS-BCI is easier for subjects to use and is more simple and intuitive, making it more applicable in real-life and long-term application scenarios.

[0003] Current EEG decoding methods for inner speech primarily employ deep learning to decode inner speech (IS) and contrastive learning-based EEG decoding methods. These deep learning methods include: using CNNs to capture spatiotemporal dependencies in EEG signals at multiple scales; decoding nine words using convolutional and recurrent neural networks; building a BCI control system using CNNs that can detect and decode inner speech (IS); and using CNN-based capsule neural networks to decode distichous, nasals, vowels, and consonants. Due to the high time cost of collecting IS EEG data and the widespread individual variability of EEG data, model training faces challenges with insufficient data and overfitting. Contrastive learning-based EEG decoding methods include: data augmentation by adjusting the order of different slices in the EEG sample, which has applications in sleep staging and pathology detection; and data augmentation by generating subgraphs using a restarted random walk technique, which allows the generated samples to capture local structure and contextual information. However, current graph augmentation methods are all based on random mechanisms, which disrupt the inherent connectivity of brain networks, resulting in weak decoding performance. Summary of the Invention

[0004] To address the existing technical issues of insufficient data and overfitting in model training due to the high time cost of collecting IS EEG data and the widespread individual variability of EEG, as well as the poor decoding performance caused by the random mechanism of graph enhancement methods, the present invention provides a method and device for inner speech decoding based on brain network constraints and graph contrast learning. The technical solution is as follows:

[0005] In one aspect, a method for inner speech decoding based on brain network constraints and graph contrast learning is provided. The method is implemented by an inner speech decoding device based on brain network constraints and graph contrast learning, and the method comprises:

[0006] S1. Obtain EEG signal sample dataset;

[0007] S2. Constructing an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions comprises: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain networks, and a spatial function fusion module;

[0008] S3. Training the initial graph contrast learning model constrained by brain network spatial functions based on the EEG signal sample data set, performing local feature extraction using the graph-based EEG representation building module to obtain local features of nodes; performing network feature extraction using the graph-based EEG representation building module to obtain network features; combining the local features of the nodes with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation;

[0009] S4. Inputting the spatial-based EEG map representation into the dual-stream map contrast learning module constrained by the brain network for contrast learning to obtain a spatial-based EEG map representation after contrast learning; inputting the functional-based EEG map representation into the dual-stream map contrast learning module constrained by the brain network for contrast learning to obtain a functional-based EEG map representation after contrast learning;

[0010] S5. Inputting the spatial-based EEG graph representation after contrastive learning and the functional-based EEG graph representation after contrastive learning into the spatial-functional fusion module to obtain inner speech decoding prediction results; and obtaining a trained graph contrastive learning model constrained by the spatial function of the brain network based on the inner speech decoding prediction results;

[0011] S6. Obtain EEG signal data to be decoded; input the EEG signal data to be decoded into the trained graph contrast learning model constrained by brain network spatial function to obtain the final inner speech decoding result.

[0012] In another aspect, a device for inner speech decoding based on brain network constraints and graph contrast learning is provided. The device is applied to the inner speech decoding method based on brain network constraints and graph contrast learning. The device comprises:

[0013] An acquisition unit, used for acquiring an EEG signal sample data set;

[0014] A construction unit, configured to construct an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions comprises: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain networks, and a spatial function fusion module;

[0015] a training unit for training the initial graph contrast learning model constrained by the spatial function of the brain network according to the EEG signal sample data set, extracting local features through the graph-based EEG representation construction module to obtain local features of the nodes; extracting network features through the graph-based EEG representation construction module to obtain network features; combining the local features of the nodes with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation; inputting the space-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning to obtain a space-based EEG graph representation after contrast learning; inputting the function-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning to obtain a function-based EEG graph representation after contrast learning; inputting the space-based EEG graph representation after contrast learning and the function-based EEG graph representation after contrast learning into the spatial function fusion module to obtain an internal speech decoding prediction result; and obtaining a trained graph contrast learning model constrained by the spatial function of the brain network according to the internal speech decoding prediction result;

[0016] The decoding unit is used to obtain EEG signal data to be decoded; input the EEG signal data to be decoded into the trained graph contrast learning model constrained by the brain network spatial function to obtain the final inner speech decoding result.

[0017] On the other hand, an inner speech decoding device based on brain network constraints and graph contrast learning is provided, and the inner speech decoding device based on brain network constraints and graph contrast learning comprises: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the inner speech decoding methods based on brain network constraints and graph contrast learning is implemented.

[0018] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned inner speech decoding methods based on brain network constraints and graph contrast learning.

[0019] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0020] The embodiment of the present invention first obtains an EEG signal sample data set; constructs an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions includes: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain network, and a spatial function fusion module; secondly, the initial graph contrast learning model constrained by brain network spatial functions is trained based on the EEG signal sample data set, and local features are extracted by the graph-based EEG representation construction module to obtain local features of nodes; network features are extracted by the graph-based EEG representation construction module to obtain network features; the local features of the nodes are combined with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation; The spatial-based EEG map representation and the function-based EEG map representation are input into the dual-stream graph contrastive learning module constrained by the brain network for contrastive learning, thereby obtaining the spatial-based EEG map representation and the function-based EEG map representation after contrastive learning; the spatial-based EEG map representation and the function-based EEG map representation after contrastive learning are input into the spatial-functional fusion module to obtain the inner speech decoding prediction result; finally, based on the inner speech decoding prediction result, a trained graph contrastive learning model constrained by the spatial function of the brain network is obtained; the EEG signal data to be decoded is obtained; the EEG signal data to be decoded is input into the trained graph contrastive learning model constrained by the spatial function of the brain network to obtain the final inner speech decoding result.

[0021] The embodiments of the present invention introduce prior knowledge of brain networks into the feasibility of decoding inner speech. The embodiments of the present invention can comprehensively learn robust representations from functional and spatial perspectives. To comprehensively represent the brain state related to inner speech, the embodiments of the present invention design a graph-based EEG representation construction module to learn local and network features. To alleviate the impact of data scarcity, the embodiments of the present invention design a dual-stream graph contrast learning module constrained by brain networks to generate more brain-like enhanced samples. By designing a superposition mechanism to fuse the graph representations from the spatial stream and the functional stream, the decoding performance is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 Schematic diagram of a graph comparison learning method constrained by brain network spatial function provided by an embodiment of the present invention;

[0024] Figure 2 This is a flow chart of an inner speech decoding method based on brain network constraints and graph contrast learning provided by an embodiment of the present invention;

[0025] Figure 3 It is a schematic diagram of the structure of the space-function fusion module provided by an embodiment of the present invention;

[0026] Figure 4 This is a visual brain topography map provided by an embodiment of the present invention;

[0027] Figure 5 This is a block diagram of an inner speech decoding device based on brain network constraints and graph contrast learning provided by an embodiment of the present invention;

[0028] Figure 6 This is a structural diagram of an inner speech decoding device based on brain network constraints and graph contrast learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0030] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0031] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0032] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0033] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0034] The embodiment of the present invention provides an inner speech decoding method based on brain network constraints and graph contrast learning. The method can be implemented by an inner speech decoding device based on brain network constraints and graph contrast learning. The inner speech decoding device based on brain network constraints and graph contrast learning can be a terminal or a server. Figure 1 is a structural diagram of a graph contrast learning method constrained by brain network space function provided by an embodiment of the present invention, such as Figure 2 The flowchart of the inner speech decoding method based on brain network constraint and graph contrast learning is shown. The processing flow of the method may include the following steps:

[0035] S1. Obtain an EEG signal sample dataset.

[0036] Among them, EEG (Electroencephalography) signals are a way to measure brain activity. They record the electrical activity of the cerebral cortex by placing electrodes on the scalp. EEG signals reflect the synchronous discharge of neurons and can provide real-time information about the functional state of the brain.

[0037] S2. Construct an initial graph contrast learning model constrained by the spatial functions of the brain network; the initial graph contrast learning model constrained by the spatial functions of the brain network includes: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by the brain network, and a spatial function fusion module.

[0038] Optionally, the brain network constrained dual-stream graph contrast learning module of S2 includes: a brain network constrained graph enhancement module, an encoder based on a graph isomorphism network, and a projection layer.

[0039] Among them, the dual-stream graph contrast learning module constrained by the brain network also includes: a loss function.

[0040] Optionally, the spatial function fusion module of S2 includes: two basic classifiers and a meta-classifier.

[0041] S3. The initial graph contrast learning model constrained by the spatial function of the brain network is trained based on the EEG signal sample dataset. Local features are extracted through the graph-based EEG representation construction module to obtain local features of the nodes; network features are extracted through the graph-based EEG representation construction module to obtain network features; the local features of the nodes are combined with the network features to obtain space-based EEG graph representation and function-based EEG graph representation.

[0042] Optionally, the specific implementation process of S3 may include S31-S34:

[0043] S31. Based on the concept of graph, the EEG information is lightweight represented in the form of a graph structure; according to a preset number of frames, the electrode signals in the EEG signal sample data set are divided into multiple frames, and the signal power of each frequency band is calculated according to a preset number of frequency bands to obtain power characteristics of multiple EEG electrodes;

[0044] In a feasible implementation, in order to enable the graph contrastive learning model constrained by the spatial function of the brain network to fully represent the local information of the electrodes and the global brain network topology information while reducing the burden of model training as much as possible, the embodiment of the present invention proposes a graph-based brain representation construction module to obtain the EEG representation of the EEG signal sample data set; in order to avoid the subsequent contrastive learning model from having too many parameters to be trained, which may lead to overfitting of the model, the embodiment of the present invention is based on the concept of graph. , the EEG information is expressed in a lightweight form in the form of a graph structure, where Feature representation of the representation graph; Indicates inclusion A collection of node features, each node corresponds to an electrode one-to-one; Represents the adjacency matrix of the graph; A set of edges representing the existence of spatial or functional connections between electrodes, containing key network features of the brain.

[0045] S32. Based on the power features of multiple EEG electrodes, a one-dimensional convolutional layer is used to shrink the power features of multiple EEG electrodes to a one-dimensional vector, and output local feature representations of multiple nodes; wherein the local feature representation of the node can be obtained by the following formula (1):

[0046] (1)

[0047] in, For the Local feature representation of nodes; Indicates the The power characteristics of the EEG electrodes, , F represents the number of frequency bands, T represents the number of frames; R represents a set of real numbers; represents the weight matrix, This is a hyperparameter representing the size of the convolution kernel, and the value set is the same as the stride; Represents the bias vector to be trained; represents the linear rectification function.

[0048] Optionally, S3 extracts network features through a graph-based EEG representation building module to obtain network features; combines local features of nodes with network features to obtain space-based EEG graph representation and function-based EEG graph representation, including:

[0049] Among them, since there are a lot of duplications in the third-order neighbors of each node, its own characteristic information will be lost and the same eigenvalue will be obtained, which will lead to the over-smoothing problem. In order to avoid the above situation and further reduce the burden of model training, when constructing EEG representation based on the graph, the edges of the graph are sparsely operated to make it lightweight.

[0050] S33. According to the graph structure representation, a spatial connectivity matrix is ​​calculated based on a predefined spatial relationship; a random initial first trainable mask matrix is ​​used to perform adjacency matrix sparsification on the spatial connectivity matrix to obtain a space-based adjacency matrix; a functional connectivity matrix is ​​calculated based on the Pearson correlation coefficient, and an adjacency matrix sparsification is performed on the functional connectivity matrix using a random initial second trainable mask matrix to obtain a function-based adjacency matrix; wherein the process of obtaining the space-based adjacency matrix can be expressed by the following formula (2):

[0051] (2)

[0052] in, Represents a space-based adjacency matrix; represents the angle between the p-th electrode and the q-th electrode; represents the p-th electrode and the q-th electrode; represents the cosine function of the angle between the pth electrode and the qth electrode; represents the first trainable mask matrix randomly initialized; p and q represent the number of electrodes;

[0053] The process of obtaining the function-based adjacency matrix can be expressed by the following formula (3):

[0054] (3)

[0055] in, represents the EEG signal sequence of the pth electrode; represents the EEG signal sequence of the qth electrode; represents the standard deviation of the EEG signal sequence of the pth electrode; represents the standard deviation of the EEG signal sequence of the qth electrode; represents the covariance between the EEG signal sequence of the p-th electrode and the EEG signal sequence of the q-th electrode; represents the randomly initialized second trainable mask matrix; , The value range is (-1,1), The value range is (-1, 1) and does not exceed r%, and the value of the element r∈(0, 100) is greater than 0; r represents a pre-set hyperparameter used to control the sparsity of the adjacency matrix; N represents the total number of EEG electrodes.

[0056] Among them, the space-based EEG graph representation can be expressed as ; The function-based EEG graph representation can be expressed as .

[0057] The predefined spatial relationship may be defined as a ratio of a cosine value of an angle between two electrodes and a Euclidean distance.

[0058] The Pearson correlation coefficient is a conventional technical means, and the embodiment of the present invention does not elaborate on the calculation process of the Pearson correlation coefficient.

[0059] S34. Based on the local feature representation of multiple nodes and the space-based adjacency matrix, a set of edges based on spatial connections between nodes are obtained; the local feature representation of multiple nodes, the space-based adjacency matrix and the set of edges based on spatial connections between nodes are combined to obtain a space-based EEG map representation; based on the local feature representation of multiple nodes and the function-based adjacency matrix, a set of edges based on functional connections between nodes are obtained; the local feature representation of multiple nodes, the function-based adjacency matrix and the set of edges based on functional connections between nodes are combined to obtain a function-based EEG map representation.

[0060] In a feasible implementation, the edges in the graph are determined by nodes and an adjacency matrix based on the representation of the graph structure; wherein the adjacency matrix is ​​a two-dimensional array, wherein the rows and columns represent the nodes in the graph, and when an element in the matrix is ​​non-zero, there is an edge between the nodes; based on the local features of multiple nodes and the space-based adjacency matrix, a set of edges of spatial connections between nodes are obtained, which contain key network features of the brain; based on the local features of multiple nodes and the function-based adjacency matrix, a set of edges of functional connections between nodes are obtained.

[0061] S4. Input the space-based EEG map representation into the dual-stream map contrast learning module constrained by the brain network for contrast learning, and obtain the space-based EEG map representation after contrast learning; input the function-based EEG map representation into the dual-stream map contrast learning module constrained by the brain network for contrast learning, and obtain the function-based EEG map representation after contrast learning.

[0062] Among them, the brain network is a complex network composed of spatial structure and functional organization; the physical connections of the brain network in space are non-random, mainly manifested as a small-world architecture, including highly localized clustering and short path length. Based on this brain network characteristic as a constraint, when performing comparative learning on spatial branches, the connection relationships that are closer in spatial positions should be retained with a greater probability to avoid destroying the small-world architecture of spatial connectivity during data enhancement.

[0063] Optionally, the specific implementation process of S4 may include S41-S43:

[0064] S41. Input the spatial-based EEG graph representation into the graph enhancement module constrained by the brain network. Using the brain network characteristics as the constraint condition, the graph data of the spatial branch is enhanced to obtain the EEG graph representation based on spatial enhancement. The process of enhancing the graph data of the spatial branch can be expressed by the following formula (4):

[0065] (4)

[0066] in, Represents the adjacency matrix based on spatial enhancement; express The absolute values ​​of the non-zero elements in the array are sorted at the top. The element set of , k represents the hyperparameter that controls the proportion of the elements retained in the original adjacency matrix; Represents Random number matrices of the same size, with elements ranging from 0 to 1; Indicates from A random selection from the zero elements in The edge set corresponding to the element;

[0067] Among them, according to The non-zero elements in the node and the edge Update to obtain the EEG map representation based on spatial enhancement, which can be expressed as .

[0068] S42. Input the function-based EEG graph representation into the graph enhancement module constrained by the brain network. Based on the brain network characteristics, the graph data of the functional branch is enhanced to obtain the EEG graph representation based on function enhancement. The process of enhancing the graph data of the functional branch can be expressed by the following formula (5):

[0069] (5)

[0070] in, Represents the adjacency matrix based on feature enhancement; Indicates from A random selection from the zero elements in The edge set corresponding to the element;

[0071] Among them, according to The non-zero elements in the node and the edge Update to obtain the EEG map representation based on functional enhancement, which can be expressed as .

[0072] Among them, there is a lot of overlap between the functional connectivity and spatial connectivity of the brain network, and neurons with spatial connections have stronger functional connectivity than unconnected neurons; using this brain network characteristic as a constraint, when conducting comparative learning on the functional branched brain network, the connections contained in the spatially enhanced graph representation are more likely to be retained, and new samples are generated on the premise of containing effective functional connections.

[0073] S43, inputting the space-based EEG graph representation, the function-based EEG graph representation, the space-enhanced EEG graph representation, and the function-enhanced EEG graph representation into the encoder and projection layer composed of the graph isomorphism network for comparative learning, and obtaining the graph representation of the EEG signal sample data set after comparative learning; wherein, the graph representation of the EEG signal sample data set after comparative learning includes: the space-based EEG graph representation after comparative learning and the function-based EEG graph representation after comparative learning

[0074] Among them, the spatial stream and the functional stream are compared and learned based on the same network structure, and the spatial EEG map is represented , Function-based EEG graph representation , EEG image representation based on spatial enhancement and EEG graph representation based on functional enhancement , forming four groups of image samples; in the comparative learning stage, using and The original samples and and Positive pairs are created by pairing each sample with other samples in the same set, and negative pairs are created by pairing each sample with other samples in the same set. The update rule for the node representation of each graph representation in the graph isomorphic network can be expressed by the following formula (6):

[0075] (6)

[0076] in, represents the updated node representation; Represents the node representation, represents the adjacency matrix of the EEG graph, represents the identity matrix, It is a learnable parameter that controls the influence of each node's own attributes. MLP stands for Multi-layer Perceptron model. Among them, the four groups of graph representations obtained by the encoder are subjected to graph pooling operation to obtain graph sample representation, which is then mapped to the projection layer consisting of two fully connected layers and one ReLU layer, also known as the projection head. The potential space in .

[0077] In a feasible implementation, the spatial flow loss and the functional flow loss are calculated separately according to the contrast loss function; the cosine similarity of the image representation before and after enhancement is calculated in the latent space to measure the enhancement effect of the image sample; the overall loss function is calculated according to the spatial flow loss and the functional flow loss to guide the simultaneous training of the image enhancement module constrained by the brain network; wherein the overall loss function can be expressed by the following formula (7):

[0078] (7)

[0079] Among them, cos represents cosine similarity, represents the spatial flow, Represents functional flow; Represents the overall loss function; Represents the loss function of the branch. When i=1, it is the loss function based on the spatial branch. When i=2, it is the loss function based on the functional branch.

[0080] Among them, the contrast loss function is a loss function commonly used in machine learning. A common form of contrast loss is the triple loss including: anchor point, positive sample and negative sample.

[0081] Among them, the cosine similarity of the graph representation before and after enhancement is calculated in the latent space to measure the enhancement effect of the graph sample and determine the influence of this branch on the overall loss function.

[0082] S5. Input the space-based EEG graph representation after contrastive learning and the function-based EEG graph representation after contrastive learning into the spatial function fusion module to obtain the inner speech decoding prediction result; based on the inner speech decoding prediction result, obtain a trained graph contrastive learning model constrained by the spatial function of the brain network.

[0083] Contrastive learning is an unsupervised learning method that learns by comparing different views or variants of data. The specific implementation process of contrastive learning may include: data preprocessing: First, preprocess the raw data, including normalization and denoising, to facilitate subsequent processing; contrastive learning requires paired data; in a positive pair, the two samples are similar, typically generated from the same raw data through data augmentation techniques including rotation, scaling, and cropping; in contrast, the two samples in a negative pair are dissimilar; they can be randomly selected from a dataset or generated through various data augmentation techniques; a neural network or other model is used to extract feature representations of the samples; the feature representations can capture the essential properties of the data; the neural network can be a CNN neural network model; a loss function is defined to train the model. The loss function typically encourages the model to bring the feature representations of the positive pair closer together while pushing the feature representations of the negative pair further apart; in this embodiment of the present invention, the overall loss function is calculated based on spatial and functional flow losses; the model parameters are updated through a backpropagation algorithm to minimize the loss function; and the above steps are repeated until the model achieves satisfactory performance on the contrastive task.

[0084] Among them, the spatial function fusion module can fuse the knowledge of spatial branches and functional branches, maximize the use of brain network information, and further improve the internal speech decoding performance; the spatial function fusion module can integrate the information of spatial brain networks and functional brain networks to achieve complementary advantages and further improve the internal speech decoding performance of the model; the embodiment of the present invention designs a classifier based on a stacking mechanism to fuse spatial streams and functional streams to generate the final decoding results.

[0085] In a feasible implementation, the embodiment of the present invention uses support vector machine (SVM) as the basic classifier for internal speech decoding, and allocates appropriate weights to spatial branches and functional branches through a stacking mechanism to promote the effective fusion of information from spatial streams and functional streams.

[0086] Among them, support vector machine (SVM) is a commonly used supervised learning algorithm that can be used for classification and regression problems. In model fusion technology, support vector machine (SVM) can be used as a base learner in the stacking mechanism, which is a commonly used ensemble learning method. The implementation process of the stacking mechanism may include:

[0087] (1) Selecting a set of base learners, which may be different types of models, including decision trees, neural networks, and SVMs; in the embodiment of the present invention, SVM is selected;

[0088] (2) Use K-fold cross-validation to train the base learner and generate prediction results for the training set and test set;

[0089] (3) The prediction results are used as new features and combined with the original features to form a new feature matrix;

[0090] (4) Use a meta-learner, which can be an SVM or other model, to train on the new feature matrix;

[0091] (5) On the test set, use the trained base learner to make predictions, and use the prediction results as the input of the meta-learner to obtain the final prediction.

[0092] Optionally, the specific implementation steps of S5 may include S51-S55:

[0093] S51, inputting the graph representation of the EEG signal sample data set after contrastive learning into the spatial function fusion module, clustering the graph representation of the spatial branches using the K-means algorithm to obtain graph representation cluster centers based on the spatial branches and clusters based on the spatial clustering, and clustering the graph representation of the functional branches using the K-means algorithm to obtain graph representation cluster centers based on the functional branches and clusters based on the functional clustering;

[0094] In a feasible implementation, Figure 3 FIG. 1 is a schematic diagram of the structure of the space-function fusion module provided by an embodiment of the present invention. The K-means algorithm is used to cluster the graph representation of the spatial branch and the graph representation of the functional branch, respectively, to obtain the clustering center of the graph representation based on the spatial branch and the clusters after spatial clustering, as well as the clustering center of the graph representation based on the functional branch and the clusters after functional clustering. The clustering process can be expressed by the following formula (8):

[0095] (8)

[0096] in, The representation graph represents the cluster centers; Represents the cluster after clustering; Represents the projection layer; represents the encoder; Node representation of EEG graph; Represents the adjacency matrix of the EEG graph; represents the number of clusters, which is consistent with the number of internal speech instructions to be decoded; i represents the branch subscript, where i=1 represents the spatial branch and i=2 represents the functional branch;

[0097] S52. Calculating a space-based fitness score based on the spatial branch-based graph representation cluster center;

[0098] The cluster centers are represented by a graph based on functional branches, and the function-based fitness scores are calculated;

[0099] In a feasible implementation, the fitness score calculation formula can be expressed by the following formula (9):

[0100] (9)

[0101] in, Indicates the The first branch clusters, ; express A sample image in Indicates the The first branch the center of a cluster; represents the fitness score, when It indicates the fitness score based on space; represents the fitness score based on the function.

[0102] In a feasible implementation, the weight of the space-based branch is calculated according to the space-based fitness score; the weight of the function-based branch is calculated according to the function-based fitness score. The calculation process of the weight of the space-based branch and the weight of the function-based branch can be expressed by the following formula (10):

[0103] (10)

[0104] in, represents the fitness score of the spatial branch; Represents the fitness score of the functional branch; Indicates the weight of the output result of the branch basic classifier in the meta-classifier, The value range is 1 or 2, where i=1 represents the spatial branch and i=2 represents the functional branch.

[0105] S53, inputting the clusters after spatial clustering into the first basic classifier, and outputting a spatial decoding result; using the weights based on the spatial branches, performing weighted processing on the spatial decoding result to obtain a weighted spatial decoding result;

[0106] S54, inputting the clusters after function-based clustering into the second basic classifier, and outputting the function-based decoding results; using the weights based on the function branches, performing weighted processing on the function-based decoding results to obtain weighted function-based decoding results;

[0107] S55. Input the weighted spatial-based decoding result and the weighted functional-based decoding result into a meta-classifier to obtain an inner speech decoding prediction result.

[0108] S6. Obtain EEG signal data to be decoded; input the EEG signal data to be decoded into a trained graph contrast learning model constrained by the spatial function of the brain network to obtain the final inner speech decoding result.

[0109] In a feasible implementation, the embodiment of the present invention is compared with five currently advanced methods; wherein, the experimental data is the BCI-Competition V-3, BCI V3 dataset. The BCI V3 dataset collects EEG data when subjects imagine daily-related phrases / vocabulary, which may include: hello, help me, stop, thank you and yes. The embodiment of the present invention needs to complete the five categories of IS on the BCI V3 dataset. The BCI V3 dataset requires subjects to imagine that they are performing real speech movements, but without moving any vocal organs or making any sounds. The internal speech inducements of the BCI V3 dataset involve both language function and motor planning. 15 healthy subjects aged 20-30 years participated in the experiment. 64 EEG channels were recorded at a sampling rate of 256Hz.

[0110] Among them, Thinking out loud (TOL) collects EEG data when subjects produce direction-related words, including three language tasks: inner speech, vocalized speech, and visual state. The data of the inner speech task is used in the comparative experiment. The embodiment of the present invention needs to complete the four-category IS on the TOL dataset. The TOL dataset requires subjects to imagine their own voice issuing instructions and repeating them continuously, and explicitly requires them not to focus on imagining the movements of their mouth and tongue, but to focus on inducing inner speech only through high-level language functions. Ten healthy subjects participated in the experiment with an average age of 34 years and a standard deviation of 10 years. 128 EEG channels and 8 EOG / EMG channels were recorded at a sampling rate of 1024Hz.

[0111] In a feasible implementation, the five currently advanced methods include: EEGNet, CNN, deep metric learning CNN, multi-scale neural network and classification principal component analysis; the comparison methods used include: machine learning, deep learning based on representation learning and deep neural network model; as shown in Figure 1, the results of the BCI Competition V-3 dataset.

[0112] Table 1

[0113]

[0114] Among them, the embodiment of the present invention can naturally preserve the EEG topology through graph representation learning, and obtain a more robust EEG representation through unsupervised contrastive learning, further achieving optimal decoding performance and stability.

[0115] Among them, the performance of five state-of-the-art methods applied to the TOL dataset was compared, including: EEGNet, bidirectional long short-term memory network, CNN, SVM and subject-independent meta-learning; the compared methods included machine learning methods, traditional deep learning models and meta-learning frameworks; the results of the Thinking Out Loud dataset are shown in Table 2.

[0116] Table 2

[0117]

[0118] Compared to the BCI V3 dataset, the decoding performance of the TOL dataset significantly decreased. This is because the TOL dataset explicitly requires subjects to focus on the IS content rather than the motor imagery of the vocal organs. The embodiments of the present invention construct EEG representations that more concisely represent local and network EEG information, mitigating the negative impact of small sample sizes. They also introduce functional connectivity and spatial structure to more comprehensively characterize brain network topology, resulting in improved optimal decoding performance and stability compared to existing methods.

[0119] Among them, Figure 4 An embodiment of the present invention provides a visualized brain topography map; a visual analysis is performed based on the visualized brain topography map; the brain topography map is used to visualize which brain regions are more important in IS decoding, providing interpretability for the proposed model; and the node representation corresponding to each electrode learned by the embodiment of the present invention is correlated with the internal speech marker to obtain a brain topography map for each subject. Figure 4The average correlation across all subjects is shown. Key brain regions for the BCI V3 dataset are the frontal lobe (①) and the left temporal lobe (②) near the central region; the corresponding key brain region for the TOL is the temporal lobe (③). This discrepancy arises from different methods of inner speech invocation in the two datasets. The premotor area, corresponding to ①, is one of the brain regions most associated with motor imagery and is involved in motor planning and preparation. The inferior frontal gyrus, corresponding to ②, contains the motor speech center, responsible for generating coordinated articulatory programs. Because the BCI V3 dataset requires subjects to imagine performing real speech actions without actually moving their vocal organs, language-related motor imagery is involved. Therefore, ① and ② are important for decoding inner speech in the BCI V3 dataset. The temporal lobe, where ③ is located, plays a key role in speech perception and processing. The TOL dataset requires subjects to imagine themselves issuing commands, assuming that the human brain may process silent speech information in a manner similar to that of external speech. We identified region ④ in the occipital lobe as a common key brain region for inner speech decoding. The occipital lobe is an intermediate hub between visual information and language processing. It is reasonable to assume that generating inner speech automatically associates it with visual imagery, such as related words or scenes. Thus, BSFG can identify key brain regions associated with IS, including both dataset-specific and common regions, showing generalization capabilities across different triggers, i.e., inner speech decoding relying solely on language functions or including motor planning.

[0120] The embodiment of the present invention first obtains an EEG signal sample data set; constructs an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions includes: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain network, and a spatial function fusion module; secondly, the initial graph contrast learning model constrained by brain network spatial functions is trained based on the EEG signal sample data set, and local features are extracted by the graph-based EEG representation construction module to obtain local features of nodes; network features are extracted by the graph-based EEG representation construction module to obtain network features; the local features of the nodes are combined with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation; The spatial-based EEG map representation and the function-based EEG map representation are input into the dual-stream graph contrastive learning module constrained by the brain network for contrastive learning, thereby obtaining the spatial-based EEG map representation and the function-based EEG map representation after contrastive learning; the spatial-based EEG map representation and the function-based EEG map representation after contrastive learning are input into the spatial-functional fusion module to obtain the inner speech decoding prediction result; finally, based on the inner speech decoding prediction result, a trained graph contrastive learning model constrained by the spatial function of the brain network is obtained; the EEG signal data to be decoded is obtained; the EEG signal data to be decoded is input into the trained graph contrastive learning model constrained by the spatial function of the brain network to obtain the final inner speech decoding result.

[0121] The embodiments of the present invention introduce prior knowledge of brain networks into the feasibility of decoding inner speech. The embodiments of the present invention can comprehensively learn robust representations from functional and spatial perspectives. To comprehensively represent the brain state related to inner speech, the embodiments of the present invention design a graph-based EEG representation construction module to learn local and network features. To alleviate the impact of data scarcity, the embodiments of the present invention design a dual-stream graph contrast learning module constrained by brain networks to generate more brain-like enhanced samples. By designing a superposition mechanism to fuse the graph representation information from the spatial stream and the functional stream, the decoding performance is improved.

[0122] Figure 5 This is a block diagram of an inner speech decoding device based on brain network constraints and graph contrast learning according to an exemplary embodiment. The device is used in an inner speech decoding method based on brain network constraints and graph contrast learning. Figure 5 The device includes an acquisition unit 310, a construction unit 320, a training unit 330, and a decoding unit 340.

[0123] An acquisition unit 310 is configured to acquire an EEG signal sample data set;

[0124] A construction unit 320 is configured to construct an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions comprises: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain networks, and a spatial function fusion module;

[0125] The training unit 330 is configured to train the initial graph contrast learning model constrained by the spatial function of the brain network according to the EEG signal sample data set, extract local features through the graph-based EEG representation construction module to obtain local features of the nodes; extract network features through the graph-based EEG representation construction module to obtain network features; combine the local features of the nodes with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation; input the space-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning to obtain a space-based EEG graph representation after contrast learning; input the function-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning to obtain a function-based EEG graph representation after contrast learning; input the space-based EEG graph representation after contrast learning and the function-based EEG graph representation after contrast learning into the spatial function fusion module to obtain an inner speech decoding prediction result; and obtain a trained graph contrast learning model constrained by the spatial function of the brain network according to the inner speech decoding prediction result.

[0126] The decoding unit 340 is used to obtain EEG signal data to be decoded; input the EEG signal data to be decoded into the trained graph contrast learning model constrained by the brain network spatial function to obtain the final inner speech decoding result.

[0127] Optionally, extracting local features using the graph-based EEG representation building module to obtain local features of nodes includes:

[0128] Based on the concept of graph, EEG information is lightweight represented in the form of a graph structure; according to the preset number of frames, the electrode signals in the EEG signal sample data set are divided into multiple frames, and the signal power of each frequency band is calculated according to the preset number of frequency bands to obtain the power characteristics of multiple EEG electrodes;

[0129] According to the power features of the multiple EEG electrodes, a one-dimensional convolution layer is used to shrink the power features of the multiple EEG electrodes to a one-dimensional vector, and output local feature representations of multiple nodes; wherein the local feature representation of the node can be obtained by the following formula (1):

[0130] (1)

[0131] in, For the Local feature representation of nodes; Indicates the The power characteristics of the EEG electrodes, , F represents the number of frequency bands, T represents the number of frames; R represents a set of real numbers; represents the weight matrix, This is a hyperparameter representing the size of the convolution kernel, and the value set is the same as the stride; Represents the bias vector to be trained; represents the linear rectification function.

[0132] Optionally, extracting network features through the graph-based EEG representation building module to obtain network features; combining the local features of the nodes with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation, includes:

[0133] According to the graph structure representation, a spatial connectivity matrix is ​​calculated based on a predefined spatial relationship; a random initial first trainable mask matrix is ​​used to perform adjacency matrix sparsification on the spatial connectivity matrix to obtain a space-based adjacency matrix; a functional connectivity matrix is ​​calculated based on the Pearson correlation coefficient, and a random initial second trainable mask matrix is ​​used to perform adjacency matrix sparsification on the functional connectivity matrix to obtain a function-based adjacency matrix; wherein the process of obtaining the space-based adjacency matrix can be expressed by the following formula (2):

[0134] (2)

[0135] in, Represents a space-based adjacency matrix; represents the angle between the p-th electrode and the q-th electrode; represents the Euclidean distance between the p-th electrode and the q-th electrode; represents the cosine function of the angle between the pth electrode and the qth electrode; represents the first trainable mask matrix randomly initialized;

[0136] The process of obtaining the function-based adjacency matrix can be expressed by the following formula (3):

[0137] (3)

[0138] in, Represents a function-based adjacency matrix; represents the EEG signal sequence of the pth electrode; represents the EEG signal sequence of the qth electrode; represents the standard deviation of the EEG signal sequence of the pth electrode; represents the standard deviation of the EEG signal sequence of the qth electrode; represents the covariance between the EEG signal sequence of the p-th electrode and the EEG signal sequence of the q-th electrode; represents the randomly initialized second trainable mask matrix; , The value range is (-1,1), The value range is (-1, 1) and does not exceed r%, and the value of the element r∈(0, 100) is greater than 0; r represents a pre-set hyperparameter used to control the sparsity of the adjacency matrix; N represents the total number of EEG electrodes.

[0139] According to the local feature representations of multiple nodes and the space-based adjacency matrix, a group of edges based on spatial connections between nodes are obtained; the local feature representations of multiple nodes, the space-based adjacency matrix and the group of edges based on spatial connections between nodes are combined to obtain a space-based EEG map representation; according to the local feature representations of multiple nodes and the function-based adjacency matrix, a group of edges based on functional connections between nodes are obtained; the local feature representations of multiple nodes, the function-based adjacency matrix and the group of edges based on functional connections between nodes are combined to obtain a function-based EEG map representation.

[0140] Optionally, the dual-stream graph contrast learning module constrained by the brain network includes: a graph enhancement module constrained by the brain network, an encoder based on a graph isomorphism network, and a projection layer.

[0141] Optionally, the spatial-based EEG map representation is input into the dual-stream map contrast learning module constrained by the brain network for contrast learning to obtain a spatial-based EEG map representation after contrast learning; the functional-based EEG map representation is input into the dual-stream map contrast learning module constrained by the brain network for contrast learning to obtain a functional-based EEG map representation after contrast learning, including:

[0142] The spatial-based EEG graph representation is input into the graph enhancement module constrained by the brain network. The graph data of the spatial branch is enhanced with the brain network characteristics as the constraint condition to obtain the EEG graph representation based on spatial enhancement. The process of enhancing the graph data of the spatial branch can be expressed by the following formula (4):

[0143] (4)

[0144] in, Represents the adjacency matrix based on spatial enhancement; express The absolute values ​​of the non-zero elements in the array are sorted at the top. The element set of , k represents the hyperparameter that controls the proportion of the elements retained in the original adjacency matrix; Represents Random number matrices of the same size, with elements ranging from 0 to 1; Indicates from A random selection from the zero elements in The edge set corresponding to the element;

[0145] In the function-based EEG graph representation and the graph enhancement module constrained by the brain network, the graph data of the functional branch is enhanced with the brain network characteristics as the constraint to obtain the EEG graph representation based on function enhancement. The process of enhancing the graph data of the functional branch can be expressed by the following formula (5):

[0146] (5)

[0147] in, Represents the adjacency matrix based on feature enhancement; Indicates from A random selection from the zero elements in The edge set corresponding to the element;

[0148] The space-based EEG graph representation, the function-based EEG graph representation, the space-enhanced EEG graph representation, and the function-enhanced EEG graph representation are input into the encoder composed of the graph isomorphism network and the projection layer for comparative learning to obtain a graph representation of the EEG signal sample data set after comparative learning; wherein, the graph representation of the EEG signal sample data set after comparative learning includes: the space-based EEG graph representation after comparative learning and the function-based EEG graph representation after comparative learning, wherein, the update rule of the node representation of each graph representation in the graph isomorphism network can be expressed by the following formula (6):

[0149] (6)

[0150] in, represents the updated node representation; Represents the node representation, represents the adjacency matrix of the EEG graph, represents the identity matrix, It is a learnable parameter that controls the influence of each node's own attributes. MLP represents the multi-layer perceptron model.

[0151] Optionally, the spatial function fusion module includes: two basic classifiers and one meta-classifier.

[0152] Optionally, inputting the spatial-based EEG map representation and the functional-based EEG map representation into the spatial-functional fusion module to obtain an inner speech decoding prediction result includes:

[0153] The graph representation of the EEG signal sample data set after contrastive learning is input into the spatial function fusion module, and the K-means algorithm is used to cluster the graph representation of the spatial branch to obtain the graph representation cluster center based on the spatial branch and the clusters based on the spatial clustering. The K-means algorithm is used to cluster the graph representation of the functional branch to obtain the graph representation cluster center based on the functional branch and the clusters based on the functional clustering. The clustering process can be expressed by the following formula (7):

[0154] (7)

[0155] in, The representation graph represents the cluster centers; Represents the cluster after clustering; Represents the projection layer; represents the encoder; Node representation of EEG graph; Represents the adjacency matrix of the EEG graph; represents the number of clusters, which is consistent with the number of internal speech instructions to be decoded; i represents the branch subscript, where i=1 represents the spatial branch and i=2 represents the functional branch;

[0156] According to the graph based on spatial branches, cluster centers are represented and the space-based fitness score is calculated. According to the graph based on functional branches, cluster centers are represented and the function-based fitness score is calculated. The fitness score calculation formula can be expressed by the following formula (8):

[0157] (8)

[0158] in, Indicates the The first branch clusters, ; express A sample image in Indicates the The first branch the center of a cluster; represents the fitness score, when It indicates the fitness score based on space; When represents the fitness score based on function;

[0159] Among them, the weight of the space-based branch is calculated according to the space-based fitness score; the weight of the function-based branch is calculated according to the function-based fitness score. The calculation process of the weight based on the space branch and the weight based on the function branch can be expressed by the following formula (9):

[0160] (9)

[0161] in, represents the fitness score of the spatial branch; Represents the fitness score of the functional branch;

[0162] Indicates the weight of the output result of the branch basic classifier in the meta-classifier, The value range of is 1 or 2, where i=1 represents spatial branch and i=2 represents functional branch;

[0163] Inputting the clusters after spatial clustering into the first basic classifier, and outputting spatial decoding results; using weights based on spatial branches, performing weighted processing on the spatial decoding results to obtain weighted spatial decoding results;

[0164] Inputting the clusters obtained by function clustering into a second basic classifier and outputting a function-based decoding result; performing weighted processing on the function-based decoding result using weights based on function branches to obtain a weighted function-based decoding result;

[0165] The weighted spatial-based decoding results and the weighted functional-based decoding results are input into the meta-classifier to obtain the inner speech decoding prediction results.

[0166] The embodiment of the present invention first obtains an EEG signal sample data set; constructs an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions includes: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain network, and a spatial function fusion module; secondly, the initial graph contrast learning model constrained by brain network spatial functions is trained based on the EEG signal sample data set, and local features are extracted by the graph-based EEG representation construction module to obtain local features of nodes; network features are extracted by the graph-based EEG representation construction module to obtain network features; the local features of the nodes are combined with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation; The spatial-based EEG map representation and the function-based EEG map representation are input into the dual-stream graph contrastive learning module constrained by the brain network for contrastive learning, thereby obtaining the spatial-based EEG map representation and the function-based EEG map representation after contrastive learning; the spatial-based EEG map representation and the function-based EEG map representation after contrastive learning are input into the spatial-functional fusion module to obtain the inner speech decoding prediction result; finally, based on the inner speech decoding prediction result, a trained graph contrastive learning model constrained by the spatial function of the brain network is obtained; the EEG signal data to be decoded is obtained; the EEG signal data to be decoded is input into the trained graph contrastive learning model constrained by the spatial function of the brain network to obtain the final inner speech decoding result. The embodiments of the present invention introduce prior knowledge of brain networks into the feasibility of decoding inner speech. The embodiments of the present invention can comprehensively learn robust representations from functional and spatial perspectives. To comprehensively represent the brain state related to inner speech, the embodiments of the present invention design a graph-based EEG representation construction module to learn local and network features. To alleviate the impact of data scarcity, the embodiments of the present invention design a dual-stream graph contrast learning module constrained by brain networks to generate more brain-like enhanced samples. By designing a superposition mechanism to fuse the graph representation information from the spatial stream and the functional stream, the decoding performance is improved.

[0167] Figure 6 is a structural diagram of an inner speech decoding device based on brain network constraint and graph contrast learning provided by an embodiment of the present invention, such as Figure 6 As shown, the inner speech decoding device based on brain network constraint and graph contrast learning may include the above Figure 5 The inner speech decoding device based on brain network constraints and graph contrast learning is shown. Optionally, the inner speech decoding device based on brain network constraints and graph contrast learning 410 may include a first processor 2001.

[0168] Optionally, the inner speech decoding device 410 based on brain network constraint and graph contrast learning may further include a memory 2002 and a transceiver 2003 .

[0169] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0170] The following combination Figure 6 The components of the inner speech decoding device 410 based on brain network constraints and graph contrast learning are described in detail:

[0171] The first processor 2001 is the control center of the inner speech decoding device 410 based on brain network constraint and graph contrast learning, and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), or application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0172] Optionally, the first processor 2001 can perform various functions of the internal speech decoding device 410 based on brain network constraints and graph contrast learning by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0173] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 6 CPU0 and CPU1 are shown in FIG.

[0174] In a specific implementation, as an embodiment, the inner speech decoding device 410 based on brain network constraint and graph contrast learning may also include multiple processors, such as Figure 6 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0175] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0176] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and accessed through the interface circuit ( Figure 6 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0177] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0178] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 6 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0179] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or can exist independently and can be connected to the internal speech decoding device 410 based on brain network constraint and graph contrast learning ( Figure 6 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0180] It should be noted that Figure 6 The structure of the internal speech decoding device 410 based on brain network constraints and graph contrast learning shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0181] In addition, the technical effects of the internal speech decoding device 410 based on brain network constraints and graph contrast learning can refer to the technical effects of the internal speech decoding method based on brain network constraints and graph contrast learning described in the above method embodiment, and will not be repeated here.

[0182] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0183] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0184] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0185] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0186] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0187] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0188] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0189] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0190] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0191] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0192] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0193] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0194] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for inner speech decoding based on brain network constraints and graph contrast learning, characterized in that: The method comprises: S1. Obtain EEG signal sample dataset; S2. Constructing an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions comprises: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain networks, and a spatial function fusion module; S3. Training the initial graph contrast learning model constrained by brain network spatial functions based on the EEG signal sample data set, extracting local features through the graph-based EEG representation building module to obtain local features of nodes; extracting network features through the graph-based EEG representation building module to obtain network features; combining the local features of the nodes with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation; S4. Inputting the spatial-based EEG map representation into the dual-stream map contrast learning module constrained by the brain network for contrast learning to obtain a spatial-based EEG map representation after contrast learning; inputting the functional-based EEG map representation into the dual-stream map contrast learning module constrained by the brain network for contrast learning to obtain a functional-based EEG map representation after contrast learning; S5. Inputting the spatial-based EEG graph representation after contrastive learning and the functional-based EEG graph representation after contrastive learning into the spatial-functional fusion module to obtain inner speech decoding prediction results; and obtaining a trained graph contrastive learning model constrained by the spatial function of the brain network based on the inner speech decoding prediction results; S6. Obtain EEG signal data to be decoded; input the EEG signal data to be decoded into the trained graph contrast learning model constrained by brain network spatial function to obtain the final inner speech decoding result.

2. The inner speech decoding method based on brain network constraint and graph contrast learning according to claim 1, characterized in that The S3 extracts local features through the graph-based EEG representation building module to obtain local features of the node, including: S31. Based on the concept of graph, the EEG information is lightweight represented in the form of a graph structure; according to a preset number of frames, the electrode signals in the EEG signal sample data set are divided into multiple frames, and the signal power of each frequency band is calculated according to a preset number of frequency bands to obtain power characteristics of multiple EEG electrodes; S32. Based on the power features of the multiple EEG electrodes, a one-dimensional convolutional layer is used to shrink the power features of the multiple EEG electrodes into a one-dimensional vector, and output local feature representations of the multiple nodes; wherein the local feature representation of the node is obtained by the following formula (1): （1） in, For the Local feature representation of nodes; Indicates the The power characteristics of the EEG electrodes, , F represents the number of frequency bands, T represents the number of frames; R represents a set of real numbers; represents the weight matrix, This is a hyperparameter representing the size of the convolution kernel, and the value set is the same as the stride; Represents the bias vector to be trained; represents the linear rectification function.

3. The inner speech decoding method based on brain network constraint and graph contrast learning according to claim 1, characterized in that: The S3 extracts network features through the graph-based EEG representation building module to obtain network features; Combining the local features of the nodes with the network features to obtain a space-based EEG map representation and a function-based EEG map representation, comprising: S33. According to the graph structure representation, a spatial connectivity matrix is ​​calculated based on predefined spatial relationships; an adjacency matrix is ​​sparsified on the spatial connectivity matrix using a random initial first trainable mask matrix to obtain a space-based adjacency matrix; a functional connectivity matrix is ​​calculated based on the Pearson correlation coefficient, and an adjacency matrix is ​​sparsified on the functional connectivity matrix using a random initial second trainable mask matrix to obtain a function-based adjacency matrix; The process of obtaining the space-based adjacency matrix is ​​expressed by the following formula (2): （2） in, Represents a space-based adjacency matrix; represents the angle between the p-th electrode and the q-th electrode; represents the Euclidean distance between the p-th electrode and the q-th electrode; represents the cosine function of the angle between the pth electrode and the qth electrode; represents the first trainable mask matrix randomly initialized; p and q represent the number of electrodes; The process of obtaining the function-based adjacency matrix is ​​expressed by the following formula (3): （3） in, Represents a function-based adjacency matrix; represents the EEG signal sequence of the pth electrode; represents the EEG signal sequence of the qth electrode; represents the standard deviation of the EEG signal sequence of the pth electrode; represents the standard deviation of the EEG signal sequence of the qth electrode; represents the covariance between the EEG signal sequence of the p-th electrode and the EEG signal sequence of the q-th electrode; represents the randomly initialized second trainable mask matrix; , The value range is (-1,1), The value range is (-1, 1) and does not exceed r%. r represents a pre-set hyperparameter used to control the sparsity of the adjacency matrix. The value of the element r∈(0, 100) is greater than 0. N represents the total number of EEG electrodes. S34. Based on the local feature representation of multiple nodes and the space-based adjacency matrix, a set of edges based on spatial connections between nodes are obtained; the local feature representation of multiple nodes, the space-based adjacency matrix and the set of edges based on spatial connections between nodes are combined to obtain a space-based EEG map representation; based on the local feature representation of multiple nodes and the function-based adjacency matrix, a set of edges based on functional connections between nodes are obtained; the local feature representation of multiple nodes, the function-based adjacency matrix and the set of edges based on functional connections between nodes are combined to obtain a function-based EEG map representation.

4. The inner speech decoding method based on brain network constraint and graph contrast learning according to claim 1, characterized in that The brain network-constrained dual-stream graph contrast learning module of S2 includes: a brain network-constrained graph enhancement module, an encoder based on a graph isomorphism network, and a projection layer.

5. The inner speech decoding method based on brain network constraint and graph contrast learning according to claim 4, characterized in that: The step S4 of inputting the space-based EEG map representation into the dual-stream map contrast learning module constrained by the brain network for contrast learning to obtain a space-based EEG map representation after contrast learning; and inputting the function-based EEG map representation into the dual-stream map contrast learning module constrained by the brain network for contrast learning to obtain a function-based EEG map representation after contrast learning, includes: S41. Inputting the spatial-based EEG graph representation into the graph enhancement module constrained by the brain network, enhancing the graph data of the spatial branch with the brain network characteristics as the constraint condition, and obtaining the EEG graph representation based on spatial enhancement; wherein the process of enhancing the graph data of the spatial branch is expressed by the following formula (4): （4） in, Represents the adjacency matrix based on spatial enhancement; express The absolute values ​​of the non-zero elements in the array are sorted at the top. The element set of , k represents the hyperparameter that controls the proportion of the elements retained in the original adjacency matrix; Represents Random number matrices of the same size, with elements ranging from 0 to 1; Indicates from A random selection from the zero elements in The edge set corresponding to the element; S42. Input the function-based EEG graph representation into the graph enhancement module constrained by the brain network, and enhance the graph data of the functional branch with the brain network characteristics as the constraint to obtain the EEG graph representation based on function enhancement. The process of enhancing the graph data of the functional branch is expressed by the following formula (5): （5） in, Represents the adjacency matrix based on feature enhancement; Indicates from A random selection from the zero elements in The edge set corresponding to the element; S43. Input the space-based EEG graph representation, the function-based EEG graph representation, the space-enhanced EEG graph representation, and the function-enhanced EEG graph representation into the encoder and projection layer composed of the graph isomorphism network for comparative learning, and obtain the graph representation of the EEG signal sample data set after comparative learning; wherein, the graph representation of the EEG signal sample data set after comparative learning includes: the space-based EEG graph representation after comparative learning and the function-based EEG graph representation after comparative learning, wherein, the update rule of the node representation of each graph representation in the graph isomorphism network is expressed by the following formula (6): （6） in, represents the updated node representation; Represents the node representation, represents the adjacency matrix of the EEG graph, represents the identity matrix, It is a learnable parameter that controls the influence of each node's own attributes. MLP represents the multi-layer perceptron model.

6. The inner speech decoding method based on brain network constraint and graph contrast learning according to claim 1, characterized in that: The spatial function fusion module of S2 includes two basic classifiers and a meta-classifier.

7. The inner speech decoding method based on brain network constraint and graph contrast learning according to claim 6, characterized in that: The step S5 inputs the space-based EEG map representation after contrastive learning and the function-based EEG map representation after contrastive learning into the space-function fusion module to obtain an inner speech decoding prediction result, including: S51. Input the graph representation of the EEG signal sample data set after contrastive learning into the spatial function fusion module, use the K-means algorithm to cluster the graph representation of the spatial branches, obtain the graph representation clustering center based on the spatial branches and the clusters based on the spatial clustering, and use the K-means algorithm to cluster the graph representation of the functional branches, obtain the graph representation clustering center based on the functional branches and the clusters based on the functional clustering; wherein the clustering process is expressed by the following formula (7): （7） in, The representation graph represents the cluster centers; Represents the cluster after clustering; Represents the projection layer; represents the encoder; Node representation of EEG graph; Represents the adjacency matrix of the EEG graph; represents the number of clusters, which is consistent with the number of internal speech instructions to be decoded; i represents the branch subscript, where i=1 represents the spatial branch and i=2 represents the functional branch; S52. Calculating a space-based fitness score based on the spatial branch-based graph representation cluster center; According to the cluster center represented by the graph based on the functional branch, the function-based fitness score is calculated; wherein, the fitness score calculation formula is expressed by the following formula (8): （8） in, Indicates the The first branch clusters, ; express A sample image from Indicates the The first branch the center of a cluster; Represents the fitness score, when It indicates the fitness score based on space; When represents the fitness score based on function; The weight of the space-based branch is calculated based on the space-based fitness score; the weight of the function-based branch is calculated based on the function-based fitness score. The calculation process of the weight of the space-based branch and the weight of the function-based branch is expressed by the following formula (9): （9） in, represents the fitness score of the spatial branch; Represents the fitness score of the functional branch; Indicates the weight of the output result of the branch basic classifier in the meta-classifier, The value range of is 1 or 2, where i=1 represents spatial branch and i=2 represents functional branch; S53, inputting the clusters after spatial clustering into the first basic classifier, and outputting a spatial decoding result; using the weights based on the spatial branches, performing weighted processing on the spatial decoding result to obtain a weighted spatial decoding result; S54, inputting the clusters after function-based clustering into the second basic classifier, and outputting the function-based decoding results; using the weights based on the function branches, performing weighted processing on the function-based decoding results to obtain weighted function-based decoding results; S55. Input the weighted spatial-based decoding result and the weighted functional-based decoding result into a meta-classifier to obtain an inner speech decoding prediction result.

8. An inner speech decoding device based on brain network constraints and graph contrast learning, wherein the inner speech decoding device based on brain network constraints and graph contrast learning is used to implement the inner speech decoding method based on brain network constraints and graph contrast learning as described in any one of claims 1 to 7, characterized in that: The device comprises: An acquisition unit, used for acquiring an EEG signal sample data set; A construction unit, configured to construct an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions comprises: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain networks, and a spatial function fusion module; a training unit for training the initial graph contrast learning model constrained by the spatial function of the brain network according to the EEG signal sample data set, extracting local features through the graph-based EEG representation construction module to obtain local features of the nodes; extracting network features through the graph-based EEG representation construction module to obtain network features; combining the local features of the nodes with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation; inputting the space-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning to obtain a space-based EEG graph representation after contrast learning; inputting the function-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning to obtain a function-based EEG graph representation after contrast learning; inputting the space-based EEG graph representation after contrast learning and the function-based EEG graph representation after contrast learning into the spatial function fusion module to obtain an internal speech decoding prediction result; and obtaining a trained graph contrast learning model constrained by the spatial function of the brain network according to the internal speech decoding prediction result; The decoding unit is used to obtain EEG signal data to be decoded; input the EEG signal data to be decoded into the trained graph contrast learning model constrained by the brain network spatial function to obtain the final inner speech decoding result.

9. An inner speech decoding device based on brain network constraint and graph contrast learning, characterized in that The inner speech decoding device based on brain network constraint and graph contrast learning includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images