Internal speech decoding method based on brain network constraint and graph contrast learning

Abstract

The invention provides an internal speech decoding method and device based on brain network constraint and graph contrast learning, and relates to the technical field of electroencephalogram signal intelligent decoding. The method comprises the following steps: acquiring an EEG signal sample data set; constructing an initial graph contrast learning model constrained by a brain network space function; the model comprises a graph-based electroencephalogram representation construction module, a double-flow graph comparison learning module constrained by a brain network and a space function fusion module. Training the model according to the EEG signal sample data set to obtain a trained graph contrast learning model constrained by the brain network space function; obtaining EEG signal data to be decoded; and inputting EEG signal data to be decoded into the trained graph contrast learning model constrained by the brain network space function to obtain a final internal speech decoding result. According to the invention, the decoding performance can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent decoding of electroencephalogram signals, and in particular to an inner speech decoding method and device based on brain network constraints and graph contrast learning. Background Art

[0002] Inner speech (IS) is the internalized process of a person thinking in pure meaning, also known as inner monologue. It is an important and common way to express intentions and occupies an obvious and important position in human cognition. As a promising technology field, brain-computer interface (BCI) can build an information exchange pathway between the brain and the external environment by collecting and decoding electroencephalogram (EEG) signals from brain neural activity. Compared with existing BCI paradigms, such as P300, SSVEP or motor imagery, IS-BCI is easier to use and more concise and intuitive for subjects, and has stronger applicability in real life and long-term application scenarios.

[0003] At present, the EEG decoding methods for inner speech mainly use deep learning methods to decode IS and EEG decoding methods based on contrastive learning; among them, the deep learning methods include: using CNN to capture the spatiotemporal dependencies at multiple scales in EEG signals, using convolutional and recurrent neural networks to decode 9 words, using CNN to build a BCI control system that can detect and decode IS, and using CNN-based capsule neural networks to decode dissonants, nasals, vowels, and consonants respectively; due to the high time cost of collecting IS EEG data and the individual variability of EEG, model training faces the problems of insufficient quantity and overfitting; among them, the EEG decoding methods based on contrastive learning include: one is to enhance the data by adjusting the order of appearance of different slices in the EEG sample, which is applied to sleep staging and pathological detection; the other is to generate subgraphs for data enhancement by restarting random walk technology, allowing the generated samples to capture local structure and context information. However, the current graph enhancement methods are all based on random mechanisms, which will destroy the inherent connectivity of the brain network, resulting in weak decoding performance. Summary of the invention

[0004] In order to solve the technical problems in the prior art, such as the high time cost of collecting IS EEG data and the individual variability of EEG, which leads to insufficient data and overfitting in model training, and the random mechanism of graph enhancement method, which leads to poor decoding performance, the embodiment of the present invention provides an inner speech decoding method and device based on brain network constraints and graph contrast learning. The technical solution is as follows:

[0005] In one aspect, a method for inner speech decoding based on brain network constraints and graph contrast learning is provided, the method being implemented by an inner speech decoding device based on brain network constraints and graph contrast learning, the method comprising:

[0006] S1. Obtain EEG signal sample data set;

[0007] S2. constructing an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions comprises: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain networks, and a spatial function fusion module;

[0008] S3, training the initial graph contrast learning model constrained by brain network spatial function according to the EEG signal sample data set, extracting local features through the graph-based EEG representation construction module to obtain local features of nodes; extracting network features through the graph-based EEG representation construction module to obtain network features; combining the local features of the nodes with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation;

[0009] S4, inputting the space-based EEG image representation into the dual-stream image contrast learning module constrained by the brain network for contrast learning, and obtaining the space-based EEG image representation after contrast learning; inputting the function-based EEG image representation into the dual-stream image contrast learning module constrained by the brain network for contrast learning, and obtaining the function-based EEG image representation after contrast learning;

[0010] S5, inputting the space-based EEG graph representation after contrastive learning and the function-based EEG graph representation after contrastive learning into the space-function fusion module to obtain the inner speech decoding prediction result; according to the inner speech decoding prediction result, obtaining a trained graph contrastive learning model constrained by the brain network space function;

[0011] S6. Obtain EEG signal data to be decoded; input the EEG signal data to be decoded into the trained graph contrast learning model constrained by the brain network spatial function to obtain the final inner speech decoding result.

[0012] On the other hand, an inner speech decoding device based on brain network constraint and graph contrast learning is provided, and the device is applied to the inner speech decoding method based on brain network constraint and graph contrast learning, and the device comprises:

[0013] An acquisition unit, used for acquiring an EEG signal sample data set;

[0014] A construction unit, used to construct an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions comprises: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain network, and a spatial function fusion module;

[0015] A training unit, used for training the initial graph contrast learning model constrained by the spatial function of the brain network according to the EEG signal sample data set, extracting local features through the graph-based EEG representation construction module to obtain local features of nodes; extracting network features through the graph-based EEG representation construction module to obtain network features; combining the local features of the nodes with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation; inputting the space-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning to obtain a space-based EEG graph representation after contrast learning; inputting the function-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning to obtain a function-based EEG graph representation after contrast learning; inputting the space-based EEG graph representation after contrast learning and the function-based EEG graph representation after contrast learning into the spatial function fusion module to obtain an internal speech decoding prediction result; according to the internal speech decoding prediction result, a trained graph contrast learning model constrained by the spatial function of the brain network is obtained;

[0016] The decoding unit is used to obtain the EEG signal data to be decoded; input the EEG signal data to be decoded into the trained graph contrast learning model constrained by the brain network spatial function to obtain the final inner speech decoding result.

[0017] On the other hand, an inner speech decoding device based on brain network constraints and graph contrast learning is provided, and the inner speech decoding device based on brain network constraints and graph contrast learning comprises: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the inner speech decoding methods based on brain network constraints and graph contrast learning is implemented.

[0018] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned inner speech decoding methods based on brain network constraints and graph contrast learning.

[0019] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0020] The embodiment of the present invention first obtains an EEG signal sample data set; constructs an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions includes: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain network, and a spatial function fusion module; secondly, the initial graph contrast learning model constrained by brain network spatial functions is trained according to the EEG signal sample data set, and local features are extracted by the graph-based EEG representation construction module to obtain local features of nodes; network features are extracted by the graph-based EEG representation construction module to obtain network features; the local features of the nodes are combined with the network features to obtain space-based EEG graph representation and function-based EEG graph representation; The space-based EEG map representation and the function-based EEG map representation are input into the dual-stream graph contrast learning module constrained by the brain network for contrast learning, so as to obtain the space-based EEG map representation and the function-based EEG map representation after contrast learning; the space-based EEG map representation and the function-based EEG map representation after contrast learning are input into the spatial function fusion module to obtain the inner speech decoding prediction result; finally, according to the inner speech decoding prediction result, a trained graph contrast learning model constrained by the spatial function of the brain network is obtained; the EEG signal data to be decoded is obtained; the EEG signal data to be decoded is input into the trained graph contrast learning model constrained by the spatial function of the brain network to obtain the final inner speech decoding result.

[0021] The embodiment of the present invention introduces brain network prior knowledge into the feasibility of inner speech decoding. The embodiment of the present invention can comprehensively learn robust representations from functional and spatial perspectives. In order to comprehensively characterize the brain state related to inner speech, the embodiment of the present invention designs a graph-based EEG representation building module to learn local and network features. In order to alleviate the impact of data scarcity, the embodiment of the present invention designs a dual-stream graph contrast learning module constrained by brain networks to generate more brain-like enhanced samples, and improves decoding performance by designing a superposition mechanism to fuse graph representations from spatial streams and functional streams. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 It is a structural schematic diagram of a graph contrast learning method constrained by brain network space function provided by an embodiment of the present invention;

[0024] Figure 2 It is a flow chart of an inner speech decoding method based on brain network constraint and graph contrast learning provided by an embodiment of the present invention;

[0025] Figure 3 It is a schematic diagram of the structure of the space-function fusion module provided in an embodiment of the present invention;

[0026] Figure 4 It is a visual brain topography map provided by an embodiment of the present invention;

[0027] Figure 5 It is a block diagram of an inner speech decoding device based on brain network constraint and graph contrast learning provided by an embodiment of the present invention;

[0028] Figure 6 It is a structural schematic diagram of an inner speech decoding device based on brain network constraints and graph contrast learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0030] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0031] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0032] In the embodiments of the present invention, sometimes the subscripts such as W 1 It may be written in non-subscript form such as W1. When the difference is not emphasized, the meaning is the same.

[0033] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0034] The embodiment of the present invention provides an inner speech decoding method based on brain network constraints and graph contrast learning, which can be implemented by an inner speech decoding device based on brain network constraints and graph contrast learning, and the inner speech decoding device based on brain network constraints and graph contrast learning can be a terminal or a server. Figure 1 is a structural schematic diagram of a graph contrast learning method constrained by brain network space function provided by an embodiment of the present invention, such as Figure 2 The flowchart of the inner speech decoding method based on brain network constraint and graph contrast learning is shown in the figure. The processing flow of the method may include the following steps:

[0035] S1. Obtain EEG signal sample dataset.

[0036] Among them, EEG (Electroencephalography) signals are a way to measure brain activity. They record the electrical activity of the cerebral cortex by placing electrodes on the scalp. EEG signals reflect the synchronous discharge of neurons and can provide real-time information about the functional state of the brain.

[0037] S2. Construct an initial graph contrast learning model constrained by the spatial functions of the brain network; the initial graph contrast learning model constrained by the spatial functions of the brain network includes: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by the brain network, and a spatial function fusion module.

[0038] Optionally, the brain network constrained dual-stream graph contrast learning module of S2 includes: a brain network constrained graph enhancement module, an encoder based on a graph isomorphic network, and a projection layer.

[0039] Among them, the dual-stream graph contrast learning module constrained by the brain network also includes: a loss function.

[0040] Optionally, the spatial function fusion module of S2 includes: two basic classifiers and one meta classifier.

[0041] S3. The initial graph contrast learning model constrained by the spatial function of the brain network is trained according to the EEG signal sample data set, and local features are extracted through the graph-based EEG representation construction module to obtain local features of the nodes; network features are extracted through the graph-based EEG representation construction module to obtain network features; the local features of the nodes are combined with the network features to obtain space-based EEG graph representation and function-based EEG graph representation.

[0042] Optionally, the specific implementation process of S3 may include S31-S34:

[0043] S31. According to the concept of graph, the EEG information is lightweight represented in the form of a graph structure; according to a preset number of frames, the electrode signal in the EEG signal sample data set is divided into multiple frames, and the signal power of each frequency band is calculated according to a preset number of frequency bands to obtain power characteristics of multiple EEG electrodes;

[0044] In a feasible implementation, in order to enable the graph contrastive learning model constrained by the spatial function of the brain network to comprehensively characterize the local information of the electrode and the global brain network topology information while reducing the burden of model training as much as possible, the embodiment of the present invention proposes a graph-based brain representation construction module to obtain the EEG representation of the EEG signal sample data set; in order to avoid the subsequent contrastive learning model from having too many parameters to be trained, resulting in overfitting of the model, the embodiment of the present invention is based on the concept of graph , the EEG information is expressed in a lightweight form in the form of a graph structure, where Feature representation of the representation graph; Indicates that it contains A collection of node features, each node corresponds to an electrode one by one; Represents the adjacency matrix of a graph; A set of edges representing the existence of spatial or functional connections between electrodes, containing key network features present in the brain.

[0045] S32. According to the power features of the multiple EEG electrodes, a one-dimensional convolution layer is used to shrink the power features of the multiple EEG electrodes to a one-dimensional vector, and the local feature representations of the multiple nodes are output; wherein the local feature representation of the node can be obtained by the following formula (1):

[0046] (1)

[0047] in, For the Local feature representation of nodes; Indicates The power characteristics of the EEG electrodes, , F represents the number of frequency bands, T represents the number of frames; R represents a real number set; represents the weight matrix, This is a hyperparameter representing the size of the convolution kernel, and the value set is the same as the stride; represents the bias vector to be trained; represents the linear rectification function.

[0048] Optionally, S3 extracts network features through a graph-based EEG representation building module to obtain network features; combines local features of nodes with network features to obtain space-based EEG graph representation and function-based EEG graph representation, including:

[0049] Among them, since there are a large number of repetitions in the third-order neighbors of each node, its own characteristic information will be lost and the same eigenvalue will be obtained, which will lead to the over-smoothing problem; in order to avoid the above situation and further reduce the burden of model training, when constructing EEG representation based on the graph, the edges of the graph are sparsely operated to make it lightweight.

[0050] S33, according to the graph structure representation, the spatial connectivity matrix is ​​calculated based on the predefined spatial relationship; the spatial connectivity matrix is ​​sparsely processed by using a random initial first trainable mask matrix to obtain a space-based adjacency matrix; the functional connectivity matrix is ​​calculated based on the Pearson correlation coefficient, and the functional connectivity matrix is ​​sparsely processed by using a random initial second trainable mask matrix to obtain a function-based adjacency matrix; wherein the process of obtaining the space-based adjacency matrix can be expressed by the following formula (2):

[0051] (2)

[0052] in, Represents a space-based adjacency matrix; represents the angle between the p-th electrode and the q-th electrode; represents the p-th electrode and the q-th electrode; represents the cosine function of the angle between the pth electrode and the qth electrode; represents the first trainable mask matrix randomly initialized; p and q represent the number of electrodes;

[0053] The process of obtaining the function-based adjacency matrix can be expressed by the following formula (3):

[0054] (3)

[0055] in, represents the EEG signal sequence of the pth electrode; represents the EEG signal sequence of the qth electrode; represents the standard deviation of the EEG signal sequence of the pth electrode; represents the standard deviation of the EEG signal sequence of the qth electrode; represents the covariance between the EEG signal sequence of the p-th electrode and the EEG signal sequence of the q-th electrode; represents a randomly initialized second trainable mask matrix; , The value range is (-1,1), The value range is (-1, 1) and does not exceed r%, the value of the element r∈(0, 100) is greater than 0; r represents a pre-set hyperparameter used to control the sparsity of the adjacency matrix; N represents the total number of EEG electrodes.

[0056] Among them, the spatial EEG graph representation can be expressed as ; The function-based EEG graph representation can be expressed as .

[0057] The predefined spatial relationship may be defined as the ratio of the cosine value of the angle between the two electrodes and the Euclidean distance.

[0058] The Pearson correlation coefficient is a conventional technical means, and the embodiment of the present invention does not elaborate on the calculation process of the Pearson correlation coefficient.

[0059] S34. According to the local feature representation of multiple nodes and the space-based adjacency matrix, a set of edges based on spatial connection between nodes is obtained; the local feature representation of multiple nodes, the space-based adjacency matrix and the set of edges based on spatial connection between nodes are combined to obtain a space-based EEG graph representation; according to the local feature representation of multiple nodes and the function-based adjacency matrix, a set of edges based on functional connection between nodes is obtained; the local feature representation of multiple nodes, the function-based adjacency matrix and the set of edges based on functional connection between nodes are combined to obtain a function-based EEG graph representation.

[0060] In a feasible implementation, the edges in the graph are determined by nodes and an adjacency matrix based on the representation of the graph structure; wherein the adjacency matrix is ​​a two-dimensional array, wherein rows and columns represent nodes in the graph, and when an element in the matrix is ​​non-zero, there is an edge between the nodes; based on the local features of multiple nodes and a space-based adjacency matrix, a set of edges of spatial connections between nodes are obtained, which contain key network features of the brain; based on the local features of multiple nodes and a function-based adjacency matrix, a set of edges of functional connections between nodes are obtained.

[0061] S4. Input the space-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning, and obtain the space-based EEG graph representation after contrast learning; input the function-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning, and obtain the function-based EEG graph representation after contrast learning.

[0062] Among them, the brain network is a complex network composed of spatial structure and functional organization; the physical connections of the brain network in space are non-random, which is mainly reflected in the small-world architecture, including highly local clustering and short path length. Taking this brain network characteristic as a constraint, when performing comparative learning on spatial branches, the connection relationships that are more adjacent in spatial positions should be retained with a greater probability, so as to avoid destroying the small-world architecture of spatial connectivity during data enhancement.

[0063] Optionally, the specific implementation process of S4 may include S41-S43:

[0064] S41, inputting the spatial EEG graph representation into the graph enhancement module constrained by the brain network, enhancing the graph data of the spatial branch with the brain network characteristics as the constraint condition, and obtaining the EEG graph representation based on spatial enhancement; wherein the process of enhancing the graph data of the spatial branch can be expressed by the following formula (4):

[0065] (4)

[0066] in, Represents the adjacency matrix based on spatial enhancement; express The absolute values ​​of the non-zero elements in the array are sorted at the top. The element set of k represents the hyperparameter that controls the proportion of the elements retained in the original adjacency matrix; Representation and Random number matrices of the same size, with elements ranging from 0 to 1; Indicates from A random selection from the zero elements in The edge set corresponding to the element;

[0067] Among them, according to The non-zero elements in the node and edge Update to obtain the EEG image representation based on spatial enhancement, which can be expressed as .

[0068] S42, inputting the function-based EEG graph representation into the graph enhancement module constrained by the brain network, and enhancing the graph data of the functional branch with the brain network characteristics as the constraint to obtain the EEG graph representation based on function enhancement; wherein the process of enhancing the graph data of the functional branch can be expressed by the following formula (5):

[0069] (5)

[0070] in, Represents the adjacency matrix based on feature enhancement; Indicates from A random selection from the zero elements in The edge set corresponding to the element;

[0071] Among them, according to The non-zero elements in the node and edge Update to obtain the EEG graph representation based on functional enhancement, which can be expressed as .

[0072] Among them, there is a lot of overlap between the functional connections and spatial connections of the brain network, and neurons with spatial connections have stronger functional connectivity than unconnected neurons. Taking this brain network characteristic as a constraint, when conducting comparative learning on the functional branched brain network, the connections contained in the spatially enhanced graph representation are more likely to be retained, and new samples are generated on the premise of containing effective functional connections.

[0073] S43, inputting the space-based EEG graph representation, the function-based EEG graph representation, the space-enhanced EEG graph representation and the function-enhanced EEG graph representation into the encoder and projection layer composed of the graph isomorphism network for contrastive learning, and obtaining the graph representation of the EEG signal sample data set after contrastive learning; wherein the graph representation of the EEG signal sample data set after contrastive learning includes: the space-based EEG graph representation after contrastive learning and the function-based EEG graph representation after contrastive learning

[0074] Among them, the spatial stream and the functional stream are compared and learned based on the same network structure, and the spatial EEG map is represented , Function-based EEG representation , EEG image representation based on spatial enhancement And EEG image representation based on functional enhancement , forming four groups of image samples; in the contrastive learning stage, using and The original samples and and Positive pairs are created by pairing each sample with other samples in the same set, and negative pairs are created by pairing each sample with other samples in the same set. The update rule for the node representation of each graph representation in the graph isomorphic network can be expressed by the following formula (6):

[0075] (6)

[0076] in, represents the updated node representation; Represents the node representation, represents the adjacency matrix of the EEG graph, represents the identity matrix, It is a learnable parameter that controls the influence of each node's own attributes. MLP stands for multi-layer perceptron model. The four groups of graph representations obtained by the encoder are subjected to graph pooling operations to obtain graph sample representations, which are then mapped to a projection layer consisting of two fully connected layers and one ReLU layer, also known as a projection head. The potential space in .

[0077] In a feasible implementation, the spatial flow loss and the functional flow loss are calculated respectively according to the contrast loss function; the cosine similarity of the graph representation before and after enhancement is calculated in the latent space respectively, which can measure the enhancement effect of the graph sample; the overall loss function is calculated according to the spatial flow loss and the functional flow loss, which is used to guide the simultaneous training of the graph enhancement module constrained by the brain network; wherein the overall loss function can be expressed by the following formula (7):

[0078] (7)

[0079] Among them, cos represents cosine similarity, represents the spatial flow, Represents the functional flow; Represents the overall loss function; Represents the loss function of the branch. When i=1, it is the loss function based on the spatial branch. When i=2, it is the loss function based on the functional branch.

[0080] Among them, the contrast loss function is a commonly used loss function in machine learning. A common form of contrast loss is the triple loss including: anchor point, positive sample and negative sample.

[0081] Among them, the cosine similarity of the graph representation before and after enhancement is calculated in the latent space to measure the enhancement effect of the graph sample and determine the influence of this branch on the overall loss function.

[0082] S5. Input the space-based EEG graph representation after contrastive learning and the function-based EEG graph representation after contrastive learning into the spatial function fusion module to obtain the inner speech decoding prediction result; according to the inner speech decoding prediction result, obtain the trained graph contrastive learning model constrained by the spatial function of the brain network.

[0083] Among them, contrastive learning is an unsupervised learning method, which learns by comparing different views or variants of data. The specific implementation process of contrastive learning may include: data preprocessing: first, preprocess the original data, including normalization and denoising, which can facilitate subsequent processing; contrastive learning requires paired data; in a positive sample pair, the two samples are similar, usually through data enhancement techniques including rotation, scaling and cropping, to generate positive sample pairs from the same original data; relative to the positive sample pair, the two samples in the negative sample pair are dissimilar; they can be randomly selected from the data set, or negative sample pairs can be generated through different data enhancement techniques; use a neural network or other model to extract the feature representation of the sample; the feature representation can capture the essential attributes of the data; wherein the neural network can use a CNN neural network model; define a loss function to train the model. The loss function usually encourages the model to bring the feature representation of the positive sample pair closer and push the feature representation of the negative sample pair farther; in the embodiment of the present invention, the overall loss function is calculated based on the loss of the spatial flow and the loss of the functional flow; the parameters of the model are updated through the back propagation algorithm to minimize the loss function; repeat the above steps until the model achieves satisfactory performance on the contrast task.

[0084] Among them, the spatial function fusion module can fuse the knowledge of spatial branches and functional branches, which can maximize the use of brain network information and further improve the internal speech decoding performance; the spatial function fusion module can integrate the information of spatial brain network and functional brain network to achieve complementary advantages and further improve the internal speech decoding performance of the model; the embodiment of the present invention designs a classifier based on a stacking mechanism to fuse spatial streams and functional streams to generate the final decoding results.

[0085] In a feasible implementation manner, the embodiment of the present invention uses support vector machine (SVM) as the basic classifier for internal speech decoding, and allocates appropriate weights to the spatial branch and the functional branch through a stacking mechanism, thereby promoting the effective fusion of information from the spatial stream and the functional stream.

[0086] Among them, support vector machine SVM is a commonly used supervised learning algorithm, which can be used for classification and regression problems; in the model fusion technology, support vector machine SVM can be used as a base learner to participate in the stacking mechanism, which is a commonly used ensemble learning method. The implementation process of the stacking mechanism may include:

[0087] (1) selecting a set of base learners, which may be different types of models, including decision trees, neural networks, and SVMs, etc.; in the embodiment of the present invention, SVM is selected;

[0088] (2) Use K-fold cross validation to train the base learner and generate prediction results for the training set and test set;

[0089] (3) The prediction results are used as new features and combined with the original features to form a new feature matrix;

[0090] (4) Use a meta-learner, which can be an SVM or other model, to train on the new feature matrix;

[0091] (5) On the test set, use the trained base learner to make predictions and use the prediction results as the input of the meta-learner to obtain the final prediction.

[0092] Optionally, the specific implementation steps of S5 may include S51-S55:

[0093] S51, inputting the graph representation of the EEG signal sample data set after contrast learning into the spatial function fusion module, clustering the graph representation of the spatial branch using the K-means algorithm to obtain the graph representation clustering center based on the spatial branch and the clusters based on the spatial clustering, clustering the graph representation of the functional branch using the K-means algorithm to obtain the graph representation clustering center based on the functional branch and the clusters based on the functional clustering;

[0094] In a feasible implementation, Figure 3 The figure is a schematic diagram of the structure of the space-function fusion module provided by an embodiment of the present invention; the K-means algorithm is used to cluster the graph representation of the spatial branch and the graph representation of the functional branch respectively, to obtain the clustering center of the graph representation based on the spatial branch and the clusters based on the spatial clustering, as well as the clustering center of the graph representation based on the functional branch and the clusters based on the functional clustering. The clustering process can be expressed by the following formula (8):

[0095] (8)

[0096] in, The representation graph represents the cluster centers; Represents the cluster after clustering; Represents the projection layer; represents the encoder; Node representation representing the EEG graph; Represents the adjacency matrix of the EEG graph; represents the number of clusters, which is consistent with the number of internal speech instructions to be decoded; i represents the branch subscript, where i=1 represents the spatial branch and i=2 represents the functional branch;

[0097] S52, calculating a space-based fitness score according to the cluster center represented by the graph based on the space branch;

[0098] According to the graph representation based on functional branches, cluster centers are calculated and the function-based fitness scores are calculated;

[0099] In a feasible implementation, the fitness score calculation formula can be expressed by the following formula (9):

[0100] (9)

[0101] in, Indicates The branch clusters, ; express A sample of the graph in Indicates The branch The center of a cluster; represents the fitness score, when It indicates the fitness score based on space when represents the function-based fitness score.

[0102] In a feasible implementation, the weight of the space-based branch is calculated according to the space-based fitness score; the weight of the function-based branch is calculated according to the function-based fitness score, wherein the calculation process of the weight based on the space branch and the weight based on the function branch can be expressed by the following formula (10):

[0103] (10)

[0104] in, represents the fitness score of the spatial branch; Represents the fitness score of the functional branch; It represents the weight of the output result of the branch basic classifier in the meta-classifier. The value range is 1 or 2, where i=1 represents the spatial branch and i=2 represents the functional branch.

[0105] S53, inputting the clusters after spatial clustering into the first basic classifier, and outputting the spatial decoding results; using the weights based on the spatial branches, weighting the spatial decoding results to obtain the weighted spatial decoding results;

[0106] S54, inputting the clusters after function-based clustering into the second basic classifier, and outputting the function-based decoding results; using the weights based on the function branches, weighting the function-based decoding results to obtain weighted function-based decoding results;

[0107] S55. Input the weighted spatial-based decoding result and the weighted functional-based decoding result into a meta-classifier to obtain an inner speech decoding prediction result.

[0108] S6. Obtain the EEG signal data to be decoded; input the EEG signal data to be decoded into a trained graph contrast learning model constrained by the spatial function of the brain network to obtain the final inner speech decoding result.

[0109] In a feasible implementation, the embodiment of the present invention is compared with five currently advanced methods; wherein, the experimental data is BCI-Competition V-3, BCI V3 dataset, and the BCI V3 dataset collects EEG data when the subjects imagine daily-related phrases / vocabulary, which may include: hello, help me, stop, thank you and yes. The embodiment of the present invention needs to complete the five classifications of IS on the BCI V3 dataset. The BCI V3 dataset requires the subjects to imagine that they are performing real speech movements, but do not move any vocal organs or make any sounds. The inducement of internal speech in the BCI V3 dataset involves both language function and motor planning. 15 healthy subjects aged 20-30 years participated in the experiment. 64 EEG channels were recorded at a sampling rate of 256Hz.

[0110] Among them, Thinking out loud, TOL collects EEG data when subjects produce orientation-related words, including three language tasks: inner speech, vocalized speech and visual state. The data of the inner speech task are used in the comparative experiment. The embodiment of the present invention needs to complete the four classifications of IS on the TOL dataset. The TOL dataset requires subjects to imagine their own voice issuing instructions and repeating them continuously, and explicitly requires not to focus on imagining the movements of the mouth and tongue, but to focus on inducing inner speech only through high-level language functions. 10 healthy subjects participated in the experiment, with an average age of 34 years old and a standard deviation of 10 years old. 128 EEG channels and 8 EOG / EMG channels were recorded at a sampling rate of 1024Hz.

[0111] In a feasible implementation, the five currently advanced methods include: EEGNet, CNN, deep metric learning CNN, multi-scale neural network and classification principal component analysis; the comparative methods used include: machine learning, deep learning based on representation learning and deep neural network model; as shown in Figure 1, the results of the BCI Competition V-3 dataset.

[0112] Table 1

[0113]

[0114] Among them, the embodiment of the present invention can naturally retain the EEG topology through graph representation learning, and obtain a more robust EEG representation through unsupervised contrastive learning, and further obtain the best decoding performance and stability.

[0115] Among them, the performance of five state-of-the-art methods applied to the TOL dataset is compared, including: EEGNet, bidirectional long short-term memory network, CNN, SVM and subject-independent meta-learning; the compared methods include machine learning methods, traditional deep learning models and meta-learning frameworks; the results of the Thinking Out Loud dataset are shown in Table 2.

[0116] Table 2

[0117]

[0118] Among them, compared with the BCI V3 dataset, the decoding performance of the TOL dataset is significantly reduced; this is because the TOL dataset explicitly requires the subjects to focus on the IS content rather than the movement image of the vocal organs. The embodiments of the present invention can more concisely represent the local and network information of the EEG when constructing the EEG representation, can reduce the negative impact of the small sample size, and introduce functional connectivity and spatial structure to more comprehensively characterize the topological structure of the brain network. Compared with the existing methods, the best decoding performance and stability can be improved.

[0119] Among them, Figure 4 An embodiment of the present invention provides a visualized brain topography map; a visual analysis is performed based on the visualized brain topography map; the brain topography map is used to visualize which brain regions are more important in IS decoding, providing interpretability for the proposed model; the node representation corresponding to each electrode learned by the embodiment of the present invention is correlated with the internal speech marker to obtain a brain topography map for each subject. Figure 4The average correlation of brain regions across all subjects is shown. The key brain regions for the BCI V3 dataset are the frontal lobe part near the central area (①) and the part near the left temporal lobe (②); the key brain region corresponding to TOL is the temporal lobe (③); the reason for this difference is that the two datasets have different methods of inner speech invocation. The premotor area corresponding to ① is one of the brain regions most associated with motor imagery and is involved in the planning and preparation of movement. The inferior frontal gyrus corresponding to ② contains the motor speech center, which is responsible for generating coordinated articulatory programs. Since the BCI V3 dataset requires subjects to imagine that they are performing real speech actions without actually moving the vocal organs, language-related motor imagery is involved; therefore, ① and ② are very important for the decoding of inner speech in the BCI V3 dataset; the temporal lobe where ③ is located plays a key role in speech perception and processing. The TOL dataset requires subjects to imagine that they are making command sounds, assuming that the human brain may process silent speech information in a similar way to processing external speech. The occipital lobe region ④ was determined to be the common key brain region for inner speech decoding. The occipital lobe is an intermediate hub between visual information and language processing. It is reasonable to assume that generating inner speech is automatically associated with visual images, such as related words or scenes. Thus, BSFG can identify key brain regions associated with IS, including both dataset-specific and common regions, showing generalization capabilities to different triggers, i.e., inner speech decoding that relies solely on language functions or includes motor planning.

[0120] The embodiment of the present invention first obtains an EEG signal sample data set; constructs an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions includes: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain network, and a spatial function fusion module; secondly, the initial graph contrast learning model constrained by brain network spatial functions is trained according to the EEG signal sample data set, and local features are extracted by the graph-based EEG representation construction module to obtain local features of nodes; network features are extracted by the graph-based EEG representation construction module to obtain network features; the local features of the nodes are combined with the network features to obtain space-based EEG graph representation and function-based EEG graph representation; The space-based EEG map representation and the function-based EEG map representation are input into the dual-stream graph contrast learning module constrained by the brain network for contrast learning, so as to obtain the space-based EEG map representation and the function-based EEG map representation after contrast learning; the space-based EEG map representation and the function-based EEG map representation after contrast learning are input into the spatial function fusion module to obtain the inner speech decoding prediction result; finally, according to the inner speech decoding prediction result, a trained graph contrast learning model constrained by the spatial function of the brain network is obtained; the EEG signal data to be decoded is obtained; the EEG signal data to be decoded is input into the trained graph contrast learning model constrained by the spatial function of the brain network to obtain the final inner speech decoding result.

[0121] The embodiment of the present invention introduces brain network prior knowledge into the feasibility of inner speech decoding. The embodiment of the present invention can comprehensively learn robust representation from functional and spatial perspectives. In order to comprehensively characterize the brain state related to inner speech, the embodiment of the present invention designs a graph-based EEG representation construction module to learn local and network features. In order to alleviate the impact of data scarcity, the embodiment of the present invention designs a dual-stream graph contrast learning module constrained by brain networks to generate more brain-like enhanced samples, and improves decoding performance by designing a superposition mechanism to fuse the graph representation information from the spatial stream and the functional stream.

[0122] Figure 5 is a block diagram of an inner speech decoding device based on brain network constraints and graph contrast learning according to an exemplary embodiment, wherein the device is used in an inner speech decoding method based on brain network constraints and graph contrast learning. Figure 5 The device includes an acquisition unit 310, a construction unit 320, a training unit 330 and a decoding unit 340. Wherein:

[0123] An acquisition unit 310 is used to acquire an EEG signal sample data set;

[0124] A construction unit 320 is used to construct an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions includes: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain networks, and a spatial function fusion module;

[0125] The training unit 330 is used to train the initial graph contrast learning model constrained by the spatial function of the brain network according to the EEG signal sample data set, extract local features through the graph-based EEG representation construction module to obtain local features of the nodes; extract network features through the graph-based EEG representation construction module to obtain network features; combine the local features of the nodes with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation; input the space-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning to obtain a space-based EEG graph representation after contrast learning; input the function-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning to obtain a function-based EEG graph representation after contrast learning; input the space-based EEG graph representation after contrast learning and the function-based EEG graph representation after contrast learning into the spatial function fusion module to obtain an internal speech decoding prediction result; obtain a trained graph contrast learning model constrained by the spatial function of the brain network according to the internal speech decoding prediction result;

[0126] The decoding unit 340 is used to obtain the EEG signal data to be decoded; input the EEG signal data to be decoded into the trained graph contrast learning model constrained by the brain network spatial function to obtain the final inner speech decoding result.

[0127] Optionally, the extracting local features through the graph-based EEG representation building module to obtain local features of the node includes:

[0128] According to the concept of graph, EEG information is lightweight represented in the form of graph structure; according to the preset number of frames, the electrode signal in the EEG signal sample data set is divided into multiple frames, and the signal power of each frequency band is calculated according to the preset number of frequency bands to obtain the power characteristics of multiple EEG electrodes;

[0129] According to the power features of the multiple EEG electrodes, a one-dimensional convolution layer is used to shrink the power features of the multiple EEG electrodes to a one-dimensional vector, and local feature representations of multiple nodes are output; wherein the local feature representation of the node can be obtained by the following formula (1):

[0130] (1)

[0131] in, For the Local feature representation of nodes; Indicates The power characteristics of the EEG electrodes, , F represents the number of frequency bands, T represents the number of frames; R represents a real number set; represents the weight matrix, This is a hyperparameter representing the size of the convolution kernel, and the value set is the same as the stride; represents the bias vector to be trained; represents the linear rectification function.

[0132] Optionally, the extracting network features through the graph-based EEG representation building module to obtain network features; combining the local features of the nodes with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation, comprises:

[0133] According to the graph structure representation, a spatial connectivity matrix is ​​calculated based on a predefined spatial relationship; a random initial first trainable mask matrix is ​​used to perform adjacency matrix sparsification on the spatial connectivity matrix to obtain a space-based adjacency matrix; a functional connectivity matrix is ​​calculated based on the Pearson correlation coefficient, and an adjacency matrix sparsification is performed on the functional connectivity matrix using a random initial second trainable mask matrix to obtain a function-based adjacency matrix; wherein the process of obtaining the space-based adjacency matrix can be expressed by the following formula (2):

[0134] (2)

[0135] in, Represents a space-based adjacency matrix; represents the angle between the p-th electrode and the q-th electrode; represents the Euclidean distance between the p-th electrode and the q-th electrode; represents the cosine function of the angle between the pth electrode and the qth electrode; represents the first trainable mask matrix randomly initialized;

[0136] The process of obtaining the function-based adjacency matrix can be expressed by the following formula (3):

[0137] (3)

[0138] in, Represents a function-based adjacency matrix; represents the EEG signal sequence of the pth electrode; represents the EEG signal sequence of the qth electrode; represents the standard deviation of the EEG signal sequence of the pth electrode; represents the standard deviation of the EEG signal sequence of the qth electrode; represents the covariance between the EEG signal sequence of the p-th electrode and the EEG signal sequence of the q-th electrode; represents a randomly initialized second trainable mask matrix; , The value range is (-1,1), The value range is (-1, 1) and does not exceed r%, the value of the element r∈(0, 100) is greater than 0; r represents a pre-set hyperparameter used to control the sparsity of the adjacency matrix; N represents the total number of EEG electrodes.

[0139] According to the local feature representations of multiple nodes and the space-based adjacency matrix, a group of edges based on spatial connections between nodes are obtained; the local feature representations of multiple nodes, the space-based adjacency matrix and the group of edges based on spatial connections between nodes are combined to obtain a space-based EEG graph representation; according to the local feature representations of multiple nodes and the function-based adjacency matrix, a group of edges based on functional connections between nodes are obtained; the local feature representations of multiple nodes, the function-based adjacency matrix and the group of edges based on functional connections between nodes are combined to obtain a function-based EEG graph representation.

[0140] Optionally, the dual-stream graph contrast learning module constrained by the brain network includes: a graph enhancement module constrained by the brain network, an encoder based on a graph isomorphic network, and a projection layer.

[0141] Optionally, the space-based EEG image representation is input into the dual-stream image contrast learning module constrained by the brain network for contrast learning to obtain the space-based EEG image representation after contrast learning; the function-based EEG image representation is input into the dual-stream image contrast learning module constrained by the brain network for contrast learning to obtain the function-based EEG image representation after contrast learning, including:

[0142] The spatial-based EEG graph representation is input into the graph enhancement module constrained by the brain network, and the graph data of the spatial branch is enhanced with the brain network characteristics as the constraint condition to obtain the spatial-enhanced EEG graph representation; wherein the process of enhancing the graph data of the spatial branch can be expressed by the following formula (4):

[0143] (4)

[0144] in, Represents the adjacency matrix based on spatial enhancement; express The absolute values ​​of the non-zero elements in the array are sorted at the top. The element set of k represents the hyperparameter that controls the proportion of the elements retained in the original adjacency matrix; Representation and Random number matrices of the same size, with elements ranging from 0 to 1; Indicates from A random selection from the zero elements in The edge set corresponding to the element;

[0145] In the function-based EEG graph representation and the graph enhancement module constrained by the brain network, the graph data of the functional branch is enhanced with the brain network characteristics as the constraint to obtain the EEG graph representation based on function enhancement; wherein the process of enhancing the graph data of the functional branch can be expressed by the following formula (5):

[0146] (5)

[0147] in, Represents the adjacency matrix based on feature enhancement; Indicates from A random selection from the zero elements in The edge set corresponding to the element;

[0148] The space-based EEG graph representation, the function-based EEG graph representation, the space-enhanced EEG graph representation and the function-enhanced EEG graph representation are input into the encoder composed of the graph isomorphism network and the projection layer for comparative learning to obtain a graph representation of the EEG signal sample data set after comparative learning; wherein the graph representation of the EEG signal sample data set after comparative learning includes: a space-based EEG graph representation after comparative learning and a function-based EEG graph representation after comparative learning, wherein the update rule of the node representation of each graph representation in the graph isomorphism network can be expressed by the following formula (6):

[0149] (6)

[0150] in, represents the updated node representation; Represents the node representation, represents the adjacency matrix of the EEG graph, represents the identity matrix, It is a learnable parameter that controls the influence of each node's own attributes. MLP represents a multi-layer perceptron model.

[0151] Optionally, the spatial function fusion module includes: two basic classifiers and one meta classifier.

[0152] Optionally, the space-based EEG map representation and the function-based EEG map representation are input into the space-function fusion module to obtain the inner speech decoding prediction result, including:

[0153] The graph representation of the EEG signal sample data set after contrastive learning is input into the spatial function fusion module, and the K-means algorithm is used to cluster the graph representation of the spatial branch to obtain the graph representation clustering center based on the spatial branch and the clusters based on the spatial clustering. The K-means algorithm is used to cluster the graph representation of the functional branch to obtain the graph representation clustering center based on the functional branch and the clusters based on the functional clustering. The clustering process can be expressed by the following formula (7):

[0154] (7)

[0155] in, The representation graph represents the cluster centers; Represents the cluster after clustering; Represents the projection layer; represents the encoder; Node representation representing the EEG graph; Represents the adjacency matrix of the EEG graph; represents the number of clusters, which is consistent with the number of internal speech instructions to be decoded; i represents the branch subscript, where i=1 represents the spatial branch and i=2 represents the functional branch;

[0156] According to the graph based on spatial branches, the cluster centers are represented and the space-based fitness score is calculated; according to the graph based on functional branches, the cluster centers are represented and the function-based fitness score is calculated; wherein, the fitness score calculation formula can be expressed by the following formula (8):

[0157] (8)

[0158] in, Indicates The branch clusters, ; express A sample of the graph in Indicates The branch The center of a cluster; represents the fitness score, when It indicates the fitness score based on space when represents the fitness score based on the function;

[0159] Among them, the weight based on the space branch is calculated according to the fitness score based on the space; the weight based on the function branch is calculated according to the fitness score based on the function. The weight calculation process based on the space branch and the weight calculation process based on the function branch can be expressed by the following formula (9):

[0160] (9)

[0161] in, represents the fitness score of the spatial branch; Represents the fitness score of the functional branch;

[0162] It represents the weight of the output result of the branch basic classifier in the meta-classifier. The value range of is 1 or 2, where i=1 represents the spatial branch and i=2 represents the functional branch;

[0163] Inputting the clusters after the space-based clustering into the first basic classifier, and outputting the space-based decoding results; using the weights based on the space branches, weighting the space-based decoding results to obtain weighted space-based decoding results;

[0164] Inputting the clusters after function-based clustering into the second basic classifier, and outputting the function-based decoding results; using weights based on function branches to perform weighted processing on the function-based decoding results, and obtaining weighted function-based decoding results;

[0165] The weighted spatial-based decoding results and the weighted functional-based decoding results are input into a meta-classifier to obtain inner speech decoding prediction results.

[0166] The embodiment of the present invention first obtains an EEG signal sample data set; constructs an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions includes: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain network, and a spatial function fusion module; secondly, the initial graph contrast learning model constrained by brain network spatial functions is trained according to the EEG signal sample data set, and local features are extracted by the graph-based EEG representation construction module to obtain local features of nodes; network features are extracted by the graph-based EEG representation construction module to obtain network features; the local features of the nodes are combined with the network features to obtain space-based EEG graph representation and function-based EEG graph representation; The space-based EEG map representation and the function-based EEG map representation are input into the dual-stream graph contrast learning module constrained by the brain network for contrast learning, so as to obtain the space-based EEG map representation and the function-based EEG map representation after contrast learning; the space-based EEG map representation and the function-based EEG map representation after contrast learning are input into the spatial function fusion module to obtain the inner speech decoding prediction result; finally, according to the inner speech decoding prediction result, a trained graph contrast learning model constrained by the spatial function of the brain network is obtained; the EEG signal data to be decoded is obtained; the EEG signal data to be decoded is input into the trained graph contrast learning model constrained by the spatial function of the brain network to obtain the final inner speech decoding result. The embodiment of the present invention introduces brain network prior knowledge into the feasibility of inner speech decoding. The embodiment of the present invention can comprehensively learn robust representation from functional and spatial perspectives. In order to comprehensively characterize the brain state related to inner speech, the embodiment of the present invention designs a graph-based EEG representation construction module to learn local and network features. In order to alleviate the impact of data scarcity, the embodiment of the present invention designs a dual-stream graph contrast learning module constrained by brain networks to generate more brain-like enhanced samples, and improves decoding performance by designing a superposition mechanism to fuse the graph representation information from the spatial stream and the functional stream.

[0167] Figure 6 is a structural schematic diagram of an inner speech decoding device based on brain network constraint and graph contrast learning provided by an embodiment of the present invention, such as Figure 6 As shown, the inner speech decoding device based on brain network constraint and graph contrast learning may include the above Figure 5 The inner speech decoding device based on brain network constraints and graph contrast learning is shown. Optionally, the inner speech decoding device based on brain network constraints and graph contrast learning 410 may include a first processor 2001.

[0168] Optionally, the inner speech decoding device 410 based on brain network constraint and graph contrast learning may further include a memory 2002 and a transceiver 2003 .

[0169] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0170] Combine the following Figure 6 The components of the internal speech decoding device 410 based on brain network constraint and graph contrast learning are specifically introduced as follows:

[0171] The first processor 2001 is the control center of the internal speech decoding device 410 based on brain network constraint and graph contrast learning, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention, such as one or more microprocessors (digital signal processor, DSP), or one or more field programmable gate arrays (field programmable gate array, FPGA).

[0172] Optionally, the first processor 2001 can perform various functions of the internal speech decoding device 410 based on brain network constraints and graph contrast learning by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0173] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 6 CPU0 and CPU1 are shown in FIG.

[0174] In a specific implementation, as an embodiment, the inner speech decoding device 410 based on brain network constraint and graph contrast learning may also include multiple processors, such as Figure 6 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0175] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0176] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 6 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0177] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0178] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 6 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0179] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 6 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0180] It should be noted that Figure 6 The structure of the internal speech decoding device 410 based on brain network constraints and graph contrast learning shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0181] In addition, the technical effects of the internal speech decoding device 410 based on brain network constraints and graph contrast learning can refer to the technical effects of the internal speech decoding method based on brain network constraints and graph contrast learning described in the above method embodiment, and will not be repeated here.

[0182] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0183] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0184] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0185] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0186] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0187] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0188] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0189] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0190] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0191] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0193] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0194] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An inner speech decoding method based on brain network constraints and graph contrast learning, characterized in that: The method comprises: S1. Obtain EEG signal sample data set; S2. constructing an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions comprises: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain networks, and a spatial function fusion module; S3, training the initial graph contrast learning model constrained by brain network spatial function according to the EEG signal sample data set, extracting local features through the graph-based EEG representation construction module to obtain local features of nodes; extracting network features through the graph-based EEG representation construction module to obtain network features; combining the local features of the nodes with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation; S4, inputting the space-based EEG image representation into the dual-stream image contrast learning module constrained by the brain network for contrast learning, and obtaining the space-based EEG image representation after contrast learning; inputting the function-based EEG image representation into the dual-stream image contrast learning module constrained by the brain network for contrast learning, and obtaining the function-based EEG image representation after contrast learning; S5, inputting the space-based EEG graph representation after contrastive learning and the function-based EEG graph representation after contrastive learning into the spatial function fusion module to obtain the inner speech decoding prediction result; according to the inner speech decoding prediction result, obtaining a trained graph contrastive learning model constrained by the spatial function of the brain network; S6. Obtain EEG signal data to be decoded; input the EEG signal data to be decoded into the trained graph contrast learning model constrained by the brain network spatial function to obtain the final inner speech decoding result.

2. The inner speech decoding method based on brain network constraint and graph contrast learning according to claim 1, characterized in that: The S3 extracts local features through the graph-based EEG representation building module to obtain local features of the node, including: S31. According to the concept of graph, the EEG information is lightweight represented in the form of a graph structure; according to a preset number of frames, the electrode signal in the EEG signal sample data set is divided into multiple frames, and the signal power of each frequency band is calculated according to a preset number of frequency bands to obtain power characteristics of multiple EEG electrodes; S32. According to the power features of the multiple EEG electrodes, a one-dimensional convolution layer is used to shrink the power features of the multiple EEG electrodes to a one-dimensional vector, and the local feature representations of the multiple nodes are output; wherein the local feature representation of the node is obtained by the following formula (1): （1） in, For the Local feature representation of nodes; Indicates The power characteristics of the EEG electrodes, , F represents the number of frequency bands, T represents the number of frames; R represents a real number set; represents the weight matrix, This is a hyperparameter representing the size of the convolution kernel, and the value set is the same as the stride; represents the bias vector to be trained; represents the linear rectification function.

3. The inner speech decoding method based on brain network constraint and graph contrast learning according to claim 1, characterized in that: The S3 extracts network features through the graph-based EEG representation building module to obtain network features; Combining the local features of the node with the network features to obtain a space-based EEG map representation and a function-based EEG map representation, including: S33, according to the graph structure representation, the spatial connectivity matrix is ​​calculated based on the predefined spatial relationship; the spatial connectivity matrix is ​​sparsely processed by using a random initial first trainable mask matrix to obtain a space-based adjacency matrix; the functional connectivity matrix is ​​calculated based on the Pearson correlation coefficient, and the functional connectivity matrix is ​​sparsely processed by using a random initial second trainable mask matrix to obtain a function-based adjacency matrix; The process of obtaining the space-based adjacency matrix is ​​expressed by the following formula (2): （2） in, Represents a space-based adjacency matrix; represents the angle between the p-th electrode and the q-th electrode; represents the Euclidean distance between the p-th electrode and the q-th electrode; represents the cosine function of the angle between the pth electrode and the qth electrode; represents the first trainable mask matrix randomly initialized; p and q represent the number of electrodes; The process of obtaining the function-based adjacency matrix is ​​expressed by the following formula (3): （3） in, Represents a function-based adjacency matrix; represents the EEG signal sequence of the pth electrode; represents the EEG signal sequence of the qth electrode; represents the standard deviation of the EEG signal sequence of the pth electrode; represents the standard deviation of the EEG signal sequence of the qth electrode; represents the covariance between the EEG signal sequence of the p-th electrode and the EEG signal sequence of the q-th electrode; represents a randomly initialized second trainable mask matrix; , The value range is (-1,1). The value range is (-1, 1) and does not exceed r%. r represents a pre-set hyperparameter used to control the sparsity of the adjacency matrix. The value of the element r∈(0, 100) is greater than 0. N represents the total number of EEG electrodes. S34. According to the local feature representation of multiple nodes and the space-based adjacency matrix, a set of edges based on spatial connection between nodes is obtained; the local feature representation of multiple nodes, the space-based adjacency matrix and the set of edges based on spatial connection between nodes are combined to obtain a space-based EEG graph representation; according to the local feature representation of multiple nodes and the function-based adjacency matrix, a set of edges based on functional connection between nodes is obtained; the local feature representation of multiple nodes, the function-based adjacency matrix and the set of edges based on functional connection between nodes are combined to obtain a function-based EEG graph representation.

4. The inner speech decoding method based on brain network constraint and graph contrast learning according to claim 1, characterized in that: The brain network-constrained dual-stream graph contrast learning module of S2 includes: a brain network-constrained graph enhancement module, an encoder based on a graph isomorphic network, and a projection layer.

5. The inner speech decoding method based on brain network constraint and graph contrast learning according to claim 4, characterized in that: The step S4 of inputting the space-based EEG image representation into the dual-stream image contrast learning module constrained by the brain network for contrast learning to obtain the space-based EEG image representation after contrast learning; and inputting the function-based EEG image representation into the dual-stream image contrast learning module constrained by the brain network for contrast learning to obtain the function-based EEG image representation after contrast learning, includes: S41, inputting the spatial-based EEG graph representation into the graph enhancement module constrained by the brain network, enhancing the graph data of the spatial branch with the brain network characteristics as the constraint condition, and obtaining the EEG graph representation based on spatial enhancement; wherein the process of enhancing the graph data of the spatial branch is expressed by the following formula (4): （4） in, Represents the adjacency matrix based on spatial enhancement; express The absolute values ​​of the non-zero elements in the array are sorted at the top. The element set of k represents the hyperparameter that controls the proportion of the elements retained in the original adjacency matrix; Representation and Random number matrices of the same size, with elements ranging from 0 to 1; Indicates from A random selection from the zero elements in The edge set corresponding to the element; S42, inputting the function-based EEG graph representation into the graph enhancement module constrained by the brain network, enhancing the graph data of the functional branch with the brain network characteristics as the constraint, and obtaining the EEG graph representation based on function enhancement; wherein the process of enhancing the graph data of the functional branch is expressed by the following formula (5): （5） in, Represents the adjacency matrix based on feature enhancement; Indicates from A random selection from the zero elements in The edge set corresponding to the element; S43, inputting the space-based EEG graph representation, the function-based EEG graph representation, the space-enhanced EEG graph representation and the function-enhanced EEG graph representation into the encoder and projection layer composed of the graph isomorphism network for comparative learning, and obtaining the graph representation of the EEG signal sample data set after comparative learning; wherein the graph representation of the EEG signal sample data set after comparative learning includes: the space-based EEG graph representation after comparative learning and the function-based EEG graph representation after comparative learning, wherein the update rule of the node representation of each graph representation in the graph isomorphism network is expressed by the following formula (6): （6） in, represents the updated node representation; Represents the node representation, represents the adjacency matrix of the EEG graph, represents the identity matrix, It is a learnable parameter that controls the influence of each node's own attributes. MLP represents a multi-layer perceptron model.

6. The inner speech decoding method based on brain network constraint and graph contrast learning according to claim 1, characterized in that: The spatial function fusion module of S2 includes: two basic classifiers and one meta classifier.

7. The inner speech decoding method based on brain network constraint and graph contrast learning according to claim 6, characterized in that: The step S5 inputs the space-based EEG image representation after contrastive learning and the function-based EEG image representation after contrastive learning into the space-function fusion module to obtain the inner speech decoding prediction result, including: S51, inputting the graph representation of the EEG signal sample data set after contrast learning into the spatial function fusion module, using the K-means algorithm to perform clustering processing on the graph representation of the spatial branch, obtaining the graph representation clustering center based on the spatial branch and the clusters based on the spatial clustering, and using the K-means algorithm to perform clustering processing on the graph representation of the functional branch, obtaining the graph representation clustering center based on the functional branch and the clusters based on the functional clustering; wherein the clustering process is represented by the following formula (7): （7） in, The representation graph represents the cluster centers; Represents the cluster after clustering; Represents the projection layer; represents the encoder; Node representation representing the EEG graph; Represents the adjacency matrix of the EEG graph; represents the number of clusters, which is consistent with the number of internal speech instructions to be decoded; i represents the branch subscript, where i=1 represents the spatial branch and i=2 represents the functional branch; S52, calculating a space-based fitness score according to the cluster center represented by the graph based on the space branch; According to the graph representation of cluster centers based on functional branches, the function-based fitness score is calculated; wherein the fitness score calculation formula is expressed by the following formula (8): （8） in, Indicates The branch clusters, ; express A sample of the graph in Indicates The first branch The center of a cluster; represents the fitness score, when It indicates the fitness score based on space when represents the fitness score based on the function; According to the fitness score based on space, the weight of the space-based branch is calculated; according to the fitness score based on function, the weight of the function-based branch is calculated. The calculation process of the weight based on space branch and the weight based on function branch is expressed by the following formula (9): （9） in, represents the fitness score of the spatial branch; Represents the fitness score of the functional branch; It represents the weight of the output result of the branch basic classifier in the meta-classifier. The value range of is 1 or 2, where i=1 represents the spatial branch and i=2 represents the functional branch; S53, inputting the clusters after spatial clustering into the first basic classifier, and outputting the spatial decoding results; using the weights based on the spatial branches, weighting the spatial decoding results to obtain the weighted spatial decoding results; S54, inputting the clusters after function-based clustering into the second basic classifier, and outputting the function-based decoding results; using the weights based on the function branches, weighting the function-based decoding results to obtain weighted function-based decoding results; S55. Input the weighted spatial-based decoding result and the weighted functional-based decoding result into a meta-classifier to obtain an inner speech decoding prediction result.

8. An inner speech decoding device based on brain network constraints and graph contrast learning, the inner speech decoding device based on brain network constraints and graph contrast learning is used to implement the inner speech decoding method based on brain network constraints and graph contrast learning as described in any one of claims 1-7, characterized in that: The device comprises: An acquisition unit, used for acquiring an EEG signal sample data set; A construction unit, used to construct an initial graph contrast learning model constrained by brain network spatial functions; the initial graph contrast learning model constrained by brain network spatial functions comprises: a graph-based EEG representation construction module, a dual-stream graph contrast learning module constrained by brain network, and a spatial function fusion module; A training unit, used for training the initial graph contrast learning model constrained by the spatial function of the brain network according to the EEG signal sample data set, extracting local features through the graph-based EEG representation construction module to obtain local features of nodes; extracting network features through the graph-based EEG representation construction module to obtain network features; combining the local features of the nodes with the network features to obtain a space-based EEG graph representation and a function-based EEG graph representation; inputting the space-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning to obtain a space-based EEG graph representation after contrast learning; inputting the function-based EEG graph representation into the dual-stream graph contrast learning module constrained by the brain network for contrast learning to obtain a function-based EEG graph representation after contrast learning; inputting the space-based EEG graph representation after contrast learning and the function-based EEG graph representation after contrast learning into the spatial function fusion module to obtain an internal speech decoding prediction result; according to the internal speech decoding prediction result, a trained graph contrast learning model constrained by the spatial function of the brain network is obtained; The decoding unit is used to obtain the EEG signal data to be decoded; input the EEG signal data to be decoded into the trained graph contrast learning model constrained by the brain network spatial function to obtain the final inner speech decoding result.

9. An inner speech decoding device based on brain network constraint and graph contrast learning, characterized in that: The inner speech decoding device based on brain network constraint and graph contrast learning includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images