Brain-computer interface signal recognition method, system and electronic equipment

Abstract

This specification discloses a brain-computer interface signal recognition method, system and electronic device, which can achieve high-precision recognition and classification of MI-EEG / MEG signals. The method includes: acquiring original EEG / MEG signals and preprocessing them to generate EEG / MEG measurement signals, extracting corresponding global feature information in the measurement space; converting the EEG / MEG measurement signals into EEG / MEG source signals, extracting corresponding detail feature information in the source space; using a cross-space fusion convolutional neural network to obtain fusion feature information of global feature information and detail feature information, and classifying according to the fusion feature information to determine the movement intention corresponding to the original EEG / MEG signal. The system includes an original signal acquisition module, a frequency band analysis module, a global feature extraction module, a brain source imaging module, a detail feature extraction module and a cross-space fusion module. The processor in the electronic device is used to execute the brain-computer interface signal recognition method.

Description

Technical Field

[0001] The present invention relates to the field of brain information processing technology, and in particular to a brain-computer interface signal recognition method, system and electronic equipment. Background Art

[0002] A brain-computer interface (BCI) is a technology that enables information exchange between the brain and external devices by building pathways that are independent of peripheral nerves and muscle tissue. It has broad application prospects in fields such as gaming and entertainment, industrial processes, aerospace, and rehabilitation medicine and engineering. BCIs based on non-invasive electroencephalogram (EEG) / magnetoencephalography (MEG) signals can monitor large-scale neuronal activity throughout the brain adjacent to the skull in a cost-effective and risk-free manner, and are widely used in the field of brain activity recording. The motor imagery (MI) paradigm is one of the main BCI paradigms. Motor imagery electroencephalogram (MI-EEG / MEG) signals have a spatial distribution characteristic: different motor imagery tasks result in different activation areas in the subject's cerebral cortex, and the corresponding EEG / MEG signals also differ in their spatial distribution. Methods based on this characteristic to identify and decode motor intentions are widely used in MI-BCI systems.

[0003] One decoding method for MI-EEG / MEG is BCI decoding based on the measurement space, which extracts and identifies features based on multi-lead EEG / MEG in the scalp space. This decoding method does not accurately express the fine spatial features of the BCI signal, which restricts further improvement in classification accuracy. Another decoding method for MI-EEG / MEG is BCI decoding based on the source space. This method uses EEG / MEG Source Imaging (ESI) technology to map the signal on the scalp to the source distribution of the cortex to complete the EEG / MEG signal tracing. While retaining high temporal resolution, it also improves spatial resolution. However, this decoding method has the limitations of non-realistic signals and the inability to represent global information. Summary of the Invention

[0004] In view of this, the embodiments of this specification provide a brain-computer interface signal recognition method, system and electronic equipment, which can achieve high-precision recognition and classification of MI-EEG / MEG signals.

[0005] In one aspect, an embodiment of this specification provides a method for identifying brain-computer interface signals, comprising:

[0006] Acquire raw EEG / MEG signals, and pre-process the raw EEG / MEG signals to generate EEG / MEG measurement signals;

[0007] Determine a main rhythm passband of the EEG / MEG measurement signal by performing frequency band analysis on the EEG / MEG measurement signal, and extract a main rhythm measurement signal corresponding to the main rhythm passband based on the EEG / MEG measurement signal;

[0008] Dividing the main rhythm measurement signal into multiple categories of sub-signals, and performing co-spatial feature extraction on the multiple categories of sub-signals to generate global feature information;

[0009] Using a brain source imaging algorithm to convert the EEG / MEG measurement signal into an EEG / MEG source signal;

[0010] A clustering algorithm is used to screen and determine key areas that are highly relevant to the task in the EEG / MEG source signals, and time series signals of the key areas are extracted as detailed feature information;

[0011] A cross-space fusion convolutional neural network is used to obtain fused feature information of the global feature information and the detailed feature information, and classification is performed according to the fused feature information to determine the movement intention corresponding to the original EEG / MEG signal.

[0012] Optionally, preprocessing the original EEG / MEG signal to generate an EEG / MEG measurement signal includes:

[0013] Filtering the original EEG / MEG signal, removing eye movement artifacts, and performing baseline correction processing to generate an EEG / MEG corrected signal;

[0014] Data amplification is performed on the EEG / MEG correction signal to generate the EEG / MEG measurement signal.

[0015] Optionally, determining the main rhythm passband of the EEG / MEG measurement signal by performing frequency band analysis on the EEG / MEG measurement signal includes:

[0016] Performing time-frequency decomposition on the EEG / MEG measurement signal, dividing the EEG / MEG measurement signal into a plurality of sub-frequency bands, and determining sub-band energy coefficients corresponding to the plurality of sub-frequency bands;

[0017] The sub-frequency band with the largest sub-band energy coefficient is used as a seed point, and a seed growth method is adopted to select a plurality of sub-frequency bands to form the main rhythm passband.

[0018] Optionally, dividing the main rhythm measurement signal into multiple categories of sub-signals, and performing co-spatial feature extraction on the multiple categories of sub-signals to generate global feature information, includes:

[0019] Grouping the main rhythm measurement signals and performing common space feature extraction so as to maximize the difference between the sub-signals of each group and the sub-signals of other groups in the grouping results;

[0020] The spatial features corresponding to the sub-signals are determined, and the spatial features corresponding to multiple groups of the sub-signals are spliced to generate the global feature information.

[0021] Optionally, using a brain source imaging algorithm to convert the EEG / MEG measurement signal into an EEG / MEG source signal includes:

[0022] Determining the original data source corresponding to the EEG / MEG measurement signal, and determining the head model and source model corresponding to the original data source;

[0023] registering the EEG / MEG poles corresponding to the EEG / MEG measurement signals with the head model;

[0024] determining a conduction matrix between a measurement space and a source space based on the head model and the source model;

[0025] converting the EEG / MEG measurement signal into the EEG / MEG source signal according to the conduction matrix;

[0026] Wherein, the original data sources include private data sets and public data sets;

[0027] The determining of the header model and the source model corresponding to the source of the original data includes:

[0028] In response to the source of the original data being the public dataset, selecting a common header model and source model corresponding to the public dataset;

[0029] In response to the original data source being the private dataset, magnetic resonance anatomical information of a subject corresponding to the private dataset is acquired, and a private head model and a source model corresponding to the subject are created based on the magnetic resonance anatomical information.

[0030] Optionally, the EEG / MEG source signal includes current dipole distribution information in the source space;

[0031] A clustering algorithm is used to screen and identify key areas of high task relevance in the EEG / MEG signal, including:

[0032] A clustering algorithm is used to cluster multiple current dipoles in the source space using activation intensity and distribution position as constraints, multiple concentrated clusters in a strongly activated state are screened and determined, and the key area is determined based on the multiple concentrated clusters.

[0033] Optionally, using a cross-space fusion convolutional neural network to obtain fused feature information of the global feature information and the detailed feature information, and performing classification based on the fused feature information to determine the movement intention corresponding to the original EEG / MEG signal, including:

[0034] Extracting implicit global feature information from the global feature information using a convolutional neural network;

[0035] Extracting implicit detail feature information from the detail feature information using a convolutional neural network;

[0036] Fusing the implicit global feature information with the implicit detail feature information to generate a fused feature matrix;

[0037] Deep feature mining and integration are performed on the fusion feature matrix, and classification processing is performed on the integrated deep features, and the movement intention is determined according to the classification results.

[0038] Optionally, extracting implicit detail feature information from the detail feature information using a convolutional neural network includes:

[0039] The detail feature information is processed successively using convolutional layers with different receptive fields, and detail time information and detail space information are extracted from the detail feature information, wherein the implicit detail feature information includes the detail time information and the detail space information.

[0040] In a second aspect, the embodiments of this specification further provide a brain-computer interface signal recognition system, comprising:

[0041] The original signal acquisition module is used to obtain the original EEG / MEG signals and pre-process the original EEG / MEG signals to generate EEG / MEG measurement signals;

[0042] a frequency band analysis module, configured to perform frequency band analysis on the EEG / MEG measurement signal, determine a main rhythm passband of the EEG / MEG measurement signal, and extract a main rhythm measurement signal corresponding to the main rhythm passband based on the EEG / MEG measurement signal;

[0043] A global feature extraction module is used to divide the main rhythm measurement signal into multiple categories of sub-signals, and perform common space feature extraction on the multiple categories of sub-signals to generate global feature information;

[0044] A brain source imaging module, configured to convert the EEG / MEG measurement signals into EEG / MEG source signals using a brain source imaging algorithm;

[0045] A detail feature extraction module is used to use a clustering algorithm to screen and determine key areas that are highly relevant to the task in the EEG / MEG source signals, and extract time series signals of the key areas as detail feature information;

[0046] A cross-space fusion module is used to use a cross-space fusion convolutional neural network to obtain fused feature information of the global feature information and the detailed feature information, and to classify the fused feature information to determine the movement intention corresponding to the original EEG / MEG signal.

[0047] In a third aspect, an embodiment of this specification also provides a cross-space fused MI-EEG / MEG recognition electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the brain-computer interface signal recognition method as described in the first aspect is implemented.

[0048] As can be seen from the above, the brain-computer interface signal recognition method, system, and electronic device provided in the embodiments of this specification have the following beneficial technical effects:

[0049] The original EEG / MEG signal is preprocessed to generate the EEG / MEG measurement information. In the measurement space, the corresponding specific main rhythm is determined for the EEG / MEG measurement signal, the corresponding main rhythm measurement signal is extracted, and the global feature information in the measurement space is determined based on the main rhythm feature signal. For the source space, the EEG / MEG measurement signal is first mapped into the EEG / MEG source signal in the source space, and then the key area in the EEG / MEG source signal is determined, and the time series signal corresponding to the key area is extracted as the detail feature information in the source space. Then, a cross-space fusion convolutional neural network is used to extract the global feature information and the implicit feature information corresponding to the detail feature information and fuse them. Finally, classification and identification are performed based on the fused feature information that covers both the global feature information of the measurement space and the detail feature information of the source space. In this way, the global feature information of the measurement space is fused with the detail feature information of the source space. This method can overcome the non-real and non-global limitations of the source space decoding method, improve the spatial resolution of the measurement space decoding method, and achieve high-precision recognition and classification of MI-EEG / MEG signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0051] Figure 1 A schematic diagram of a brain-computer interface signal recognition method provided by one or more optional embodiments of this specification is shown;

[0052] Figure 2A schematic diagram of a method for preprocessing the original EEG / MEG signals in a brain-computer interface signal recognition method provided by one or more optional embodiments of this specification is shown;

[0053] Figure 3 A schematic diagram of a method for determining a main rhythm passband in a brain-computer interface signal recognition method provided by one or more optional embodiments of this specification is shown;

[0054] Figure 4 A schematic diagram of a method for generating global feature information in a brain-computer interface signal recognition method provided by one or more optional embodiments of this specification is shown;

[0055] Figure 5 A schematic diagram of a method for converting the EEG / MEG measurement signal into an EEG / MEG source signal in a brain-computer interface signal recognition method provided by one or more optional embodiments of this specification is shown;

[0056] Figure 6 A schematic diagram of a method for extracting fusion feature information and determining motion intention by fusing convolutional neural networks across space in a brain-computer interface signal recognition method provided by one or more optional embodiments of this specification is shown;

[0057] Figure 7 A schematic diagram of a cross-space fusion convolutional neural network structure in a brain-computer interface signal recognition method provided by one or more optional embodiments of this specification is shown;

[0058] Figure 8 A schematic diagram of the structure of a brain-computer interface signal recognition system provided by one or more optional embodiments of this specification is shown;

[0059] Figure 9 A schematic diagram of the structure of a cross-space fusion MI-EEG / MEG recognition electronic device provided by one or more optional embodiments of this specification is shown. DETAILED DESCRIPTION

[0060] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0061] Brain-computer interface (BCI) is a technology that enables information exchange between the brain and external devices by building pathways that are independent of peripheral nerves and muscle tissue. It has broad application prospects in gaming and entertainment, industrial processes, aerospace, and rehabilitation medicine and engineering. BCI based on non-invasive EEG / MEG (Electroencephalogram) signals can monitor large-scale neuronal activity of the entire brain adjacent to the skull in a low-cost and risk-free manner, and is widely used in brain activity recording methods in the field of BCI. The motor imagery (MI) paradigm is one of the main paradigms of BCI. Motor imagery EEG / MEG signals (MI-EEG / MEG) have the characteristic of spatial distribution, that is, the corresponding activation areas of the subject's cerebral cortex are different for different motor imagery tasks, and the corresponding EEG / MEG acquired will also have differences in spatial distribution. Based on this characteristic, methods for identifying and decoding motor intentions are widely used in MI-BCI systems.

[0062] One approach to decoding MI-EEG / MEG is measurement-space BCI decoding, which extracts and identifies features from multi-channel EEG / MEG signals in the scalp space. In recent years, methods for signal classification based on extracting the time, frequency, and spatial characteristics of multi-channel EEG / MEG signals have made considerable progress. One study has achieved three-dimensional control of a virtual helicopter by directly extracting the difference in the spectral amplitude of the mu rhythm between left and right EEG / MEG signals. Despite the rapid development of measurement-space BCI in recent years, various limitations remain. While EEG / MEG possesses extremely high temporal resolution, its spatial resolution is low relative to the number of intracranial neural sources, and this cannot be addressed by simply increasing the number of electrodes. Because the signals recorded by each EEG / MEG electrode are the result of coupling from multiple intracranial neural sources, the BCI signal's fine spatial features are inaccurately represented, hindering further improvements in classification accuracy.

[0063] Another decoding approach for MI-EEG / MEG is source-space-based BCI decoding. This approach uses EEG / MEG Source Imaging (ESI) to map scalp signals onto cortical source distributions, tracing the EEG / MEG signal's origin. This approach improves spatial resolution while maintaining high temporal resolution. However, ESI is inherently a model-based neuroimaging technique. On the one hand, the signals it provides are computationally derived rather than real. On the other hand, the introduction of a large number of dipoles in the source space can easily lead to overfitting. Therefore, it is necessary to manually select dipoles that are highly relevant to the MI task, a process that inevitably introduces artifacts or spurious information. Overall, some brain regions after this selection process fail to represent global cortical information. While source-space-based decoding offers numerous advantages and has demonstrated certain superiority, overcoming its limitations of being unrealistic and localized remains a challenge.

[0064] In response to the above problems, the purpose of the embodiments of this specification is to propose a cross-space fusion convolutional neural network (CS-CNN) algorithm to extract customized intrinsic features of MI-EEG / MEG signals in the measurement space and source space respectively, and feed the spatiotemporal and frequency domain features of the two spaces into the convolutional neural network. By mining and fusing deep-level features, MI-EEG / MEG signal recognition and classification can be achieved.

[0065] Based on the above objectives, on one hand, an embodiment of this specification provides a brain-computer interface signal recognition method.

[0066] like Figure 1 As shown, one or more optional embodiments of this specification provide a brain-computer interface signal recognition method, including:

[0067] S1: Acquire original EEG / MEG signals, and pre-process the original EEG / MEG signals to generate EEG / MEG measurement signals.

[0068] The raw EEG / MEG signals are EEG / MEG signals in the measurement space acquired using multi-lead scalp electrodes. These raw EEG / MEG signals can be acquired from public and private datasets. Public datasets can include BCI competition data, while private datasets are collected through MI experiments on multiple subjects.

[0069] After the original EEG / MEG signals are acquired, they may be pre-processed to extract signals containing subject information and filter out interfering signals.

[0070] S2: Determine the main rhythm passband of the EEG / MEG measurement signal by performing frequency band analysis on the EEG / MEG measurement signal, and extract the main rhythm measurement signal corresponding to the main rhythm passband based on the EEG / MEG measurement signal.

[0071] Frequency band analysis can be performed on the EEG / MEG measurement signal to determine the dominant rhythm corresponding to the signal portion of the EEG / MEG measurement signal containing the main characteristic information, thereby extracting the dominant rhythm measurement signal based on the dominant rhythm signal. The dominant rhythm measurement signal contains most of the signal energy of the EEG / MEG measurement signal.

[0072] S3: dividing the main rhythm measurement signal into multiple categories of sub-signals, and performing common space feature extraction on the multiple categories of sub-signals to generate global feature information;

[0073] The main rhythm measurement signals are grouped and subjected to Common Spatial Pattern (CSP) feature extraction to maximize the differences between different main rhythm measurement signals and highlight the characteristics of each type of main rhythm measurement signal, thereby determining the global feature information of the EEG / MEG signal in the measurement space.

[0074] S4: using a brain source imaging algorithm to convert the EEG / MEG measurement signal into an EEG / MEG source signal.

[0075] The EEG / MEG measurement signal in the measurement space can be mapped and converted into the cerebral cortex source space by using the brain source imaging algorithm ESI to obtain the EEG / MEG source signal.

[0076] S5: using a clustering algorithm to screen and determine key areas that are highly relevant to the task in the EEG / MEG source signals, and extracting time series signals of the key areas as detailed feature information;

[0077] The EEG / MEG source signal includes the distribution of multiple current dipoles. Clustering can be performed based on the activation strength and classification of the multiple current dipoles to identify key regions highly relevant to the MI task. Furthermore, feature extraction can be performed on these key regions to obtain detailed feature information in the source space.

[0078] S6: Utilize a cross-space fusion convolutional neural network to obtain fused feature information of the global feature information and the detailed feature information, and perform classification based on the fused feature information to determine the movement intention corresponding to the original EEG / MEG signal.

[0079] A cross-space fusion convolutional neural network is used to extract implicit features for the global feature information and the detailed feature information respectively, and the implicit global feature information and the implicit detailed feature information are further fused across space. Finally, classification and identification are performed based on the fused feature information covering both the global feature information of the measurement space and the detailed feature information of the source space, so as to determine the movement intention corresponding to the original EEG / MEG signal according to the classification result.

[0080] The brain-computer interface signal recognition method preprocesses the original EEG / MEG signal to generate the EEG / MEG measurement information. In the measurement space, the corresponding specific main rhythm is determined for the EEG / MEG measurement signal, the corresponding main rhythm measurement signal is extracted, and the global feature information in the measurement space is determined based on the main rhythm feature signal. For the source space, the EEG / MEG measurement signal is first mapped and converted into the EEG / MEG source signal in the source space, and then the key area in the EEG / MEG source signal is determined, and the time series signal corresponding to the key area is extracted as the detail feature information in the source space. Then, a cross-space fusion convolutional neural network is used to extract the global feature information and the implicit feature information corresponding to the detail feature information and fuse them. Finally, classification and recognition are performed based on the fused feature information that covers both the global feature information of the measurement space and the detail feature information of the source space. In this way, the global feature information of the measurement space is fused with the detail feature information of the source space. This method can overcome the non-real and non-global limitations of the source space decoding method, improve the spatial resolution of the measurement space decoding method, and achieve high-precision recognition and classification of MI-EEG / MEG signals.

[0081] like Figure 2 As shown, in a brain-computer interface signal recognition method provided in one or more optional embodiments of this specification, the original EEG / MEG signal is preprocessed to generate an EEG / MEG measurement signal, including:

[0082] S201: Filtering the original EEG / MEG signal, removing eye movement artifacts, and performing baseline correction processing to generate an EEG / MEG corrected signal.

[0083] The original EEG / MEG signals may be sequentially subjected to 50 Hz notch filtering and 0.1-32 Hz band-pass filtering to extract effective signal portions from the original EEG / MEG signals.

[0084] After filtering, ICA to remove eye movement artifacts and baseline correction are performed to generate the EEG / MEG corrected signal. This approach can eliminate interference factors in the original EEG / MEG signal, facilitating further signal processing.

[0085] S202: Amplify the EEG / MEG correction signal data to generate the EEG / MEG measurement signal.

[0086] Data amplification can be performed on the EEG / MEG correction signal using a sliding window strategy of time series shearing. The step size and sampling window width are set, and multiple signal segments are cut out according to the set step size to serve as the EEG / MEG measurement signal.

[0087] For example, for the acquisition period corresponding to the EEG / MEG correction signal, the step size can be set to an interval of 50 sampling points, the sampling window width is 1s, and the process of data expansion of the EEG / MEG correction signal is:

[0088] Starting from the first sampling point, a 1-second window is slid forward in steps of 50 sampling points to sequentially crop the EEG / MEG correction signal acquisition period, with each signal segment lasting 1 second. Formally, we define the EEG / MEG measurement signal for a single trial as C*T.

[0089] The original EEG / MEG signal is a multi-channel signal corresponding to multiple electrodes, and the corrected EEG / MEG signal generated after filtering and correction is also a multi-channel signal. For example, for a single EEG / MEG trial, the EEG / MEG correction signal has a data dimension of 64*2048*1 before amplification, while the EEG / MEG measurement signal generated after amplification using the above method has a data dimension of 64*512*11.

[0090] Data amplification through a sliding window strategy of time series shearing can greatly increase the data volume of the electroencephalogram / magnetic brain measurement signal, and can effectively improve the accuracy of the final MI-EEG / MEG recognition and classification.

[0091] Those skilled in the art will appreciate that methods for data amplification of the electroencephalogram (EEG) / magnetic brain correction signal include, but are not limited to, a method for amplification using a sliding window strategy.

[0092] like Figure 3 As shown, in a brain-computer interface signal recognition method provided in one or more optional embodiments of this specification, determining the main rhythm passband of the EEG / MEG measurement signal by performing frequency band analysis on the EEG / MEG measurement signal includes:

[0093] S301: Perform time-frequency decomposition on the EEG / MEG measurement signal, divide the EEG / MEG measurement signal into multiple sub-bands, and determine sub-band energy coefficients corresponding to the multiple sub-bands.

[0094] The EEG / MEG measurement signal can be divided into multiple sub-bands using wavelet packet decomposition. For example, an EEG / MEG signal in the 0.1-32 Hz frequency band can be divided into 16 sub-bands with a width of 2 Hz after four layers of wavelet packet decomposition.

[0095] The sub-band energy coefficient corresponding to the sub-band can be expressed as:

[0096]

[0097] Among them, E j represents the sub-band energy coefficient of the j-th sub-band, c j represents the wavelet packet coefficient of the j-th sub-band, k is the sampling point, and J represents the number of the sub-bands.

[0098] Those skilled in the art will appreciate that time-frequency decomposition of the EEG / MEG measurement signals includes but is not limited to wavelet packet decomposition, and for example, signal decomposition methods such as fast multidimensional empirical mode decomposition may also be used.

[0099] S302: Taking the sub-frequency band with the largest sub-band energy coefficient as a seed point, adopting a seed growth method to select a plurality of sub-frequency bands to form the main rhythm passband.

[0100] Select E m =max(E j ) as seed points, and the growth direction alternates between high-frequency and low-frequency. Adjacent sub-bands are gradually absorbed to form a connected band set. When the total energy in the connected band set exceeds a threshold δ, the growth stops.

[0101] The growth process can be described as:

[0102]

[0103] Wherein, V represents the connected belt set, P m+s The threshold δ can be flexibly set according to actual conditions, and generally, setting the threshold δ to 0.90 is the best effect.

[0104] Taking the wavelet packet time-frequency decomposition method as an example, after four layers of wavelet packet decomposition, it can be divided into 16 sub-bands with a width of 2Hz. j Represents the wavelet packet coefficient of the jth (j=1,2,…,16) sub-band node in the 4th layer.

[0105] First, define the energy coefficient vector E T =[E1,E2,E3,…,E 16], where the energy coefficient E corresponding to the j-th sub-band node j The calculation formula is shown in formula (1):

[0106]

[0107] Seed growth can form a signal set with specific properties based on a given growth criterion. We use this principle to select the maximum energy seed point. The goal is to grow adjacent frequency subbands according to the established rules to extract the set of interest. V contains q elements, where the subscripts p1, p1, ..., p q ∈[1,2,3,…,16].

[0108] With the energy coefficient E j The sub-band node where the maximum value is located defines the seed point m, and its corresponding energy coefficient is recorded as E m , as shown in formula (2):

[0109]

[0110] Starting from the seed node m, the network grows alternately in the high frequency (+) and low frequency (-) directions. The growth displacement vector D T =[0,1,-1,2,-2,…,-15], d i Corresponding D T The i-th element in , and always satisfies the constraint 1≤m+d i ≤16, if the displacement in any direction exceeds the above range, the growth in that direction will be stopped. When the sum of the sub-band energy coefficients reaches the energy threshold δ, all growth will be stopped. The corresponding maximum displacement at this time is the boundary, and its index i max The calculation formula is shown in formula (3):

[0111]

[0112] Constructing a collection of interest The elements in V are the wavelet packet coefficients contained in the growth region, representing the results of adaptive frequency selection, that is, the most active personalized frequency components in the subject's data. The main rhythm passband is determined by the set of interest V.

[0113] As a specific embodiment, the seed-growing method is used to select a plurality of sub-frequency bands to form the main rhythm passband, and the following steps may be used:

[0114] Taking the 16 sub-bands with a width of 2Hz obtained by wavelet packet decomposition as an example, the wavelet packet coefficient of the j-th sub-band is c j (k), where k is the number of sampling points, 1 <j<16。

[0115] Step (1): Calculate the energy coefficient E of each sub-band j :

[0116]

[0117] Step (2): Select E j The sub-band represented by the maximum value is used as the initial seed point:

[0118] S j =E j max

[0119] Step (3): Grow in the direction of increasing node number and update the energy parameter F j :

[0120] F j =S j +S j+1

[0121] Step (4): Set the energy threshold as δ, and judge F j Whether the growth stop conditions are met:

[0122] F j >δ

[0123] Step (5): If the condition is met, stop the growth algorithm; if not, continue growing in another direction:

[0124] F j =S j +S j-1

[0125] Step (6): Repeat step (4). If not satisfied, update the seed point S. j =F j .

[0126] Step (7): Repeat steps (3)(4)(5)(6) until the growth conditions are met, complete the adaptive frequency selection, and determine the main rhythm passband.

[0127] The connected band set determined using the seed-growing method, wherein multiple sub-bands constitute a primary rhythm passband, and the corresponding primary rhythm measurement signal contains the main signal energy of the EEG / MEG measurement signal. For example, taking the threshold δ as 0.90, the energy contained in the primary rhythm passband determined using the seed-growing method exceeds 90% of the total energy of the EEG / MEG measurement signal. Therefore, the primary rhythm measurement signal corresponding to the primary rhythm passband contains the main signal information of the EEG / MEG measurement signal.

[0128] like Figure 4As shown, in a brain-computer interface signal recognition method provided in one or more optional embodiments of this specification, the main rhythm measurement signal is divided into multiple categories of sub-signals, and co-spatial feature extraction is performed on the multiple categories of sub-signals to generate global feature information, including:

[0129] S401: Grouping the main rhythm measurement signals and performing common space feature extraction to maximize the difference between the sub-signals in each group and the sub-signals in other groups in the grouping results.

[0130] S402: Determine the measurement space features corresponding to each group of sub-signals, and combine the measurement space features corresponding to multiple groups of sub-signals to form the global feature information.

[0131] For the four-category MI task, a one-over-many common spatial pattern (OVRCSP) can be used to convert the four-category task into four binary classification tasks. Define u, u′∈{1,2,3,4} as four-category MI-EEG / MEG, where u contains one sub-signal and u′ contains the remaining three sub-signals. For example, when u = 2, u′ = 1, 3, 4. This method distinguishes the u = 2 class from the remaining three classes, making it easier to extract the best features that distinguish the u = 2 class from the other classes as the spatial features corresponding to the u = 2 sub-signal.

[0132] In this way, the main rhythm measurement signal is divided into four groups, and the spatial features of the four groups are extracted in turn. The spatial features corresponding to the four groups of sub-signals can be determined respectively, and the spatial features corresponding to the four groups of sub-signals can be combined as the global feature information.

[0133] like Figure 5 As shown, in a brain-computer interface signal recognition method provided in one or more optional embodiments of this specification, a brain source imaging algorithm is used to convert the EEG / MEG measurement signal into an EEG / MEG source signal, including:

[0134] S501: Determine the original data source corresponding to the electroencephalogram (EEG) / magnetic brain measurement signal, and determine the head model and source model corresponding to the original data source.

[0135] Wherein, the original data sources include private data sets and public data sets;

[0136] The determining of the header model and the source model corresponding to the source of the original data includes:

[0137] In response to the source of the original data being the public dataset, selecting a general model corresponding to the public dataset;

[0138] In response to the original data source being the private dataset, magnetic resonance anatomical information of a subject corresponding to the private dataset is acquired, and a private head model and a source model corresponding to the subject are created based on the magnetic resonance anatomical information.

[0139] Taking into account the differences in brain control, learning abilities, and physiology among subjects, some optional embodiments utilize MRI anatomical information from the subject to create a personalized head and source model. Subsequent determination of the conduction matrix and signal conversion is based on this personalized model. This approach can enhance the robustness of the method and further improve the stability and accuracy of signal recognition results.

[0140] For example, the private dataset could be data from 10 subjects, with an average age of 33 and no BCI experience. These subjects collected 64-lead EEG / MEG signals using a Biosemi device, along with personalized MRI slice information. Correspondingly, the public dataset could be composed of 9 subjects from Group 2a of the 4th Brain-Computer Interface Competition, without MRI data, using a public head model from Brainstorm.

[0141] S502: Aligning the EEG / MEG poles corresponding to the EEG / MEG measurement signals with the head model.

[0142] S503: Determine a conduction matrix mapping from the measurement space to the source space based on the head model and the source model.

[0143] In combination with the head model, a finite element method (FEM) may be used to determine a conduction matrix that maps from the measurement space of the scalp signal to the source space of the cerebral cortex signal.

[0144] It should be noted that boundary element method, finite difference method and the like can also be used to construct the conduction matrix.

[0145] S504: Convert the EEG / MEG measurement signal into the EEG / MEG source signal according to the conduction matrix.

[0146] The relationship between the EEG / MEG measurement signal and the EEG / MEG source signal can be expressed as:

[0147] M=LC+N

[0148] Wherein, M represents the EEG / MEG measurement signal in the measurement space, C represents the EEG / MEG source signal in the source space, L represents the conduction matrix, and N represents measurement noise.

[0149] During source imaging, the equivalent current dipole model can be used to calculate the distribution of current dipoles in the source space using minimum norm estimation, thereby determining the EEG / MEG source signal at a finer spatial scale. This EEG / MEG source signal has a more precise spatial resolution, providing more detailed spatial information for subsequent analysis.

[0150] In a brain-computer interface signal recognition method provided in one or more optional embodiments of this specification, the EEG / MEG source signal includes current dipole distribution information in the source space;

[0151] A clustering algorithm is used to screen and identify key areas of high task relevance in the EEG / MEG signal, including:

[0152] A clustering algorithm is used to cluster multiple current dipoles in the source space using activation intensity and distribution position as constraints, multiple concentrated clusters in a strongly activated state are screened and determined, and the key area is determined based on the multiple concentrated clusters.

[0153] In some optional embodiments, a duplex mean-shift clustering (DMSClustering) method can be used to screen out the key regions of interest (ROIs) that are highly relevant to the MI task. Before clustering, the four types of source signals are superimposed separately to form sub-ROI clusters for different MI tasks. The final ROI is the union of the four sub-ROIs.

[0154] DMSClustering is a data-driven method that clusters dipoles by simultaneously considering the constraints of activation intensity and distribution location, ensuring that the dipoles in the screened ROI are concentrated clusters in a strongly activated state.

[0155] By simultaneously constraining the distance factor S q and intensity factor I t , establish an abstract spherical window with core and radius ct and r respectively. ,

[0156] The clustering process can be expressed as:

[0157]

[0158]

[0159] S k ={(x,y,z,A)|S q (x,y,z)+I t (A) <r}

[0160] Where (x, y, z) represents the position coordinates of the dipole, A represents the activation intensity, N is the total number of dipoles, S k The radius r of the spherical window can be set to 0.1.

[0161] After the spherical window is determined, the center of mass drifts toward the average value of the dipole strength within the spherical window until convergence. Each drift is determined by an average displacement vector M, which always goes from the most excited state to the second most excited state.

[0162] The average displacement vector M can be expressed as:

[0163]

[0164] Where n represents the spherical window S k The number of dipoles in .

[0165] We start drifting by taking the dipole with the highest activation intensity as the initial centroid, and the clustering process ends when the number of visited dipoles exceeds the Nth percentile.

[0166] It should be noted that when screening and determining the key areas that are highly relevant to the task in the EEG / MEG source signals, clustering algorithms such as K-means clustering and spectral clustering can also be used to achieve this.

[0167] like Figure 6 As shown, in a brain-computer interface signal recognition method provided in one or more optional embodiments of this specification, a cross-space fusion convolutional neural network is used to obtain fusion feature information of the global feature information and the detail feature information, and classification is performed based on the fusion feature information to determine the movement intention corresponding to the original EEG / MEG signal, including:

[0168] S601: Extracting implicit global feature information from the global feature information using a convolutional neural network.

[0169] For the global feature information in the measurement space, a convolutional neural network can be used to further learn and extract implicit feature information. In some optional embodiments, a shallow network structure containing multiple layers of convolution can be used to further learn the implicit global spatial feature information, wherein the number of convolution kernels in each convolution layer increases in sequence according to the learning rule from low complexity to high complexity. Figure 7 As shown in FIG, a schematic diagram of the structure of a cross-space fusion convolutional neural network is shown. For the global feature information, a convolutional neural network is used to extract the implicit global feature information.

[0170] For example, a three-layer convolutional neural network (Conv_11, Conv_12, and Conv_13) can be set up for feature learning. A one-dimensional convolution kernel (1×5) is selected to match the input feature size (1×40). The kernel length is equivalent to the length of the CSP spatial filter. The number of convolution kernels in the three-layer convolutional network is set to 8, 16, and 32, respectively. The input feature information is not compressed throughout the entire process, so the feature matrix size of the measurement space after convolution remains (1×40).

[0171] S602: Extracting implicit detail feature information from the detail feature information using a convolutional neural network.

[0172] For the detailed feature information in the source space, spatially separable convolution and variable receptive field strategies are adopted to better learn and extract implicit feature information.

[0173] In some optional embodiments, the detail feature information is processed sequentially using convolutional layers with different receptive fields to extract detail temporal information and detail spatial information from the detail feature information, where the implicit detail feature information includes the detail temporal information and the detail spatial information. Specifically, an n×n convolution kernel can be split into 1×n and n×1 kernels to respectively extract temporal detail features and spatial detail features from the detail feature information.

[0174] like Figure 7 As shown in the figure, for both temporal detail feature extraction and spatial detail feature extraction, a multi-layer convolutional neural network can be set up to achieve it. It is composed of kernels of different sizes in the temporal and spatial directions to prevent repeated learning of local areas and prevent redundancy. As the receptive field increases, the more information obtained, the better the global features. Different receptive fields can learn rich and diverse features at different levels. The number of convolution kernels in the convolutional neural network set in the temporal and spatial directions can adopt the same configuration as the convolutional neural network in the measurement space.

[0175] After multiple consecutive convolutional layers in time and space (Conv_21~Conv_23, Conv_24~Conv_26), two maximum pooling layers (Max_pooling_1, Max_pooling_1) are set respectively to reduce the amount of data and parameters and prevent overfitting. At the same time, data dimensionality reduction is also achieved to ensure that the data dimension output by the source space feature extraction matches the output data dimension of the measurement space, so as to facilitate subsequent feature fusion extraction.

[0176] In some optional embodiments, for multi-layer convolutional network processing in time and space, normalization is performed after each convolution. Normalization is to prevent overfitting in network training, avoid the problem of gradient disappearance, and speed up the convergence of the network. The features after convolution are normalized so that the input of each layer can maintain the same distribution. After normalization, the rectified linear units (ReLu) activation function is also selected for processing, which has the advantage of sparsity. The activation function can be expressed as:

[0177]

[0178] The cross entropy loss function is used to measure the difference between prediction effects. We use the Adam algorithm as the optimizer, with an initial network learning rate of 1×10 -5 The loss function is minimized under the condition of . The maximum epoch is set to 128, and the learning rate is decayed by a factor of 0.9 every 9 epochs to ensure accelerated convergence in the early stage and stable performance in the later stage.

[0179] In the training process of the cross-space fusion convolutional neural network, the cross entropy loss function is used to measure the difference between the prediction effects. In some optional embodiments, the Adam algorithm can be used as the optimizer, with an initial network learning rate of 1×10 -5 The loss function is minimized under the condition of , the maximum epoch is set to 128, and the learning rate is decayed by a factor of 0.9 every 9 epochs to ensure accelerated convergence in the early stage and stable performance in the later stage.

[0180] S603: Fusing the implicit global feature information with the implicit detail feature information to generate a fused feature matrix.

[0181] The Concat connection layer can be used to fuse the two feature matrices of the measurement space and the source space.

[0182] S604: performing deep feature mining and integration on the fusion feature matrix, performing classification processing on the integrated deep features, and determining the motion intention according to the classification results.

[0183] To further mine the deeper layers of the concatenated matrix, a residual network module (ResNet Block) is added after generating the fused feature matrix. The ResNet Block consists of two layers of convolutional neural networks (Conv_1 and Conv_2). The residual network allows for direct transfer of shallow features to deeper layers, enabling rapid feedback and feature fusion without losing the original information.

[0184] The last two fully connected layers (Fc1, Fc2) are used to integrate the above features and map the learned feature representations to the sample label space. Finally, the four-classification processing is performed through the softmax layer. The softmax layer is used as the output (Output). The four neurons [y 1 ,y 2 ,y 3 ,y 4 ] is mapped to the interval (0,1).

[0185] The output of the Softmax layer represents the relative probability between different categories, which can be defined as follows.

[0186]

[0187] According to the output of the four neurons determined by the Softmax layer classification process, the movement intention can be determined. 1 ,y 2 ,y 3 ,y 4 ] is [0,0,0,1], which means the movement intention is neuron y 4 The corresponding direction.

[0188] For the brain-computer interface signal recognition method provided in the embodiments of this specification, accuracy (ACC), Kappa value, and confusion matrix can be used to evaluate the signal recognition and classification results. Specifically, the effectiveness of the cross-space fusion convolutional neural network (CS-CNN) algorithm proposed in the signal recognition method of this specification can be verified using a self-test private dataset and the BCI competition IV-2a public dataset. Five-fold cross-validation is used to ensure the randomness of the significance test results.

[0189] In order to verify the superiority of the CS-CNN algorithm proposed in the technical solution of the embodiment of this specification, we compared its results with the advanced algorithms in some related technical solutions. To avoid the influence of data quality, we used the classification results of public datasets to verify the performance. We collected relevant studies in recent years that used BCI Competition IV-2a data for four-category classification as a control. Zhang et al. amplified the data by adding noise and input it into an improved CNN structure containing inception and ResNet modules. Amin et al. proposed a CNN framework based on the inception-attention mechanism to extract spatial background information and dynamic features, and connected bi-LSTM to process time series information. Altuwaijri et al. classified EEG / MEG signals by adopting a multi-branch CNN model with different convolution kernels, achieving end-to-end classification without preprocessing. Zhao et al. generated a three-dimensional representation by keeping MI-EEG in a two-dimensional array sequence of the spatial distribution of sampling electrodes. For the three-dimensional representation, a multi-branch three-dimensional CNN was designed. Raza et al. developed a framework that combines neural structure learning (NSL) and EEGNet to standardize neural network training by using relational information in the data.

[0190] As shown in Table 1 below, the performance data of the technical solution of the embodiment of this specification compared with other advanced algorithms.

[0191] Table 1

[0192]

[0193] Table 1 shows an overall performance comparison between CS-CNN and state-of-the-art algorithms for four task categories based on the BCI Competition IV-2a dataset. The CS-CNN algorithm described in the technical solutions of the embodiments of this specification achieved the best performance among all algorithms, achieving an average signal recognition accuracy of 90.37% and an average Kappa value of 0.88. Compared to state-of-the-art algorithms, our algorithm improved the average accuracy by 1.98%. Furthermore, we calculated the SD of the classification accuracy of each algorithm across nine subjects. The results show that CS-CNN achieved the lowest SD of 1.91%, at least 5.15% lower than the other algorithms.

[0194] In summary, the brain-computer interface signal recognition method provided in the embodiments of this specification can not only improve the overall classification and recognition performance of MI-EEG / MEG signals, but also effectively reduce the adverse effects of subject specificity on the classification effect and enhance the robustness of the algorithm.

[0195] It should be noted that the methods of one or more embodiments of this specification can be performed by a single device, such as a computer or server. The methods of this embodiment can also be applied in a distributed scenario, where multiple devices cooperate to perform the method. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the methods of one or more embodiments of this specification, and the multiple devices will interact with each other to complete the method.

[0196] It should be noted that the foregoing description of this specification is based on specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0197] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the embodiments of this specification also provide a brain-computer interface signal recognition system.

[0198] refer to Figure 8 , the brain-computer interface signal recognition system comprises:

[0199] The original signal acquisition module is used to obtain the original EEG / MEG signals and pre-process the original EEG / MEG signals to generate EEG / MEG measurement signals;

[0200] a frequency band analysis module, configured to perform frequency band analysis on the EEG / MEG measurement signal, determine a main rhythm passband of the EEG / MEG measurement signal, and extract a main rhythm measurement signal corresponding to the main rhythm passband based on the EEG / MEG measurement signal;

[0201] A global feature extraction module is used to divide the main rhythm measurement signal into multiple categories of sub-signals, and perform common space feature extraction on the multiple categories of sub-signals to generate global feature information;

[0202] A brain source imaging module, configured to convert the EEG / MEG measurement signals into EEG / MEG source signals using a brain source imaging algorithm;

[0203] A detail feature extraction module is used to use a clustering algorithm to screen and determine key areas that are highly relevant to the task in the EEG / MEG source signals, and extract time series signals of the key areas as detail feature information;

[0204] A cross-space fusion module is used to use a cross-space fusion convolutional neural network to obtain fused feature information of the global feature information and the detailed feature information, and to classify the fused feature information to determine the movement intention corresponding to the original EEG / MEG signal.

[0205] In a brain-computer interface signal recognition system provided by one or more optional embodiments of the present specification, the original signal acquisition module is further used to filter the original EEG / MEG signals, remove eye movement artifacts and perform baseline correction processing to generate EEG / MEG correction signals; and perform data amplification on the EEG / MEG correction signals to generate the EEG / MEG measurement signals.

[0206] In a brain-computer interface signal recognition system provided by one or more optional embodiments of the present specification, the frequency band analysis module is further used to perform time-frequency decomposition on the EEG / MEG measurement signal, divide the EEG / MEG measurement signal into multiple sub-bands, and determine the corresponding sub-band energy coefficients of the multiple sub-bands; taking the sub-band with the largest sub-band energy coefficient as the seed point, a seed growth method is used to select a combination of multiple sub-bands to form the main rhythm passband.

[0207] In a brain-computer interface signal recognition system provided by one or more optional embodiments of the present specification, the global feature extraction module is further used to group the main rhythm measurement signals so that the difference between the sub-signals of each group and the sub-signals of other groups in the grouping results is maximized; common spatial feature extraction is performed on multiple groups of sub-signals respectively to determine the corresponding spatial features of the sub-signals, and the spatial features corresponding to the multiple groups of sub-signals are spliced to generate the global feature information.

[0208] In a brain-computer interface signal recognition system provided in one or more optional embodiments of this specification, the brain source imaging module is further used to determine the source of the original data corresponding to the EEG / MEG measurement signal, determine the head model and source model corresponding to the original data source; align the EEG / MEG poles corresponding to the EEG / MEG measurement signal with the head model; determine the conduction matrix mapping from the measurement space to the source space based on the head model and the source model; and convert the EEG / MEG measurement signal into the EEG / MEG source signal according to the conduction matrix. The original data source includes a private dataset and a public dataset. The brain source imaging module is further used to, when the original data source is the public dataset, select the universal head model and source model corresponding to the public dataset; when the original data source is the private dataset, obtain the magnetic resonance anatomical information of the subject corresponding to the private dataset, and create a private head model and source model corresponding to the subject based on the magnetic resonance anatomical information.

[0209] In a brain-computer interface signal recognition system provided in one or more optional embodiments of this specification, the EEG / MEG source signal includes current dipole distribution information in a source space. The detail feature extraction module is further configured to employ a clustering algorithm to cluster multiple current dipoles in the source space using activation intensity and distribution position as constraints, screen and identify multiple concentrated clusters in a strongly activated state, and determine the key region based on the multiple concentrated clusters.

[0210] In a brain-computer interface signal recognition system provided by one or more optional embodiments of the present specification, the cross-space fusion module is further used to extract implicit global feature information from the global feature information using a convolutional neural network; extract implicit detail feature information from the detail feature information using a convolutional neural network; fuse the implicit global feature information with the implicit detail feature information to generate a fusion feature matrix; perform deep feature mining and integration on the fusion feature matrix, perform classification processing on the integrated deep features, and determine the motion intention based on the classification results.

[0211] In a brain-computer interface signal recognition system provided by one or more optional embodiments of the present specification, the cross-space fusion module is also used to successively process the detail feature information using convolutional layers of different receptive fields, and extract detail time information and detail space information from the detail feature information, and the implicit detail feature information includes the detail time information and the detail space information.

[0212] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing one or more embodiments of this specification, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0213] The apparatus of the above embodiment is used to implement the corresponding method in the above embodiment and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0214] Figure 9 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0215] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0216] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0217] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0218] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0219] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0220] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0221] The electronic devices of the above embodiments are used to implement the corresponding methods in the above embodiments and have the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0222] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the brain-computer interface signal recognition method described in any of the above embodiments.

[0223] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0224] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the brain-computer interface signal recognition method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0225] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). The storage medium can also include a combination of the above-mentioned types of memory.

[0226] In the early days of technological development, improvements to a technology could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using a hardware module. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, eliminating the need for a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0227] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0228] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0229] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0230] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0231] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0232] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0233] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present specification as described above, which are not provided in detail for the sake of simplicity.

[0234] In addition, to simplify the description and discussion, and so as not to obscure one or more embodiments of the present specification, well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in block diagram form to avoid obscuring one or more embodiments of the present specification, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which one or more embodiments of the present specification will be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that one or more embodiments of the present specification may be implemented without these specific details or with variations in these specific details. Accordingly, these descriptions should be considered illustrative rather than restrictive.

[0235] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0236] The one or more embodiments of this specification are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included within the scope of protection of this disclosure.

Claims

1. A brain-computer interface signal recognition method, characterized in that: include: Acquire raw EEG / MEG signals, and pre-process the raw EEG / MEG signals to generate EEG / MEG measurement signals; Determine a main rhythm passband of the EEG / MEG measurement signal by performing frequency band analysis on the EEG / MEG measurement signal, and extract a main rhythm measurement signal corresponding to the main rhythm passband based on the EEG / MEG measurement signal; Dividing the main rhythm measurement signal into multiple categories of sub-signals, and performing co-spatial feature extraction on the multiple categories of sub-signals to generate global feature information; Using a brain source imaging algorithm to convert the EEG / MEG measurement signal into an EEG / MEG source signal; A clustering algorithm is used to screen and determine key areas that are highly relevant to the task in the EEG / MEG source signals, and time series signals of the key areas are extracted as detailed feature information; A cross-space fusion convolutional neural network is used to obtain fused feature information of the global feature information and the detailed feature information, and classification is performed according to the fused feature information to determine the movement intention corresponding to the original EEG / MEG signal.

2. The method according to claim 1, characterized in that Preprocessing the original EEG / MEG signal to generate an EEG / MEG measurement signal includes: Filtering the original EEG / MEG signal, removing eye movement artifacts, and performing baseline correction processing to generate an EEG / MEG corrected signal; Data amplification is performed on the EEG / MEG correction signal to generate the EEG / MEG measurement signal.

3. The method according to claim 1, characterized in that Determining the main rhythm passband of the EEG / MEG measurement signal by performing frequency band analysis on the EEG / MEG measurement signal includes: Performing time-frequency decomposition on the EEG / MEG measurement signal, dividing the EEG / MEG measurement signal into a plurality of sub-frequency bands, and determining sub-band energy coefficients corresponding to the plurality of sub-frequency bands; The sub-frequency band with the largest sub-band energy coefficient is used as a seed point, and a seed growth method is adopted to select a plurality of sub-frequency bands to form the main rhythm passband.

4. The method according to claim 1, wherein The main rhythm measurement signal is divided into multiple categories of sub-signals, and common space feature extraction is performed on the multiple categories of sub-signals to generate global feature information, including: Grouping the main rhythm measurement signals and performing common space feature extraction so that the difference between the sub-signals of each group and the sub-signals of other groups in the grouping results is the largest; The spatial features corresponding to the sub-signals are determined, and the spatial features corresponding to multiple groups of the sub-signals are spliced to generate the global feature information.

5. The method according to claim 1, wherein The EEG / MEG measurement signal is converted into an EEG / MEG source signal using a brain source imaging algorithm, including: Determining the original data source corresponding to the EEG / MEG measurement signal, and determining the head model and source model corresponding to the original data source; registering the EEG / MEG poles corresponding to the EEG / MEG measurement signals with the head model; determining a conduction matrix between a measurement space and a source space based on the head model and the source model; converting the EEG / MEG measurement signal into the EEG / MEG source signal according to the conduction matrix; Wherein, the original data sources include private data sets and public data sets; The determining of the header model and the source model corresponding to the source of the original data includes: In response to the source of the original data being the public dataset, selecting a common header model and source model corresponding to the public dataset; In response to the original data source being the private dataset, magnetic resonance anatomical information of a subject corresponding to the private dataset is acquired, and a private head model and a source model corresponding to the subject are created based on the magnetic resonance anatomical information.

6. The method according to claim 5, characterized in that The EEG / MEG source signal includes current dipole distribution information in the source space; A clustering algorithm is used to screen and identify key areas of high task relevance in the EEG / MEG signal, including: A clustering algorithm is used to cluster multiple current dipoles in the source space using activation intensity and distribution position as constraints, multiple concentrated clusters in a strongly activated state are screened and determined, and the key area is determined based on the multiple concentrated clusters.

7. The method according to claim 1, characterized in that Obtaining fused feature information of the global feature information and the detailed feature information using a cross-space fusion convolutional neural network, and performing classification based on the fused feature information to determine the movement intention corresponding to the original EEG / MEG signal, including: Extracting implicit global feature information from the global feature information using a convolutional neural network; Extracting implicit detail feature information from the detail feature information using a convolutional neural network; Fusing the implicit global feature information with the implicit detail feature information to generate a fused feature matrix; Deep feature mining and integration are performed on the fusion feature matrix, and classification processing is performed on the integrated deep features, and the movement intention is determined according to the classification results.

8. The method according to claim 7, characterized in that Extracting implicit detail feature information from the detail feature information using a convolutional neural network, including: The detail feature information is processed successively using convolutional layers with different receptive fields, and detail time information and detail space information are extracted from the detail feature information, wherein the implicit detail feature information includes the detail time information and the detail space information.

9. A brain-computer interface signal recognition system, characterized in that: include: The original signal acquisition module is used to acquire the original EEG / MEG signals and pre-process the original EEG / MEG signals to generate EEG / MEG measurement signals; a frequency band analysis module, configured to perform frequency band analysis on the EEG / MEG measurement signal, determine a main rhythm passband of the EEG / MEG measurement signal, and extract a main rhythm measurement signal corresponding to the main rhythm passband based on the EEG / MEG measurement signal; A global feature extraction module is used to divide the main rhythm measurement signal into multiple categories of sub-signals, and perform common space feature extraction on the multiple categories of sub-signals to generate global feature information; A brain source imaging module, configured to convert the EEG / MEG measurement signals into EEG / MEG source signals using a brain source imaging algorithm; A detail feature extraction module is used to use a clustering algorithm to screen and determine key areas that are highly relevant to the task in the EEG / MEG source signals, and extract time series signals of the key areas as detail feature information; A cross-space fusion module is used to use a cross-space fusion convolutional neural network to obtain fused feature information of the global feature information and the detailed feature information, and to classify according to the fused feature information to determine the movement intention corresponding to the original EEG / MEG signal.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the program.

Images