Two-hand coupling brain-computer interface normal form, method, device and equipment

Abstract

The invention relates to the technical field of brain-computer interfaces, and discloses a two-hand coupling brain-computer interface normal form, method, device and equipment, and the normal form is a brain-computer interface normal form which is used for collecting electroencephalogram signals when a user performs motion execution or performs motion imagination on two-hand coupling motion so as to obtain neural decoding signals of motion execution or motion imagination of the user; the method comprises the following steps: inputting electroencephalogram characteristics collected under the condition of coupling motion of both hands of a user into a decoding model to obtain a motion intention; wherein the model acquisition comprises the following steps: inputting collected electroencephalogram characteristics of subjects into an electroencephalogram learning network model so as to obtain a decoding result of each subject; obtaining an accuracy rate according to a decoding result; if the accuracy rate is smaller than the preset accuracy rate, the model is controlled to conduct self-learning till the accuracy rate is larger than or equal to the preset accuracy rate, the electroencephalogram learning network model after self-learning is completed serves as a decoding model of the two-hand coupling brain-computer interface normal form, and the decoding accuracy rate can be improved through the normal form and the method.

Description

Technical Field

[0001] The present application relates to the field of brain-computer interface technology, and in particular to a two-handed coupled brain-computer interface paradigm, method, device and equipment. Background Art

[0002] The existing brain-computer interface paradigms include steady-state visual evoked potential (SSVEP), P300 and motor imagery (MI); among them, although the SSVEP and P300 paradigms can output external control commands of higher dimensions, the long-term presentation of visual stimulation during the walking process will cause visual fatigue in patients. At the same time, such paradigms do not match the habitual control mode of the human body, and their application potential is very limited. In addition, for the MI paradigm, different commands are mapped by imagining different parts of the body, which is difficult to match with the movement direction control of external devices, and there are limitations such as unnatural interactions. In addition, the training and modeling process is extremely time-consuming, and it is difficult to be effectively promoted in clinical practice. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a two-handed coupled brain-computer interface paradigm, method, apparatus and device, which can improve the decoding accuracy by decoding and identifying the collaboratively coupled motion signals of both hands in different directions.

[0004] In a first aspect, an embodiment of the present application provides a bimanually coupled brain-computer interface paradigm, which collects EEG signals when a user performs bimanual coupled movements to obtain neural decoding signals of the user's movement execution or movement imagination.

[0005] In a first aspect, an embodiment of the present application provides a decoding model acquisition method for a two-hand coupled brain-computer interface paradigm, comprising:

[0006] Inputting the collected EEG features of the subject into the EEG learning network model to obtain the decoding results of the movement direction of each subject; wherein the EEG features are the features of the movement-related cortical potentials extracted from the subject based on the above-mentioned two-hand coupled brain-computer interface paradigm;

[0007] Compare each of the motion direction decoding results with a preset result to obtain a result accuracy rate;

[0008] If the accuracy rate is lower than the preset accuracy rate, the EEG learning network model is controlled to perform self-learning according to the accuracy rate until the accuracy rate is greater than or equal to the preset accuracy rate, and the EEG learning network model after self-learning is used as a decoding model of the two-hand coupled brain-computer interface paradigm.

[0009] In some embodiments, the EEG learning network model includes a two-dimensional convolutional layer, a deep convolutional layer, a separation convolutional layer, a fully connected layer, and an activation layer;

[0010] The step of inputting the collected EEG features of the subject into the EEG learning network model to obtain the decoding result of the motion direction of each subject includes:

[0011] Inputting the EEG features of the subject into the two-dimensional convolution layer and the deep convolution layer of the EEG learning network model in sequence to extract the temporal features and spatial features of the EEG features respectively;

[0012] Enhance the learning ability of each of the temporal features and spatial features through the separable convolutional layer of the EEG learning network model;

[0013] After reshaping the feature dimension of the enhanced EEG features, the features are input into the fully connected layer to output the category probability;

[0014] The Softmax activation function is used to process the category probabilities to obtain a decoding result of the two-hand coupled motion direction.

[0015] In some embodiments, the step of sequentially inputting the EEG features of the subject into the two-dimensional convolution layer and the deep convolution layer of the EEG learning network model to extract the temporal features and spatial features of the EEG features, respectively, comprises:

[0016] Inputting the EEG features of the subject into the two-dimensional convolutional layer of the EEG learning network model;

[0017] The EEG features of each channel are convolved by the two-dimensional convolution layer to extract the time features in the EEG features; wherein the convolution kernel size of the two-dimensional convolution layer is 1 / 2 of the number of sampling points;

[0018] Performing batch normalization processing on the EEG features output by the two-dimensional convolutional layer;

[0019] Inputting the batch normalized EEG features into the deep convolution layer to extract spatial features in the EEG features of each channel;

[0020] The EEG features output by the deep convolutional layer are normalized and processed to prevent overfitting.

[0021] In some embodiments, after enhancing the learning ability of each of the temporal features and spatial features through the separable convolutional layer of the EEG learning network model, the method further includes:

[0022] The features output by the separable convolutional layer are normalized and then anti-overfitting is performed.

[0023] In some embodiments, before inputting the collected EEG features of the subject into the EEG learning network model, the method further includes:

[0024] Re-referencing the collected EEG signal; wherein the EEG signal is a neural decoding signal collected by the subject based on a two-hand coupled brain-computer interface paradigm;

[0025] The re-referenced EEG signal is subjected to a bandpass filter with a preset frequency and then subjected to a global average reference process;

[0026] Performing artifact removal processing on the EEG signal after global average reference to obtain an artifact-free EEG signal;

[0027] Segmenting the artifact-free EEG signal according to the trigger synchronization label;

[0028] The artifact-free EEG signals with amplitude exceeding a preset amplitude in each segment are removed for baseline calibration;

[0029] After baseline calibration, the artifact-free EEG signals in each segment were low-pass filtered and EEG features were extracted.

[0030] In some embodiments, performing artifact removal processing on the EEG signal after global average reference to obtain the artifact-free EEG signal includes:

[0031] The myoelectric artifact signal in the EEG signal after the global average reference is removed by an artifact component reconstruction algorithm; and the artifact component of the EEG signal after the global average reference is identified and removed by independent component analysis and a classifier.

[0032] In a third aspect, an embodiment of the present application provides a method for identifying the direction of motion intention based on two-handed coupling motion, including:

[0033] Input the EEG characteristics of the target user into the decoding model of the bimanual coupling brain-computer interface paradigm to obtain the output result;

[0034] The movement direction intention of the target user is determined according to the output result; wherein the decoding model of the two-handed coupled brain-computer interface paradigm is obtained using the decoding model acquisition method of the two-handed coupled brain-computer interface paradigm mentioned above.

[0035] In a fourth aspect, an embodiment of the present application provides a decoding model acquisition device for a two-hand coupled brain-computer interface paradigm, comprising:

[0036] A decoding result acquisition module, used for inputting the collected EEG features of the subject into the EEG learning network model to obtain the decoding results of the movement direction of each of the subjects; wherein the EEG features are the features of the movement-related cortical potentials extracted by the subject based on the above-mentioned two-hand coupled brain-computer interface paradigm;

[0037] A result accuracy determination module, used for comparing each of the motion direction decoding results with a preset result to obtain a result accuracy;

[0038] The decoding model generation module is used to control the EEG learning network model to perform self-learning according to the accuracy rate if the accuracy rate is less than the preset accuracy rate, until the accuracy rate is greater than or equal to the preset accuracy rate, and use the EEG learning network model after self-learning as the decoding model of the two-handed coupled brain-computer interface paradigm.

[0039] In a fifth aspect, an embodiment of the present application provides a brain-computer terminal device, which includes a processor and a memory, the memory stores a computer program, and the processor is used to execute the computer program to implement the above method.

[0040] The embodiments of the present application have the following beneficial effects: the present application proposes a paradigm based on a two-hand coupled brain-computer interface, a method for identifying movement intentions and a method for acquiring a model thereof, that is, a paradigm of natural movement brain-computer interface based on two-hand coupling is proposed, through which the intention of movement direction is identified, and when the EEG learning network model is trained, the EEG features used are the characteristics of the movement-related cortical potentials extracted when the subject directly imagines that both hands move in different directions at the same time during the execution of the movement or the process of movement imagination, so that the decoding model of the two-hand coupled brain-computer interface paradigm is obtained after the training of the EEG learning network model is completed, and when the user's movement intention needs to be obtained, the collected EEG features of the user are input into the decoding model of the two-hand coupled brain-computer interface paradigm. The present application simulates the natural action of moving objects with both hands, so that the user feels more natural and comfortable when using the brain-computer interface system, and can reduce the time of learning and adaptation. In addition, the paradigm can improve the decoding accuracy by decoding and identifying the coordinated movement signals of both hands, thereby improving the control accuracy of the BCI system (brain-computer interface system), so that the response of the rehabilitation equipment or movement enhancement equipment applying the method is more in line with the user's intention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A schematic diagram of the brain-computer interface paradigm in the prior art is shown;

[0043] Figure 2 A first flow chart of a method for acquiring a decoding model of a two-hand coupled brain-computer interface paradigm according to an embodiment of the present application is shown;

[0044] Figure 3 A schematic diagram of an embodiment of the present application is shown in which an object is held upward, downward, leftward, rightward, forward or backward with both hands imagined at the same time;

[0045] Figure 4 A second flow chart of the method for acquiring a decoding model of a two-hand coupled brain-computer interface paradigm according to an embodiment of the present application is shown;

[0046] Figure 5 A third flow chart of the method for acquiring a decoding model of a two-hand coupled brain-computer interface paradigm according to an embodiment of the present application is shown;

[0047] Figure 6 A schematic diagram of an EEG learning network model according to an embodiment of the present application is shown;

[0048] Figure 7 A schematic diagram of a technical route for intention recognition of two-handed coupled motion directions according to an embodiment of the present application is shown;

[0049] Figure 8 A schematic diagram showing a comparison of decoding accuracy between the two-handed coupling paradigm of an embodiment of the present application and the single-handed paradigm of the prior art is shown;

[0050] Fig. 9 A schematic diagram showing the time-frequency analysis results of the two-hand coupling paradigm using an embodiment of the present application is shown;

[0051] Fig.10 A schematic diagram of the brain topography of the beta frequency band and its statistical analysis results using the bimanual coupling paradigm of an embodiment of the present application is shown;

[0052] Fig.11 A structural schematic diagram of a decoding model acquisition device of a two-hand coupled brain-computer interface paradigm according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0054] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0055] Hereinafter, the terms "including", "having" and their cognates that can be used in various embodiments of the present application are intended only to indicate specific features, numbers, steps, operations, elements, components or a combination of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or a combination of the foregoing items or increasing the possibility of one or more features, numbers, steps, operations, elements, components or a combination of the foregoing items. In addition, the terms "first", "second", "third" and the like are only used to distinguish descriptions and cannot be understood as indicating or implying relative importance.

[0056] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0057] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0058] Brain-computer interface (BCI) technology does not rely on the normal physiological output pathways of the brain (peripheral nervous system and muscle tissue), and can decode the patient's motor intention by detecting neural signals. It is currently used as a common method of intention recognition. It has been increasingly widely used in rehabilitation medicine, smart home control, entertainment and games.

[0059] Non-invasive BCI is a technology that can obtain EEG signals without surgery and convert them into external device control signals. Compared with invasive methods (such as implanted electrodes), non-invasive BCI has the advantages of high safety and low risk, and is therefore more widely used. The current paradigms based on non-invasive BCI mainly include Figure 1The three types shown are steady-state visual evoked potential (SSVEP), P300 and motorimagery (MI); among them, SSVEP is the synchronous electrical activity generated by the visual cortex when the user looks at visual stimuli that flicker at a specific frequency; P300 refers to the positive fluctuation of brain waves that appears in about 300 milliseconds when the user notices a rare stimulus in a series of regular stimuli; MI refers to the user imagining different parts of the body (such as hands, feet or tongue) to induce cortical event-related desynchronization and synchronization; different paradigms induce different neural patterns, and the collected signals are decoded by algorithms to generate instructions, thereby realizing the control of external devices.

[0060] However, for the SSVEP and P300 paradigms in the existing technology, although they can output external control commands of higher dimensions, the training of presenting visual stimulation for a long time during walking will cause visual fatigue in patients. At the same time, such paradigms do not match the habitual control mode of the human body, and their application potential is very limited. In addition, for the MI paradigm, different commands are mapped by imagining different parts of the body, which is difficult to match with the movement direction control of external devices, and there are limitations such as unnatural interactions. In addition, the training and modeling process is extremely time-consuming, and it is difficult to be effectively promoted in clinical practice.

[0061] Furthermore, the brain-computer interface technologies in the prior art all perform movements or imagine movements of different body parts, utilize the spatial mapping relationship between the body and the cerebral cortex, and extract spatial features, such as event-related desynchronization (ERD) and event-related synchronization (ERS), to classify different body parts that perform or imagine movements, and then map the classification results to actual control instructions, such as up, down, left, right, front, back, etc. This method decodes individual parts of the body (such as the tongue, left hand, right hand, both hands and both feet), and actually controls the movement of the object in different directions. It is therefore an indirect mapping control. Therefore, when both hands need to control the movement at the same time, it will cause inaccurate and unnatural control problems. Based on this, the present application proposes a two-handed coupled brain-computer interface paradigm, method, device and terminal. The two-handed coupled brain-computer interface paradigm of the present application is a brain-computer interface paradigm in which EEG signals are collected when the user executes or imagines coupled movements of both hands to obtain neural decoding signals of the user's movement execution or movement imagination. It can be understood that the present application directly imagines the simultaneous movement of both hands, that is, imagines that both hands move up, down, left, right, etc. at the same time, and then collects EEG signals and decodes the EEG signals to achieve the purpose of obtaining movement intention. The method of the present application belongs to a brain-computer interface paradigm that is directly oriented to the task object, conforms to the natural human habit of manipulating objects or controlling the movement of objects, and by simulating the natural movement of both hands moving objects, the user feels more natural and comfortable when using the BCI system, reducing the time for learning and adaptation.

[0062] The following is an explanation of the decoding model acquisition method of the two-handed coupled brain-computer interface paradigm with reference to some specific embodiments.

[0063] Figure 2 A flowchart of a decoding model acquisition method of a two-handed coupled brain-computer interface paradigm of an embodiment of the present application is shown. Exemplarily, the decoding model acquisition method based on a two-handed coupled brain-computer interface includes the following steps:

[0064] Step S100, inputting the collected EEG features of the subjects into the EEG learning network model to obtain the motion direction decoding result of each subject.

[0065] The brain-computer interface paradigm mentioned in this embodiment is a two-handed coupled motion execution and motion imagination paradigm. This brain-computer interface paradigm is that during motion execution or motion imagination, both hands move synchronously, such as up, down, left, right, forward, backward, etc. at the same time. That is, the brain-computer interface paradigm in this application is mainly to solve the problem of uncoordinated motion of both hands caused by the prior art paradigm. Therefore, the paradigm in this embodiment is based on the actions of using both hands simultaneously in real life to grab, hold or carry objects. Therefore, the acquired EEG features are the features of motion-related cortical potentials extracted based on the two-handed coupled brain-computer interface paradigm, that is, the features of motion-related cortical potentials extracted when the subject or target user directly imagines that both hands move simultaneously during motion execution or motion imagination. It can be understood that the EEG features acquired in this embodiment are imagined moving both hands at the same time, such as Figure 3 As shown in the figure, imagine holding an object with both hands and moving it upward, downward, left, right, forward or backward, etc. This is more in line with the human habit of naturally manipulating objects or controlling the movement of objects. Figure 4 As shown, before step S100, step S500 is also included, in which the subject performs EEG acquisition while performing coupled movements of both hands in motor execution or motor imagination, and then preprocesses the acquired EEG signals to obtain EEG features. The process of preprocessing the acquired EEG signals to obtain EEG features in step S500 specifically includes:

[0066] Step S510, re-reference the collected EEG signal, and down-sample the collected EEG signal to a preset frequency.

[0067] The EEG signal is a neural decoding signal collected based on a bimanually coupled brain-computer interface paradigm, that is, a neural decoding signal collected when the subject is performing a bimanually coupled movement or a motor imagination.

[0068] Step S520, performing bandpass filtering of a preset frequency on the EEG signal, and then performing global average reference processing.

[0069] In some specific embodiments, the EEG signal of this embodiment is illustrated by taking EEG as an example. After the EEG signal (EEG) of the subject is collected; since the EEG signal includes brain activities at least at two electrode positions, it is necessary to select a suitable reference electrode to effectively improve the quality of the collected EEG signal and reduce noise interference, so that the final decoding result is more accurate; in the embodiment of the present application, the collected EEG signal is re-referenced through binaural reference electrodes, and the collected EEG signal is downsampled to 128Hz; then the re-referenced EEG signal is band-pass filtered at 0.01-30Hz and then globally averaged for reference to improve the signal-to-noise ratio of the EEG signal.

[0070] Step S530, performing artifact removal processing on the EEG signal after global average reference to obtain an artifact removed EEG signal.

[0071] Since the muscle activity of the head and neck will generate electromyographic signals, and the electromyographic EEG signals generally have a high frequency, usually above 25HZ, with a large amplitude, and appear as an explosive dense waveform, which will interfere with the EEG signals. Therefore, in the process of preprocessing the EEG signals, the electromyographic artifacts need to be removed. In addition, when blinking, the eyelids cover the eyeballs and then leave the eyeballs, which will cause a sharp change in the dipole potential difference. This potential difference change is captured by the EEG acquisition system, forming a blink artifact; in addition, the up and down or left and right movement of the eyeballs will bring about changes in the dipole potential difference, which will cause eye movement artifacts; the above-mentioned blink artifacts and eye movement artifacts are both called electrooculographic artifacts. When preprocessing the EEG signals, in order to avoid the influence of electrooculographic artifacts, the electrooculographic artifacts need to be removed in the embodiments of the present application.

[0072] It can be understood that in order to suppress the influence of artifacts, the embodiment of the present application also needs to perform artifact removal processing on the EEG signal after global average reference to obtain an artifact-free EEG signal; when performing artifact removal processing, in order to suppress large-scale motion artifacts, this embodiment applies an artifact component reconstruction (ASR) algorithm to remove myoelectric artifacts; in addition, in order to remove electrooculogram artifacts, independent component analysis (ICA) is performed on the EEG data, and ICLabel (an IC classifier) ​​is used to identify and remove artifact components caused by blinking and eye movement.

[0073] Step S540 , segmenting the artifact-free EEG signal into trials according to the trigger synchronization labels.

[0074] Step S550, removing the artifact-removed EEG signals in each segment whose amplitude exceeds a preset amplitude and performing baseline calibration.

[0075] After the EEG signal is processed for artifact removal, the artifact-free EEG signal is obtained, and then the trials are segmented according to the trigger synchronization label to obtain the segmented EEG signal; wherein, the specific number of segments can be set as needed, and is not limited here. The EEG signal in each trial (each segment) is checked. If the amplitude of the EEG signal exceeds 75μV, the segment with an amplitude exceeding 75μV is removed as a bad segment, and then the EEG data of each trial is baseline calibrated. The purpose of baseline calibration of the EEG signal is to adjust the baseline of the EEG signal (the average level of the signal), which can eliminate or reduce signal fluctuations caused by unstable factors, thereby making the EEG signal more accurate and reliable.

[0076] Step S560, performing low-pass filtering on the artifact-free EEG signals in each segment after baseline calibration, and extracting EEG features.

[0077] Finally, the EEG signal in each trail is low-pass filtered at 0.1-3 Hz to extract EEG features, where the EEG features can be MRCP features. EEG features (MRCP features) are changes in EEG signals before and after motor imagination or actual operation. They provide important information about the upcoming movement. Through low-pass filtering at 0.1-3 Hz, high-frequency noise can be removed, leaving only low-frequency components of 0.1-3 Hz, which helps capture potential changes related to movement.

[0078] It can be understood that the above steps S10-S60 are the preprocessing process of the collected EEG signals; in some embodiments, when collecting EEG signals from subjects, non-invasive techniques such as EEG (electroencephalogram), MEG (magnetoencephalogram), fNIRS (functional magnetic resonance imaging), fMRI (functional near-infrared imaging) or PET (positron emission tomography) can be used, semi-embedded techniques such as ECoG (electrocortical mapping), and invasive techniques such as LFP (local field potential) can also be used. It can be understood that the above-mentioned techniques for collecting EEG signals are only a few examples, and other techniques that can be used to collect brain signals can also be used, which will not be listed one by one here.

[0079] The following is an explanation of the channels and sampling points involved in EEG signal acquisition. The acquisition of EEG signals usually involves multiple channels and sampling points to ensure that the information of brain activity is fully captured. The acquisition of EEG signals mainly depends on the number and placement of electrodes. These factors directly affect the quality and quantity of the collected signals. The number and position of electrodes usually follow a certain standard layout, such as the international 10-20 system, which is a widely accepted electrode placement standard designed to optimize the efficiency and accuracy of signal acquisition. This layout takes into account different areas and functions of the brain to ensure that the most relevant electrical activity can be captured.

[0080] The number of channels refers to the number of electrodes used simultaneously; multi-channel acquisition can provide more comprehensive brain activity information and help analyze the synergy and independent activities between different regions. For example, for basic brain wave activity monitoring, fewer channels may be used, while for more sophisticated cognitive or motor research, more channels may be needed to capture more subtle changes.

[0081] Among them, each sampling point represents a data point collected from a specific electrode; the number of sampling points depends on the sampling rate and signal processing requirements. The sampling rate refers to the number of data points collected per second, which directly affects the temporal resolution of the data. A high sampling rate can capture faster changes in EEG activity, while a low sampling rate is suitable for analyzing slower changes or average activity levels; the choice of sampling points is also affected by hardware limitations and technical requirements. For example, a higher sampling rate requires more advanced data storage and processing capabilities.

[0082] Therefore, the channels and sampling points for EEG signal collection can be determined as needed.

[0083] In some embodiments, Figure 5 As shown, step S100 includes:

[0084] Step S110, the EEG features of the subject are sequentially input into the two-dimensional convolution layer and the deep convolution layer of the EEG learning network model to respectively extract the temporal features and spatial features of the EEG features.

[0085] The decoding model of the bimanual coupling brain-computer interface paradigm obtained subsequently is obtained by training the EEG learning network model. The EEG learning network model can be EEGNet, support vector machine (SVM), linear discriminant analysis (LDA), ShallowConvNet or DeepConvNet. Preferably, the EEG learning network model in this embodiment adopts EEGNet. Figure 6 As shown, it includes a two-dimensional convolutional layer, a depth convolutional layer, a separable convolutional layer, a fully connected layer, and a Softmax activated output layer.

[0086] The dimension of the EEG feature is (number of channels, number of sampling points). For example, if the number of channels is 64 and the number of sampling points is 128, the dimension of the EEG feature is (64, 128).

[0087] In this step, firstly, two-dimensional convolution is applied to the EEG features of each channel through the two-dimensional convolution layer of the EEG learning network model to extract the temporal features of the EEG features; the convolution kernel size of the two-dimensional convolution layer is 1 / 2 of the number of sampling points, that is, if the number of sampling points is 128, the convolution kernel size of the two-dimensional convolution layer is 64. Then, the EEG features output by the two-dimensional convolution layer are batch normalized to standardize the output of the convolution layer, and the training process of the EEG learning network model can be accelerated and the stability can be improved; then, the batch normalized EEG features are input into the deep convolution layer, and spatial convolution is applied to the EEG features of each channel to extract the spatial features of the EEG features; wherein, the size of the convolution kernel of the deep convolution layer is equal to the number of channels, and if the number of channels is 64, the size of the convolution kernel of the deep convolution layer is 64; then, the EEG features output by the deep convolution layer are normalized, and Dropout is applied to prevent overfitting; wherein, the Dropout rate is 0.3-0.5.

[0088] Among them, the temporal feature is the feature of the EEG feature in the temporal dimension; the spatial feature is the feature of the EEG feature in the spatial dimension.

[0089] Step S120, enhancing the learning ability of each temporal feature and spatial feature through the separable convolutional layer of the EEG learning network model.

[0090] In step S110, after the deep convolution layer data, the normalized and Dropout processed EEG signal is input into the separable convolution layer, in which the input data is subjected to deep convolution and point-by-point convolution respectively to enhance the learning ability of each temporal feature and spatial feature. Then the features output by the separable convolution layer are normalized again, and Dropout is applied to prevent overfitting.

[0091] In step S130, the enhanced EEG features are reshaped into feature dimensions and then input into a fully connected layer to output category probabilities.

[0092] Step S140, using a Softmax activation function to perform processing according to the category probability to obtain a decoding result of the two-hand coupled motion direction.

[0093] In step S130, the multi-dimensional output is flattened into a one-dimensional feature vector, input into a fully connected layer to output the category probability, and then Softmax activation is used to output the decoding result of the two-handed coupled motion direction, such as front, back, left, and right.

[0094] Step S200, comparing the decoding result of each motion direction with a preset result to obtain the result accuracy.

[0095] Step S300, determining whether the accuracy is greater than or equal to a preset accuracy.

[0096] If the accuracy rate is less than the preset accuracy rate, return to step S100, re-collect the subject's EEG features for training, until the accuracy rate of the subject's decoding result is greater than or equal to the preset accuracy rate, then use the current EEG learning network model as the decoding model of the two-handed coupled brain-computer interface paradigm of this embodiment; if the accuracy rate is greater than or equal to the preset accuracy rate, execute step S400, and use the EEG learning network model after self-learning as the decoding model of the two-handed coupled brain-computer interface paradigm.

[0097] After the decoding results of each subject are obtained in step S100, the decoding results are compared with the preset decoding results of each subject. If they are consistent, it means that the result is accurate. If they are inconsistent, it means that the result is inaccurate. The decoding result accuracy of all subjects is obtained by statistics. If the decoding result accuracy of the subject is less than the preset accuracy compared with the preset accuracy, it means that the current EEG learning network model does not meet the requirements. In addition, since the EEG learning network model has self-learning ability, the EEG learning network model returns to step S100 after self-learning according to the obtained accuracy.

[0098] It can be understood that steps S100 to S400 are the process of training the bimanual coupled brain-computer interface paradigm decoding model. After the bimanual coupled brain-computer interface paradigm decoding model is obtained through steps S100 to S400, when in use, the collected EEG features of the target user can be directly input into the decoding model to obtain the target user's movement intention.

[0099] It should be noted that if other support vector machines (SVM), linear discriminant analysis (LDA), ShallowConvNet (shallow convolutional neural network) or DeepConvNet (deep convolutional neural network) models are used, the collected EEG features of the subjects are also input into the model, and then the decoding results are input, and the decoding results are compared with the preset results to obtain the accuracy, and the model is self-learned with higher accuracy until the accuracy meets the requirements.

[0100] The embodiment of the present application also proposes a method for identifying the direction intention of a two-handed coupled motion. Exemplarily, the method for identifying the direction intention of a two-handed coupled motion includes: inputting the EEG characteristics of the target user into a two-handed coupled brain-computer interface paradigm decoding model to obtain an output result;

[0101] The movement direction intention of the target user is determined according to the output result; wherein, the decoding model of the two-handed coupled brain-computer interface paradigm is obtained using the two-handed coupled brain-computer interface paradigm decoding model acquisition method of the above-mentioned embodiment.

[0102] It can be understood that after obtaining a decoding model of the two-handed coupled brain-computer interface paradigm with a higher decoding result accuracy through the two-handed coupled brain-computer interface decoding paradigm model acquisition method of the above-mentioned embodiment, when it is necessary to decode the movement direction intention of the target user, the EEG signal of the target user can be collected, and then the collected EEG information can be preprocessed (wherein the preprocessing process is consistent with the above-mentioned preprocessing process for the EEG signal of the subject) to obtain EEG features, and then the EEG features are input into the decoding model of the two-handed coupled brain-computer interface paradigm, and the movement direction intention of the target user can be determined according to the result output by the model.

[0103] In some embodiments, the technical route of the two-hand coupled motion direction intention recognition is as follows: Figure 7 As shown, the EEG signals of the target user when imagining the direction of the coupled movement of both hands are collected, and then the collected EEG signals are preprocessed to obtain EEG features (MRCP features) (wherein the preprocessing of the EEG signals is the same as the preprocessing process of the EEG signals of the subject in the above embodiment), and then the EEG features are input into the decoding model of the two-hand coupled brain-computer interface paradigm for direction decoding to obtain the decoded direction information, which is used as a control instruction to control the movement direction of the peripheral device (such as the end handle of the rehabilitation robot, the end of the exoskeleton, etc.).

[0104] It can be understood that the options in the above embodiment are also applicable to this embodiment, so they will not be described again here.

[0105] The method of this embodiment was used to conduct experiments on two-hand coupled motor imagery and one-handed motor imagery on multiple subjects, and the experimental results were as follows: Figure 8 As shown, according to Figure 8 It can be seen that no matter which model / algorithm is used, the decoding accuracy of the two-hand coupling paradigm of the present application is significantly improved compared with the one-hand paradigm. In addition, according to the time-frequency analysis, Fig. 9 As shown in Figure 2, the bimanual coupling paradigm not only showed significant ERD in the alpha frequency band, but also in the beta frequency band. Fig.10 As shown in the figure, compared with single-handed movement, bimanual coupling showed stronger ERD phenomenon in the beta frequency band of the left and right hemispheres, and there was a significant statistical difference, among which, Fig.10 The red dots indicate significant differences at p < 0.05.

[0106] The present application proposes a two-handed coupling brain-computer interface paradigm, a method for identifying movement intentions, and a method for obtaining a model, that is, a brain-computer interface paradigm based on two-handed coupling is proposed, through which the intention of movement direction is identified. When the EEG learning network model is trained, the EEG features used are the characteristics of the movement-related cortical potentials extracted when the subject directly imagines that both hands move simultaneously during movement execution or movement imagination. In this way, after the training of the EEG learning network model is completed, a decoding model of the two-handed coupling brain-computer interface paradigm is obtained. When it is necessary to obtain the user's movement direction intention, the collected EEG features of the user are input into the decoding model of the two-handed coupling brain-computer interface paradigm. The brain-computer interface paradigm based on two-handed coupling of the present application refers to the task of using both hands at the same time to grasp, hold or carry objects in life. This task involves the coordinated movement of both hands, which can ensure the stability of the object during the carrying process. That is, this paradigm is a movement execution and movement imagination paradigm that is more in line with the natural operation habits of the human body for objects, which can improve the accuracy of human movement intention recognition, thereby providing a potential way for rehabilitation equipment or movement enhancement equipment.

[0107] Fig.11 A schematic diagram of a structure of a device for acquiring a paradigm decoding model of a two-handed coupled brain-computer interface according to an embodiment of the present application is shown. Exemplarily, the device for acquiring a paradigm decoding model of a two-handed coupled brain-computer interface comprises:

[0108] The decoding result acquisition module 100 is used to input the collected EEG features of the subject into the EEG learning network model to obtain the decoding results of the movement direction of each subject; wherein the EEG features are the features of the movement-related cortical potentials extracted based on the bimanual coupling brain-computer interface paradigm;

[0109] A result accuracy determination module 200, for comparing the decoding result of each motion direction with a preset result to obtain a result accuracy;

[0110] The decoding model generation module 300 is used to control the EEG learning network model to perform self-learning according to the accuracy rate if the accuracy rate is less than the preset accuracy rate, until the accuracy rate is greater than or equal to the preset accuracy rate, and use the EEG learning network model after self-learning as the decoding model of the two-hand coupled brain-computer interface paradigm.

[0111] It can be understood that the device of this embodiment corresponds to the method for acquiring the bimanual coupling brain-computer interface paradigm decoding model of the above-mentioned embodiment, and the optional items in the above-mentioned embodiment are also applicable to this embodiment, so they will not be repeated here.

[0112] The present application also provides a brain-computer terminal device. Exemplarily, the brain-computer terminal device includes a processor and a memory, wherein the memory stores a computer program, and the processor runs the computer program, thereby enabling the brain-computer terminal device to execute the above method.

[0113] Among them, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU) and a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or at least one of other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0114] The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory is used to store a computer program, and the processor may execute the computer program accordingly after receiving an execution instruction.

[0115] The present application also provides a computer-readable storage medium for storing the computer program used in the above-mentioned device. For example, the computer-readable storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0116] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of boxes in the structure diagram and / or the flow diagram, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] In addition, the functional modules or units in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0118] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application.

[0119] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A bimanually coupled brain-computer interface paradigm, characterized in that: The paradigm is a brain-computer interface paradigm in which electroencephalogram signals are collected when the user performs motor execution or motor imagination coupled movements of both hands, so as to obtain neural decoding signals of the user's motor execution or motor imagination.

2. A decoding model acquisition method for a two-handed coupled brain-computer interface paradigm, characterized in that: include: Inputting the collected EEG features of the subject into the EEG learning network model to obtain the decoding results of the movement direction of each of the subjects; wherein the EEG features are the features of the movement-related cortical potentials extracted by the subject based on the two-hand coupled brain-computer interface paradigm described in claim 1; Compare each of the motion direction decoding results with a preset result to obtain a result accuracy rate; If the accuracy rate is lower than the preset accuracy rate, the EEG learning network model is controlled to perform self-learning according to the accuracy rate until the accuracy rate is greater than or equal to the preset accuracy rate, and the EEG learning network model after self-learning is used as a decoding model of the two-hand coupled brain-computer interface paradigm.

3. The decoding model acquisition method of the two-handed coupling brain-computer interface paradigm according to claim 2 is characterized in that: The EEG learning network model includes a two-dimensional convolutional layer, a deep convolutional layer, a separation convolutional layer, a fully connected layer and an activation layer; The step of inputting the collected EEG features of the subject into the EEG learning network model to obtain a decoding result for each subject includes: Inputting the EEG features of the subject into the two-dimensional convolution layer and the deep convolution layer of the EEG learning network model in sequence to extract the temporal features and spatial features of the EEG features respectively; Enhance the learning ability of each of the temporal features and spatial features through the separable convolutional layer of the EEG learning network model; After reshaping the feature dimension of the enhanced EEG features, the features are input into the fully connected layer to output the category probability; The Softmax activation function is used to process the category probabilities to obtain a decoding result of the two-hand coupled motion direction.

4. The decoding model acquisition method of the two-hand coupled brain-computer interface paradigm according to claim 3 is characterized in that: The step of sequentially inputting the EEG features of the subject into the two-dimensional convolution layer and the deep convolution layer of the EEG learning network model to extract the temporal features and spatial features of the EEG features, respectively, comprises: Inputting the EEG features of the subject into the two-dimensional convolutional layer of the EEG learning network model; The EEG features of each channel are convolved by the two-dimensional convolution layer to extract the time features in the EEG features; wherein the convolution kernel size of the two-dimensional convolution layer is 1 / 2 of the number of sampling points; Performing batch normalization processing on the EEG features output by the two-dimensional convolutional layer; Inputting the batch normalized EEG features into the deep convolution layer to extract spatial features in the EEG features of each channel; The EEG features output by the deep convolutional layer are normalized and processed to prevent overfitting.

5. The decoding model acquisition method of the two-handed coupling brain-computer interface paradigm according to claim 3 is characterized in that: After enhancing the learning ability of each of the temporal features and spatial features through the separable convolutional layer of the electroencephalogram learning network model, the method further includes: The features output by the separable convolutional layer are normalized and then anti-overfitting is performed.

6. The decoding model acquisition method of the two-handed coupling brain-computer interface paradigm according to claim 2 is characterized in that: Before inputting the collected EEG features of the subject into the EEG learning network model, the method further includes: Re-referencing the collected EEG signal; wherein the EEG signal is a neural decoding signal collected by the subject based on a two-hand coupled brain-computer interface paradigm; The re-referenced EEG signal is subjected to a bandpass filter with a preset frequency and then subjected to a global average reference process; Performing artifact removal processing on the EEG signal after global average reference to obtain an artifact-free EEG signal; Segmenting the artifact-free EEG signal according to the trigger synchronization label; The artifact-free EEG signals with amplitude exceeding a preset amplitude in each segment are removed for baseline calibration; After baseline calibration, the artifact-free EEG signals in each segment were low-pass filtered and EEG features were extracted.

7. The decoding model acquisition method of the two-handed coupling brain-computer interface paradigm according to claim 6 is characterized in that: The step of performing artifact removal processing on the EEG signal after global average reference to obtain an artifact-free EEG signal includes: The myoelectric artifact signal in the EEG signal after the global average reference is removed by an artifact component reconstruction algorithm; and the artifact component of the EEG signal after the global average reference is identified and removed by independent component analysis and a classifier.

8. A method for identifying the direction of intention based on two-hand coupled motion, characterized in that: include: Input the EEG characteristics of the target user into the decoding model of the bimanual coupling brain-computer interface paradigm to obtain the output result; The movement intention of the target user is determined according to the output result; wherein the decoding model of the two-handed coupled brain-computer interface paradigm is obtained by using the model acquisition method of the two-handed coupled brain-computer interface decoding paradigm as described in any one of claims 2-7.

9. A decoding model acquisition device for a two-handed coupled brain-computer interface paradigm, characterized in that: include: A decoding result acquisition module, used for inputting the collected EEG features of the subject into the EEG learning network model to obtain the decoding results of the movement direction of each of the subjects; wherein the EEG features are the features of the movement-related cortical potentials extracted by the subject based on the two-hand coupled brain-computer interface paradigm described in claim 1; A result accuracy determination module, used for comparing each of the motion direction decoding results with a preset result to obtain a result accuracy; The decoding model generation module is used to control the EEG learning network model to perform self-learning according to the accuracy rate if the accuracy rate is less than the preset accuracy rate, until the accuracy rate is greater than or equal to the preset accuracy rate, and use the EEG learning network model after self-learning as the decoding model of the two-handed coupled brain-computer interface paradigm.

10. A brain-computer terminal device, characterized in that: The terminal device comprises a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the method according to any one of claims 2 to 8.

Images