Multi-modal brain-computer interface-based motor imagery recognition method, device and system

Abstract

The invention discloses a motor imagery recognition method, device and system based on a multi-modal brain-computer interface, and the method comprises the steps: obtaining multi-modal feature data of a user, the multi-modal feature data comprising at least two of the following data: electroencephalogram feature data, electrocardio feature data and myoelectricity feature data; determining multi-modal fusion feature data based on preset weight information and the multi-modal feature data; and inputting the multi-modal fusion feature data into a classification model to obtain motor imagery of the user, the motor imagery being a target body part of the user imagining to move in a plurality of body parts of the user. Therefore, the motor imagery of the user is identified based on the multiple modal data, the data anti-interference capability is improved by utilizing complementarity among the multiple modal data, and the stability of the data is relatively good, so that the accuracy of the motor imagery identified based on the multiple modal data is relatively good.

Description

Technical Field

[0001] The present application belongs to the field of information processing technology, and in particular, relates to a method, device and system for motor imagery recognition based on a multimodal brain-computer interface. Background Art

[0002] In the prior art, brain-computer interface technology mainly extracts information and recognizes intentions based on single-modal data, thereby identifying the user's motor imagination. However, single-modal data has poor stability and is easily disturbed by various factors, such as environmental factors and the user's body movements. Therefore, the accuracy of identifying the user's motor imagination based on single-modal data for information extraction and intention recognition is low. Summary of the invention

[0003] The embodiment of the present application provides an implementation solution different from the prior art to solve the technical problem of low accuracy in identifying the user's motion imagination in the prior art.

[0004] In a first aspect, the present application provides a method for identifying motor imagery based on a multimodal brain-computer interface, comprising: obtaining multimodal feature data of a user, wherein the multimodal feature data comprises at least two of the following data: EEG feature data, ECG feature data, and EMG feature data; determining multimodal fusion feature data based on preset weight information and the multimodal feature data; inputting the multimodal fusion feature data into a classification model to obtain the user's motor imagery, wherein the motor imagery refers to a target body part that the user imagines moving among multiple body parts of the user.

[0005] In the second aspect, the present application provides a motion imagery recognition device based on a multimodal brain-computer interface, comprising: an acquisition unit, used to acquire a user's multimodal feature data, the multimodal feature data including at least two of the following data: EEG feature data, ECG feature data, and EMG feature data; a determination unit, used to determine multimodal fusion feature data based on preset weight information and the multimodal feature data; an input unit, used to input the multimodal fusion feature data into a classification model to obtain the user's motion imagery, the motion imagery refers to a target body part that the user imagines moving among multiple body parts of the user.

[0006] In the third aspect, the present application provides a motor imagery recognition system based on a multimodal brain-computer interface, including: a data processing module for obtaining a user's multimodal feature data, the multimodal feature data including at least two of the following data: EEG feature data, ECG feature data, and EMG feature data; determining multimodal fusion feature data based on preset weight information and the multimodal feature data; inputting the multimodal fusion feature data into a classification model to obtain the user's motor imagery, the motor imagery referring to a target body part that the user imagines moving among multiple body parts of the user.

[0007] In a fourth aspect, the present application provides an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any method in the first aspect or any possible implementation scheme of the first aspect by executing the executable instructions.

[0008] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program implements any method in the first aspect or any possible implementation manner of the first aspect.

[0009] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect or any possible implementation manner of the first aspect.

[0010] The solution provided by the present application obtains the multimodal feature data of the user, wherein the multimodal feature data includes at least two of the following data: EEG feature data, ECG feature data, and EMG feature data; determines the multimodal fusion feature data based on the preset weight information and the multimodal feature data; and inputs the multimodal fusion feature data into the classification model to obtain the user's motor imagination, wherein the motor imagination refers to the scheme of the target body part that the user imagines moving among the multiple body parts of the user. The target body part that the user imagines moving is identified based on the multiple modal data, and the complementarity between the multiple modal data is utilized to improve the data anti-interference ability, so that the data stability is better, thereby making the accuracy of the motor imagination identified based on the multiple modal data better. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0012] Figure 1 A flowchart of a method for motor imagery recognition based on a multimodal brain-computer interface provided in one embodiment of the present application;

[0013] Figure 2 A schematic diagram of the structure of a motor imagery recognition device based on a multimodal brain-computer interface is provided for one embodiment of the present application;

[0014] Figure 3 A schematic diagram of the structure of a motor imagery recognition system based on a multimodal brain-computer interface provided in one embodiment of the present application;

[0015] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, but cannot be understood as limiting the present application.

[0017] The terms "first" and "second" etc. in the specification, claims and drawings of the embodiments of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0018] First, some terms in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0019] EEG: (Electroencephalography) is a medical examination technology used to measure the electrical activity of the brain. By placing electrodes on the scalp, EEG is able to record electrical signals between neurons, reflecting the functional state of the brain and neural activity.

[0020] EMG: (Electromyography) is a technology used to measure the electrical activity of muscles. EMG can record and analyze the electrical signals generated by muscles during contraction or relaxation, providing important information about muscle function and nerve conduction status.

[0021] ECG: (Electrocardiography) is a medical technology used to record the electrical activity of the heart. ECG can monitor the electrical signals generated by the heart with each heartbeat through electrodes, providing important information about the health and function of the heart.

[0022] BCI: (Brain-Computer Interface) mainly refers to the communication or control with external devices through a single signal input mode.

[0023] Motor imagery: refers to the process in which the user imagines and simulates movement through thinking without actual movement.

[0024] CT: (Computed Tomography) is a medical imaging technology that uses X-rays and computer technology to generate cross-sectional images of structures inside the body.

[0025] MRI: (Magnetic Resonance Imaging) is a medical imaging technique that uses strong magnetic fields and radio waves to produce detailed images of tissues and organs in the body.

[0026] Multimodal data of brain-computer interface: refers to the combination of data from different signal sources or sensors, for example, at least two of the following data: EEG signals, EMG signals, and ECG signals.

[0027] Brain-computer interface technology is widely used in many fields, such as medical field, rehabilitation field and industrial safety field. Traditional brain-computer interface technology is mainly based on single modality data (such as EEG signal) for information extraction and intention recognition. Thus, the user's motor imagination can be identified. However, single modality data has poor stability and is easily disturbed by various factors, such as environmental factors and the user's body movement. Therefore, the accuracy of identifying the user's motor imagination by brain-computer interface technology based on single modality data for information extraction and intention recognition is low. For example, in some studies and applications related to motor imagination, when only relying on EEG signals to judge the user's motor imagination, due to the high individual differences of EEG signals, the EEG signal characteristics of different individuals may show large differences under the same motor imagination, and electromagnetic interference in the environment, the user's body movement artifacts, etc. may interfere with the EEG signal, resulting in a decrease in signal quality, so that the accuracy of judging the user's motor imagination by relying solely on EEG signals is low. That is, the accuracy of traditional brain-computer interface technology is low, and it has large individual differences and poor stability.

[0028] In addition, in the field of medical diagnosis, there are technologies that fuse multiple medical imaging data (such as CT, MRI, etc.) to improve the accuracy of disease diagnosis. However, these fusion technologies are mainly aimed at image data, and their fusion methods and processing processes are very different from the multimodal data fusion in brain-computer interface technology. Therefore, they cannot be directly applied to the fusion of multimodal data to improve the accuracy of brain-computer interface technology. In addition, in intelligent transportation systems, there are also applications of multi-sensor data fusion, such as fusing vehicle speed, position, camera images and other information to achieve automatic driving or traffic flow monitoring. However, the types and characteristics of these data are quite different from the types and characteristics of multimodal data in brain-computer interface technology, so they cannot be directly applied to the fusion of multimodal victory data to improve the accuracy of brain-computer interface technology.

[0029] In summary, the existing brain-computer interface technology based on single-modal data has poor anti-interference ability, poor accuracy and stability, and is difficult to meet the high-precision requirements for the recognition of the user's motor imagination intention in practical applications. In addition, there is a lack of effective fusion methods for multi-modal data of brain-computer interfaces, and it is impossible to fully utilize the advantages of multi-modal data of brain-computer interfaces.

[0030] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0031] Figure 1 A flowchart of a method for motor imagery recognition based on a multimodal brain-computer interface is provided as an exemplary embodiment of the present application. The execution subject of the method may be a server. The method includes at least the following steps:

[0032] S101, obtaining multimodal feature data of a user, wherein the multimodal feature data includes at least two of the following data: electroencephalogram feature data, electrocardiogram feature data, and electromyography feature data;

[0033] Optionally, the multimodal feature data refers to representative data extracted from multiple original modal data to illustrate the properties or characteristics of the original modal data, wherein the multimodal feature data corresponds one-to-one to the multiple original modal data. Specifically, EEG feature data refers to representative data extracted from EEG signals, ECG feature data refers to representative data extracted from ECG signals, and muscle point feature data refers to representative data extracted from electromyography signals.

[0034] Optionally, in the aforementioned S101, before obtaining the multimodal feature data of the user, the method further includes obtaining an initial multimodal signal of the user, wherein the initial multimodal signal includes at least two of the following data: an initial EEG signal, an initial ECG signal, and an initial EMG signal;

[0035] Optionally, the initial multimodal signal can be acquired through the multimodal data acquisition module, and the acquired initial multimodal signal can be sent to the server so that the server can extract the multimodal feature data of the initial multimodal signal. Specifically, the initial EEG signal can be acquired through the EEG signal acquisition device in the multimodal data acquisition module, and the initial EEG signal can be sent to the server; the initial EMG signal can be acquired through the EMG signal acquisition device in the multimodal data acquisition module, and the initial EMG signal can be sent to the server; the initial ECG signal can be acquired through the ECG signal acquisition device in the multimodal data acquisition module, and the initial ECG signal can be sent to the server.

[0036] The aforementioned step of obtaining the multimodal feature data of the user in S101 includes the following steps S01-S02:

[0037] S01. performing data preprocessing on the initial multimodal signal to obtain a target multimodal signal;

[0038] Optionally, the target multimodal signal includes at least two of the following data: a target EEG signal, a target ECG signal, and a target myoelectric signal, and the target multimodal signal corresponds one-to-one to the initial multimodal signal. For example, if the initial multimodal signal includes an initial EEG signal and an initial ECG signal, then the target multimodal signal includes a target EEG signal and a target ECG signal; if the initial multimodal signal includes an initial EEG signal, an initial ECG signal, and an initial myoelectric signal, then the target multimodal signal includes a target EEG signal, a target ECG signal, and a target myoelectric signal.

[0039] Optionally, the data preprocessing refers to performing data processing on the initial multimodal signal to remove noise in the data, or processing the data to a specified data range to improve the usability of the data.

[0040] Optionally, if the initial multimodal signal includes an initial EEG signal, performing data preprocessing on the initial EEG signal to obtain a target EEG signal includes the following steps S010-S013:

[0041] S010, performing bandpass filtering on the initial EEG signal based on a preset first bandpass range to obtain a first EEG signal;

[0042] Optionally, the preset first passband range may be set to be no less than 0.5 Hz and no more than 45 Hz to remove high-frequency noise and DC drift. Specifically, the preset first passband range may also be set to other ranges, which are not specifically limited here.

[0043] S011, analyzing and processing the first EEG signal using an independent component analysis algorithm to obtain a second EEG signal;

[0044] Optionally, an independent component analysis algorithm is used to decompose the multivariate signal into independent non-Gaussian signal components. The independent component analysis algorithm includes a variety of algorithms, which can be found in the prior art and will not be described in detail here. The independent component analysis algorithm can be used to remove false signals such as eye movements and blinks to remove noise in the EEG signal and improve the purity of the EEG signal.

[0045] S012, performing re-reference processing on the second EEG signal to obtain a third EEG signal;

[0046] Optionally, when re-referencing the second EEG signal, an average reference electrode or a specific brain region electrode may be used as a reference electrode so that the reference electrodes of the third EEG signal are all the same, thereby enhancing the comparability of the EEG signals. The specific brain region electrode may be a mastoid electrode.

[0047] Optionally, the specific processing procedure of how to perform re-reference processing on the second EEG signal based on the average reference electrode or the specific brain area electrode as the reference electrode can be referred to in the prior art, which will not be described in detail here.

[0048] S013. Segment the third EEG signal based on a preset duration to obtain multiple EEG signal segments, and use the multiple EEG signal segments as target EEG signals, wherein the duration of each EEG signal segment is equal to the preset duration.

[0049] Optionally, the preset duration can be 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 6 seconds, etc., and the preset duration can be set according to actual needs.

[0050] Optionally, if the initial multimodal data includes an initial electromyographic signal, performing data preprocessing on the initial electromyographic signal to obtain a target electromyographic signal includes the following steps S0110-S0114:

[0051] S0110, performing high-pass filtering on the initial electromyographic signal based on a preset frequency threshold to obtain a first electromyographic signal;

[0052] Optionally, the preset frequency threshold is a cut-off frequency, and the cut-off frequency may be a value such as 18 Hz, 19 Hz, 20 Hz, or 21 Hz.

[0053] Specifically, in S0110, high-pass filtering is performed on the initial electromyographic signal based on a preset frequency threshold to obtain a first electromyographic signal including:

[0054] The preset frequency threshold is used as the cutoff frequency of the initial electromyographic signal, and the signal in the initial electromyographic signal that is higher than the preset frequency threshold is removed to obtain a first electromyographic signal, wherein the frequency in the first electromyographic signal is not greater than the preset frequency threshold.

[0055] S0111, rectifying and smoothing the first electromyographic signal to obtain a second electromyographic signal;

[0056] Optionally, the specific implementation process of rectifying and smoothing the first electromyographic signal can be referred to the prior art, which will not be described in detail here.

[0057] S0112, performing low-pass filtering on the second electromyographic signal to obtain a first envelope;

[0058] S0113, normalizing the first envelope to obtain a second envelope;

[0059] Optionally, the amplitude in the first envelope may be normalized according to maximum voluntary contraction or other suitable methods to facilitate data comparison between different muscle groups and different individuals.

[0060] S0114. Segment the second envelope based on the preset duration to obtain multiple envelope segments, and use the multiple envelope segments as target electromyographic signals, wherein the duration of each envelope segment is equal to the preset duration, that is, equal to the duration of each EEG signal segment.

[0061] Optionally, the preset duration is the aforementioned preset duration.

[0062] Optionally, if the initial multimodal data includes an initial ECG signal, performing data preprocessing on the initial ECG signal to obtain a target ECG signal includes the following steps S01110-S01111:

[0063] S01110, performing bandpass filtering on the initial electrocardiogram signal based on a preset second bandpass range to obtain a first electrocardiogram signal;

[0064] Optionally, the preset second passband range may be set to be no less than 0.5 Hz and no more than 40 Hz to remove high-frequency noise and baseline drift. Specifically, the preset second passband range may also be set to other ranges, which are not specifically limited here.

[0065] S01111. Segment the first ECG signal based on the preset duration to obtain multiple segments of second ECG signals, and use the multiple segments of second ECG signals as target ECG signals, wherein the duration of each segment of the second ECG signal is equal to the preset duration, that is, equal to the duration of each segment of the EEG signal.

[0066] Optionally, the preset duration is the aforementioned preset duration.

[0067] S02. Perform feature extraction processing on the target multimodal signal to obtain the multimodal feature data.

[0068] Optionally, the multimodal feature data includes at least two of the following data: EEG feature data, ECG feature data, and EMG feature data, and the multimodal feature data corresponds one-to-one to the target multimodal signal. For example, if the target multimodal signal includes a target EEG signal and a target ECG signal, the multimodal feature data includes EEG feature data and ECG feature data; if the target multimodal signal includes a target EEG signal, a target ECG signal, and a target EMG signal, the multimodal feature data includes EEG feature data, ECG feature data, and EMG feature data.

[0069] Optionally, if the target multimodal signal includes a target EEG signal, and the target EEG signal includes multiple EEG signal segments, then the method performs feature extraction processing on the target EEG signal to obtain EEG feature data, including: performing feature extraction processing on each segment of the target EEG signal to obtain first feature data, thereby obtaining multiple segments of first feature data, and using the multiple segments of first feature data as EEG feature data.

[0070] Optionally, extracting features from each segment of the target EEG signal to obtain first feature data includes the following steps S021-S024:

[0071] S021, extracting event-related potential components from the electroencephalogram signal to obtain at least one event-related potential component;

[0072] Optionally, the at least one event-related potential component may be a movement-related potential component. Specifically, the meanings and extraction processes of the event-related potential component and the movement-related potential component can be found in the prior art, and will not be described in detail here.

[0073] S022, performing calculation processing on the at least one event-related potential component to obtain at least one latency and at least one amplitude;

[0074] Alternatively, latency refers to the period during which an event-related potential component appears, such as from the moment preparation for the event begins to the moment the event begins.

[0075] Optionally, amplitude refers to the magnitude of the maximum negative or positive value of an event-related potential component.

[0076] S023. Processing the EEG signal using a time-frequency analysis method to obtain at least three energy distributions and at least three power spectrum densities, wherein the at least three energy distributions include at least an energy distribution of an alpha frequency band, an energy distribution of a beta frequency band, and an energy distribution of a gamma frequency band, and the at least three power spectrum densities include at least a power spectrum density of an alpha frequency band, a power spectrum density of a beta frequency band, and a power spectrum density of a gamma frequency band;

[0077] Optionally, the relevant explanations about alpha, beta and gamma can be found in the prior art, which will not be elaborated here.

[0078] Optionally, the time-frequency analysis method may adopt a wavelet transform method, or may adopt other time-frequency analysis methods in the prior art.

[0079] S024. Use the at least one latent period, the at least one amplitude, the at least three energy distributions and the at least three power spectrum densities as first characteristic data.

[0080] Optionally, if the target multimodal signal includes a target electromyographic signal, and the target electromyographic signal includes multiple segments of electromyographic signals, then in the method, feature extraction processing is performed on the target electromyographic signal to obtain electromyographic feature data, including: feature extraction processing is performed on each segment of the electromyographic signal in the target electromyographic signal (each segment of the envelope in multiple segments of envelopes) to obtain second feature data, thereby obtaining multiple segments of second feature data, and using the multiple segments of second feature data as electromyographic feature data.

[0081] Optionally, extracting features from each segment of the target electromyographic signal to obtain second feature data includes the following steps S0221-S0223:

[0082] S0221. Calculate the root mean square value of the electromyographic signal to obtain an electromyographic root mean square value;

[0083] Optionally, the method for calculating the root mean square value of the electromyographic signal may refer to the root mean square value calculation method in the prior art, which will not be described in detail here.

[0084] Optionally, during motor imagery, different imagined movements may cause the RMS value of the corresponding muscle group to change, so the calculated RMS value of the electromyography can reflect the overall intensity of muscle activity.

[0085] S0222, extracting the zero-crossing rate of the electromyographic signal to obtain the electromyographic zero-crossing rate;

[0086] Optionally, the zero-crossing rate is related to the contraction speed of the muscle and different movements, so the frequency characteristics of the muscle activity can be better reflected by the myoelectric zero-crossing rate.

[0087] S0223. Use the myoelectric root mean square value and the myoelectric zero-crossing rate as the second characteristic data.

[0088] Optionally, if the target multimodal signal includes a target ECG signal, and the target ECG signal includes multiple segments of ECG signals, then the method performs feature extraction processing on the target ECG signal to obtain ECG feature data, including: performing feature extraction processing on each segment of the target ECG signal to obtain third feature data, thereby obtaining multiple segments of third feature data, and using the multiple segments of third feature data as ECG feature data.

[0089] Optionally, extracting features from each segment of the target ECG signal to obtain third feature data includes the following steps S02221-S02224:

[0090] S02221. Process the electrocardiogram signal using a QRS wave detection algorithm to obtain an R wave;

[0091] Optionally, the relevant explanation and processing process of the QRS wave detection algorithm can be found in the prior art, which will not be described in detail here.

[0092] S0222. Obtaining heart rate and heart rate variability based on the R wave;

[0093] S0223. Obtaining a time domain index and a frequency domain index based on the heart rate variability, wherein the time domain index includes at least SDNN and RMSSD, and the frequency domain index includes at least a high frequency component, a low frequency component, a ratio of the low frequency component to the high frequency component, and a total power;

[0094] Specifically, SDNN refers to the standard deviation of the intervals between adjacent R waves, and RMSSD refers to the root mean square of the differences between adjacent RR intervals.

[0095] S0224. Use the heart rate, time domain index and frequency domain index as third feature data.

[0096] S102, determining multimodal fusion feature data based on preset weight information and the multimodal feature data;

[0097] Optionally, in the aforementioned S102, the preset weight information includes a first weight value corresponding to the EEG characteristic data, a second weight value corresponding to the ECG characteristic data, and a third weight value corresponding to the EMG characteristic data.

[0098] Optionally, the first weight value, the second weight value and the third weight value are all the same, partially the same, or different.

[0099] Optionally, the first weight value, the second weight value and the third weight value may all be 1 / 3.

[0100] Optionally, the first weight value, the second weight value and the third weight value can also be set according to the importance of different feature data. The higher the importance of the feature data, the larger the corresponding weight value. For example, if the EEG feature data is more important, the first weight value can be set to a higher value, higher than the second weight value and the third weight value. If the ECG feature data is more important than the EMG feature data, the second weight value is higher than the third weight value.

[0101] Optionally, in the aforementioned S102, the determining of the multimodal fusion feature data based on the preset weight information and the multimodal feature data includes the following steps S001-S004:

[0102] S001. Obtaining EEG fusion feature data based on the first weight value and the EEG feature data;

[0103] Optionally, the EEG feature data includes a plurality of first feature data, and a weight value for each feature data in the EEG feature data is a value obtained by dividing the first weight value by the number of first feature data in the plurality of first feature data.

[0104] Optionally, the aforementioned S001 obtains EEG fusion feature data based on the first weight value and the EEG feature data, including: obtaining EEG fusion feature data based on a preset fusion method, the first weight value and the EEG feature data, wherein the preset fusion method includes at least one of the following methods: a weighted fusion method and a principal component analysis method.

[0105] S002. Obtaining ECG fusion feature data based on the second weight value and the ECG feature data;

[0106] Optionally, the electrocardiogram characteristic data includes a plurality of third characteristic data, and a weight value for each third characteristic data in the electrocardiogram characteristic data is a value obtained by dividing the second weight value by the number of the third characteristic data in the plurality of third characteristic data.

[0107] Optionally, the aforementioned S002 obtains ECG fusion feature data based on the second weight value and the ECG feature data, including: obtaining ECG fusion feature data based on a preset fusion method, the second weight value and the ECG feature data, wherein the preset fusion method includes at least one of the following methods: a weighted fusion method and a principal component analysis method.

[0108] S003, obtaining electromyographic fusion feature data based on the third weight value and the electromyographic feature data;

[0109] Optionally, the electromyographic feature data includes a plurality of second feature data, and the weight value for each second feature data in the electromyographic feature data is the value obtained by dividing the third weight value by the number of second feature data in the plurality of second feature data.

[0110] Optionally, in the aforementioned S003, electromyographic fusion feature data is obtained based on the third weight value and the electromyographic feature data, including: based on a preset fusion method, the third weight value and the electromyographic feature data, electromyographic fusion feature data is obtained, wherein the preset fusion method includes at least one of the following methods: a weighted fusion method and a principal component analysis method.

[0111] S004. Determine multimodal fusion feature data based on the electroencephalogram fusion feature data, the electrocardiogram fusion feature data, and the electromyography fusion feature data.

[0112] Optionally, in the aforementioned S004, determining the multimodal fusion feature data based on the EEG fusion feature data, the EKG fusion feature data and the EMG fusion feature data includes: using the EEG fusion feature data, the EKG fusion feature data and the EMG fusion feature data as multimodal fusion feature data.

[0113] S103: Input the multimodal fusion feature data into a classification model to obtain the user's movement imagination, where the movement imagination refers to the target body part that the user imagines moving among the user's multiple body parts.

[0114] Optionally, the multiple body parts include a left hand part, a right hand part, a head part, a left foot part, a right foot part, a left leg part and a right leg part.

[0115] Optionally, the method further comprises the following steps S0001-S0002:

[0116] S0001, generating a motion control instruction based on the motion imagery;

[0117] S0002. Control the target device through the motion control instruction, so that the target device executes the motion corresponding to the motion control instruction, or assists the user to execute the motion corresponding to the motion imagination.

[0118] Optionally, the target device may be a prosthesis, a wheelchair, a smart home device, etc.

[0119] Optionally, the motion control instruction may be sent to the target device via wireless communication technology, so as to control the target device based on the motion control instruction.

[0120] Optionally, the method also includes: obtaining motion state information after the target device executes the motion corresponding to the motion control instruction, or assisting the user to execute the motion corresponding to the motion imagination; optimizing the classification model based on the motion state information to improve the accuracy and stability of the classification model prediction.

[0121] Optionally, in the field of rehabilitation medicine, this solution can assist stroke patients, spinal cord injured persons, etc. in effective rehabilitation training and assisted motion control; in the field of smart home control, this solution can enable users to more reliably control home appliances through motor imagination; in the field of virtual reality and augmented reality interaction, this solution provides a more natural and accurate interaction method, enhancing the user experience.

[0122] Optionally, the method also includes: training an initial classification model based on sample data to obtain the classification model, wherein the sample data includes multiple multimodal sample feature data and labels corresponding to each multimodal sample feature data, and the initial classification model is any one of the following models: a support vector machine, a neural network, and a deep learning model.

[0123] Optionally, the neural network may be a multilayer perceptron or a convolutional neural network, and the deep learning model may be a long short-term memory network.

[0124] Optionally, in the process of training the initial classification model based on the sample data and obtaining the classification model, techniques such as cross-validation and regularization may be used to train the initial classification model to prevent overfitting and improve the generalization ability of the classification model.

[0125] In summary, the present application provides a method for obtaining multimodal feature data of a user, wherein the multimodal feature data includes at least two of the following data: EEG feature data, ECG feature data, and EMG feature data; multimodal fusion feature data is determined based on preset weight information and the multimodal feature data; the multimodal fusion feature data is input into a classification model to obtain the user's motor imagination, wherein the motor imagination refers to a scheme of a target body part that the user imagines moving among multiple body parts of the user. The target body part that the user imagines moving is identified based on multiple modal data, and the complementarity between multiple modal data is utilized to improve the data's anti-interference ability, so that the data has better stability, thereby improving the accuracy of the motor imagination identified based on multiple modal data.

[0126] Specifically, EEG signals reflect the brain's cognitive and motor planning processes, EMG signals can predict the readiness of muscle movements in advance, and changes in ECG signals are related to the body's stress and excitement states. Fusion of multimodal data such as EEG, EMG, and ECG can predict the user's motor imagery from multiple perspectives such as brain neural activity, muscle movement preparation, and cardiovascular system response, and can more accurately judge the user's motor imagery intention. Different modal data have different sensitivities to noise and interference. When the EEG signal is subject to electromagnetic interference, the EMG and ECG signals may not be affected or be less affected. Through data fusion, adaptive adjustments can be made based on the reliability of each modal data, and good performance can still be maintained when some data is interfered with, thereby improving the stability and robustness of predicting the user's motor imagery and reducing the occurrence of misjudgment. The weight values ​​of each modal data in the multimodal data can be personalized. For users with different muscle strength, cardiovascular function, and EEG signal characteristics, different weight values ​​can be set for feature data fusion to adapt to individual differences, providing a more accurate and personalized way to predict the user's motor imagery.

[0127] Figure 2 A schematic diagram of a motor imagery recognition device based on a multimodal brain-computer interface is provided as a structural diagram of an exemplary embodiment of the present application; wherein the device comprises:

[0128] An acquisition unit 21 is used to acquire multimodal feature data of a user, wherein the multimodal feature data includes at least two of the following data: EEG feature data, ECG feature data, and EMG feature data;

[0129] A determination unit 22, configured to determine multimodal fusion feature data based on preset weight information and the multimodal feature data;

[0130] The input unit 23 is used to input the multimodal fusion feature data into the classification model to obtain the user's movement imagination, where the movement imagination refers to the target body part that the user imagines moving among the user's multiple body parts.

[0131] Optionally, in the device, the preset weight information includes a first weight value corresponding to the EEG characteristic data, a second weight value corresponding to the ECG characteristic data, and a third weight value corresponding to the EMG characteristic data.

[0132] Optionally, in the device, the first weight value, the second weight value and the third weight value are all the same, partially the same, or different.

[0133] Optionally, the device is used to determine the multimodal fusion feature data based on preset weight information and the multimodal feature data, specifically for: obtaining EEG fusion feature data based on the first weight value and the EEG feature data; obtaining EKG fusion feature data based on the second weight value and the EKG feature data; obtaining EMG fusion feature data based on the third weight value and the EMG feature data; and determining the multimodal fusion feature data based on the EEG fusion feature data, the EKG fusion feature data and the EMG fusion feature data.

[0134] Optionally, the apparatus is also used to: generate motion control instructions based on the motor imagination; control a target device through the motion control instructions so that the target device performs the motion corresponding to the motion control instructions, or assists the user in performing the motion corresponding to the motor imagination.

[0135] Optionally, the device is also used to: obtain an initial multimodal signal of the user, the initial multimodal signal including at least two of the following data: an initial EEG signal, an initial ECG signal, and an initial EMG signal; the device is used to obtain the user's multimodal feature data, specifically to: perform data preprocessing on the initial multimodal signal to obtain a target multimodal signal; perform feature extraction processing on the target multimodal signal to obtain the multimodal feature data.

[0136] Optionally, the device is also used to: train an initial classification model based on sample data to obtain the classification model, wherein the sample data includes multiple multimodal sample feature data and labels corresponding to each multimodal sample feature data, and the initial classification model is any one of the following models: a support vector machine, a neural network, and a deep learning model.

[0137] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the device may perform the above method embodiment, and the above and other operations and / or functions of each module in the device are the corresponding processes in each method in the above method embodiment, respectively, and no further description is given here for the sake of brevity.

[0138] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0139] Figure 3 A schematic diagram of a motor imagery recognition system based on a multimodal brain-computer interface is provided as a structural diagram of an exemplary embodiment of the present application; wherein the system comprises:

[0140] The data processing module 31 is used to obtain the multimodal feature data of the user, wherein the multimodal feature data includes at least two of the following data: EEG feature data, ECG feature data and EMG feature data; determine multimodal fusion feature data based on preset weight information and the multimodal feature data; input the multimodal fusion feature data into a classification model to obtain the user's motor imagination, wherein the motor imagination refers to the target body part that the user imagines moving among multiple body parts of the user.

[0141] Optionally, the multimodal feature data refers to representative data extracted from multiple original modal data to illustrate the properties or characteristics of the original modal data, wherein the multimodal feature data corresponds one-to-one to the multiple original modal data. Specifically, EEG feature data refers to representative data extracted from EEG signals, ECG feature data refers to representative data extracted from ECG signals, and muscle point feature data refers to representative data extracted from electromyography signals.

[0142] Alternatively, if Figure 3 As shown, the system also includes: a multimodal data acquisition module 32, which is used to acquire the user's initial multimodal signal, and the initial multimodal signal includes at least two of the following data: an initial EEG signal, an initial ECG signal, and an initial EMG signal; the data processing module 31 is used to obtain the user's multimodal feature data, specifically used to: perform data preprocessing on the initial multimodal signal to obtain a target multimodal signal; perform feature extraction processing on the target multimodal signal to obtain the multimodal feature data.

[0143] Alternatively, if Figure 3 As shown, the multimodal data acquisition module 32 is further used to send the acquired initial multimodal signal to the server so that the server can extract multimodal feature data of the initial multimodal signal.

[0144] Optionally, the multimodal data acquisition module 32 includes an EEG signal acquisition device 321, an ECG signal acquisition device 322, and an EMG signal acquisition device 323. Specifically, the EEG signal acquisition device 321 is used to acquire initial EEG signals and send the initial EEG signals to a server; the EMG signal acquisition device 323 is used to acquire initial EMG signals and send the initial EMG signals to a server; the ECG signal acquisition device 322 is used to acquire initial ECG signals and send the initial ECG signals to a server.

[0145] Optionally, the EEG signal acquisition device 321 can use a highly sensitive multi-channel EEG acquisition device, and the electrodes of the multi-channel EEG acquisition device can be distributed on the scalp according to the international 10-20 system to ensure that all functional areas of the brain can be fully covered. The sampling rate of the multi-channel EEG acquisition device can be set to a sampling rate of 500 Hz or higher to accurately capture the rapid changes of the initial EEG signal. The multi-channel EEG acquisition device can use active electrode technology to reduce the impedance between the electrode and the scalp, improve signal quality, and reduce the impact of motion artifacts. Among them, the channels of the multi-channel EEG acquisition device can be set to 64 channels or 128 channels or other higher number of channels.

[0146] Optionally, the surface electromyographic sensor of the electromyographic signal acquisition device 323 is attached to the surface of the muscle group related to motor imagery, and the sampling rate of the electromyographic signal acquisition device can be set to 1000 Hz or higher to accurately record the subtle changes in muscle electrical activity. The sensor of the electromyographic signal acquisition device can adopt differential amplification technology to enhance the anti-interference ability of the signal; the electromyographic signal acquisition device can be equipped with a high-precision analog-to-digital converter to convert the analog signal into a digital signal. Among them, the surface of the muscle group can be the biceps brachii, triceps brachii, etc. of the upper limbs, and the quadriceps femoris, gastrocnemius, etc. of the lower limbs.

[0147] Optionally, the ECG signal acquisition device 322 may be a three-lead or multi-lead ECG acquisition device; the electrodes of the ECG signal acquisition device are placed at appropriate positions on the chest, such as the right arm, left arm, and left leg. The sampling rate of the ECG signal acquisition device may be set to 250 Hz or higher to ensure that the waveform changes of the ECG signal can be accurately monitored. The ECG signal acquisition device may use filtering technology to remove common noise such as power frequency interference and improve the quality of the initial ECG signal.

[0148] Optionally, the target multimodal signal includes at least two of the following data: a target EEG signal, a target ECG signal, and a target myoelectric signal, and the target multimodal signal corresponds one-to-one to the initial multimodal signal. For example, if the initial multimodal signal includes an initial EEG signal and an initial ECG signal, then the target multimodal signal includes a target EEG signal and a target ECG signal; if the initial multimodal signal includes an initial EEG signal, an initial ECG signal, and an initial myoelectric signal, then the target multimodal signal includes a target EEG signal, a target ECG signal, and a target myoelectric signal.

[0149] Optionally, the data preprocessing refers to performing data processing on the initial multimodal signal to remove noise in the data, or processing the data to a specified data range to improve the usability of the data.

[0150] Optionally, if the initial multimodal signal includes an initial EEG signal, the data processing module 31, when used to perform data preprocessing on the initial EEG signal to obtain a target EEG signal, is specifically used to perform bandpass filtering on the initial EEG signal based on a preset first bandpass range to obtain a first EEG signal; analyze and process the first EEG signal using an independent component analysis algorithm to obtain a second EEG signal; re-reference the second EEG signal to obtain a third EEG signal; segment the third EEG signal based on a preset duration to obtain multiple EEG signals, and use the multiple EEG signals as target EEG signals, wherein the duration of each segment of the EEG signal is equal to the preset duration.

[0151] Optionally, the preset first passband range may be set to be no less than 0.5 Hz and no more than 45 Hz to remove high-frequency noise and DC drift. Specifically, the preset first passband range may also be set to other ranges, which are not specifically limited here.

[0152] Optionally, an independent component analysis algorithm is used to decompose the multivariate signal into independent non-Gaussian signal components. The independent component analysis algorithm includes a variety of algorithms, which can be found in the prior art and will not be described in detail here. The independent component analysis algorithm can be used to remove false signals such as eye movements and blinks to remove noise in the EEG signal and improve the purity of the EEG signal.

[0153] Optionally, when re-referencing the second EEG signal, an average reference electrode or a specific brain region electrode may be used as a reference electrode so that the reference electrodes of the third EEG signal are all the same, thereby enhancing the comparability of the EEG signals. The specific brain region electrode may be a mastoid electrode.

[0154] Optionally, the preset duration can be 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 6 seconds, etc., and the preset duration can be set according to actual needs.

[0155] Optionally, if the initial multimodal data includes an initial electromyographic signal, the data processing module 31, when used to perform data preprocessing on the initial electromyographic signal to obtain a target electromyographic signal, is specifically used to: high-pass filter the initial electromyographic signal based on a preset frequency threshold to obtain a first electromyographic signal; rectify and smooth the first electromyographic signal to obtain a second electromyographic signal; low-pass filter the second electromyographic signal to obtain a first envelope; normalize the first envelope to obtain a second envelope; segment the second envelope based on the preset duration to obtain multiple envelope segments, and use the multiple envelope segments as the target electromyographic signal, wherein the duration of each envelope segment is equal to the preset duration, that is, equal to the duration of each segment of the EEG signal.

[0156] Optionally, the preset frequency threshold is a cut-off frequency, and the cut-off frequency may be a value such as 18 Hz, 19 Hz, 20 Hz, or 21 Hz.

[0157] Optionally, the specific implementation process of rectifying and smoothing the first electromyographic signal can be referred to the prior art, which will not be described in detail here.

[0158] Optionally, the amplitude in the first envelope may be normalized according to maximum voluntary contraction or other suitable methods to facilitate data comparison between different muscle groups and different individuals.

[0159] Optionally, the preset duration is the aforementioned preset duration.

[0160] Optionally, if the initial multimodal data includes an initial ECG signal, the data processing module 31, when used to perform data preprocessing on the initial ECG signal to obtain a target ECG signal, is specifically used to: perform bandpass filtering on the initial ECG signal based on a preset second bandpass range to obtain a first ECG signal; perform segmented processing on the first ECG signal based on the preset time length to obtain multiple segments of second ECG signals, and use the multiple segments of second ECG signals as target ECG signals, wherein the duration of each segment of the second ECG signal is equal to the preset time length, that is, equal to the duration of each segment of the EEG signal.

[0161] Optionally, the preset second passband range may be set to be no less than 0.5 Hz and no more than 40 Hz to remove high-frequency noise and baseline drift. Specifically, the preset second passband range may also be set to other ranges, which are not specifically limited here.

[0162] Optionally, the preset duration is the aforementioned preset duration.

[0163] Optionally, the multimodal feature data includes at least two of the following data: EEG feature data, ECG feature data, and EMG feature data, and the multimodal feature data corresponds one-to-one to the target multimodal signal. For example, if the target multimodal signal includes a target EEG signal and a target ECG signal, the multimodal feature data includes EEG feature data and ECG feature data; if the target multimodal signal includes a target EEG signal, a target ECG signal, and a target EMG signal, the multimodal feature data includes EEG feature data, ECG feature data, and EMG feature data.

[0164] Optionally, if the target multimodal signal includes a target EEG signal, and the target EEG signal includes multiple EEG signal segments, then the data processing module 31, when used to perform feature extraction processing on the target EEG signal to obtain EEG feature data, is specifically used to: perform feature extraction processing on each segment of the target EEG signal respectively to obtain first feature data, thereby obtaining multiple segments of first feature data, and using the multiple segments of first feature data as EEG feature data.

[0165] Optionally, when the data processing module 31 is used to perform feature extraction on each segment of the target EEG signal to obtain the first feature data, it is specifically used to: extract the event-related potential component in the EEG signal to obtain at least one event-related potential component; perform calculation processing on the at least one event-related potential component to obtain at least one latency and at least one amplitude; process the EEG signal using a time-frequency analysis method to obtain at least three energy distributions and at least three power spectral densities, wherein the at least three energy distributions include at least the energy distribution of the alpha band, the energy distribution of the beta band, and the energy distribution of the gamma band, and the at least three power spectral densities include at least the power spectral density of the alpha band, the power spectral density of the beta band, and the power spectral density of the gamma band; and use the at least one latency, at least one amplitude, at least three energy distributions, and at least three power spectral densities as the first feature data.

[0166] Optionally, the at least one event-related potential component may be a movement-related potential component. Specifically, the meanings and extraction processes of the event-related potential component and the movement-related potential component can be found in the prior art, and will not be described in detail here.

[0167] Alternatively, latency refers to the period during which an event-related potential component appears, such as from the moment preparation for the event begins to the moment the event begins.

[0168] Optionally, amplitude refers to the magnitude of the maximum negative or positive value of an event-related potential component.

[0169] Optionally, the relevant explanations about alpha, beta and gamma can be found in the prior art, which will not be elaborated here.

[0170] Optionally, the time-frequency analysis method may adopt a wavelet transform method, or may adopt other time-frequency analysis methods in the prior art.

[0171] Optionally, if the target multimodal signal includes a target electromyographic signal, and the target electromyographic signal includes multiple segments of electromyographic signals, then the data processing module 31, when used to perform feature extraction processing on the target electromyographic signal to obtain electromyographic feature data, is specifically used to: perform feature extraction processing on each segment of the electromyographic signal in the target electromyographic signal (each segment of the envelope in multiple segments of envelopes) to obtain second feature data, thereby obtaining multiple segments of second feature data, and using the multiple segments of second feature data as electromyographic feature data.

[0172] Optionally, when the data processing module 31 is used to perform feature extraction on each segment of the target electromyographic signal to obtain the second feature data, it is specifically used to: calculate the root mean square value of the electromyographic signal to obtain the electromyographic root mean square value; extract the zero-crossing rate of the electromyographic signal to obtain the electromyographic zero-crossing rate; and use the electromyographic root mean square value and the electromyographic zero-crossing rate as the second feature data.

[0173] Optionally, the method for calculating the root mean square value of the electromyographic signal may refer to the root mean square value calculation method in the prior art, which will not be described in detail here.

[0174] Optionally, during motor imagery, different imagined movements may cause the RMS value of the corresponding muscle group to change, so the calculated RMS value of the electromyography can reflect the overall intensity of muscle activity.

[0175] Optionally, the zero-crossing rate is related to the contraction speed of the muscle and different movements, so the frequency characteristics of the muscle activity can be better reflected by the myoelectric zero-crossing rate.

[0176] Optionally, if the target multimodal signal includes a target ECG signal, and the target ECG signal includes multiple segments of ECG signals, then the data processing module 31, when used to perform feature extraction processing on the target ECG signal to obtain ECG feature data, is specifically used to: perform feature extraction processing on each segment of the ECG signal in the target ECG signal respectively to obtain third feature data, thereby obtaining multiple segments of third feature data, and using the multiple segments of third feature data as ECG feature data.

[0177] Optionally, when the data processing module 31 is used to extract features from each segment of the target ECG signal to obtain the third feature data, it is specifically used to: process the ECG signal using a QRS wave detection algorithm to obtain an R wave; obtain heart rate and heart rate variability based on the R wave; obtain time domain indicators and frequency domain indicators based on the heart rate variability, wherein the time domain indicators include at least SDNN and RMSSD, and the frequency domain indicators include at least high-frequency components, low-frequency components, the ratio of low-frequency components to high-frequency components, and total power; and use the heart rate, time domain indicators, and frequency domain indicators as the third feature data.

[0178] Optionally, the relevant explanation and processing process of the QRS wave detection algorithm can be found in the prior art, which will not be described in detail here.

[0179] Specifically, SDNN refers to the standard deviation of the intervals between adjacent R waves, and RMSSD refers to the root mean square of the differences between adjacent RR intervals.

[0180] Optionally, in the system, the preset weight information includes a first weight value corresponding to the EEG characteristic data, a second weight value corresponding to the ECG characteristic data, and a third weight value corresponding to the EMG characteristic data.

[0181] Optionally, in the system, the first weight value, the second weight value and the third weight value are all the same, partially the same, or different.

[0182] Optionally, the first weight value, the second weight value and the third weight value may all be 1 / 3.

[0183] Optionally, the first weight value, the second weight value and the third weight value can also be set according to the importance of different feature data. The higher the importance of the feature data, the larger the corresponding weight value. For example, if the EEG feature data is more important, the first weight value can be set to a higher value, higher than the second weight value and the third weight value. If the ECG feature data is more important than the EMG feature data, the second weight value is higher than the third weight value.

[0184] Optionally, the data processing module 31 in the system is used to determine the multimodal fusion feature data based on the preset weight information and the multimodal feature data, specifically for: obtaining EEG fusion feature data based on the first weight value and the EEG feature data; obtaining EKG fusion feature data based on the second weight value and the EKG feature data; obtaining EMG fusion feature data based on the third weight value and the EMG feature data; and determining the multimodal fusion feature data based on the EEG fusion feature data, the EKG fusion feature data and the EMG fusion feature data.

[0185] Optionally, the EEG feature data includes a plurality of first feature data, and a weight value for each feature data in the EEG feature data is a value obtained by dividing the first weight value by the number of first feature data in the plurality of first feature data.

[0186] Optionally, when the data processing module 31 is used to obtain EEG fusion feature data based on the first weight value and the EEG feature data, it is specifically used to: obtain EEG fusion feature data based on a preset fusion method, the first weight value and the EEG feature data, wherein the preset fusion method includes at least one of the following methods: a weighted fusion method and a principal component analysis method.

[0187] Optionally, the electrocardiogram characteristic data includes a plurality of third characteristic data, and a weight value for each third characteristic data in the electrocardiogram characteristic data is a value obtained by dividing the second weight value by the number of the third characteristic data in the plurality of third characteristic data.

[0188] Optionally, when the data processing module 31 is used to obtain ECG fusion feature data based on the second weight value and the ECG feature data, it is specifically used to: obtain ECG fusion feature data based on a preset fusion method, the second weight value and the ECG feature data, wherein the preset fusion method includes at least one of the following methods: a weighted fusion method and a principal component analysis method.

[0189] Optionally, the electromyographic feature data includes a plurality of second feature data, and the weight value for each second feature data in the electromyographic feature data is the value obtained by dividing the third weight value by the number of second feature data in the plurality of second feature data.

[0190] Optionally, when the data processing module 31 is used to obtain electromyographic fusion feature data based on the third weight value and the electromyographic feature data, it is specifically used to: obtain electromyographic fusion feature data based on a preset fusion method, the third weight value and the electromyographic feature data, wherein the preset fusion method includes at least one of the following methods: a weighted fusion method and a principal component analysis method.

[0191] Optionally, the multiple body parts include a left hand part, a right hand part, a head part, a left foot part, a right foot part, a left leg part and a right leg part.

[0192] Optionally, the data processing module 31 is also used to: generate motion control instructions based on the motor imagination; control the target device through the motion control instructions, so that the target device performs the movement corresponding to the motion control instructions, or assists the user to perform the movement corresponding to the motor imagination.

[0193] Optionally, the system may include a target device, or the target device may be an external device not included in the system, and the motion control instruction may be sent to the target device via wireless communication technology to control the target device based on the motion control instruction. The target device may be a prosthesis, a wheelchair, a smart home device, etc.

[0194] Optionally, the data processing module 31 is also used to: obtain motion state information after the target device executes the motion corresponding to the motion control instruction, or assists the user in executing the motion corresponding to the motion imagination; and optimize the classification model based on the motion state information to improve the accuracy and stability of the classification model prediction.

[0195] Optionally, in the field of rehabilitation medicine, this solution can assist stroke patients, spinal cord injured persons, etc. in effective rehabilitation training and assisted motion control; in the field of smart home control, this solution can enable users to more reliably control home appliances through motor imagination; in the field of virtual reality and augmented reality interaction, this solution provides a more natural and accurate interaction method, enhancing the user experience.

[0196] Optionally, the data processing module 31 also includes a model training module, which is used to: train an initial classification model based on sample data to obtain the classification model, wherein the sample data includes multiple multimodal sample feature data and labels corresponding to each multimodal sample feature data, and the initial classification model is any one of the following models: select a support vector machine, a neural network, and a deep learning model.

[0197] Optionally, the initial classification model may be a support vector machine, a neural network or a deep learning model, etc. The neural network may be a multilayer perceptron or a convolutional neural network, etc., and the deep learning model may be a long short-term memory network.

[0198] Optionally, the data processing module 31 may train the initial classification model based on the sample data to obtain the classification model by using cross-validation, regularization and other techniques to prevent overfitting and improve the generalization ability of the classification model.

[0199] Figure 4 is a schematic block diagram of an electronic device provided in an embodiment of the present application, and the electronic device may include:

[0200] The memory 301 and the processor 302, the memory 301 is used to store the computer program and transmit the program code to the processor 302. In other words, the processor 302 can call and run the computer program from the memory 301 to implement the method in the embodiment of the present application.

[0201] For example, the processor 302 may be configured to execute the above method embodiments according to instructions in the computer program.

[0202] In some embodiments of the present application, the processor 302 may include but is not limited to:

[0203] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0204] In some embodiments of the present application, the memory 301 includes but is not limited to:

[0205] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0206] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 301 and executed by the processor 302 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0207] like Figure 4 As shown, the electronic device may also include:

[0208] The transceiver 303 may be connected to the processor 302 or the memory 301 .

[0209] The processor 302 may control the transceiver 303 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 303 may include a transmitter and a receiver. The transceiver 303 may further include an antenna, and the number of antennas may be one or more.

[0210] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0211] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, which, when executed by a computer, enables the computer to perform the method of the above method embodiment.

[0212] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0213] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0214] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0215] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0216] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A motor imagery recognition method based on a multimodal brain-computer interface, characterized in that: include: Acquire multimodal feature data of the user, wherein the multimodal feature data includes at least two of the following data: electroencephalogram feature data, electrocardiogram feature data, and electromyography feature data; Determining multimodal fusion feature data based on preset weight information and the multimodal feature data; The multimodal fusion feature data is input into a classification model to obtain the user's motion imagination, where the motion imagination refers to a target body part that the user imagines moving among multiple body parts of the user.

2. The method according to claim 1, characterized in that The preset weight information includes a first weight value corresponding to the EEG characteristic data, a second weight value corresponding to the ECG characteristic data, and a third weight value corresponding to the EMG characteristic data.

3. The method according to claim 2, characterized in that The first weight value, the second weight value, and the third weight value are all the same, partially the same, or different.

4. The method according to claim 2, characterized in that: The determining of the multimodal fusion feature data based on the preset weight information and the multimodal feature data includes: Obtaining EEG fusion feature data based on the first weight value and the EEG feature data; Obtaining ECG fusion feature data based on the second weight value and the ECG feature data; Based on the third weight value and the myoelectric feature data, obtaining myoelectric fusion feature data; Multimodal fusion feature data is determined based on the electroencephalogram fusion feature data, the electrocardiogram fusion feature data and the electromyography fusion feature data.

5. The method according to claim 1, characterized in that The method further comprises: generating a movement control instruction based on the movement imagery; The target device is controlled by the motion control instruction, so that the target device performs the motion corresponding to the motion control instruction, or assists the user to perform the motion corresponding to the motion imagination.

6. The method according to claim 1, characterized in that The method further comprises: Acquire an initial multimodal signal of the user, wherein the initial multimodal signal includes at least two of the following data: an initial electroencephalogram signal, an initial electrocardiogram signal, and an initial electromyography signal; The obtaining of multimodal feature data of the user includes: Performing data preprocessing on the initial multimodal signal to obtain a target multimodal signal; Perform feature extraction processing on the target multimodal signal to obtain the multimodal feature data.

7. The method according to claim 1, characterized in that The method further comprises: An initial classification model is trained based on sample data to obtain the classification model, wherein the sample data includes multiple multimodal sample feature data and labels corresponding to each multimodal sample feature data, and the initial classification model is any one of the following models: a support vector machine, a neural network, and a deep learning model.

8. A motor imagery recognition device based on a multimodal brain-computer interface, characterized in that: include: An acquisition unit, used for acquiring multimodal feature data of a user, wherein the multimodal feature data includes at least two of the following data: EEG feature data, ECG feature data, and EMG feature data; A determination unit, configured to determine multimodal fusion feature data based on preset weight information and the multimodal feature data; The input unit is used to input the multimodal fusion feature data into a classification model to obtain the user's movement imagination, where the movement imagination refers to the target body part that the user imagines moving among the user's multiple body parts.

9. A motor imagery recognition system based on a multimodal brain-computer interface, characterized in that: include: A data processing module, used for acquiring multimodal characteristic data of a user, wherein the multimodal characteristic data includes at least two of the following data: EEG characteristic data, ECG characteristic data and EMG characteristic data; Determining multimodal fusion feature data based on preset weight information and the multimodal feature data; The multimodal fusion feature data is input into a classification model to obtain the user's motion imagination, where the motion imagination refers to a target body part that the user imagines moving among multiple body parts of the user.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images