Irregular action recognition method, system and equipment for short-time sequence electroencephalogram and myoelectricity fusion

Abstract

The invention discloses an irregular action recognition method, system and device for short-time sequence electroencephalogram and electromyography fusion, and the method comprises the steps: collecting electroencephalogram and electromyography signals from motor imagery to execution in an incomplete asynchronous manner through a wireless electroencephalogram signal collection device and a distributed wireless surface electromyography signal collection device; processing noise in the electroencephalogram signals and the surface electromyogram signals; in combination with electroencephalogram signals in a motor imagery period, extracting electroencephalogram signal space domain features in a motor execution period, extracting time domain features of surface electromyogram signals, and in combination with a mixed fusion mode of feature layer fusion and decision-making layer fusion, fusing short-time sequence electroencephalogram and electromyogram signals in the motor execution period; according to the method, the motion imagination period recognition result and the motion execution period recognition result are subjected to confidence analysis, irregular actions of the user are quickly and accurately recognized and serve as control instructions to be input into the exoskeleton robot, the exoskeleton robot is controlled to assist the user in completing the target actions, and the initiative of the user is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of rehabilitation medicine and motion intention recognition, and particularly to a method, a system and a device for recognizing irregular motions by fusing short-time series brain and muscle electrophysiological signals. Background Art

[0002] For a large number of people with motor function disorders caused by nervous system diseases after illness, early scientific motor rehabilitation training can effectively help restore limb motor function. However, at present, the proportion of practicing (including assistant) physicians in the national rehabilitation medical industry in the overall physician scale is less than 0.5%. Thus, it is urgent to research intelligent wearable rehabilitation devices such as exoskeleton robots, which can not only assist patients to complete rehabilitation training, improve the rehabilitation effect, but also relieve the pressure on rehabilitation physicians and effectively solve the problem of insufficient rehabilitation physicians. As one of the effective means of treating depression, intelligent wearable rehabilitation devices such as exoskeleton robots can efficiently assist patients in mobilizing the body to regulate hormone levels, strengthen the physiological stress response, give patients a positive emotional impact while enhancing their immunity, and eliminate negative emotions. Summary of the Invention

[0003] The main object of the present invention is to provide a method, a system and a device for recognizing irregular motions by fusing short-time series brain and muscle electrophysiological signals, aiming to solve the problems in the prior art that the exoskeleton robot has insufficient perception of the user's irregular motions and high latency.

[0004] To achieve the above object, the present invention provides the following technical solutions: A method for recognizing irregular motions by fusing short-time series brain and muscle electrophysiological signals, the specific steps are as follows:

[0005] S1. Using a wireless electroencephalogram signal acquisition device and a distributed wireless surface electromyogram signal acquisition device to collect brain and muscle electrophysiological signals during the whole process from motor imagination to execution in an incompletely asynchronous manner, and obtaining electroencephalogram signal data during the 1 s motor imagination period before the initiation of the surface electromyogram signal and muscle signal data during the 200 ms motor execution period after that;

[0006] S2. According to the characteristics of the electroencephalogram signal and its acquisition device and according to the characteristics of the surface electromyogram signal and its acquisition device, respectively process the noise in the electroencephalogram signal and the surface electromyogram signal;

[0007] S3. Combine the EEG signals during the motor imagery period, extract the spatial domain features of the EEG signals during the motor execution period, and extract the time domain features of the surface electromyography (sEMG) signals. Then fuse the two to construct a feature vector for the motor execution period. At the same time, use the method of Filter Band Common Spatial Pattern (FBCSP) to extract the features of the EEG signals during the motor imagery period to construct a feature vector for the EEG signals during the motor imagery period. Input the two feature vectors into the trained classifier respectively, and adopt the Bayesian decision fusion strategy to analyze the confidence levels of the classification results of the two, and output the category with a high confidence level as the final output action instruction;

[0008] Another aspect of the present invention provides a non-regular action online recognition system for short-time series brain-muscle electrical fusion, including:

[0009] An incomplete asynchronous data acquisition module, which is used to collect and receive the brain-muscle electrical signals of the user from the whole process of motor imagery to execution collected by the wireless EEG device and the distributed wireless sEMG device in an incomplete asynchronous manner, preprocess the data and perform non-regular action recognition, and transmit the data to the information processing terminal;

[0010] A data preprocessing module, which preprocesses the data collected by the incomplete asynchronous data acquisition module and divides it into EEG signal data during the motor imagery period, short-time EEG sequences during the motor execution period, and short-time sEMG sequences;

[0011] A lower limb motion intention recognition module for hybrid brain-muscle electrical fusion, which extracts the features of the EEG signals during the motor imagery period to construct a feature vector for the motor imagery period, extracts the features of the short-time EEG sequences during the motor execution period and the features of the short-time sEMG sequences during the motor execution period, and fuses the features of the short-time brain-muscle electrical sequences during the motor execution period to construct a feature vector for the motor execution period. Subsequently, recognize the feature vectors for the motor imagery period and the motor execution period respectively, and use the Bayesian decision fusion strategy to analyze the confidence levels of the recognition results of the two, and output the result with a high confidence level as the control instruction for the exoskeleton robot.

[0012] As a preferred solution of the present invention, the lower limb motion intention recognition module for hybrid brain-muscle electrical fusion includes a feature extraction unit for EEG signals during the motor imagery period, a feature extraction unit for short-time series brain-muscle electrical signals during the motor execution period, a brain-muscle electrical signal feature fusion unit, and a Bayesian decision fusion unit.

[0013] Another aspect of the present invention provides a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the above-mentioned non-regular action recognition method for short-time series brain-muscle electrical fusion can be implemented.

[0014] Another aspect of the present invention provides a computer-readable storage medium that can store computer-executable instructions. When the executable instructions are executed by a control processor, the above-mentioned online recognition method for irregular actions of short-time series brain-muscle electrical signal fusion is implemented.

[0015] Another aspect of the present invention provides a device for an online recognition method of irregular actions, including one or more processors, a memory for storing one or more computer programs, a communication interface for enabling connection and communication between internal components of the electronic device, and a display for displaying information processed in the electronic device terminal. Among them, the above-mentioned one or more processors execute the above-mentioned one or more computer programs to implement the steps of the above-mentioned method.

[0016] Compared with the prior art, the present invention solves the problems in the prior art that the exoskeleton robot has insufficient perception of the user's irregular actions and high latency, and effectively improves the initiative of the user, and has the following beneficial effects:

[0017] 1. In terms of data acquisition: Compared with the current mainstream synchronous data acquisition method, this patent uses an incomplete asynchronous acquisition method to acquire brain-muscle electrical signals during the preparation movement to the movement execution process, which can quickly and effectively obtain the electroencephalogram signals of motor imagery during the movement preparation period, and synchronously acquire the brain-muscle electrical signals during the movement execution period, so as to improve the data processing speed of the online system.

[0018] 2. In terms of feature extraction of short-time series electroencephalogram signals: Compared with the electroencephalogram signal feature extraction methods involved in current research, most of them require long-time series electroencephalogram signals for analysis. This patent combines the electroencephalogram signals of motor imagery during the user's movement preparation period to extract the features of short-time series electroencephalogram signals during the movement execution period, which is beneficial to increasing the decoding stability of short-time series electroencephalogram signals and reducing the action estimation deviation.

[0019] 3. In terms of fusion strategy: Compared with most current research using a single-level fusion strategy, this patent combines feature-level fusion and decision-level fusion, and uses a hybrid fusion strategy to enhance the feature differences of short-time series during the movement execution period between different action categories. At the same time, the recognition results during the motor imagery period and the recognition results during the movement execution period are combined for decision-level fusion to improve the reliability of the final output result.

[0020] 4. In terms of online recognition: Compared with most current research that only considers shortening the model prediction time from the model level to improve the real-time performance of the system. Considering that the human reaction time is about 300 ms, this patent starts from the data acquisition level, collects data for 200 ms during the movement execution period for analysis, greatly reducing the time required for data acquisition. From the model level, this patent uses traditional machine learning methods to build the model, shortening the model prediction duration while ensuring the recognition accuracy. This patent starts from both the data acquisition level and the model level, shortening the data length and reducing the model prediction duration, improving the reliability and real-time performance of the online system.

[0021] 5. The present invention can be widely applied to fields such as stroke rehabilitation and exercise intervention therapy. Compared with the current passive rehabilitation exercise process, the present invention enhances the rehabilitation initiative of patients, can effectively improve the rehabilitation effect, and can even be extended to the assisted movement in daily life, bringing more convenience and happiness to the lives of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flowchart of a preferred embodiment of the lower limb movement intention recognition method of the present invention;

[0023] Figure 2 The specific execution process of the movement intention recognition module in the lower limb movement intention recognition method of the present invention

[0024] Figure 3 is a structural diagram of a preferred embodiment of the lower limb movement intention recognition method of the present invention;

[0025] Figure 4 is a schematic diagram of the operating environment of a preferred embodiment of the device of the present invention; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] Embodiment 1

[0027] Combined with Figure 1 and Figure 4 as shown, the present invention provides a technical solution, a non-regular action recognition method for short-time series brain-muscle electrofusion, and the specific steps are as follows:

[0028] S1. Use a wireless electroencephalogram (EEG) signal acquisition device and a distributed wireless surface electromyogram (EMG) signal acquisition device to collect the whole-process EEG-EMG signals from motor imagery to execution in an incompletely asynchronous manner, and obtain the EEG signal data during the 1-s motor imagery period before the initiation of the surface EMG signal and the EEG-EMG signal data during the 200-ms motor execution period thereafter. To ensure the accuracy of the recognition results for non-regular actions, a multi-channel signal acquisition method is adopted. Among them, according to the 10-20 international standard lead system, EEG electrodes are arranged on the FCz, FC1, FC2, FC3, FC4, T7, T8, Cz, C1, C2, C3, C4, CP1, CP2, CP3, and CP4 channels corresponding to the somatosensory-cognitive activity network to collect the user's EEG signals. For the EMG electrodes, considering the possible hemiplegia of the user, they are arranged on the corresponding muscles of the user's unilateral healthy limb, including the commonly used muscles during movement. Taking the lower limb as an example, such as the rectus femoris, vastus medialis, vastus lateralis, long head of the biceps femoris, medial gastrocnemius, lateral gastrocnemius, lateral soleus, and tibialis anterior, etc., to collect the surface EMG signals generated during the user's limb movement.

[0029] S2. According to the characteristics of the EEG signal and its acquisition device, and the characteristics of the surface EMG signal and its acquisition device, an EEG signal preprocessing unit and a surface EMG signal preprocessing unit are provided to process the noise in the EEG signal and the surface EMG signal respectively.

[0030] S3. A non-regular action recognition module for hybrid EEG-EMG fusion combines the EEG signals during the motor imagery period, extracts the spatial domain features of the EEG signals during the motor execution period, and extracts the time domain features of the surface EMG signals. The two are fused to construct a feature vector for the motor execution period. At the same time, the method of Filter Band Common Spatial Pattern (FBCSP) is used to extract the features of the EEG signals during the motor imagery period to construct a feature vector of the EEG signals during the motor imagery period. The two are respectively input into the trained classifier, and the Bayesian decision fusion strategy is adopted to analyze the confidence levels of the two classification results, and the category with a high confidence level is output as the final output action instruction. Among them, taking the lower limb as an example, the non-regular action instruction set includes: walking, crossing an obstacle, going up the steps, going down the steps, sitting down, and standing up, etc.

[0031] The present invention combines a hybrid fusion method of feature-level fusion and decision-level fusion, fuses the short-time sequence EEG-EMG signals during the motor execution period, and performs confidence analysis on the recognition results during the motor imagery period and the motor execution period, so as to quickly and accurately recognize the user's non-regular actions and input them as control instructions into the exoskeleton robot to control the exoskeleton robot to assist the user to complete the target actions, effectively improving the user's initiative, and having good application prospects in the assisted treatment of patients with depression, stroke, etc.

[0032] In this embodiment, an incomplete asynchronous method is adopted to collect the whole-process electroencephalogram and electromyogram signals from motor imagination to execution. It is necessary to judge the starting period of the surface electromyogram signal. The present invention combines the Teager-Kaiser energy operator and the double-threshold determination method to detect it. The detection method and steps are described as follows:

[0033] S11, intercept the surface electromyogram signal with a time window length of 200 ms, and calculate the amplitude average value of all channels. The formula is as follows:

[0034]

[0035] where x c [n] represents the surface electromyogram signal data of the c-th channel with a time window length of 200 ms, and C represents the number of channels.

[0036] S12, calculate the Teager-Kaiser energy operator of to analyze the instantaneous energy change of the signal, extract the instantaneous amplitude information, and at the same time, perform band-pass filtering on it in the range of 20 - 150 Hz to remove unnecessary frequency components such as noise. The formula is as follows:

[0037]

[0038] Ψ filter [n] = F(Ψ[n])

[0039] where Ψ[n] represents the Teager-Kaiser energy operator, N represents the number of samples within a 200-ms time window, and Ψ filter [n] represents the filtered Teager-Kaiser energy operator, and F(·) represents the filtering function of a 20 - 150 Hz band-pass filter.

[0040] S13, use the Hilbert transform to calculate the signal envelope Ψ filter [n] of Ψ helbert [n], and set the amplitude threshold Th1 and the length threshold Th2, and use the double-threshold determination method to judge whether this segment belongs to the action starting period. Among them, the amplitude threshold is obtained during system calibration. The surface electromyogram signal of the user in the static state is collected, and its signal envelope is obtained through the above steps. The peak value of the signal envelope is detected and the sum of its mean value and 20 times the variance is calculated, denoted as Th1. And the selected length threshold Th2 is the data volume contained in a time length of 150 ms. When the data length of the signal envelope Ψ helbert [n] greater than Th1 is greater than Th2, it is determined that this 200-ms window is the starting period of the surface electromyogram signal.

[0041] After obtaining the exercise initiation period, S14 synchronously records the electroencephalogram (EEG) signals and surface electromyogram (sEMG) signals within a 200 ms window length of this period, and obtains the EEG signals of the previous 1 s from a readable storage medium, thus completing the data acquisition.

[0042] Compared with the current mainstream synchronous data acquisition method, the present invention uses an incompletely asynchronous acquisition method to collect EEG and sEMG signals during the preparation movement to the movement execution process, which can quickly and effectively obtain the EEG signals of motor imagery during the movement preparation period, and synchronously collect the EEG and sEMG signals during the movement execution period, so as to improve the data processing speed of the online system.

[0043] In this embodiment, the noise in the EEG signals and surface electromyogram signals is processed as follows:

[0044] The signal processing steps of the EEG signal preprocessing unit for the set EEG signal preprocessing unit include removing baseline drift, removing 50 Hz power frequency interference, 1 - 50 Hz band-pass filtering, and Z-Score normalization;

[0045] The signal processing steps of the surface electromyogram signal preprocessing unit for the set surface electromyogram signal preprocessing unit include removing baseline drift, removing 50 Hz power frequency interference, 20 - 200 Hz band-pass filtering, downsampling, and Z-Score normalization.

[0046] In this embodiment, the S3 step is divided into two stages: model training and real-time recognition.

[0047] Among them, the overall process of model training is described in detail as follows:

[0048] S31, collect the EEG signal data and surface electromyogram signal data of the subject in an incompletely asynchronous acquisition method, form a data set, and perform data processing on it through a data preprocessing module to improve the data quality, and divide it into a training set and a validation set for model training;

[0049] S32, subsequently, extract the features of the EEG signals during the motor imagery period through the EEG signal feature extraction unit during the motor imagery period, and use the method of Filter Band Common Spatial Pattern (FBCSP) to construct the feature matrix MI_Feature of the EEG signals during the motor imagery period;

[0050] S33. Next, the short-time sequence brain-muscle EMG signal feature extraction unit in the movement execution period uses a regularization method to extract the spatial domain features of the EEG signal in the first 200 ms before movement execution in combination with 1 s of EEG signal data during the motor imagery period. Considering that there are multiple action categories, the one-versus-many method is adopted to extract the spatial domain features of the short-time sequence brain-muscle EMG signal in the movement execution period, that is, the common spatial pattern (CSP) features, to form the feature matrix ME_CSP;

[0051] S34. At the same time, the short-time sequence brain-muscle EMG signal feature extraction unit in the movement execution period completes the feature extraction of the preprocessed surface EMG signal, mainly extracting the time domain features of the signal, including the absolute average value, root mean square value, waveform length, number of zero crossings, etc., to form the feature matrix ME_sEMG;

[0052] S35. Then, the brain-muscle EMG signal feature fusion unit fuses the features extracted from the EEG signal and the surface EMG signal in the movement execution period to form the feature matrix ME_Feature in the movement execution period:

[0053] ME_Feature = [ME_CSP ME_sEMG]

[0054] S36. The Bayesian decision fusion unit constructs two classifiers Model_MI and Model_ME based on traditional machine learning methods (such as random forest classifiers), inputs the MI_Feature and ME_Feature extracted from the training set into them for training respectively, and verifies the performance of the classifiers through the validation set, calculates the confusion matrices of the two, and measures the error distribution of each network for each class:

[0055]

[0056] Among them, represents the number of samples of the i-th class predicted as the j-th class by the k-th classifier, M represents the number of action categories, and K represents the number of classifiers.

[0057] According to the above confusion matrix, calculate the conditional probability of each classifier This conditional probability represents the probability that the output of the k-th classifier is j when the true class is i, where, represents the event that the true output class of the k-th classifier is i, e k represents the predicted output class of the classifier, represents the event that the predicted output class of the k-th classifier is j.

[0058] The projection matrix in the short-time sequence brain-muscle EMG signal feature extraction unit in the movement execution period The conditional probability matrices P of the two classifiers MI and P MEAnd the trained classifiers Model_MI and Model_ME are recorded in a computer-readable storage medium to complete the model training phase.

[0059] The overall process of the real-time recognition phase is described as follows:

[0060] S37, Read the projection matrix from the computer-readable storage medium The conditional probability matrix P MI and P ME and the trained classifiers Model_MI and Model_ME.

[0061] S38, According to the projection matrix Calculate the spatial domain features of the EEG signals during the movement execution period and construct a feature vector. At the same time, calculate the time domain features of the surface EMG signals during the movement execution period and construct a feature vector.

[0062] S39, The brain-muscle EEG signal fusion unit fuses the above-mentioned spatial domain features of the EEG signals and the time domain features of the surface EMG signals to construct a feature vector ME_Feature during the movement execution period. At the same time, the FBCSP method is used to extract the features of the EEG signals during the movement imagination period and construct a feature vector MI_Feature during the movement imagination period. And input the two into the classifiers Model_MI and Model_ME respectively for prediction to obtain the prediction results c MI and c ME .

[0063] Subsequently, according to the conditional probability matrix P MI and P ME Calculate the Bayesian probability:

[0064]

[0065] Among them,

[0066] S310, Select the action category c i corresponding to the maximum probability value of Bel(c i ) as the output instruction and transmit it to the exoskeleton robot, and then control the exoskeleton robot to assist the user to complete the action.

[0067] Compared with the EEG signal feature extraction methods involved in the current research, most of them require long-time series EEG signals for analysis. The present invention combines the movement imagination EEG signals during the user's movement preparation period to extract the features of short-time series EEG signals during the movement execution period, which is beneficial to increasing the decoding stability of short-time series EEG signals and reducing the action estimation deviation.

[0068] In this embodiment, the calculation method of step S33 is as follows:

[0069] S33.1. Obtain the EEG signals at different times after being processed through step S31, and calculate the sum of the spatial covariance matrices for each type of action during the motor imagery period and the motor execution period respectively. The formula is as follows:

[0070]

[0071] where, SMI c represents the sum of the spatial covariance matrices during the motor imagery period for the c-th type of action, and SME c represents the sum of the spatial covariance matrices during the motor execution period for the c-th type of action. Classnum represents the total number of action categories, M represents the total number of trials for the c-th type of action, and E (c,m) represents the EEG signal of the m-th trial for the c-th type of action, whose dimension is N×T, where N is the number of channels and T is the number of samples.

[0072] S33.2. According to the sum of the spatial covariances SMI c and SME c for each type of action obtained in step S32.1, calculate the regularized average spatial covariance matrix for each type of action. The formula is as follows:

[0073]

[0074] where, β and γ are two regularization parameters, and 0 ≤ β, γ ≤ 1. Ι is the N×N identity matrix, and the definition formula is as follows:

[0075]

[0076] S33.3. According to the regularized average spatial covariance matrix for the c-th type of action and the regularized average spatial covariance matrix for other types of actions calculated respectively in step S32.2, calculate the composite spatial covariance matrix and decompose it into the following form:

[0077]

[0078] where, represents the eigenvector matrix, represents the diagonal matrix of the corresponding eigenvalues.

[0079] S33.4. According to the eigenvector matrix and the diagonal matrix of the corresponding eigenvalues obtained in step S32.3, calculate the whitening matrix and decompose the regularized average spatial covariance matrix after whitening for the c-th type of action. The formula is as follows:

[0080]

[0081] The eigenvector matrix obtained from the singular value decomposition of the regularized average spatial covariance matrix of the c-th type of action after whitening according to step S32.4 and the whitening matrix Calculate the full projection matrix The formula is as follows:

[0082]

[0083] For the most discriminative patterns, only retain the first α columns and the last α columns of to form the projection matrix

[0084] S33.6. According to the projection matrix obtained in step S32.5 Extract features from the EEG signals of each trial in the dataset, and its formula is expressed as follows:

[0085]

[0086] where represents the projection of the EEG signal of the m-th trial. Here, E (c,m) is the EEG signal during the movement execution period, represents the q-th column in m and ME_CSP

[0087] Embodiment 2

[0088] Combined with Figure 2 as shown, an online recognition system for non-regular actions of short-time series brain-muscle electrofusion in this embodiment is used for rapid recognition of non-regular actions and includes:

[0089] Incomplete asynchronous data acquisition module, configured to: collect electroencephalogram (EEG) and electromyogram (EMG) signals during the whole process from motor imagery to execution by using a wireless EEG signal acquisition device and a distributed wireless surface EMG signal acquisition device, and obtain 1 s of EEG signal data during the motor imagery period and synchronously record 200 ms of EEG and EMG signal data during the motor execution period through the S1 incomplete asynchronous acquisition method; wherein, the wireless EEG signal acquisition device includes but is not limited to NeuSen.W64 produced by Borui Kang Co., Ltd., and the distributed wireless surface EMG signal acquisition device includes but is not limited to Picolite produced by Cometa Co., Ltd. To ensure the accuracy of the recognition results of non-regular movements, a multi-channel signal acquisition method is adopted. Among them, according to the 10-20 international standard lead system, EEG electrodes are arranged on the FCz, FC1, FC2, FC3, FC4, T7, T8, Cz, C1, C2, C3, C4, CP1, CP2, CP3, and CP4 channels corresponding to the somatosensory-cognitive activity network for collecting the user's EEG signals. For EMG electrodes, considering that the user may have hemiplegia, they are arranged on the corresponding muscles of the user's unilateral healthy limb, including the muscles commonly used during movement. Taking the lower limb as an example, such as the rectus femoris, vastus medialis, vastus lateralis, long head of the biceps femoris, medial gastrocnemius, lateral gastrocnemius, lateral soleus, and tibialis anterior, etc., for collecting the surface EMG signals generated when the user's limb moves.

[0090] Data preprocessing module, configured to: preprocess the data collected by the incomplete asynchronous data acquisition module and segment it into EEG signal data during the motor imagery period, short-term EEG sequences during the motor execution period, and short-term EMG sequences.

[0091] Lower limb motor intention recognition module for hybrid EEG and EMG fusion, configured to: extract the features of EEG signals during the motor imagery period, construct a feature vector during the motor imagery period, extract the features of short-term EEG sequences and short-term EMG sequences during the motor execution period, fuse the features of short-term EEG and EMG sequences during the motor execution period to construct a feature vector during the motor execution period. Subsequently, respectively identify the feature vector during the motor imagery period and the feature vector during the motor execution period, analyze the confidence levels of the two recognition results by using the Bayesian decision fusion strategy, and output the result with a high confidence level as the control instruction for the exoskeleton robot.

[0092] Furthermore, the lower limb movement intention recognition module with hybrid brain-muscle electroencephalogram (EEG) fusion includes an electroencephalogram signal feature extraction unit during the motor imagery period, a short-time sequence brain-muscle EEG signal feature extraction unit during the motor execution period, a brain-muscle EEG signal feature fusion unit, and a Bayesian decision fusion unit; it extracts the electroencephalogram signal features during the motor imagery period, combines the electroencephalogram signal during the motor imagery period to extract the short-time sequence spatial domain features of the electroencephalogram signal during the motor execution period, extracts the time domain features of the surface electromyogram signal, fuses the features of the electroencephalogram signal and the surface electromyogram signal during the motor execution period, and performs Bayesian decision fusion on the classification results of the electroencephalogram signal during the motor imagery period and the classification results of the brain-muscle EEG signal during the motor execution period, and finally outputs non-regular action instructions. Among them, taking the lower limb as an example, the non-regular action instruction set includes: walking, crossing obstacles, going up stairs, going down stairs, sitting down, and standing up, etc.

[0093] Embodiment 3

[0094] Combined with Figure 3 and Figure 4 As shown, this embodiment provides a computer-readable storage medium with a program stored thereon, and when the program is executed by a processor, it implements all steps in the above non-regular action recognition method with short-time sequence brain-muscle EEG fusion.

[0095] The electronic device may include a processor, a memory, a communication interface, a display, and a bus, and may also include a computer program stored in the memory and executable on the processor, including a data acquisition program, a data preprocessing program, and a non-regular action recognition program.

[0096] The memory includes but is not limited to a readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card-type memory (such as an SD memory, etc.). In some embodiments, it may also be an internal storage unit of the electronic device, such as the mobile hard disk of the electronic device. In other embodiments, it may also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart memory card, a flash memory card, etc. equipped on the electronic device. The memory may also include both the internal storage unit of the electronic device and the external storage device. It can not only be used to store the application software installed on the electronic device and various types of data, such as the preprocessing program and the movement intention recognition program, but also be used to temporarily store the data that has been output or will be output.

[0097] In some embodiments, the processor may be composed of integrated circuits. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor is the core of the electronic device, connecting various components of the entire electronic device through various interfaces and circuits. By running or executing programs or modules stored in the memory (such as programs for preprocessing and motion intention recognition), and by calling data stored in the memory, it performs various functions of the electronic device and processes data.

[0098] The communication interface includes a wired communication interface and a wireless communication interface (such as a WI-FI interface, a Bluetooth interface, etc.). It is usually used to establish a communication connection between this electronic device and other electronic devices (such as an exoskeleton robot), and to achieve connection communication between internal components of the electronic device.

[0099] The display can be an LED display screen, a liquid crystal display screen, a touch liquid crystal display screen, an OLED touch screen, etc. It is used to display the information processed in the electronic device (such as the motion intention recognition result) and to display a visual user interface.

[0100] The bus can be a peripheral component interconnect standard bus or an extended industry standard architecture bus. It is divided into an address bus, a data bus, a control bus, etc. It is set to achieve connection communication between the memory and at least one processor, etc.

[0101] Figure 3 Only the electronic device with components is shown. Those skilled in the art can understand that the shown structure does not constitute a limitation on the electronic device. It may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0102] The memory in the electronic device stores the preprocessing program and the motion intention recognition program, which is a combination of multiple instructions. When the processor runs, it can achieve:

[0103] Perform the whole process of motor imagery to executed whole process electroencephalogram and electromyogram signals in a not completely asynchronous manner, and complete the filtering process of the collected electroencephalogram signals and surface electromyogram signals through the preprocessing program to obtain the filtered electroencephalogram and electromyogram signals;

[0104] Combine the filtered electroencephalogram signals during the motor imagery period to extract features from the filtered short-time sequence electroencephalogram signals during the motor execution period to obtain the spatial domain features of the short-time sequence electroencephalogram signals;

[0105] Extract features from the filtered surface electromyogram signals to obtain the time domain features of the surface electromyogram signals;

[0106] Extract the features of the electroencephalogram (EEG) signals during the motor imagery period to obtain the feature set during the motor imagery period and perform action recognition.

[0107] Construct a non-regular action recognition model for short-time sequence electroencephalogram and electromyogram fusion, fuse the spatial domain features of EEG signals and the time domain features of surface electromyogram signals, construct the feature set during the motor execution period and perform action recognition, and use the Bayesian decision fusion method to fuse the recognition results during the motor imagery period and the motor execution period for confidence analysis. Finally, quickly and accurately obtain the category of non-regular actions that the user is about to perform in a hybrid fusion manner of feature-level fusion and decision-level fusion.

[0108] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for recognizing irregular actions by fusing short-time sequence brain and muscle electroencephalogram, characterized in that , including the following steps: S1, using wireless EEG signal acquisition equipment and distributed wireless surface EMG signal acquisition equipment to collect EEG signals from the entire process of motor imagination to execution in a non-completely asynchronous manner, and obtaining EEG signal data of the motor imagination period 1s before the initiation of surface EMG signals and EEG signal data of the motor imagination period 200ms after the initiation of surface EMG signals; S2, according to the characteristics of the EEG signal and its acquisition equipment, and the characteristics of the surface electromyography signal and its acquisition equipment, an EEG signal preprocessing unit and a surface electromyography signal preprocessing unit are provided to process the noise in the EEG signal and the surface electromyography signal respectively; S3, combined with the EEG signals during the motor imagery period, the spatial domain features of the EEG signals during the motor execution period are extracted, and the time domain features of the surface electromyography signals are extracted, and the two are fused to construct the feature vector of the motor execution period. At the same time, the filter group co-spatial pattern method is used to extract the EEG signal features during the motor imagery period to construct the feature vector of the EEG signal during the motor imagery period. The two are input into the trained classifier respectively, and the Bayesian decision fusion strategy is used to analyze the confidence of the classification results of the two, and the action category with high confidence is output as the final output action instruction.

2. The non-regular motion recognition method for short-time sequence brain-muscle electroencephalogram fusion according to claim 1, wherein The EEG signal collection and surface electromyography signal collection adopt a wireless multi-channel signal collection method.

3. The non-regular action recognition method for short-time sequence electroencephalogram and electromyogram fusion according to claim 1, wherein: The surface electromyography signal initiation period is detected by combining the Teager-Kaiser energy operator and the double threshold judgment method. The detection method and steps are described as follows: S11, intercept the surface electromyographic signal with a time window length of 200ms, and calculate the amplitude average of all channels. The formula is as follows: where x c [n] represents the surface electromyogram signal data of the c-th channel with a time window length of 200 ms, and C represents the number of channels; S12, calculates the Teager-Kaiser energy operator of X[n], which is used to analyze the instantaneous energy change of the signal and extract the instantaneous amplitude information. At the same time, it is subjected to 20-150Hz bandpass filtering to remove unnecessary frequency components such as noise. The formula is as follows: Ψ filter [n] = F(Ψ[n]) Among them, Ψ[n] represents the Teager-Kaiser energy operator, N represents the number of samples within a 200 ms time window, and Ψ filter [n] represents the filtered Teager-Kaiser energy operator, and F(·) represents the filtering function of a 20 - 150 Hz band-pass filter; S13. Calculate Ψ using the Hilbert transform filter The signal envelope Ψ helbert [n], and set the amplitude threshold Th1 and the length threshold Th2. Use the double-threshold decision method to determine whether this segment belongs to the action initiation period; among them, the amplitude threshold is obtained during system calibration. The surface electromyogram signal of the user in the static state is collected, and its signal envelope is obtained through the above steps. The peak value of the signal envelope is detected and the sum of its mean value and 20 times the variance is calculated, denoted as Th1, and the selected length threshold Th2 is the data volume included in a 150 ms time length; when the data length of the signal envelope Ψ helbert [n] greater than Th1 is greater than Th2, then it is determined that this 200 ms window is the surface electromyogram signal initiation period; After obtaining the movement initiation period, S14 synchronously records the EEG signal and surface electromyography signal of the 200ms window length in the period, and obtains the EEG signal of the previous 1s from the readable storage medium, thus completing the data collection.

4. The non-regular motion recognition method for short-time sequence brain-computer electromyography fusion according to claim 1, characterized in that: The EEG signal preprocessing unit sets the signal processing steps of the EEG signal preprocessing unit, including removing baseline drift, removing 50Hz power frequency interference, 1-50Hz bandpass filtering and Z-Score standardization; The surface electromyography signal preprocessing unit sets the signal processing steps of the surface electromyography signal preprocessing unit to include removing baseline drift, removing 50Hz power frequency interference, 20-200Hz bandpass filtering, downsampling and Z-Score standardization.

5. The non-regular motion recognition method for short-time sequence electroencephalogram and electromyogram fusion according to claim 1, characterized in that: Step S3 is divided into two stages: model training and real-time recognition. The specific steps are as follows. The overall process of model training is described in detail as follows: S31, collecting the EEG signal data and surface electromyography signal data of the subject in an incompletely asynchronous collection mode to form a data set, and processing the data through a data preprocessing module to improve the data quality, and dividing the data into a training set and a validation set for model training; S32. Subsequently, the feature extraction unit for EEG signals during the motor imagery period extracts the features of the EEG signals during the motor imagery period. Using the filter bank common spatial pattern method, a feature matrix MI_Feature of the EEG signals during the motor imagery period is constructed. S33. Next, the feature extraction unit for short-term sequential EEG-EMG signals during the motor execution period uses a regularization method to extract the spatial domain features of the EEG signals in the first 200 ms before the motor execution period by combining the EEG signal data in the 1 s during the motor imagery period. Using a one-to-many method, the spatial domain features of the short-term sequential EEG-EMG signals during the motor execution period, that is, the common spatial pattern (CSP) features, are extracted to form an EEG signal feature matrix ME_CSP. S34. Meanwhile, the feature extraction unit for short-term sequential EEG-EMG signals during the motor execution period completes the feature extraction of the preprocessed surface electromyogram (sEMG) signals, mainly extracting the time domain features of the signals, including the absolute average value, root mean square value, waveform length, number of zero crossings, etc., to form a surface electromyogram signal feature matrix ME_sEMG. S35. Then, the EEG-EMG signal feature fusion unit fuses the features extracted from the EEG signals and surface electromyogram signals during the motor execution period to form a feature matrix ME_Feature: ME_Feature = [ME_CSP ME_sEMG] S36. The Bayesian decision fusion unit constructs two classifiers Model_MI and Model_ME based on traditional machine learning methods, inputs the MI_Feature and ME_Feature extracted from the training set into them for training respectively, verifies the performance of the classifiers through the validation set, calculates their confusion matrices, and measures the error distribution of each network for each class. Among them, represents the number of samples of the \(i\)-th class predicted as the \(j\)-th class by the \(k\)-th classifier, \(M\) represents the number of action categories, and \(K\) represents the number of classifiers; Calculate the conditional probability of each classifier based on the above confusion matrix This conditional probability represents the probability that the output of the k-th classifier is j when the true class is i, where represents the event that the true output class of the k-th classifier is i, e k represents the output class predicted by the classifier represents the event that the predicted output class of the k-th classifier is j; The projection matrix in the short-time series brain-computer myoelectric signal feature extraction unit during movement execution The conditional probability matrices P of the two classifiers MI and P ME as well as the already trained classifiers Model_MI and Model_ME are recorded in a computer-readable storage medium to complete the model training stage; The overall process of the real-time recognition stage is described as follows: S37, Read the projection matrix from the computer-readable storage medium Conditional probability matrix P MI and P ME and the trained classifiers Model_MI and Model_ME; S38, according to the projection matrix Calculate the spatial domain features of the EEG signals during the movement execution period, and construct the feature vector ME_CSP; at the same time, calculate the time domain features of the sEMG signals during the movement execution period, and construct the feature vector ME_sEMG; S39, the brain-muscle electrical signal fusion unit fuses the above-mentioned spatial features of electroencephalogram signals and temporal features of surface electromyogram signals to construct a feature vector ME_Feature during the movement execution period. At the same time, it extracts the features of electroencephalogram signals during the movement imagination period in the FBCSP manner to construct a feature vector MI_Feature; and inputs the two into the classifiers Model_MI and Model_ME respectively for prediction to obtain the prediction results c MI and c ME ; Subsequently, according to the conditional probability matrices P MI and P ME calculate the Bayesian probability: Among them, S310, select the action category c corresponding to the largest probability value Bel(c i ) as the output instruction and transmit it to the exoskeleton robot, thereby controlling the exoskeleton robot to assist the user to complete the action. i ​ 6. The non-regular motion recognition method for short-time sequence electroencephalogram and electromyogram fusion according to claim 5, wherein The calculation method described in step S33 is as follows: S33.

1. Obtain the EEG signals in different periods processed by step S31, and calculate the sum of the spatial covariance matrices of each type of action during the motor imagery period and the motor execution period respectively. The formula is as follows: Among them, SMI c represents the sum of the spatial covariance matrices during the motor imagery period of the c-th type of movement, SME c the sum of the spatial covariance matrices during the motor execution period of the c-th type of movement, Classnum represents the total number of movement categories, M represents the total number of trials of the c-th type of movement, E (c,m) represents the electroencephalogram signal of the m-th trial of the c-th type of movement, whose dimension is N×T, N is the number of channels, and T is the number of samples; S33.

2. Obtain the sum SMI of the spatial covariance at different periods of various actions obtained in step S33.1 c and SME c , and calculate the regularized average spatial covariance matrix of each type of action. The formula is as follows: where β and γ are two regularization parameters, and 0 ≤ β, γ ≤ 1, I is an N×N identity matrix, and The definition is as follows: S33.

3. Calculate the regularized average spatial covariance matrix of the c-th type of action respectively according to step S33.2 and the regularized average spatial covariance matrix of other types of actions Calculate the composite spatial covariance matrix and decompose it into the following form: Among them, represents the eigenvector matrix, represents the diagonal matrix of the corresponding eigenvalues; S33.

4. The eigenvector matrix obtained according to step S32.3 and the diagonal matrix of the corresponding eigenvalues Calculate the whitening matrix and decompose the regularized average spatial covariance matrix after whitening for the c-th type of action. The formula is as follows: The eigenvector matrix obtained from the singular value decomposition of the regularized average spatial covariance matrix of the c-th type of action after whitening according to step S33.4 and the whitening matrix Calculate the full projection matrix The formula is as follows: For the most discriminative patterns, only the first α columns and the last α columns of are retained to form a projection matrix with size N×Q, where Q = 2α; S33.

6. The projection matrix obtained according to step S33.5 Feature extraction is performed on the EEG signals of each trial in the dataset, and its formula is expressed as follows: Among them, represents the projection of the electroencephalogram signal of the m-th trial, where E (c,m) is the electroencephalogram signal during the movement execution period, represents the q-th column in m ME_CSP represents the CSP eigenvector of the m-th trial.

7. An online recognition system for irregular actions by fusing short-time series of brain and myoelectric signals, characterized in that, Including: An incomplete asynchronous data acquisition module, which is used to collect and receive the EEG-EMG signals of the user from the whole process of motor imagery to execution collected by the wireless EEG device and the distributed wireless EMG device in an incomplete asynchronous manner, preprocess the data and identify non-regular actions, and transmit the data to the information processing terminal. A data preprocessing module, which preprocesses the data collected by the incomplete asynchronous data acquisition module and divides it into EEG signal data during the motor imagery period, short-term EEG sequences and short-term EMG sequences during the motor execution period. A lower limb movement intention recognition module for hybrid EEG-EMG fusion extracts the features of the EEG signals during the motor imagery period, constructs a feature vector during the motor imagery period, extracts the features of the short-term EEG sequences and short-term EMG sequences during the motor execution period, fuses the features of the short-term EEG-EMG sequences during the motor execution period to construct a feature vector during the motor execution period. Subsequently, the feature vectors during the motor imagery period and the motor execution period are identified respectively, and the Bayesian decision fusion strategy is used to analyze the confidence levels of the recognition results of the two, and the result with a high confidence level is output as the control instruction for the exoskeleton robot.

8. An on-line recognition system for irregular actions by short-time sequence brain-muscle electroencephalogram fusion according to claim 6, characterized in that: The lower limb movement intention recognition module based on hybrid electroencephalogram and electromyogram fusion includes an electroencephalogram signal feature extraction unit during the motor imagery period, a short-time sequence electroencephalogram and electromyogram signal feature extraction unit during the motor execution period, an electroencephalogram and electromyogram signal feature fusion unit, and a Bayesian decision fusion unit.

9. A computer-readable storage medium that can store computer-executable instructions, characterized in that: When the executable instruction is executed by the control processor, it realizes the online recognition method for non-regular actions based on short-time sequence electroencephalogram and electromyogram fusion as claimed in any one of claims 1 to 6.

10. An apparatus for an online recognition method of non-regular actions, characterized in that, It includes at least one data acquisition program, at least one data preprocessing program, and at least one non-regular action recognition program, and also includes a memory for storing the at least one data acquisition program, the at least one data preprocessing program, and the at least one non-regular action recognition program; the memory stores instructions executable by the at least one data acquisition program, instructions executable by the at least one data preprocessing program, and instructions executable by the at least one non-regular action recognition program; When at least one of the instructions executed by the data acquisition program, the instructions executed by the data preprocessing program, and the instructions executed by the non-regular action recognition program is executed by the processor, the at least one control processor can execute the online recognition method for non-regular actions as claimed in any one of claims 1 to 6.

Images