Motor imagery brain-computer interface control system and method based on feature fusion and pose feedback

Abstract

The invention discloses a motor imagery brain-computer interface control system and method based on feature fusion and pose feedback. The motor imagery brain-computer interface control system comprises a controller system and a motion execution mechanism. The controller system comprises a master controller and a slave controller; the main controller comprises an electroencephalogram signal acquisition module and an electroencephalogram signal processing module; the electroencephalogram signal acquisition module is used for acquiring multi-modal electroencephalogram signals; the electroencephalogram signal processing module is used for feature matrix fusion, space diagonalization feature extraction and pattern classification; the slave controller comprises a motion control module which is used for comprehensively generating a motion instruction according to the motion intention and spatial pose correction and driving a motion execution mechanism; and the motion execution mechanism is used for specifically realizing motion imagination electroencephalogram signal actions. According to the invention, the accuracy and response time of motor imagery mode classification are improved. And the practicability of motion integration control with an external execution mechanism is enhanced. And the accuracy, the real-time performance and the flexibility of the motion of the robot in a complex space are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of brain science and information, and in particular to a motor imagery brain-computer interface control system and method based on feature fusion and posture feedback. Background Art

[0002] The potential changes of the cerebral cortex neuron cluster reflect the thinking intention. The main research method of brain-computer interface technology is to convert this potential change into relevant control instructions and transmit it to the external control unit to achieve interaction with the external environment. The current mainstream brain-computer interface application is the brain-computer interface technology based on the rhythm of motor imagery. It has strong real-time performance and can flexibly adapt to the scene. However, in the classification and processing of rhythm-based motor imagery EEG signal features, there are problems such as unclear feature extraction, inaccurate classification, and inability to accurately capture the movement intention, and the spatial posture information of the movement mechanism is not effectively integrated. Summary of the invention

[0003] The purpose of the present invention is to provide a motor imagery brain-computer interface control system and method based on feature fusion and posture feedback, so as to solve the aforementioned problems existing in the prior art.

[0004] The present invention proposes a feature fusion algorithm of local spectral normalization (LSN) and online regression-Empirical Mode Reconstruction (OR-EMR), which effectively solves the problem that the recognition accuracy of EEG signals is not high and the stability is difficult to meet the requirements due to the limitations of single feature extraction and feature classification algorithms. The feature fusion algorithm effectively improves the control ability of the brain-computer interface, enhances the robustness and adaptability of practical application scenarios, and realizes the integration of motor imagination, thinking, intention control and movement. In addition, on the basis of realizing EEG signal processing, the calculation process of the motion state of the actuator and the current spatial posture feedback is integrated, the motion trend and spatial positioning are comprehensively evaluated and predicted, the motion amplitude and range of the actuator are effectively controlled, the control stability in the entire control process is maintained, and the coherence and coordination of the actuator motion control are realized.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A motor imagery brain-computer interface control system based on feature fusion and posture feedback, comprising:

[0007] Controller systems and motion actuators;

[0008] The controller system includes: a master controller and a slave controller; the master controller includes an EEG signal acquisition module and an EEG signal processing module; the EEG signal acquisition module is used to acquire multimodal EEG signals; the EEG signal processing module is used for feature matrix fusion, spatial diagonal feature extraction and pattern classification;

[0009] The slave controller includes: a motion control module for comprehensively generating motion instructions and driving motion actuators according to motion intention and spatial posture correction;

[0010] The motion execution mechanism is used to realize the specific actions of motor imagery EEG signals.

[0011] In some specific embodiments, it also includes:

[0012] RS485 serial communication interface is used to connect the controller system and the slave controller and realize reliable data transmission.

[0013] In some specific embodiments, the EEG signal processing module includes: preprocessing, feature matrix fusion and feature extraction classification;

[0014] The EEG signal processing module is mainly used to complete the preprocessing of the collected EEG signals, then extract the signal features that reflect the subject's intentions, and finally classify the extracted features and convert them into control commands.

[0015] In some specific embodiments, the preprocessing is used to filter, denoise, segment, baseline correct, remove FastICA artifacts, and perform transient time-frequency joint feature matrix fusion, spatial diagonalization feature extraction, and pattern recognition classification of the raw EEG data after re-reference preprocessing;

[0016] Feature matrix fusion is used to fuse the feature matrices extracted by the local spectrum normalization algorithm and the online regression modal reconstruction algorithm in the joint feature extraction process in a serial manner to construct a new high-quality feature fusion vector;

[0017] Feature extraction classification is used to train and classify the support vector machine for hyperplane classification based on the feature vectors corresponding to the feature values ​​after the new fusion feature matrix space is diagonalized.

[0018] In some specific embodiments, the motion control module includes:

[0019] The spatial posture estimation module is used to integrate the calculation process of motion trend and initial working posture, estimate the spatial posture, solve its current spatial posture to prevent excessive movement of the motion actuator after reaching the limit position, and realize the coordinated control of the drive unit;

[0020] The spatial posture positioning module is used to integrate the motion state of the actuator and the current spatial posture based on the classification results of the motion imagination pattern and the strength of the motion imagination intention signal, calculate the correction amount of the current spatial posture to adjust to the target posture, comprehensively evaluate and predict the motion trend, and generate real-time micro-torque adaptive coupling motion control instructions;

[0021] Servo drive and mechanical motion control module,Servo drive and mechanical motion control module are used to achieve the consistency and coordination of motion actuator control.

[0022] In some specific embodiments, the preprocessing specifically includes: performing quantitative calculation of event-related desynchronization (ERD) / event-related synchronization (ERS) phenomenon according to the judgment and identification criteria of the electrooculographic independent components and using formula (1);

[0023]

[0024] In formula (1), R and A are the power spectrum values ​​of the specific frequency band of the EEG signal before and after the motor imagery task, respectively. ERD / ERS <0, it means that the power spectrum value of a certain frequency band decreases after the motor imagery task, proving that the ERD phenomenon occurs; on the contrary, when C ERD / ERS >0, it means that ERS phenomenon occurs.

[0025] In some specific embodiments, feature matrix fusion is specifically as follows: the EEG signal after filtering, denoising, segmentation, baseline correction, FastICA artifact removal, and re-reference preprocessing is subjected to local spectral normalization (LSN) and online regression-Empirical Mode Reconstruction (OR-EMR), and then connected in series end to end using linear weighting to construct a feature connection matrix that integrates local spectral phase characteristics and time domain amplitude space diagonalization characteristics, thereby obtaining the final feature matrix for feature extraction and pattern recognition classification.

[0026] A motor imagery brain-computer interface control method based on feature fusion and posture feedback based on the same conception includes the following steps:

[0027] S100, the user wears a high-precision multi-modal signal acquisition head cap, and performs mental thinking and imagination according to the movement intention. The EEG signal acquisition module obtains the spontaneous EEG related potential signal of the occipital lobe area of ​​the brain;

[0028] S200, after the induced signal is picked up by the acquisition head cap, it is amplified by the amplifier and connected to the main controller through the RS485 serial communication interface. The EEG signal processing module performs preprocessing, independent component extraction, feature fusion reconstruction, feature extraction, and feature pattern recognition on the original EEG data;

[0029] S300, the main controller converts the motion intention into an instruction according to the processed classification result and transmits it to the slave controller, and after receiving the motion intention instruction, the slave controller integrates the motion state and current spatial posture of the motion actuator to generate a real-time micro-torque adaptive coupling motion control instruction;

[0030] S400, the slave controller converts the generated micro-torque adaptive coupling motion control instruction into the code of the servo drive encoder, and inputs the code into the servo drive controller through the RS485 communication bus, and drives the servo motor of the motion actuator;

[0031] S500, the motion actuator feeds back the motion result to the main controller.

[0032] In some specific embodiments, the specific method of independent component extraction is:

[0033] Based on the prior knowledge of statistical independence between source signals, the ICA method is used to accurately separate statistically independent source signals from mixed signals under the condition that both the source signals and the mixing matrix are unknown, thereby realizing blind source separation of motion imagery signals.

[0034] In some specific embodiments, the specific method of feature pattern recognition is:

[0035] The support vector machine hyperplane classification algorithm (SVM) based on constrained convex optimization is used as the core processing method for pattern recognition and feature classification. The feature vector calculated by diagonalizing the fusion feature matrix space is used to classify motor imagery.

[0036] The beneficial effects of the present invention are as follows: the present invention discloses a motor imagery brain-computer interface control system and method based on feature fusion and posture feedback, comprising: a controller system and a motion actuator; the controller system comprises: a master controller and a slave controller; the master controller comprises an EEG signal acquisition module and an EEG signal processing module; the EEG signal acquisition module is used to acquire multimodal EEG signals; the EEG signal processing module is used for feature matrix fusion, spatial diagonal feature extraction and pattern classification; the slave controller comprises: a motion control module, which is used to comprehensively generate motion instructions and drive the motion actuator according to motion intention and spatial posture correction; the motion actuator is used for the specific implementation of motor imagery EEG signal actions.

[0037] (1) Compared with the existing brain-computer interface control system and motion target position parameterized control method, the present invention provides a control method and control method that can directly convert the operator's thoughts into identifiable and perceptible motion target positions and motion trends, thereby improving the accuracy and response time of motor imagery pattern classification.

[0038] (2) During use, the operator can generate real-time micro-torque adaptive coupling motion control instructions based on the classification results of the classifier, the motion state of the integrated actuator, and the current spatial posture, effectively control the motion amplitude and range of the actuator, and enhance the practicality of motion integration control with external actuators.

[0039] (3) The present invention uses the movement, thinking and intention signals of the human brain to accurately locate the target position, and controls the complex movement of multi-degree-of-freedom actuators in real time and accurately, greatly improving the accuracy, real-time and flexibility of the robot's movement in complex spaces, and enhancing its comprehensive motion control performance. It can be widely used in surgical robots, industrial application robots, military robots, household robots and other aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a functional module structure diagram of the brain-computer interface system of the present invention;

[0041] Figure 2 is a structural diagram of the brain-computer interface system of the present invention;

[0042] Figure 3 This is a calculation process for the spatial posture of a brain-computer interface system provided by an embodiment of the present invention;

[0043] Figure 4 It is a signal processing structure block diagram of the present invention;

[0044] Figure 5 It is the brain topography of independent component energy distribution of C3, C4 and Cz channels after preprocessing of the present invention;

[0045] Figure 6 It is the power spectrum of the independent source component (IC) of the C3 channel signal of the present invention;

[0046] Figure 7 It is the power spectrum of the independent source component (IC) of the C4 channel signal of the present invention;

[0047] Figure 8 It is a time-frequency analysis diagram of the C3 channel of the present invention;

[0048] Fig. 9 It is a time-frequency analysis diagram of the C4 channel of the present invention;

[0049] Fig.10It is a time-frequency analysis diagram of the Cz channel of the present invention;

[0050] Fig.11 is the C3 channel amplitude map after preprocessing of the present invention;

[0051] Fig.12 is a C4 channel amplitude map after preprocessing of the present invention;

[0052] Fig.13 It is the waveform diagram of C3, C4 and Cz channels before filtering of the present invention;

[0053] Fig.14 It is the waveform diagram after filtering of C3, C4 and Cz channels of the present invention;

[0054] Fig.15 is the maximum ERP component power spectrum of the present invention;

[0055] Fig.16 It is the first two-order ISM decomposition general diagram of the left-hand imagined movement signal of the present invention;

[0056] Fig.17 It is the first two-order ISM decomposition general diagram of the right hand imagined movement signal of the present invention;

[0057] Fig.18 It is a decomposition diagram of the left hand imaginary movement signal OR-EMR of the C3 channel of the present invention;

[0058] Fig.19 It is a decomposition diagram of the left hand imaginary movement signal OR-EMR of the C4 channel of the present invention;

[0059] Fig. 20 It is a decomposition diagram of the right hand imaginary movement signal OR-EMR of the C3 channel of the present invention;

[0060] Fig.21 It is the OR-EMR decomposition diagram of the right hand imagined movement signal of the C4 channel of the present invention;

[0061] Fig. 22 It is the power spectrum of the 8th order ISM components of the left hand in the C3 channel of the present invention before imaginary movement;

[0062] Fig.23 It is the power spectrum of the 8th order ISM components of the left hand in the C4 channel of the present invention before imaginary movement;

[0063] Fig.24 It is the power spectrum of the 8th order ISM components before the imaginary movement of the right hand in the C3 channel of the present invention;

[0064] Fig.25 It is the power spectrum of the 8th order ISM components before the C4 channel of the present invention performs imagined movement of the right hand;

[0065] Fig.26 It is a comparison diagram of EMD and OR-EMR signal waveforms of the present invention;

[0066] Fig. 27 It is a comparison diagram of the persistent spectrum and scale of the present invention;

[0067] Fig.28 is a persistent spectrum and scale comparison diagram of another embodiment of the present invention;

[0068] Fig.29 It is a scatter plot of spatial feature extraction of the LSN method of the present invention;

[0069] Fig.30 It is a scatter plot of spatial feature extraction using the OR-EMR method of the present invention;

[0070] Fig.31 It is a scatter plot of spatial feature extraction using the LSN+OR-EMR method of the present invention;

[0071] Fig.32 This is a flow chart of a motor imagery brain-computer interface control method based on feature fusion and posture feedback of the present invention.

[0072] In the attached figure, 1. operator; 2. EEG signal acquisition unit; 3. brain-computer signal transmission cable; 4. brain-computer signal converter; 5. EEG signal processing unit; 6. spatial posture estimation and motion situation perception unit; 7. motion controller unit; 8. motion control signal transmission cable; 9. actuator mechanical body unit. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0074] Reference Figure 1 A motion imagery brain-computer interface control system based on feature fusion and posture feedback is shown, including a controller system and a motion actuator.

[0075] The controller system includes a master controller and a slave controller. The master controller includes an EEG signal acquisition module and an EEG signal processing module. The EEG signal acquisition module is used to acquire multimodal EEG signals. The EEG signal processing module is used for feature matrix fusion, spatial diagonal feature extraction and pattern classification. The slave controller includes a motion control module, which is used to generate motion instructions and drive the motion actuator according to the motion intention and spatial posture correction. The motion actuator is used to implement the specific action of the EEG signal of the motor imagination.

[0076] Reference Figure 2The present invention also includes a hardware system, which includes: an EEG signal acquisition unit 2, a brain-computer signal transmission cable 3, a brain-computer signal converter 4 (EEG signal analog-to-digital conversion unit), an EEG signal processing unit 5, a spatial posture estimation and motion situation perception unit 6, a motion controller unit 7, a motion control signal transmission cable 8, and an actuator mechanical body unit 9. The operator 1 wears the EEG signal acquisition unit 2 on the top of the head, and the various hardware are connected together through the brain-computer signal transmission cable 3 to transmit signals.

[0077] Reference Figure 3 The calculation process of the spatial posture shown is as follows: first, the posture of the worker is initialized and adjusted, the target posture is set, the target adjustment value is calculated through the inverse velocity motion algorithm, and the target posture is adjusted. Finally, the adjusted posture is determined through the forward kinematics algorithm, and the posture is fed back to the target posture.

[0078] Reference Figure 4 As shown in the signal processing structure block diagram, the preprocessing stage mainly uses ICA to identify various artifact components in the original EEG signal and use IC Label to remove them.

[0079] Reference Figure 5 The following are brain topography maps of independent component energy distribution of C3, C4, and Cz channels after preprocessing (left: C3, middle: C4, right: Cz). The left picture is the brain topography map of independent component energy distribution of C3 channel after preprocessing; the middle picture is the brain topography map of independent component energy distribution of C4 channel after preprocessing; the right picture is the brain topography map of independent component energy distribution of Cz channel after preprocessing; the independent component energy distribution of different channels is different and the location of their energy distribution is different.

[0080] Reference Figure 6 and Figure 7 Shown is the independent component (IC) power spectrum of C3 and C4 channels.

[0081] in, Figure 6 is the power spectrum of the independent source component (IC) of the C3 channel signal; Figure 7 This is the power spectrum of the independent source component (IC) of the C4 channel signal. The C3 channel and the C4 channel show the characteristics of event-related desynchronization (ERD).

[0082] Reference Figure 8 , Fig. 9 and Fig.10 The time-frequency analysis diagram of C3, C4 and Cz channels shown;

[0083] Figure 8 This is the time-frequency analysis diagram of the C3 channel; Fig. 9 This is the time-frequency analysis diagram of the C4 channel; Fig.10This is the time-frequency analysis diagram of the Cz channel. There is a significant correlation in the joint analysis of time and frequency.

[0084] Reference Fig.11 and Fig.12 The amplitude diagram of C3 and C4 channels after preprocessing is shown: Fig.11 It is the power spectrum amplitude diagram of the left-hand C3 channel; Fig.12 This is the power spectrum amplitude diagram of the right-hand C4 channel.

[0085] 1. Introduction to the EEG signal acquisition module:

[0086] The present invention adopts a 32-lead electrode cap to collect brain EEG signals. The electrode positions are in accordance with the positioning standard of the international standard 10-20 system. The EEG signals are collected using C3, C4, and Cz channels. The reference electrodes select mastoid positions M1 and M2, and record horizontal and vertical electrooculograms. The contact resistance between the active electrode and the scalp is maintained below 5K ohms. The bandwidth is 1 to 30 Hz, the sampling rate is 1000 Hz, and the EEG signals are amplified by an amplifier and transmitted to a main control unit at a data collection end. The main control unit records the collected and amplified signals.

[0087] In some specific embodiments, an RS485 serial communication interface is also included to connect the controller system and the slave controller and realize reliable data transmission.

[0088] 2. Introduction to the EEG signal processing module:

[0089] The EEG signal processing module includes: preprocessing, feature matrix fusion and feature extraction and classification. It mainly completes the preprocessing of the collected brain signals, then extracts the signal features that reflect the subject's intention, and finally classifies the extracted features and converts them into control commands.

[0090] The EEG signal processing module is mainly used to complete the preprocessing of the collected EEG signals, then extract the signal features that reflect the subject's intentions, and finally classify the extracted features and convert them into control commands.

[0091] In some specific embodiments, preprocessing is used to filter (bandpass filtering), denoise (power frequency and other environmental noise), segment, baseline correct, FastICA artifact removal (remove physiological artifacts such as electrocardiogram, electrooculogram, and electromyography), perform transient time-frequency joint feature matrix fusion of mu rhythm, spatial diagonalization feature extraction, and pattern recognition classification after re-reference preprocessing of raw EEG data;

[0092] The feature matrices extracted by the local spectrum normalization (LSN) algorithm and the online regression modal reconstruction (OR-EMR) algorithm are fused in a serial (linear weighted) manner to construct a new high-quality feature fusion vector and generate a fusion feature matrix in the joint feature extraction process. This fusion feature matrix not only reflects the phase characteristics of the local spectrum, but also has time-amplitude characteristics, which is more suitable for the co-space pattern analysis of the simultaneous diagonalization of the covariance matrix and enhances the significance of the difference in motor imagery fusion features.

[0093] Feature extraction classification is used to train and classify the support vector machine based on the eigenvector corresponding to the eigenvalue after the diagonalization of the new fusion feature matrix space, which is input into the hyperplane classification to improve the response speed of the motor imagery brain-computer interface system and the accuracy of pattern recognition.

[0094] In some specific embodiments, the motion control module includes a spatial posture estimation module, which is used to integrate the motion trend and the calculation process of the initial working posture, estimate the spatial posture, and solve its current spatial posture to prevent excessive movement of the motion actuator after reaching the limit position, thereby realizing coordinated control of the drive unit.

[0095] In this embodiment, the core calculation process is to use forward kinematics to solve the current posture of the actuator and inverse kinematics to solve and control the actuator to move to the initial working posture. When the relative error between the adjusted posture and the target posture exceeds the preset error level, the posture feedback is triggered to determine the correction amount for adjusting the current posture to the target posture until the final target posture is reached.

[0096] The spatial posture positioning module is used to integrate the motion state and current spatial posture of the actuator based on the classification results of the motor imagination pattern and the strength of the motor imagination intention signal, calculate the correction amount for adjusting the current spatial posture to the target posture, comprehensively evaluate and predict the motion trend, and generate real-time micro-torque adaptive coupling motion control instructions.

[0097] Servo drive and mechanical motion control module,Servo drive and mechanical motion control module are used to achieve the consistency and coordination of motion actuator control.

[0098] In this embodiment, the servo drive encoder driven actuator converted from the micro-torque adaptive coupling motion control instructions generated by the controller (ARM processor) and its peripheral circuits effectively controls the motion amplitude and range of the actuator, achieving the continuity and coordination of the actuator motion control.

[0099] 2.1 Pre-processing

[0100] The specific preprocessing is as follows: according to the judgment and identification criteria of the independent components of electrooculogram, the quantitative calculation of event-related desynchronization (ERD) / event-related synchronization (ERS) phenomenon is performed using formula (1);

[0101]

[0102] In formula (1), R and A are the power spectrum values ​​of the specific frequency band of the EEG signal before and after the motor imagery task, respectively. ERD / ERS <0, it means that the power spectrum value of a certain frequency band decreases after the motor imagery task, proving that the ERD phenomenon occurs; on the contrary, when C ERD / ERS >0, it means that ERS phenomenon occurs.

[0103] In this embodiment, the identification of artifact components in the EEG signal is performed based on the following two criteria:

[0104] 1) The smoothly decreasing power spectrum is a typical feature of electrooculographic artifacts;

[0105] 2) The projection close to the forehead in the EEG topography is another characteristic of the electrooculogram component.

[0106] According to the calculation of the power spectrum, if the power spectrum of the component shows a smooth decline, the independent component can be determined as an electrooculographic artifact.

[0107] First, the original EEG data in the 100ms period before motor imagery was selected for 8-32Hz bandpass filtering (bandpass filter based on finite impulse response) to remove a series of power frequency and DC interference (EEG signal denoising). According to the mark label, the data before and after the stimulation event point was segmented (epoch). After segmentation, the original EEG data in the 100ms period before motor imagery was averaged, and the result of the average calculation was used as the reference value for baseline correction. The corrected baseline value was subtracted from the sample point value in the segment to eliminate the influence of baseline drift, reduce signal glitches, and improve the signal-to-noise ratio. The baseline-corrected signal was identified and removed using the independent component analysis (ICA) algorithm (removing or shielding artifact components such as electrooculogram) to obtain the significant frequency band signal of motor imagery ERD / ERS of mu rhythm (8-12Hzrhythm). In order to reduce the dependence on the position of the reference electrode, necessary re-reference (re-reference the average value) was performed, that is, the average value of all electrodes (linear combination) was subtracted from each electrode. Fig.13 The waveforms of C3, C4 and Cz channels before filtering are shown. Fig.14 Shown are the waveforms of the C3, C4, and Cz channels after filtering, segmentation, and baseline correction. Fig.15 Shown is the maximum ERP (independent component) power spectrum.

[0108] In some specific embodiments, feature matrix fusion is specifically as follows: the EEG signal after filtering, denoising, segmentation, baseline correction, FastICA artifact removal, and re-reference preprocessing is subjected to local spectral normalization (LSN) and online regression-Empirical Mode Reconstruction (OR-EMR), and then connected in series end to end using linear weighting to construct a feature connection matrix that integrates local spectral phase characteristics and time domain amplitude space diagonalization characteristics, thereby obtaining the final feature matrix for feature extraction and pattern recognition classification.

[0109] Reference Fig.32 A motor imagery brain-computer interface control method based on feature fusion and posture feedback is shown, comprising the following steps:

[0110] S100. The user wears a high-precision multimodal signal acquisition head cap and performs mental thinking and imagination according to the movement intention. The EEG signal acquisition module obtains the spontaneous EEG-related potential signals of the occipital lobe area of ​​the brain.

[0111] S200, after the induced signal is picked up by the acquisition head cap, it is amplified by the amplifier and connected to the main controller through the RS485 serial communication interface. The EEG signal processing module pre-processes the original EEG data, extracts independent components, reconstructs feature fusion, extracts features, and recognizes feature patterns.

[0112] S300, the main controller converts the motion intention into an instruction according to the processed classification result and transmits it to the slave controller. After receiving the motion intention instruction, the slave controller integrates the motion state and current spatial posture of the motion actuator to generate a real-time micro-torque adaptive coupling motion control instruction. The generation of motion control instructions includes the positioning, estimation and integration of motion trends of spatial posture. The integration process includes calculating the initial working posture and solving its current spatial posture to prevent excessive movement of the actuator (robot arm forearm) after reaching the limit position, so as to realize the coordinated control of the drive unit.

[0113] S400, the slave controller (ARM processor) converts the generated micro-torque adaptive coupling motion control instruction into the code of the servo drive encoder, and inputs the code into the servo drive controller through the RS485 communication bus, and drives the servo motor of the motion actuator.

[0114] S500, the motion actuator feeds back the motion result to the main controller.

[0115] The invention proposes for the first time that the feature matrix extracted by local spectrum normalization and the feature matrix extracted by online regression modal reconstruction are serially weighted fused to obtain new fusion features, which significantly enhances the difference of the feature matrices, improves the accuracy and real-time performance of the motor imagery classification algorithm, and further provides a motor imagery brain-computer interface control method based on feature fusion and posture feedback.

[0116] In some specific embodiments, the specific method of independent component extraction is:

[0117] Based on the prior knowledge of statistical independence between source signals, the ICA method is used to accurately separate statistically independent source signals from mixed signals under the condition that both the source signals and the mixing matrix are unknown, thereby realizing blind source separation of motion imagery signals.

[0118] In this embodiment, the present invention uses the information maximization criterion and combines the judgment and recognition criterion of the independent components of electrooculography to identify and remove the artifact components such as electrooculography.

[0119] FastICA algorithm based on information maximization statistical independence criterion

[0120] The independent component extraction algorithm used in the present invention is the FastICA algorithm based on the information transfer maximization criterion (Information-Maximization, Informax) algorithm proposed by Bell and Sejnowski, which integrates the nonlinear irrelevance rule. Using the maximum information transfer criterion, the mutual information I(x, y) between the input signal x and the output signal y is maximized by adjusting the separation matrix W. The mutual information is defined as follows according to formula (2):

[0121] I(x,y)=H(y)-H(y / x) (2)

[0122] Where H(y) is the joint output entropy of the output signal, and H(y / x) is the conditional entropy of the output. Find the maximum value of I(x,y).

[0123] 2.2 Modal Normalization Reconstruction Based on Local Spectral Phase Transform (LSN)

[0124] The local spectral normalization (LSN) method based on local spectral phase transformation proposed in the present invention is a time-frequency analysis method with local spectral phase characteristics, which effectively combines the local phase information of the absolute reference (the peak value of the local spectral amplitude and the phase outside the peak value, the phase information refers to the parameters of the sine wave at time zero) and the progressive resolution of the time-frequency space phase. The method retains the absolute reference phase information. The proposed method can not only estimate the local power spectrum, but also estimate the local phase spectrum of the complex-valued time series.

[0125] Based on the Short-Time Fourier Transformation (STFT), the present invention creatively proposes to use a local sliding convolution time window function (as shown in formula (5)) whose width is proportional to the inverse of the frequency for analysis. The integral interval of the sliding convolution is centered on τ and the phase factor e is used. j2πfτ The multiplication operation is performed to characterize the position of the local spectrum transformation function on the transformation time axis. This improves the defect of fixed integral difference. The sliding convolution function is related to the frequency f. As the frequency f changes, the integral interval of the sliding convolution function changes accordingly, thereby improving the defect of the STFT fixed window.

[0126] The signal x(n) (channel*sampling point*trails) used in the present invention is transformed from LSN to TFS, resulting in data dimensionality increase. In signal processing, the one-dimensional signal x(i) (the time series of each channel of all trails, that is, the time series of C3 and C4 channels) is transformed from LSN to obtain a two-dimensional complex variable matrix L s (frequency sampling point * time sampling point), L s T Input to PCA to extract components, select the required number of components (k=3) according to the setting of parameter k to obtain new feature data (time sampling point*component component), and fuse the component component with the channel to form a three-dimensional matrix X_lsn (component component*sampling point*trials).

[0127] 2.2.1LSN Algorithm Process

[0128] Step 1: Perform Fast Fourier Transform (FFT) on the input signal x(t) to obtain the Fourier spectrum of the input signal.

[0129] Step 2 calculates the local sliding convolution time window function according to the sampling point position.

[0130] Step 3 removes the sliding window at the edge of the 5% Hanning taper.

[0131] Step 4 calculates the frequency domain results, multiplies the Fourier coefficient of each frequency component by the sliding convolution function corresponding to the frequency, and highlights the local time-frequency characteristics of the EEG signal near each frequency component.

[0132] Step 5: Perform Fourier transform on the frequency domain result to obtain the time domain LSN expression.

[0133] Step 6: Repeat steps 2, 3 and 4 to obtain the corresponding LSN results of all frequency sampling points.

[0134] Step 7 performs principal component analysis (PCA) and dimensionality reduction on the results of LSN.

[0135] 2.3.2LSN Algorithm Steps

[0136] A variable local sliding convolution time window function obtains the local spectrum of the input signal. The local sliding convolution time window function ω(t) is defined as follows according to formula (3):

[0137]

[0138] In formula (3), the scale factor δ is used to change the width of the local sliding convolution time window function to extract different time-frequency components, and the scale factor δ is related to the frequency component f according to formula (4).

[0139]

[0140] Therefore, the local sliding convolution time window function can be defined as follows according to formula (5):

[0141]

[0142] The length l of the Hanning taper is defined as follows according to formula (6), and 5% of the Hanning taper edge is removed according to formula (7).

[0143] l=floor(n×0.05) (6)

[0144]

[0145] In formula (7), h(l) is the Hanning window, and the floor(.) function means rounding down to an integer. The local spectrum phase transform of the signal x(t) is defined as follows according to formula (8):

[0146]

[0147] Where, L s(τ, f0) is the time frequency spectrum (TFS) obtained after the transformation of the input signal x(t), which is a one-dimensional time function of the constant spectrum f0, indicating the change of the amplitude and phase of the specific spectrum (Exact Frequency) over time. According to formula (9), the discretized LSN processing is performed to construct the discrete spectrum formula:

[0148]

[0149] In formula (9), The Fourier transform of the N-point time series x[kT]. j,m,n=0,1,...,N-1, the sampling method of the local spectrum phase transformation There is one sampling point at the time sampling point and one sampling point at the Fourier frequency sampling point.

[0150] 2.3 Online Modal Regression Reconstruction (OR-EMR)

[0151] The present invention innovatively proposes and uses an online regression window to decompose the signal into a finite number of online regression intrinsic spectral modes (OR-ISM) on a spectral scale. Each order ISM component represents an oscillation mode with different amplitudes and frequencies, contains corresponding frequency domain information, and changes dynamically with the change of the signal, and has adaptive characteristics. The present invention creatively proposes to use an online regression window for frequency band decomposition and reconstruction (as shown in formula (11) and formula (12)). The decomposed signal is more in line with and adapts to the calculation of spatial diagonalization and eigenvalues ​​in amplitude, highlighting the contribution of the main components of the signal and significantly reflecting the characteristics of data diagonalization. The number of ISMs after decomposition is concentrated between 7 and 9, among which the energy of ISM1, ISM2 and ISM3 is relatively concentrated and within the range of mu rhythm. The decomposed ISM1, ISM2 and ISM3 components are reconstructed as the main components, and the modal aliasing is effectively eliminated after reconstruction.

[0152] 2.3.1 OR-EMR algorithm steps

[0153] The OR-EMR method adds a regression sliding window to the EMD method, and realizes the online modal regression reconstruction of EMR through its translation, thereby reducing the aliasing of the inherent modes of the reconstructed signal to a certain extent.

[0154] For the input one-dimensional signal x(t) (the time series of each channel of all trails, i.e., the time series of C3 and C4 channels), it is first preprocessed and standardized, and the data z is obtained by the z-score standardization formula according to formula (10).

[0155]

[0156] The Hann window is obtained by equation (11), and the signal x'(t) after the amplitude is calculated is obtained by equation (12).

[0157]

[0158] x′(t)=z(t).*h(n) (12)

[0159] The ISM components decomposed by OR-EMR meet two conditions: ① In the entire component, the number of extreme value points and the number of intersections of zero value points are equal or at most one different. ② At any data position, the average value of the upper and lower envelopes formed by the local extreme values ​​is zero.

[0160] Each trail obtains its ISM component c(t) and residual component r(t) through the local extreme value screening method.

[0161] The steps of ISM component decomposition are as follows:

[0162] Step 1 Input data x′(t) and obtain all local maximum values ​​e max (t) and the minimum value e mim (t), forming a local extreme value.

[0163] Step 2: Using the cubic spline interpolation method, all local maximum and minimum values ​​form the upper and lower envelopes respectively, and the envelope mean m1(t) is calculated according to formula (13).

[0164]

[0165] Step 3: Subtract the envelope mean from the input data according to equation (14) to obtain the first component h1(t).

[0166] h1(t)=x(t)-m1(t) (14)

[0167] Step 4 determines whether h1(t) satisfies the ISM condition. If so, jump to step 6, otherwise jump to step 5.

[0168] Step 5: Take h1(t) as input data, perform secondary screening, repeat steps 1 to 4, and obtain the envelope mean m. 11 (t) and the component h 11 (t), repeat the screening k times until the ISM condition or the screening stop condition is met. The condition judgment basis is as shown in formula (15), which is usually set as the standard deviation parameter SD is less than the set threshold value, which is 0.2~0.3.

[0169]

[0170] Step 6: If the ISM condition or the screening stop condition is met, x′(t) is the first-order ISM component c1(t), and the residual component r1(t) can be obtained according to formula (16).

[0171] r1(t)=x′(t)-c1(t) (16)

[0172] Step 7: Take the residual component r1(t) as the new input data, execute steps 1 to 6, and obtain the residual component r2(t), and repeat until the nth residual component r n (t) is a monotonic function or a constant. The process stops when it cannot be decomposed.

[0173] Therefore, as shown in Equation (17), the original input data x'(t) is decomposed into the sum of all ISM components and an average trend component r n (t).

[0174]

[0175] The first three-order ISM components of each channel of all trials are combined and sampled to form a matrix C×N×T, where C is the sum of the ISM components of each channel, N is the time sampling point, and T is the trails. Then the data of all trails are calculated to form a three-dimensional matrix X_or-emr (component component*sampling point*trails).

[0176] 2.3.2 OR-EMR reconstruction results

[0177] Fig.16 , Fig.17 , Fig.18 , Fig.19 , Fig. 20 and Fig.21 The ISM component map is obtained by OR-EMR decomposition of EEG signal. Fig. 22 , Fig.23 , Fig.24 and Fig.25 is the power spectrum of the ISM component of the EEG signal, Fig.26 Comparison of signal waveforms after EMD and OR-EMR reconstruction. Fig. 27 and Fig.28 For comparison of persistence spectrum and scale map.

[0178] 2.4 Feature Matrix Fusion

[0179] In this embodiment, after LSN and OR-EMR reconstruction, linear weighting is used to connect them in series to construct a feature connection matrix that integrates local spectrum phase characteristics and time domain amplitude space diagonalization characteristics, and finally obtains the feature matrix for feature extraction and pattern recognition classification.

[0180] 2.4.1 Feature Matrix Fusion

[0181] The components of the datasets X_lsn (component*sampling point*trails) and X_or-emr (component*sampling point*trails) are converted into X_lsn′ (channel*sampling point*trails) and X_or-emr′ (channel*sampling point*trails) by linear weighting. The two datasets represent different characteristics, but their intrinsic connections in representing different categories (left-handed category, right-handed category) are the same.

[0182] 2.4.2 Spatial diagonal feature extraction

[0183] In this embodiment, the mature CSP algorithm based on the idea of ​​simultaneous diagonalization of two types of covariance matrices is used to project and spatially diagonalize the matrix (feature fusion matrix) that fuses the local spectrum phase characteristics and the time domain amplitude spatial diagonal characteristics, and solve the eigenvalues ​​and eigenvectors of the fusion matrix (spatial domain feature extraction). The matrix data set after feature fusion is divided into a training set X_train and a test set X_test, and the training labels Y_train and the test labels Y_test are obtained for spatial domain feature extraction.

[0184] 2.4.3 Space diagonalization steps

[0185] The CSP algorithm linear transformation projects multi-channel EEG data into a low-dimensional subspace to obtain a projection matrix. Each row of the projection matrix is ​​weighted by the weighting factor of the channel to obtain a composite covariance matrix. The composite covariance matrix is ​​diagonally decomposed to obtain a diagonal matrix of eigenvalues, so that the eigenvector corresponding to the maximum eigenvalue of the left-hand motor imagination in the two types of motor imagination corresponds to the minimum eigenvalue of the right-hand motor imagination. The variance of the composite covariance matrix is ​​maximized, thereby increasing the obviousness of the feature difference.

[0186] Step 1: Calculate the covariance matrix R of each signal of each class according to formula (18): i (R L : Left hand, R R : Right hand). The covariance matrix reflects the pairwise correlation between channels.

[0187]

[0188] Note: trace() represents the trace of the matrix, that is, the sum of the diagonal elements.

[0189] According to formula (19), the two types of covariance matrix data are averaged and normalized to obtain the average total covariance matrix R c .

[0190]

[0191] Step 2 Average total covariance matrix R c The significance of performing eigenvalue decomposition according to formula (20) is to use sample points to fit a straight line. The direction of the eigenvector corresponding to the maximum eigenvalue is the direction of the sample point fitting straight line, and the direction of the normal vector of the minimum eigenvalue is the direction of the fitting straight line.

[0192] R c =U c λU c T (20)

[0193] In formula (20), U c is the eigenvector matrix, and λ is the diagonal matrix of eigenvalues. Arrange the eigenvalues ​​in descending order according to formula (21) to obtain the whitening matrix P.

[0194]

[0195] Step 3: Perform whitening transformation according to equations (22) and (23) to obtain the average covariance whitening transformation matrix S for the left-hand class and the right-hand class respectively. L and S R As shown in equations (24) and (25), matrix whitening can transform the covariance matrix into a unit matrix, making the variances of its vectors the same, removing the correlation between the signals, simplifying the subsequent component extraction process, and having convergence.

[0196]

[0197] Where S L and S R They have a common eigenvector, that is, the sum of the eigenvalue diagonal matrices after eigenvalue decomposition is the unit matrix.

[0198]

[0199] λ L +λ R =I (26)

[0200] U L =U R =U (27)

[0201] It can be seen from formula (26) that S L The eigenvector with the largest eigenvalue corresponds to S R The eigenvector of the minimum eigenvalue is thus realized when one type of eigenvalue is maximized while the other type of eigenvalue is minimized. At the same time, according to formula (27), S L and S R have the same eigenvectors.

[0202] Step 4 defines the projection matrix W as a spatial filter, which is defined as follows according to formula (28):

[0203] W=U T P (28)

[0204] According to formula (29), the input data X i (The training set X_train and the test set X_test are input separately) After this filter, the projected feature matrix Z is obtained i , extract the feature vector f based on the logarithmic power feature according to formula (30) i .

[0205] Z i =W·X i (29)

[0206]

[0207] Reference Fig.30 , Fig.31 and Fig.32 Shown are the scatter plots of spatial feature extraction using three methods: LSN, OR-EMR, and LSN+OR-EMR feature fusion. From the figure, it can be seen that the scatter plot distribution of feature extraction using the LSN+OR-EMR method is more approximately symmetrical.

[0208] In some specific embodiments, the specific method of feature pattern recognition is:

[0209] This embodiment uses a support vector machine hyperplane classification algorithm (SVM) based on constrained convex optimization as the core processing method for pattern recognition and feature classification, and uses the feature vectors calculated by diagonalizing the fusion feature matrix space to classify motor imagery. The SVM classification algorithm converts the optimization problem of inequality constraints into an equality constraint problem by the Lagrange multiplier method, and then converts the equality constraint problem into an unconstrained extreme value optimization problem.

[0210] The sub-process is divided into four sub-steps: (1) transforming constrained convex optimization into unconstrained extreme value optimization; (2) transforming the original problem into a dual problem; (3) using kernel functions to map high-dimensional space; and (4) using the SMO algorithm to iteratively solve the problem.

[0211] The beneficial effects of the present invention are as follows: the present invention discloses a motor imagery brain-computer interface control system and method based on feature fusion and posture feedback, comprising: a controller system and a motion actuator; the controller system comprises: a master controller and a slave controller; the master controller comprises an EEG signal acquisition module and an EEG signal processing module; the EEG signal acquisition module is used to acquire multimodal EEG signals; the EEG signal processing module is used for feature matrix fusion, spatial diagonal feature extraction and pattern classification; the slave controller comprises: a motion control module, which is used to comprehensively generate motion instructions and drive the motion actuator according to motion intention and spatial posture correction; the motion actuator is used for the specific implementation of motor imagery EEG signal actions.

[0212] (1) Compared with the existing brain-computer interface control system and motion target position parameterized control method, the present invention provides a control method and control method that can directly convert the operator's thoughts into identifiable and perceptible motion target positions and motion trends, thereby improving the accuracy and response time of motor imagery pattern classification.

[0213] (2) During use, the operator can generate real-time micro-torque adaptive coupling motion control instructions based on the classification results of the classifier, the motion state of the integrated actuator, and the current spatial posture, effectively control the motion amplitude and range of the actuator, and enhance the practicality of motion integration control with external actuators.

[0214] (3) The present invention uses the movement, thinking and intention signals of the human brain to accurately locate the target position, and controls the complex movement of multi-degree-of-freedom actuators in real time and accurately, greatly improving the accuracy, real-time and flexibility of the robot's movement in complex spaces, and enhancing its comprehensive motion control performance. It can be widely used in surgical robots, industrial application robots, military robots, household robots and other aspects.

[0215] After experimental comparison and statistical analysis of application scenarios, it can be seen that the average classification accuracy of the LSN+OR-EMR scheme is 11% higher than that of the LSN scheme, and 5% higher than that of the OR-EMR scheme. It is obvious that the accuracy has been greatly improved, and the response time is also significantly better than the response time of the single method feature extraction. Table 1 shows the average classification accuracy, response time and training time of the three schemes.

[0216] Table 1 Comparison of average classification accuracy, response time and training time

[0217]

Claims

1. A motor imagery brain-computer interface control system based on feature fusion and posture feedback, characterized in that: include: Controller systems and motion actuators; The controller system comprises: a master controller and a slave controller; the master controller comprises an EEG signal acquisition module and an EEG signal processing module; the EEG signal acquisition module is used to acquire multimodal EEG signals; the EEG signal processing module is used for feature matrix fusion, spatial diagonal feature extraction and pattern classification; The slave controller includes: a motion control module, which is used to comprehensively generate motion instructions and drive the motion actuator according to the motion intention and spatial posture correction; The motion execution mechanism is used for the specific implementation of the motor imagery EEG signal action.

2. The motor imagery brain-computer interface control system based on feature fusion and posture feedback according to claim 1 is characterized in that: Also includes: The RS485 serial communication interface is used to connect the controller system and the slave controller and realize reliable data transmission.

3. The motor imagery brain-computer interface control system based on feature fusion and posture feedback according to claim 2 is characterized in that: The EEG signal processing module includes: preprocessing, feature matrix fusion and feature extraction classification; The EEG signal processing module is mainly used to complete the preprocessing of the collected EEG signals, then extract the signal features reflecting the subject's intention, and finally classify the extracted features and convert them into control commands.

4. The motor imagery brain-computer interface control system based on feature fusion and posture feedback according to claim 3 is characterized in that: The preprocessing is used to filter, denoise, segment, baseline correct, remove FastICA artifacts, and perform mu rhythm transient time-frequency joint feature matrix fusion, spatial diagonal feature extraction, and pattern recognition classification on the raw EEG data after re-reference preprocessing; The feature matrix fusion is used to fuse the feature matrices extracted by the local spectrum normalization algorithm and the online regression modal reconstruction algorithm in a joint feature extraction process in a serial manner to construct a new high-quality feature fusion vector; The feature extraction classification is used to train and classify a support vector machine for hyperplane classification based on the feature vectors corresponding to the feature values ​​after the new fusion feature matrix space is diagonalized and input.

5. The motor imagery brain-computer interface control system based on feature fusion and posture feedback according to claim 4 is characterized in that: The motion control module comprises: A spatial posture estimation module, which is used to integrate the calculation process of the motion trend and the initial working posture, estimate the spatial posture, and solve its current spatial posture to prevent excessive movement of the motion actuator after reaching the limit position, so as to realize the coordinated control of the drive unit; A spatial posture positioning module, which is used to integrate the motion state of the actuator and the current spatial posture based on the motor imagery pattern classification result and the motor imagery intention signal strength, calculate the correction amount of the current spatial posture to adjust to the target posture, comprehensively evaluate and predict the motion trend, and generate real-time micro-torque adaptive coupling motion control instructions; A servo drive and mechanical motion control module, wherein the servo drive and mechanical motion control module is used to achieve the consistency and coordination of the motion actuator control.

6. The motor imagery brain-computer interface control system based on feature fusion and posture feedback according to claim 4 is characterized in that: The preprocessing is specifically as follows: according to the judgment and identification criteria of the independent components of electrooculogram, quantitative calculation of event-related desynchronization (ERD) / event-related synchronization (ERS) phenomenon is performed using formula (1); In formula (1), R and A are the power spectrum values ​​of the specific frequency band of the EEG signal before and after the motor imagery task, respectively. ERD / ERS <0, it means that the power spectrum value of a certain frequency band decreases after the motor imagery task, proving that the ERD phenomenon occurs; on the contrary, when C ERD / ERS >0, it means that ERS phenomenon occurs.

7. The motor imagery brain-computer interface control system based on feature fusion and posture feedback according to claim 6 is characterized in that: The feature matrix fusion is specifically as follows: the EEG signal after filtering, denoising, segmentation, baseline correction, FastICA artifact removal, and re-reference preprocessing is subjected to local spectral normalization (LSN) and online regression-Empirical Mode Reconstruction (OR-EMR), and then connected in series end to end using linear weighting to construct a feature connection matrix that integrates local spectral phase characteristics and time domain amplitude space diagonalization characteristics, thereby obtaining the final feature matrix for feature extraction and pattern recognition classification.

8. A motor imagery brain-computer interface control method based on feature fusion and posture feedback, characterized in that: The following steps are involved: S100, the user wears a high-precision multi-modal signal acquisition head cap and performs mental thinking and imagination according to the movement intention, and the EEG signal acquisition module obtains the spontaneous EEG related potential signal of the occipital lobe area of ​​the brain; S200, after the induced signal is picked up by the acquisition head cap, it is amplified by the amplifier and connected to the main controller through the RS485 serial communication interface, and the EEG signal processing module performs preprocessing, independent component extraction, feature fusion reconstruction, feature extraction, and feature pattern recognition on the original EEG data; S300, the master controller converts the motion intention into an instruction according to the processed classification result and transmits it to the slave controller, and after receiving the motion intention instruction, the slave controller integrates the motion state and current spatial posture of the motion actuator to generate a real-time micro-torque adaptive coupling motion control instruction; S400, the slave controller converts the generated micro-torque adaptive coupling motion control instruction into the code of the servo drive encoder, and inputs the code into the servo drive controller through the RS485 communication bus, and drives the servo motor of the motion actuator; S500: The motion execution mechanism feeds back the motion result to the main controller.

9. The motor imagery brain-computer interface control method based on feature fusion and posture feedback according to claim 8 is characterized in that: The specific method of extracting the independent components is: Based on the prior knowledge of statistical independence between source signals, the ICA method is used to accurately separate statistically independent source signals from mixed signals under the condition that both the source signals and the mixing matrix are unknown, thereby realizing blind source separation of motion imagery signals.

10. The motor imagery brain-computer interface control method based on feature fusion and posture feedback according to claim 8, characterized in that: The specific method of the characteristic pattern recognition is: The support vector machine hyperplane classification algorithm (SVM) based on constrained convex optimization is used as the core processing method for pattern recognition and feature classification. The feature vector calculated by diagonalizing the fused feature matrix space is used to classify motor imagery.

Images