AR-based SSVEP-mi control and rehabilitation training wheelchair

Abstract

The application relates to an AR-based SSVEP-MI control and rehabilitation training wheelchair, which comprises an electric wheelchair and further comprises: an electroencephalogram acquisition module, which is used for collecting SSVEP electroencephalogram signals and / or MI electroencephalogram signals in real time; an AR guiding module, which is used for displaying a stimulation guiding interface for inducing SSVEP electroencephalogram signals and guiding MI electroencephalogram signals; an algorithm module, which is used for acquiring the SSVEP electroencephalogram signals and the MI electroencephalogram signals, identifying the SSVEP electroencephalogram signals and the MI electroencephalogram signals respectively, generating display information based on the identification results, and generating display control instructions for the stimulation guiding interface and / or action control instructions for the electric wheelchair. Compared with the prior art, the application has the advantages of taking into account the needs of rehabilitation training and motion control, high reliability, and improving the application range of the brain-controlled wheelchair.

Description

Technical Field

[0001] This invention relates to the technical field of brain-computer interfaces, specifically to an AR-based SSVEP-MI control and rehabilitation training wheelchair. Background Technology

[0002] For people with mobility impairments, wheelchairs are convenient assistive mobile devices that can serve as a means of transportation, expand their activity area, and improve their quality of life. Currently, common types of wheelchairs on the market include manual wheelchairs, which are pushed manually by the user or by someone else, and electric wheelchairs, which are controlled by joysticks or buttons. However, some special groups of people are unable to use these two common types of wheelchairs independently. For a small number of paralyzed individuals who have lost communication and limb mobility, they cannot control the wheelchair's movement with their upper limbs or use joysticks to control the electric wheelchair. These individuals can only move around with the help of caregivers, leading to a reduction in their living space, a decline in their quality of life, and a loss of enthusiasm for rehabilitation.

[0003] Taking stroke as an example, stroke is a serious chronic non-communicable disease that severely endangers health, characterized by five major features: high incidence, high disability rate, high mortality rate, high recurrence rate, and high economic burden. Even survivors of stroke often experience nerve damage, affecting motor function; in severe cases, they may completely lose control of their limbs. Most patients retain clear thinking but are unable to control limb movement through their own neural circuits. Therefore, stroke patients often experience mobility issues, requiring caregivers for daily life and rehabilitation after surgery, which can place a burden on families and lead to social problems.

[0004] Currently, with the continuous development of Brain-Computer Interaction (BCI) technology, its application in the field of intelligent healthcare has become a trend. Therefore, brain-controlled wheelchairs have emerged, applying BCI technology to traditional wheelchairs to assist patients in autonomously controlling them and improving their quality of life. Currently, common brain-controlled wheelchairs primarily use Steady-State Visual Evoked Potentials (SSVEP) BCIs for wheelchair control, which generally has high accuracy. However, this method is limited in scope and does not provide rehabilitation benefits for motor nerves. Motor Imagery (MI) BCIs have a rehabilitative effect on damaged nerves, but their accuracy is not ideal and they cannot be used as control signals for wheelchair control.

[0005] Therefore, new brain-controlled wheelchair technology needs to be designed. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an AR-based SSVEP-MI control and rehabilitation training wheelchair that meets the needs of rehabilitation training and motor control and has high reliability.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] An AR-based SSVEP-MI control and rehabilitation training wheelchair, including an electric wheelchair, and also including:

[0009] The EEG acquisition module is used to acquire SSVEP EEG signals and / or MI EEG signals in real time;

[0010] The AR guidance module is used to display a stimulation guidance interface for inducing SSVEP EEG signals and guiding MI EEG signals;

[0011] The algorithm module is used to acquire the SSVEP EEG signal and the MI EEG signal, identify the SSVEP EEG signal and the MI EEG signal respectively, generate display information based on the identification results, and generate display control commands for the stimulation guidance interface and / or motion control commands for the electric wheelchair.

[0012] As a preferred embodiment of the present invention, the EEG acquisition module includes a 64-conductive electrode EEG cap.

[0013] As a preferred embodiment of the present invention, the EEG acquisition module is connected to the algorithm module via TCP / IP communication.

[0014] As a preferred technical solution of the present invention, the AR guidance module is connected to the algorithm module via UDP communication.

[0015] As a preferred technical solution of the present invention, the stimulation guidance interface includes a main interface and a secondary interface. The main interface is the SSVEP stimulation interface, which includes five flashing stimulation blocks representing forward, backward, left turn, right turn and stop respectively. The secondary interface is the MI guidance interface.

[0016] As a preferred technical solution of the present invention, each of the flashing stimulation blocks flashes at different frequencies.

[0017] As a preferred technical solution of the present invention, the algorithm module includes:

[0018] The SSVEP decoding unit is used to decode the SSVEP EEG signal using the FBCCA algorithm when the SSVEP EEG signal is received, and to obtain the SSVEP classification result.

[0019] The judgment unit is used to judge the SSVEP classification result and execute corresponding actions based on the judgment result. Specifically, when the SSVEP classification result is forward, backward or stop, the corresponding display information and corresponding action control command for the electric wheelchair are generated at the same time. When the SSVEP classification result is left turn or right turn, the display control command for the stimulus guidance interface is generated, so that the stimulus guidance interface is displayed as a secondary interface.

[0020] The MI decoding unit is used to decode the MI EEG signal using the CSP-LDA algorithm when the MI EEG signal is received, obtain the MI classification result, compare the MI classification result with the SSVEP classification result, and if they match, generate the corresponding display information and corresponding action control command; otherwise, the MI EEG signal is repeatedly acquired.

[0021] The return unit is used to generate display control commands for the stimulus guidance interface, so that the stimulus guidance interface displays the main interface.

[0022] As a preferred technical solution of the present invention, both the SSVEP decoding unit and the MI decoding unit include a preprocessing subunit for signal preprocessing, which includes various methods such as filtering, downsampling, and removal of electrooculography artifacts.

[0023] As a preferred embodiment of the present invention, the AR guidance module includes AR glasses.

[0024] As a preferred embodiment of the present invention, the electric wheelchair includes a wheelchair controller connected to the algorithm module via a serial port.

[0025] Research on brain-controlled wheelchairs is still in its developmental stage, lacking a fixed model, but possesses significant potential research value. Compared with existing technologies, this invention has the following beneficial effects:

[0026] (1) This invention combines MI and SSVEP, combining the advantages of the two brain-computer interfaces and weakening their disadvantages, and builds a multimodal BCI system with two parallel paradigms. It takes into account the needs of patient rehabilitation training and motor control, has high reliability and strong scalability, and improves the application scope of brain-controlled wheelchairs.

[0027] (2) Based on AR technology, this invention designs two types of brain-computer interface paradigms into a two-level interface, making the two more closely integrated and the interaction more natural and user-friendly during use. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall framework of the present invention;

[0029] Figure 2 The interface and algorithm flowchart are provided to stimulate the user.

[0030] Figure 3 Here is the flowchart for the FBCCA algorithm;

[0031] Figure 4 Here is a flowchart of the CSP-LDA algorithm;

[0032] Figure 5 This is a schematic diagram of the hardware structure of the present invention. Detailed Implementation

[0033] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0034] like Figure 1 As shown, this embodiment provides an AR-based SSVEP-MI control and rehabilitation training wheelchair, including an electric wheelchair, an EEG acquisition module, an AR guidance module, and an algorithm module. The EEG acquisition module is used to acquire SSVEP EEG signals and / or MI EEG signals in real time. The AR guidance module is used to display a stimulation guidance interface to induce SSVEP EEG signals and guide MI EEG signals. The algorithm module is used to acquire the SSVEP and MI EEG signals, identify them respectively, generate display information based on the identification results, and generate display control commands for the stimulation guidance interface and / or movement control commands for the electric wheelchair. The wheelchair can analyze and identify the two types of EEG signals, control the wheelchair's forward, backward, left, and right turns, and simultaneously rehabilitate the patient's brain's neurological function. The aforementioned wheelchair device integrates SSVEP and MI, combining the advantages of both brain-computer interfaces while mitigating their disadvantages. It achieves a novel wheelchair device that analyzes and identifies two types of EEG signals, controls the wheelchair's forward, backward, left, and right turns, and simultaneously rehabilitates the patient's brain's neural function. It is multifunctional, highly scalable, and provides new ideas for the development of brain-controlled wheelchairs.

[0035] In this embodiment, the algorithm module is housed in a host computer. The EEG acquisition module includes a 64-conductor wet electrode EEG cap, a signal amplifier, a router (intelligent synchronization center), and a labeler (multi-parameter synchronizer). The EEG cap is used to contact the scalp and detect weak potential changes for real-time acquisition of the user's EEG signals. The signal amplifier amplifies the signals detected by the electrodes. The router transmits the acquired EEG data, and the labeler performs hardware labeling, tagging the data for each trial. The router connects to the signal amplifier and labeler via WiFi, the labeler establishes serial communication with the host computer via a USB interface, and the router establishes TCP communication with the host computer. Through these devices, the host computer can acquire real-time EEG signals and transmit them to the algorithm module for real-time analysis via TCP / IP communication.

[0036] In a specific implementation, the EEG cap used is the Borricon 64 moisture-conducting electrode EEG cap.

[0037] In this embodiment, the AR guidance module is connected to the algorithm module via UDP communication. The stimulation guidance interface displayed in the AR guidance module is used to induce SSVEP EEG signals and guide MI EEG signals. The stimulation guidance interface includes a main interface and a secondary interface. The main interface is the SSVEP stimulation interface, which includes five flashing stimulation blocks representing forward, backward, left turn, right turn, and stop, respectively. The secondary interface is the MI guidance interface. In a specific implementation, each of the flashing stimulation blocks flashes at a different frequency to better guide the user's selection and more accurately determine the user's choice.

[0038] In this embodiment, the algorithm module includes an SSVEP decoding unit, a judgment unit, an MI decoding unit, and a return unit. The SSVEP decoding unit is the main algorithm unit, and the MI decoding unit is the secondary algorithm unit. The switching between the main and secondary algorithms is synchronized with the switching of the stimulus guidance interface. The SSVEP decoding unit decodes the received SSVEP EEG signal using the FBCCA algorithm to obtain the SSVEP classification result. The judgment unit judges the SSVEP classification result and executes corresponding actions based on the judgment result. Specifically, when the SSVEP classification result is forward, backward, or stop, corresponding display information and corresponding action control commands for the electric wheelchair are generated simultaneously. When the SSVEP classification result is left turn or right turn, display control commands for the stimulation guidance interface are generated, causing the stimulation guidance interface to display as a secondary interface. The MI decoding unit decodes the received MI EEG signal using the CSP-LDA algorithm to obtain the MI classification result. The MI classification result is compared with the SSVEP classification result. If they match, corresponding display information and corresponding action control commands are generated; otherwise, the MI EEG signal is repeatedly acquired. The return unit generates display control commands for the stimulation guidance interface, causing the stimulation guidance interface to display the main interface.

[0039] Specifically, such as Figure 2 As shown, the AR guidance module displays a stimulus guidance interface. It first shows five flashing stimulus blocks on the main interface to induce SSVEP brainwave signals. The user focuses on one of the flashing blocks, sending the brainwave signal to the algorithm module for analysis and classification using the SSVEP algorithm. When the classification result is forward, backward, or stop, the algorithm module sends the result to the electric wheelchair, instructing it to perform the corresponding action and providing feedback to the stimulus guidance interface for display. Then, the next round of SSVEP signal induction begins. When the classification result is left or right, the algorithm module switches to the MI algorithm for analysis, and the stimulus guidance interface enters a secondary interface to guide the generation of MI signals. The user performs motor imagery according to the guidance. The MI algorithm in the algorithm module analyzes and classifies the brainwave signal until the motor imagery result matches the classification result (i.e., the classification result is correct). The algorithm module then sends a left or right turn command to the electric wheelchair and provides feedback to the stimulus guidance interface for display. The stimulus guidance interface then returns to the SSVEP main interface, and the algorithm module returns to the main thread of the SSVEP processing algorithm, starting the next round of SSVEP signal induction.

[0040] The algorithm module mainly consists of two parts: the first part is the FBCCA algorithm for analyzing SSVEP signals, and the second part is the CSP-LDA algorithm for analyzing MI signals. FBCCA is the main thread algorithm, which calculates five classification results based on the acquired EEG signals. When the SSVEP signal classification result is forward, backward, or stop, the classification result is directly sent to the electric wheelchair controller via serial port for control and is also displayed on the stimulation guidance interface. When the SSVEP signal classification result is left turn or right turn, it jumps to the CSP-LDA algorithm to analyze the MI EEG signal. The CSP-LDA algorithm will obtain two classification results: correct or incorrect. If the classification result is correct, the control command is sent to the electric wheelchair controller for control; if the classification result is incorrect, the motor imagery continues until the imagery is correct.

[0041] The specific process of decoding and analyzing SSVEP EEG signals using FBCCA is as follows:

[0042] Before classifying SSVEP EEG signals, preprocessing is necessary to reduce noise. Methods such as filtering, downsampling, and baseline drift removal can be used.

[0043] (1) Filtering: Since EEG signals are often distributed in the range of 1-100Hz, components that are too low or too high can be regarded as noise. The common method is to use bandpass filters and notch filters. Bandpass filters are mainly used to filter out high-frequency and low-frequency components contained in EEG signals, while notch filters are mainly used to filter out 50Hz power frequency interference.

[0044] (2) Removal of electrooculogram (EOG) artifacts: EOG artifacts are often present in MI EEG signals and are generated by the subject's blinking. They can be removed using the Independent Component Analysis (ICA) algorithm, which decomposes the raw EEG data into multiple independent components, thereby removing the electromyography / EOG artifact components of the EEG. However, specific parameters need to be adjusted; otherwise, components containing MI EEG signals may be filtered out.

[0045] (3) Downsampling: In some cases, the processor used has weak computing power, and an excessively high sampling rate can lead to excessively long calculation time or algorithm errors. In such cases, it is necessary to downsample the original EEG data to reduce the amount of data and computational complexity. However, it should be noted that excessively reducing the sampling rate can distort the signal and lose effective information.

[0046] The CCA algorithm is used to measure the correlation between two sets of variables. It can analyze the linear relationship between two groups of variables and give their correlation coefficient.

[0047] The FBCCA algorithm is an improved version of the CCA algorithm, characterized by high stability and accuracy. Currently, the FBCCA algorithm is commonly used in SSVEP EEG signal processing. This algorithm combines a filtering bank with the CCA algorithm, fully utilizing the harmonic characteristics of the SSVEP EEG signal, combining the fundamental and harmonic components to further improve the algorithm's accuracy. The FBCCA algorithm steps are as follows: Figure 3 As shown, the specific steps include:

[0048] (1) Construct the corresponding subband filter bank, filter the preprocessed SSVEP EEG signal using the filter bank, and obtain the subband components of the signal.

[0049] (2) Apply the CCA algorithm to each sub-band component to obtain the corresponding correlation coefficient.

[0050] (3) Multiply the squares of the correlation coefficients of each subband by the weights and add them together to obtain the correlation coefficients of the corresponding frequencies. Take the frequency with the largest correlation coefficient as the result.

[0051] The specific process of analyzing MI EEG signals using the CSP-LDA algorithm is as follows:

[0052] MI EEG signals contain a large amount of noise and artifacts. In order to improve the accuracy of classification, the raw signals must first undergo preprocessing operations similar to those used for SSVEP signals.

[0053] CSP is a feature extraction algorithm that essentially utilizes matrix diagonalization. It cannot be used alone and must be used in conjunction with a classification algorithm, such as Linear Discriminant Analysis (LDA) or Support Vector Machine (SVM). The main idea of ​​CSP is to construct a projection matrix to linearly transform multi-channel EEG signals, maximizing or minimizing the variance of the transformed signal in a specific direction. The extracted feature vectors are then used in a subsequent classifier for classification. The CSP algorithm extracts spatial domain features from MI (Minimum Injection) EEG signals.

[0054] Based on the modeling process of left and right hand MI EEG signals, the CSP algorithm is analyzed and its process is explained. Assume the data from a single trial is E... N*T Where N is the number of channels and T is the number of sampling points. The specific steps of CSP are as follows: Figure 4 As shown:

[0055] (1) First calculate the normalized covariance of a single trial data. trace(X) represents the trace of matrix X, which is the sum of the elements on the diagonal of the matrix.

[0056]

[0057] (2) Calculate the mean covariance C for the left and right hands respectively.l C r and the covariance of the mixed space C c :

[0058]

[0059] Among them, C c It is the average covariance matrix of the experiment.

[0060] (3) For the mean covariance matrix C c Perform eigenvalue decomposition:

[0061]

[0062] Among them, Λ c It is an eigenvalue diagonal matrix, U c It is the eigenvector matrix.

[0063] (4) Calculate the average covariance matrix C c Whitening matrix P:

[0064]

[0065] (5) For C l C r Whitening is performed to obtain the spatial coefficient matrix S. l S r :

[0066]

[0067] (6) For the spatial coefficient matrix S l S r Perform eigenvalue decomposition:

[0068]

[0069] (7) Calculate the spatial filter W:

[0070] W = (B T P) T (7)

[0071] (8) Use spatial filter W to filter E N*T Perform filtering:

[0072] Z N*T =W N*N E N*T (8)

[0073] (9) Calculate the feature vector f. The number of feature pairs of the feature vector f needs to be manually selected, and its maximum value cannot exceed the number of EEG channels N. In fact, when extracting the feature vector, the first m rows and the last m rows of Z are extracted (2m < N). m is the hyperparameter that needs to be manually adjusted:

[0074]

[0075] var(X) represents the calculation of the variance of sample X.

[0076] After CSP feature extraction, the final result is f = {f1, f2, ..., f...} 2m}, where m is the manually selected feature logarithm. The first m dimensions and the last m dimensions, if one is a maximum, the other is a minimum. The feature vectors can then be classified using algorithms such as LDA.

[0077] like Figure 5 As shown, in this embodiment, the AR guidance module specifically adopts an AR headset, including AR glasses; the electric wheelchair includes a wheelchair controller connected to the algorithm module via a serial port; the algorithm module is located in the host computer. The AR headset is used to display the stimulus guidance interface and induce EEG signals, the host computer is used for EEG signal analysis and running the stimulus guidance interface, and the wheelchair controller receives control commands transmitted by the host computer via a serial port to control the movement of the wheelchair.

[0078] When using the aforementioned wheelchair, the user wears an EEG cap. The stimulation-guided interface sends a start flag to the algorithm module to ensure synchronization, and begins to induce or guide the generation of real-time EEG signals. Simultaneously, the EEG acquisition module performs hardware tagging on the EEG signals according to the stimulation-guided interface, ensuring that the EEG signals are synchronized with the timestamps on the interface. The EEG acquisition module sends the acquired EEG signals to the algorithm module on the host computer via TCP for analysis. The analysis results are then sent to the stimulation-guided interface via UDP for feedback display, and simultaneously sent to the wheelchair controller via serial port for wheelchair movement control.

[0079] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An AR-based SSVEP-MI control and rehabilitation training wheelchair, including an electric wheelchair, characterized in that, Also includes: The EEG acquisition module is used to acquire SSVEP EEG signals and / or MI EEG signals in real time; The AR guidance module is used to display a stimulation guidance interface for inducing SSVEP EEG signals and guiding MI EEG signals; The algorithm module is used to acquire the SSVEP EEG signal and the MI EEG signal, identify the SSVEP EEG signal and the MI EEG signal respectively, generate display information based on the identification results, and generate display control commands for the stimulation guidance interface and / or motion control commands for the electric wheelchair. The stimulation guidance interface includes a main interface and a secondary interface. The main interface is the SSVEP stimulation interface, which includes five flashing stimulation blocks that represent forward, backward, left turn, right turn and stop respectively. The secondary interface is the MI guidance interface. The algorithm module includes: The SSVEP decoding unit is used to decode the SSVEP EEG signal using the FBCCA algorithm when the SSVEP EEG signal is received, and to obtain the SSVEP classification result. The judgment unit is used to judge the SSVEP classification result and execute corresponding actions based on the judgment result. Specifically, when the SSVEP classification result is forward, backward or stop, the corresponding display information and corresponding action control command for the electric wheelchair are generated at the same time. When the SSVEP classification result is left turn or right turn, the display control command for the stimulus guidance interface is generated, so that the stimulus guidance interface is displayed as a secondary interface. The MI decoding unit is used to decode the MI EEG signal using the CSP-LDA algorithm when the MI EEG signal is received, obtain the MI classification result, compare the MI classification result with the SSVEP classification result, and if they match, generate the corresponding display information and corresponding action control command; otherwise, the MI EEG signal is repeatedly acquired. The return unit is used to generate display control commands for the stimulus guidance interface, so that the stimulus guidance interface displays the main interface.

2. The AR-based SSVEP-MI control and rehabilitation training wheelchair according to claim 1, characterized in that, The EEG acquisition module includes a 64-moisture conductive electrode EEG cap.

3. The AR-based SSVEP-MI control and rehabilitation training wheelchair according to claim 1, characterized in that, The EEG acquisition module is connected to the algorithm module via TCP / IP communication.

4. The AR-based SSVEP-MI control and rehabilitation training wheelchair according to claim 1, characterized in that, The AR guidance module is connected to the algorithm module via UDP communication.

5. The AR-based SSVEP-MI control and rehabilitation training wheelchair according to claim 1, characterized in that, Each of the aforementioned flashing stimulus blocks flashes at a different frequency.

6. The AR-based SSVEP-MI control and rehabilitation training wheelchair according to claim 1, characterized in that, Both the SSVEP decoding unit and the MI decoding unit include a preprocessing subunit for signal preprocessing, which includes various methods such as filtering, downsampling, and removal of electrooculography artifacts.

7. The AR-based SSVEP-MI control and rehabilitation training wheelchair according to claim 1, characterized in that, The AR guidance module includes AR glasses.

8. The AR-based SSVEP-MI control and rehabilitation training wheelchair according to claim 1, characterized in that, The electric wheelchair includes a wheelchair controller that is connected to the algorithm module via a serial port.

Images