Multi-modal closed-loop rehabilitation training system based on brain-computer interface and eye movement

Abstract

The invention discloses a multi-modal closed-loop rehabilitation training system based on a brain-computer interface and eye movement, which comprises a stimulation unit used for providing sensory stimulation for a subject, and an acquisition unit used for acquiring an eye movement signal and an electroencephalogram signal of the subject and outputting the signals to a multi-modal information processing unit; the multi-modal information processing unit is used for identifying a brain control state and an idle state based on a preset machine learning algorithm according to the eye movement signal and the electroencephalogram signal so as to obtain a motion intention generated by the subject based on sensory stimulation, generating a control instruction based on the motion intention and sending the control instruction to the second stimulation unit; the second stimulation unit outputs mechanical stimulation to the subject based on the control instruction. According to the system, high-accuracy and low-delay brain-machine switch identification is realized based on eye movement information, and high-accuracy intention identification of different actions is realized based on visual evoked potential, so that collaborative feedback is performed on a subject based on the second stimulation unit to establish a closed-loop system, the degree of participation of the subject is improved, and neural recombination is promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of brain-computer interface, eye movement and rehabilitation training, and in particular to a multi-modal closed-loop rehabilitation training system based on brain-computer interface and eye movement. Background Art

[0002] With the aging process, the number of patients with brain injury and spinal cord injury increases year by year. After being out of danger, patients will have varying degrees of motor, speech and other functional disorders, which brings a huge burden to their families and society. Early motor function rehabilitation training is of great significance to the functional recovery of patients.

[0003] At present, motor function rehabilitation training is assisted by rehabilitation therapists to perform limb movements, which is a heavy task and the rehabilitation effect is average; the emergence of rehabilitation robots has relieved some of the pressure of rehabilitation therapists, but the rehabilitation effect has not been significantly improved. The reason is that the existing motor function rehabilitation training is mainly passive training, and the patient's active participation is low, which makes it difficult to effectively promote neural reorganization.

[0004] The brain-computer interface (BCI) can achieve direct information interaction between the brain and external devices or computers without relying on human neuromuscular channels, providing a new idea for motor function rehabilitation. Among them, the brain-computer interface technology based on motor imagery is widely used in the field of rehabilitation: patients actively imagine movements of their left or right hands or feet according to visual cues, the system collects the patient's EEG signals in real time, extracts and recognizes features, and sends control instructions to the rehabilitation robot or electrical stimulation device based on the recognition results to give the patient limb movement sensory feedback. This technology simulates the process of normal human limb movement for motor relearning, integrating the patient's active movement intention in order to promote neural plasticity. However, there are large individual differences in motor imagery ability, a long training cycle, general recognition of EEG features, and it is difficult to meet the timeliness requirements of neural plasticity, resulting in large differences in the improvement of rehabilitation effects. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the brain-computer interface-based training systems in the prior art, such as poor recognition of EEG features, difficulty in meeting the timeliness requirements of neural plasticity, and poor training effects, and to provide a multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement.

[0006] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0007] A multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement, comprising: a first stimulation unit, a collection unit, a multimodal information processing unit and a second stimulation unit;

[0008] Wherein, the signal output end of the first stimulation unit is connected to the signal input end of the acquisition unit, the signal output end of the acquisition unit is connected to the signal input end of the multimodal information processing unit, the signal output end of the multimodal information processing unit is connected to the signal input end of the second stimulation unit, and the signal output end of the second stimulation unit is connected to the signal input end of the stimulation unit;

[0009] The first stimulation unit is used to provide sensory stimulation to the subject, and the sensory stimulation includes: visual and / or auditory and / or tactile stimulation;

[0010] The acquisition unit includes: an EEG acquisition module and an eye movement acquisition module, which are used to acquire eye movement signals and EEG signals generated by the subject based on the sensory stimulation, and output them to the multimodal information processing unit;

[0011] The multimodal information processing unit is used to identify the brain-controlled state and the idle state according to the eye movement signal and the electroencephalogram signal based on a preset machine learning algorithm, so as to obtain the subject's movement intention based on the sensory stimulation, and to generate a control instruction based on the movement intention and send it to the second stimulation unit; the second stimulation unit is used to output mechanical stimulation to the subject based on the control instruction.

[0012] According to a specific implementation, in the above-mentioned multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement, the stimulation unit includes: a visual stimulation module and an auditory stimulation module;

[0013] The visual stimulation module includes: a display or an AR module;

[0014] The auditory stimulation module includes: headphones or speakers.

[0015] According to a specific implementation method, in the above-mentioned multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement, different motion observation paradigms in the visual stimulation module have different frame rates.

[0016] According to a specific implementation method, in the above-mentioned multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement, the eye movement acquisition module is a non-invasive binocular movement tracking sensor, which is used to collect in real time the eye coordinate position and time of the subject's gaze under the visual stimulation provided by the visual stimulation module, as well as the pupil diameter. The sampling frequency of the eye movement acquisition module is 1200Hz.

[0017] According to a specific implementation method, in the above-mentioned multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement, the sampling frequency of the EEG acquisition module exceeds 250 Hz.

[0018] According to a specific implementation, in the above-mentioned multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement, the second stimulation unit includes: an electrical stimulation device and a mechanical stimulation device;

[0019] Electrical stimulation includes: neuromuscular electrical stimulation and functional electrical stimulation; mechanical devices include: end effectors, exoskeleton robots, and pneumatic robots.

[0020] According to a specific implementation, in the above-mentioned multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement, the multimodal information processing unit is configured as:

[0021] Pre-delineating a limb movement area with a fixed movement frequency in the first stimulation module as a region of interest;

[0022] Receive eye movement signals and EEG signals, obtain the position and duration of the subject's gaze on the area of ​​interest in real time based on the eye movement signals, and determine whether the obtained position and duration are greater than the preset eye movement threshold based on a preset period. If not, it is in an idle state; if so, it enters a brain control state; receive the EEG signals, perform bandpass filtering on the EEG signals, and process the EEG signals in real time based on canonical correlation analysis, extended canonical correlation analysis, task-related component analysis, task discrimination pattern analysis, and convolutional neural network algorithms to determine which visual stimulation frame rate component is contained in the EEG signal: if the action observation paradigm corresponding to the determined frame rate component is consistent with the limb movement marked by the AOI, then it is determined that the subject's movement intention at this time is the action in the movement observation paradigm, and the corresponding instruction is sent to the external feedback device control module; if the action observation paradigm corresponding to the determined frame rate component is inconsistent with the limb movement marked by the AOI, then it is determined that the subject is in a state of non-concentration, and the system continues to perform the above judgment; if the EEG recognition results are different from the eye movement AOI mark for three consecutive times, the auditory stimulation module provides auditory stimulation to guide the subject to observe the action when the visual stimulation module presents the visual stimulation.

[0023] According to a specific implementation method, in the above-mentioned multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement, the eye movement threshold is determined by a working characteristic curve.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The system provided in the embodiment of the present invention collects the eye movement information and EEG information of the subject under the sensory stimulation provided by the first stimulation unit through the collection unit, realizes the "brain-computer switch" recognition based on the eye movement information, and at the same time cooperates with the preset EEG signal recognition algorithm to perform high-accuracy and low-latency recognition of the EEG signal, recognizes the subject's action intention, and based on the recognized action intention, controls the second stimulation module to provide electrical stimulation and physical movement of the mechanical device to the subject, thereby realizing the coordinated training of the subject's vision, hearing, and touch, establishing a closed-loop system, improving the subject's active participation, and promoting neural reorganization and reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement as described in one embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention is further described in detail below in conjunction with test examples and specific implementation methods. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0028] Example 1

[0029] Figure 1 A multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement according to an exemplary embodiment of the present invention is shown, comprising: a first stimulation unit, a collection unit, a multimodal information processing unit and a second stimulation unit;

[0030] Wherein, the signal output end of the first stimulation unit is connected to the signal input end of the acquisition unit, the signal output end of the acquisition unit is connected to the signal input end of the multimodal information processing unit, the signal output end of the multimodal information processing unit is connected to the signal input end of the second stimulation unit, and the signal output end of the second stimulation unit is connected to the signal input end of the stimulation unit;

[0031] The first stimulation unit is used to provide sensory stimulation to the subject, and the sensory stimulation includes: visual and / or auditory and / or tactile stimulation;

[0032] The acquisition unit includes: an EEG acquisition module and an eye movement acquisition module, which are used to acquire eye movement signals and EEG signals generated by the subject based on the sensory stimulation, and output them to the multimodal information processing unit;

[0033] The multimodal information processing unit is used to identify the brain-controlled state and the idle state according to the eye movement signal and the electroencephalogram signal based on a preset machine learning algorithm, so as to obtain the subject's movement intention based on the sensory stimulation, and to generate a control instruction based on the movement intention and send it to the second stimulation unit; the second stimulation unit is used to output mechanical stimulation to the subject based on the control instruction.

[0034] In this embodiment, highly accurate and low-latency "brain-computer switch" recognition is achieved based on eye movement information, and highly accurate intention recognition of different actions is achieved based on visual evoked potentials; the control mode of the external feedback module can provide the subjects with coordinated feedback of electrical stimulation and physical movement of the mechanical device, establish a closed-loop system, improve the subjects' active participation, and promote neural reorganization and reconstruction.

[0035] Example 2

[0036] Reference Figure 1 A multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement includes a first stimulation unit (including: a visual stimulation module, an auditory stimulation module), a first output end of a collection unit (including: an eye movement collection module and an electroencephalogram collection module) is connected to an input end of a multimodal information processing unit, a first output end of the multimodal information processing unit is connected to an input end of a second stimulation unit (including: an electrical stimulation device and a mechanical device), and a second output end of the multimodal information processing unit is connected to a first input end of the first stimulation unit.

[0037] The first stimulation unit includes a visual stimulation module presented by a display or augmented reality (AR) glasses, an auditory stimulation module provided by headphones or speakers, and a tactile stimulation module provided by a vibrator; wherein different motion observation paradigms in the visual stimulation module have different frame rates, such as: the frame rate of the left hand grasping movement is 6Hz, the frame rate of the left arm forward stretching movement is 8.57Hz, the frame rate of the left arm upward stretching movement is 10Hz, the frame rate of the left arm abduction movement is 15Hz, etc.; if the EEG signal recognition result based on the visual area is different from the eye movement recognition result for three consecutive times, an auditory stimulation matching the visual stimulation is added; if the event-related desynchronization response of the motor area of ​​the brain is lower than the set threshold after ten minutes of training, a tactile stimulation is added to the end of the limb.

[0038] The eye movement acquisition module realizes binocular tracking by binocular tracking sensors, realizes natural interaction between the subject and the visual stimulation in a non-invasive manner, and collects the eyeball coordinate position and time when the subject is looking at the visual stimulation provided by the visual stimulation module, as well as the pupil diameter in real time, with a sampling frequency of 1200Hz;

[0039] The EEG signal acquisition module is in accordance with the international standard 10 / 20 system, with the forehead Fpz as the ground pole and the left earlobe as the reference, to collect the EEG signals of the subject's Fp1, Fp2, AFz, F7, F3, Fz, F4, F8, FC5, FC1, FC2, FC4, T7, C3, Cz, C4, T8, CP5, CP1, CP2, CP6, P3, Pz, P4, PO7, PO3, POz, PO4, PO8, O1, Oz and O2, a total of 32 channels distributed on the scalp surface of the frontal lobe, parietal lobe, occipital lobe, temporal lobe, etc., with a sampling frequency of 250 Hz or above;

[0040] The multimodal information processing unit includes an eye movement signal processing module and an electroencephalogram signal processing module;

[0041] The specific multimodal information processing process is:

[0042] First, the limb movement area with a fixed movement frequency in the visual stimulation module is delineated as the area of ​​interest (AOI). Then the eye movement information processing module records the time of the first gaze at the AOI and calculates the position and duration of the gaze point in real time. When the position and duration of the gaze point are greater than the preset thresholds (Threshold 1 and Threshold 2), the system determines that the subject enters the brain-controlled state, otherwise it is in the idle state. The time interval between each judgment is less than 100 milliseconds; Threshold 1 and Threshold 2 are determined by the receiver operating characteristic curve (ROC curve).

[0043] When it is determined that the subject enters the brain control state, the EEG signal processing module performs bandpass filtering (3Hz to 40Hz) on the EEG signals of the visual areas (PO7, PO3, POz, PO4, PO8, O1, Oz and O2) of the brain based on canonical correlation analysis (CCA), extended canonical correlation analysis (eCCA), task-related component analysis (TRCA), task-discriminant component analysis (TDCA), Or convolutional neural network (CNN) and other algorithms for real-time processing to determine which visual stimulus frame rate component is contained in the EEG signal: if the action observation paradigm corresponding to the determined frame rate component is consistent with the limb movement marked by AOI, the subject's movement intention at this time is determined to be the action in the movement observation paradigm, and the corresponding instruction is sent to the external feedback device control module; if the action observation paradigm corresponding to the determined frame rate component is inconsistent with the limb movement marked by AOI, the subject is determined to be in a state of non-concentration, and the system continues to make the above judgments; if the EEG recognition results are different from the eye movement AOI mark for three consecutive times, the auditory stimulation module provides auditory stimulation to guide the subject to observe the action when the visual stimulation module presents the visual stimulation;

[0044] The second stimulation unit has been listening to whether there are any control instructions sent in the TCP / IP communication or serial communication channel. When receiving the control instructions, it controls the electrical stimulation device and the mechanical device to output and move according to the pre-set parameters. The electrical stimulation device takes neuromuscular electrical stimulation as an example, and adopts kilohertz high-frequency electrical stimulation - modulated by a low-frequency mode, wherein the modulation frequency is 40Hz, the carrier frequency is 5kHz, the single pulse width is 100μs, and the bidirectional square wave. By keeping the frequency and pulse width unchanged and changing the intensity of the stimulation current, induced movement based on electrical stimulation is achieved. Taking the upper limb exoskeleton robot arm as an example of the mechanical device, the device performs impedance control after receiving the control instructions, and uses the interaction force between the upper arm and the mechanical arm and the current intensity as input to control the movement of the mechanical arm to drive the upper arm to complete the predetermined action, such as the left arm stretching forward, the left arm stretching upward, etc., to ensure the synergistic effect of electrical stimulation and the mechanical device.

[0045] In a possible implementation, the multimodal information processing unit is an electronic device (which may be a computer, a tablet, a PC, etc.), which includes a processor, a network interface, and a memory, wherein the processor, the network interface, and the memory are interconnected, wherein the memory is used to store a computer program, and the computer program includes program instructions, and the processor is configured to call the program instructions to execute the above-mentioned cutting and fusion method.

[0046] In the embodiments of the present invention, the processor may be an integrated circuit chip having the ability to process signals. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0047] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0048] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0049] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0050] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus DRAM (DRRAM).

[0051] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0052] It should be understood that the system disclosed in the present invention can be implemented in other ways. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the communication connection between modules can be through some interfaces, indirect coupling or communication connection of servers or units, which can be electrical or other forms.

[0053] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one processing unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional unit.

[0054] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement, characterized in that: include: A first stimulation unit, a collection unit, a multimodal information processing unit, and a second stimulation unit; Wherein, the signal output end of the first stimulation unit is connected to the signal input end of the acquisition unit, the signal output end of the acquisition unit is connected to the signal input end of the multimodal information processing unit, the signal output end of the multimodal information processing unit is connected to the signal input end of the second stimulation unit, and the signal output end of the second stimulation unit is connected to the signal input end of the stimulation unit; The first stimulation unit is used to provide sensory stimulation to the subject, and the sensory stimulation includes: visual and / or auditory stimulation; The acquisition unit includes: an EEG acquisition module and an eye movement acquisition module, which are used to acquire eye movement signals and EEG signals generated by the subject based on the sensory stimulation, and output them to the multimodal information processing unit; The multimodal information processing unit is used to identify the brain-controlled state and the idle state according to the eye movement signal and the electroencephalogram signal based on a preset algorithm, so as to obtain the subject's movement intention based on the sensory stimulation, and to generate a control instruction based on the movement intention and send it to the second stimulation unit; the second stimulation unit is used to output mechanical stimulation to the subject based on the control instruction.

2. The multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement according to claim 1 is characterized in that: The stimulation unit includes: a visual stimulation module and an auditory stimulation module; The visual stimulation module includes: a display or an AR module; The auditory stimulation module includes: headphones or speakers.

3. The multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement according to claim 2 is characterized in that: Different motion observation paradigms in the visual stimulation module have different frame rates.

4. The multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement according to claim 1, characterized in that: The eye movement acquisition module is a non-invasive binocular tracking sensor, which is used to collect in real time the eye coordinate position and time of the subject's gaze under the visual stimulation provided by the visual stimulation module, as well as the pupil diameter. The sampling frequency of the eye movement acquisition module is 1200Hz.

5. The multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement according to claim 1 is characterized in that: The sampling frequency of the EEG acquisition module exceeds 250 Hz.

6. The multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement according to claim 1 is characterized in that: The second stimulation unit includes: an electrical stimulation device and a mechanical stimulation device; Electrical stimulation includes: neuromuscular electrical stimulation and functional electrical stimulation; mechanical devices include: end effectors, exoskeleton robots, and pneumatic robots.

7. The multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement according to any one of claims 1 to 6, characterized in that: The multimodal information processing unit is configured as follows: Pre-delineating a limb movement area with a fixed movement frequency in the first stimulation module as a region of interest; Receive eye movement signals and EEG signals, obtain the position and duration of the subject's gaze on the area of ​​interest in real time based on the eye movement signals, and determine whether the obtained position and duration are greater than a preset eye movement threshold based on a preset period. If not, it is in an idle state; if so, it enters a brain control state; The EEG signal is received, band-pass filtered, and the EEG signal is processed in real time based on a preset algorithm to determine which frame rate component of visual stimulation is contained in the EEG signal: if the action observation paradigm corresponding to the determined frame rate component is consistent with the limb movement marked by the AOI, the subject's movement intention at this time is determined to be the action in the action observation paradigm, and a corresponding instruction is sent to the external feedback device control module; if the action observation paradigm corresponding to the determined frame rate component is inconsistent with the limb movement marked by the AOI, the subject is determined to be in a state of non-concentration, and the system continues to perform the above judgment; if the EEG recognition results are different from the eye movement AOI mark for three consecutive times, the first stimulation module is feedback-controlled to send out the visual stimulation module and the auditory stimulation module to guide the subject to observe the action.

8. The multimodal closed-loop rehabilitation training system based on brain-computer interface and eye movement according to claim 7, characterized in that: The eye movement threshold is determined by a working characteristic curve.

Images