Asynchronous robot hybrid control method and system based on motor imagery electroencephalogram and electro-oculogram

By combining the asynchronous control method of electroencephalogram and electroophthalmic signals of motor imagination, the problem of insufficient multi-command and stability in the brain control system is solved, asynchronous multi-command control is realized, and the user's operation comfort and system stability are improved.

CN119088220AActive Publication Date: 2024-12-06SUZHOU R&D CENT OF NO 214 RES INST OF CHINA NORTH IND GRP
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411191821.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-12-06
Estimated Expiration
2044-08-28

AI Technical Summary

Technical Problem

The existing brain-controlled robot systems lack multi-command control and stability when used, resulting in inconvenience in user operation and low system response rate. Especially in complex scenarios, a single fixed control method is difficult to meet the needs.

Method used

Adopting a hybrid control method of asynchronous robots based on motion imagination EEG and EEG, by acquiring and preprocessing the user's motion imagine EEG and EEG signals, combining multi-scale global convolutional neural network and multi-threshold detection function to realize asynchronous control , add 'baseline' category to relieve user burden and convert signals into robot control commands.

Benefits of technology

Multi-command stable control is realized, improving the comfort of human-computer interaction and system stability. Users can decide independently when to control the robot, reducing the false touch rate and improving the response rate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088220A_ABST
    Figure CN119088220A_ABST
Patent Text Reader

Abstract

The invention discloses an asynchronous robot hybrid control method and system based on motor imagery electroencephalogram and electro-oculogram. The system comprises a signal acquisition module, a signal preprocessing and decoding module and a robot control module. Firstly, signals of a subject are collected through electroencephalogram equipment, the signals comprise electroencephalogram signals and electro-oculogram signals, and the electro-oculogram signals are added to enrich a control instruction set of a system. Then, the collected signals are processed in two stages: in the first stage, signal preprocessing is carried out, and original signals are normalized into required forms; and the second stage is signal decoding which comprises two branches of electroencephalogram signal identification and electro-oculogram signal identification, and an identification result is converted into a robot control command. And finally, the robot device obtains the recognition result of the signal through a wireless data receiving module and then converts the recognition result into a control instruction to control the robot to make corresponding actions, and the next command is guided to be sent according to feedback of the actions of the robot, so that the comfort of man-machine interaction is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an asynchronous robot hybrid control method and system based on motor imagery electroencephalogram and electrooculogram, belonging to the field of robot control. Background Art

[0002] As a technology for direct interaction between the human brain and the machine, the brain-computer interface (BCI) can directly output instructions from the brain to the machine without the assistance of the muscle system, so that the elderly and the disabled can independently control external devices to assist in daily activities without the help of limbs or others, such as controlling a robotic arm to grab objects and controlling the movement of a wheelchair. The emergence of BCI technology provides a new way of interaction between people and machines, which is of great significance to improving the quality of life of the elderly and the disabled. However, the current brain-controlled external device system does not fully consider the user's comfort and the stability of the system. Previous studies, such as brain control systems based on steady-state visual evoked potentials and brain control systems based on visual P300, all require visual stimulation to induce EEG signals, which greatly limits the portability and natural interaction of BCI applications. Although the brain control system based on motor imagery does not require visual stimulation, current studies use synchronous control to enable users to perform imagination tasks at a fixed time. This method reduces the naturalness of human-computer interaction, and the existing research has few control instructions, and the stability of the system has not been considered.

[0003] Therefore, there is an urgent need for a multi-instruction asynchronous and stable brain-controlled external device method, so that users can control the robot to perform tasks or put it on standby at any time according to their needs when using the brain control system. At the same time, the brain control system has good signal recognition performance and high stability, ensuring a high response rate and low false touch rate during task execution.

[0004] The shortcomings of current technology mainly lie in the way the human brain interacts with external devices and the way brain-controlled systems are controlled.

[0005] At present, the control method of brain-controlled robot systems based on motor imagery is single, and the comfort of users using the system is not fully considered. Especially in complex real-life scenarios, the single fixed control method will inevitably bring about the problem of poor operability. Summary of the invention

[0006] The present invention provides an asynchronous robot hybrid control method and system based on motor imagery electroencephalogram and electrooculogram, which solves the problems disclosed in the background technology. The system is equipped with multi-command control, and by introducing a "baseline" class (signal when motor imagery and eye movement are not performed), the user can independently decide whether to perform motor imagery or eye movement tasks, so as to realize asynchronous control and reduce the burden on the user.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: an asynchronous robot hybrid control method based on motor imagery EEG and EOG, comprising:

[0008] Obtain the user's motor imagery EEG signals and EOG signals;

[0009] Preprocessing the motor imagery EEG signal and the electrooculogram signal;

[0010] The preprocessed motor imagery EEG signal is input into a pre-built EEG recognition model to obtain the user's motor imagery category;

[0011] The preprocessed electrooculogram signal is input into a pre-built electrooculogram recognition model to identify the user's blinking or non-blinking action;

[0012] Converting user movement imagery categories and blinking or non-blinking actions into corresponding robot control commands;

[0013] Control the robot according to the corresponding robot control commands.

[0014] Furthermore, the method for preprocessing the motor imagery EEG signal is as follows:

[0015] Downsampling the EEG signal to 250 Hz, and filtering the EEG signal by reducing the sampling points, including 0.5 Hz-100 Hz bandpass filtering and 50 Hz notch filtering;

[0016] The filtered EEG signal is segmented and the preset time window in the motor imagery stage is divided into the task class and the preset time window in the rest stage is divided into the non-task class.

[0017] Furthermore, the method for preprocessing the electrooculogram signal is:

[0018] The electrooculogram signal is downsampled to 62.5 Hz, and a 0.5 Hz-30 Hz bandpass filter and a 50 Hz notch filter are performed on the electrooculogram signal; segmented processing is performed, and the preset time window of the blinking stage is divided as a blink class.

[0019] Furthermore, the band-pass filtering adopts a Butterworth filter, which is represented by the following formula of square of amplitude to frequency:

[0020]

[0021] in, is the filter order, set , that is, the fourth-order Butterworth filter, is the cut-off frequency, is the passband edge frequency, It is a parameter that controls the degree of filter passband fluctuation.

[0022] Furthermore, the notch filter adopts a second-order infinite impulse response filter, whose amplitude response is zero at a certain frequency, and is used to eliminate a certain frequency component, and its function is:

[0023]

[0024] in, is the notch digital frequency (rad), is the notch frequency (Hz), is the sampling frequency (Hz).

[0025] Furthermore, the method of constructing the EEG recognition model is:

[0026] A multi-scale global convolutional neural network is used. The multi-scale global convolutional neural network is set as a one-dimensional convolutional neural network. The convolution kernel slides only in the time dimension. The network architecture of the multi-scale global convolutional neural network includes a multi-scale convolutional layer and two global convolutional layers. After receiving the input, the multi-scale convolutional layer and the global convolutional layer extract the EEG signal features in parallel. After the convolution pooling operation, two feature vectors are obtained. The two feature vectors are connected and fused and input into the fully connected layer to complete the final classification.

[0027] The multi-scale convolution layer is used to capture more local feature information in the EEG signal by setting three different kernel sizes. The convolution of is the time window size covered by the convolution, is the number of channels covered by the convolution; extract multi-scale features:

[0028]

[0029] in, is the result of the interaction between the convolution kernel and the data. is the convolution kernel weight, is the data value, is the bias term, is the activation function;

[0030] The global convolution layer is used to extract the global features of the EEG signal. By building an expanded convolution layer, it has a larger receptive field. ,in is the convolution kernel size, is the expansion rate.

[0031] Furthermore, the method of constructing the electrooculographic recognition model is:

[0032] A multi-threshold detection function is used. The function design is based on three rapid blinks, including: peak value, peak position, trough value, trough position, and interval time. Conscious blinking has a higher peak value than unconscious blinking. By setting the peak threshold To distinguish between them; between two active blinks of the user, there is often a trough, and the interval time between the peak position and the trough position is calculated. , to determine whether the two blinks are continuous actions, in order to distinguish two slow blinks; for three fast blinks, it is necessary to determine whether there are two blinks within the interval threshold. within , exceeding In range Treated as non-continuous action; by setting the trough threshold Limit the recognition range of the trough; the calculation process of the multi-threshold detection function:

[0033]

[0034] in, It is 0 or 1, representing the non-blinking or blinking category. is the number of peaks, is the number of troughs, is the number of changes from peak to trough.

[0035] Further, the method of converting the user's motor imagery category and blinking or non-blinking action into a corresponding robot control command includes:

[0036] If the recognition result is a blink, the process enters the electrooculogram command determination phase; otherwise, the process enters the EEG signal recognition phase. In the electrooculogram command determination phase, the corresponding command is issued according to the robot state: if the robot is in state 1 and , a command to start the brain-controlled robot system is issued; if the robot is in state 1 and Or if the robot is in state 2, a command to shut down the brain-controlled robot system is issued; if the robot is in state 3, a control conversion command is issued, converting the robot movement control to the robot arm control; is a flag variable, indicating the current state of the system: 0 indicates that the system is not started, and 1 indicates that the system is running; state 1 is the initial state of the robot, state 3 is the state when the robot reaches the target grasping location, and the rest of the time is in state 2; EEG signal recognition stage: first determine whether the robot is in state 2; if the robot is in state 2, identify the EEG signal and send the corresponding control command to the robot according to the recognition result; otherwise, continue to perform EOG signal recognition.

[0037] Further, the method of converting the motor imagery category into the corresponding robot control command includes a robot chassis control stage and a robot arm grasping control stage;

[0038] The robot chassis control stage includes: imagining the left hand movement to control the robot to turn to the left target grasping location; imagining the right hand movement to control the robot to turn to the right target grasping location; imagining the foot movement to control the robot to go to the target grasping location; imagining the tongue movement to control the robot to return to the initial position; if the robot is in the turning stage, the foot movement and tongue movement categories are invalid; otherwise, the left hand movement and right hand movement categories are invalid;

[0039] The robotic arm grasping control stage includes: imagining the left hand movement to control the robotic arm to move one position to the left; imagining the right hand movement to control the robotic arm to move one position to the right; imagining the foot movement to control the robotic arm to grasp an object; and imagining the tongue movement to control the robotic arm to put back the grasped object.

[0040] Furthermore, the method of converting a blinking or non-blinking action into a corresponding robot control command includes: converting only the blinking action into an actual control command; when the robot is in state 1 and When the blinking control brain-controlled robot system starts; when the robot is in state 1 and , blinking controls the brain-controlled robot system to shut down; when the robot is in state 2, the robot must first cancel the current action and return to the initial position, and then shut down the system; when the robot is in state 3, blinking controls the movement imagination EEG signal to control the robotic arm.

[0041] Accordingly, the present invention also provides an asynchronous robot hybrid control system based on motor imagery electroencephalogram and electrooculogram, comprising:

[0042] Signal acquisition module, which acquires the user's motor imagery EEG signal and eye contact signal;

[0043] A signal preprocessing module, used for preprocessing the motor imagery EEG signal and the electrooculogram signal;

[0044] The signal decoding module is used to input the preprocessed motor imagery EEG signal into a pre-built EEG recognition model to obtain the user's motor imagery category; input the preprocessed electrooculogram signal into a pre-built electrooculogram recognition model to identify the user's blinking or non-blinking action;

[0045] A command conversion module converts the user's motor imagery categories and blinking or non-blinking actions into corresponding robot control commands;

[0046] The robot control module is used to control the robot according to corresponding robot control commands.

[0047] The beneficial effects achieved by the present invention are:

[0048] The present invention proposes a multi-scale global convolutional neural network and a multi-threshold detection function decoding method for motor imagery EEG and EOG signals, and integrates motor imagery EEG and EOG signals to achieve asynchronous, multi-command stable control of an external robot.

[0049] The present invention combines motor imagery EEG and EOG to enrich the control instruction set. The newly added "baseline" class (non-task class) assists in realizing the asynchronous control of the robot, which can improve the comfort of human-machine interaction compared with the previous synchronous control method;

[0050] The present invention decodes signals in an asynchronous manner, allowing the user to independently decide when to control the robot to operate, and adopts a category high threshold setting to improve the stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of an asynchronous robot hybrid control system based on motor imagery EEG and EOG of the present invention;

[0052] Figure 2 The present invention is a flow chart of the asynchronous robot hybrid control method based on motor imagery EEG and EOG. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0054] Embodiment 1:

[0055] like Figure 1 As shown, this embodiment provides an asynchronous robot hybrid control method based on motor imagery EEG and EOG, comprising the following steps:

[0056] Step 1: Obtaining EEG and EOG signals by using EEG and EOG signal acquisition equipment;

[0057] Step 2: According to the signal characteristics and task requirements, perform signal clipping, downsampling, filtering, and noise reduction operations on the EEG and EOG signals;

[0058] Step 3: Input the EEG and EOG data obtained by the preprocessing in step 2 into their respective recognition methods, decode the signal categories, and convert them into corresponding robot control commands.

[0059] Step 4: Combine the signal decoding in step 3 and formulate a signal decoding process to ensure the asynchrony and stability of the robot control system.

[0060] The specific process of step one is as follows:

[0061] The present invention adopts NeuSen W, the latest wireless EEG device developed by Boricon, to collect EEG and EOG signals, and selects 19 channels including FC3, FC1, FCz, FC2, FC4, C5, C3, C1, Cz, C2, C4, C6, CP3, CP1, CP2, CP4, P3, Pz, and P4 as EEG signal recording electrodes, and selects VEOU as the EOG signal recording electrode, with a sampling rate of 1000 Hz.

[0062] The specific process of step 2 is as follows:

[0063] 1. Preprocessing of EEG signals for motor imagery

[0064] First, the collected EEG signal is downsampled to 250Hz to reduce the amount of calculation and improve the efficiency of the algorithm. And by reducing the sampling points, the overall shape of the signal is smoother, so that some slight fluctuations in the signal will not have a significant impact on the overall characteristics. Then, the EEG signal is filtered, including 0.5Hz-100Hz bandpass filtering and 50Hz notch filtering. The bandpass filter uses the Butterworth filter, which is a smooth filter characterized by no obvious ripples between the passband and the stopband. The design of this filter is achieved by mapping the frequency response function to the plane of poles and zeros. The Butterworth filter can be expressed by the following formula of amplitude squared to frequency:

[0065]

[0066] in, is the order of the filter. This study sets , that is, the fourth-order Butterworth filter, is the cut-off frequency, is the passband edge frequency, It is a parameter that controls the degree of filter passband fluctuation.

[0067] The notch filter uses a simple second-order infinite impulse response (IIR) filter whose amplitude response is zero at a certain frequency and can be used to eliminate a certain frequency component. Its system function is:

[0068]

[0069] in, is the notch digital frequency (rad), is the notch frequency (Hz), is the sampling frequency (Hz).

[0070] Finally, the EEG data were segmented and the 3-s time window during the motor imagery phase was divided into the task class and the 3-s time window during the rest phase was divided into the “baseline” class (non-task class).

[0071] 2. Electrooculogram signal preprocessing

[0072] The electrooculogram signal is a physiological signal that records eye movement. Since blinking produces a large peak, a single electrode close to the eye can be used to detect the electrooculogram signal directly. To maintain the consistency of signal preprocessing, the electrooculogram signal preprocessing process is the same as that of the motor imagery EEG signal. First, the electrooculogram signal is downsampled to 62.5Hz. The sampling rate is lower than that of the EEG signal in order to reduce the irrelevant fluctuations caused by the high-frequency components in the signal, make the electrooculogram signal waveform smoother, and improve the robustness of the detection algorithm. Then, the electrooculogram signal is bandpass filtered at 0.5Hz-30Hz and notch filtered at 50Hz. Finally, segmented processing is performed to divide the 1.5s time window in the blinking stage as the blinking class.

[0073] The specific process of step three is as follows:

[0074] After signal preprocessing, it needs to be further input into the decoding module for recognition. To this end, a multi-scale global convolutional neural network (MGCNN) is constructed for motor imagery EEG classification, and a multi-threshold detection function (MTDF) is designed for eye movement recognition.

[0075] 1. Multi-scale global convolutional neural network

[0076] In order to enable the network model to learn multi-level and comprehensive EEG signal feature information, a multi-scale global convolutional neural network was constructed. At the same time, since the EEG signal features are mainly reflected in the time dimension, the multi-scale global convolutional neural network is set as a one-dimensional convolutional neural network, and the convolution kernel only slides in the time dimension to maintain the integrity of the time domain features. The network architecture of the multi-scale global convolutional neural network includes a multi-scale convolutional layer and two global convolutional layers. After receiving the input, the multi-scale convolutional layer and the global convolutional layer extract the EEG signal features in parallel, and after the convolution pooling operation, two feature vectors are obtained. The two feature vectors are connected and fused and input into the fully connected layer to complete the final classification.

[0077] The multi-scale convolution layer is used to capture more local feature information in the EEG signal by setting three different kernel sizes. The convolution of is the time window size covered by the convolution, is the number of channels covered by the convolution) to extract multi-scale features:

[0078]

[0079] in, is the result of the interaction between the convolution kernel and the data. is the convolution kernel weight, is the data value, is the bias term, is the activation function.

[0080] The global convolution layer is used to extract the global features of the EEG signal. By building an expanded convolution layer, it has a larger receptive field. ,in is the convolution kernel size, is the expansion rate.

[0081] 2. Multi-threshold detection function

[0082] The function design is based on three rapid eye blinks, including: peak value, peak position, trough value, trough position, and interval time. Conscious blinking has a higher peak value than unconscious blinking. By setting the peak threshold Between two active blinks of the user, there is often a trough, and the interval between the peak position and the trough position is calculated. , to determine whether the two blinks are continuous actions, in order to distinguish two slow blinks. For three fast blinks, it is necessary to determine whether there are two blinks within the interval threshold. within , exceeding In range In addition, to avoid the interference of high-amplitude troughs, the trough threshold is set Limit the recognition range of the trough. The calculation process of the multi-threshold detection function:

[0083]

[0084] in, It is 0 or 1, representing the non-blinking or blinking category. is the number of peaks, is the number of troughs, is the number of changes from peak to trough.

[0085] The specific process of step 4 is as follows:

[0086] The robot used is the Alpha Pro mobile grasping robot, which is configured with key components including a mobile chassis, collaborative robotic arm, large gripper, Realsense camera, and router antenna.

[0087] like Figure 2As shown in the figure, the robot's one-action decoding process consists of three modules: Module 1 is signal acquisition and preprocessing; Module 2 is EOG signal recognition; Module 3 is EEG signal recognition. First, module 1 collects signals in real time and performs preprocessing operations to obtain single-channel EOG signals and three-channel EEG signals. Then, MTDF is called to identify EOG signals: if the recognition result is blinking, the EOG command determination stage is entered; otherwise, the EEG signal recognition stage is entered. In the EOG command determination stage, the corresponding command is issued according to the robot state: if it is in state 1 and , then issue a command to start the brain-controlled robot system; if it is in state 1 and Or if it is in state 2, a command to shut down the brain-controlled robot system is issued; if it is in state 3, a control conversion command is issued, converting the robot movement control to the robot arm control. is a flag variable, indicating the current state of the system: 0 means the system is not started, 1 means the system is running; state 1 is the initial state of the robot, state 3 is the state when the robot reaches the target grasping location, and the rest of the time is in state 2. EEG signal recognition stage: Since EEG signals are only meaningful in state 2, it is first determined whether the robot is in state 2. If it is in state 2, the MGCNN model is called to recognize EEG signals, and the corresponding control command is issued to the robot according to the recognition result; otherwise, the electrooculogram signal recognition is continued. Finally, the robot is controlled through the decoding signals coordinated by the three modules.

[0088] As shown in Table 1, the correspondence between the motor imagery EEG signal categories and the robot control commands is divided into two stages: robot chassis control and robot arm grasping control. Robot chassis control stage: imagine the left hand movement to control the robot to turn to the left target grasping location; imagine the right hand movement to control the robot to turn to the right target grasping location; imagine the foot movement to control the robot to go to the target grasping location; imagine the tongue movement to control the robot to return to the initial position. This stage is divided into two small stages: turning and moving: if the robot is in the turning stage, the foot and tongue categories are invalid; otherwise, the left hand and right hand categories are invalid. Robot arm grasping control stage: imagine the left hand movement to control the robot arm to move one position to the left; imagine the right hand movement to control the robot arm to move one position to the right; imagine the foot movement to control the robot arm to grasp an object; imagine the tongue movement to control the robot arm to put back the grasped object.

[0089] Table 1

[0090]

[0091] As shown in Table 2, the correspondence between the types of electrooculogram signals and the robot control commands, only the blinking type has actual control commands. When the blinking control brain-controlled robot system starts; when the robot is in state 1 and , blinking directly controls the brain-controlled robot system to shut down; when the robot is in state 2, the robot must first cancel the current action and return to the initial position, and then shut down the system; when the robot is in state 3, blinking controls the movement imagination and converts the EEG signal into controlling the robotic arm.

[0092] Table 2

[0093]

[0094] In terms of robot control, the setting of the category threshold is directly related to the sensitivity of the robot's instructions. A smaller target category threshold indicates a higher instruction sensitivity, and vice versa. For online robot control, if the instruction sensitivity is too high, the number of erroneous instructions will increase, affecting system reliability. Therefore, this embodiment raises the threshold, and signals with unclear features are more likely to be regarded as non-blinking or "baseline" categories. Specifically, the eye contact signal blinking category must meet the following requirements: the peak value is greater than 400, the trough value is less than 200, and the interval time between the peak and the trough is between 0.048s and 0.32s; the four task categories of motor imagery EEG signals must meet the following requirements: the class probability is greater than 70%. In addition, once the robot receives an instruction, it will no longer process the EEG signal during the execution of the robot until the specified action is completed.

[0095] Example 2

[0096] This embodiment provides an asynchronous robot hybrid control system based on motor imagery EEG and EOG, including:

[0097] Signal acquisition module, which acquires the user's motor imagery EEG signal and eye contact signal;

[0098] A signal preprocessing module, used for preprocessing the motor imagery EEG signal and the electrooculogram signal;

[0099] The signal decoding module is used to input the preprocessed motor imagery EEG signal into a pre-built EEG recognition model to obtain the user's motor imagery category; input the preprocessed electrooculogram signal into a pre-built electrooculogram recognition model to identify the user's blinking or non-blinking action;

[0100] A command conversion module converts the user's motor imagery categories and blinking or non-blinking actions into corresponding robot control commands;

[0101] The robot control module is used to control the robot according to corresponding robot control commands.

[0102] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0103] A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, enable the computing device to perform an asynchronous robot hybrid control method based on motor imagery electroencephalography and electrooculography.

[0104] A computing device includes one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing an asynchronous robot hybrid control method based on motor imagery electroencephalography and electrooculography.

[0105] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0109] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A hybrid control method for asynchronous robots based on motor imagery EEG and EOG, characterized in that: include: Obtain the user's motor imagery EEG signals and EOG signals; Preprocessing the motor imagery EEG signal and the electrooculogram signal; Input the preprocessed motor imagery EEG signal into a pre-built EEG recognition model to obtain the user's motor imagery category; The preprocessed electrooculogram signal is input into a pre-built electrooculogram recognition model to identify the user's blinking or non-blinking action; Converting user movement imagery categories and blinking or non-blinking actions into corresponding robot control commands; Control the robot according to the corresponding robot control commands.

2. The asynchronous robot hybrid control method based on motor imagery EEG and EOG according to claim 1 is characterized in that: The method for preprocessing the motor imagery EEG signal is as follows: Downsampling the EEG signal to 250 Hz, and filtering the EEG signal by reducing the sampling points, including 0.5 Hz-100 Hz bandpass filtering and 50 Hz notch filtering; The filtered EEG signal is segmented and the preset time window in the motor imagery stage is divided into the task class and the preset time window in the rest stage is divided into the non-task class.

3. The asynchronous robot hybrid control method based on motor imagery EEG and EOG according to claim 1 is characterized in that: The method for preprocessing the electrooculogram signal is as follows: The electrooculogram signal is downsampled to 62.5 Hz, and a 0.5 Hz-30 Hz bandpass filter and a 50 Hz notch filter are performed on the electrooculogram signal; segmented processing is performed, and the preset time window of the blinking stage is divided as a blink class.

4. The asynchronous robot hybrid control method based on motor imagery EEG and EOG according to claim 2 or 3, characterized in that: The band-pass filtering adopts a Butterworth filter, which is expressed by the following formula of square amplitude versus frequency: ; in, is the filter order, set , that is, the fourth-order Butterworth filter, is the cut-off frequency, is the passband edge frequency, It is a parameter that controls the degree of filter passband fluctuation.

5. The asynchronous robot hybrid control method based on motor imagery EEG and EOG according to claim 2 or 3, characterized in that: The notch filter adopts a second-order infinite impulse response filter, whose amplitude response is zero at a certain frequency, and is used to eliminate a certain frequency component. Its function is: ; in, is the notch digital frequency (rad), is the notch frequency (Hz), is the sampling frequency (Hz).

6. The asynchronous robot hybrid control method based on motor imagery EEG and EOG according to claim 1 is characterized in that: The method of constructing the EEG recognition model is: A multi-scale global convolutional neural network is used. The multi-scale global convolutional neural network is set as a one-dimensional convolutional neural network. The convolution kernel slides only in the time dimension. The network architecture of the multi-scale global convolutional neural network includes a multi-scale convolutional layer and two global convolutional layers. After receiving the input, the multi-scale convolutional layer and the global convolutional layer extract the EEG signal features in parallel. After the convolution pooling operation, two feature vectors are obtained. The two feature vectors are connected and fused and input into the fully connected layer to complete the final classification. The multi-scale convolution layer is used to capture more local feature information in the EEG signal by setting three different kernel sizes. The convolution of is the time window size covered by the convolution, is the number of channels covered by the convolution; extract multi-scale features: ; in, is the result of the interaction between the convolution kernel and the data. is the convolution kernel weight, is the data value, is the bias term, is the activation function; The global convolution layer is used to extract the global features of the EEG signal. By building an expanded convolution layer, it has a larger receptive field. ,in is the convolution kernel size, is the expansion rate.

7. The asynchronous robot hybrid control method based on motor imagery EEG and EOG according to claim 1 is characterized in that: The method of constructing the electrooculographic recognition model is: A multi-threshold detection function is used. The function design is based on three rapid blinks, including: peak value, peak position, trough value, trough position, and interval time. Conscious blinking has a higher peak value than unconscious blinking. By setting the peak threshold To distinguish between them; between two active blinks of the user, there is often a trough, and the interval time between the peak position and the trough position is calculated. , to determine whether the two blinks are continuous actions, in order to distinguish two slow blinks; for three fast blinks, it is necessary to determine whether there are two blinks within the interval threshold. within , exceeding In range Treated as non-continuous action; by setting the trough threshold Limit the recognition range of the trough; the calculation process of the multi-threshold detection function: ; in, It is 0 or 1, representing the non-blinking or blinking category. is the number of peaks, is the number of troughs, is the number of changes from peak to trough.

8. The asynchronous robot hybrid control method based on motor imagery EEG and EOG according to claim 1 is characterized in that: The method of converting the user's motor imagery categories and blinking or non-blinking actions into corresponding robot control commands includes: If the recognition result is a blink, the process enters the electrooculogram command determination phase; otherwise, the process enters the EEG signal recognition phase. In the electrooculogram command determination phase, the corresponding command is issued according to the robot state: if the robot is in state 1 and , a command to start the brain-controlled robot system is issued; if the robot is in state 1 and Or if the robot is in state 2, a command to shut down the brain-controlled robot system is issued; if the robot is in state 3, a control conversion command is issued, converting the robot movement control to the robot arm control; is a flag variable, indicating the current state of the system: 0 indicates that the system is not started, and 1 indicates that the system is running; state 1 is the initial state of the robot, state 3 is the state when the robot reaches the target grasping location, and the rest of the time is in state 2; EEG signal recognition stage: first determine whether the robot is in state 2; if the robot is in state 2, identify the EEG signal and send the corresponding control command to the robot according to the recognition result; otherwise, continue to perform EOG signal recognition.

9. The asynchronous robot hybrid control method based on motor imagery EEG and EOG according to claim 8 is characterized in that: The method of converting the motor imagery category into the corresponding robot control command includes a robot chassis control stage and a robot arm grasping control stage; The robot chassis control stage includes: imagining the left hand movement to control the robot to turn to the left target grasping location; imagining the right hand movement to control the robot to turn to the right target grasping location; imagining the foot movement to control the robot to go to the target grasping location; imagining the tongue movement to control the robot to return to the initial position; if the robot is in the turning stage, the foot movement and tongue movement categories are invalid; otherwise, the left hand movement and right hand movement categories are invalid; The robotic arm grasping control stage includes: imagining the left hand movement to control the robotic arm to move one position to the left; imagining the right hand movement to control the robotic arm to move one position to the right; imagining the foot movement to control the robotic arm to grasp an object; and imagining the tongue movement to control the robotic arm to put back the grasped object.

10. The asynchronous robot hybrid control method based on motor imagery EEG and EOG according to claim 8 is characterized in that: The method for converting a blinking or non-blinking action into a corresponding robot control command includes: converting only the blinking action into an actual control command; when the robot is in state 1 and When the blinking control brain-controlled robot system starts; when the robot is in state 1 and , blinking controls the brain-controlled robot system to shut down; when the robot is in state 2, the robot must first cancel the current action and return to the initial position, and then shut down the system; when the robot is in state 3, blinking controls the movement imagination EEG signal to control the robotic arm.

11. Asynchronous robot hybrid control system based on motor imagery EEG and EOG, characterized by: include: Signal acquisition module, which acquires the user's motor imagery EEG signal and eye contact signal; A signal preprocessing module, used for preprocessing the motor imagery EEG signal and the electrooculogram signal; The signal decoding module is used to input the preprocessed motor imagery EEG signal into a pre-built EEG recognition model to obtain the user's motor imagery category; input the preprocessed electrooculogram signal into a pre-built electrooculogram recognition model to identify the user's blinking or non-blinking action; A command conversion module converts the user's motor imagery categories and blinking or non-blinking actions into corresponding robot control commands; The robot control module is used to control the robot according to corresponding robot control commands.

Citation Information

Patent Citations

  • Mechanical arm control method and system based on electroencephalogram signals and electro-oculogram signals

    CN114327048A

  • Motor imagery and eyeball tracking mixed brain-computer interface cursor control system

    CN117348771A

  • Multi-task-oriented brain-controlled composite robot control system and method

    CN117718962A

  • Joint driving control system and method based on electroencephalogram signal motion intention analysis

    CN118664625A

  • Systems and methods for a hybrid brain interface for robotic swarms using EEG signals and an input device

    US20200057498A1