Asynchronous robot hybrid control method and system based on motor imagery electroencephalogram and electrooculogram

Abstract

The application discloses an asynchronous robot hybrid control method and system based on motor imagery electroencephalogram and electrooculogram, and the system comprises three modules of signal acquisition, signal preprocessing and decoding and robot control. First, the signals of a subject are collected through an electroencephalogram device, including electroencephalogram signals and electrooculogram signals, and the electrooculogram signals are added to enrich the control instruction set of the system. Then, the collected signals are processed, and the processing is divided into two stages: the first stage is signal preprocessing, and the original signals are normalized into the required form; the second stage is signal decoding, including two branches of electroencephalogram signal recognition and electrooculogram signal recognition, and the recognition results are converted into robot control commands. Finally, the robot device obtains the recognition results of the signals through a wireless data receiving module, and then converts the recognition results into control commands to control the robot to make corresponding actions, and the feedback of the robot actions is used to guide the issuance of the next command, so that the comfort of human-computer interaction is improved.

Description

TECHNICAL FIELD

[0001] The application relates to an asynchronous robot hybrid control method and system based on motor imagery electroencephalogram and electrooculogram, and belongs to the field of robot control. BACKGROUND

[0002] As a technology of direct interaction between human brain and machine, the brain-computer interface (BCI) can directly output instructions from the brain to the machine without the aid of the muscle system, so that the elderly and the disabled can independently control external devices to assist in daily activities, such as controlling a mechanical arm to grab an object, controlling a wheelchair to move, and the like. The emergence of the BCI technology provides a new interaction mode between human and machine, and has great significance for improving the quality of life of the elderly and the disabled. However, the current brain-controlled external device system does not fully consider the comfort of users and the stability of the system. In the past, researches, such as a brain-controlled system based on steady-state visual evoked potential and a brain-controlled system based on visual P300, all need visual stimulation to induce electroencephalogram signals, which greatly limits the portability and natural interaction mode of BCI application. Although the brain-controlled system based on motor imagery does not need visual stimulation, the current researches make users perform the imagination task at a fixed time through synchronous control, which reduces the naturalness of human-computer interaction, and the existing researches have few control instructions, and the stability of the system is not considered.

[0003] Therefore, there is an urgent need for a multi-instruction asynchronous stable brain-controlled external device method, which enables users to control the robot to perform tasks or make it standby at any time according to their own needs when using the brain-controlled system, and the brain-controlled system has good signal recognition performance and high stability, ensuring high response rate and low false touch rate during task execution.

[0004] The disadvantages of the current technology mainly lie in the interaction mode between the human brain and the external device and the control mode of the brain-controlled system.

[0005] The current brain-controlled robot system based on motor imagery has a single control mode, and the comfort of users using the system is not fully considered, especially in complex real-world scenarios. The single fixed control mode will inevitably cause poor operability. SUMMARY

[0006] The application provides an asynchronous robot hybrid control method and system based on motor imagery electroencephalogram and electrooculogram, which solves the problems disclosed in the background art. The system has multi-instruction control, and by introducing a “baseline” class (signals when no motor imagery and eye movement are performed), users can autonomously decide whether to perform motor imagery or eye movement tasks to achieve asynchronous control and reduce the burden on users.

[0007] In order to solve the above technical problems, the technical scheme adopted by the present application is: an asynchronous robot mixed control method based on motor imagery electroencephalogram and electrooculogram, comprising:

[0008] Obtaining motor imagery electroencephalogram signals and electrooculogram signals of a user;

[0009] Preprocessing the motor imagery electroencephalogram signals and electrooculogram signals;

[0010] Inputting the preprocessed motor imagery electroencephalogram signals into a pre-constructed electroencephalogram recognition model to obtain a motor imagery category of the user;

[0011] Inputting the preprocessed electrooculogram signals into a pre-constructed electrooculogram recognition model to identify a blink or non-blink action of the user;

[0012] Converting the motor imagery category of the user and the blink or non-blink action into corresponding robot control commands;

[0013] Controlling a robot according to the corresponding robot control commands.

[0014] Further, the method for preprocessing the motor imagery electroencephalogram signals is:

[0015] Downsampling the electroencephalogram signals to 250Hz, and filtering the electroencephalogram signals by reducing sampling points, including band-pass filtering of 0.5Hz-100Hz and notch filtering of 50Hz;

[0016] Segmenting the filtered electroencephalogram signals for processing, and cutting a motor imagery stage preset time window as a task category and a rest stage preset time window as a non-task category.

[0017] Further, the method for preprocessing the electrooculogram signals is:

[0018] Downsampling the electrooculogram signals to 62.5Hz, and performing band-pass filtering of 0.5Hz-30Hz and notch filtering of 50Hz on the electrooculogram signals; performing segmentation processing, and cutting a blink stage preset time window as a blink category.

[0019] Further, the band-pass filter adopts a Butterworth filter, and the Butterworth filter is expressed by a formula of amplitude square to frequency as follows:

[0020]

[0021] wherein, is an order of the filter, and is set to 4 , that is, a fourth-order Butterworth filter, is a cutoff frequency, is a passband edge frequency, A parameter for controlling the degree of passband fluctuation of the filter.

[0022] Further, the notch filter adopts a second-order infinite impulse response filter, whose amplitude response is zero at a certain frequency, to eliminate a certain frequency component, and the function is:

[0023]

[0024] wherein, is the notch digital frequency (rad), is the notch frequency (Hz), is the sampling frequency (Hz).

[0025] Further, the method for constructing the electroencephalogram recognition model is:

[0026] A multi-scale global convolutional neural network is adopted, the multi-scale global convolutional neural network is set as a one-dimensional convolutional neural network, the convolution kernel only slides in the time dimension, and the network architecture of the multi-scale global convolutional neural network includes a multi-scale convolutional layer and two global convolutional layers; after the multi-scale convolutional layer and the global convolutional layer receive input, the electroencephalogram signal features are extracted in parallel, two feature vectors are obtained after convolution and pooling operations, and the two feature vectors are connected and fused to be input to a fully connected layer to complete final classification.

[0027] The multi-scale convolutional layer is used to capture more local feature information in the electroencephalogram signal, three convolution kernels with different sizes are set, is the size of a time window covered by convolution, is the number of channels covered by convolution; and multi-scale features are extracted.

[0028]

[0029] wherein, is the result of the convolution kernel acting on the data, is the convolution kernel weight, is the data value, is the bias term, is the activation function.

[0030] The global convolutional layer is used to extract global features of the electroencephalogram signal, an inflation convolutional layer is built, and a larger receptive field is obtained, is the size of the convolution kernel, is the expansion rate.

[0031] Further, the method for constructing the electroencephalogram recognition model is:

[0032] Adopting multi-threshold detection function, the function is designed based on three times of fast blinking, including: wave crest value, wave crest position, wave trough value, wave trough position, interval time; the wave crest value of conscious blinking is higher than that of unconscious blinking, and the wave crest threshold is set to distinguish; the user often accompanies a wave trough between two times of active blinking, and the interval time between the wave crest position and the wave trough position is calculated to judge whether the two times of blinking are continuous actions to distinguish two times of slow blinking actions; for three times of fast blinking actions, it is needed to judge that two times are within the interval threshold , and the actions beyond the range are regarded as non-continuous actions; the wave trough threshold is set to limit the recognition range of the wave trough; the calculation process of the multi-threshold detection function is as follows:

[0033]

[0034] wherein, is 0 or 1, representing non-blinking or blinking, is the wave crest number, is the wave trough number, is the change number from wave crest to wave trough.

[0035] Further, the method for converting the user motion imagination category and the blinking or non-blinking action into corresponding robot control commands comprises:

[0036] if the recognition result is blinking, entering the electro-oculogram command determination stage; otherwise, entering the electroencephalogram signal recognition stage; in the electro-oculogram command determination stage, issuing corresponding commands according to the robot state: if the robot is in state 1 and , issuing the command of starting the brain-controlled robot system; if the robot is in state 1 and or the robot is in state 2, issuing the command of closing the brain-controlled robot system; if the robot is in state 3, issuing the command of control conversion, converting the robot movement control to the robot arm control; wherein, is a flag variable, indicating the current state of the system: 0 indicates that the system is not started, and 1 indicates that the system is running; state 1 is the initial state of the robot, and state 3 is the state when the robot reaches the target grabbing place, and the rest of the time is in state 2; in the electroencephalogram signal recognition stage: firstly, judging whether the robot is in state 2; if the robot is in state 2, recognizing the electroencephalogram signal, and issuing corresponding control commands to the robot according to the recognition result; otherwise, continuing the electro-oculogram signal recognition.

[0037] ​​​Further, the method for converting the motor imagery categories into corresponding robot control commands comprises a robot chassis control stage and a mechanical arm grabbing control stage.

[0038] The robot chassis control stage comprises: controlling the robot to turn to a left target grabbing site by imagining left-hand movement; controlling the robot to turn to a right target grabbing site by imagining right-hand movement; controlling the robot to go to the target grabbing site by imagining foot movement; controlling the robot to return to an initial position by imagining tongue movement; if the robot is in a turning stage, the foot movement and tongue movement categories are invalid; otherwise, the left-hand movement and right-hand movement categories are invalid.

[0039] The mechanical arm grabbing control stage comprises: controlling the mechanical arm to move to the left by one position by imagining left-hand movement; controlling the mechanical arm to move to the right by one position by imagining right-hand movement; controlling the mechanical arm to grab an article by imagining foot movement; and controlling the mechanical arm to put back the grabbed article by imagining tongue movement.

[0040] Further, the method for converting the blink or non-blink action into corresponding robot control commands comprises: converting the only-blink action into actual control commands; when the robot is in state 1 and , the blink controls the brain-controlled robot system to start; when the robot is in state 1 and , the blink controls the brain-controlled robot system to stop; when the robot is in state 2, the robot first needs to cancel the current action and return to the initial position, and then stop the system; when the robot is in state 3, the blink controls the motor imagery electroencephalogram to be converted into control of the mechanical arm.

[0041] Correspondingly, the application also provides an asynchronous robot hybrid control system based on motor imagery electroencephalogram and electrooculogram, comprising:

[0042] A signal acquisition module acquires motor imagery electroencephalogram signals and electrooculogram signals of a user;

[0043] A signal preprocessing module is used for preprocessing the motor imagery electroencephalogram signals and electrooculogram signals;

[0044] A signal decoding module is used for inputting the preprocessed motor imagery electroencephalogram signals into a pre-constructed electroencephalogram recognition model to obtain motor imagery categories of the user; and inputting the preprocessed electrooculogram signals into a pre-constructed electrooculogram recognition model to identify blink or non-blink actions of the user;

[0045] A command conversion module converts the motor imagery categories and the blink or non-blink actions into corresponding robot control commands;

[0046] A robot control module is used for controlling the robot according to the corresponding robot control commands.

[0047] The application has the following beneficial effects:

[0048] The application proposes a multi-scale global convolutional neural network and a multi-threshold detection function decoding method for motor imagery electroencephalogram and electrooculogram signals, and fuses motor imagery electroencephalogram and electrooculogram signals to realize asynchronous and multi-command stable control of an external robot.

[0049] The application combines motor imagery electroencephalogram and electrooculogram, enriches the control command set, adds a 'baseline' class (non-task class), and assists in realizing asynchronous control of the robot, compared with the previous synchronous control mode, and can improve the comfort of human-computer interaction.

[0050] The application decodes signals based on an asynchronous mode, enables the user to decide when to control the robot operation, and adopts a high threshold setting of the category to improve the stability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 It is a schematic diagram of the asynchronous robot hybrid control system based on motor imagery electroencephalogram and electrooculogram of the application.

[0052] Figure 2 It is a schematic diagram of the asynchronous robot hybrid control method flow based on motor imagery electroencephalogram and electrooculogram of the application. DETAILED DESCRIPTION

[0053] The application will be further described below in combination with the drawings. The following examples are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0054] Example 1

[0055] As shown in the drawings, the application provides an asynchronous robot hybrid control method based on motor imagery electroencephalogram and electrooculogram, which includes the following steps: Figure 1

[0056] Step one, electroencephalogram and electrooculogram signal acquisition, acquiring signals through electroencephalogram and electrooculogram signal acquisition equipment;

[0057] Step two, according to the signal characteristics and task requirements, performing signal clipping, downsampling, filtering, and noise reduction operations on the electroencephalogram and electrooculogram signals;

[0058] Step three, inputting the electroencephalogram and electrooculogram data obtained in step two into respective recognition methods, decoding the signal categories, and converting them into corresponding robot control commands.

[0059] Step four, combining the signal decoding of step three, formulating a signal decoding process to ensure the asynchrony and stability of the robot control system.

[0060] The specific process of step one is as follows:

[0061] ​The application adopts the latest wireless electroencephalogram device NeuSen W developed by Brain Products Company to collect electroencephalogram and electrooculogram signals, selects 19 channels of FC3, FC1, FCz, FC2, FC4, C5, C3, C1, Cz, C2, C4, C6, CP3, CP1, CP2, CP4, P3, Pz and P4 as electroencephalogram signal recording electrodes, selects VEOU as an electrooculogram signal recording electrode, and the sampling rate is 1000 Hz.

[0062] The specific process of step two is as follows:

[0063] 1. Motor imagery electroencephalogram signal preprocessing

[0064] Firstly, the collected electroencephalogram signal is down-sampled to 250 Hz to reduce the calculation amount and improve the algorithm efficiency. Moreover, by reducing the sampling points, the overall shape of the signal is smoother, so that some subtle fluctuations in the signal will not have a significant impact on the overall characteristics. Then, the electroencephalogram signal is filtered, including 0.5 Hz-100 Hz band-pass filtering and 50 Hz notch filtering. The band-pass filter adopts a Butterworth filter, which is a smooth filter, and its characteristic is that there is no obvious ripple between the passband and the stopband. The design of this filter is realized by mapping the frequency response function to the plane of poles and zeros. The Butterworth filter can be expressed by the following formula of the square of the amplitude versus frequency:

[0065]

[0066] wherein, is the order of the filter, and in the present study, i.e. a fourth-order Butterworth filter, is the cutoff frequency, is the passband edge frequency, is a parameter for controlling the degree of passband fluctuation of the filter.

[0067] The notch filter adopts a simple second-order infinite impulse response (IIR) filter, which has a zero amplitude response at a certain frequency and can be used to eliminate a certain frequency component. Its system function is:

[0068]

[0069] wherein, is the notch digital frequency (rad), is the notch frequency (Hz), is the sampling frequency (Hz).

[0070] Finally, the electroencephalogram data is segmented and processed, and the 3s time window of the motor imagery stage is cut as the task class, and the 3s time window of the rest stage is cut as the “baseline” class (non-task class).

[0071] 2. Electrooculogram preprocessing

[0072] The electrooculogram is a physiological signal recording eye movement. Since the blink movement will produce a large peak, the electrooculogram can be directly detected by using a single electrode close to the eye position. In order to maintain the consistency of signal preprocessing, the electrooculogram preprocessing process is the same as the motor imagery electroencephalogram. First, the electrooculogram is down-sampled to 62.5 Hz, and a lower sampling rate than the electroencephalogram is selected to reduce irrelevant fluctuations caused by high-frequency components in the signal, making the electrooculogram waveform smoother and improving the robustness of the detection algorithm. Then, the electrooculogram is subjected to 0.5 Hz-30 Hz band-pass filtering and 50 Hz notch filtering. Finally, the segmented processing is performed, and the 1.5 s time window of the blink phase is cut as the blink class.

[0073] The specific process of step three is as follows:

[0074] After signal preprocessing, it needs to be further input to the decoding module for recognition. For this purpose, a multi-scale global convolutional neural network (MGCNN) is constructed for motor imagery electroencephalogram classification, and a multi-threshold detection function (MTDF) is designed for electrooculogram recognition.

[0075] 1. Multi-scale global convolutional neural network

[0076] In order to make the network model learn multi-level comprehensive electroencephalogram feature information, a multi-scale global convolutional neural network is constructed. At the same time, since the electroencephalogram signal features are mainly reflected in the time dimension, the multi-scale global convolutional neural network is set to one-dimensional convolutional neural network, and the convolution kernel only slides in the time dimension to maintain the integrity of the time domain features. The network architecture of the multi-scale global convolutional neural network includes a multi-scale convolutional layer and two global convolutional layers. The multi-scale convolutional layer and the global convolutional layer receive the input and extract the electroencephalogram signal features in parallel. After convolution and pooling operations, two feature vectors are obtained. The two feature vectors are connected and fused, and then input to the fully connected layer to complete the final classification.

[0077] The multi-scale convolutional layer is used to capture more local feature information in the electroencephalogram signal. By setting three different kernel sizes of convolution , the time window size covered by the convolution, and the number of channels covered by the convolution, multi-scale features are extracted:

[0078]

[0079] wherein, is the result of the convolution kernel acting on the data, is the convolution kernel weight, is the data value, is the bias term, is an activation function.

[0080] The global convolutional layer is used to extract the global features of the electroencephalogram signal, and by building an expanded convolutional layer, it has a larger receptive field wherein is the size of the convolution kernel, is the size of the expansion rate.

[0081] 2. Multi-threshold detection function

[0082] The function design is based on three rapid blinks, including: peak value, peak position, trough value, trough position, interval time. The peak value of conscious blinking is higher than that of unconscious blinking, which is distinguished by setting the peak threshold. The user often accompanies a trough between two active blinks, and the interval time between the peak position and the trough position is calculated to determine whether the two blinks are continuous actions to distinguish two slow blinking actions. For three rapid blinking actions, it is necessary to determine that there are two within the interval threshold , and within the range is considered as a non-continuous action. In addition, in order to avoid the interference of high amplitude trough, the recognition range of trough is limited by setting the trough threshold . The calculation process of the multi-threshold detection function is as follows:

[0083]

[0084] wherein, is 0 or 1, representing non-blinking or blinking, is the number of peaks, is the number of troughs, is the number of changes from peak to trough.

[0085] The specific process of step four is as follows:

[0086] The robot used is Alpha Pro mobile grabbing robot, and its configuration includes mobile chassis, collaborative mechanical arm, large atlas gripper, Realsense camera and router antenna and other key components.

[0087] As Figure 2 ​As shown, the robot one-time action decoding process consists of three modules: module 1 is signal acquisition and preprocessing; module 2 is electrooculogram signal recognition; and module 3 is electroencephalogram signal recognition. First, through module 1, signals are acquired in real time, and preprocessing operations are performed to obtain single-channel electrooculogram signals and three-channel electroencephalogram signals. Then, the MTDF is called to recognize the electrooculogram signals: if the recognition result is blinking, the electrooculogram command determination stage is entered; otherwise, the electroencephalogram signal recognition stage is entered. In the electrooculogram command determination stage, corresponding commands are issued according to the robot state: if in state 1 and , the command to start the brain-controlled robot system is issued; if in state 1 and or in state 2, the command to close the brain-controlled robot system is issued; and if in state 3, the command to control the conversion is issued, which converts robot movement control to robot arm control. Among them, is a flag variable representing the current state of the system: 0 indicates that the system is not started, and 1 indicates that the system is running; state 1 is the initial state of the robot, and state 3 is the state when the robot reaches the target grabbing location, and the rest of the time is in state 2. In the electroencephalogram signal recognition stage, since the electroencephalogram signal only has practical significance in state 2, first, it is determined whether the robot is in state 2. If in state 2, the MGCNN model is called to recognize the electroencephalogram signal, and corresponding control commands are issued to the robot according to the recognition result; otherwise, electrooculogram signal recognition is continued. Finally, through the decoding signals of the three modules coordinated with each other, the control of the robot is completed.

[0088] As shown in Table 1, the correspondence between the motor imagery electroencephalogram signal categories and the robot control commands is divided into two stages: robot chassis control and robot arm grabbing control. In the robot chassis control stage: imagining left hand movement controls the robot to turn to the left target grabbing location; imagining right hand movement controls the robot to turn to the right target grabbing location; imagining foot movement controls the robot to move to the target grabbing location; and imagining tongue movement controls the robot to return to the initial position. This stage is divided into two small stages: turning and moving. If the robot is in the turning stage, the foot and tongue categories are invalid; otherwise, the left hand and right hand categories are invalid. In the robot arm grabbing control stage: imagining left hand movement controls the robot arm to move one position to the left; imagining right hand movement controls the robot arm to move one position to the right; imagining foot movement controls the robot arm to grab an object; and imagining tongue movement controls the robot arm to put back the grabbed object.

[0089] Table 1

[0090]

[0091] As shown in Table 2, the correspondence between the electrooculogram signal categories and the robot control commands is that only the blinking category has practical control commands. When the robot is in state 1 and When the robot is in state 1, the blink controls the brain-controlled robot system to start; when the robot is in state 2, the blink directly controls the brain-controlled robot system to close; when the robot is in state 3, the blink controls the motion imagination electroencephalogram signal to be converted into a control mechanical arm.

[0092] Table 2

[0093]

[0094] In terms of robot control, the setting of the category threshold is directly related to the sensitivity of the robot instruction. A smaller target category threshold indicates higher instruction sensitivity, and vice versa. For online robot control, if the instruction sensitivity is too high, the number of incorrect instructions will increase, affecting the reliability of the system. Therefore, the embodiment adjusts the high threshold, and for signals with unclear characteristics, it will be more likely to be considered as a non-blinking category or a "baseline" category. Specifically, the eye signal blinking category needs to meet: the peak value is greater than 400, the trough value is less than 200, and the interval time between the peak and the trough is between 0.048s~0.32s; the four task categories of motion imagination electroencephalogram signal need to meet: the category probability is greater than 70%. In addition, once the robot receives the instruction, it will no longer process the electroencephalogram signal during the execution of the robot action until the specified action is completed.

[0095] Embodiment 2

[0096] The embodiment provides an asynchronous robot hybrid control system based on motion imagination electroencephalogram and eye signals, which comprises:

[0097] A signal acquisition module acquires user motion imagination electroencephalogram signals and eye signals.

[0098] A signal preprocessing module is configured to preprocess the motion imagination electroencephalogram signals and the eye signals.

[0099] A signal decoding module is configured to input the preprocessed motion imagination electroencephalogram signals into a pre-constructed electroencephalogram recognition model to obtain user motion imagination categories; and input the preprocessed eye signals into a pre-constructed eye signal recognition model to identify user blinking or non-blinking actions.

[0100] A command conversion module converts the user motion imagination categories and the blinking or non-blinking actions into corresponding robot control commands.

[0101] A robot control module is configured to control a robot according to the corresponding robot control commands.

[0102] ​The above merely describes the preferred embodiments of the present application, and it should be noted that those skilled in the art can make several improvements and modifications without departing from the technical principles of the present application, and these improvements and modifications should also be considered as falling within the protection scope of the present application.

[0103] A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by a computing device, cause the computing device to perform a method of asynchronous robot hybrid control based on motor imagery electroencephalography and electrooculography.

[0104] A computing device comprising one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing a method of asynchronous robot hybrid control based on motor imagery electroencephalography and electrooculography.

[0105] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0106] The present application is described in reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowchart

[0107] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction means, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowchart

[0108] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide processes for implementing the functions specified in the flowcharts Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0109] The above merely describes the embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. An asynchronous robot hybrid control method based on motor imagery EEG and electrooculography, characterized in that, include: Acquire the user's electroencephalogram (EEG) and electrooculogram (EOG) signals related to motor imagery; The electroencephalogram (EEG) and electrooculogram (EOG) signals of the motor imagery are preprocessed; The preprocessed motor imagery EEG signals are input into a pre-built EEG recognition model to obtain the user's motor imagery category; The pre-processed electrooculogram (EOG) signal is input into a pre-built EOG recognition model to identify the user's blinking or non-blinking actions; The user's imagined movement categories and blinking or non-blinking actions are converted into corresponding robot control commands; Control the robot according to the corresponding robot control commands; Methods for translating user motion imagery categories and blinking or non-blinking actions into corresponding robot control commands include: If the recognition result is blinking, the robot proceeds to the eye-command determination stage; otherwise, it proceeds to the brain-electroencephalogram (EEG) signal recognition stage. In the eye-command determination stage, corresponding commands are issued based on the robot's state: if the robot is in state 1 and... If the system is in robot state 1 and..., then a command to activate the brain-controlled robot system is issued; Alternatively, if the robot is in state 2, a command to shut down the brain-controlled robot system is issued; if the robot is in state 3, a control transition command is issued, switching from robot movement control to robotic arm control; where, The system is a flag variable representing the current state of the system: 0 indicates that the system is not started, and 1 indicates that the system is running. State 1 is the initial state of the robot, and state 3 is the state when the robot reaches the target grasping location. At other times, the robot is in state 2. In the EEG signal recognition stage, it is first determined whether the robot is in state 2. If the robot is in state 2, the EEG signal is recognized, and the corresponding control command is issued to the robot according to the recognition result. Otherwise, the electrooculogram (EOG) signal recognition continues.

2. The asynchronous robot hybrid control method based on motor imagery EEG and EEG according to claim 1, characterized in that, The method for preprocessing the motor imagery EEG signals is as follows: The EEG signal is downsampled to 250Hz, and the EEG signal is filtered by reducing the number of sampling points, including bandpass filtering of 0.5Hz-100Hz and notch filtering of 50Hz. The filtered EEG signal was segmented, with the motor imagery phase being designated as the task category and the rest phase as the non-task category.

3. The asynchronous robot hybrid control method based on motor imagery EEG and EEG according to claim 1, characterized in that, The method for preprocessing the electrooculogram signal is as follows: The electrooculogram (EOG) signal was downsampled to 62.5 Hz, and bandpass filtering (0.5 Hz - 30 Hz) and notch filtering (50 Hz) were applied to the EOG signal. The signal was then segmented, and a preset time window for each blink phase was defined as a blink category.

4. The asynchronous robot hybrid control method based on motor imagery EEG and EEG according to claim 2 or 3, characterized in that, The bandpass filter uses a Butterworth filter, which is expressed by the following formula: the square of the amplitude versus the frequency. ； in, Set the filter order. That is, a fourth-order Butterworth filter. The cutoff frequency, The passband edge frequency, These are parameters used to control the passband ripple of the filter.

5. The asynchronous robot hybrid control method based on motor imagery EEG and EEG according to claim 2 or 3, characterized in that, The notch filter employs a second-order infinite impulse response filter, whose amplitude response is zero at a certain frequency, used to eliminate a specific frequency component. Its function is: ； in, For notch filtering digital frequency, The notch frequency, This represents the sampling frequency.

6. The asynchronous robot hybrid control method based on motor imagery EEG and EEG according to claim 1, characterized in that, The method for constructing an EEG recognition model is as follows: A multi-scale global convolutional neural network is adopted, which is set as a one-dimensional convolutional neural network. The convolutional kernels only slide in the time dimension. The network architecture of the multi-scale global convolutional neural network includes a multi-scale convolutional layer and two global convolutional layers. After receiving the input, the multi-scale convolutional layer and the global convolutional layer extract the EEG signal features in parallel. After convolution pooling operation, two feature vectors are obtained. The two feature vectors are concatenated and fused and then input into the fully connected layer to complete the final classification. Multi-scale convolutional layers are used to capture more local feature information in EEG signals by setting three different kernel sizes. convolution, The size of the time window covered by the convolution. The number of channels covered by the convolution; extracting multi-scale features: ； in, The result of the interaction between the convolution kernel and the data. For convolution kernel weights, For data values, For bias terms, For activation functions; Global convolutional layers are used to extract global features from EEG signals. By constructing dilated convolutional layers, a larger receptive field can be obtained. ,in The kernel size is [size]. This represents the magnitude of the expansion rate.

7. The asynchronous robot hybrid control method based on motor imagery EEG and EEG according to claim 1, characterized in that, The method for constructing the electrooculogram (EOG) recognition model is as follows: A multi-threshold detection function is employed, designed based on three rapid blinks, including: peak value, peak position, trough value, trough position, and interval time. Conscious blinks have a higher peak value than involuntary blinks; this is addressed by setting a peak threshold. To differentiate them; between two active blinks, a trough often appears, which can be calculated by the time interval between the peak and the trough. To determine whether two blinks are consecutive, distinguish between two slow blinks; for three rapid blinks, identify two blinks that occur within a certain interval threshold. within Exceeding Within range This is considered a discontinuous action; by setting a trough threshold... Limiting the recognition range of troughs; the calculation process of the multi-threshold detection function: ； in, A value of 0 or 1 indicates either a non-blinking or blinking type. The number of peaks, For the number of troughs, This represents the number of changes from the peak to the trough.

8. The asynchronous robot hybrid control method based on motor imagery EEG and EEG according to claim 1, characterized in that, The method for converting the motion imagination category into the corresponding robot control command includes a robot chassis control stage and a robotic arm grasping control stage. The robot chassis control phase includes: imagining left hand movement to control the robot to turn to the left target grasping location; imagining right hand movement to control the robot to turn to the right target grasping location; imagining foot movement to control the robot to move to the target grasping location; imagining tongue movement to control the robot to return to the initial position; if the robot is in the turning phase, the foot movement and tongue movement categories are invalid; otherwise, the left hand movement and right hand movement categories are invalid. The robotic arm grasping control phase includes: imagining the left hand moving to control the robotic arm to move one position to the left; imagining the right hand moving to control the robotic arm to move one position to the right; imagining the foot moving to control the robotic arm to grasp the object; and imagining the tongue moving to control the robotic arm to put the grasped object back.

9. The asynchronous robot hybrid control method based on motor imagery EEG and EEG according to claim 1, characterized in that, Methods for converting blinking or non-blinking actions into corresponding robot control commands include: converting only blinking actions into actual control commands; and converting blinking actions into actual control commands when the robot is in state 1 and When blinking activates the brain-controlled robot system; when the robot is in state 1 and Blinking controls the brain-controlled robot system to shut down; when the robot is in state 2, the robot first cancels the current action and returns to the initial position, and then shuts down the system; when the robot is in state 3, blinking controls the conversion of motor imagery brain signals into control of the robotic arm.

10. An asynchronous robot hybrid control system based on motor imagery EEG and electrooculography, characterized in that, include: The signal acquisition module acquires the user's electroencephalogram (EEG) and electrooculogram (EOG) signals related to motor imagery. The signal preprocessing module is used to preprocess the motor imagery EEG signals and electrooculogram signals; The signal decoding module is used to input the pre-processed motor imagery EEG signal into a pre-built EEG recognition model to obtain the user's motor imagery category; and to input the pre-processed electrooculogram (EOG) signal into a pre-built EOG recognition model to identify the user's blinking or non-blinking actions. The command conversion module converts the user's imagined motion categories and blinking or non-blinking actions into corresponding robot control commands; The robot control module is used to control the robot according to the corresponding robot control commands. Methods for translating user motion imagery categories and blinking or non-blinking actions into corresponding robot control commands include: If the recognition result is blinking, the robot proceeds to the eye-command determination stage; otherwise, it proceeds to the brain-electroencephalogram (EEG) signal recognition stage. In the eye-command determination stage, corresponding commands are issued based on the robot's state: if the robot is in state 1 and... If the system is in robot state 1 and..., then a command to activate the brain-controlled robot system is issued; Alternatively, if the robot is in state 2, a command to shut down the brain-controlled robot system is issued; if the robot is in state 3, a control transition command is issued, switching from robot movement control to robotic arm control; where, The system is a flag variable representing the current state of the system: 0 indicates that the system is not started, and 1 indicates that the system is running. State 1 is the initial state of the robot, and state 3 is the state when the robot reaches the target grasping location. At other times, the robot is in state 2. In the EEG signal recognition stage, it is first determined whether the robot is in state 2. If the robot is in state 2, the EEG signal is recognized, and the corresponding control command is issued to the robot according to the recognition result. Otherwise, the electrooculogram (EOG) signal recognition continues.

Images