A calibration-free and low-stimulation brain-eye asynchronous virtual robotic arm control system

Abstract

The present invention provides a calibration-free and low-stimulation brain-eye asynchronous virtual robotic arm control system. The system first determines the left and right areas where a stimulation block flashes through eye movement signals, and then classifies the electroencephalogram (EEG) signals through a training-free classification algorithm to determine the instruction selected by the user. Before the system is used, there is no need to perform system calibration and algorithm training on the eye movement signals, and a good classification accuracy rate can be achieved. The method does not require any training and calibration of the EEG signals and the eye movement signals, thus omitting the calibration time before the system is used and improving the convenience of the system. At the same time, real-time feedback from the system can be received, making it convenient for the user to understand the execution status of the system in real time, improving the friendliness of the system and facilitating the practical application of the system.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction and brain-computer interface, and specifically relates to a brain-eye asynchronous virtual robotic arm control system with no calibration and little stimulation. Background Art

[0002] Brain-computer interface (BCI) technology establishes a direct connection between the human brain and external devices, with electroencephalogram (EEG) signals being the most widely used in BCI applications. Steady-state visual evoked potentials (SSVEPs) are gaining increasing attention due to their advantages, such as stable evoked signals and high signal-to-noise ratio. However, SSVEP-based BCIs often require users to stare at a screen with numerous flickering stimuli for extended periods, which can increase user fatigue and discomfort. Synchronous systems with fixed stimulus intervals still dominate BCI research. These systems lack flexibility and user autonomy, hindering practical application. Furthermore, single-modality human-computer interaction approaches using EEG signals are often limited and limited in their application. Therefore, BCI systems continue to face challenges in improving information transmission rates, human-computer compatibility, stability, and ease of use. Integrating virtual reality (VR) technology as a new platform with BCI systems, the design of a brain-eye asynchronous control system promises to address these challenges. Exploring these issues includes enhancing user autonomy, alleviating visual fatigue, and improving system convenience and user friendliness. Summary of the Invention

[0003] In view of this, the present invention provides a calibration-free and low-stimulation brain-eye asynchronous virtual robotic arm control system, which can better combine brain-eye signals to achieve asynchronous control, improve recognition accuracy, reduce user visual fatigue, and improve the user's operating system autonomy as well as the convenience and friendliness of the system.

[0004] A calibration-free and low-stimulation brain-eye asynchronous virtual robotic arm control system, comprising hardware equipment, a virtual scene robotic arm module, a brain-eye asynchronous control module, and a control instruction sending module;

[0005] The hardware equipment includes VR equipment, EEG equipment and eye movement equipment;

[0006] The virtual scene robotic arm module is used to render a virtual reality 3D scene in a VR device, including a robotic arm and a virtual plane; the virtual plane is divided into a left area, a central area, and a right area; a set number of flashing stimulation blocks are provided in each of the left area and the right area, and each flashing stimulation block represents an instruction for the robotic arm to perform an action;

[0007] The brain-eye asynchronous control module is used to:

[0008] The eye movement signals collected by the eye movement device are processed, and it is determined whether the area where the user's eyes are gazing is the left area, the right area or the central area; if it is determined that the gaze is on the left area, a command is sent to the virtual scene robotic arm module via the control instruction sending module to control the flashing stimulus block in the left area to start flashing; if it is determined that the gaze is on the right area, a command is sent to the virtual scene robotic arm module via the control instruction sending module to control the flashing stimulus block in the right area to start flashing; if it is determined that the user closes his eyes for more than a set time during the flashing of the stimulus block, a command is sent to the virtual scene robotic arm module via the control instruction sending module, and the robotic arm refuses to execute the instruction represented by the currently selected stimulus block;

[0009] The user's EEG signals collected by the EEG device are processed to obtain the target classification result, that is, the flickering stimulus block that the user is looking at or selected; then a command is sent to the virtual scene robotic arm module via the control instruction sending module to control the robotic arm to execute the instruction represented by the selected flickering stimulus block.

[0010] Preferably, the brain-eye asynchronous control module is further configured to: if it is determined that the user is not gazing or is gazing at the central area, then the flickering stimulus block does not flicker.

[0011] Preferably, the brain-eye asynchronous control module filters and removes the baseline preprocessing operation on the EEG signal, and then uses the filter bank canonical correlation analysis algorithm to classify the EEG signal in real time to obtain the EEG coefficient corresponding to each flickering stimulus block, and selects the one with the largest EEG coefficient as the target classification result.

[0012] Preferably, the brain-eye asynchrony control module determines the variance value of the eye movement data, and if the variance is less than a set threshold, it is regarded as gaze.

[0013] Preferably, the control instruction sending module compiles the classification results obtained by the brain-eye asynchronous control module into control instructions and transmits them to the virtual scene robotic arm module.

[0014] Preferably, the virtual scene robotic arm module also presents the control instructions sent by the control instruction sending module to the user in real time through visual and auditory means.

[0015] Preferably, the virtual scene robotic arm module completes the transmission of control instructions through the TCP / IP communication protocol, thereby realizing the interaction among the virtual scene robotic arm module, the brain-eye asynchronous control module and the control instruction sending module.

[0016] Preferably, the VR device is a head-mounted virtual reality device.

[0017] Preferably, the eye tracking device adopts an embedded infrared eye tracker.

[0018] The present invention has the following beneficial effects:

[0019] (1) The present invention provides a calibration-free and low-stimulation brain-eye asynchronous virtual robotic arm control system, which first determines the left and right areas of the flashing stimulus block through eye movement signals, and then classifies the EEG signals through a training-free classification algorithm to determine the instructions selected by the user. Before the system is used, there is no need to calibrate the eye movement signals and train the algorithm, and a good classification accuracy can be achieved. This method does not require any training and calibration of EEG signals and eye movement signals, omitting the calibration time before the system is used and improving the convenience of the system. At the same time, real-time feedback from the system can be received, which makes it easier for users to understand the execution status of the system in real time, improves the friendliness of the system, and is conducive to the practical application of the system.

[0020] (2) The present invention triggers the flashing of a unilateral stimulation block by selecting the fixation area through eye movement, which reduces the time the user's eyes receive the flashing stimulation and the number of EEG classification targets, thereby alleviating the visual fatigue caused by the stimulation and improving the overall accuracy of the system.

[0021] (3) The present invention can detect whether the user is gazing and the gaze area of ​​the eye movement, and the user can choose the time to start stimulation according to his or her own wishes, thereby improving the autonomy and friendliness of the system operation.

[0022] (4) The present invention can realize eye closure detection during the flicker stimulation stage, and the user can autonomously reject erroneous instructions generated by false triggering and other reasons, thereby improving the autonomy and robustness of system control. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the system structure of the present invention.

[0024] Figure 2 Schematic diagram of the division of the attention area of ​​the present invention. DETAILED DESCRIPTION

[0025] The present invention is described in detail below with reference to the accompanying drawings and embodiments. The described embodiments are only part of the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0026] like Figure 1 As shown, the present invention provides a calibration-free brain-eye asynchronous virtual robotic arm control system, which mainly includes hardware equipment, a virtual scene robotic arm module, a brain-eye asynchronous control module, and a control instruction sending module.

[0027] The hardware device's function is to present the user with a virtual scene of the robotic arm in use and collect EEG and eye movement signals. This section includes: VR equipment, EEG equipment, and eye movement equipment, specifically head-mounted virtual reality equipment, wireless EEG acquisition equipment, and embedded infrared eye trackers.

[0028] The virtual scene robotic arm module primarily implements real-time rendering of virtual reality 3D scenes and executes robotic arm operation commands. It consists of two main parts: the virtual scene component, which provides a virtual robotic arm and application scenario, executes robotic arm control commands, and provides visual and auditory feedback to the user; and the virtual stimulation component, which provides an area containing flickering stimulation blocks. The left and right areas are equipped with flickering stimulation blocks, while the central area is not equipped with flickering stimulation blocks.

[0029] When the user is detected gazing at the left or right stimulation area, the stimulation block on the corresponding side will flash; when the user is not detected gazing or is detected gazing at the central area, the stimulation block will not flash. This method provides a small amount of visual stimulation to the user.

[0030] The brain-eye asynchronous control module mainly processes EEG and eye movement signals and determines the target classification results. This module includes three functions:

[0031] The first part involves gaze detection. The eye movement device does not require system calibration. Eye movement data is acquired directly from the device in real time, and the variance of the data over a fixed duration is calculated. If the variance is less than a threshold, it is considered a gaze. If the gaze point falls on the left or right stimulation area, the control command sending module sends a command to trigger the stimulation blocks in the corresponding areas of the virtual scene robotic arm module to flash. The eye movement signal processing algorithm does not require calibration.

[0032] The second is the EEG signal processing part. When the stimulus block flashes, the EEG signal is collected and pre-processed such as filtering and baseline removal is performed on the collected EEG signal. Then, the filter group canonical correlation analysis algorithm is used to directly classify the EEG signal in real time without training, and the EEG coefficient of each target is obtained. The one with the largest EEG coefficient is selected as the classification result. In the present invention, only one side of the stimulus block flashes during target classification, and thus the EEG signal processing part does not need to classify all targets. The number of targets that need to be classified is only half of the total number. In addition, the use of the system does not require calibration or training of eye movement signals and EEG signals. Eye movement signals do not require calibration of hardware equipment data acquisition and calibration of classification algorithms. The eye gaze detection part only uses eye movement coordinates to distinguish between left and right gaze areas, without the need to accurately determine the gaze target; and the EEG signal processing part uses a filter group canonical correlation analysis classification algorithm that does not require training.

[0033] Third, if the user wishes to reject the robot arm's command due to a false trigger or other reason, this module detects whether the user's eyes are closed during the flickering stimulus. If it detects that the eyes have been closed for longer than a set time, it sends a message to the control command sending module rejecting the command. Eye closure detection is achieved by calculating the proportion of non-zero pupil diameter sampling points within a fixed time window.

[0034] The control command transmission module's main function is to compile the classification results obtained by the brain-eye asynchronous control module into control commands and transmit them to the virtual scene robotic arm module. There are three types of control commands: the first, the stimulus start command, which initiates the flickering stimulation of the virtual scene robotic arm module; the second, the execution command, which is an executable command for the robotic arm; and the third, the rejection command, which is an error caused by a false trigger or other reason. For erroneous commands caused by false triggers, the eye movement gaze detection component detects eye closure and rejects the erroneous command.

[0035] Furthermore, the virtual scene robotic arm module, in addition to presenting 3D virtual scenes and providing visual stimulation, can also provide real-time visual and auditory feedback to the user. This module is implemented using the Unity engine.

[0036] Furthermore, the control instruction sending module completes the transmission of control instructions through a customized TCP / IP communication protocol, thereby realizing the interaction between the virtual scene robotic arm module, the brain-eye asynchronous control module and the control instruction sending module.

[0037] Example:

[0038] like Figure 2 As shown, the gaze area is primarily divided into a central area, a left stimulation area, and a right stimulation area. The virtual robotic arm is located in the central area. The left and right stimulation areas contain eight visual stimulation blocks, corresponding to the eight actions the robotic arm can perform. When the user's gaze is focused on the central area or when they close their eyes and rest, the stimulation blocks remain stationary. However, they only flash when the user's gaze is focused on the left or right stimulation area.

[0039] Before using the system, the eye tracking equipment does not need to be calibrated, and the EEG and eye tracking signal processing algorithms do not require training. During operation, the brain-eye asynchronous control module detects the user's gaze. When gaze is detected and the center of gaze falls within the left or right stimulation area, it transmits a command to the virtual scene robotic arm module and emits a beep. The four stimulation blocks corresponding to the gaze area then begin to flash. After the flashing stimulation ends, the brain-eye asynchronous control module classifies the collected EEG signals. The control instruction transmission module then compiles the classification results into control instructions, and the virtual robotic arm executes the corresponding action, completing the execution of one action. The system then enters the next eye tracking detection cycle and repeats the above process until the robotic arm is fully controlled.

[0040] The brain-eye asynchronous control module processes the left and right eye gaze coordinates within the sliding time window. First, it removes zero eye gaze coordinates. Next, it calculates the variance of the left and right eye gaze coordinates (x, y) and sums them. If this sum is less than the gaze threshold, gaze is detected. The mean of the left and right eye gaze coordinates is then calculated and compared with the left and right boundary thresholds to determine the gaze region. If the gaze region is identified as the left or right stimulation region, a command is sent to the virtual scene robotic arm module to start flashing the stimulation block on the corresponding side.

[0041] The brain-eye asynchronous control module processes and classifies the valid EEG data it reads. First, a digital filter is used to limit the effective range of the EEG signal frequency band and remove power frequency interference. Secondly, the baseline is removed. Finally, a filter bank canonical correlation analysis algorithm is used to obtain the EEG classification coefficient. The signal is divided into different frequency bands using a filter bank. Canonical correlation analysis is then used to calculate the correlation coefficients with the four target frequency template signals in each frequency band. The weighted sum of the correlation coefficients across all frequency bands is then used to obtain the EEG classification coefficient.

[0042] The virtual scene robotic arm module provides users with three forms of visual and auditory feedback: ① The robotic arm performs corresponding actions when executing instructions; ② The recognized stimulus block will change from white to blue; ③ Chinese voice prompts of the execution action are played.

[0043] The control instruction sending module transmits control instructions from the brain-eye asynchronous control module to the virtual scene robotic arm module via a custom TCP / IP protocol, enabling interaction between the virtual scene robotic arm module, the brain-eye asynchronous control module, and the control instruction sending module. The communication protocol specifies the format of communication data packets, encoding and decoding methods, and transmitted variables, and implements real-time duplex communication.

[0044] Furthermore, the system may experience false triggering, where the stimulus block flashes without the user intending to send a command. To address this issue, the user can close their eyes for the entire duration of the flashing stimulus to deny the incorrect command for the current trial. Eye closure detection is achieved by calculating the ratio of the effective pupil diameters of both eyes within a fixed time window.

[0045] In summary, the above are merely preferred embodiments of the present invention, intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the present invention and implement it accordingly. They are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A calibration-free and low-stimulation brain-eye asynchronous virtual robotic arm control system, characterized in that: Including hardware equipment, virtual scene robotic arm module, brain-eye asynchronous control module, and control instruction sending module; The hardware equipment includes VR equipment, EEG equipment and eye movement equipment; The virtual scene robotic arm module is used to render a virtual reality 3D scene in a VR device, including a robotic arm and a virtual plane; the virtual plane is divided into a left area, a central area, and a right area; a set number of flashing stimulation blocks are provided in each of the left area and the right area, and each flashing stimulation block represents an instruction for the robotic arm to perform an action; The brain-eye asynchronous control module is used to: The eye movement signals collected by the eye movement device are processed, and the eye movement device does not require system calibration before use. The system determines whether the area where the user's eyes are gazing is the left area, the right area, or the central area; if it is determined that the gaze is on the left area, a command is sent to the virtual scene robotic arm module via the control instruction sending module to control the flashing stimulus block in the left area to start flashing; if it is determined that the gaze is on the right area, a command is sent to the virtual scene robotic arm module via the control instruction sending module to control the flashing stimulus block in the right area to start flashing; if it is determined that the user closes his eyes for more than a set time during the flashing of the stimulus block, a command is sent to the virtual scene robotic arm module via the control instruction sending module, and the robotic arm refuses to execute the instruction represented by the currently selected stimulus block; If the gaze was judged to be non-fixated or focused on the central area, the flickering stimulus block did not flicker; When the stimulus block flashes, EEG signals are collected, the user's EEG signals collected by the EEG device are processed, and a target classification result is obtained, i.e., the flickering stimulus block that the user is looking at or selecting; the EEG signals are filtered and pre-processed to remove the baseline, and then the EEG signals are classified in real time using a filter bank canonical correlation analysis algorithm to obtain the EEG coefficients corresponding to each flickering stimulus block, and the one with the largest EEG coefficient is selected as the target classification result; Then, a command is sent to the virtual scene robotic arm module via the control instruction sending module to control the robotic arm to execute the instruction represented by the selected flashing stimulus block.

2. The calibration-free and low-stimulation brain-eye asynchronous virtual robotic arm control system according to claim 1, characterized in that: The brain-eye asynchronous control module determines the variance value of the eye movement data, and if the variance is less than a set threshold, it is considered as gaze.

3. The calibration-free and low-stimulation brain-eye asynchronous virtual robotic arm control system according to claim 1, characterized in that: The control instruction sending module compiles the classification results obtained by the brain-eye asynchronous control module into control instructions and transmits them to the virtual scene robotic arm module.

4. The calibration-free and low-stimulation brain-eye asynchronous virtual robotic arm control system according to claim 1, characterized in that: The virtual scene robotic arm module also presents the control instructions sent by the control instruction sending module to the user in real time through visual and auditory means.

5. The calibration-free and low-stimulation brain-eye asynchronous virtual robotic arm control system according to claim 1, characterized in that: The virtual scene robotic arm module completes the transmission of control instructions through the TCP / IP communication protocol, realizing the interaction between the virtual scene robotic arm module, the brain-eye asynchronous control module and the control instruction sending module.

6. The calibration-free and low-stimulation brain-eye asynchronous virtual robotic arm control system according to claim 1, characterized in that: The VR device is a head-mounted virtual reality device.

7. The calibration-free and low-stimulation brain-eye asynchronous virtual robotic arm control system according to claim 1, characterized in that: The eye tracking device adopts an embedded infrared eye tracker.

Images