Method and system of combining three-dimensional line-of-sight tracking and brain-computer interface to control manipulator to grasp object

Abstract

The invention discloses a method and a system of combining three-dimensional line-of-sight tracking and brain-computer interface technology to control a manipulator to grasp an object. A process thereof includes: collecting line-of-sight signals of a user by an eye tracker, sending data to a controller, calculating three-dimensional coordinates of a gaze point by the controller, and sending the same to the manipulator; and at the same time, collecting human-brain electroencephalogram signal changes by an electroencephalogram helmet, sending the same to the controller, decoding user motion imagination and relaxation status by the controller, and thus triggering grasping and releasing action of the manipulator on the gaze object. According to the method and the system, the dual advantages ofhuman line-of-sight and mind control are exerted, the deficiency that existing 2D line-of-sight tracking technology must rely on a displayer is overcome at the same time, intuitive human-robot interaction can be carried out in a real scene, assisting or rehabilitating manipulators can be provided for paraplegic/hemiplegic patients, movement-inconvenient elderly persons and the like by applying the system of the invention, and autonomy thereof is improved or rehabilitation is helped in everyday life.

Description

technical field [0001] The invention belongs to the technical field of human-computer interaction in a three-dimensional real scene, and in particular relates to a method and system for controlling a mechanical arm to grab an object by combining three-dimensional line of sight tracking and brain-computer interface technology. Background technique [0002] According to relevant statistics, there are at least 8 million people in the country with limb paralysis of varying degrees, and the number of elderly people exceeds 20 million. They cannot take care of themselves. Robotic arms, to enhance their autonomy or perform rehabilitation in daily life, have great social significance. [0003] Some existing implementation methods include brain-computer interface technology and eye-tracking interaction technology. Among them, the brain-computer interface technology can extract the patient's active movement intention without the actual movement of the limbs, thus providing the possib...

AI Technical Summary

Problems solved by technology

Among them, the brain-computer interface technology can extract the patient's active movement intention without the actual movement of the limbs, thus providing the possibility for patients whose limbs cannot move to control the robot, but the current method of controlling the position of the robot arm based on the brain-computer interface technology is extremely unintuitive, and The intent recognition rate is lower when the number of instructions required is large

The interaction method based on gaze tracking can provide intuitive and accurate position control. However, this method lacks the means to extract the patient's active intention, and it is very easy to mistakenly trigger the next action of the robot arm when the patient stares for a long time but does not intend to move.

[0004] Therefore, it is urgent to combine brain-computer interface and eye-tracking technology to develop a human-robot interaction system that obeys the patient's intention in daily life. In recent years, some researchers have developed human-robot interaction that combines brain-computer interface and 2D eye-tracking technology. system, but the 2D-based gaze tracking method can only obtain the two-dimensional coordinates on the screen, so the user must rely on the display, and cannot realize the natural interaction process in the real three-dimensional space

Images