Robot brain-computer interaction method and system based on visual perception and brain-controlled intention decoding

Abstract

This invention discloses a robot brain-computer interface method and system based on visual perception and brain-controlled intention decoding. The method includes: acquiring image sequences to extract skeletal modality data and three-dimensional gaze vectors; inputting these data into a motion intention prediction model to obtain a first action intention and its probability of occurrence, and combining the gaze vectors to determine a spatial target; acquiring EEG signals and inputting them into a hierarchical deep convolutional network with a four-branch parallel architecture to obtain a second action intention and its confidence level, wherein the four branches include time domain, frequency domain, time-frequency domain, and spatial domain branches; and using a full-time shared control model based on dynamic weight allocation to dynamically adjust the decision weights of the visual and EEG modalities according to the probability of occurrence of the first action intention and the confidence level of the second action intention, and performing weighted fusion to generate a globally optimal motion intention containing the finally determined action intention category and spatial target, thereby planning the motion trajectory of the robot's end effector to the spatial target.

Images