Rehabilitation robot brain-machine fusion control method and system

Abstract

This invention relates to a brain-computer interface (BCI) control method and system for a rehabilitation robot, comprising the following steps: S1: Using a target image stimulation paradigm, visual evoked signals are induced in the operator, and the operator's electroencephalogram (EEG) signals are simultaneously acquired. The EEG signals include two types: a first type is steady-state visual evoked potential (VEP) EEG signals, and a second type is hand fine motor intention EEG signals; S2: The steady-state VEP EEG signals are decoded to identify the target object and its spatial position in the image being viewed by the operator, generating a target selection command. By employing the above technical solution, this invention simultaneously acquires steady-state VEP EEG signals and hand fine motor intention EEG signals, which are used for target position selection and robot hand movement type determination, respectively. This achieves decision fusion of dual-modal EEG signals, overcoming the problems of low dimensionality and poor flexibility in single-mode control.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology, and in particular relates to a brain-computer fusion control method and system for rehabilitation robots. Background Technology

[0002] The rehabilitation robot for the disabled is an intelligent medical assistive robot that integrates biomedical engineering, artificial intelligence and robotics technology. It is designed to help patients with upper limb dysfunction regain their ability to live independently by assisting with fine motor skills such as grasping and pinching.

[0003] However, existing assistive rehabilitation robots cannot determine tasks and goals based on the patient's intentions. They can only select goals through predefined rules and do not yet possess advanced human intelligent decision-making abilities. Their flexibility and adaptability in complex environments are still far behind those of humans, so they cannot help disabled people regain basic self-care abilities.

[0004] Brain control technology is an emerging technology that uses electrical signals from the cerebral cortex as the source of control information. This technology extracts brain signals under different control intentions of the operator and converts them into control commands for peripheral devices through decoding methods, thus establishing a direct connection between the operator and the peripheral devices. However, existing research on brain control technology focuses on improving the accuracy of brain control command decoding and the information transmission rate of brain control systems, while the control strategies for peripheral devices are relatively simple and the execution efficiency is still lacking. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides a brain-computer interface control method and system for rehabilitation robots. Based on traditional brain control methods, this invention aims to improve the maneuverability of assistive robots and their ability to adapt to complex environments. It combines brain-computer interface technology based on biological intelligence with motion control methods for assistive rehabilitation robots based on upper limb fine motor skills, which can be applied to various complex daily activity training tasks.

[0006] The first objective of this invention is to provide a brain-computer interface control method for a rehabilitation robot, comprising the following steps: S1: Using the target image stimulation paradigm, visual induction is performed on the operator, and the operator's EEG signals are collected simultaneously. The EEG signals include two types: the first type is steady-state visual evoked potential EEG signals, and the second type is hand fine motor intention EEG signals. S2: Decode the steady-state visual evoked potential EEG signal, identify the target object being gazed at by the operator and its spatial location, and generate a target selection command; S3: Decode the EEG signal of the hand fine motor intention, identify the type of hand fine motor imagined or performed by the operator, and generate action type instructions; S4: Perform decision-level fusion of the target selection instruction and the action type instruction to generate a fused control instruction containing the spatial coordinates of the target object and the action primitives; S5: Based on the fusion control command, combined with the environmental map, path planning and motion control are performed to drive the rehabilitation robot to perform fine upper limb movements corresponding to the action type command.

[0007] Further configuration of the present invention: In step S1, the acquisition method of the steady-state visual evoked potential EEG signal is as follows: after grayscale processing of the target object image, the target object image is flashed at different frequencies, the operator gazes at the target object image, and the EEG signals of the O1, O2, and Oz channels in the occipital lobe are acquired; the acquisition method of the fine motor intention EEG signal is as follows: after the operator gazes at the target object image, he actively performs fine motor imagination of the hand associated with the object, that is, motor imagination EEG paradigm. Since the motor imagination is actively initiated by the operator, as long as the operator actively imagines the corresponding grasping action, the event-related EEG signal of the brain's motor cortex can be stably evoked; taking a water cup as an example, after the operator gazes at the image of the water cup, he actively imagines the intention of grasping the water cup with his whole hand in a fist, and the EEG signals of the C3, C4, and Cz channels in the central area of ​​the brain's motor cortex are acquired.

[0008] A further provision of the present invention: In step S2, the decoding method for the steady-state visual evoked potential EEG signal is a hybrid decoding method combining convolutional neural networks and genetic algorithms, or a decoding method based on time-frequency transformation and canonical correlation analysis.

[0009] A further provision of the present invention: In step S3, the decoding method for the EEG signal of fine motor intention of the hand is a hybrid decoding method combining convolutional neural network and genetic algorithm, or a method based on wavelet transform or short-time Fourier transform to extract time-frequency features and then combining them with support vector machine for classification.

[0010] A further feature of this invention is that the decoding model of the EEG signal of the fine motor intention of the hand is established through offline training. During training, the operator views the schematic diagram of the fine motor intention of the hand and actually performs or actively imagines the corresponding fine motor intention of the hand. The EEG signals of the C3, C4 and Cz channels in the central area of ​​the motor cortex are collected and the signal characteristics corresponding to different fine motor intentions are recorded. After the training is completed, the decoding model is fixed in the EEG signal processing unit and does not need to be retrained during online control.

[0011] A second objective of this invention is to provide a brain-computer interface control system for a rehabilitation robot to implement the above-described method, comprising: a robot unit, a visual evoked unit, an EEG signal acquisition unit, and an EEG signal processing unit; The robot unit includes a vision perception module, an industrial control computer module, a motion control module, and a DC servo motor module; The visual perception module is used to acquire target image information; The visual evoked unit is used to process the target image in grayscale and flash the target object image at different frequencies to induce the operator to generate steady-state visual evoked EEG signals. The EEG signal acquisition unit is used to simultaneously acquire the operator's steady-state visual evoked potential EEG signals and fine motor intention EEG signals. The EEG signal processing unit is used to decode the steady-state visual evoked potential EEG signal and the fine motor intention EEG signal of the hand, and output target selection instructions and action type instructions; The industrial control computer module is used to perform decision-level fusion of the target selection instruction and the action type instruction to generate fused control instructions, and to generate motion control instructions in combination with the environmental map; The motion control module is used to convert the motion control commands into drive commands for the DC servo motor module.

[0012] A further feature of the present invention includes a serial communication module, wherein the industrial control computer module is connected to the vision perception module via the serial communication module, and the industrial control computer module is also wirelessly connected to the motion control module.

[0013] A further feature of the present invention is that the industrial control computer module transmits the target image information acquired by the visual perception module to the visual induction unit for grayscale processing.

[0014] A further provision of the present invention: the EEG signal processing unit decodes the EEG signal by: filtering the received EEG signal, removing trend terms, and then performing feature extraction and pattern recognition.

[0015] A further feature of the present invention is that the EEG signal acquisition unit is a portable 32-channel wireless EEG acquisition device, wherein the O1, O2, and Oz channels in the occipital lobe are used to acquire steady-state visual evoked potential signals, and the C3, C4, and Cz channels in the central motor cortex are used to acquire fine motor intention signals of the hand.

[0016] Compared with the prior art, the present invention has at least the following beneficial effects: 1. This invention achieves decision fusion of dual-modal EEG signals by simultaneously acquiring steady-state visual evoked potential (SSVEP) EEG signals and fine motor intention EEG signals of the hand, which are used for target location selection and robot hand action type determination, respectively. This overcomes the problems of low control dimension and poor flexibility of single EEG mode, and significantly improves the interactive flexibility and control accuracy of the rehabilitation robot for the disabled.

[0017] 2. This invention employs a hybrid decoding method combining convolutional neural networks and genetic algorithms to perform high-precision decoding of visually evoked EEG signals. By establishing a mapping model between frequency and action through offline training, the operator only needs to gaze at the target image and actively imagine the corresponding fine hand movements during online control to generate fine hand movement intention signals, thus lowering the barrier to entry and reducing operator fatigue.

[0018] 3. This invention combines human intent with the robot's autonomous environmental perception, path planning, and motion control. The operator only needs to look at the target image, and the robot automatically completes target recognition, spatial positioning, motion mapping, trajectory planning, and execution, realizing natural human-computer interaction. It is especially suitable for rehabilitation training and daily living assistance for patients with upper limb dysfunction.

[0019] 4. This invention extracts images of target objects in the environment in real time through a visual perception module and presents them by flashing at different frequencies, so that the SSVEP stimulus content is consistent with the actual operation object, avoiding the cognitive load caused by the use of abstract symbols in traditional SSVEP brain control, and improving control accuracy and user experience. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the system structure of the rehabilitation robot in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the visual induction process in an embodiment of the present invention; Figure 3 This is a schematic diagram of the robot unit in an embodiment of the present invention; Figure 4 This is a schematic diagram showing the position of the electrodes of the EEG signal acquisition unit on the operator's head in an embodiment of the present invention; Figure 5 This is a schematic diagram of the visual evoked EEG signals of steady-state visual evoked potentials in an embodiment of the present invention; Figure 6 These are schematic diagrams illustrating six types of fine hand movements in embodiments of the present invention.

[0022] In the diagram: 310, Robot Unit; 320, Visual Evocation Unit; 330, EEG Signal Acquisition Unit; 340, EEG Signal Processing Unit; 350, Wireless Communication Unit; 360, Bluetooth Communication Unit; 510, Body Structure; 520, Visual Perception Module; 530, Industrial Control Computer Module; 540, Motion Control Module; 550, DC Servo Motor Module; 560, Serial Communication Module. Detailed Implementation

[0023] The technical solutions in this embodiment will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] This invention provides a brain-computer interface control method and system for a rehabilitation robot, such as... Figure 1-6 As shown, the system structure includes: a robot unit 310, a visual evoked unit 320, an EEG signal acquisition unit 330, an EEG signal processing unit 340, a wireless communication unit 350, and a Bluetooth communication unit 360.

[0025] The robot unit 310 includes a six-axis robotic arm body structure 510, a vision perception module 520, an industrial control computer module 530, a motion control module 540, and a DC servo motor module 550.

[0026] The visual perception module 520 uses a depth camera, such as LeTMC-520, which is installed on the top of the six-axis robotic arm body structure 510. It is used to collect real-time visual information of the environment and visual information such as the recognition and positioning of objects placed on the table, and transmits it to the industrial control computer module 530 through the serial communication module 560.

[0027] The industrial control computer module 530 uses an embedded industrial control computer to handle image processing, map building, decision-level fusion, and motion planning. Specifically, based on indoor environmental information, the industrial control computer module 530 uses the ORB-SLAM algorithm for three-dimensional spatial localization and map building of the indoor environment, and the YOLO V4 algorithm for target detection and two-dimensional localization of objects placed on the desktop. After converting the three-dimensional map into a two-dimensional map image through top-view orthogonal projection and rasterization quantization on the desktop plane, the KD-Tree algorithm is used to extract the target image from the two-dimensional map image, providing the visual evoked unit 320 with accurate, clear, and standardized target images, and providing a precise spatial position reference for the path planning of the rehabilitation robot for the disabled.

[0028] The motion control module 540 uses an STM32-407 microcontroller to drive the DC servo motor module 550 through pulse width modulation (PWM) waves. The DC servo motor module 550 can drive the DC servo motors on each joint of the six-axis robotic arm body structure 510 to rotate, so as to realize the various actions of the six-axis robotic arm body structure 510. The visual evoked unit 320 receives the target image from the industrial control computer module 530 through the wireless communication unit 350. After grayscale processing of the target image, the target image is flashed at different frequencies using a sinusoidal stimulation mode. The operator expresses their target intention by looking at different flashing targets. Specifically, the grayscale processing adopts a weighted average method, converting the color image into an 8-bit grayscale image according to the standard brightness formula R:G:B=0.299:0.587:0.114, with an output grayscale range of 0-255. The purpose of grayscale processing is to eliminate the influence of color vision differences on the occipital visual cortex response, ensuring that target images with different background colors can induce a stable and pure SSVEP response.

[0029] The EEG signal acquisition unit 330 employs a portable 32-channel wireless EEG acquisition device, such as the Neusen W-32, with electrodes placed according to the international standard 10 / 20-lead system. Figure 4 As shown, the O1, O2, and Oz channels in the occipital lobe are used to acquire steady-state visual evoked potential (SSVEP) EEG signals; the C3, C4, and Cz channels in the central motor cortex are used to acquire EEG signals of fine hand movement intentions, with the CPz position selected as the reference electrode.

[0030] The EEG signal processing unit 340 performs preprocessing on the received EEG signal, such as filtering and removing trend terms, and then performs feature extraction and pattern recognition to decode the target selection instruction and action type instruction respectively. The decoded instructions are then transmitted to the industrial control computer module 530 via the Bluetooth communication unit 360.

[0031] The industrial control computer module 530 performs decision-level fusion of target selection instructions and action type instructions to generate fused control instructions containing the spatial coordinates of the target object and action primitives. Combined with the environmental map, it uses a fast random tree search (RRT) algorithm to generate a collision-free navigation path and employs a PID controller to achieve smooth motion control of speed and acceleration. Finally, it generates motion control instructions, which are sent to the motion control module 540 via wireless communication. The motion control module 540 converts the motion control instructions into pulse width modulation (PWM) waves, driving the DC servo motor module 550 to complete the precise motion control of the rehabilitation robot.

[0032] (1) SSVEP decoding model training (this embodiment uses a decoding method combining convolutional neural networks and genetic algorithms as an example): The operator wears an EEG acquisition device and gazes at multiple desktop target object images that flash at different frequencies (6-12Hz), such as water cups, remote controls, etc. Each frequency corresponds to one object image. SSVEP signals from the O1, O2, and Oz channels of the occipital lobe are collected. Each frequency is collected for no less than 100 trials. The frequency classification model is trained using a method combining convolutional neural networks and genetic algorithms. This model can identify the frequency corresponding to the object gazed at by the operator based on the EEG signal. The trained model is then fixed in the EEG signal processing unit.

[0033] It should be noted that this training is not required if the CCA decoding method is used.

[0034] (2) Training of fine motor intention decoding model of hand Manipulator watching Figure 6 The diagram illustrates fine motor skills of the hand, including various hand gestures such as clenching the fist, extending all five fingers, pinching with the thumb and index finger, pinching with three fingers, pinching with the tips of the thumb and index finger together, and pointing with the index finger. Participants were instructed to actually perform or visualize these movements according to the diagram's prompts, repeating each movement at least 100 times, with each trial lasting 4 seconds. Electroencephalogram (EEG) signals from the C3, C4, and Cz channels of the central motor cortex were collected at a sampling rate of 250 Hz. The raw signals were preprocessed by bandpass filtering (2 Hz-30 Hz) and detrending, with a 0.5-second truncated segment taken after the start of the movement. A 2.5-second window is used as a single sample (32 channels × 500 time points). A hybrid decoding method combining convolutional neural networks (CNN) and genetic algorithms (GA) is used for training. The CNN structure includes temporal convolutional layers, spatial convolutional layers, separable convolutional layers, global average pooling layers, and fully connected output layers (6 types). The genetic algorithm is used to optimize hyperparameters, including kernel size, dropout rate, learning rate, and batch size. After training, the decoding model is embedded in the EEG signal processing unit and does not need to be retrained during online control.

[0035] It should be noted that the canonical correlation analysis (CCA) method is designed specifically for SSVEP frequency recognition and is not suitable for decoding EEG signals of fine motor intentions of the hand (motor imagery).

[0036] (3) Establish mapping relationship Establishing a frequency-target object mapping table: Since the target objects placed on the desktop are different in different scenarios, the frequency mapping needs to be dynamically and automatically generated according to the current environment. During system initialization, the visual perception module 520 identifies the target objects on the current desktop, and the industrial control computer module 530 automatically assigns a unique flashing frequency to each target object from the preset frequency pool (6Hz, 7Hz, 8Hz, 9Hz, 10Hz, 12Hz), establishes a three-level mapping table of "frequency-target object category-spatial coordinates" and stores it in the industrial control computer module 530. When the visual perception module 520 detects changes in desktop objects (addition or reduction), the industrial control computer module 530 updates the mapping table in real time, dynamically adding or deleting frequency allocations to ensure that the mapping relationship is always consistent with the current scene before each use. Taking a certain task as an example: 6Hz - water cup (coordinates X1, Y1, Z1), 7Hz - remote control (coordinates X2, Y2, Z2). Establish a mapping table of motion types and robot motion primitives: such as clenching a fist - gripping and grasping, extending five fingers - opening and releasing, etc., and store it in the industrial control computer module.

[0037] Online control phase: Taking the example of the operator entering a complex indoor environment and using brain-computer interface to control the rehabilitation robot to perform a specific upper limb fine motor task, the online control process of the present invention is explained in detail. This phase is carried out on the premise that offline training has been completed, that is, the decoding model has been solidified in the EEG signal processing unit.

[0038] Step S1: Using the target image stimulation paradigm, visual evoked signals are generated in the operator, and the operator's EEG signals (including SSVEP and fine motor intention EEG signals) are collected simultaneously. Specifically, this includes: S1.1 Environmental Perception and Target Image Extraction: The visual perception module 520 (depth camera, such as LeTMC-520) of the assistive rehabilitation robot unit 310 collects real-time information on the complex indoor environment and the visual information of objects placed on the table, including: collecting indoor environmental structure information and ORB feature points, contour feature points, and localization feature points of objects on the table; using the ORB-SLAM algorithm for three-dimensional spatial localization and map construction of the indoor environment; and using the YOLO V4 algorithm for target detection and two-dimensional localization of objects placed on the table, identifying target objects on the table (such as water cups, remote controls, pens, mobile phones, etc.) and obtaining their categories and two-dimensional bounding boxes. After converting the 3D map into a 2D map image through top-down orthogonal projection and rasterization on a desktop plane, the KD-Tree algorithm is used to extract the target object image from the 2D map image. This provides the visual evoked unit with a precise, clear, and standardized target image, and at the same time provides a precise spatial position reference for robot path planning. Specifically, the purpose of the 3D map (ORB-SLAM construction) is to obtain the precise 3D spatial coordinates of the target object for subsequent path planning and motion control of the robot unit 310. The purpose of the 2D processing is to obtain a standardized 2D image of the target object, which is then transmitted to the visual evoked unit 320 for grayscale processing and flashing, presenting and inducing the operator's SSVEP response. The two processes have a clear division of labor and complement each other.

[0039] The aforementioned visual information is transmitted to the industrial control computer module 530 via the serial communication module 560.

[0040] S1.2 Grayscale processing and frequency allocation of target object image: The industrial control computer module 530 transmits the extracted target object image to the visual induction unit 320 through the wireless communication unit 350. The visual evoked unit 320 performs grayscale processing on each target object image: using a weighted average method, the color image is converted into an 8-bit grayscale image according to the weights of the red, green and blue channels in the ratio of 0.299:0.587:0.114, and the output grayscale range is 0-255. The purpose of grayscale processing is to eliminate the influence of color vision differences on the response of the occipital visual cortex, ensure that target object images with different background colors can induce stable and pure SSVEP responses, and reduce the signal variation introduced by color vision differences between individual operators.

[0041] Meanwhile, a unique sinusoidal flicker frequency is assigned to each target object image through a frequency-target object mapping table pre-stored in the industrial control computer module 530. In this embodiment, six target object images are used as an example, corresponding to six different flicker frequencies: 6Hz, 7Hz, 8Hz, 9Hz, 10Hz and 12Hz. All of the above frequencies are within the effective SSVEP induction range (usually 4-40Hz), and the 6-12Hz low-frequency SSVEP paradigm has the advantages of high signal-to-noise ratio and good subject comfort. The frequency interval is not less than 1Hz to avoid frequency aliasing and harmonic interference. The flicker brightness range is set to 0-255 gray levels, and the flicker amplitude (contrast) is not less than 80% to ensure a sufficiently strong SSVEP response.

[0042] The mapping relationship between each target object image and its corresponding frequency is pre-established and stored in the industrial computer module 530 through a frequency-target object mapping table during system initialization. When the vision perception module 520 recognizes a new desktop target object, the industrial computer module 530 automatically assigns the new target object image to the currently idle frequency channel and updates the mapping table in real time to ensure that each target object image corresponds one-to-one with a unique frequency.

[0043] S1.3 Simultaneous acquisition of visual evoked and bimodal EEG signals: The visual evoked unit 320 sinusoidally flashes various grayscale target object images at different assigned frequencies, such as... Figure 5 As shown, a water cup flashes at 6Hz, a remote control flashes at 7Hz, etc. The operator looks at the image of the target object that the robot wants to grasp, such as looking at the image of a "water cup" flashing at 6Hz, according to the task requirements.

[0044] The EEG signal acquisition unit 330 is a portable 32-channel wireless EEG acquisition device, such as the Neusen W-32, which simultaneously acquires two types of EEG signals from the operator according to the international standard 10 / 20-lead system. Steady-state visual evoked potential (SSVEP) EEG signals: signals from the O1, O2, and Oz channels in the occipital lobe were acquired. The reference electrode was selected at the CPz position, which is located in the midline of the parieto-occipital region and at a moderate distance from the target electrode in the occipital visual cortex. The CPz response to SSVEP stimulation is weak in the resting state, which can effectively suppress common-mode noise and maximize the signal-to-noise ratio of the target channel. EEG signal of fine motor intention of the hand: The signals of C3, C4 and Cz channels in the central area of ​​the motor cortex are collected. After the operator looks at the image of the target object, he actively performs fine motor imagery of the hand associated with the target object (i.e. motor imagery EEG paradigm). For example, when looking at a water glass, he actively imagines his whole hand making a fist, and when looking at a remote control, he actively imagines his index finger pointing. Since the motor imagery is initiated entirely by the operator's subjective will, as long as the operator actively imagines the corresponding action, the event-related potential of the motor cortex can be stably induced. This intention signal is the EEG signal of fine motor intention of the hand. The two types of EEG signals are collected synchronously in time and have a clear division of labor in logic: the SSVEP signal is used to determine which target to control, i.e., target selection, and the motor imagery signal is used to determine what action to use to control, i.e. action type selection.

[0045] The sampling rate of the above-mentioned EEG signals is 250Hz, and the acquired bimodal EEG signals are transmitted in real time to the EEG signal processing unit 340 through the wireless communication unit 350.

[0046] Step S2: Decode the steady-state visual evoked potential EEG signal, identify the target object and its spatial location in the image being viewed by the operator, and generate a target selection command, specifically including: S2.1 SSVEP signal preprocessing: The EEG signal processing unit 340 preprocesses the received SSVEP signal from the occipital lobe: 2Hz~30Hz bandpass filtering to filter out low-frequency drift and high-frequency electromyographic noise; detrending term to eliminate interference from slow changes such as electrode polarization; and extracting a time window (500 time points) of 0.5~2.5 seconds after the start of stimulation as the effective signal segment.

[0047] S2.2 SSVEP Signal Decoding: The preprocessed 32-channel × 500 time-point EEG signals are sent to the SSVEP decoding module. This system supports the following decoding methods: Method 1: A hybrid decoding method combining convolutional neural networks and genetic algorithms (CNN-GA hybrid decoding). Preprocessed EEG signals are input into a pre-trained CNN-GA hybrid decoding model. The structure of this model has been determined during the offline training phase, including: Temporal convolutional layer: kernel size 1×64, number of kernels 16, stride 1, used to extract temporal local features of EEG signals; Spatial convolutional layer: The kernel size is the number of channels × 1 (i.e., 32 × 1). Each temporal convolutional kernel corresponds to 2 spatial convolutional kernels. Depth-separable convolution is used to achieve spatial filtering between electrodes. The number of convolutional kernels is 32. Separable convolutional layer: kernel size 1×16, number of kernels 32, stride 16, used to further extract frequency domain features; Global average pooling layer: performs average pooling on each feature map along the time dimension, with an output dimension of 32; Fully connected output layer: The number of output nodes is 6 (corresponding to the stimulation frequency categories of the 6 target object images respectively), the activation function is the Softmax function, and the output is the probability distribution of each frequency category; The activation function of each convolutional layer is the ELU function. The batch normalization layer is placed after each convolutional layer. The model outputs a probability distribution vector of 6 frequency categories, and the frequency category with the highest probability is taken as the initial decoding result.

[0048] Method 2: Canonical Correlation Analysis (CCA) Decoding Method: Perform canonical correlation analysis on the preprocessed multi-channel EEG signal matrix and the fundamental frequency and harmonic sine / cosine reference signal matrix of each target frequency, calculate the canonical correlation coefficient corresponding to each frequency, and take the frequency corresponding to the largest canonical correlation coefficient as the preliminary decoding result.

[0049] Method 3: Short-Time Fourier Transform (STFT) Time-Frequency Feature Extraction and Classification Method: Perform a short-time Fourier transform on the EEG signal of each channel, calculate the amplitude features in the time-frequency joint domain, transform the abstract EEG signal into low-dimensional quantized features that match the stimulation frequency, and then use a pre-trained classifier, such as a support vector machine (SVM) or a lightweight neural network, to classify the features and output the frequency category.

[0050] S2.3 Validity Threshold Verification: To prevent erroneous operations caused by blinking, electromyography interference, random EEG noise, or when the operator is not focused on the target, the system employs a corresponding threshold verification strategy based on the selected decoding method: If Method 1 (CNN-GA hybrid decoding) is used, the validity threshold verification method is as follows: take the maximum Softmax probability distribution of the CNN output under the random gaze state of the operator in the offline calibration stage as the benchmark, and take its mean plus 1.5 times the standard deviation as the upper bound of the probability threshold; during online recognition, if the Softmax probability of all categories is lower than the threshold, it is judged as invalid input; if the maximum Softmax probability exceeds the threshold, the corresponding frequency category is taken as the valid recognition result. If Method 2, Canonical Correlation Analysis (CCA) decoding, is used, the validity threshold verification method is as follows: During the offline calibration phase, the CCA coefficient distribution of each operator at each target frequency is collected. The maximum value distribution of the CCA coefficient at each frequency when the operator randomly gazes (without a target) is used as the benchmark, and the mean plus 1.5 times the standard deviation is taken as the upper bound of the validity threshold (i.e., the noise upper bound). During the online recognition phase, if the CCA coefficients corresponding to all frequencies are lower than the validity threshold, it is determined to be invalid input, that is, the operator has not actively gazed at any target, and the system remains in a waiting state and issues a prompt tone or visual flash feedback to the operator. If there is a maximum CCA coefficient that exceeds the threshold, it is determined to be valid recognition, and the brain control intention command is output at the frequency corresponding to the maximum CCA coefficient.

[0051] S2.4 Generate target selection instruction: The verified valid frequency is converted through the preset "frequency-target object-spatial position" mapping table in the industrial control computer module 530: frequency-target object category and its three-dimensional spatial coordinates in the environmental map (obtained by the visual perception module in S1.1), and finally a target selection instruction is generated, which includes the semantic category and precise spatial position of the target object.

[0052] Step S3: Decode the EEG signal of the fine motor intention of the hand, identify the type of fine motor movement of the hand actively imagined by the operator, and generate action type instructions, specifically including: S3.1 Preprocessing of motor intention signals: The EEG signal processing unit 340 performs the same preprocessing on the received signals from the C3, C4, and Cz channels of the central motor cortex: 2-30Hz bandpass filtering, detrending term removal, and truncating a time window of 0.5-2.5 seconds after the start of stimulation (500 time points). S3.2 Motor Intent Decoding: The preprocessed motor cortex EEG signal is sent to the motor intent decoding module. This decoding model has been established through offline training. During online decoding, the model outputs the probability distribution of 6 types of fine hand motor intents and takes the action type corresponding to the highest probability. Similarly, a validity threshold can be set to verify and filter out invalid or ambiguous intent recognition results. S3.3 Generate motion type instructions: Generate motion type instructions through the preset "motion type - robot motion primitive" mapping table.

[0053] Step S4: Perform decision-level fusion of the target selection command and the action type command to generate a fused control command containing the target object's spatial coordinates and action primitives, specifically including: The EEG signal processing unit 340 transmits the target selection instruction generated in step S2 and the action type instruction generated in step S3 to the industrial control computer module 530 via the Bluetooth communication unit 360. The industrial control computer module 530 performs decision-level fusion, integrating the two in the logical order of target localization followed by action confirmation: first, the semantic category of the target object and its three-dimensional spatial coordinates are determined by the target selection instruction obtained by SSVEP decoding (provided by step S2.4). Next, the operation action primitive is determined by the action type instruction obtained from the motion intent decoding (provided by step S3.3); The two are merged to generate a structured fusion control instruction, in the format: {target object category, three-dimensional spatial coordinates (x, y, z), action primitive type}, for example {water cup, (0.35m, 0.12m, 0.08m), full hand clenched fist}; this fusion instruction provides a complete task description for the robot's path planning and fine motion control, based on which the robot automatically completes the entire process of target recognition, spatial localization, motion mapping, trajectory planning and execution.

[0054] The advantage of this decision-level fusion is that the two signals are independent and do not interfere with each other. Even if the quality of one signal degrades, the other can still continue to work, which has a certain degree of redundancy and robustness.

[0055] Step S5: Based on the fusion control commands and combined with the environmental map, perform path planning and motion control to drive the assistive rehabilitation robot to execute upper limb fine motor movements corresponding to the action type commands, including: S5.1 Path planning: The industrial control computer module 530 generates a collision-free navigation path from the robot's current end effector position to the target position based on the spatial coordinates of the target object in the fused control command and the environmental map constructed in step S1. S5.2 Velocity and Acceleration Planning: A PID controller is used to plan the velocity and acceleration curves during motion, achieving smooth start and stop and avoiding impacts. Velocity and acceleration planning ensures smooth robot movement and prevents impacts or object drops caused by sudden acceleration or deceleration. S5.3 Motion control command generation and execution: After the planning is completed, a complete set of motion control commands is generated, including the path point sequence, angular velocity of each joint, angular acceleration, etc. The industrial computer module 530 sends the motion control commands to the motion control module 540 through wireless communication. The motion control module 540 converts motion control commands into pulse width modulation (PWM) waves, which drive the DC servo motors of each joint in the DC servo motor module 550, enabling the six-axis robotic arm body structure 510 to sequentially complete the following sequence of actions: Move to above the target object; adjust the end effector posture to adapt to grasping; execute the upper limb fine motor corresponding to the action type command; lift the object; move it to the operator's hand or a designated location; if the action type is open release, release the object.

[0056] S5.4 Task Completion Feedback: After the robot unit 310 completes the expected movement, it can provide feedback to the operator through visual feedback and biological perception. Specifically, visual feedback: the industrial control computer module 530 indicates task completion via a display or indicator light; auditory feedback: a prompt sound is emitted through a speaker; tactile feedback: the robot lightly touches the operator's hand to indicate that the task has been completed.

[0057] At this point, a complete online brain-computer interface control cycle ends. The operator can then focus on the next target object image and repeat steps S1 to S5 to achieve continuous, multi-task assistive rehabilitation control.

Claims

1. A brain-computer interface control method for a rehabilitation robot, characterized in that: Includes the following steps: S1: Using the target image stimulation paradigm, visual induction is performed on the operator, and the operator's EEG signals are collected simultaneously. The EEG signals include two types: the first type is steady-state visual evoked potential EEG signals, and the second type is hand fine motor intention EEG signals. S2: Decode the steady-state visual evoked potential EEG signal, identify the target object being gazed at by the operator and its spatial location, and generate a target selection command; S3: Decode the EEG signal of the hand fine motor intention, identify the type of hand fine motor imagined or performed by the operator, and generate action type instructions; S4: Perform decision-level fusion of the target selection instruction and the action type instruction to generate a fused control instruction containing the spatial coordinates of the target object and the action primitives; S5: Based on the fusion control command, combined with the environmental map, path planning and motion control are performed to drive the rehabilitation robot to perform fine upper limb movements corresponding to the action type command.

2. The method according to claim 1, characterized in that, In step S1, the method for acquiring the steady-state visual evoked potential EEG signal is as follows: after processing the grayscale of the target object image, the target object image is flashed at different frequencies, the operator stares at the target object image, and the EEG signals of the O1, O2, and Oz channels in the occipital lobe are acquired. The method for collecting the EEG signals of the fine motor intention of the hand is as follows: after the operator looks at the image of the target object, he actively imagines the fine motor intention of the hand associated with the object, and collects the EEG signals of the C3, C4 and Cz channels in the central area of ​​the motor cortex of the brain.

3. The method according to claim 1, characterized in that, In step S2, the decoding method for steady-state visual evoked potential EEG signals is either a hybrid decoding method combining convolutional neural networks and genetic algorithms, or a decoding method based on time-frequency transformation and canonical correlation analysis.

4. The method according to claim 1 or 3, characterized in that, In step S3, the decoding method for the EEG signal of fine motor intention of the hand is either a hybrid decoding method combining convolutional neural network and genetic algorithm, or a method based on wavelet transform or short-time Fourier transform to extract time-frequency features and then combining them with support vector machine for classification.

5. The method according to claim 4, characterized in that, The decoding model of the EEG signal of the intention of fine hand movement is established through offline training. During training, the operator watches the diagram of fine hand movement and actually performs or actively imagines the corresponding fine hand movement. The EEG signals of the C3, C4 and Cz channels in the central area of ​​the motor cortex are collected and the signal characteristics corresponding to different fine movements are recorded.

6. A brain-computer interface control system for a rehabilitation robot used to implement the method of any one of claims 1-5, characterized in that, include: Robot unit (310), visual evoked unit (320), EEG signal acquisition unit (330) and EEG signal processing unit (340); The robot unit (310) includes a vision perception module (520), an industrial control computer module (530), a motion control module (540), and a DC servo motor module (550). The visual perception module (520) is used to acquire target image information; The visual evoked unit (320) is used to process the target image in grayscale and flash the target object image at different frequencies to induce the operator to generate steady-state visual evoked EEG signals. The EEG signal acquisition unit (330) is used to simultaneously acquire the steady-state visual evoked potential EEG signal and the fine motor intention EEG signal of the operator. The EEG signal processing unit (340) is used to decode the steady-state visual evoked potential EEG signal and the fine motor intention EEG signal of the hand, and output the target selection instruction and the action type instruction; The industrial control computer module (530) is used to perform decision-level fusion of the target selection instruction and the action type instruction to generate fused control instructions, and to generate motion control instructions in combination with the environment map; The motion control module (540) is used to convert the motion control commands into drive commands for the DC servo motor module (550).

7. The system according to claim 6, characterized in that, It also includes a serial communication module (560), the industrial control computer module (530) is connected to the vision perception module (520) through the serial communication module (560), and the industrial control computer module (530) is also wirelessly connected to the motion control module (540).

8. The system according to claim 7, characterized in that, The industrial control computer module (530) extracts the target image information obtained by the visual perception module (520) and transmits it to the visual induction unit (320).

9. The system according to claim 6, characterized in that, The EEG signal processing unit (340) decodes the EEG signal by filtering the received EEG signal, removing trend terms, and then performing feature extraction and pattern recognition.

10. The system according to claim 6, characterized in that, The EEG signal acquisition unit (330) is a portable 32-channel wireless EEG acquisition device, wherein the O1, O2, and Oz channels in the occipital lobe are used to acquire steady-state visual evoked potential signals, and the C3, C4, and Cz channels in the central area of ​​the motor cortex are used to acquire fine motor intention signals of the hand.

Images