Self-adaptive SSMVEP-MI fusion brain-computer interface method and system based on eye movement tracking

Abstract

The invention provides an adaptive SSMVEP-MI fusion brain-computer interface method and system based on eye movement tracking. The method comprises the following steps: determining a current effective fixation area of a user in real time through eye movement tracking; based on the region, only activating a corresponding local SSMVEP visual stimulation target, and generating associated specific motor imagery prompt information; electroencephalogram signals are synchronously collected, and SSMVEP features and MI features are extracted; self-adaptively selecting a decoding path according to whether the accumulated number of the MI effective samples with the labels reaches a threshold value or not: if not, independently decoding by using the SSMVEP features; if yes, fusion decoding based on signal quality evaluation is carried out on the bimodal features; and finally outputting a control signal to drive external equipment or update an interactive interface. According to the method, the visual interference is remarkably reduced, the recognition accuracy and the training efficiency at the initial stage of motor imagery are improved, and the robustness and the availability of the system are ensured through a dual adaptive mechanism of the sample number and the signal quality.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface technology, and in particular to an adaptive SSMVEP–MI fusion brain-computer interface method and system based on eye tracking. Background Technology Stroke, traumatic brain injury, and other neurological disorders often lead to impaired motor function, severely impacting patients' quality of life. The core goal of rehabilitation therapy is to promote neuroplasticity and motor function recovery by actively stimulating motor neural pathways through the brain. In recent years, brain-computer interface (BCI) technology, as a novel approach to neurorehabilitation, decodes patients' electroencephalogram (EEG) signals to recognize motor intentions and control limbs, providing a direct neurofeedback mechanism for rehabilitation training.

[0002] Existing brain-computer interfaces for rehabilitation are mainly based on the following signaling pathways: Steady-state visual evoked potentials (SSVEP / SSMVEP) brain-computer interfaces: These interfaces induce brain electrical signals through fixed-frequency flashing or motion visual stimulation. The signals are stable, the response is strong, and the recognition accuracy is high, making them suitable for most people. However, traditional SSVEP typically uses full-screen or bilateral multi-target flashing stimulation, which can lead to cross-target interference, visual fatigue, and limited effectiveness in stimulating motor function when the patient lacks voluntary motor intention.

[0003] Imagery-based brain-computer interface (MI): Users imagine limb movements with their brains without actually performing them; EEG captures the corresponding brainwave features for classification. This method can activate the brain's motor areas, improve neuroplasticity, and is suitable for rehabilitation training. However, it usually requires long-term training data collection, requires patients to remain at rest, avoid electromyographic artifacts, leading to significant fatigue; and the initial recognition accuracy is low, which can easily affect patient confidence.

[0004] Multimodal fusion attempts: Some studies have attempted to fuse SSVEP with MI, using visual stimulation to guide motor imagery and improve the separability of MI. However, there are problems such as significant interference from full-screen / bilateral stimulation, distraction of attention; lack of adaptive control over the eye movement fixation area, resulting in insufficient MI representation discrimination; and lack of plug-and-play capability for patients in the early stages of rehabilitation, requiring additional training.

[0005] In summary, existing technologies suffer from the following drawbacks: visual stimuli are highly interfering and easily lead to fatigue; the initial accuracy of motor imagery recognition is low and training is time-consuming; and there is a lack of gaze-based adaptive stimulus design to fully guide brain region activation. Therefore, there is an urgent need for a new method and system that can achieve high-accuracy lateral MI decoding and SSMVEP fusion control under low-interference, low-fatigue, and plug-and-play conditions. Summary of the Invention

[0006] The purpose of this invention is to provide an adaptive SSMVEP–MI fusion brain-computer interface method and system based on eye tracking, which can solve existing problems.

[0007] The technical problems solved by this invention include at least the following: 1. How to improve the accuracy of initial motor imagery (MI) recognition: In traditional brain-computer interfaces (MIs), the initial brainwave signal characteristics of users are not obvious, and the classification accuracy is usually only around 50% to 60%, leading to unreliable operation and affecting the rehabilitation training effect. Therefore, the problem that this invention aims to solve is how to use visual attention guidance and local stimulation strategies to improve the separability and recognition accuracy of MI signals in the early stages of training, and realize a plug-and-play brain-computer interface.

[0008] 2. How to reduce visual stimulation-induced interference and fatigue: Traditional SSVEP full-screen or multi-sided target flicker can easily lead to cross-target interference and visual fatigue, and may even pose a risk of epilepsy to some patients. The problem this invention aims to solve is how to adaptively activate local SSVEP stimulation based on the user's gaze side, thereby reducing the number of stimuli and interference, and improving comfort and signal stability.

[0009] 3. How to shorten the MI training sample collection time and efficiently acquire labeled data: Traditional motion imagery (MI) data acquisition phases typically last 15 to 20 minutes or longer, requiring users to remain still and repeatedly visualize actions, a time-consuming and tedious process. The problem this invention aims to solve is how to combine SSMVEP classification results with the fixation side to achieve synchronized motion imagery cues and data acquisition, thereby efficiently generating labeled MI samples and reducing the training burden.

[0010] 4. How to achieve adaptive fusion and robust control: Existing multimodal SSVEP–MI fusion methods lack real-time adaptive mechanisms, and their recognition performance tends to degrade when eye-tracking gaze or EEG signal quality is low. The problem this invention aims to solve is how to design an adaptive fusion strategy based on signal quality to automatically switch or adjust when SSVEP or MI signals fail, ensuring system robustness and continuous control capabilities.

[0011] The technical solution of the present invention is as follows: According to one aspect of the present invention, an adaptive SSMVEP–MI fusion brain-computer interface method based on eye tracking is disclosed, comprising the following steps: The coordinates of the user's gaze point are obtained in real time and mapped to at least three preset gaze regions to determine the current effective gaze region; In response to the current effective gaze area, local stimulus control is performed: only one local SSMVEP visual stimulus target corresponding to the current effective gaze area is activated, while generating specific motion imagery cue information associated with that area; wherein, SSMVEP visual stimulus targets outside the current effective gaze area are inactive. Simultaneously collect the user's electroencephalogram (EEG) signals; From the EEG signals, SSMVEP features related to the local SSMVEP visual stimulus target and MI features related to the motor imagery cue information were extracted respectively. Determine whether the number of valid MI samples that have been collected and labeled with motion imagery prompts has reached a preset threshold; Perform adaptive decoding based on the judgment result: If the number of valid MI samples does not reach the preset threshold, the first control command is generated based on the SSMVEP feature. If the number of valid MI samples reaches a preset threshold, the second control command is generated based on the fusion result of SSMVEP features and MI features. Output control signals to drive associated external devices or update the interactive interface according to a first control command or a second control command.

[0012] In some implementations, decoding is performed based on the fusion result of SSMVEP features and MI features, including: The first identification result based on SSMVEP features and its first quality index, and the second identification result based on MI features and its second quality index are obtained respectively. Based on the first and second quality indicators, the weights of the first and second identification results in the final decision are dynamically adjusted and then weighted and fused.

[0013] In some implementations, the eye-tracking-based adaptive SSMVEP–MI fusion brain-computer interface method further includes an adaptive mode switching step: Continuously monitor the primary and secondary quality indicators; When the first quality index remains below the first switching threshold, the activation of the local SSMVEP visual stimulus target is paused, and the mode of decoding based on MI features is switched. When the second quality index remains below the second switching threshold, switch to the mode that decodes based on SSMVEP features.

[0014] In some implementations, the first quality metric includes the signal-to-noise ratio or classification confidence of the SSMVEP feature; the second quality metric includes the classification confidence of the MI feature.

[0015] In some implementations, the specific motion imagery cue is at least one of a visual cue, an auditory cue, or a tactile cue, used to guide the user to perform a specific motion imagery task associated with the currently effective gaze area.

[0016] According to another aspect of the present invention, an adaptive SSMVEP–MI fusion brain-computer interface system based on eye tracking is also disclosed, which can be used to implement the above-described adaptive SSMVEP–MI fusion brain-computer interface method based on eye tracking. The system includes at least: The eye-tracking module is configured to acquire the coordinates of the user's gaze point in real time. The gaze region determination module is configured to map the gaze point coordinates to at least three preset gaze regions and output the currently valid gaze region. The stimulus presentation and control module is configured to activate only one local SSMVEP visual stimulus target corresponding to the current effective gaze area in response to the current effective gaze area, and generate associated specific motion imagery cue information; wherein, SSMVEP visual stimulus targets outside the current effective gaze area are inactive. The EEG acquisition module is configured to acquire the user's EEG signals. The feature extraction module is configured to extract SSMVEP features and MI features from EEG signals, respectively. The adaptive fusion decoding module is configured as follows: Record and determine whether the number of valid MI samples labeled with motion imagery prompts has reached a preset threshold. When the number of valid MI samples does not reach the preset threshold, the first control command is generated based on the SSMVEP feature. When the number of valid MI samples reaches a preset threshold, the second control command is generated by decoding based on the fusion result of SSMVEP features and MI features. The control output module is configured to output control signals to drive associated external devices or update the interactive interface according to a first control command or a second control command.

[0017] In some implementations, the adaptive fusion decoding module includes a quality assessment unit and a fusion decision unit; The quality assessment unit is configured to assess the first quality index corresponding to the SSMVEP feature and the second quality index corresponding to the MI feature. The fusion decision unit is configured to: when the number of valid MI samples reaches a preset threshold, acquire the first identification result based on SSMVEP features and the second identification result based on MI features, and perform weighted fusion of the two based on the first quality index and the second quality index.

[0018] In some implementations, the adaptive fusion decoding module is also configured to perform adaptive mode switching: When the first quality index continues to be lower than the first switching threshold, an instruction is sent to the stimulus presentation and control module to pause the SSMVEP visual stimulus and control the fusion decision unit to switch to the mode of decoding based on MI features. When the second quality index continues to be lower than the second switching threshold, the control fusion decision unit switches to the mode of decoding based on SSMVEP features.

[0019] In some implementations, the system also includes a motion imagery cue module configured to generate specific motion imagery cue information in the form of visual, auditory, or tactile cues, based on instructions from the stimulus presentation and control module.

[0020] In some implementations, the adaptive fusion decoding module is configured to: each time the local SSMVEP visual stimulus target is activated and the EEG signal acquisition is valid, associate and store the synchronously acquired EEG signal fragment with the motor imagery task label determined by the currently valid gaze area, and accumulate it as a valid MI sample.

[0021] The beneficial effects of this invention are as follows: 1. Significantly improves initial MI recognition accuracy and training efficiency: By linking eye-tracking-guided local stimulation with associated MI tasks, the specific activation of the brain's motor cortex is enhanced, improving the separability of MI signals. Simultaneously, the real-time high-precision recognition results of SSMVEP are used to automatically and accurately label the synchronously generated MI EEG data, greatly shortening the supervised data acquisition time required for MI model training, achieving "plug-and-play" functionality and rapid performance improvement.

[0022] 2. Effectively reduces visual interference and fatigue: It abandons the traditional full-screen or multi-target continuous flashing mode and adopts the strategy of "gaze-driven, local activation" and "stimulus and cue space partitioning". It significantly reduces visual confusion and cognitive load from both the stimulus source and cue information levels, reduces user visual fatigue, and improves the comfort and long-term usability of the system.

[0023] 3. Enhance the system's adaptability and robustness: Through decoding path switching based on the number of samples and adaptive fusion and mode switching mechanisms based on signal quality assessment, the system can automatically adjust its working mode when the SSMVEP signal is interfered with (such as blinking or gaze deviation) or the MI signal quality is unstable, ensuring the continuity and stability of control command output in various practical application scenarios.

[0024] 4. Wide range of applications: This system is not only suitable for limb motor function rehabilitation training for stroke patients, but also for prosthesis and rehabilitation robot control, as well as various human-computer interaction scenarios that require high-precision and robust intent recognition, such as virtual reality / augmented reality interaction and smart home control.

[0025] In summary, this invention discloses an adaptive SSMVEP-MI fusion brain-computer interface method and system based on eye tracking, belonging to the field of brain-computer interface technology. The method includes: determining the user's current effective gaze area in real time through eye tracking; based on this area, activating only one corresponding local SSMVEP visual stimulus target and generating associated specific motor imagery prompts; simultaneously acquiring EEG signals and extracting SSMVEP and MI features; adaptively selecting a decoding path based on whether the accumulated number of labeled MI valid samples reaches a threshold: if not, decoding using only SSMVEP features; if so, performing fusion decoding of the dual-modal features based on signal quality assessment; and finally outputting a control signal to drive external devices or update the interactive interface. This invention significantly reduces visual interference and improves the recognition accuracy and training efficiency in the initial stages of motor imagery through gaze-driven local stimulation and task guidance. Furthermore, the dual adaptive mechanism of sample quantity and signal quality ensures the robustness and usability of the system, making it widely applicable in the fields of neurorehabilitation, assistive control, and human-computer interaction. Attached Figure Description

[0026] Figure 1 This is a structural block diagram of the core module of the eye-tracking-based adaptive SSMVEP–MI fusion brain-computer interface system of the present invention; Figure 2 This is a system architecture block diagram of the adaptive SSMVEP–MI fusion brain-computer interface system based on eye tracking of the present invention; Figure 3 This is a schematic diagram of the core steps of the eye-tracking-based adaptive SSMVEP–MI fusion brain-computer interface method of the present invention; Figure 4 This is a detailed flowchart of the adaptive SSMVEP–MI fusion brain-computer interface method based on eye tracking disclosed in Embodiment 1 of the present invention.

[0027] Figure 5 This is a detailed flowchart of the adaptive fusion decoding module in Embodiment 1 of the present invention; Figure 6 This is a schematic diagram of one type of stimulation interface (divided into three regions: left, middle, and right) of the present invention; Figure 7 yes Figure 6 The diagram shows the activation of a local stimulus in the middle region of the stimulus interface.

[0028] Figure labels: 101-Eye tracking module; 102-Gaze ​​area determination module; 103-Stimulus presentation and control module; 104-Motor imagery prompting module; 105-EEG acquisition module; 106-Feature extraction module; 107-Adaptive fusion decoding module; 108-Control output module. Detailed Implementation

[0029] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0030] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0031] like Figure 1-7 As shown, this invention provides an adaptive SSMVEP–MI fusion brain-computer interface method and system based on eye tracking. The core of the invention lies in using eye tracking to achieve spatial and temporal synchronization of "gaze-stimulus-MI task" and designing an adaptive data acquisition and fusion decoding process to solve the pain points in the existing technology.

[0032] like Figure 1-2 As shown, this invention discloses an adaptive SSMVEP–MI fusion brain-computer interface system based on eye tracking, the core architecture of which includes: The eye-tracking module 101 is configured to acquire the user's gaze coordinates in real time. It is used to acquire binocular images in real time and calculate the gaze point, gaze stability, and gaze area. The eye-tracking module 101 includes a camera, infrared illumination, a pupil detection algorithm, and a gaze mapping algorithm. The eye-tracking module 101 can be referenced from the Tobii device described in CN113419628A. The gaze point refers to the focal position of the eye's gaze, commonly used in eye tracking or visual analysis, representing the specific coordinates of the line of sight on a screen or in space. In eye-tracking technology, the gaze point is calculated by capturing eye movements through sensors or cameras to determine the projection point of the gaze on a target surface (such as a screen). The gaze mapping algorithm is mainly used to estimate the direction of human eye gaze or the gaze point; common methods include geometric methods, appearance methods, and hybrid methods.

[0033] The gaze region determination module 102 is configured to map the gaze point coordinates to at least three preset gaze regions and output the currently valid gaze region; for example, the gaze region determination module 102 is used to map the user's gaze point to three predefined regions on the left, center and right of the screen and output parameters such as gaze confidence and gaze stabilization time.

[0034] The stimulus presentation and control module 103 is configured to activate only one local SSMVEP visual stimulus target corresponding to the current effective gaze area in response to the current effective gaze area, and generate associated specific motion imagery cue information; wherein, the SSMVEP visual stimulus target outside the current effective gaze area is inactive. The EEG acquisition module 105 is configured to acquire the user's EEG signals; preferably, it is an EEG acquisition device that can acquire EEG signals from the user's occipital lobe, central region, motor cortex, etc., and supports acquisition of 8 to 32 channels.

[0035] Feature extraction module 106 is configured to extract SSMVEP features and MI features from EEG signals, respectively; The adaptive fusion decoding module 107 is configured as follows: Record and determine whether the number of valid MI samples labeled with motion imagery prompts has reached a preset threshold. When the number of valid MI samples does not reach the preset threshold, the first control command is generated based on the SSMVEP feature. When the number of valid MI samples reaches a preset threshold, the second control command is generated by decoding based on the fusion result of SSMVEP features and MI features. The control output module 108 is configured to output control signals to drive associated external devices or update the interactive interface according to a first control command or a second control command.

[0036] Preferably, the adaptive fusion decoding module 107 includes a quality assessment unit and a fusion decision unit; The quality assessment unit is configured to assess the first quality index corresponding to the SSMVEP feature and the second quality index corresponding to the MI feature. The fusion decision unit is configured to: when the number of valid MI samples reaches a preset threshold, acquire the first identification result based on SSMVEP features and the second identification result based on MI features, and perform weighted fusion of the two based on the first quality index and the second quality index.

[0037] Preferably, the adaptive fusion decoding module 107 is also configured to perform adaptive mode switching: When the first quality index continues to be lower than the first switching threshold, an instruction is sent to the stimulus presentation and control module 103 to pause the SSMVEP visual stimulus and control the fusion decision unit to switch to the mode of decoding based on MI features. When the second quality index continues to be lower than the second switching threshold, the control fusion decision unit switches to the mode of decoding based on SSMVEP features.

[0038] Preferably, the system also includes a motion imagery prompting module 104, configured to generate specific motion imagery prompting information in the form of visual, auditory, or tactile sensations according to the instructions of the stimulus presentation and control module 103.

[0039] Preferably, the adaptive fusion decoding module 107 is configured to: when the local SSMVEP visual stimulus target is activated and the EEG signal acquisition is effective, associate and store the synchronously acquired EEG signal segment with the motor imagery task label determined by the current effective gaze area, and accumulate it as a valid MI sample.

[0040] like Figure 3-4 As shown, this invention discloses an adaptive SSMVEP–MI fusion brain-computer interface method based on eye tracking, which includes at least: The coordinates of the user's gaze point are obtained in real time and mapped to at least three preset gaze regions to determine the current effective gaze region; In response to the current effective gaze area, local stimulus control is performed: only one local SSMVEP visual stimulus target corresponding to the current effective gaze area is activated, while generating specific motion imagery cue information associated with that area; wherein, SSMVEP visual stimulus targets outside the current effective gaze area are inactive. Simultaneously collect the user's electroencephalogram (EEG) signals; From the EEG signals, SSMVEP features related to the local SSMVEP visual stimulus target and MI features related to the motor imagery cue information were extracted respectively. Determine whether the number of valid MI samples that have been collected and labeled with motion imagery prompts has reached a preset threshold; Perform adaptive decoding based on the judgment result: If the number of valid MI samples does not reach the preset threshold, the first control command is generated based on the SSMVEP feature. If the number of valid MI samples reaches a preset threshold, the second control command is generated based on the fusion result of SSMVEP features and MI features. Output control signals to drive associated external devices or update the interactive interface according to a first control command or a second control command.

[0041] Preferably, decoding is performed based on the fusion result of SSMVEP features and MI features, including: The first identification result based on SSMVEP features and its first quality index, and the second identification result based on MI features and its second quality index are obtained respectively. Based on the first and second quality indicators, the weights of the first and second identification results in the final decision are dynamically adjusted and then weighted and fused.

[0042] As a preferred embodiment, the eye-tracking-based adaptive SSMVEP–MI fusion brain-computer interface method further includes an adaptive mode switching step: Continuously monitor the primary and secondary quality indicators; When the first quality index remains below the first switching threshold, the activation of the local SSMVEP visual stimulus target is paused, and the mode of decoding based on MI features is switched. When the second quality index remains below the second switching threshold, switch to the mode that decodes based on SSMVEP features.

[0043] Preferably, the first quality metric includes the signal-to-noise ratio or classification confidence of the SSMVEP feature; the second quality metric includes the classification confidence of the MI feature.

[0044] Preferably, the specific motion imagery prompt is at least one of visual, auditory, or tactile prompts, used to guide the user to perform a specific motion imagery task associated with the currently effective gaze area.

[0045] The present invention will be further described below through specific embodiments.

[0046] Example 1 like Figure 2As shown, the adaptive SSMVEP-MI fusion brain-computer interface system based on eye tracking in this embodiment includes: an eye tracking module 101, a gaze region determination module 102, a stimulus presentation and control module 103, a motor imagery prompting module 104, an EEG acquisition module 105, a feature extraction module 106, an adaptive fusion decoding module 107, and a control output module 108. These modules work collaboratively to achieve a closed loop from user intent perception to control signal output.

[0047] Figure 4 This paper demonstrates a complete embodiment of the eye-tracking-based adaptive SSMVEP–MI fusion brain-computer interface method of the present invention, combined with the following... Figure 4 A detailed explanation of the adaptive SSMVEP–MI fusion brain-computer interface method based on eye tracking is provided: Step S401, System Initialization: The user wears an integrated or separate eye tracker and EEG acquisition device, facing the stimulation screen. After the system starts, eye track calibration (such as 9-point calibration) and EEG electrode impedance detection are performed first to ensure signal acquisition quality.

[0048] Step S402, Real-time Eye Tracking and Region Determination: The eye tracking module 101 continuously acquires images of the user's eyes and calculates the coordinates of the gaze point in the screen coordinate system in real time using algorithms such as pupil-corneal reflection. The gaze region determination module 102 divides the screen into at least three predefined regions (this embodiment uses left, center, and right regions as an example, e.g.) Figure 3 As shown in the diagram, the module maps the real-time gaze coordinates to the corresponding region. The module also calculates gaze stability (such as the variance of the gaze point within a certain time window) and confidence. Only when gaze stability exceeds a certain time (such as 200ms) and the confidence is higher than a threshold is it determined to be the "current effective gaze region".

[0049] Step S403, Local Stimulus Activation and MI Cue Generation: This is one of the key steps of the present invention. The stimulus presentation and control module 103 receives information about the currently effective gaze region. For example... Figure 6 As shown, three potential stimulus positions—left, center, and right—are preset on the screen. The core logic executed by module 103 is to make only the stimulus target corresponding to the currently effective fixation area flash or move according to a preset specific frequency and encoding pattern to induce SSMVEP; simultaneously, it immediately turns off or keeps the stimulus targets in other non-fixation areas static or dark. Simultaneously, the motion imagery cue module 104 generates explicit and associated MI task cues based on the fixation area.

[0050] like Figure 7 As shown, the system's display interface is specially designed to spatially separate stimuli from cues, thereby minimizing visual interference: Central region: Used to present the local SSMVEP visual stimulus target (e.g., highlighted graphics, dynamically changing patterns). Users complete the SSMVEP gaze recognition and selection function by looking at the target in this region.

[0051] Side regions (e.g., the left and right sides of the screen): These are used to automatically generate and display corresponding lateral MI cues based on the current effective gaze area (left or right). For example, when gazing to the left, the left side region displays a weak visual cue related to "left hand" (such as a lit left-hand icon and a softly flashing border); when gazing to the right, the right side region displays a "right hand" cue. These MI cues and the central SSMVEP stimulus are displayed in spatially partitioned sections, without overlapping or covering each other.

[0052] Prompt format: MI prompts include the aforementioned weak visual prompts, and can be combined with optional voice prompts (such as playing "Imagine with your left hand") to enhance task guidance in a multimodal manner.

[0053] This clear spatial partitioning design of "central stimulus and lateral cues" ensures that the specific frequency stimuli of SSMVEP, the overall visual cognitive load, and the MI task cues do not interfere with each other. This guarantees both the signal-to-noise ratio of the SSMVEP signal and provides clear, low-interference guidance for the user to perform the MI task. This ensures a high degree of spatial and intentional consistency between the visual attention focus, the SSMVEP stimulus source, and the MI cognitive task.

[0054] Step S404, Synchronous EEG Acquisition: During stimulus activation and MI cue presentation, the EEG acquisition module 105 synchronously acquires continuous EEG signals.

[0055] Step S405, Feature Extraction: After preprocessing the acquired EEG signal (such as filtering and artifact removal), the feature extraction module 106 processes the two modes in parallel: SSMVEP Feature Extraction: For EEG data within a short time window (e.g., 1-4 seconds) after the stimulus begins, methods such as filter bank canonical correlation analysis (FBCCA) are used to calculate the correlation between the data and each preset stimulus frequency and its harmonics, and to obtain feature values ​​or confidence levels.

[0056] MI Feature Extraction: For EEG data within a relatively long time window (e.g., 2-6 seconds) after the occurrence of the MI cue, extract time-domain, frequency-domain, or time-frequency-domain features related to motion imagery, such as the energy variations of event-related desynchronization / synchronization (ERD / ERS) in the μ (8-13Hz) and β (13-30Hz) frequency bands. These features can be input into a lightweight classifier (e.g., EEGNet) for preliminary classification.

[0057] The classification confidence of SSMVEP features can be characterized by the maximum correlation coefficient with the target frequency calculated by the FBCCA algorithm or by the normalized score; the classification confidence of MI features can be characterized by the probability of the corresponding imagined category output by classifiers such as EEGNet or the decision function value. These confidence values ​​are directly used for subsequent quality assessment and fusion decisions.

[0058] Step S406, Sample Management and Quantity Judgment: This function is performed by the adaptive fusion decoding module 107. Each time steps S403-S405 are executed, if the current effective gaze region is reliably determined and the quality of the synchronously acquired EEG signal is acceptable, the system automatically generates a labeled MI sample. The label represents the motion imagery task category determined by the current effective gaze region (e.g., "left-hand imagery"). The adaptive fusion decoding module 107 continuously counts the number N of such effective samples.

[0059] Step S407, Adaptive Fusion Decoding and Output: This is the core step of this embodiment, and its detailed logic is as follows: Figure 5 As shown.

[0060] The adaptive fusion decoding module 107 first determines whether the accumulated number of valid MI samples N has reached the preset threshold N_th (e.g., N_th = 30 for each type of task).

[0061] Path A (N < N_th, MI model not ready): At this stage, the MI classifier is unreliable or does not exist. The adaptive fusion decoding module 107 will ignore the MI features and decode solely based on the SSMVEP features. For example, the region corresponding to the stimulus frequency with the largest feature value is selected as the recognition result. This result is used to generate the first control command. The system can still provide stable BCI control at this stage, while continuously accumulating high-quality labeled MI samples in the background.

[0062] Path B (N≥N_th, MI model ready): The system switches to fusion mode at this point. The adaptive fusion decoding module 107 simultaneously receives the SSMVEP decoding result (e.g., target class C_ssmvep and its confidence Conf_s) and the MI decoding result (e.g., target class C_mi and its confidence Conf_m). It executes the adaptive fusion mechanism: Consistency check and direct output: If C_ssmvep and C_mi are consistent, then the consistency result is directly output as the final instruction.

[0063] Weighted Fusion in Case of Inconsistency: If the two metrics are inconsistent, a signal quality metric is introduced for arbitration. The signal-to-noise ratio (SNR_s) of the SSMVEP signal is calculated as its quality metric Q_s, and the MI classification confidence score Conf_m is taken as its quality metric Q_m. The final decision D can be obtained through weighted voting, with the weights being functions of Q_s and Q_m. This ensures that the higher-quality mode dominates the decision-making process.

[0064] Adaptive Mode Switching: The adaptive fusion decoding module 107 continuously monitors Q_s and Q_m. If Q_s remains below a low threshold (e.g., low SNR, possibly due to frequent blinking or eye movement by the user), while Q_m remains high, the module can automatically trigger mode switching, sending a command to the stimulus presentation and control module 103 to temporarily pause SSMVEP stimulation. The system enters "MI-only" mode, relying solely on MI for control. Conversely, if Q_m remains too low, it switches to "SSMVEP-dominated" mode. Once signal quality recovers, it automatically switches back to fusion mode.

[0065] A second control instruction is generated based on the fusion or arbitration result.

[0066] Step S408: Output control signal: The control output module 108 outputs a corresponding control signal according to the generated control command. This signal can be used to drive external devices such as rehabilitation robots and prostheses, and can also be used to update the content and status of interactive interfaces such as virtual reality scenes and rehabilitation software interfaces to realize the user's intentions.

[0067] Step S409, Loop Judgment: The system determines whether the current training or control task has ended. If it has not ended, the process returns to step S402 and continues to the next trial; if it has ended, the current run is terminated.

[0068] Through the above embodiments, this invention achieves a seamless transition from stable control relying on SSMVEP in the initial stage to high-precision dual-modal fusion control in the later stage. Local stimulus activation and spatial partitioning cue logic significantly reduce visual interference and enhance MI signals; adaptive decoding path selection based on sample number ensures the availability and performance growth of the system throughout the entire process; and the fusion and switching mechanism based on signal quality ensures the robustness of the system in the face of real-world interference.

[0069] The advantages of the eye-tracking-based adaptive SSMVEP–MI fusion brain-computer interface method and system of the present invention compared with the prior art are as follows: 1. Significantly improves MI recognition accuracy in the early stages of training: The eye-tracking-SSMVEP-MI three-channel consistency enhances the MI signal characteristics, enabling the system to achieve accuracy far exceeding that of traditional MI BCI with very few training samples.

[0070] 2. Effectively reduces visual stimuli, fatigue, and risks: Local activation of SSMVEP stimulation reduces irrelevant flicker, improving comfort and long-term usability.

[0071] 3. Significantly reduce MI annotation data collection time: Through automated and synchronized tag generation, reduce MI data collection time by 50%–80%.

[0072] 4. Stable and reliable fusion decoding: Supports mechanisms such as eye movement loss compensation and SSMVEP signal failure switching, improving the reliability of practical applications.

[0073] 5. Suitable for multiple application scenarios such as rehabilitation, interaction, and enhanced control: It can significantly improve the performance of intent recognition in complex control tasks.

[0074] It is understood that the above specific embodiments are merely examples, and the scope of protection of this invention is not limited to the above embodiments. For example, the gaze area can be divided into upper, middle, and lower quadrants, or a finer grid (four quadrants, etc.), as long as the number is at least three; MI cues can also take other forms such as tactile vibration. Any modifications, equivalent substitutions, and improvements made within the principles and design concepts disclosed in this invention should be included within the scope of protection of the claims of this invention.

Claims

1. An adaptive SSMVEP–MI fusion brain-computer interface method based on eye tracking, characterized in that, Includes the following steps: The coordinates of the user's gaze point are obtained in real time and mapped to at least three preset gaze regions to determine the current effective gaze region; In response to the current effective gaze area, local stimulus control is performed: only one local SSMVEP visual stimulus target corresponding to the current effective gaze area is activated, and specific motion imagery cue information associated with that area is generated; wherein, SSMVEP visual stimulus targets outside the current effective gaze area are inactive. Simultaneously collect the user's electroencephalogram (EEG) signals; From the EEG signals, SSMVEP features related to the local SSMVEP visual stimulus target and MI features related to the motor imagery cue information are extracted respectively. Determine whether the number of valid MI samples that have been collected and labeled with the aforementioned motion imagery prompt information has reached a preset threshold; Perform adaptive decoding based on the judgment result: If the number of valid MI samples does not reach the preset threshold, then the first control command is generated based on the SSMVEP feature. If the number of valid MI samples reaches a preset threshold, then the second control command is generated based on the fusion result of the SSMVEP feature and the MI feature. Output control signals to drive associated external devices or update the interactive interface according to the first control command or the second control command.

2. The adaptive SSMVEP–MI fusion brain-computer interface method based on eye tracking according to claim 1, characterized in that, The decoding based on the fusion result of SSMVEP features and MI features includes: A first identification result and its first quality index based on the SSMVEP feature are obtained, as well as a second identification result and its second quality index based on the MI feature. Based on the first quality index and the second quality index, the weights of the first identification result and the second identification result in the final decision are dynamically adjusted and weighted fusion is performed.

3. The adaptive SSMVEP–MI fusion brain-computer interface method based on eye tracking according to claim 2, characterized in that, It also includes an adaptive mode switching step: Continuously monitor the first and second quality indicators; When the first quality index continues to be lower than the first switching threshold, the activation of the local SSMVEP visual stimulus target is paused, and the mode of decoding based on the MI feature is switched. When the second quality index remains below the second switching threshold, the system switches to a mode that decodes based on the SSMVEP feature.

4. The adaptive SSMVEP–MI fusion brain-computer interface method based on eye tracking according to claim 2 or 3, characterized in that, The first quality metric includes the signal-to-noise ratio or classification confidence of the SSMVEP feature; the second quality metric includes the classification confidence of the MI feature.

5. The method according to claim 1, characterized in that, The specific motion imagery prompt is at least one of visual, auditory, or tactile prompts, used to guide the user to perform a specific motion imagery task associated with the currently effective gaze area.

6. An adaptive SSMVEP-MI fusion brain-computer interface system based on eye tracking, used to implement the adaptive SSMVEP-MI fusion brain-computer interface method based on eye tracking as described in any one of claims 1-5, characterized in that, The system includes: The eye-tracking module is configured to acquire the coordinates of the user's gaze point in real time. The gaze region determination module is configured to map the coordinates of the gaze point to at least three preset gaze regions and output the currently valid gaze region; The stimulus presentation and control module is configured to, in response to the current effective gaze area, activate only one local SSMVEP visual stimulus target corresponding to the current effective gaze area and generate associated specific motion imagery cue information; wherein, SSMVEP visual stimulus targets outside the current effective gaze area are inactive. The EEG acquisition module is configured to acquire the user's EEG signals. The feature extraction module is configured to extract SSMVEP features and MI features from the electroencephalogram (EEG) signals, respectively. The adaptive fusion decoding module is configured as follows: Record and determine whether the number of valid MI samples labeled with the aforementioned motion imagery prompts reaches a preset threshold; When the number of valid MI samples does not reach the preset threshold, decoding is performed based on the SSMVEP feature to generate a first control command. When the number of valid MI samples reaches a preset threshold, the second control command is generated based on the fusion result of the SSMVEP feature and the MI feature. The control output module is configured to output control signals to drive associated external devices or update the interactive interface according to the first control command or the second control command.

7. The adaptive SSMVEP–MI fusion brain-computer interface system based on eye tracking according to claim 6, characterized in that, The adaptive fusion decoding module includes a quality assessment unit and a fusion decision unit; The quality assessment unit is configured to assess a first quality index corresponding to the SSMVEP feature and a second quality index corresponding to the MI feature. The fusion decision unit is configured to: when the number of valid MI samples reaches a preset threshold, acquire a first identification result based on the SSMVEP feature and a second identification result based on the MI feature, and perform weighted fusion of the two based on the first quality index and the second quality index.

8. The adaptive SSMVEP–MI fusion brain-computer interface system based on eye tracking according to claim 7, characterized in that, The adaptive fusion decoding module is also configured to perform adaptive mode switching: When the first quality index is continuously lower than the first switching threshold, an instruction is sent to the stimulus presentation and control module to pause the SSMVEP visual stimulus and control the fusion decision unit to switch to the mode of decoding based on the MI feature. When the second quality index continues to be lower than the second switching threshold, the fusion decision unit is controlled to switch to the mode of decoding based on the SSMVEP features.

9. The adaptive SSMVEP–MI fusion brain-computer interface system based on eye tracking according to claim 6, characterized in that, The system also includes a motion imagery prompting module, configured to generate specific motion imagery prompting information in visual, auditory, or tactile form according to the instructions of the stimulus presentation and control module.

10. The adaptive SSMVEP–MI fusion brain-computer interface system based on eye tracking according to claim 6, characterized in that, The adaptive fusion decoding module is configured to: each time the local SSMVEP visual stimulus target is activated and the EEG signal acquisition is effective, associate and store the synchronously acquired EEG signal segment with the motor imagery task label determined by the current effective gaze area, and accumulate it as a valid MI sample.

Images