Adaptive mode switching method and system based on SSMVEP-MI fusion brain-computer interface system, and storage medium

Abstract

The invention provides an adaptive mode switching method and system based on an SSMVEP-MI fusion brain-computer interface system and a storage medium, and belongs to the technical field of brain-computer interfaces. According to the method, the SSMVEP signal quality, the MI signal quality and the eye movement tracking quality are monitored in real time and compared with preset threshold values, and the system is driven to be intelligently switched among three modes of bimodal fusion, SSMVEP dominant mode and MI dominant mode. And in the fusion mode, dynamically distributing fusion weights according to the real-time quality score. The problems that an existing fusion brain-computer interface is poor in robustness and prone to control interruption when signal interference or single-mode failure occurs are solved, fault sensing, autonomous fault tolerance and refined decision making are achieved, the control continuity and reliability of a system in a complex environment are remarkably improved, and the method is particularly suitable for long-time and high-requirement application scenes such as neural rehabilitation.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface technology, and in particular to an adaptive mode switching method, system, and storage medium based on the SSMVEP–MI fusion brain-computer interface system. Background Technology Brain-computer interface (BCI) technology offers a revolutionary approach to neurological rehabilitation and alternative control. Currently, there are two main technical approaches: one is based on steady-state visual evoked potentials (SSMVEP / SSMVEP), which has the advantages of strong signals and less training, but full-screen stimulation can easily lead to visual fatigue and interference; the other is based on motor imagery (MI), which can actively activate the motor cortex and promote neuroplasticity, but suffers from low initial recognition rates and long training times.

[0002] To combine the advantages of both, existing technologies attempt to fuse ssmvep and MI. However, most existing fusion methods are static or simple weighted fusions, lacking the ability to adapt to environmental changes and user physiological states. Their main drawbacks include: 1. Static fusion strategy has poor robustness: When the signal quality of a certain modality (such as SSMVEP due to blinking, gaze shift, or MI due to attentional distraction) temporarily decreases, the fixed-weight fusion method will introduce noise, causing the overall recognition performance to drop sharply or even interrupt the control.

[0003] 2. Lack of fault detection and switching mechanism: The existing system cannot detect signal quality degradation events in real time. When critical signals (such as eye tracking or SSMVEP signals) completely fail, the system cannot switch to an available backup mode on its own, resulting in user experience interruption.

[0004] 3. Coarse-grained fusion decision-making: When both dual-modal signals are available, the system fails to perform refined and dynamic weight allocation based on the real-time "credibility" of the signals (such as signal-to-noise ratio and classification confidence), which limits further improvement in fusion accuracy.

[0005] Therefore, there is an urgent need for a fusion mechanism that can sense the quality of multimodal signals in real time and make intelligent decisions and adaptive switching accordingly, in order to solve the problems of insufficient robustness and discontinuous control in existing technologies. Summary of the Invention

[0006] The purpose of this invention is to provide an adaptive mode switching method, system, and storage medium based on the SSMVEP–MI fusion brain-computer interface system, which can at least solve one of the existing problems.

[0007] The technical solution of the present invention is as follows: According to a first aspect of the present invention, an adaptive mode switching method based on an SSMVEP–MI fusion brain-computer interface system is disclosed, comprising at least: Real-time signal quality monitoring steps: Real-time calculation and output of a normalized first score characterizing SSMVEP signal quality, a second score characterizing MI signal quality, and a third score characterizing eye-tracking quality; Adaptive decision-making steps: Compare the first score, second score, and third score with the preset first threshold, second threshold, and third threshold, respectively; Mode switching execution steps: Based on the comparison results, the control system switches between dual-modal fusion mode, SSMVEP-dominated mode and MI-dominated mode; Specifically, when the third score is lower than the third threshold and the first score is lower than the first threshold, the switch to MI-dominated mode is triggered. When the second score falls below the second threshold, a switch to SSMVEP-dominated mode is triggered.

[0008] In some implementations, in the dual-modal fusion mode, the system dynamically calculates the weights of the SSMVEP mode and the fusion weights of the MI mode based on the first score, the second score, and the third score, and performs weighted fusion of the decoding results of the two modes based on the two weights to generate the final control command.

[0009] In some implementations, the dynamic calculation of fusion weights employs a normalization strategy based on linear fractions, where the weights w of the SSMVEP modes... ssmvep and the weights w of the MI mode mi Calculate using the following formulas respectively:

[0010] Wherein, SNR_ssmvep is the first score, GSS is the second score, Conf_MI is the third score, and α, β, γ are preset non-negative weighting coefficients; The weighting coefficients α, β, and γ take values ​​in the following ranges: α∈[0.4, 1.0], β∈[0.1, 0.6], and γ∈[0.5, 1.2], respectively, and α+β+γ=C, where C is a constant.

[0011] In some implementations, the first score is the SSMVEP signal-to-noise ratio score normalized to the [0-1] interval, calculated based on the ratio of signal power to background noise power at the stimulus frequency; the second score is the MI classification confidence score normalized to the [0-1] interval, obtained based on the maximum posterior probability value output by the classifier; and the third score is the eye movement stability score normalized to the [0-1] interval, calculated based on the standard deviation of the fixation point coordinates.

[0012] In some implementations, the third score is obtained through an eye-tracking module, which is used to acquire images of the user's eyes in real time and calculate fixation parameters.

[0013] In some implementations, the values ​​of the first threshold, the second threshold, and the third threshold are all in the range of 0.1-0.6.

[0014] In some implementations, the first threshold ranges from 0.2 to 0.4 of the SSMVEP signal-to-noise ratio score; the second threshold ranges from 0.3 to 0.5 of the MI classification confidence score; and the third threshold ranges from 0.2 to 0.4 of the eye movement stability score.

[0015] In some implementations, the mode switching execution step also includes a recovery mechanism: When the system is in MI-dominated mode, if the third score is detected to be higher than the third threshold and the first score is higher than the first threshold, the system will trigger a switch back to dual-modal fusion mode. When the system is in SSMVEP dominant mode, if the second score is detected to be higher than the second threshold, it will trigger a switch back to dual-modal fusion mode.

[0016] In some implementations, the adaptive mode switching method based on the SSMVEP–MI fusion brain-computer interface system also includes a mode initialization step: after the system starts, it defaults to the dual-modal fusion mode, and in this mode, the SSMVEP decoding results are used to provide training labels for the synchronously acquired MI EEG data.

[0017] In some implementations, the fusion decoding and control module is further configured to: when the SSMVEP visual stimulus target is activated and the EEG signal acquisition is effective, associate the synchronously acquired MI EEG signal fragments with the motor imagery task labels determined by the currently effective gaze area, store and accumulate them as labeled MI training samples, and use them for online updates of the MI decoding model.

[0018] According to a second aspect of the present invention, an SSMVEP–MI fusion brain-computer interface system is also disclosed, characterized in that it comprises at least: The EEG acquisition module is used to acquire the user's EEG signals; The eye-tracking module is used to collect the user's eye movement signals in real time; The stimulus and cue presentation module is used to present SSMVEP stimuli and motion imagery cues to the user; The signal processing module is used to extract and decode SSMVEP and MI features from EEG signals; The signal quality monitoring module is used to calculate and output normalized scores in real time, representing the first score of SSMVEP signal quality, the second score of MI signal quality, and the third score of eye-tracking quality. An adaptive mode switching module, connected to the signal quality monitoring module, is used to compare the first score, the second score, and the third score with preset thresholds, and output system operating mode instructions based on the comparison results. The operating modes include: dual-modal fusion mode, SSMVEP-dominated mode, and MI-dominated mode. Specifically, when the third score is lower than the third threshold and the first score is lower than the first threshold, an instruction to switch to the MI-dominated mode is output; when the second score is lower than the second threshold, an instruction to switch to the SSMVEP-dominated mode is output. The integrated decoding and control module is connected to the signal processing module and the adaptive mode switching module. It is used to select the appropriate decoding strategy and generate control commands according to the system's operating mode instructions.

[0019] In some implementations, the adaptive mode switching module includes a state machine, whose states include a dual-modal fusion state, an SSMVEP-dominated state, and a MI-dominated state. The state transition is driven by a comparison logic between the score output by the signal quality monitoring module and a preset threshold.

[0020] In some implementations, the stimulus and cue presentation module is connected to the eye-tracking module to activate local SSMVEP stimuli only in the corresponding area that the user is looking at, based on the gaze area information, and simultaneously present other motion imagery cues on the same side. In some implementations, the fusion decoding and control module and / or signal processing module are further configured to: When the SSMVEP stimulus target is successfully decoded, the synchronously acquired MI EEG signal fragments are associated with the motor imagery task labels corresponding to the current decoding results, and used as labeled MI training samples to update the MI decoding model.

[0021] According to a third aspect of the present invention, a computer-readable storage medium is also disclosed, on which a computer program is stored, which, when executed by a processor, implements the above-described adaptive mode switching method based on the SSMVEP–MI fusion brain-computer interface system.

[0022] The beneficial effects of this invention are as follows: 1. Achieve real-time adaptation and significantly improve system robustness: Through real-time monitoring and threshold judgment, the system can actively perceive and respond to changes in signal quality caused by eye movement loss, visual interference, user fatigue, etc., and automatically switch to a reliable working mode, fundamentally solving the problem of performance drop when the existing static fusion method fluctuates.

[0023] 2. Ensure control continuity under extreme conditions: The mode switching process is smooth and fast, ensuring that the brain-computer interface control task is not interrupted in extreme cases where SSMVEP or MI monomodal mode temporarily fails, greatly improving the availability of the system and the user experience.

[0024] 3. Achieve intelligent and precise decision-making: When both modes are available, a dynamic weight model is introduced, enabling the system to precisely allocate decision weights based on the real-time "credibility" of the signal, achieving higher accuracy and situational adaptability than a fixed fusion strategy.

[0025] 4. Enhance the practical application potential of the technology: This method enables the SSMVEP-MI fusion system to cope with complex and ever-changing real-world environments, promoting the technology from the laboratory to clinical and daily applications. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of dynamic fusion weight calculation in one embodiment of the present invention.

[0027] Figure 2 This is a flowchart illustrating the overall process of an adaptive mode switching method provided in one embodiment of the present invention.

[0028] Figure 3 This is a structural block diagram of the SSMVEP-MI fusion brain-computer interface system provided in one embodiment of the present invention.

[0029] Figure 4 This is a state transition diagram of the internal state machine of the adaptive mode switching module in one embodiment of the present invention.

[0030] Figure 5 This is a flowchart illustrating the recovery mechanism included in the mode switching execution step in one embodiment of the present invention.

[0031] Figure 6 This is a schematic diagram of the online self-learning and tag generation process shown in Embodiment 3 of the present invention. Detailed Implementation

[0032] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0033] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0034] like Figure 1-6 As shown, this invention provides an adaptive mode switching method, system, and storage medium based on a SSMVEP–MI fusion brain-computer interface system, belonging to the field of brain-computer interface (BCI) technology. Specifically, it relates to a method for decoding and fusing multimodal EEG signals (especially steady-state motor visual evoked potentials (SSMVEP) and motor imagery (MI)), and particularly an adaptive mode switching mechanism based on real-time signal quality monitoring to improve the robustness, control continuity, and interaction reliability of the BCI system in complex and variable environments. This invention can be applied to neurorehabilitation therapy, assistive control devices (such as prostheses and rehabilitation robots), and high-precision human-computer interaction.

[0035] like Figure 1-2 and Figure 4-6 As shown, this invention discloses an adaptive mode switching method based on an SSMVEP–MI fusion brain-computer interface system, comprising at least: Real-time signal quality monitoring steps: Real-time calculation and output of normalized first score SNR_ssmvep representing SSMVEP signal quality, second score Conf_Mi representing MI signal quality, and third score GSS representing eye tracking quality; Adaptive decision-making steps: Compare the first score SNR_ssmvep, the second score Conf_Mi, and the third score GSS with the preset first threshold Th_ssmvep, the second threshold Th_mi, and the third threshold Th_gaze, respectively; Mode switching execution steps: Based on the comparison results, the control system switches between dual-modal fusion mode, SSMVEP-dominated mode and MI-dominated mode; Specifically, when the third score is lower than the third threshold and the first score is lower than the first threshold, the switch to MI-dominated mode is triggered. When the second score falls below the second threshold, a switch to SSMVEP-dominated mode is triggered.

[0036] In some implementations, in the dual-modal fusion mode, the system dynamically calculates the weights of the SSMVEP mode and the fusion weights of the MI mode based on the first score, the second score, and the third score, and performs weighted fusion of the decoding results of the two modes based on the two weights to generate the final control command.

[0037] In some implementations, the dynamic calculation of fusion weights employs a normalization strategy based on linear fractions, where the weights w of the SSMVEP modes... ssmvep and the weights w of the MI mode mi Calculate using the following formulas respectively:

[0038] Wherein, SNR_ssmvep is the first score, GSS is the second score, Conf_MI is the third score, and α, β, γ are preset non-negative weighting coefficients; The weighting coefficients α, β, and γ take values ​​in the following ranges: α∈[0.4, 1.0], β∈[0.1, 0.6], and γ∈[0.5, 1.2], respectively, and α+β+γ=C, where C is a constant.

[0039] Preferably, the first score is the SSMVEP signal-to-noise ratio score normalized to the [0-1] interval, calculated based on the ratio of signal power to background noise power at the stimulus frequency; the second score is the MI classification confidence score normalized to the [0-1] interval, obtained based on the maximum posterior probability value output by the classifier; and the third score is the eye movement stability score normalized to the [0-1] interval, calculated based on the standard deviation of the fixation point coordinates.

[0040] As a preferred option, the third score is obtained through an eye-tracking module, which is used to acquire images of the user's eyes in real time and calculate the gaze point parameters.

[0041] Preferably, the values ​​of the first threshold, the second threshold, and the third threshold are all in the range of 0.1-0.6.

[0042] As a further preferred option, the first threshold value range corresponds to 0.2-0.4 of the SSMVEP signal-to-noise ratio score; the second threshold value range corresponds to 0.3-0.5 of the MI classification confidence score; and the third threshold value range corresponds to 0.2-0.4 of the eye movement stability score.

[0043] Preferably, the mode switching execution steps also include a recovery mechanism: When the system is in MI-dominated mode, if the third score is detected to be higher than the third threshold and the first score is higher than the first threshold, the system will trigger a switch back to dual-modal fusion mode. When the system is in SSMVEP dominant mode, if the second score is detected to be higher than the second threshold, it will trigger a switch back to dual-modal fusion mode.

[0044] As a preferred embodiment, the adaptive mode switching method based on the SSMVEP–MI fusion brain-computer interface system also includes a mode initialization step: after the system starts, it enters the dual-modal fusion mode by default, and in this mode, the SSMVEP decoding results are used to provide training labels for the synchronously acquired MI EEG data.

[0045] Preferably, the fusion decoding and control module is also configured to: when the SSMVEP visual stimulus target is activated and the EEG signal acquisition is effective, associate the synchronously acquired MI EEG signal fragments with the motor imagery task labels determined by the current effective gaze area, store and accumulate them as labeled MI training samples, and use them for online updates of the MI decoding model.

[0046] like Figure 3 As shown, this invention discloses an SSMVEP-MI fusion brain-computer interface system, characterized in that it includes at least: The EEG acquisition module is used to acquire the user's EEG signals; The eye-tracking module is used to collect the user's eye movement signals in real time; The stimulus and cue presentation module is used to present SSMVEP stimuli and motion imagery cues to the user; The signal processing module is used to extract and decode SSMVEP and MI features from EEG signals; The signal quality monitoring module is used to calculate and output normalized scores in real time, representing the first score of SSMVEP signal quality, the second score of MI signal quality, and the third score of eye-tracking quality. An adaptive mode switching module, connected to the signal quality monitoring module, is used to compare the first score, the second score, and the third score with preset thresholds, and output system operating mode instructions based on the comparison results. The operating modes include: dual-modal fusion mode, SSMVEP-dominated mode, and MI-dominated mode. Specifically, when the third score is lower than the third threshold and the first score is lower than the first threshold, an instruction to switch to the MI-dominated mode is output; when the second score is lower than the second threshold, an instruction to switch to the SSMVEP-dominated mode is output. The integrated decoding and control module is connected to the signal processing module and the adaptive mode switching module. It is used to select the appropriate decoding strategy and generate control commands according to the system's operating mode instructions.

[0047] Preferably, the adaptive mode switching module has a state machine. The states of the state machine include dual-modal fusion state, SSMVEP dominant state, and MI dominant state. The state transition is driven by the comparison logic between the score output by the signal quality monitoring module and the preset threshold.

[0048] Preferably, the stimulus and cue presentation module is connected to the eye-tracking module to activate local SSMVEP stimulation only in the corresponding area that the user is looking at, based on the gaze area information, and simultaneously present other motion imagery cues on the same side. Preferably, the fusion decoding and control module and / or signal processing module are further configured as follows: When the SSMVEP stimulus target is successfully decoded, the synchronously acquired MI EEG signal fragments are associated with the motor imagery task labels corresponding to the current decoding results, and used as labeled MI training samples to update the MI decoding model.

[0049] Preferably, the eye-tracking module is configured to acquire the user's gaze coordinates in real time. The eye-tracking module is used to acquire binocular images in real time and calculate the gaze point, gaze stability, and gaze area. The eye-tracking module includes a camera, infrared illumination, a pupil detection algorithm, and a gaze mapping algorithm. The eye-tracking module can refer to the Tobii device in CN113419628A. The gaze point refers to the focal position of the eye's gaze, commonly used in eye tracking or visual analysis, representing the specific coordinates of the line of sight on a screen or in space. In eye-tracking technology, the gaze point is calculated by capturing eye movements through sensors or cameras and determining the projection point of the line of sight onto a target surface (such as a screen). The gaze mapping algorithm is mainly used to estimate the direction of human eye gaze or the gaze point; common methods include geometric methods, appearance methods, and hybrid methods.

[0050] The gaze region determination module is configured to map the gaze point coordinates to at least three preset gaze regions and output the currently valid gaze regions; for example, the gaze region determination module 102 is used to map the user's gaze point to three predefined regions on the left, center and right of the screen and output parameters such as gaze confidence and gaze stabilization time.

[0051] The stimulus presentation and control module is configured to activate only one local SSMVEP visual stimulus target corresponding to the current effective gaze area in response to the current effective gaze area, and generate associated specific motion imagery cue information; wherein, SSMVEP visual stimulus targets outside the current effective gaze area are inactive. The EEG acquisition module is configured to acquire the user's EEG signals; preferably, it is an EEG acquisition device that can acquire EEG signals from the user's occipital lobe, central lobe, motor cortex, and other locations, and supports acquisition of 8 to 32 channels.

[0052] The feature extraction module is configured to extract SSMVEP features and MI features from the EEG signals, respectively; The fusion decoding and control module is configured as follows: Record and determine whether the number of valid MI samples labeled with motion imagery prompts has reached a preset threshold. When the number of valid MI samples does not reach the preset threshold, the first control command is generated based on the SSMVEP feature. When the number of valid MI samples reaches a preset threshold, the second control command is generated by decoding based on the fusion result of SSMVEP features and MI features. Output control signals to drive associated external devices or update the interactive interface according to a first control command or a second control command.

[0053] Preferably, the motion imagery cue is at least one of a visual cue, an auditory cue, or a tactile cue, used to guide the user to perform a specific motion imagery task associated with the currently effective gaze area.

[0054] This invention discloses a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the above-described adaptive mode switching method based on the SSMVEP–MI fusion brain-computer interface system.

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the following embodiments are for illustrative purposes only and are not intended to limit the invention.

[0056] Example 1: Core Process of Adaptive Mode Switching Method This embodiment elaborates on the core of the invention—the adaptive mode switching mechanism. For example... Figure 2 As shown, this method is continuously executed in a loop during system operation and mainly includes the following core steps: Step 1: Real-time quantization of signal quality The system runs three quality assessment threads in parallel: SSMVEP quality assessment: Spectral analysis is performed on the EEG signal in the current time window (e.g., the most recent 1 second). The power P_signal at the target stimulus frequency and its harmonics is calculated and compared with the average power P_noise of the adjacent background frequency band to obtain the raw signal-to-noise ratio SNR_raw. This is then normalized to the [0,1] interval using a sigmoid function or linear mapping to obtain the first score SNR_ssmvep. For example, SNR_raw is linearly mapped in the range of 0dB-15dB.

[0057] MI quality assessment: Input MI-related features (such as ERS / ERD of C3 and C4 channels) from the current time window (e.g., the last 2 seconds) into a pre-trained MI classifier (e.g., EEGNet). Take the maximum value P_max in the softmax probability vector output by the classifier as the original confidence score. Since P_max itself is in [0,1], it can be directly used as the second score Conf_Mi, or it can be fine-tuned through a calibration function.

[0058] Eye movement quality assessment: Obtain the screen coordinates (x_i, y_i) of the fixation points for the most recent N sampling points (e.g., corresponding to 200 ms). Calculate their standard deviations σ_x and σ_y, and the gaze stability Gaze_Stability = 1 / (1 + k * (σ_x + σ_y)), where k is the scaling factor. At the same time, obtain the confidence score Conf_gaze provided by the eye tracker. Finally, the third score GSS = Gaze_Stability * Conf_gaze.

[0059] Step 2: Threshold-based adaptive decision logic The system pre-sets three configurable thresholds: Th_ssmvep (e.g., 0.3), Th_mi (e.g., 0.4), Th_gaze (e.g., 0.25). The decision logic is as follows (see Figure 4 state machine): Initial / default state: After the system starts up, it defaults to the "dual-modal fusion mode".

[0060] Trigger the SSMVEP-dominant mode: If it is detected that Conf_Mi < Th_mi, indicating that the current MI signal is unreliable, immediately migrate to the "SSMVEP-dominant state" regardless of the visual signal. In this state, the fusion decoding and control module sets w_mi to 0 and w_ssmvep to 1.

[0061] Trigger the MI-dominant mode: If it is detected that GSS < Th_gaze and SNR_ssmvep < Th_ssmvep, indicating that eye movement tracking fails (e.g., the user closes their eyes or their line of sight leaves the screen) and the visual evoked potential signal is also weak, immediately migrate to the "MI-dominant state". In this state, w_ssmvep is set to 0 and w_mi is set to 1.

[0062] Maintain or restore to the fusion mode: When the system is in the single-modal dominant state, continuously monitor the relevant scores. For example, in the MI-dominant state, if GSS recovers to ≥ Th_gaze and SNR_ssmvep recovers to ≥ Th_ssmvep, then switch back to the "dual-modal fusion state". In the SSMVEP-dominant state, if Conf_Mi recovers to ≥ Th_mi, then switch back to the "dual-modal fusion state".

[0063] Step 3: Dynamic weight fusion (in the fusion mode) As Figure 1 shown, when the system is in the dual-modal fusion mode, the following formula is used to dynamically calculate the fusion weights: w_ssmvep = (α*SNR_ssmvep + γ*GSS) / Z w_mi=(β*Conf_Mi) / Z Where Z = α*SNR_ssmvep + β* Conf_Mi + γ*GSS is the normalization factor.

[0064] The weighting coefficients α, β, and γ can be optimized based on experimental data. A typical setting is: α = 0.7 (emphasizing the quality of SSMVEP itself), β = 0.3 (emphasizing the quality of MI itself), and γ = 0.5 (emphasizing the contribution of eye movement stability).

[0065] Mechanism Explanation: When both SNR_ssmvep and GSS are high, w_ssmvep increases, and the system trusts SSMVEP decoding more.

[0066] When Conf_Mi is high, w_mi increases, and the system trusts MI decoding more.

[0067] When GSS is low (instable gaze), w_ssmvep will also decrease (because SSMVEP requires fixation), even if SNR_ssmvep is high. This reflects the "veto" tendency of eye movement quality on visual modality, which is consistent with physiological logic.

[0068] Example 2: A complete brain-computer interface system including a mode switching mechanism like Figure 3 As shown, this embodiment provides an SSMVEP-MI fusion brain-computer interface system that implements the above method, specifically including: EEG acquisition module: Employs a multi-channel (e.g., 16-channel) EEG acquisition device, focusing on covering the occipital lobe (O1, Oz, O2 for SSMVEP) and the sensorimotor cortex (C3, Cz, C4 for MI).

[0069] Eye-tracking module: Employs an infrared camera eye tracker (such as the Tobii series), integrated below the display, to output the user's real-time gaze coordinates (x_gaze, y_gaze) and confidence level at a frequency of at least 60Hz.

[0070] Stimulus and Cue Presentation Module: Running on the main control computer, it generates a graphical user interface. The interface is divided into three logical areas: left, center, and right. The key innovation lies in the fact that this module receives the gaze area judgment result (such as "left") from the eye-tracking module, activates only the local SSMVEP stimulus of the corresponding area (such as a graphic that contracts / rotates at a specific frequency), and simultaneously presents the corresponding MI cue on the side of the screen (such as highlighting the "Left Hand Imagination" icon).

[0071] Signal processing module: contains two parallel processing pipelines.

[0072] The SSMVEP pipeline performs bandpass filtering (e.g., 5-30Hz) on the occipital channel signal, uses a short time window (e.g., 1 second) of canonical correlation analysis (CCA) or filter bank CCA (FBCCA) algorithm to identify the target frequency that the user is looking at, and outputs the decoding result command _ssmvep (e.g., left, center, right).

[0073] The MI pipeline performs bandpass filtering (e.g., 8-30Hz) on the sensorimotor cortex channel signal, extracts event-related desynchronization / synchronization (ERD / ERS) features, classifies them using a lightweight classifier (e.g., EEGNet), and outputs the decoding result command _mi (e.g., left hand, right hand).

[0074] Signal quality monitoring module: As described in Example 1, it receives the raw signal and intermediate processing results in real time, calculates and outputs three scores: SNR_ssmvep, Conf_Mi, and GSS.

[0075] Adaptive mode switching module: The core of this module is a finite state machine (FSM), and its state transition diagram is as follows. Figure 4 As shown. It receives three quality scores, drives the state to transition between "fusion", "SSMVEP dominant", and "MI dominant" according to the decision logic described in Example 1, and outputs an instruction (Mode_Cmd) representing the current mode.

[0076] The fusion decoding and control module receives the commands _ssmvep, _mi, and Mode_Cmd. It selects the operation based on Mode_Cmd. If it is in "fusion mode", the weight calculation unit is invoked to calculate w_ssmvep and w_mi based on the current three quality scores, and then weighted fusion is performed.

[0077] If it is "SSMVEP dominant mode", then the command _ssmvep will be output directly.

[0078] If it is "MI-dominant mode", the command _mi will be output directly.

[0079] The final generated control commands are used to drive external devices, such as robotic arms, wheelchairs, or characters in virtual reality scenes. The final generated control commands can also be used to control and switch interactive interfaces.

[0080] Example 3: Initialization process including online self-learning and tag generation This embodiment further details how the system initialization phase works in conjunction with the adaptive mode switching mechanism to address the issues of limited training data and difficulty in obtaining labels during the initial stages of MI.

[0081] like Figure 6As shown, after the system starts and completes device calibration, it defaults to dual-modal fusion mode. In the initial stage, because the MI model has not been fully trained, its Conf_Mi score may be naturally low, easily triggering a switch to "SSMVEP-dominated mode". But this is precisely the ingenious aspect of the invention: In SSMVEP-dominant mode, the system relies on the highly reliable SSMVEP signal for stable control.

[0082] Key steps: Each time SSMVEP successfully decodes a valid instruction (e.g., "left"), the system will automatically tag the MI EEG signal segments collected within a time window before and after triggering the instruction (e.g., 0.5 seconds before the instruction to 2.5 seconds after the instruction) with the motor imagery label corresponding to the instruction (e.g., "left hand imagery").

[0083] These automatically generated, high-quality label data are stored in a buffer pool for online incremental updates of the MI classifier model.

[0084] As training samples accumulate, the performance of the MI classifier gradually improves, and the Conf_Mi score also increases accordingly.

[0085] Once Conf_Mi stably exceeds the second threshold Th_mi, the system can run more frequently and stably in the true "dual-modal fusion mode" and enjoy the performance improvement brought by fusion.

[0086] This process expands the adaptive mode switching mechanism from a simple "fault response" strategy into an enabling framework that supports the system to learn and optimize online from scratch, greatly enhancing the system's practicality and ease of use.

[0087] Example 4: Computer-readable storage medium This embodiment provides a computer-readable storage medium, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, on which a computer program is stored. When this program is executed by a processor (such as the CPU of a computer, embedded system, or a dedicated main control chip for brain-computer interfaces), it can implement all the steps of the adaptive mode switching method described in any of embodiments 1, 2, and 3 above. For example, the program module may include code segments for implementing functions such as signal quality assessment, threshold comparison, state machine logic, and weight calculation.

[0088] It is important to emphasize that the innovative aspect of the adaptive mode switching mechanism described in this invention lies in its organic integration of signal quality quantization, threshold-based multimodal decision-making logic, and dynamic weight allocation into a closed-loop, adaptive control system. It is not merely an improvement at the algorithm level, but rather a manifestation of system-level reliability design principles. Any adaptive modifications and substitutions made by those skilled in the art, based on the core idea of ​​this invention (i.e., adaptive mode switching and fusion based on real-time multimodal signal quality), to the threshold setting method, weight calculation formula, quality score calculation method, etc., should be included within the scope of protection of this invention.

[0089] The advantages of the adaptive mode switching method and system based on the SSMVEP–MI fusion brain-computer interface system of the present invention compared with the prior art are as follows: 1. The system's robustness has been fundamentally improved: By monitoring the quality of multimodal signals in real time and performing adaptive mode switching, the system possesses "fault perception" and "autonomous fault tolerance" capabilities. When a mode fails due to environmental interference or poor user conditions, the system can seamlessly switch to another reliable mode, fundamentally avoiding control interruption and ensuring control continuity under extreme conditions.

[0090] 2. A refined intelligent decision-making mechanism is provided: When both modalities are available, a dynamic weight fusion model based on real-time quality scoring is introduced. This model can finely allocate decision weights according to the instantaneous "credibility" of the signal, significantly improving the accuracy and context adaptability of fusion decoding compared to fixed weights or simple voting strategies.

[0091] 3. Enhanced the practical potential of the technology: This invention enables the SSMVEP-MI fusion brain-computer interface system to adapt to complex and ever-changing application scenarios in the real world (such as changes in lighting, user fatigue, temporary visual obstruction, etc.), reducing the dependence on the stability of the external environment and the high degree of standardization of user operation, and greatly promoting the technology from the laboratory to clinical and daily applications.

[0092] 4. Optimized user experience: The adaptive switching process is smooth, fast, and transparent to the user. Users do not need to actively intervene or restart the system when the signal is poor, thus obtaining a smoother and more reliable control experience, which is especially beneficial for rehabilitation training scenarios that require long-term use.

[0093] This invention provides an adaptive mode switching method, system, and storage medium based on an SSMVEP–MI fusion brain-computer interface system, belonging to the field of brain-computer interface technology. The method intelligently switches between three modes—bimodal fusion, SSMVEP-dominant, and MI-dominant—by real-time monitoring of SSMVEP signal quality, MI signal quality, and eye-tracking quality, and comparing them with preset thresholds. In fusion mode, fusion weights are dynamically allocated based on real-time quality scores. This invention solves the problems of poor robustness and easy control interruption in existing fusion brain-computer interfaces when there is signal interference or single-modal failure. It achieves fault perception, autonomous fault tolerance, and refined decision-making, significantly improving the system's control continuity and reliability in complex environments, and is particularly suitable for long-duration, high-requirement applications such as neurorehabilitation.

[0094] It is understood that the above specific embodiments are merely examples, and the scope of protection of this invention is not limited to the above embodiments. For example, the gaze area can be divided into upper, middle, and lower quadrants, or a finer grid (four quadrants, etc.), as long as the number is at least three; MI cues can also take other forms such as tactile vibration. Any modifications, equivalent substitutions, and improvements made within the principles and design concepts disclosed in this invention should be included within the scope of protection of the claims of this invention.

Claims

1. An adaptive mode switching method based on an SSMVEP–MI fusion brain-computer interface system, characterized in that, At least including: Real-time signal quality monitoring steps: Real-time calculation and output of a normalized first score characterizing SSMVEP signal quality, a second score characterizing MI signal quality, and a third score characterizing eye-tracking quality; Adaptive decision-making steps: Compare the first score, the second score, and the third score with preset first threshold, second threshold, and third threshold, respectively; Mode switching execution steps: Based on the comparison results, the control system switches between dual-modal fusion mode, SSMVEP-dominated mode and MI-dominated mode; Specifically, when the third score is lower than the third threshold and the first score is lower than the first threshold, the switch to the MI-dominated mode is triggered. When the second score is lower than the second threshold, a switch to the SSMVEP dominant mode is triggered.

2. The adaptive mode switching method based on the SSMVEP–MI fusion brain-computer interface system according to claim 1, characterized in that, In the dual-modal fusion mode, the system dynamically calculates the weights of the SSMVEP mode and the fusion weights of the MI mode based on the first score, the second score, and the third score, and performs weighted fusion of the decoding results of the two modes according to the two weights to generate the final control command.

3. The adaptive mode switching method based on the SSMVEP–MI fusion brain-computer interface system according to claim 2, characterized in that, The dynamically calculated fusion weights employ a normalization strategy based on linear fractions, and the weights w of the SSMVEP mode... ssmvep and the weights w of the MI mode mi Calculate using the following formulas respectively: Wherein, SNR_ssmvep is the first score, GSS is the second score, Conf_MI is the third score, and α, β, γ are preset non-negative weighting coefficients; The weighting coefficients α, β, and γ take values ​​in the following ranges: α∈[0.4, 1.0], β∈[0.1, 0.6], γ∈[0.5, 1.2], and α+β+γ=C, where C is a constant.

4. The adaptive mode switching method based on the SSMVEP–MI fusion brain-computer interface system according to claim 1, characterized in that, The first score is the SSMVEP signal-to-noise ratio score normalized to the [0-1] interval, calculated based on the ratio of signal power to background noise power at the stimulus frequency; the second score is the MI classification confidence score normalized to the [0-1] interval, obtained based on the maximum posterior probability value output by the classifier; the third score is the eye movement stability score normalized to the [0-1] interval, calculated based on the standard deviation of the fixation point coordinates.

5. The adaptive mode switching method based on the SSMVEP–MI fusion brain-computer interface system according to claim 1, characterized in that, The third score is obtained through an eye-tracking module, which is used to collect user eye images in real time and calculate fixation point parameters.

6. The adaptive mode switching method based on the SSMVEP–MI fusion brain-computer interface system according to any one of claims 1-5, characterized in that, The values ​​of the first threshold, the second threshold, and the third threshold are all in the range of 0.1-0.

6.

7. The adaptive mode switching method based on the SSMVEP–MI fusion brain-computer interface system according to any one of claims 1-5, characterized in that, The mode switching execution step also includes a recovery mechanism: When the system is in the MI-dominated mode, if the third score is detected to be higher than the third threshold and the first score is higher than the first threshold, then a switch back to the dual-modal fusion mode is triggered. When the system is in the SSMVEP dominant mode, if the second score is detected to be higher than the second threshold, a switch back to the dual-modal fusion mode is triggered.

8. The adaptive mode switching method based on the SSMVEP–MI fusion brain-computer interface system according to any one of claims 1-5, characterized in that, It also includes a mode initialization step: after the system starts, it enters the dual-modal fusion mode by default, and in this mode, the SSMVEP decoding results are used to provide training labels for the synchronously acquired MI EEG data.

9. A SSMVEP–MI fused brain-computer interface system, characterized in that, At least including: The EEG acquisition module is used to acquire the user's EEG signals; The eye-tracking module is used to collect the user's eye movement signals in real time; The stimulus and cue presentation module is used to present SSMVEP stimuli and motion imagery cues to the user; The signal processing module is used to extract and decode SSMVEP features and MI features from the electroencephalogram (EEG) signals; The signal quality monitoring module is used to calculate and output normalized scores in real time, representing the first score of SSMVEP signal quality, the second score of MI signal quality, and the third score of eye-tracking quality. An adaptive mode switching module, connected to the signal quality monitoring module, is used to compare the first score, the second score, and the third score with preset thresholds, and output a system operating mode command based on the comparison results. The operating modes include: dual-modal fusion mode, SSMVEP-dominated mode, and MI-dominated mode. Specifically, when the third score is lower than the third threshold and the first score is lower than the first threshold, an instruction to switch to the MI-dominated mode is output; when the second score is lower than the second threshold, an instruction to switch to the SSMVEP-dominated mode is output. The fusion decoding and control module is connected to the signal processing module and the adaptive mode switching module, and is used to select the corresponding decoding strategy and generate control commands according to the system working mode instructions.

10. The SSMVEP–MI fusion brain-computer interface system according to claim 9, characterized in that, The adaptive mode switching module is equipped with a state machine. The states of the state machine include dual-modal fusion state, SSMVEP dominant state, and MI dominant state. The state transition is driven by the comparison logic between the score output by the signal quality monitoring module and the preset threshold.

Images