A brain-computer interface data processing method fusing steady-state motion visual evoked potential and motor imagery

Abstract

This invention relates to the field of brain-computer interface (BCI) data processing technology, and discloses a BCI data processing method that integrates steady-state motor visual evoked potentials and motor imagery. The method includes: generating a bimodal task configuration table; generating a bimodal synchronized stimulation sequence; acquiring and labeling multi-lead EEG signals; extracting visual response features and motor imagery response features; performing motor intention recognition; and generating rehabilitation feedback information. Compared to the single visual evoked paradigm in existing technologies, especially under conditions of significant differences in subjects' motor imagery abilities and low EEG noise ratios, this invention addresses the technical problem of failing to achieve stable motor intention recognition. By synchronously binding motor visual stimulation with motor imagery tasks and jointly extracting visual frequency response and motor imagery desynchronization features, stable recognition of motor intention is achieved, improving the recognition accuracy of rehabilitation BCI training.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface data processing technology, and in particular to a brain-computer interface data processing method that integrates steady-state motion visual evoked potentials and motor imagery. Background Technology

[0002] Currently, brain-computer interfaces based on a single EEG modality, such as steady-state visual evoked potential brain-computer interfaces or motor imagery brain-computer interfaces, have significant shortcomings in rehabilitation applications.

[0003] For example, while steady-state visual evoked potential (BCP) brain-computer interfaces (BCIs) offer high recognition accuracy and short training times, their instruction set is limited by the number of distinguishable stimulus frequencies, and prolonged fixation on flashing stimuli can easily lead to visual fatigue. Motor imagery BCPs, on the other hand, possess a more natural control paradigm, but their recognition accuracy is greatly affected by individual differences, training cycles are long, and stability is relatively poor. Existing technologies cannot fully meet the comprehensive requirements of rehabilitation training for high accuracy, high instruction diversity, low user fatigue, and long-term stability. Therefore, there is an urgent need for a method that can achieve efficient, stable, and user-friendly motor intention recognition even in multimodal fusion scenarios, especially when facing complex rehabilitation tasks, to improve the practicality and efficacy of BCPs in the field of neurorehabilitation. Summary of the Invention

[0004] To address the aforementioned technical shortcomings, the present invention aims to propose a brain-computer interface data processing method that integrates steady-state motion visual evoked potentials and motor imagery. This method is intended to solve the technical problem that the single visual evoked paradigm in the existing technology cannot achieve stable motor intention recognition, especially under conditions where there are large differences in the motor imagery abilities of subjects and the brain-computer signal-to-noise ratio is low.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides a brain-computer interface data processing method that integrates steady-state motion visual evoked potentials and motor imagery.

[0006] The brain-computer interface data processing method that integrates steady-state motion visual evoked potentials and motor imagery includes: Step S10: Obtain the subject's rehabilitation training task information and generate a bimodal task configuration table based on the rehabilitation training task information; Step S20: Generate a bimodal synchronous stimulation sequence according to the bimodal task configuration table; Step S30: Acquire and label multi-lead EEG signals according to the dual-modal synchronous stimulation sequence to obtain synchronous EEG timing data; Step S40: Extract visual response features and motor imagery response features from the synchronized EEG time-series data to obtain a bimodal EEG feature set; Step S50: Perform motion intention recognition based on the bimodal EEG feature set to obtain the motion intention recognition result.

[0007] Preferably, step S10, which involves obtaining the subject's rehabilitation training task information and generating a bimodal task configuration table based on the rehabilitation training task information, specifically includes: Step S101: Obtain the rehabilitation training task information of the subject. The rehabilitation training task information includes training stage identifiers, limbs to be trained, motor imagery actions to be performed, and the number of target instructions. The training stage identifiers include individual calibration stage identifiers and rehabilitation training stage identifiers. Step S102: Determine multiple motion visual stimulus targets according to the number of target instructions, and assign each motion visual stimulus target to a motion imagination action to be executed; Step S103: Configure different motion frequencies for each motion visual stimulus target, and write the motion visual stimulus target, the motion imagination action to be performed, the motion frequency and the training stage identifier into the same mapping record to obtain a bimodal task configuration table.

[0008] Preferably, step S20, the step of generating a bimodal synchronous stimulus sequence based on the bimodal task configuration table, specifically includes: Step S201: Read the motion visual stimulus target, the motion imagery action to be performed, the motion frequency and the training stage identifier in the bimodal task configuration table, and assign a stimulus display area to each motion visual stimulus target in the stimulus display interface; Step S202: Based on the motor imagery action to be performed, load at least one type of motor imagery guidance content, including limb outline pattern, limb movement animation, text prompt, or voice prompt, into the corresponding stimulus display area; Step S203: Control the corresponding motion visual stimulus target to perform periodic spatial motion according to the motion frequency, and make the motion imagination guidance content and the periodic spatial motion synchronously presented within the same stimulus time window to obtain a dual-modal synchronous stimulus sequence.

[0009] Preferably, step S203, which involves controlling the corresponding moving visual stimulus target to perform periodic spatial motion according to the motion frequency, specifically includes: The periodic spatial motion includes translational motion, rotational motion, or scaling motion; The pattern content of the motion visual stimulus target includes checkerboard pattern, zebra stripe pattern, arrow pattern or limb outline pattern. The motion frequency interval between different moving visual stimuli is not less than 0.5 Hz, and the motion frequency is in the range of 6 Hz to 12 Hz; The animation switching state of the moving visual stimulus target is determined by the following formula:

[0010] in, Indicates the first A moving visual stimulus target at time Animation switching states; Indicates the first The motion frequency of a visual stimulus target; Indicates the timing of stimulus presentation; Represents a symbolic function.

[0011] Preferably, step S30, which involves acquiring and labeling multi-lead EEG signals based on the dual-modal synchronous stimulation sequence to obtain synchronous EEG timing data, specifically includes: Step S301: When the dual-modal synchronous stimulation sequence begins to be presented, a stimulation initiation marker is generated, and the multi-lead EEG signal of the subject is acquired at a sampling rate of not less than 250Hz; Step S302: Based on the stimulation initiation marker and the preset stimulation duration, the multi-lead EEG signal is segmented in several trials to obtain EEG segments; Step S303: Align the EEG segments with the corresponding motion visual stimulus targets, motion frequencies, motion imagery actions to be performed, and training phase markers to obtain synchronized EEG time-series data; The electrode points for collecting the multi-lead EEG signals include at least the occipital visual cortex electrode points O1, Oz, and O2, and the sensorimotor cortex electrode points C3, Cz, and C4.

[0012] Preferably, the electrode points for acquiring the multi-lead EEG signals also include adjacent electrode points PO3, POz, PO4, PO5, and PO6 in the occipital visual cortex area, and adjacent electrode points FC3, CP3, FCz, CPz, FC4, and CP4 in the sensorimotor cortex area. When the training phase is identified as an individual calibration phase, the synchronized EEG timing data includes multiple task-state EEG samples collected in a fixed stimulus sequence. When the training phase is identified as a rehabilitation training phase, the synchronized EEG timing data includes rehabilitation training EEG samples collected according to the motor imagery content selected by the subject.

[0013] Preferably, step S40, which involves extracting visual response features and motor imagery response features from the synchronized EEG time-series data to obtain a bimodal EEG feature set, specifically includes: Step S401: Extract the occipital visual cortex lead signal from the synchronized EEG time series data, and calculate the spectral energy at each motion frequency and its harmonic frequency to obtain the frequency response characteristics of steady-state motion visual evoked potentials. Step S402: Extract sensorimotor cortex lead signals from the synchronized EEG time-series data, and calculate the energy changes of mu rhythm and beta rhythm between the baseline segment and the stimulation segment before stimulation to obtain motor imagery event-related desynchronization features; Step S403: The steady-state motion visual evoked potential frequency response features, the motion imagery event-related desynchronization features, and the corresponding label information are concatenated to obtain a dual-modal EEG feature set; The step of identifying motor intentions based on the bimodal EEG feature set to obtain the motor intention recognition result specifically includes: calculating the visual response score of each motor visual stimulus target based on the steady-state motor visual evoked potential frequency response features in the bimodal EEG feature set; calculating the motor imagination score of each motor imagination action to be performed based on the motor imagination event correlation desynchronization features in the bimodal EEG feature set; weightedly fusing the visual response score and the motor imagination score to obtain the fusion score of each candidate motor intention, and determining the candidate motor intention with the highest fusion score as the motor intention recognition result; The fusion score is calculated using the following formula:

[0014] in, Indicates the first A fusion score of candidate motion intentions; Indicates the first Visual response scores corresponding to each candidate motion intent; Indicates the first The motion imagery score corresponding to each candidate motion intention; Indicates visual response weights; Indicates the weight of motion imagination; This represents a constant to prevent the denominator from being zero. and These represent the minimum and maximum visual response scores for each candidate motion intention, respectively. and These represent the minimum and maximum scores for the motor imagery of each candidate motor intention, respectively; and... ; Rehabilitation feedback information is generated based on the motor intention recognition results. The rehabilitation feedback information includes at least one of visual feedback, auditory feedback, progress bar feedback, or rehabilitation assessment feedback. The rehabilitation assessment feedback includes the accuracy of steady-state motor visual evoked potential recognition and the motor imagery event-related desynchronization index.

[0015] This invention also provides a brain-computer interface data processing system that integrates steady-state motion visual evoked potentials and motor imagery, comprising: The task configuration module is used to obtain the rehabilitation training task information of the subject and generate a bimodal task configuration table based on the rehabilitation training task information. The stimulus generation module is used to generate a bimodal synchronous stimulus sequence based on the bimodal task configuration table. The signal acquisition and annotation module is used to acquire and annotate multi-lead EEG signals according to the dual-modal synchronous stimulation sequence to obtain synchronous EEG timing data. The feature extraction module is used to extract visual response features and motor imagery response features based on the synchronous EEG time-series data to obtain a bimodal EEG feature set; The intention recognition feedback module is used to perform motor intention recognition based on the bimodal EEG feature set, obtain motor intention recognition results, and generate rehabilitation feedback information based on the motor intention recognition results.

[0016] The present invention also provides a brain-computer interface data processing device that integrates steady-state motor visual evoked potentials and motor imagery. The brain-computer interface data processing device that integrates steady-state motor visual evoked potentials and motor imagery includes: a memory, a processor, and a brain-computer interface data processing program that integrates steady-state motor visual evoked potentials and motor imagery stored in the memory and executable on the processor. When the brain-computer interface data processing program that integrates steady-state motor visual evoked potentials and motor imagery is executed by the processor, it implements the above-described method.

[0017] The present invention also provides a computer program product, the computer program product including a brain-computer interface data processing program that integrates steady-state motor visual evoked potentials and motor imagery, the brain-computer interface data processing program that integrates steady-state motor visual evoked potentials and motor imagery implements the above method when executed by a processor.

[0018] The beneficial effects of this invention are as follows: 1. This invention achieves task-level synchronous fusion of steady-state motor visual evoked potentials and motor imagery, and generates spatiotemporally aligned synchronous stimulus sequences based on a dual-modality task configuration table. This enables the simultaneous induction and acquisition of EEG responses from both modalities within a single trial. This design not only increases the amount of information acquired per unit time but also effectively enhances the accuracy of motor intent recognition in complex environments through the complementarity of the two modalities, overcoming the shortcomings of single-modality recognition which is susceptible to interference from individual states or environmental noise.

[0019] 2. This invention designs a feature extraction and fusion recognition framework for bimodal EEG signals. It extracts steady-state motor visual evoked potential frequency response features and motor imagery event-related desynchronization features from the occipital visual cortex and sensorimotor cortex, respectively, and employs a weighted fusion strategy for intent discrimination. This method can adaptively balance the contributions of the two modalities in different subjects or at different task stages, providing core technical support for constructing a high-instruction-throughput, low-cognitive-load brain-computer interface system for rehabilitation training. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the first embodiment of a brain-computer interface data processing method that integrates steady-state motion visual evoked potentials and motor imagery according to the present invention. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0022] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1: As Figure 1 The diagram shown is a flowchart illustrating the first embodiment of a brain-computer interface data processing method that integrates steady-state motor visual evoked potentials and motor imagery according to the present invention.

[0024] In the first embodiment, the brain-computer interface data processing method integrating steady-state motion visual evoked potentials and motor imagery includes: Step S10: Obtain the subject's rehabilitation training task information and generate a bimodal task configuration table based on the rehabilitation training task information; The "rehabilitation training task information" in this step refers to the training parameters set by the rehabilitation physician or based on the subject's specific situation. These parameters include training stage identifiers (e.g., distinguishing between individual calibration and rehabilitation training stages), the limbs to be trained (e.g., left hand, right hand, both feet), the motor imagery actions to be performed (e.g., clenching a fist, extending fingers, dorsiflexing the ankle), and the number of target instructions. The "bimodal task configuration table" is a structured mapping table built upon this information. Its core function is to bind visual stimulus targets, motor imagery content, stimulus frequency, and training stages one-to-one.

[0025] This step serves as the initialization and configuration stage of the entire data processing flow. Its technical effect lies in transforming abstract rehabilitation training goals into a set of bimodal task parameters that can be accurately parsed and executed by the computer. It ensures that all subsequent stimulus presentation, signal acquisition, and feature analysis have a clear task context and goal orientation.

[0026] Compared to the relatively fixed or independent configuration of stimulation tasks in traditional brain-computer interfaces, this step generates a "dual-modal task configuration table," achieving unified planning and associated configuration of steady-state motion visual evoked potential stimulation and motion imagery guidance tasks. This avoids potential fragmentation or conflict between the two modalities at the task level, ensuring deep integration and synergy of the two modalities from the task design stage, laying a solid foundation for subsequent efficient synchronous data acquisition and feature fusion recognition.

[0027] For example, for a stroke patient requiring rehabilitation of their right-hand grasping function, the rehabilitation training task information might be set as follows: the training phase is "Rehabilitation Training Phase," the limb to be trained is "right hand," the motor imagery action to be performed is "clenching a fist," and the number of target instructions is 4 (corresponding to the four directions: left, right, up, and down). Step S10 will generate a bimodal task configuration table based on this, containing four records. Each record associates a motion visual stimulus target (such as an arrow moving to the right) with the imagery action of "clenching a fist with the right hand," a specific motion frequency (such as 8.0Hz, 8.5Hz, 9.0Hz, 9.5Hz), and the "Rehabilitation Training Phase" identifier. This table becomes the core configuration driving the entire operation.

[0028] Step S20: Generate a bimodal synchronous stimulation sequence according to the bimodal task configuration table; The "bimodal synchronous stimulation sequence" in this step refers to a sequence of visual-motor stimuli and motor imagery guidance content that is strictly synchronized in time, generated according to a configuration table. Specifically, an independent stimulus display area is assigned to each visual-motor stimulus target on the stimulus display interface (such as a screen). Within each display area, on the one hand, the stimulus target (such as a checkerboard or arrow) is controlled to perform periodic spatial movements such as translation and rotation according to a specified motion frequency to induce steady-state visual-motor evoked potentials; on the other hand, motor imagery guidance content corresponding to the stimulus target is loaded synchronously, such as limb contour animation, text prompts like "imagine your right hand making a fist," or voice prompts, to guide the subject to perform specific motor imagery.

[0029] The technical advantage of this step lies in achieving precise synchronization between external visual stimuli and internal cognitive tasks (motor imagery) within a time window. This synchronous presentation mechanism ensures that the acquired EEG signals simultaneously include steady-state visual evoked potential responses induced by periodic visual motion, as well as event-related desynchronization / synchronization responses induced by the synchronous motor imagery cognitive process, thus creating conditions for subsequent extraction of highly temporally correlated bimodal features.

[0030] Compared to traditional methods that may present visual stimuli and imagery cues in a time-separated manner, or only provide simple static cues, this step achieves spatiotemporal coupling of stimulation and guidance by generating a "dual-modal synchronous stimulation sequence." This not only more effectively guides users into the correct imagery state and improves performance on motor imagery tasks, but also ensures a high degree of temporal overlap between the two modal EEG responses, reducing signal confusion and noise caused by task switching or time delays, and greatly improving the signal-to-noise ratio and effectiveness of subsequent feature extraction and fusion.

[0031] For example, based on the aforementioned configuration table, four arrow patterns are displayed in the four corners of the screen. When the task begins, the arrow in the upper right corner starts rotating clockwise at a frequency of 9.0 Hz (visual-motor stimulation), while a dynamic animation of a right hand clenching into a fist is superimposed next to the arrow, accompanied by the voice prompt "Imagine your right hand clenching into a fist." The rotating animation and the guided imagery start, continue, and end simultaneously, constituting a complete bimodal synchronous stimulation trial. The stimulation targets in the other three corners are also presented synchronously with their respective frequencies (e.g., 8.0 Hz, 8.5 Hz, 9.5 Hz) and corresponding guided imagery.

[0032] Step S30: Acquire and label multi-lead EEG signals according to the dual-modal synchronous stimulation sequence to obtain synchronous EEG timing data; The "synchronized EEG timing data" in this step refers to the dataset obtained by precisely aligning and labeling the acquired raw multi-lead EEG signals with the presentation events of the bimodal synchronous stimulation sequence. The specific process includes: marking the beginning of the stimulation sequence with a "stimulus start marker"; synchronously acquiring EEG signals covering the occipital visual cortex (e.g., O1, Oz, O2, PO3, POz, etc.) and the sensorimotor cortex (e.g., C3, Cz, C4, FC3, CP3, etc.) at a high sampling rate (e.g., not less than 250Hz); segmenting the continuous EEG signals into trial segments based on the markers and preset stimulation durations; and finally labeling each EEG segment with its corresponding motor visual stimulus target, movement frequency, to-be-executed motor imagery action, and training stage identifier.

[0033] The technical advantage of this step lies in the precise correlation between time-synchronized stimulus events in the physical world and the acquisition of bioelectrical signals at the data level. The resulting "synchronized EEG temporal data" not only includes the original sequence of EEG voltage changes over time but also includes complete task label information. This allows subsequent feature extraction and model training to clearly identify which bimodal task induced each segment of EEG data, forming the foundation for supervised learning and accurate intent recognition.

[0034] Compared to traditional EEG acquisition methods that may focus solely on signal quality while neglecting precise alignment with task events, or that only annotate a single modality, this step emphasizes "synchronization" and "dual-modal annotation." It ensures that the occipital lobe response evoked by visual stimuli and the sensorimotor cortex response evoked by motor imagery are fully recorded within the same timeframe and possess unified, accurate, multi-dimensional labels. This high-quality data annotation method provides reliable data support for subsequent feature extraction and effective fusion of the two modalities, avoiding feature confusion and recognition errors caused by unclear or asynchronous annotations.

[0035] For example, in one trial, a start marker is placed at time t0, and a 9.0Hz rotating arrow and a right fist clenching are displayed as guidance for 3 seconds. The EEG device simultaneously acquires signals from all leads during these 3 seconds. After the trial, a data segment is generated containing EEG data from t0 to t0+3 seconds, labeled as: visual target = "upper right arrow", frequency = 9.0Hz, imagined action = "right fist clenching", stage = "rehabilitation training". Data segments from all trials are processed according to this rule, collectively forming a synchronous EEG time-series database for model training or online recognition.

[0036] Step S40: Extract visual response features and motor imagery response features from the synchronized EEG time-series data to obtain a bimodal EEG feature set; The "bimodal EEG feature set" in this step refers to the set of feature vectors formed by concatenating two types of features extracted in parallel from synchronous EEG time-series data. Specifically, it includes: 1) "Steady-state motor visual evoked potential frequency response features": obtained by calculating the spectral energy at each stimulus frequency and its harmonic frequencies from occipital visual cortex lead signals (e.g., O1, O2) using methods such as Fast Fourier Transform or Power Spectral Density Analysis, reflecting the brain's phase-locked response strength to specific frequency visual-motor stimuli. 2) "Motor imagery event-related desynchronization features": obtained by calculating the energy decrease (i.e., desynchronization) of the mu rhythm (8-13Hz) and beta rhythm (13-30Hz) from sensorimotor cortex lead signals (e.g., C3, C4) during the stimulus presentation segment relative to the pre-stimulus resting baseline segment, reflecting the activation state of the sensorimotor cortex during specific motor imagery. After extraction, these two types of features are concatenated with corresponding task labels (e.g., target category) to form the final feature sample.

[0037] The technical advantage of this step lies in achieving dimensionality reduction and information condensation of the original high-dimensional EEG time-series data, transforming the key neural response patterns related to the bimodal task contained in the signal into feature vectors that are easy for computers to process and recognize. Complementary features are extracted from both spatial (different brain regions) and frequency (different rhythms) dimensions, preserving to the greatest extent possible the frequency domain locking information of SSMVEP and the time-frequency domain event correlation desynchronization information of MI, providing rich and highly discriminative input for subsequent fusion classification.

[0038] Compared to traditional methods that may only extract features from a single modality or perform simple mixing of multimodal signals, this step designs a specialized feature extraction method tailored to the characteristics of SSMVEP and MI, which originate in different brain regions and have different neurophysiological mechanisms. This targeted feature extraction can capture the response characteristics of each modality more purely and effectively, avoiding mutual interference between different modal signals at the raw data or low-level feature level. This provides a clearer and more discriminative feature foundation for subsequent fusion recognition based on high-level semantics (intent), and is a key step in achieving high-precision dual-modal fusion recognition.

[0039] For example, for an EEG segment labeled "right hand clenched fist," the feature extraction process is as follows: First, spectral analysis of the signal in the occipital lobe reveals significant energy peaks at 9.0 Hz and its harmonic 18.0 Hz. Therefore, the energy values ​​at these two frequencies are extracted as the visual feature vector [E_9Hz, E_18Hz]. Simultaneously, time-frequency analysis is performed on the signals from leads C3 (corresponding to the right hand) and C4 (corresponding to the left hand) of the sensorimotor cortex. The energy ratio of the mu / beta rhythm during imagery relative to the baseline is calculated. It is found that the mu rhythm energy in lead C3 decreases significantly (desynchronization), while the change in lead C4 is minimal. Therefore, [C3_mu_ERD, C4_mu_ERD, C3_beta_ERD, C4_beta_ERD] are extracted as the motor imagery feature vector. Finally, these two sets of features are concatenated with the label "right hand" to form a complete bimodal feature sample.

[0040] Step S50: Perform motion intention recognition based on the bimodal EEG feature set to obtain the motion intention recognition result.

[0041] The "motor intention recognition result" in this step refers to the motor command that the subject is currently attempting to execute, determined through analysis and calculation of the bimodal EEG feature set. The specific implementation process includes: First, based on the frequency response characteristics of steady-state motor visual evoked potentials, a "visual response score" is calculated for each candidate motor visual stimulus target (corresponding to a candidate intention). This score is typically proportional to the spectral energy at the target's corresponding frequency. Second, based on the event-related desynchronization characteristics of motor imagery, a "motor imagery score" is calculated for each candidate motor imagery action to be executed (also corresponding to a candidate intention). This score is typically positively correlated with the degree of desynchronization of a specific rhythm in the contralateral sensorimotor cortex. Then, a weighted fusion formula is used to normalize and weightedly sum the visual and motor imagery scores of each candidate intention to obtain a "fusion score." Finally, the candidate intention with the highest fusion score is selected as the final recognition result. Based on this result, various forms of "rehabilitation feedback information" can be generated, such as highlighting the selected visual target, playing a success sound effect, updating the training progress bar, or outputting evaluation reports on the recognition accuracy and event-related desynchronization index of this training session.

[0042] The technical advantage of this step lies in achieving final decision-making based on multimodal information fusion, transforming the two features extracted in the preceding steps into explicit control commands or training feedback. The weighted fusion mechanism allows for dynamic adjustment of the contribution weights (α and β) of the two modalities based on the individual characteristics of the subject (e.g., a strong SSMVEP response and a weak MI response) or the needs of different training stages, thereby achieving personalized, adaptive, and robust intent recognition. The generated rehabilitation feedback forms a closed loop, instantly motivating the subject and quantifying the training effect.

[0043] Compared to traditional single-modal recognition methods that make decisions directly based on a single feature score, or the potential modal conflict issues associated with simple voting fusion, the weighted fusion recognition strategy employed in this step offers significant advantages. It not only achieves complementarity of bimodal information at the decision-making level, but also addresses situations where one modality temporarily fails (e.g., weakened SSMVEP response due to visual fatigue) through weight adjustment, ensuring that a relatively reliable judgment can still be made based on the other modality. This greatly improves stability and fault tolerance in practical use. This fusion approach makes the brain-computer interface closer to the multi-sensory integrated decision-making mechanism of biological systems, enhancing overall performance and the fluency of human-computer interaction.

[0044] For example, in a four-command online rehabilitation training session, the visual scores for the four candidate intentions (up, down, left, right) were calculated to be [85, 45, 90, 30], and the motor imagery scores were [60, 75, 40, 80]. After normalization, assuming a weighting of α=0.6 (emphasizing visual) and β=0.4 (emphasizing imagery), the fusion scores could be [0.72, 0.54, 0.68, 0.44]. The "up" instruction had the highest fusion score (0.72), and was therefore identified as the motor intention recognition result. The arrow at the top was then highlighted on the screen, an encouraging sound effect was played, and one point was added to the progress bar. Simultaneously, the system recorded: the SSMVEP recognition accuracy (based on visual score) was 100%, and the MI event-related desynchronization index (C3 lead mu rhythm) was -0.35 (significant desynchronization).

[0045] Example 2: Furthermore, the present invention provides a brain-computer interface data processing system that integrates steady-state motor visual evoked potentials (SMPs) and motor imagery, employing a brain-computer interface data processing method for integrating SMPs and motor imagery as described in the above embodiments. This system can solve the technical problem of brain-computer interface data processing that integrates SMPs and motor imagery. The beneficial effects of the brain-computer interface data processing system integrating SMPs and motor imagery provided by the present invention are the same as those of the brain-computer interface data processing method for integrating SMPs and motor imagery provided in the above embodiments. Other technical features of the brain-computer interface data processing system integrating SMPs and motor imagery are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0046] Example 3: This invention provides a brain-computer interface (BCI) data processing device integrating steady-state motor visual evoked potentials (SMPs) and motor imagery. The BCI data processing device includes: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the BCI data processing method integrating SMPs and motor imagery described in Example 1 above. The BCI data processing device integrating SMPs and motor imagery in this embodiment of the invention may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. This BCI data processing device integrating SMPs and motor imagery is merely an example and should not be construed as limiting the functionality or scope of the invention. A brain-computer interface (BCI) data processing device integrating steady-state motor visual evoked potentials (SMPs) and motor imagery may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). The RAM also stores various programs and data required for the operation of the BCI data processing device. The processing unit, ROM, and RAM are interconnected via a bus. An I / O interface is also connected to the bus. Typically, the following systems can be connected to the I / O interface: input devices including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication devices. The communication device allows the BCI data processing device to wirelessly or wiredly communicate with other devices to exchange data. While a brain-computer interface data processing device integrating steady-state motor visual evoked potentials and motor imagery with various systems has been described, it should be understood that implementation of all the described systems is not required. Alternatively, more or fewer systems may be implemented.

[0047] Example 4: This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the brain-computer interface data processing method fusing steady-state motor visual evoked potentials and motor imagery as described above. The computer program product provided by this invention can solve the technical problem of brain-computer interface data processing fusing steady-state motor visual evoked potentials and motor imagery. Compared with the prior art, the beneficial effects of the computer program product provided by this invention are the same as the beneficial effects of the brain-computer interface data processing method fusing steady-state motor visual evoked potentials and motor imagery provided in the above embodiments, and will not be repeated here.

[0048] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a read-only memory. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments disclosed in this invention.

[0049] It should be understood that the various parts disclosed in this invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0050] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A brain-computer interface data processing method integrating steady-state motion visual evoked potentials and motor imagery, characterized in that, The method includes: Step S10: Obtain the subject's rehabilitation training task information and generate a bimodal task configuration table based on the rehabilitation training task information; Step S20: Generate a bimodal synchronous stimulation sequence according to the bimodal task configuration table; Step S30: Acquire and label multi-lead EEG signals according to the dual-modal synchronous stimulation sequence to obtain synchronous EEG timing data; Step S40: Extract visual response features and motor imagery response features from the synchronized EEG time-series data to obtain a bimodal EEG feature set; Step S50: Perform motion intention recognition based on the bimodal EEG feature set to obtain the motion intention recognition result.

2. The brain-computer interface data processing method integrating steady-state motion visual evoked potentials and motor imagery as described in claim 1, characterized in that, Step S10, which involves obtaining the subject's rehabilitation training task information and generating a bimodal task configuration table based on the rehabilitation training task information, specifically includes: Step S101: Obtain the rehabilitation training task information of the subject. The rehabilitation training task information includes training stage identifiers, limbs to be trained, motor imagery actions to be performed, and the number of target instructions. The training stage identifiers include individual calibration stage identifiers and rehabilitation training stage identifiers. Step S102: Determine multiple motion visual stimulus targets according to the number of target instructions, and assign each motion visual stimulus target to a motion imagination action to be executed; Step S103: Configure different motion frequencies for each motion visual stimulus target, and write the motion visual stimulus target, the motion imagination action to be performed, the motion frequency and the training stage identifier into the same mapping record to obtain a bimodal task configuration table.

3. The brain-computer interface data processing method integrating steady-state motion visual evoked potentials and motor imagery as described in claim 1, characterized in that, Step S20, the step of generating a bimodal synchronous stimulation sequence based on the bimodal task configuration table, specifically includes: Step S201: Read the motion visual stimulus target, the motion imagery action to be performed, the motion frequency and the training stage identifier in the bimodal task configuration table, and assign a stimulus display area to each motion visual stimulus target in the stimulus display interface; Step S202: Based on the motor imagery action to be performed, load at least one type of motor imagery guidance content, including limb outline pattern, limb movement animation, text prompt, or voice prompt, into the corresponding stimulus display area; Step S203: Control the corresponding motion visual stimulus target to perform periodic spatial motion according to the motion frequency, and make the motion imagination guidance content and the periodic spatial motion synchronously presented within the same stimulus time window to obtain a dual-modal synchronous stimulus sequence.

4. The brain-computer interface data processing method integrating steady-state motion visual evoked potentials and motor imagery as described in claim 3, characterized in that, Step S203, which involves controlling the corresponding moving visual stimulus target to perform periodic spatial motion according to the motion frequency, specifically includes: The periodic spatial motion includes translational motion, rotational motion, or scaling motion; The pattern content of the motion visual stimulus target includes checkerboard pattern, zebra stripe pattern, arrow pattern or limb outline pattern. The motion frequency interval between different moving visual stimuli is not less than 0.5 Hz, and the motion frequency is in the range of 6 Hz to 12 Hz; The animation switching state of the moving visual stimulus target is determined by the following formula: in, Indicates the first A moving visual stimulus target at time Animation switching states; Indicates the first The motion frequency of a visual stimulus target; Indicates the timing of stimulus presentation; Represents a symbolic function.

5. The brain-computer interface data processing method integrating steady-state motion visual evoked potentials and motor imagery as described in claim 1, characterized in that, Step S30, which involves acquiring and labeling multi-lead EEG signals based on the dual-modal synchronous stimulation sequence to obtain synchronized EEG timing data, specifically includes: Step S301: When the dual-modal synchronous stimulation sequence begins to be presented, a stimulation initiation marker is generated, and the multi-lead EEG signal of the subject is acquired at a sampling rate of not less than 250Hz; Step S302: Based on the stimulation initiation marker and the preset stimulation duration, the multi-lead EEG signal is segmented in several trials to obtain EEG segments; Step S303: Align the EEG segments with the corresponding motion visual stimulus targets, motion frequencies, motion imagery actions to be performed, and training phase markers to obtain synchronized EEG time-series data; The electrode points for collecting the multi-lead EEG signals include at least the occipital visual cortex electrode points O1, Oz, and O2, and the sensorimotor cortex electrode points C3, Cz, and C4.

6. The brain-computer interface data processing method integrating steady-state motion visual evoked potentials and motor imagery as described in claim 5, characterized in that, The electrode points for collecting the multi-lead EEG signals also include adjacent electrode points PO3, POz, PO4, PO5, and PO6 in the occipital visual cortex area, and adjacent electrode points FC3, CP3, FCz, CPz, FC4, and CP4 in the sensorimotor cortex area. When the training phase is identified as an individual calibration phase, the synchronized EEG timing data includes multiple task-state EEG samples collected in a fixed stimulus sequence. When the training phase is identified as a rehabilitation training phase, the synchronized EEG timing data includes rehabilitation training EEG samples collected according to the motor imagery content selected by the subject.

7. The brain-computer interface data processing method integrating steady-state motion visual evoked potentials and motor imagery as described in claim 1, characterized in that, Step S40, which involves extracting visual response features and motor imagery response features from the synchronized EEG time-series data to obtain a bimodal EEG feature set, specifically includes: Step S401: Extract the occipital visual cortex lead signal from the synchronized EEG time series data, and calculate the spectral energy at each motion frequency and its harmonic frequency to obtain the frequency response characteristics of steady-state motion visual evoked potentials. Step S402: Extract sensorimotor cortex lead signals from the synchronized EEG time-series data, and calculate the energy changes of mu rhythm and beta rhythm between the baseline segment and the stimulation segment before stimulation to obtain motor imagery event-related desynchronization features; Step S403: The steady-state motion visual evoked potential frequency response features, the motion imagery event-related desynchronization features, and the corresponding label information are concatenated to obtain a bimodal EEG feature set.

8. A brain-computer interface data processing system integrating steady-state motor visual evoked potentials and motor imagery, applied to the brain-computer interface data processing method integrating steady-state motor visual evoked potentials and motor imagery as described in any one of claims 1 to 7, characterized in that, The brain-computer interface data processing system includes: The task configuration module is used to obtain the rehabilitation training task information of the subject and generate a bimodal task configuration table based on the rehabilitation training task information. The stimulus generation module is used to generate a bimodal synchronous stimulus sequence based on the bimodal task configuration table. The signal acquisition and annotation module is used to acquire and annotate multi-lead EEG signals according to the dual-modal synchronous stimulation sequence to obtain synchronous EEG timing data. The feature extraction module is used to extract visual response features and motor imagery response features based on the synchronous EEG time-series data to obtain a bimodal EEG feature set; The intention recognition feedback module is used to perform motor intention recognition based on the bimodal EEG feature set, obtain motor intention recognition results, and generate rehabilitation feedback information based on the motor intention recognition results.

9. A brain-computer interface data processing device integrating steady-state motion visual evoked potentials and motor imagery, characterized in that, The brain-computer interface data processing device integrating steady-state motor visual evoked potentials and motor imagery includes: a memory, a processor, and a brain-computer interface data processing program integrating steady-state motor visual evoked potentials and motor imagery stored in the memory and executable on the processor. When the brain-computer interface data processing program integrating steady-state motor visual evoked potentials and motor imagery is executed by the processor, it implements a brain-computer interface data processing method integrating steady-state motor visual evoked potentials and motor imagery as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a brain-computer interface data processing program that integrates steady-state motor visual evoked potentials and motor imagery. When the brain-computer interface data processing program that integrates steady-state motor visual evoked potentials and motor imagery is executed by the processor, it implements a brain-computer interface data processing method that integrates steady-state motor visual evoked potentials and motor imagery as described in any one of claims 1 to 7.

Images