A multi-mode fusion-based brain-computer interface control system and method

Abstract

This invention discloses a brain-computer interface control system and method based on multimodal fusion, relating to the interdisciplinary field of artificial intelligence. The method includes: loading a pre-trained generative model that decouples neural representations into three categories: individual physiology, general cognition, and task intent; then guiding the user to perform efficient calibration through active learning, simultaneously acquiring EEG and generating initial virtual multimodal signals to form a user feature set; using this feature set to directionally adjust model parameters to update the individual physiological mapping, constructing a personalized generator; this generator can convert real-time EEG into virtual signals and fuse them with the original signals to form a unified representation; decoding this fused representation to parse task intent and output control commands, supporting online updates based on interactive feedback and rapid cross-task transfer. This invention, based on lightweight single-modal acquisition, achieves virtual multimodal fusion and rapid personalization through generative AI, solving the problems of bulky equipment, time-consuming calibration, and task rigidity in traditional solutions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence interdisciplinary technology, and in particular to a brain-computer interface control system and method based on multi-modal fusion. Background Technology

[0002] Brain-computer interface (BCI) technology has evolved over decades, moving from single-modal signal analysis to a new stage of multimodal fusion and enhancement. Traditional methods primarily involve simultaneously acquiring heterogeneous physiological signals such as electroencephalograms (EEGs) and functional near-infrared spectroscopy (FIRS), fusing them at the feature or decision layers to overcome the limitations of single-modality approaches and improve decoding robustness and control dimensionality. In recent years, the rise of generative artificial intelligence has provided a new paradigm for brain signal analysis and synthesis, driving the field towards intelligence and adaptability.

[0003] Current technologies still have significant limitations. Mainstream multimodal fusion solutions heavily rely on the simultaneous acquisition of data from multiple physical sensors, resulting in cumbersome systems, high costs, and poor user experience. Their models are typically trained jointly for specific users and tasks, lacking versatility, leading to lengthy calibration times for new users and difficulty in quickly migrating to new tasks. Furthermore, once deployed, the system becomes static and difficult to dynamically optimize online based on changes in user status or interaction feedback. These shortcomings make it difficult for existing technologies to achieve lightweight, highly generalizable, and self-evolving practical brain-computer interface systems. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a brain-computer interface control method based on multimodal fusion to solve the problems of existing multimodal fusion brain-computer interface methods, such as heavy calibration burden, reliance on multiple hardware sets, poor cross-user and cross-task transfer capabilities, and how to achieve a high-performance brain-computer interface system with lightweight acquisition, online adaptation and rapid generalization.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a brain-computer interface control method based on multi-modal fusion, characterized by comprising the following steps: Load a pre-trained decoupled generative model that separates three types of representations: individual physiology, general cognition, and task intent; The system guides users to calibrate, collects EEG signals, generates initial virtual multimodal signals, and forms a user feature set. Use feature sets to adjust model parameters, update individual physiological mappings, and build personalized generators; The personalized generator converts real-time EEG into virtual signals, which are then fused with the original signals to form a fused representation. Decode and fuse representations, parse task intent, and output control commands; The model is updated online based on interactive feedback, and the intent module is adjusted according to new task data to achieve migration.

[0007] As a preferred embodiment of the brain-computer interface control method based on multi-modal fusion described in this invention, the loading of a pre-trained decoupled generative model, which separates three types of representations—individual physiology, general cognition, and task intent—involves the following specific steps: The basic generative model is obtained from the cloud model library. This model adopts a variational autoencoder structure, and its latent space is constrained into three independent subspaces by decoupling the loss function. The model is deployed to a local edge computing device and loaded with model weights pre-trained for large-scale group data; Initialize a dynamic parameter mask matrix, which is used to identify and lock model parameters related to general cognitive representations in subsequent steps; The model is started, entering a ready-to-fit state, preparing to receive the user's EEG signal stream and perform forward computation.

[0008] As a preferred embodiment of the brain-computer interface control method based on multimodal fusion described in this invention, the steps of guiding user calibration, acquiring EEG signals and generating initial virtual multimodal signals to form a user feature set are as follows: The active learning module evaluates the expected information gain of multiple pre-trained calibration tasks based on the cognitive subspace output of the pre-trained model. Based on the evaluation results, an optimal calibration task sequence is dynamically generated and displayed to the user, which contains specific paradigmatic motor imagery and visual stimuli. High-density EEG signals were simultaneously acquired when the user performed the calibration task and input into the decoupled generative model to obtain the corresponding virtual functional near-infrared spectral signals. The collected real EEG signals and the generated virtual signals are aligned by timestamp and packaged into a structured user-specific initial feature set.

[0009] As a preferred embodiment of the brain-computer interface control method based on multi-modal fusion described in this invention, the specific steps of adjusting model parameters using feature sets, updating individual physiological mappings, and constructing a personalized generator are as follows: The user-specific initial feature set is input into the model, and forward and backward propagation is performed only on the encoder path responsible for individual physiological representation in the decoupled network. A gradient pruning fine-tuning strategy is adopted to update the connection weights of the individual physiological encoder, while the parameters related to general cognitive representation are fixed by dynamic parameter masking. Analyze the activation patterns of the fine-tuned model in the cognitive subspace, select the clusters of neurons with the highest activation intensity, and form a subset of user-strongly associated general features; The finely tuned individual physiological encoder, the locked general cognitive module, and the unchanged task intent decoder are recombined and solidified into a lightweight personalized generator file.

[0010] As a preferred embodiment of the brain-computer interface control method based on multi-modal fusion described in this invention, the personalized generator converts real-time EEG into virtual signals and fuses them with the original signals to form a fused representation. The specific steps are as follows: The user's raw EEG signals are continuously collected, and after standardized preprocessing, they are input into the personalized generator. In the personalized generator, the individual physiological encoder extracts the specific components of the signal, while the general cognitive module extracts the homeostatic components; Based on the extracted components, the decoding part of the generator synthesizes a virtual multimodal signal with temporal characteristics of hemoglobin concentration changes; The frequency domain features of the original EEG signal are concatenated with the time domain features of the virtual signal, and then dimensionality reduction and fusion are performed through a shallow neural network layer to output a unified fused feature vector.

[0011] As a preferred embodiment of the brain-computer interface control method based on multi-modal fusion described in this invention, the specific steps of decoding and fusing representations, parsing task intent, and outputting control commands are as follows: The fused feature vector is input into a pre-trained multi-level gated recurrent unit network; The first layer of the network learns the short-term temporal dependencies of the fused features, and the second layer learns the long-term contextual dependencies across modalities. The network's output layer is connected to a task-specific classifier that maps the learned deep representations to a probability distribution of a predefined intent category. The intent category with the highest probability is selected, and combined with the current device state context, it is converted into a control instruction with execution priority and parameters, and sent to the peripheral driver interface.

[0012] As a preferred embodiment of the brain-computer interface control method based on multi-mode fusion described in this invention, the system continuously monitors the command execution results and implicit user feedback, and triggers model updates when multiple consecutive low-confidence outputs or execution failures are detected. Collect recent EEG signals within a small window, generate virtual signals, and corresponding feedback labels to form an incremental dataset; Using only this incremental data, iterative updates of the decoding layer parameters related to task intent in the personalized generator are performed with a small learning rate; When a user starts a new task, the basic task intent module is loaded, and the rapid calibration and reorganization process is repeated to achieve migration to the new task without retraining the entire model.

[0013] Secondly, the present invention provides a brain-computer interface control system based on multi-modal fusion, comprising, The model initialization module loads a pre-trained decoupled generative model, which separates three types of representations: individual physiology, general cognition, and task intent. The active calibration module guides the user through calibration, collects EEG signals, and generates initial virtual multimodal signals to form a user feature set. The parameter adaptation module uses feature sets to adjust model parameters, update individual physiological mappings, and build a personalized generator. The signal fusion module and the personalized generator convert real-time EEG into virtual signals, which are then fused with the original signals to form a fused representation. The intent decoding module decodes and fuses representations, parses task intents, and outputs control commands. The online optimization module updates the model online based on interactive feedback and adjusts the intent module according to new task data to achieve migration.

[0014] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the brain-computer interface control method based on multimodal fusion as described in the first aspect of the present invention.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the brain-computer interface control method based on multi-modal fusion as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By loading a pre-trained decoupled generative model and dynamically synthesizing virtual multimodal signals using single-modal EEG signals, a "lightweight acquisition, heavy-duty analysis" architecture is constructed, fundamentally reducing the dependence on multiple physical sensors and improving the system's portability and user-friendliness; by decoupling neural representations into three categories—individual physiology, general cognition, and task intent—and using active learning-guided calibration, rapid personalized adaptation for new users is achieved, significantly shortening calibration time; by fine-tuning and reorganizing only a specific subset of parameters in the model and supporting online updates based on interactive feedback, the system possesses the ability to rapidly migrate across tasks and continuously optimize, ultimately forming a highly generalizable, evolvable, and practical integrated brain-computer interface solution. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the brain-computer interface control method based on multi-modal fusion in Example 1. Detailed Implementation

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0020] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0021] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0022] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a brain-computer interface control method based on multi-modal fusion, characterized by including the following steps: Load a pre-trained decoupled generative model that separates three types of representations: individual physiology, general cognition, and task intent; The system guides users to calibrate, collects EEG signals, generates initial virtual multimodal signals, and forms a user feature set. Use feature sets to adjust model parameters, update individual physiological mappings, and build personalized generators; The personalized generator converts real-time EEG into virtual signals, which are then fused with the original signals to form a fused representation. Decode and fuse representations, parse task intent, and output control commands; The model is updated online based on interactive feedback, and the intent module is adjusted according to new task data to achieve migration.

[0023] It should be noted that by pre-separating the three types of representations, a structural foundation is laid for accurate and rapid adaptation to individual differences, avoiding the need to train a large model from scratch for each new user, and fundamentally improving the efficiency and versatility of system initialization.

[0024] By employing an active learning strategy, the system acquires the most user-discriminative data with minimal calibration tasks, significantly reducing calibration time and alleviating user burden. Simultaneously, the process of generating virtual multimodal signals essentially involves utilizing general knowledge to perform a multi-dimensional analysis of individual user neural activity, resulting in a feature set that provides precise input for subsequent refined personalized adaptation.

[0025] By employing a strategy of "partial fine-tuning" rather than "overall retraining," the model retains the general cognitive patterns learned from massive amounts of data while incorporating individual user physiological characteristics. This achieves both precise adaptation of the generator to the user and ensures the model's lightweight nature and stability, avoiding the "catastrophic forgetting" and computational overhead caused by full-parameter training. This is key to achieving the goal of "rapid personalization."

[0026] Using a personalized generator, a single EEG signal is expanded in real time into a richer "virtual-real hybrid" dual-stream signal. This fusion is not a simple splicing, but rather a complementary and enhanced representation at the feature level. This results in a final representation that combines the high temporal resolution of EEG with the high spatial resolution / metabolic information characteristics of virtual signals, providing a more robust and comprehensive input for subsequent intent decoding.

[0027] The specially designed decoding network can effectively handle the complex spatiotemporal relationships of "virtual-real fusion" characteristics, thereby more accurately interpreting the user's instantaneous intent. Combined with instruction translation based on device context, the output instructions are not only accurate but also closely aligned with actual control scenarios, enhancing the system's practicality and reliability, and achieving a key leap from "signal recognition" to "intelligent control" in brain-computer interfaces.

[0028] Through online fine-tuning based on interactive feedback, the system can adapt to slow changes in the user's neural patterns or environmental disturbances, maintaining long-term performance stability. Modular design and targeted parameter adjustments enable rapid migration to new tasks without retraining the entire system, completely resolving the pain points of traditional brain-computer interfaces—"one task, one model"—which are rigid and difficult to expand, greatly extending the system's lifespan and applicability.

[0029] Specifically, the loading of the pre-trained decoupled generative model, which separates three types of representations—individual physiology, general cognition, and task intent—involves the following steps: The basic generative model is obtained from the cloud model library. This model adopts a variational autoencoder structure, and its latent space is constrained into three independent subspaces by decoupling the loss function. The model is deployed to a local edge computing device and loaded with model weights pre-trained for large-scale group data; Initialize a dynamic parameter mask matrix, which is used to identify and lock model parameters related to general cognitive representations in subsequent steps; The model is started, entering a ready-to-fit state, preparing to receive the user's EEG signal stream and perform forward computation.

[0030] It should be noted that the local edge computing device first sends a request to the remote cloud server via a network interface to download a specific generative model file from the centrally managed model library in the cloud. This model uses a variational autoencoder as its basic architecture, and its training process employs a special decoupling loss function. This function constrains the natural formation process of the hidden layer representations within the model, forcing the learned feature distribution to automatically separate into three statistically independent representation subspaces, corresponding to three different types of neural information: individual physiology, general cognition, and task intent.

[0031] By acquiring pre-built structured models from the cloud, centralized management and efficient distribution of prior knowledge from neural networks are achieved. The use of a variational autoencoder structure ensures the model's ability to generate stable and diverse data from a low-dimensional latent space. Crucially, pre-training with a decoupling loss function forces the model to establish a modular knowledge representation framework within its internal encoding structure that conforms to the essential characteristics of neural information, even before it encounters specific user data. This provides a vital structural prerequisite for subsequent targeted parameter fine-tuning and rapid adaptation, avoiding chaos and coupling in model representation, and forming the foundation for realizing the core innovative concept of "personalized general models."

[0032] After successfully downloading the model structure definition file, the system deploys it to the user's local edge computing device (such as an embedded system or a dedicated processor). Subsequently, the system loads a pre-trained parameter and weight file from the downloaded resource package, which is completely corresponding to the model structure. These weight parameters were obtained through large-scale pre-training in the cloud using massive amounts of EEG data from diverse user groups, and contain statistical knowledge of the general laws governing human neural activity.

[0033] Deploying the model on local edge devices ensures low latency and privacy for all subsequent signal processing and computation, meeting the basic requirements for real-time brain-computer interface interaction. Loading weights pre-trained on large-scale population data means the model doesn't start from scratch but already possesses powerful and general-purpose neural signal feature extraction and generation capabilities. This is equivalent to injecting the system with a high-starting-point "brain," enabling it to have a strong prior understanding of various neural patterns when first encountering any new user. This greatly reduces the dependence on the initial amount of data from a single user and is key to improving the system's initial performance and generalization ability.

[0034] After the model is loaded and before user data is formally processed, the system automatically generates a binary mask matrix with the same dimension as the total number of parameters in the model, based on the model's internal structure definition. The initialization logic of this matrix is ​​based on the decoupling constraint results described in step one. The system parses the specific parameter connections corresponding to the three subspaces in the model's hidden layer, marking the positions of parameters deemed primarily responsible for "general cognitive representation" in the mask matrix as locked (e.g., set to 0), while marking other parameters as adjustable (e.g., set to 1). This mask matrix will dynamically participate in calculations during subsequent training.

[0035] Initializing the dynamic parameter mask matrix is ​​a core technical preparation for implementing the "selective fine-tuning" strategy. It's equivalent to pre-drawing a "parameter importance map" within a massive neural network, clearly distinguishing which parameters encode stable, common knowledge (general cognition) across users, and which parameters encode more variable, individual- or task-specific information. By identifying and locking general cognitive parameters in advance, the mechanism ensures that the valuable general knowledge learned by the model from massive amounts of data will not be easily destroyed or overwritten during subsequent personalized adaptation. This allows the model to adapt to individual differences while firmly maintaining its generalization ability.

[0036] After completing all the above preparations, the system formally instantiates the generative model in the memory of the edge computing device, putting it into a runnable state. All parameters of the model are fixed in evaluation mode, the forward computation pathway is activated, but parameter gradient calculation and backpropagation pathways are temporarily disabled. At this point, the model, as a static function mapper, already has the ability to receive real-time input EEG signal data streams and perform feature extraction and virtual signal generation based on its current parameters (group pre-training weights), and is waiting to connect specific user data to start the subsequent personalized process.

[0037] This step marks a smooth transition from the model preparation phase to the real-time interaction preparation phase. By placing the model in a "ready-to-adapt" state, it ensures that the system can immediately begin processing signals and providing basic virtual fusion functions, while also preparing for a seamless transition into user data-based fine-tuning processes. This state design makes the system initialization process efficient and goal-oriented. Once the user wears the device, the system can provide immediate services using a general model, while the backend is already prepared with the simplest path to quickly absorb personalized user information, optimizing the continuity of the user experience.

[0038] Specifically, the process of guiding user calibration, collecting EEG signals and generating initial virtual multimodal signals to form a user feature set involves the following steps: The active learning module evaluates the expected information gain of multiple pre-trained calibration tasks based on the cognitive subspace output of the pre-trained model. Based on the evaluation results, an optimal calibration task sequence is dynamically generated and displayed to the user, which contains specific paradigmatic motor imagery and visual stimuli. High-density EEG signals were simultaneously acquired when the user performed the calibration task and input into the decoupled generative model to obtain the corresponding virtual functional near-infrared spectral signals. The collected real EEG signals and the generated virtual signals are aligned by timestamp and packaged into a structured user-specific initial feature set.

[0039] It should be noted that the active learning module runs first after system initialization. This module calls the pre-trained generative model and puts it into inference mode. Internally, the module stores a calibration task library containing multiple paradigms, such as motion imagery in different directions and visual flickering at different frequencies. For each candidate task in the library, the module simulates the EEG signal characteristics it might induce and inputs these simulated features into the pre-trained model, specifically observing the activation intensity and pattern changes of neurons in the model's internal "cognitive subspace." Using an information-theoretic metric (such as calculating the changes in the distribution of the cognitive subspace or entropy reduction caused by the simulated features), the module calculates the expected information gain from performing each candidate task, quantifying the task's potential contribution to revealing the unique cognitive patterns of the current user (relative to the general population).

[0040] This step transforms the traditional "blind traversal" calibration into "intelligent guided" calibration. By utilizing the cognitive subspace of a pre-trained model as an evaluator, the system can pre-determine the effectiveness of different calibration tasks in distinguishing user-specific cognitive patterns at the level of neural representation. This allows for the fundamental filtering out of redundant or ineffective tasks before real data collection, laying the algorithmic foundation for efficiently acquiring the user's most discriminative neural features. This is a core decision-making step that significantly shortens the overall calibration time.

[0041] After evaluating all preset tasks, the active learning module ranks them according to their expected information gain. Based on an optimization objective (e.g., maximizing total information gain under a total time constraint, or minimizing the number of tasks under a given information gain objective), the module dynamically plans an optimal task execution sequence using an algorithm. Subsequently, the system clearly presents this customized task sequence to the user through a graphical user interface or audiovisual prompts. Each task in the sequence has clear start, execution, and end prompts, such as sequentially displaying instructions and stimuli like "Imagine your left hand moving" and "Look at the flashing square in the center of the screen," guiding the user to complete the tasks in sequence.

[0042] This step transforms the intelligent assessment from the previous step into an executable action plan and enables human-computer interaction with the user. By dynamically generating optimal sequences, it ensures that every second of calibration time spent by the user is used to perform tasks that contribute most to building their personal model, completely avoiding redundancy or deficiencies caused by fixed processes. This directly and significantly increases the "information density" of the calibration process, compressing the commonly criticized traditional calibration time of tens of minutes to an extremely short range while ensuring the personalized quality of subsequent models, greatly improving the user experience and system usability.

[0043] As the user begins each calibration task according to the instructions, the system synchronously and continuously acquires raw brain voltage signals through the high-density EEG cap worn by the user. These analog signals are amplified, filtered, and converted from analog to digital to form a high-quality digital EEG time-series data stream. This data stream is fed in real time into a pre-trained, decoupled generative model. The model processes these EEG signals in a forward propagation manner, and its internal network, based on the specific patterns of the input, outputs in parallel, time-synchronized virtual functional near-infrared spectral signal data streams from its learned mapping relationships. These signals simulate the temporal process of changes in hemoglobin concentration in the cerebral cortex.

[0044] High-density EEG acquisition ensures the richness of spatial detail in the original signal. The core innovation lies in leveraging the generative capabilities of a pre-trained model to generate hemodynamic signals in real time, which would otherwise require additional, expensive, and bulky near-infrared equipment, even with only a single EEG sensor. This not only instantly creates multimodal conditions at the data level, providing a foundation for subsequent fusion, but also ensures that the generated virtual signal is precisely aligned with the EEG in time, avoiding synchronization errors that are difficult to completely eliminate with physical multi-sensor systems.

[0045] After all calibration tasks are completed, the system organizes all acquired data segments. It retains a timestamp accurate to the millisecond for each sampling point. Based on these timestamps, the processing program concatenates and aligns the real signal data segments from the EEG acquisition hardware with the virtual signal data segments output from the generative model. Then, the system may perform preliminary feature extraction on these two aligned raw signals (such as calculating the EEG band power, the trend of oxyhemoglobin changes in the virtual signal, etc.), and organize these feature values, along with corresponding task labels, timestamps, and other information, into a structured data file (such as a matrix or database record in a specific format). This file is uniquely identified and stored as the user's initial feature set.

[0046] Strict timestamp alignment ensures the consistency of causal relationships across multimodal data in the temporal domain, which is crucial for training a reliable generative model. The resulting structured feature set is a high-quality, cross-modal, labeled mini-dataset that encapsulates the most informative personalized neural activity patterns provided by the user in the shortest possible time, providing direct and sufficient input for precise and efficient parameter fine-tuning of the model.

[0047] Specifically, the steps for adjusting model parameters using feature sets, updating individual physiological mappings, and constructing a personalized generator are as follows: The user-specific initial feature set is input into the model, and forward and backward propagation is performed only on the encoder path responsible for individual physiological representation in the decoupled network. A gradient pruning fine-tuning strategy is adopted to update the connection weights of the individual physiological encoder, while the parameters related to general cognitive representation are fixed by dynamic parameter masking. Analyze the activation patterns of the fine-tuned model in the cognitive subspace, select the clusters of neurons with the highest activation intensity, and form a subset of user-strongly associated general features; The finely tuned individual physiological encoder, the locked general cognitive module, and the unchanged task intent decoder are recombined and solidified into a lightweight personalized generator file.

[0048] It should be noted that the system invokes the model training framework, loading the structured user-specific initial feature set obtained in the previous step as training data. When starting the training loop, the system configures the computation graph to ensure the complete forward propagation process, calculating the loss between the predicted output and the true label. However, during the crucial backpropagation to compute gradients, the system selectively blocks gradient backflow in most computational paths within the network, strictly limiting gradient computation and parameter updates to specific sub-network parts within the model designated as "individual physiological encoders." This means that error signals and weight adjustments only apply to those network layers that learn the user-specific physiological background.

[0049] This mechanism ensures that the capture and learning of individual user physiological characteristics can be carried out efficiently without disturbing the valuable general cognitive rules already learned by the model. This is the primary technical guarantee for achieving rapid and stable personalization, and avoids model performance fluctuations or forgetting of general knowledge caused by global updates.

[0050] When iteratively updating the parameters of the individual physiological encoder, the system applies a gradient pruning operation after each gradient calculation. This involves checking the norm of the gradient vector; if it exceeds a preset threshold, the gradient is scaled down proportionally to prevent excessively large update steps. Simultaneously, the system calls a dynamic parameter mask matrix prepared during initialization, which corresponds one-to-one with the model parameters. During the parameter update step, the system performs a dot product between the calculated update amount and the mask. For parameters marked "locked" in the mask, corresponding to general cognitive representations, their update amounts are forcibly reset to zero, thus ensuring these parameters are completely fixed and do not change with this fine-tuning.

[0051] By combining gradient pruning and dynamic parameter masking, a dual constraint of stability and protection is established during the model personalization process. Gradient pruning prevents gradient explosion or unstable updates that may occur when optimizing on limited data, ensuring smooth convergence of the learning process. Dynamic parameter masking enforces a knowledge protection strategy from the outset, ensuring that core network parameters representing cross-user common cognition remain frozen and absolutely unaffected by current user data. This dual protection allows the personalization process to flexibly adapt to individuals while firmly safeguarding the foundation of the model's generalization ability.

[0052] After fine-tuning the individual physiological encoder, the system does not directly use the entire general cognitive module. Instead, it runs an analysis process: the user-calibrated feature set is re-inputted into the current model, and the system specifically monitors and records the activation values ​​of all neurons in the model's internal "cognitive subspace" for each calibration task. The system performs cluster statistical analysis on this activation data to identify groups of neurons (i.e., "neuronal clusters") that consistently exhibit significantly higher activation levels than the mean baseline when faced with specific user data. The system records the index identifiers of these specific neurons.

[0053] It recognizes that even for general cognitive modules, different users may tend to rely on different neural pathways or combinations of features. By analyzing activation patterns, the system can automatically identify the general cognitive representations that are most relevant and contribute the most to the user's current cognitive pattern, thus forming a more compact and personalized "user-strongly correlated general feature subset." This further optimizes the efficiency of model resource utilization and provides more accurate general knowledge components for generating virtual signals that better match users' cognitive habits.

[0054] After the personalization process is complete, the system initiates the model reconstruction and export process. It first creates a new model container in memory, then assembles three parts according to predefined interfaces: first, the "individual physiological encoder" parameters updated in step two; second, the "general cognitive module" extracted from the original model, but retaining only the connections corresponding to the "user-strongly associated general feature subset" based on the indexes selected in step three; and third, the "task intent decoder" directly copied from the original model without any modifications. The system serializes and compresses this assembled new network structure and its parameters, which have a smaller parameter size than the original complete model, into a single model file and stores it in local storage.

[0055] Through modular reconfiguration, the system no longer produces a large, redundant model, but rather a compact generator customized specifically for the user, with irrelevant parts removed. This significantly reduces the computational resources and memory consumed by the model during inference, improves the speed and efficiency of real-time signal generation, and meets the stringent requirements of edge device deployment. This generated independent file serves as the core engine for all subsequent real-time brain-computer interface interactions for the user, marking the final completion of the efficient and personalized process.

[0056] Specifically, the personalized generator converts real-time EEG into virtual signals, which are then fused with the original signals to form a fused representation. The specific steps are as follows: The user's raw EEG signals are continuously collected, and after standardized preprocessing, they are input into the personalized generator. In the personalized generator, the individual physiological encoder extracts the specific components of the signal, while the general cognitive module extracts the homeostatic components; Based on the extracted components, the decoding part of the generator synthesizes a virtual multimodal signal with temporal characteristics of hemoglobin concentration changes; The frequency domain features of the original EEG signal are concatenated with the time domain features of the virtual signal, and then dimensionality reduction and fusion are performed through a shallow neural network layer to output a unified fused feature vector.

[0057] It should be noted that during the real-time interaction phase, the EEG acquisition device worn by the user continuously records scalp potential signals at a fixed sampling rate, forming a raw EEG data stream. This data stream is first fed into a preprocessing pipeline, where it undergoes bandpass filtering to retain effective frequency bands, removes power frequency interference, and is standardized with zero mean and unit variance to eliminate amplitude deviations caused by electrode contact or physiological fluctuations. The standardized EEG signal frames after preprocessing are immediately loaded into the input port of the personalized generator model, which has been instantiated and is in inference mode in the system memory.

[0058] Through this step, the system transforms continuous bioelectrical signals into a standardized data stream suitable for processing by artificial intelligence models. Crucially, the processed signal is directly fed into a "personalized generator" tailored to the current user, rather than a generic pre-trained model. This ensures that from the very first signal input stage, the system fully considers the user's unique physiological background, laying an accurate and stable data foundation for the subsequent generation of highly personalized virtual signals. This is the initial guarantee for achieving real-time, precise "virtual-real fusion."

[0059] After standardized EEG signals enter the personalized generator, they flow in parallel through two core sub-modules within the network. In the "Individual Physiological Encoder" path, the network layers, based on previously fine-tuned user-specific parameters, focus on separating and extracting features from the signal that characterize the user's unique physiological traits, such as personalized basal rhythms, distinctive noise patterns, or stable cortical response characteristics. Simultaneously, in the "General Cognitive Module" path, the network utilizes its locked parameters, which imply group commonalities, to extract stable, cross-user cognitive features from the same frame of signal, such as common neural oscillation patterns related to attention, motor readiness, or visual processing.

[0060] Through this step, the generator performs its first intelligent deconstruction of the input signal. Its core function lies in utilizing pre-constructed prior knowledge of both "individual" and "general" dual paths to decompose the mixed neural signals at their source. This parallel extraction mechanism ensures the simultaneous capture of both user-specific "specific" components and relatively stable "steady-state" cognitive components in the signal. This provides clear and complementary feature inputs for the next step of synthesizing complete and physiologically plausible virtual multimodal signals, making it a crucial feature preparation stage for achieving high-quality generation.

[0061] Two types of feature vectors, extracted by the individual physiological encoder and the general cognitive module respectively, are fed together into the decoder network of the generator. During pre-training and personalized reconstruction, this decoder has learned how to map these neurophysiological features from EEG into hemodynamic response patterns with physiological coupling. The decoder network processes these features, which integrate specific and general information, through a series of nonlinear layers, ultimately generating a continuous temporal signal at its output layer. This signal morphologically simulates the temporal process of the relative changes in the concentrations of oxyhemoglobin and deoxyhemoglobin in a specific region of the cerebral cortex—a virtual functional near-infrared spectral signal.

[0062] By using a decoder that has learned the neural-vascular coupling rules, the system successfully synthesized physiological signals of another modality (fNIRS) from EEG features of a single modality in real time. This essentially creates a "signal multiplication" effect, generating a virtual channel containing metabolic function information at the software level, even with only physical EEG hardware. The generated virtual signal is not only strictly synchronized with the original EEG in time, but also provides complementary dimensions in terms of information content, such as spatial positioning and metabolic level, greatly enriching the information sources that can be used for intent decoding.

[0063] The system performs a Fast Fourier Transform on the raw EEG signal, calculating its power spectral density in different frequency bands (e.g., δ, θ, α, β, γ waves) to form a set of frequency domain features. Simultaneously, it calculates the mean, slope, variance, and other temporal statistical features of the newly generated virtual fNIRS signal within a time window. Subsequently, these two sets of feature vectors, originating from different physical sources (real and virtual) and representing different characteristics (frequency and time), are concatenated to form a high-dimensional hybrid feature vector. This vector is then fed into a lightweight, shallow neural network (e.g., a feedforward network with one or two fully connected layers). This network learns how to filter redundant information, highlight effective correlations, and outputs a unified, fused feature vector with lower dimensionality and higher information density.

[0064] It doesn't simply mix the raw signals or decisions at the level, but rather performs deep fusion across domains (frequency and time) and sources (real and virtual) at the feature level. Through optimization and dimensionality reduction of shallow networks, it preserves the high temporal resolution and frequency characteristics of EEG while integrating the hemodynamic temporal variation information of virtual signals, generating a more robust and discriminative unified feature representation. This fused feature vector is the most direct input to the subsequent intent decoder, and its quality directly determines the final control accuracy and reliability of the entire system.

[0065] Specifically, the decoding and fusion representation, parsing of task intent, and output of control commands involve the following steps: The fused feature vector is input into a pre-trained multi-level gated recurrent unit network; The first layer of the network learns the short-term temporal dependencies of the fused features, and the second layer learns the long-term contextual dependencies across modalities. The network's output layer is connected to a task-specific classifier that maps the learned deep representations to a probability distribution of a predefined intent category. The intent category with the highest probability is selected, and combined with the current device state context, it is converted into a control instruction with execution priority and parameters, and sent to the peripheral driver interface.

[0066] It should be noted that the system calls the inference engine deployed on the edge device, taking the unified fused feature vector generated in the previous step as a continuous temporal data sequence, and inputting it frame by frame or by time window into the input layer of a multi-level gated recurrent unit network model that has been loaded into memory. This network model has been pre-trained and optimized on a large amount of multimodal neural signal data, and its parameters are fixed, specifically designed for efficient encoding of neural features with complex temporal dependencies.

[0067] Through this step, the system selects a specialized computational architecture for handling the complex spatiotemporal characteristics of "virtual-real fusion." Gated recurrent unit networks (GRUs) are employed due to their superior performance in capturing long-term dependencies in time series. Pre-training ensures the network possesses a basic ability to recognize neural temporal patterns. Inputting the fused feature vector, deeply optimized by the preceding steps, allows the network to focus on extracting the highest-level temporal information, laying a solid and specialized computational foundation for subsequent accurate intent inference. This serves as the key interface for realizing the transformation from features to intent.

[0068] During network inference, the first-layer gated recurrent unit receives the input fused feature sequence. Its internal gating mechanism (update gate and reset gate) mainly regulates the flow and forgetting of information between adjacent time steps, thus focusing on capturing short-term patterns of local, high-frequency changes in the sequence, such as transient fluctuations in power at a specific EEG frequency or rapid fluctuations in virtual signal features. The output sequence of the first layer is then fed into the second-layer gated recurrent unit, which learns dependencies across longer time windows based on longer hidden states. It also focuses on integrating the interaction and co-evolution patterns between features from two different sources, the raw EEG and the virtual signal, to form a deep representation containing rich cross-modal context.

[0069] Through this step, the system achieves layered and refined analysis of fused features over time. The first layer rapidly responds to instantaneous changes, capturing the "momentary cues" that generate intent; the second layer takes a holistic view, understanding the continuous "intent profile" outlined by multimodal signals. This layered processing mechanism effectively simulates the multi-layered temporal structure that human intent formation and expression may exhibit in neural activity, enabling the decoding network not only to perceive the user's immediate command tendencies but also to understand their coherent intent trends over a period of time, significantly improving the coherence and accuracy of intent recognition in complex or continuous tasks.

[0070] The final hidden state vector, containing multi-level temporal and cross-modal information, obtained after processing by two layers of gated recurrent units, is passed to the task-specific classification head at the end of the network. This classification head typically consists of one or more fully connected layers, and its parameters are configured or fine-tuned according to the specific control task to be performed by the current user (such as two-dimensional cursor movement, robotic arm grasping, etc.). The classification head performs nonlinear transformation and normalization on this deep spatiotemporal representation, and finally generates a probability vector at its output, where the value of each element represents the probability that the input feature sequence corresponds to a preset, discrete user intent category (such as "move up", "turn left", "confirm").

[0071] Through this step, the system completes the "translation" or "decoding" from abstract, continuous spatiotemporal neural representations to specific, discrete operational instructions. The task-specific classification head's role is to specifically associate the general temporal patterns extracted by the preceding network layers with the specific action options of the current application scenario. The output is a probability distribution rather than a single label, providing the system with confidence information for decision-making. This enables subsequent steps to perform confidence-based judgments or fault-tolerant processing, enhancing the system's ability to cope with uncertainty and distinguish ambiguous intentions.

[0072] The system reads the probability distribution output by the classification header and selects the intent category with the highest probability value as the initial recognition result. Subsequently, an instruction generation module queries the current state context information of the external controlled device, such as the current position of the robotic arm's end effector, its remaining range of motion, or whether the system is currently in a specific operating mode. Based on this context, the module semantically enriches and parameterizes the initial intent: for example, the "move forward" intent, combined with the robotic arm's current position, calculates a specific, achievable target coordinate and motion speed parameter, and assigns it an execution priority. Finally, this complete, executable instruction is encapsulated into a data packet of a specific protocol and sent to the peripheral device's drive controller via the communication interface.

[0073] This process successfully maps neural signals to abstract intentions, further grounding these abstract intentions into safe, effective, and executable action commands within a specific physical environment. By incorporating device state context, it avoids issuing dangerous commands that could lead to collisions or boundary violations, achieving intelligent error prevention. Generating commands with priorities and specific parameters makes control more precise, compliant, and aligned with practical engineering constraints, ultimately seamlessly transforming "thought control" into "device action," achieving a crucial leap for brain-computer interfaces from laboratory concepts to practical applications.

[0074] Specifically, the online model update based on interactive feedback, and the adjustment of the intent module according to new task data to achieve migration, involve the following steps: The system continuously monitors the execution results of instructions and implicit user feedback. When multiple consecutive low-confidence outputs or execution failures are detected, the model is updated. Collect recent EEG signals within a small window, generate virtual signals, and corresponding feedback labels to form an incremental dataset; Using only this incremental data, iterative updates of the decoding layer parameters related to task intent in the personalized generator are performed with a small learning rate; When a user starts a new task, the basic task intent module is loaded, and the rapid calibration and reorganization process is repeated to achieve migration to the new task without retraining the entire model.

[0075] It should be noted that the system runs an independent performance monitoring process in the background. This process receives the "intent confidence" data stream from the decoding module in real time and accesses "command execution result" feedback from external devices or environmental sensors. Simultaneously, it obtains implicit feedback by analyzing user operation intervals, recurring trigger patterns, or through simple interactive interfaces (such as confirmation / cancel buttons). This process has dynamic performance thresholds and a continuous anomaly counting mechanism. When the decoding confidence is detected to be consistently below the threshold, or when external feedback explicitly indicates that the operation has failed a preset number of times, the monitoring process generates a trigger signal, automatically waking up the system's model update process.

[0076] Through this step, the system transforms from "passive execution" to "proactive perception and self-maintenance." Its core function is to establish a closed-loop, performance-based monitoring mechanism, enabling the system to autonomously and promptly detect performance degradation in its decoding capabilities caused by user state drift, environmental interference, or long-term use. This threshold-based intelligent triggering avoids the resource waste caused by periodic or indiscriminate updates, ensuring that updates only occur when necessary. It serves as the first intelligent line of defense for maintaining the long-term stable and reliable operation of the system.

[0077] Once the update process is triggered, the system immediately activates a data collector. This collector backtracks and retrieves all relevant cached data within a sliding time window (e.g., the past few minutes): this includes raw EEG signal segments, virtual multimodal signals synchronized by the personalization generator, and "feedback labels" (such as "success" or "failure," or more granular confidence scores) associated with these data segments, provided by the performance monitoring module. The system precisely aligns these multimodal data segments and labels along the timeline, cleans and packages them to form a smaller but highly relevant and accurately labeled incremental dataset.

[0078] By selectively collecting recent "problem data" directly related to performance degradation, the system constructs a highly focused incremental dataset. Its value lies in the fact that this data directly reflects the current model's shortcomings in new situations or the user's latest neural patterns, enabling subsequent model optimization to be "targeted," using the least amount of relevant data to address the most pressing performance issues, greatly improving the efficiency and relevance of adaptive learning.

[0079] The system loads the incremental dataset into the training environment and the currently used personalized generator model. Upon initiating fine-tuning, the system strictly limits the range of parameter updates. Using a preset parameter selector, it unlocks only the network layer parameters directly related to the "task intent decoding" function (typically the last few layers or specific branches of the generator decoding part), while parameters for other parts of the model (such as the individual physiological encoder and the general cognitive module) are completely frozen. Subsequently, a very low learning rate is used for a small number of iterative training iterations on the incremental dataset, allowing the model to gently adjust its intent mapping to adapt to newly observed data patterns.

[0080] Through the dual constraints of "local fine-tuning" and "low learning rate," the system achieves extremely conservative and controllable updates. Its function is to fine-tune only the highest-level, task-specific intent-signal relationships without disrupting the established and valuable personalized physiological and general cognitive mappings. This update method is like calibrating the pointer of an instrument without altering its intricate internal mechanisms, thus rapidly adapting to new changes while maximizing the stability and consistency of the system's main functions, avoiding model crashes or drastic performance fluctuations caused by online learning.

[0081] When a user switches to a completely new control task via interface selection or system configuration, the system loads a "base task intent module" (a pre-trained, lightweight decoder subnetwork or parameter block) corresponding to the new task from the model library. The system then guides the user through a highly simplified calibration process designed for the new task (potentially involving only a few trials of key paradigms). Using this new calibration data, the system runs a rapid reconfiguration process: primarily connecting and fine-tuning the newly loaded intent module with the existing, highly personalized generator pre-processors (individual and general parts), quickly forming a complete, personalized generator variant for the new task.

[0082] By employing modular design, "task-specific knowledge" is encapsulated as pluggable, independent components. When faced with a new task, the system does not need to train a completely new, massive model from scratch; instead, it only needs to "replace the task plugin" and quickly adapt the interface. This is equivalent to rapidly learning a new "skill" based on an already built personalized "brain." This approach reduces the cost of cross-task migration from hours or even days of retraining to minutes of rapid calibration and reorganization, revolutionarily enhancing the system's versatility, flexibility, and practical value.

[0083] This embodiment also provides a brain-computer interface control system based on multi-modal fusion, including: The model initialization module loads a pre-trained decoupled generative model, which separates three types of representations: individual physiology, general cognition, and task intent. The active calibration module guides the user through calibration, collects EEG signals, and generates initial virtual multimodal signals to form a user feature set. The parameter adaptation module uses feature sets to adjust model parameters, update individual physiological mappings, and build a personalized generator. The signal fusion module and the personalized generator convert real-time EEG into virtual signals, which are then fused with the original signals to form a fused representation. The intent decoding module decodes and fuses representations, parses task intents, and outputs control commands. The online optimization module updates the model online based on interactive feedback and adjusts the intent module according to new task data to achieve migration.

[0084] This embodiment also provides a computer device applicable to the brain-computer interface control method based on multi-modal fusion, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the brain-computer interface control method based on multi-modal fusion as proposed in the above embodiment.

[0085] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0086] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the brain-computer interface control method based on multi-mode fusion as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0087] In summary, this invention constructs a "lightweight acquisition, heavy-duty analysis" architecture by loading a pre-trained decoupled generative model and dynamically synthesizing virtual multimodal signals using single-modal EEG signals. This fundamentally reduces reliance on multiple physical sensors and improves the system's portability and user-friendliness. By decoupling neural representations into three categories—individual physiology, general cognition, and task intent—and employing active learning-guided calibration, it achieves rapid personalized adaptation for new users, significantly shortening calibration time. By fine-tuning and recombining only a specific subset of parameters in the model and supporting online updates based on interactive feedback, the system possesses the ability to rapidly migrate across tasks and continuously optimize. Ultimately, this results in a highly generalizable, evolvable, and practical integrated brain-computer interface solution.

[0088] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A brain-computer interface control method based on multi-modal fusion, characterized in that, Includes the following steps: Load a pre-trained decoupled generative model that separates three types of representations: individual physiology, general cognition, and task intent; The system guides users to calibrate, collects EEG signals, generates initial virtual multimodal signals, and forms a user feature set. Use feature sets to adjust model parameters, update individual physiological mappings, and build personalized generators; The personalized generator converts real-time EEG into virtual signals, which are then fused with the original signals to form a fused representation. Decode and fuse representations, parse task intent, and output control commands; The model is updated online based on interactive feedback, and the intent module is adjusted according to new task data to achieve migration.

2. The brain-computer interface control method based on multi-modal fusion as described in claim 1, characterized in that: The pre-trained decoupled generative model is loaded, which separates three types of representations: individual physiological, general cognitive, and task intention. The specific steps are as follows: The basic generative model is obtained from the cloud model library. This model adopts a variational autoencoder structure, and its latent space is constrained into three independent subspaces by decoupling the loss function. The model is deployed to a local edge computing device and loaded with model weights pre-trained for large-scale group data; Initialize a dynamic parameter mask matrix, which is used to identify and lock model parameters related to general cognitive representations in subsequent steps; The model is started, entering a ready-to-fit state, preparing to receive the user's EEG signal stream and perform forward computation.

3. The brain-computer interface control method based on multi-modal fusion as described in claim 2, characterized in that: The process of guiding user calibration, collecting EEG signals, generating initial virtual multimodal signals, and forming a user feature set involves the following steps: The active learning module evaluates the expected information gain of multiple pre-trained calibration tasks based on the cognitive subspace output of the pre-trained model. Based on the evaluation results, an optimal calibration task sequence is dynamically generated and displayed to the user, which contains specific paradigmatic motor imagery and visual stimuli. High-density EEG signals were simultaneously acquired when the user performed the calibration task and input into the decoupled generative model to obtain the corresponding virtual functional near-infrared spectral signals. The collected real EEG signals and the generated virtual signals are aligned by timestamp and packaged into a structured user-specific initial feature set.

4. The brain-computer interface control method based on multi-modal fusion as described in claim 3, characterized in that: The specific steps for adjusting model parameters using feature sets, updating individual physiological mappings, and constructing a personalized generator are as follows: The user-specific initial feature set is input into the model, and forward and backward propagation is performed only on the encoder path responsible for individual physiological representation in the decoupled network. A gradient pruning fine-tuning strategy is adopted to update the connection weights of the individual physiological encoder, while the parameters related to general cognitive representation are fixed by dynamic parameter masking. Analyze the activation patterns of the fine-tuned model in the cognitive subspace, select the clusters of neurons with the highest activation intensity, and form a subset of user-strongly associated general features; The finely tuned individual physiological encoder, the locked general cognitive module, and the unchanged task intent decoder are recombined and solidified into a lightweight personalized generator file.

5. The brain-computer interface control method based on multi-modal fusion as described in claim 4, characterized in that: The personalized generator converts real-time EEG signals into virtual signals, which are then fused with the original signals to form a fused representation. The specific steps are as follows: The user's raw EEG signals are continuously collected, and after standardized preprocessing, they are input into the personalized generator. In the personalized generator, the individual physiological encoder extracts the specific components of the signal, while the general cognitive module extracts the homeostatic components; Based on the extracted components, the decoding part of the generator synthesizes a virtual multimodal signal with temporal characteristics of hemoglobin concentration changes; The frequency domain features of the original EEG signal are concatenated with the time domain features of the virtual signal, and then dimensionality reduction and fusion are performed through a shallow neural network layer to output a unified fused feature vector.

6. The brain-computer interface control method based on multi-modal fusion as described in claim 5, characterized in that: The decoding and fusion representation, parsing the task intent, and outputting control commands involve the following steps: The fused feature vector is input into a pre-trained multi-level gated recurrent unit network; The first layer of the network learns the short-term temporal dependencies of the fused features, and the second layer learns the long-term contextual dependencies across modalities. The network's output layer is connected to a task-specific classifier that maps the learned deep representations to a probability distribution of a predefined intent category. The intent category with the highest probability is selected, and combined with the current device state context, it is converted into a control instruction with execution priority and parameters, and sent to the peripheral driver interface.

7. The brain-computer interface control method based on multi-modal fusion as described in claim 6, characterized in that: The process involves updating the model online based on interactive feedback, adjusting the intent module according to new task data, and achieving migration. The specific steps are as follows: The system continuously monitors the execution results of instructions and implicit user feedback. When multiple consecutive low-confidence outputs or execution failures are detected, the model is updated. Collect recent EEG signals within a small window, generate virtual signals, and corresponding feedback labels to form an incremental dataset; Using only this incremental data, iterative updates of the decoding layer parameters related to task intent in the personalized generator are performed with a small learning rate; When a user starts a new task, the basic task intent module is loaded, and the rapid calibration and reorganization process is repeated to achieve migration to the new task without retraining the entire model.

8. A brain-computer interface control system based on multi-modal fusion, based on the brain-computer interface control method based on multi-modal fusion as described in any one of claims 1 to 7, characterized in that: include, The model initialization module loads a pre-trained decoupled generative model, which separates three types of representations: individual physiology, general cognition, and task intent. The active calibration module guides the user through calibration, collects EEG signals, and generates initial virtual multimodal signals to form a user feature set. The parameter adaptation module uses feature sets to adjust model parameters, update individual physiological mappings, and build a personalized generator. The signal fusion module and the personalized generator convert real-time EEG into virtual signals, which are then fused with the original signals to form a fused representation. The intent decoding module decodes and fuses representations, parses task intents, and outputs control commands. The online optimization module updates the model online based on interactive feedback and adjusts the intent module according to new task data to achieve migration.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the brain-computer interface control method based on multi-modal fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the brain-computer interface control method based on multimodal fusion as described in any one of claims 1 to 7.

Images