Brain-computer interface interactive control device and method based on multi-modal brain signal fusion

Abstract

This invention discloses a brain-computer interface (BCI) interactive control device and method based on multimodal brain signal fusion. The device includes a perception acquisition layer, an edge processing layer, a fusion decoding layer, a control application layer, and a closed-loop optimization cloud platform. The perception acquisition layer synchronously acquires multimodal brain signals; the edge processing layer uses a conditionally aligned temporal diffusion model for signal enhancement and completion; the fusion decoding layer uses a lightweight graph neural network to achieve multimodal feature fusion and intent decoding; the control application layer provides adaptive control mapping and multimodal feedback; and the closed-loop optimization cloud platform achieves continuous performance optimization through federated learning and incremental learning. This invention effectively solves the problems of single signal, insufficient accuracy, and poor practicality in traditional BCIs, significantly improving the system's decoding accuracy, real-time performance, and personalization capabilities. It can be widely applied in fields such as smart home control, medical rehabilitation training, and industrial control.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface technology, and more specifically to a brain-computer interface interactive control device and method based on multimodal brain signal fusion. Background Technology

[0002] Currently, brain-computer interface (BCI) technology, as an important bridge connecting the human brain with external devices, has received widespread attention from academia and industry in recent years. However, existing BCI technologies still face many technical bottlenecks and challenges in moving from the laboratory to practical applications.

[0003] At the signal acquisition level, traditional brain-computer interfaces rely heavily on single-modality EEG signal acquisition. This limitation of a single signal source results in insufficient and incomplete neural information acquired by the system. Although EEG signals have high temporal resolution and can capture millisecond-level changes in neural electrical activity, their spatial resolution is low and they are easily affected by physiological artifacts such as electromyography and electrooculography, and are more susceptible to environmental electromagnetic noise. This single-modality technical limitation severely restricts the decoding accuracy and reliability of brain-computer interface systems.

[0004] At the signal processing and decoding level, most existing methods use traditional machine learning algorithms or relatively simple deep learning models, which perform poorly when processing complex and variable brain signals. Summary of the Invention

[0005] To address these issues, the present invention provides a brain-computer interface interactive control device and method based on multimodal brain signal fusion, thereby solving the problems in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A brain-computer interface interactive control device based on multimodal brain signal fusion includes:

[0008] The sensing and acquisition layer is used to acquire raw data of multimodal brain signals, including EEG electrodes, functional near-infrared spectroscopy probes, and electromyography sensors.

[0009] The edge processing layer connects to the sensing and acquisition layer and is used to preprocess and enhance the raw data, and output the processed raw data. The edge processing layer includes a signal preprocessing and alignment module and a generative AI signal enhancement and completion module. The generative AI signal enhancement and completion module is built based on the conditional alignment temporal diffusion model, which can restore and enhance the signal through cross-modal information transmission when the signal quality is damaged.

[0010] The fusion decoding layer includes a lightweight graph neural network module and an intent classifier. The lightweight graph neural network module can perform deep fusion of features extracted from multimodal signals and output processed high-level feature maps. The intent classifier maps the high-level feature maps into specific, executable user intent commands.

[0011] The control application layer converts the decoded intent commands into specific control commands to enable interactive connections with external devices; the fusion decoding layer includes an external device control command module and a real-time visual / haptic feedback module.

[0012] The closed-loop optimization cloud platform is used to continuously optimize system performance and connects to the fusion decoding layer and control application layer, including a personalized model incremental learning module and a user data and model optimization module.

[0013] Furthermore, the conditionally aligned temporal diffusion model includes a conditional encoder, a diffusion processor, and a denoising network. The conditional encoder is responsible for feature extraction and encoding of the input multimodal signal, mapping signals of different modes to a unified feature space. The diffusion processor simulates the gradual degradation process of the signal from a clear state to a noisy state, as well as the inverse process of recovering from a noisy state to a clear state. The denoising network is responsible for predicting and removing noise during the recovery process, gradually reconstructing a clear signal.

[0014] Furthermore: the calculation formula for the condition encoder is:

[0015] ;

[0016] Where z is the joint embedding vector; X egg The EEG signal matrix is ​​T×C, where T is the time point and C is the number of channels; X fnirs Let fNIRS be the signal matrix T×D, where D is the number of detectors; θ is the encoder network parameter; and E represents the conditional decoder.

[0017] The formula for calculating the forward process (signal degradation) of the diffusion processor is:

[0018] ;

[0019] The formula for calculating the reverse process (signal recovery) is:

[0020] ;

[0021] Where, x t Let be the noisy signal at step t; The cumulative noise scaling factor is derived from the hyperparameter sequence {β}. t The calculation yielded the result. where i is the index variable and t is the current total number of diffusion steps. To adjust parameters in the morning, Indicates the percentage of signals retained in a single step; θ represents the noise prediction value output by the denoising neural network; N(·) is a Gaussian distribution with the mean being the first parameter in parentheses and the covariance matrix being the second parameter; I is the identity matrix.

[0022] Denoising networks can predict the original clean signal based on conditional latent variables. The calculation formula is as follows:

[0023] ;

[0024] Where z is the joint embedding vector; For decoder network parameters; x t Let be the noisy signal at step t; D represents the denoising network; This indicates a clean signal indicating reconstruction.

[0025] Furthermore: the lightweight graph neural network module adopts the NexusNet architecture; the lightweight graph neural network of the NexusNet architecture first constructs a brain functional connectivity graph. The nodes in the brain functional connectivity graph represent the positions of various electrodes or probes, and the node features include the time-frequency features and statistical features of the multimodal signals collected from that position; the edges of the brain functional connectivity graph include three types: spatial edges are established based on the physical distance between electrodes, reflecting the spatial structural relationship of the brain; functional edges are established based on the correlation between signals, reflecting the functional connection between different brain regions; and routing edges are established based on the shortest path of information transmission, simulating the information flow process inside the brain.

[0026] Furthermore: the calculation formula for the functional edge is:

[0027] ;in, The distance is Euclidean; i and j are node indices; Displacement distance attenuation coefficient;

[0028] The formula for calculating functional edges is:

[0029] ;

[0030] Among them, F ij Represents the functional connection weights; ρ(·) is the Pearson correlation coefficient; x i Let x be the signal time series of node i; j Let x be the signal time series of node j; T is the time window length; k is the time step index; x i，k Let x be the signal value of node i at time k; j，k μ is the signal value of node j at time k; iμ is the signal mean at node i; j Let be the signal mean of node j;

[0031] The formula for calculating the routing edge is:

[0032] ;

[0033] in, This means that if j is located in the set of shortest paths to node i.

[0034] Furthermore, the lightweight graph neural network employs depthwise separable graph convolution technology, which decomposes the traditional graph convolution operation into two independent steps: spatial convolution and channel convolution, significantly reducing the number of model parameters and computational complexity.

[0035] The formula for spatial convolution (aggregating neighbor information) is:

[0036]

[0037] The formula for calculating channel convolution (feature transformation) is:

[0038] ;

[0039] in, Let v be the embedding representation of node v at layer l+1; The function is a per-element nonlinear activation function; N(v) is the set of one-hop neighbors of node v; These are the parameters of the l-th layer spatial filter; Attention weights for routing; The original features of neighbor node u at layer l; DWConv is a depthwise separable convolution; Let v be the original feature of node v in layer L.

[0040] Furthermore: the lightweight graph neural network includes a routing attention mechanism; the routing attention mechanism enables the network to capture remote functional connections in the brain network;

[0041] The formula for calculating the dynamic focus on the critical path in the routing attention mechanism is:

[0042] ;

[0043] Where a is the learnable query vector; Wr is the route projection matrix; For feature splicing operations; Let be the routing projection matrix; hi, hj, and hk are the feature vectors of source node i, target node j, and target node k, respectively.

[0044] Furthermore, it also features a user proficiency adaptive algorithm that can dynamically adjust control parameters based on the user's experience. For novice users, the system adopts more conservative parameter settings to prevent accidental operation. As the user's proficiency increases, the system gradually improves control sensitivity and response speed, reduces unnecessary confirmation steps, and provides a more direct and smooth control experience.

[0045] The proficiency assessment model defines a two-dimensional state vector s=[p,r], where p is the recent task success rate and r is the reciprocal of the average response time. The proficiency level L∈{L0,L1,L2} is determined through fuzzy logic reasoning; the specific formula is as follows:

[0046] ;

[0047] in, , Membership scheduling function center parameter; final level selection is L with the largest membership degree. k L k s is the proficiency level identifier; s is the current system state feature vector; For performance indicators; For reliability indicators; This is the width parameter of the membership function.

[0048] Furthermore: the personalized model incremental learning module adopts a privacy protection mechanism, including: the cloud distributes the baseline personalized model to user devices; the devices use local data to generate personalized model updates; the encrypted personalized model updates are uploaded to the cloud; the cloud analyzes the personalized model updates of multiple devices to identify common feature patterns; and the optimized personalized model is distributed to each device.

[0049] To achieve the above objectives, the present invention also provides a brain-computer interface interactive control method based on multimodal brain signal fusion, comprising the following steps: Step 1: Multimodal signal acquisition and real-time preprocessing

[0050] The system initiates a multimodal signal synchronous acquisition process, acquiring EEG signals, monitoring changes in blood oxygen concentration, and detecting muscle activity. During the acquisition process, electrode impedance and signal quality are monitored in real time to ensure data reliability. After the raw signal is transmitted to the edge device via Bluetooth, it is immediately filtered, denoised, and time-domain aligned to eliminate power frequency interference and motion artifacts, providing high-quality standardized data for subsequent analysis.

[0051] Step 2: Generative AI Enhancement and Feature Extraction

[0052] The preprocessed signal is intelligently enhanced based on the conditionally aligned temporal diffusion model; when a certain modality signal quality degradation is detected, cross-modal reconstruction is performed using high-quality modal signals, and signal completion is achieved through the neurovascular coupling mechanism; the enhanced signal is input into the feature extraction module to construct a brain functional connectivity map containing multiple nodes, establish three types of connectivity relationships: spatial edges, functional edges, and routing edges, and form a complete brain network representation;

[0053] Step 3: Multimodal fusion and intent decoding

[0054] The lightweight graph neural network NexusNet performs deep processing on brain functional connectivity graphs, employing deep separable graph convolution and route attention mechanisms to dynamically fuse EEG temporal features and fNIRS spatial features. The intent classifier combines temporal context, task context, and user context to output the probability distribution and confidence score of multi-class intents, achieving accurate intent recognition.

[0055] Step 4: Adaptive Control and Interactive Feedback

[0056] Based on the intent recognition results and confidence levels, the system adaptively generates control commands; high-confidence results are directly mapped to device control commands, medium-confidence results trigger a confirmation process, and low-confidence results refuse execution; control parameters are dynamically adjusted according to user proficiency, with novices using conservative settings to prevent misoperation; experienced users enjoy a more direct and smooth control experience; execution results are promptly conveyed to the user through visual, tactile, and auditory multimodal feedback;

[0057] Step 5: Privacy-Preserving Data Collection and Upload

[0058] The system anonymizes the collection of key data during the interaction process while protecting user privacy; it uses differential privacy technology to process sensitive information to ensure individual privacy and security; and the processed data is regularly uploaded to the cloud optimization platform through an encrypted channel to provide a data foundation for model improvement.

[0059] Step 6: Cloud-based Federated Learning and Personalized Optimization

[0060] The cloud platform is based on a federated learning framework, which securely aggregates model updates from multiple users, extracts collective wisdom while protecting privacy, and improves the global basic model. At the same time, it builds feature profiles for individual users and generates personalized decoding models through a weighted fusion strategy, balancing generality and personalized needs. The incremental learning mechanism ensures that the model continues to improve while preventing knowledge forgetting.

[0061] Step 7: Model Distribution and Performance Monitoring

[0062] The optimized model undergoes compression and distillation to adapt to the resource constraints of embedded devices and is distributed to user terminals through a secure channel. The system monitors the performance of the new model in real-time. Based on monitoring results and user feedback, the system parameters are dynamically adjusted to ensure continuous optimization. The fault tolerance mechanism automatically activates the security confirmation process when the confidence level is low to ensure system reliability.

[0063] The present invention has the following advantages: it significantly improves decoding accuracy and intent recognition accuracy through multimodal brain signal fusion and generative AI enhancement technology; it improves real-time response speed by using a lightweight graph neural network; the personalized learning mechanism shortens user calibration time and supports three control modes: discrete, continuous and hybrid.

[0064] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0065] To more intuitively illustrate the prior art and this application, exemplary drawings are provided below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary drawings, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).

[0066] Figure 1 This is a system block diagram of the brain-computer interface interactive control device based on multimodal brain signal fusion according to the present invention.

[0067] Figure 2 This is a flowchart illustrating the implementation of a single instruction execution in the brain-computer interface interactive control device of the present invention.

[0068] Figure 3 This is a flowchart illustrating the implementation of periodic optimization in the brain-computer interface interaction control device of the present invention.

[0069] Figure 4 This is a flowchart of the brain-computer interface interactive control method based on multimodal brain signal fusion according to the present invention. Detailed Implementation

[0070] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these embodiments are merely for further explanation of the present invention and should not be construed as limiting the scope of protection of the present invention. Technical engineers in the field can make some non-essential improvements and adjustments to the present invention based on the above-described content. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] Please see Figures 1-3 A brain-computer interface interactive control device based on multimodal brain signal fusion, comprising a perception acquisition layer, an edge processing layer, a fusion decoding layer, a control application layer, and a closed-loop optimization cloud platform;

[0072] The perception and acquisition layer is responsible for collecting raw data, including multimodal brain signals, to ensure the integrity and accuracy of these signals. The edge processing layer performs preliminary processing and enhancement on the raw data collected by the perception and acquisition layer, providing a high-quality data foundation for subsequent analysis. The fusion and decoding layer is responsible for decoding the user's intent commands from the pre-processed raw data. The control and application layer converts the decoded intent commands into specific control commands to enable interaction with external devices. The closed-loop optimization cloud platform continuously improves the system's performance and personalization through continuous learning and optimization.

[0073] The sensing and acquisition layer employs a multimodal collaborative acquisition strategy to simultaneously acquire multiple types of brain signals in order to fully utilize the complementary characteristics of different signals. The sensing and acquisition layer consists of three main components: EEG electrodes, functional near-infrared spectroscopy probes, and electromyography sensors.

[0074] In this embodiment, the EEG electrodes are used, the functional near-infrared spectroscopy probe is an fNIRS optical probe, and the electromyography sensor is an EMG electromyography sensor.

[0075] The EEG electrodes are arranged according to the international 10-20 system standard, covering key brain regions such as the prefrontal cortex, motor cortex, and parietal lobe. Each EEG electrode is equipped with an independent impedance monitoring circuit to monitor the contact quality between the electrode and the scalp in real time. The sampling rate is set to 1000Hz, which can fully capture the detailed features of the EEG signal while taking into account the requirements of power consumption and data processing.

[0076] The fNIRS optical probe can monitor the dynamic changes in brain blood oxygen concentration in real time.

[0077] EMG (electromyography) sensors are used to monitor muscle activity in the face and head. This information is used for two main purposes: first, to detect and compensate for signal artifacts caused by muscle activity; and second, to provide auxiliary control information for certain brain-computer interface applications.

[0078] The edge processing layer includes a signal preprocessing and alignment module and a generative AI signal enhancement and completion module; the signal preprocessing and alignment module can preprocess the acquired raw multimodal brain signals and perform data alignment processing.

[0079] The generative AI signal enhancement and completion module, built on a conditionally aligned temporal diffusion model, can restore and enhance signals by transmitting cross-modal information even when signal quality is compromised. The core idea of ​​this module is to leverage the inherent correlation between multimodal signals; when the quality of a certain modality signal deteriorates, it uses high-quality signals from other modalities to reconstruct or enhance that signal.

[0080] Conditionally aligned temporal diffusion models are generative models based on deep learning. They establish a mapping relationship between EEG and near-infrared signals by learning the neurovascular coupling mechanism between the two signals. (Neurovascular coupling is a fundamental physiological mechanism of the brain, describing the close relationship between neural activity and cerebrovascular responses—when neural activity in a brain region increases, the metabolic demand in that region increases, leading to changes in local blood flow and oxygen concentration.) Conditionally aligned temporal diffusion models achieve mutual prediction and enhancement of cross-modal signals precisely by learning this physiological mechanism.

[0081] The conditionally aligned temporal diffusion model comprises three core components: a conditional encoder, a diffusion processor, and a denoising network. The conditional encoder is responsible for feature extraction and encoding of the input multimodal signal, mapping signals of different modes to a unified feature space. The diffusion processor simulates the gradual degradation process of the signal from a clear state to a noisy state, as well as the inverse process of recovering from a noisy state to a clear state. The denoising network is responsible for predicting and removing noise during the recovery process, gradually reconstructing a clear signal.

[0082] A conditional encoder is used to extract a unified feature representation of multimodal signals; its formula is:

[0083] ;

[0084] Where z is the joint embedding vector; X egg The EEG signal matrix is ​​T×C, where T is the time point and C is the number of channels; X fnirs Let fNIRS be the signal matrix T×D, where D is the number of detectors; θ is the encoder network parameter; and E represents the conditional decoder.

[0085] The formula for calculating the forward process (signal degradation) of the diffusion processor is:

[0086] ;

[0087] The formula for calculating the reverse process (signal recovery) is:

[0088] ;

[0089] Where, x t Let t be the noisy signal at step t (dimension: number of channels × signal). The cumulative noise scaling factor is derived from the hyperparameter sequence {β}. t The calculation yielded the result. where i is the index variable and t is the current total number of diffusion steps. To adjust parameters in the morning, Indicates the percentage of signals retained in a single step; θ represents the noise prediction value output by the denoising neural network; N(·) is a Gaussian distribution with the mean being the first parameter in parentheses and the covariance matrix being the second parameter; I is the identity matrix.

[0090] Denoising networks can predict the original clean signal based on conditional latent variables. The calculation formula is as follows:

[0091] ;

[0092] Where z is the joint embedding vector (cross-modal shared representation); For decoder network parameters; x t Let be the noisy signal at step t; D represents the denoising network; This indicates a clean signal indicating reconstruction.

[0093] The training process of the conditionally aligned temporal diffusion model adopts a supervised learning approach, using a large amount of data containing high-quality multimodal brain signals for training. Various common signal quality problems, such as motion artifacts, poor electrode contact, and environmental interference, are deliberately introduced into the training data, enabling the conditionally aligned temporal diffusion model to learn the ability to recover signals under various adverse conditions.

[0094] This generative AI signal enhancement and completion module has diverse and practical applications. For example, when a user's head movements cause a decrease in the quality of near-infrared signals, the module can utilize simultaneously acquired high-quality EEG signals to generate corresponding near-infrared signals. Conversely, when EEG signals are subject to electromagnetic interference, near-infrared signals can be used to enhance them. This cross-modal signal enhancement capability allows the system to maintain stable performance even in non-ideal experimental environments.

[0095] The fusion decoding layer is responsible for the deep fusion of features from brain signals and near-infrared signals. Instead of simply splicing or weighting features, this layer dynamically adjusts the importance of different modalities and features using learned attention weights. For example, when performing motor imagery tasks, the network may rely more on motor-related potentials (MLPs) from EEG signals; while when performing cognitive tasks, blood oxygenation information from near-infrared signals may be more important. This dynamic weight adjustment allows the network to adaptively select the most relevant information based on the specific task.

[0096] The fusion decoding layer includes a lightweight graph neural network module and an intent classifier.

[0097] The lightweight graph neural network module can perform deep fusion of features extracted from multimodal signals and output processed high-level feature maps; through an intent classifier, the high-level feature maps are mapped into specific, executable user intent commands.

[0098] The intent classifier is based not only on the current brain signal, but also on: temporal context, preceding instruction sequence; task context, current operating mode and application scenario; user context, individual habits and skill level.

[0099] The lightweight graph neural network module adopts the NexusNet architecture. The NexusNet network first constructs a brain functional connectivity graph. Nodes in the graph represent the locations of various electrodes or probes, and node features include the time-frequency and statistical characteristics of the multimodal signals acquired from that location. The edges in the brain functional connectivity graph include three types: spatial edges, established based on the physical distance between electrodes, reflecting the spatial structural relationships of the brain; functional edges, established based on the correlation between signals, reflecting the functional connections between different brain regions; and routing edges, established based on the shortest path for information transmission, simulating the information flow process within the brain.

[0100] The formula for calculating the functional edge is as follows:

[0101] ;in, The distance is Euclidean; i and j are node indices; Displacement distance attenuation coefficient;

[0102] The formula for calculating functional edges is:

[0103] ;

[0104] Among them, F ij Represents the functional connection weights; ρ(·) is the Pearson correlation coefficient; x i Let x be the signal time series of node i; j Let x be the signal time series of node j; T is the time window length; k is the time step index; xi，k Let x be the signal value of node i at time k; j，k μ is the signal value of node j at time k; i μ is the signal mean at node i; j Let be the signal mean of node j;

[0105] The formula for calculating the routing edge is:

[0106] ;

[0107] in, This means that if j is located in the set of shortest paths to node i.

[0108] Lightweight graph neural networks employ depthwise separable graph convolution techniques, which decompose traditional graph convolution operations into two independent steps: spatial convolution and channel convolution, significantly reducing the number of model parameters and computational complexity.

[0109] The formula for spatial convolution (aggregating neighbor information) is:

[0110]

[0111] The formula for calculating channel convolution (feature transformation) is:

[0112] ;

[0113] in, Let v be the embedding representation of node v at layer l+1; The function is a per-element nonlinear activation function; N(v) is the set of one-hop neighbors of node v; These are the parameters of the l-th layer spatial filter; Attention weights for routing; The original features of neighbor node u at layer l; DWConv is a depthwise separable convolution; Let v be the original feature of node v in layer L.

[0114] In addition, the NexusNet network includes a routed attention mechanism; this routed attention mechanism simulates the process of information being transmitted through specific pathways in the brain, considering not only directly connected nodes but also indirect connections transmitted through intermediate nodes when information is aggregated; this routed attention mechanism enables the network to capture remote functional connections in the brain network and to more accurately understand the brain's working patterns.

[0115] The formula for calculating dynamic focus on the critical path is:

[0116] ;

[0117] Where a is the learnable query vector; Wr is the route projection matrix; For feature splicing operations; Let be the routing projection matrix; hi, hj, and hk are the feature vectors of source node i, target node j, and target node k, respectively.

[0118] The lightweight design allows high-performance brain signal decoding algorithms to be deployed on resource-constrained platforms such as mobile phones and embedded devices, greatly expanding the application scenarios of brain-computer interface technology.

[0119] The closed-loop optimization cloud platform is a continuous learning and optimization system based on cloud computing. Through technologies such as incremental learning and personalized modeling, it enables brain-computer interface systems to continuously adapt to the usage habits of specific users and continuously improve overall performance. The closed-loop optimization cloud platform includes a personalized model incremental learning module and a user data and model optimization module.

[0120] The personalized model incremental learning module employs a privacy-preserving learning mechanism, allowing the system to continuously optimize the decoding model while protecting user privacy. The specific process is as follows: the cloud first distributes the current baseline model to user devices; the devices use local data to fine-tune the model, generating personalized model updates; then, the model updates (not the original data) are encrypted and uploaded to the cloud; the cloud analyzes model updates from multiple devices to identify common feature patterns; finally, the optimized model is redistributed to the devices.

[0121] This closed-loop optimization mechanism has multiple advantages: First, it protects user privacy because sensitive brain signal data is always stored on the user's local device; second, it can use the user's actual usage data to improve the model, making the system increasingly adaptable to the user's individual characteristics; and finally, through continuous learning and optimization, the model can continuously adapt to changes in the user and maintain optimal performance.

[0122] The user data and model optimization module is responsible for generating customized decoding models for each user. First, it constructs a detailed user profile based on the user's usage data, including brain signal feature patterns, usage habits, and task preferences. Then, based on this information, it personalizes the base model. The adaptation method employs a weighted fusion strategy, combining the base model with a user-specific model to balance generality and personalization. For new users, the system relies more on the base model to ensure basic performance. As usage time increases, the weight of the personalized model is gradually increased, making the system increasingly adaptable to the user's individual characteristics.

[0123] The control application layer translates the decoded neural intents into specific control commands and manages the user experience throughout the interaction process. The control application layer includes an external device control command module and a real-time visual / haptic feedback module.

[0124] The external device control command module supports three different types of control modes: discrete control mode, continuous control module, and hybrid control module. Discrete control mode is suitable for binary or multi-factor decision-making tasks such as switching and selection, triggered by detecting specific event-related potentials or motion visualization patterns. Continuous control mode is suitable for tasks requiring precise position control, such as cursor movement and robotic arm control, achieving smooth control by decoding the continuous characteristics of signals. Hybrid control mode combines the advantages of both discrete and continuous control, suitable for complex interactive tasks.

[0125] The real-time visual / tactile feedback module provides users with rich sensory feedback, forming a complete interactive loop.

[0126] Visual feedback displays the system's status recognition results and pending operations through a graphical interface, allowing users to understand the system's current status; tactile feedback provides gentle tactile cues through vibration motors, conveying information without interfering with visual attention; and auditory feedback uses sound signals to indicate important events, such as successful operations or error warnings.

[0127] Furthermore, this invention incorporates a user proficiency adaptive algorithm that enables the system to dynamically adjust control parameters based on the user's experience. For novice users, the system employs more conservative parameter settings, such as increasing the decision threshold, reducing control sensitivity, and introducing a confirmation mechanism to prevent accidental operation. As the user's proficiency increases, the system gradually improves control sensitivity and response speed, reduces unnecessary confirmation steps, and provides a more direct and smooth control experience. The system automatically assesses proficiency levels by continuously monitoring the user's operational accuracy and response speed, eliminating the need for manual settings by the user.

[0128] The proficiency assessment model defines a two-dimensional state vector s=[p,r]; where p is the recent task success rate (sliding window mean); and r is the reciprocal of the average response time (normalized to [0,1]). The proficiency level L∈{L0,L1,L2} is determined through fuzzy logic reasoning; the specific formula is as follows:

[0129] ;

[0130] in, , Membership scheduling function center parameter; final level selection is L with the largest membership degree. k L k s is the proficiency level identifier; s is the current system state feature vector; For performance indicators; For reliability indicators; This is the width parameter of the membership function;

[0131] The parameter adaptation rules are shown in the table below:

[0132]

[0133] In the table above, p represents the performance improvement; η represents the performance threshold; r represents the reliability decrease; ζ represents the reliability threshold; and Δτ and ΔK represent the step size adjustment in each iteration.

[0134] Furthermore, the interactive control device of the present invention is also equipped with a fault-tolerant control mechanism; when the decoding confidence is low, the device will automatically activate the confirmation process, requiring the user to confirm the operation intention through a specific EEG pattern; for critical operations, the system supports multi-level verification to prevent accidental triggering; the device also has a misoperation recovery function, providing an opportunity to undo or correct when a possible misoperation is detected; these safety mechanisms greatly improve the practicality of the device in real-world environments.

[0135] Specifically, when the decoding confidence Γ < Γmin, the misoperation protection is triggered; the protection operations include: Level 1 alarm, vibration feedback + flashing of the interface warning icon; Level 2 blocking, suspending command sending until the user actively cancels; and post-event remediation, providing a queue of the most recent N operations for cancellation.

[0136] See Figure 2 The implementation process for a single instruction execution in this invention is as follows:

[0137] The entire process of executing a single command begins with a motor imagery generated in the user's brain. When a user intends to control an external device, such as turning on a desk lamp, the motor cortex of the brain will generate a specific pattern of neural activity. This neural activity will simultaneously trigger two physiological responses: electrophysiological activity and changes in blood oxygen dynamics.

[0138] The multimodal sensors in the head-mounted device immediately begin to work; the EEG electrode array captures changes in the brain's electrical signals; at the same time, the fNIRS optical probe monitors changes in the brain's blood oxygen concentration and records the blood oxygen dynamics response; these raw multimodal signals are transmitted in real time to the edge processing device via a wireless transmission protocol.

[0139] During the edge processing stage, the system first preprocesses the received signals. The signal preprocessing module performs quality detection, evaluates the signal-to-noise ratio and integrity of the EEG and fNIRS signals, and then removes power frequency interference, electromyography artifacts and other environmental noise through digital filtering algorithms. It also performs spatiotemporal alignment processing to ensure that the millisecond-level time accuracy of the EEG can be accurately matched with the spatial positioning information of the fNIRS.

[0140] When certain signals are detected to be of poor quality, the Conditional Aligned Temporal Diffusion (CATD) model comes into play. This generative AI model can use high-quality modal signals to recover or enhance damaged signals. For example, when the fNIRS signal quality is degraded due to head movement, the system can use a clear EEG signal to generate a corresponding fNIRS signal, thereby ensuring the reliability of subsequent processing.

[0141] After signal enhancement, the system enters the real-time decoding stage. The lightweight graph neural network NexusNet receives the preprocessed multimodal features, first constructing a brain functional connectivity map, using several acquisition points as nodes, and establishing a complete brain network model through spatial edges, functional edges, and routing edges. After completing the inference calculation, the lightweight graph neural network outputs the probability distribution of the user's intention. For example, the system may recognize the intention of "left-hand motor imagery" with a 94% confidence level.

[0142] The decoding results are then sent to the control application layer, where the neural intent is mapped into specific control commands. If the recognition confidence level is higher than a preset threshold, the system directly sends the command to the smart home gateway. Upon receiving the command, the smart device immediately executes the on / off action, while simultaneously providing tactile feedback through the vibration motor of the head-mounted device and displaying visual confirmation information through the mobile application interface, completing the entire perception-action-feedback closed loop.

[0143] See Figure 3 The system evolution process of periodic optimization of the device of the present invention is as follows:

[0144] In addition to single instruction execution, the system also establishes a periodic optimization mechanism to ensure that the brain-computer interface system can continuously learn and improve; this optimization process is usually carried out on a 24-hour cycle and is executed automatically during the system's idle periods.

[0145] The first stage of the cycle optimization is data collection. Edge devices anonymize the interaction data generated in the past 24 hours, removing all information that could identify an individual, while retaining brain signal features, decoding results, and user feedback data valuable for model optimization. This anonymized data includes raw brain signal feature vectors, intent recognition results and confidence scores, user-provided correction records, and device execution status logs. The processed data is then uploaded to the cloud optimization platform via an encrypted channel.

[0146] After receiving anonymous data from multiple users, the cloud platform initiates a complex model optimization process. The first step is federated learning aggregation, where secure multi-party computation technology is used to securely aggregate model updates from different users, extracting collective wisdom while protecting individual privacy, and updating the global base model to improve generalization ability. Next is the personalized model training phase, where the system constructs an individual feature profile based on each user's historical data, analyzes unique brain signal patterns and operating habits, and generates a personalized decoding model for each user. To prevent the personalized model from forgetting old knowledge while learning new knowledge, the system also employs incremental learning technology, using an elastic weight consolidation mechanism to protect learned important parameters, and utilizing an experience replay buffer to store representative samples, ensuring that continuous model improvement does not come at the expense of existing performance.

[0147] After completing the personalized model optimization, the system enters the personalized model distribution phase. The cloud first compresses and optimizes the trained personalized decoding model, reducing its complexity through knowledge distillation technology and keeping the number of parameters within the limits of embedded devices, while ensuring that inference speed is not affected. The optimized personalized model is securely pushed to the user device via encrypted transmission, seamlessly updating the NexusNet model in the edge processing layer, and undergoing compatibility and performance verification before deployment.

[0148] The final stage is effect verification and adaptive adjustment; real-time monitoring of the new model's performance in actual use, including key indicators such as recognition accuracy and response speed, while collecting user feedback and satisfaction evaluations; if poor performance is found in certain specific scenarios, the system will make targeted parameter fine-tuning.

[0149] These two processes of the present invention together constitute a complete working loop of the brain-computer interface system: single instruction execution ensures the accuracy and timeliness of real-time interaction, meeting the user's basic need for immediate response; periodic optimization ensures the adaptability and progressiveness of the system for long-term use, continuously improving the user experience through continuous learning.

[0150] See Figure 4 A brain-computer interface interaction control method based on multimodal brain signal fusion includes the following steps:

[0151] Step 1: Multimodal signal acquisition and real-time preprocessing

[0152] The system initiates a multimodal signal synchronous acquisition process. EEG electrodes acquire brain signals, fNIRS optical probes monitor changes in blood oxygen concentration, and EMG sensors detect muscle activity. During the acquisition process, electrode impedance and signal quality are monitored in real time to ensure data reliability. After the raw signals are transmitted to the edge device via Bluetooth, they are immediately filtered, denoised, and time-domain aligned to eliminate power frequency interference and motion artifacts, providing high-quality standardized data for subsequent analysis.

[0153] Step 2: Generative AI Enhancement and Feature Extraction

[0154] Intelligent enhancement of preprocessed signals is achieved based on a conditionally aligned temporal diffusion model. When a degradation in the quality of a certain modality signal is detected, cross-modal reconstruction is performed using high-quality modal signals, and signal completion is achieved through a neurovascular coupling mechanism. The enhanced signal input feature extraction module constructs a brain functional connectivity map containing multiple nodes, establishing three types of connectivity relationships: spatial edges, functional edges, and routing edges, forming a complete brain network representation.

[0155] Step 3: Multimodal fusion and intent decoding

[0156] The lightweight graph neural network NexusNet performs deep processing on brain functional connectivity graphs, employing deep separable graph convolution and route attention mechanisms to dynamically fuse EEG temporal features with fNIRS spatial features. The intent classifier combines temporal context, task context, and user context to output the probability distribution and confidence score of multi-class intents, achieving accurate intent recognition.

[0157] Step 4: Adaptive Control and Interactive Feedback

[0158] Based on the intent recognition results and confidence levels, the system adaptively generates control commands; high-confidence results are directly mapped to device control commands, medium-confidence results trigger a confirmation process, and low-confidence results refuse execution; control parameters are dynamically adjusted according to user proficiency, with novices using conservative settings to prevent misoperation; experienced users enjoy a more direct and smooth control experience; execution results are promptly conveyed to the user through visual, tactile, and auditory multimodal feedback.

[0159] Step 5: Privacy-Preserving Data Collection and Upload

[0160] While protecting user privacy, the system anonymizes the collection of key data during the interaction process, including brain signal characteristics, recognition results, and user feedback; it uses differential privacy technology to process sensitive information to ensure individual privacy and security; and the processed data is regularly uploaded to the cloud optimization platform through an encrypted channel to provide a data foundation for model improvement.

[0161] Step 6: Cloud-based Federated Learning and Personalized Optimization

[0162] The cloud platform is based on a federated learning framework, which securely aggregates model updates from multiple users, extracts collective wisdom while protecting privacy, and improves the global basic model. At the same time, it builds feature profiles for individual users and generates personalized decoding models through a weighted fusion strategy, balancing generality and personalized needs. The incremental learning mechanism ensures that the model continues to improve while preventing knowledge forgetting.

[0163] Step 7: Model Distribution and Performance Monitoring

[0164] The optimized model undergoes compression and distillation to adapt to the resource constraints of embedded devices and is distributed to user terminals through a secure channel. The system monitors the performance of the new model in real-time, including key indicators such as recognition accuracy and response speed. Based on monitoring results and user feedback, the system parameters are dynamically adjusted to ensure continuous optimization. The fault tolerance mechanism automatically activates the security confirmation process when the confidence level is low to ensure system reliability.

[0165] This invention significantly improves decoding accuracy and intent recognition accuracy through multimodal brain signal fusion and generative AI enhancement technology; it adopts a lightweight graph neural network to improve real-time response speed; a personalized learning mechanism shortens user calibration time; it supports discrete, continuous, and hybrid control modes; and it has strong environmental adaptability, maintaining stable performance even under noise interference, providing a reliable brain-controlled interaction solution for multiple scenarios such as smart homes, medical rehabilitation, and industrial control.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A brain-computer interface interactive control device based on multimodal brain signal fusion, characterized in that, include: The sensing and acquisition layer is used to acquire raw data of multimodal brain signals, including EEG electrodes, functional near-infrared spectroscopy probes, and electromyography sensors. The edge processing layer connects to the sensing and acquisition layer and is used to preprocess and enhance the raw data, and output the processed raw data. The edge processing layer includes a signal preprocessing and alignment module that performs filtering, denoising and temporal alignment to eliminate power frequency interference and motion artifacts, and a generative AI signal enhancement and completion module. The generative AI signal enhancement and completion module is built based on a conditional alignment temporal diffusion model, which can restore and enhance the signal through cross-modal information transmission when the signal quality is damaged. The fusion decoding layer includes a lightweight graph neural network module and an intent classifier. The lightweight graph neural network module can perform deep fusion of features extracted from multimodal signals and output processed high-level feature maps. The intent classifier maps the high-level feature maps into specific, executable user intent commands. The control application layer converts the decoded intent commands into specific control commands to enable interactive connections with external devices; the fusion decoding layer includes an external device control command module and a real-time visual / haptic feedback module. The closed-loop optimization cloud platform is used to continuously optimize system performance and connects to the fusion decoding layer and control application layer, including a personalized model incremental learning module and a user data and model optimization module. The lightweight graph neural network module adopts the NexusNet architecture. The lightweight graph neural network of the NexusNet architecture first constructs a brain functional connectivity graph. The nodes in the brain functional connectivity graph represent the locations of various electrodes or probes. The node features include the time-frequency features and statistical features of the multimodal signals collected from that location. The edges of the brain functional connectivity graph include three types: spatial edges are established based on the physical distance between electrodes, reflecting the spatial structural relationships of the brain; functional edges are established based on the correlation between signals, reflecting the functional connections between different brain regions; and routing edges are established based on the shortest path of information transmission, simulating the information flow process inside the brain. The lightweight graph neural network employs depthwise separable graph convolution technology, which decomposes the traditional graph convolution operation into two independent steps: spatial convolution and channel convolution, significantly reducing the number of model parameters and computational complexity. The formula for spatial convolution is: The formula for calculating channel convolution is: ； in, Let v be the embedding representation of node v at layer l+1; The function is a per-element nonlinear activation function; N(v) is the set of one-hop neighbors of node v; These are the parameters of the l-th layer spatial filter; Attention weights for routing; The original features of neighbor node u at layer l; DWConv is a depthwise separable convolution; The original features of node v in the th layer; The lightweight graph neural network includes a routing attention mechanism; the routing attention mechanism enables the network to capture long-range functional connections in the brain network. The formula for calculating the dynamic focus on the critical path in the routing attention mechanism is: ； Where a is a learnable query vector; For feature splicing operations; h is the routing projection matrix; i h j h k Let i be the feature vectors of source node i, target node j, and target node k.

2. The brain-computer interface interactive control device based on multimodal brain signal fusion according to claim 1, characterized in that, The conditional alignment temporal diffusion model includes a conditional encoder, a diffusion processor, and a denoising network; the conditional encoder is responsible for feature extraction and encoding of the input multimodal signal, mapping signals of different modes to a unified feature space; The diffusion processor simulates the gradual degradation process of a signal from a clear state to a noisy state, and the reverse process of recovering from a noisy state to a clear state; The denoising network is responsible for predicting and removing noise during the recovery process, gradually reconstructing a clear signal.

3. The brain-computer interface interactive control device based on multimodal brain signal fusion according to claim 2, characterized in that, The calculation formula for the condition encoder is: ； Where z is the joint embedding vector; X egg The EEG signal matrix is ​​T×C, where T is the time point and C is the number of channels; X fnirs Let fNIRS be the signal matrix T×D, where D is the number of detectors; θ is the encoder network parameter; and E represents the conditional decoder. The formula for calculating the signal degradation process of the diffusion processor is: ; The calculation formula for the signal recovery process is as follows: ； Where, x t Let be the noisy signal at step t; The cumulative noise scaling factor is derived from the hyperparameter sequence {β}. t The calculation yielded the result. where i is the index variable and t is the current total number of diffusion steps. Adjust parameters for noise. Indicates the percentage of signals retained in a single step; The noise prediction value output by the denoising neural network, with parameters as follows: N(·) represents a Gaussian distribution, with the mean being the first parameter in parentheses and the covariance matrix being the second parameter; I is the identity matrix. Denoising networks can predict the original clean signal based on conditional latent variables. The calculation formula is as follows: ； Where z is the joint embedding vector; ϕ is the decoder network parameter; and D represents the denoising network. This indicates a clean signal indicating reconstruction.

4. The brain-computer interface interactive control device based on multimodal brain signal fusion according to claim 1, characterized in that, The formula for calculating the spatial edge is: ;in, The distance is Euclidean; i and j are node indices; Displacement distance attenuation coefficient; The formula for calculating functional edges is: ； Among them, F ij Represents the functional connection weights; ρ(·) is the Pearson correlation coefficient; x i Let x be the signal time series of node i; j Let be the signal time series of node j; T is the time window length; k is the time step index; Let i be the signal value of node i at time k; Let j be the signal value of node j at time k; Let be the signal mean of node i; Let be the signal mean of node j; The formula for calculating the routing edge is: ； in, This means that if j is located in the set of shortest paths to node i.

5. The brain-computer interface interactive control device based on multimodal brain signal fusion according to claim 1, characterized in that, It also features a user proficiency adaptive algorithm that can dynamically adjust control parameters based on the user's experience. For novice users, the system adopts more conservative parameter settings to prevent accidental operation. As the user's proficiency increases, the system gradually improves control sensitivity and response speed, reduces unnecessary confirmation steps, and provides a more direct and smooth control experience. The proficiency assessment model defines a two-dimensional state vector s=[p,r], where p is the recent task success rate and r is the reciprocal of the average response time. The proficiency level L∈{L0,L1,L2} is determined through fuzzy logic reasoning; the specific formula is as follows: ； in, , Membership scheduling function center parameter; final level selection is L with the largest membership degree. k L k s is the proficiency level identifier; s is the current system state feature vector; For performance indicators; For reliability indicators; This is the width parameter of the membership function.

6. The brain-computer interface interactive control device based on multimodal brain signal fusion according to claim 1, characterized in that, The personalized model incremental learning module employs a privacy protection mechanism, including: distributing the baseline personalized model to user devices via the cloud; generating personalized model updates using local data on the devices; uploading encrypted personalized model updates to the cloud; analyzing personalized model updates from multiple devices via the cloud to identify common feature patterns; and distributing the optimized personalized model to each device.

7. A brain-computer interface interactive control method based on multimodal brain signal fusion, characterized in that, Includes the following steps: Includes the following steps: Step 1: Multimodal signal acquisition and real-time preprocessing The system initiates a multimodal signal synchronous acquisition process, acquiring EEG signals, monitoring changes in blood oxygen concentration, and detecting muscle activity. During the acquisition process, electrode impedance and signal quality are monitored in real time to ensure data reliability. After the raw signal is transmitted to the edge device via Bluetooth, it is immediately filtered, denoised, and time-domain aligned to eliminate power frequency interference and motion artifacts, providing high-quality standardized data for subsequent analysis. Step 2: Generative AI Enhancement and Feature Extraction The preprocessed signal is intelligently enhanced based on the conditionally aligned temporal diffusion model; when a certain modality signal quality degradation is detected, cross-modal reconstruction is performed using high-quality modal signals, and signal completion is achieved through the neurovascular coupling mechanism; the enhanced signal is input into the feature extraction module to construct a brain functional connectivity map containing multiple nodes, establish three types of connectivity relationships: spatial edges, functional edges, and routing edges, and form a complete brain network representation; Step 3: Multimodal fusion and intent decoding The lightweight graph neural network NexusNet performs deep processing on brain functional connectivity graphs, employing deep separable graph convolution and route attention mechanisms to dynamically fuse EEG temporal features and fNIRS spatial features. The intent classifier combines temporal context, task context, and user context to output the probability distribution and confidence score of multi-class intents, achieving accurate intent recognition. The lightweight graph neural network first constructs a brain functional connectivity graph. Nodes in the brain functional connectivity graph represent the locations of various electrodes or probes, and node features include the time-frequency and statistical features of multimodal signals collected from that location. The edges of the brain functional connectivity graph include three types: spatial edges are established based on the physical distance between electrodes, reflecting the spatial structural relationships of the brain; functional edges are established based on the correlation between signals, reflecting the functional connections between different brain regions; and routing edges are established based on the shortest path for information transmission, simulating the information flow process within the brain. The lightweight graph neural network decomposes the traditional graph convolution operation into two independent steps: spatial convolution and channel convolution, which greatly reduces the number of model parameters and computational complexity. The formula for spatial convolution is: The formula for calculating channel convolution is: ； in, Let v be the embedding representation of node v at layer l+1; The function is a per-element nonlinear activation function; N(v) is the set of one-hop neighbors of node v; These are the parameters of the l-th layer spatial filter; Attention weights for routing; The original features of neighbor node u at layer l; DWConv is a depthwise separable convolution; The original features of node v at layer l; The lightweight graph neural network includes a routing attention mechanism; the routing attention mechanism enables the network to capture long-range functional connections in the brain network. The formula for calculating the dynamic focus on the critical path in the routing attention mechanism is: ； Where a is a learnable query vector; For feature splicing operations; h is the routing projection matrix; i h j h k Let be the feature vectors of source node i, target node j, and target node k; Step 4: Adaptive Control and Interactive Feedback Based on the intent recognition results and confidence levels, the system adaptively generates control commands; high-confidence results are directly mapped to device control commands, medium-confidence results trigger a confirmation process, and low-confidence results refuse execution; control parameters are dynamically adjusted according to user proficiency, with novices using conservative settings to prevent misoperation; experienced users enjoy a more direct and smooth control experience; execution results are promptly conveyed to the user through visual, tactile, and auditory multimodal feedback; Step 5: Privacy-Preserving Data Collection and Upload The system anonymizes the collection of key data during the interaction process while protecting user privacy; it uses differential privacy technology to process sensitive information to ensure individual privacy and security; and the processed data is regularly uploaded to the cloud optimization platform through an encrypted channel to provide a data foundation for model improvement. Step 6: Cloud-based Federated Learning and Personalized Optimization The cloud platform is based on a federated learning framework, which securely aggregates model updates from multiple users, extracts collective wisdom while protecting privacy, and improves the global basic model. At the same time, it builds feature profiles for individual users and generates personalized decoding models through a weighted fusion strategy, balancing generality and personalized needs. The incremental learning mechanism ensures that the model continues to improve while preventing knowledge forgetting. Step 7: Model Distribution and Performance Monitoring The optimized model undergoes compression and distillation to adapt to the resource constraints of embedded devices and is distributed to user terminals through a secure channel. The system monitors the performance of the new model in real-time. Based on monitoring results and user feedback, the system parameters are dynamically adjusted to ensure continuous optimization. The fault tolerance mechanism automatically activates the security confirmation process when the confidence level is low to ensure system reliability.

Images