A brain-computer interaction high-precision 3D intention positioning method based on task-aware bidirectional cross-modal attention
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HARBIN UNIV OF SCI & TECH
- Filing Date
- 2026-04-17
- Publication Date
- 2026-08-07
AI Technical Summary
现有的跨模态融合方法多采用启发式策略，存在以下缺陷：缺乏意图与空间的双向关联，仅支持从EEG到视觉的单向注意力；未考虑意图与坐标的几何映射特性；固定权重机制难以适应动态环境变化
Smart Images
Figure CN122526401A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of human-computer interaction technology, specifically to a three-dimensional intention localization method that integrates brain-computer interaction, motor imagery EEG, and vision, for achieving high-precision 3D spatial localization driven by intention. Background Technology
[0002] Motion imagery electroencephalography (MI-EEG) has core application value in target localization for people with disabilities and intention transmission in rehabilitation robots due to its non-invasiveness and real-time nature. However, MI-EEG suffers from low spatial resolution and poor signal-to-noise ratio due to volume conduction effects and noise interference, and can only provide directional intentions without outputting precise coordinates. In contrast, computer vision can acquire millimeter-level spatial coordinates through technologies such as RGB-D cameras, but lacks the ability to perceive user intentions and is susceptible to interference from occlusion or changes in lighting. Existing cross-modal fusion methods mostly adopt heuristic strategies, which have the following drawbacks: lack of bidirectional association between intention and space, only supporting unidirectional attention from EEG to vision; failure to consider the geometric mapping characteristics of intention and coordinates; and fixed weight mechanisms are difficult to adapt to dynamic environmental changes. Language-guided fusion models suffer from high latency due to semantic decoding, while attention networks based on feature fusion have insufficient anti-interference capabilities.
[0003] In summary, there is an urgent need to propose a novel cross-modal fusion localization method to achieve efficient bidirectional fusion of MI-EEG and visual spatial information, establish a direct mapping between neural intent and physical 3D coordinates, and adapt to modal reliability changes in dynamic scenes, thereby improving the accuracy, robustness, and real-time performance of high-precision 3D intent localization in brain-computer interaction. Summary of the Invention
[0004] The core of this invention is a high-precision 3D intent localization method for brain-computer interfaces based on task-aware bidirectional cross-modal attention, comprising the following steps:
[0005] Step 1: Construct a bimodal fusion framework of intent encoding and visual spatial representation. Simultaneously collect the user's motor imagination EEG signals and the visual spatial representation data of the target scene. After preprocessing the bimodal data, extract features and align them to 128 dimensions to directly establish a geometric mapping between neural motor intent and physical 3D coordinates.
[0006] Step 2: Construct a multi-head cross-attention task perception bidirectional cross-modal attention (TAB-CMA) fusion mechanism that uses visual spatial features as spatial prior constraints on intent features, uses intent features as feedback to calibrate visual space, and assigns fusion weights based on feature entropy confidence. After deep fusion of the two modal features, the input is given to the MLP regressor, and the output is the 3D spatial coordinates corresponding to the user intent.
[0007] Step 3: Employ a modal confidence calculation method based on feature entropy to evaluate the reliability of MI-EEG intent features and visual spatial features respectively. Calculate and allocate dual-modal fusion weights in real time according to the dynamic environment to achieve adaptive adjustment of the fusion process in complex scenarios.
[0008] Step 4: Construct an end-to-end closed-loop error feedback mechanism that uses the Euclidean distance between the predicted coordinates and the true coordinates as the error criterion. If the error exceeds the threshold, the mechanism will continuously correct the geometric mapping relationship between the neural intent and the spatial coordinates until the error is less than or equal to the preset threshold.
[0009] The beneficial effects of this invention are:
[0010] The aforementioned high-precision 3D intent localization method for brain-computer interaction based on task-aware bidirectional cross-modal attention achieves adaptive interaction between intent and spatial information through task-aware bidirectional cross-modal attention, effectively solving the problems of low localization accuracy and poor robustness in single-modal localization. On the other hand, it establishes a direct geometric mapping between neural intent and 3D coordinates, eliminating the need for semantic decoding. Dynamic weight allocation can adapt to occlusion and noise interference, and closed-loop feedback continuously optimizes localization stability, significantly improving the real-time performance and reliability of brain-computer interaction, and providing an efficient and stable solution for high-precision control of intent-driven robots. Attached Figure Description
[0011] Figure 1 This is a schematic diagram of the technical route;
[0012] Figure 2 This is a schematic diagram of the TAB-CMA fusion mechanism;
[0013] Figure 3 Here is a flowchart of the positioning strategy;
[0014] Figure 4 This is a schematic diagram of the operating space. Detailed Implementation
[0015] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.
[0016] like Figure 1 As shown, one embodiment of the present invention is a high-precision 3D intent localization method for brain-computer interaction based on task-aware bidirectional cross-modal attention, which includes the following steps:
[0017] Step 1: Construct a bimodal fusion framework of intent encoding and visual spatial representation. Simultaneously collect the user's motor imagery EEG signals and the visual spatial representation data of the target scene. After preprocessing the bimodal data, extract features and align them to 128 dimensions to directly establish a geometric mapping between neural motor intent and physical 3D coordinates. The specific implementation method is as follows:
[0018] A 64-channel EEG acquisition system was used, with electrodes arranged according to the international 10-20 system. C3, C2, and C4 were selected as key acquisition electrodes, and the sampling frequency was set to 500Hz. The electrode impedance was controlled to be lower than 100Hz. The system collects MI-EEG signals from four types of motion imagination paradigms: left hand, right hand, both feet, and tongue. These four neural patterns are mapped to the motion intentions in the four directions of left, right, back, and front in physical space, respectively. Simultaneously, an RGB-D camera is used to collect target scene data, and the scene's RGB images and depth information are acquired in real time. A relative coordinate system is constructed with the robotic arm base as the global origin to complete the benchmark positioning of the visual spatial data.
[0019] The MI-EEG signal was separated using a 7-30Hz zero-phase bandpass filter. (8-13Hz) and The (14-30Hz) rhythm was used to remove electrooculography and electromyography artifacts through independent component analysis (ICA) and then Epoch segmentation was performed to obtain pure and effective EEG signals. The visual spatial data was used to perform target detection and instance segmentation through the YOLOv8-Seg network. The coordinates of the human torso center were obtained by relying on the MediaPipe Holistic algorithm and the normalized relative spatial representation was calculated. Min-Max normalization was performed on the target coordinates. Combined with depth variance analysis, occluded targets were identified and their reliability was marked, and invalid visual data was removed.
[0020] Intent feature extraction models the preprocessed MI-EEG signals of each channel as an 8th-order autoregressive (AR) process, estimates the AR coefficients using the Yule-Walker equation, and calculates them using the Welch method. The rhythmic power spectral density (PSD) was calculated, and the Pearson correlation coefficients between the key electrodes C3, C4, and Cz were obtained. The variance and entropy features of the AR coefficients were extracted, and after normalization, they were concatenated to generate a 128-dimensional intention feature vector. Spatial feature extraction extracts the initial 3D coordinates of the target from RGB-D depth data and normalizes them to... The intervals are embedded into a high-dimensional manifold using sine and cosine encoding. A lightweight MobileNetV3 CNN is used to extract 64-dimensional image feature vectors. These vectors are then concatenated with the 64-dimensional coordinate embedding feature vectors and mapped to 128-dimensional spatial feature vectors via a 1×1 convolutional layer. This completes the 128-dimensional alignment of the bimodal features. The formula for the sine-cosine encoding is:
[0021] (1)
[0022] (2)
[0023] in, These are the normalized coordinate values. For embedding dimension indexes, For the embedded dimension.
[0024] Based on the aligned 128-dimensional intention features and spatial features, this method abandons the traditional dependence on target semantic labels and directly constructs an end-to-end geometric mapping between neuromotor intention and physical 3D coordinates. It associates the directional intention encoded by EEG signals with the spatial coordinates represented by visual signals, providing a basic mapping relationship for subsequent cross-modal fusion and 3D localization.
[0025] Step 2: Construct a TAB-CMA fusion mechanism that uses visual spatial features as spatial prior constraints on intent features, uses intent features as feedback to calibrate the visual space, and assigns fusion weights based on feature entropy confidence. Figure 2 As shown. The dual-modal features are then deeply fused and input into an MLP regressor, which outputs the 3D spatial coordinates corresponding to the user's intent. The specific implementation is as follows:
[0026] An 8-head parallel multi-head cross-attention architecture is used to construct the fusion core, with MI-EEG intent features as the query matrix. Visual spatial features are key matrices AND-value matrix The formula for calculating the multi-head cross-attention is:
[0027] (3)
[0028] in, The query matrix is a mapping of MI-EEG intent feature vectors. , These are the key matrix and value matrix, respectively, of the visual space feature vector mapping. The scaling factor for the feature dimension is used to avoid Gradient explosion occurs during computation.
[0029] The TAB-CMA fusion mechanism includes layer normalization, a feedforward network, and residual connections. The hidden layer of the feedforward network has 256 dimensions and uses GELU activation. The multi-head cross-attention output and the original intent features are processed sequentially through two residual connections, layer normalization, and feedforward network steps to obtain the fused feature vector. .
[0030] A three-layer fully connected MLP regressor was employed, with 128-dimensional fused features. For input, the hidden layer has 256 neurons and uses GELU activation; the output layer regresses to 3D spatial coordinates. Using the AdamW optimizer, initial learning rate The learning rate is adjusted using cosine annealing, and training is stopped if the loss does not improve for 15 consecutive rounds.
[0031] Step 3: Employ a modal confidence calculation method based on feature entropy to evaluate the reliability of MI-EEG intent features and visual spatial features respectively. Calculate and allocate dual-modal fusion weights in real time according to the dynamic environment to achieve adaptive adjustment of the fusion process in complex scenarios. The specific implementation method is as follows:
[0032] MI-EEG Intent Feature Vector and visual space feature vectors The elements are normalized, and the feature entropy of each mode is calculated through the probability distribution of the normalized elements. The lower the feature entropy, the higher the mode confidence.
[0033] The dynamic allocation process of modal weights is as follows: based on the confidence level of each mode and the prediction error of a single mode. , Combined with learning rate The MI-EEG modal weights were calculated. and visual modality weights ,and When vision is obstructed, Adjust to 0.7-0.8. When noise is present in MI-EEG, Adjust to 0.7-0.8;
[0034] The update formula for the modal weights is:
[0035] (4)
[0036] (5)
[0037] in, The MI-EEG modal weights are before the update.
[0038] Step 4: Construct an end-to-end closed-loop error feedback mechanism that uses the Euclidean distance between predicted and true coordinates as the error criterion. If the error exceeds a threshold, an online calibration of the fusion weight and feature mapping relationship is performed to continuously correct the geometric mapping relationship between neural intent and spatial coordinates until the error is less than or equal to a preset threshold. The specific implementation method is as follows:
[0039] The formula for calculating the Euclidean distance error between the predicted 3D coordinates and the true coordinates is:
[0040] (6)
[0041] in, To predict 3D coordinates, For true 3D coordinates; when ( When the preset error threshold is reached, closed-loop feedback is triggered, and the modal weights are recalculated based on the single-mode prediction error, and the weight allocation parameters of the TAB-CMA fusion mechanism are updated.
[0042] The closed-loop feedback adjustment step is executed in conjunction with the TAB-CMA fusion step; when the positioning error exceeds a preset threshold, the system automatically triggers a weight update operation, recalculating and allocating the fusion weights of the MI-EEG modality and the visual modality based on the single-modal prediction error, thereby achieving dynamic adaptive control of the positioning process. Figure 3 As shown.
[0043] The 3D intent localization and robot control involved in this embodiment are carried out within a preset operating space, as shown in Figure 4. This operating space constructs a relative coordinate system with the robotic arm base as the global origin, providing a standardized spatial reference for brain-computer interaction 3D localization and ensuring accurate matching between the localization results and the physical space.
Claims
1. A high-precision 3D intent localization method for brain-computer interfaces based on task-aware bidirectional cross-modal attention, characterized in that, Includes the following steps: Step 1: Construct a bimodal fusion framework of intent encoding and visual spatial representation. Simultaneously collect the user's motor imagery EEG (MI-EEG) signal and the visual spatial representation data of the target scene. After preprocessing the bimodal data, extract features and align them to 128 dimensions to directly establish a geometric mapping between neural motor intent and physical 3D coordinates. Step 2: Construct a multi-head cross-attention task perception bidirectional cross-modal attention (TAB-CMA) fusion mechanism that uses visual spatial features as spatial prior constraints on intent features, uses intent features as feedback to calibrate visual space, and assigns fusion weights based on feature entropy confidence. After deep fusion of the two modal features, the input is given to the MLP regressor, and the output is the 3D spatial coordinates corresponding to the user intent. Step 3: Employ a modal confidence calculation method based on feature entropy to evaluate the reliability of MI-EEG intent features and visual spatial features respectively. Calculate and allocate dual-modal fusion weights in real time according to the dynamic environment to achieve adaptive adjustment of the fusion process in complex scenarios. Step 4: Construct an end-to-end closed-loop error feedback mechanism that uses the Euclidean distance between the predicted coordinates and the true coordinates as the error criterion. If the error exceeds the threshold, the mechanism will continuously correct the geometric mapping relationship between the neural intent and the spatial coordinates until the error is less than or equal to the preset threshold.
2. The positioning method according to claim 1, characterized in that, In step 1, the MI-EEG signal is acquired using a 64-channel EEG acquisition system. The electrode arrangement follows the international 10-20 system, with key acquisition electrodes C3, C2, and C4. The sampling frequency is set to 500Hz, and the electrode impedance is maintained at [insert value here]. The visual spatial data described below is acquired by an RGB-D camera, simultaneously obtaining RGB images and depth data of the target scene, and constructing a relative coordinate system with the robotic arm base as the global origin.
3. The positioning method according to claim 1, characterized in that, In step 1, the preprocessing of the dual-mode data includes: separating the MI-EEG signal using a 7-30Hz zero-phase bandpass filter. Rhythm 8-13Hz and The rhythm was 14-30Hz. Independent component analysis (ICA) was used to decompose and remove artifacts such as electrooculography (EOG) and electromyography (EMG). The artifact-free signal was then segmented by epoch to obtain a clean and effective EEG signal. For visual spatial data, the YOLOv8-Seg network was used to perform target detection and segmentation. The MediaPipe Holistic estimation algorithm was used to obtain the coordinates of the human torso center and calculate the normalized relative spatial representation. The coordinates of the detected targets were normalized by Min-Max. Depth variance analysis was used to identify occluded targets and label their reliability.
4. The positioning method according to claim 1, characterized in that, In step 1, the specific process of motion intent encoding for the MI-EEG signal is as follows: the preprocessed MI-EEG signal of each channel is modeled as an 8th-order autoregressive (AR) process, and the AR coefficients are estimated using the Yule-Walker equation; the Welch method is used to calculate... Rhythm / The power spectral density (PSD) of the rhythm is calculated, and the Pearson correlation coefficients among key electrodes C3, C4, and Cz are calculated. The variance and entropy of the AR coefficients are extracted. The PSD features, electrode correlation coefficient features, and AR coefficient-derived features are normalized and then concatenated to generate a 128-dimensional intent feature vector. .
5. The positioning method according to claim 1, characterized in that, In step 1, the specific process of encoding the spatial location representation of the visual spatial data is as follows: extract the initial 3D coordinates of the target from the depth data acquired by the RGB-D camera. Normalize it to the boundary of the robotic arm's workspace. The normalized 3D coordinates are embedded into a high-dimensional manifold using sine and cosine encoding within the region. A lightweight MobileNetV3 convolutional neural network is used to extract 2D image features from the target region, generating a 64-dimensional image feature vector. This 64-dimensional image feature vector is then concatenated with the 64-dimensional coordinate embedding feature vector and mapped through a 1×1 convolutional layer to generate a 128-dimensional spatial feature vector. ; The formula for the sine and cosine encoding is: in, These are the normalized coordinate values. For embedding dimension indexes, For the embedded dimension.
6. The positioning method according to claim 1, characterized in that, In step 2, the TAB-CMA fusion mechanism adopts an 8-head parallel multi-head cross-attention structure, which projects the 128-dimensional MI-EEG intent feature vector and visual space feature vector to a 16-dimensional subspace through a linear transformation matrix to capture the multi-modal association between intent and space. The formula for calculating the multi-head cross-attention is: in, The query matrix is a mapping of MI-EEG intent feature vectors. , These are the key matrix and value matrix, respectively, of the visual space feature vector mapping. The scaling factor for the feature dimension is used to avoid Gradient explosion occurs during computation.
7. The positioning method according to claim 1, characterized in that, In step 2, the basic unit of the TAB-CMA fusion mechanism, the TAB-CMA block, includes a layer normalized network (LN), a feedforward network (FFN), and residual connections. The hidden layer dimension of the FFN is set to 256, and the Gaussian error linear unit (GELU) is used as the activation function. The feature fusion process of the TAB-CMA block is as follows: the output of the multi-head cross-attention is first residually connected to the original MI-EEG intent feature vector, followed by LN and FFN processing, and then a second residually connected and LN processing is performed to obtain the fused feature vector. .
8. The positioning method according to claim 1, characterized in that, In step 2, the MLP regressor includes three fully connected layers, and the input layer receives a 128-dimensional fused feature vector. The hidden layer has 256 neurons and uses the GELU activation function. The output layer regresses the 3D spatial coordinates of the target using 3 linear units. ; The parameters of the MLP regressor are updated using the AdamW optimizer, with an initial learning rate set to... The training process is dynamically adjusted by combining a cosine annealing strategy with an early stop mechanism: if the verification loss does not improve for 15 consecutive rounds, the training is terminated.
9. The positioning method according to claim 1, characterized in that, In step 3, the confidence calculation method based on feature entropy is as follows: For the MI-EEG intention feature vector... and visual space feature vectors The elements are normalized, and the feature entropy of each mode is calculated through the probability distribution of the normalized elements. The lower the feature entropy, the higher the mode confidence. The dynamic allocation process of modal weights is as follows: based on the confidence level of each mode and the prediction error of a single mode. , Combined with learning rate The MI-EEG modal weights were calculated. and visual modality weights ,and ; When visual obstruction occurs, Adjust to 0.7-0.
8. When noise is present in MI-EEG, Adjust to 0.7-0.8; The update formula for the modal weights is: in, The MI-EEG modal weights are before the update.
10. The positioning method according to claim 1, characterized in that, In step 4, the formula for calculating the Euclidean distance error between the predicted 3D coordinates and the true coordinates is: in, To predict 3D coordinates, For true 3D coordinates; when ( When the preset error threshold is reached, closed-loop feedback is triggered, and the modal weights are recalculated based on the single-mode prediction error, and the weight allocation parameters of the TAB-CMA fusion mechanism are updated.
11. The positioning method according to claim 1, characterized in that, The TAB-CMA fusion step employs residual connection and layer normalization operations. Residual connections preserve task-related intent and spatial features, preventing information loss during feature fusion. Layer normalization normalizes the feature vectors, ensuring the stability of feature distribution and improving the model's convergence speed.
12. The positioning method according to claim 1, characterized in that, The closed-loop feedback adjustment step is executed in conjunction with the TAB-CMA fusion step; when the positioning error exceeds the preset threshold, the weight update operation is automatically triggered, and the weights of MI-EEG mode and visual mode are recalculated and allocated according to the single-modal prediction error, so as to realize the dynamic adaptive adjustment of the positioning process.