Brain-computer interface multi-modal signal fusion method

Abstract

This invention provides a multimodal signal fusion method for brain-computer interfaces, belonging to the field of data processing technology. Specifically, it includes: acquiring electroencephalogram (EEG) signals, functional magnetic resonance imaging (fMRI) signals, and blood test data; performing time alignment based on a unified time reference; performing modality-specific preprocessing and quality control; constructing a five-level brain topology hierarchy based on brain physiological priors to achieve spatial-physiological alignment of electrode space, brain region space, and biochemical indicators; extracting multi-scale features of each modality based on the five-level brain topology hierarchy; projecting the multi-scale features onto a unified semantic space and achieving semantic alignment through cross-modal comparative learning based on physiological perception; constructing a physiologically guided cross-modal hierarchical fusion network to fuse features of each modality layer by layer from coarse to fine scales, and performing global aggregation based on physiological weights to generate multimodal fusion features. This invention improves fusion efficiency, accuracy, and adaptability.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for multimodal signal fusion in brain-computer interfaces. Background Technology

[0002] Currently, a comprehensive characterization of the brain's physiological state and functional characteristics relies on the collaborative analysis of multi-dimensional medical information, including brain electrical activity, brain region structure and function, and systemic biochemical metabolism. Among these, electroencephalography (EEG), with its advantages of being non-invasive, having high temporal resolution, being low-cost, and allowing for real-time bedside monitoring, has become the core carrier for brain-computer interface (BCI) technology to acquire dynamic brain activity information. Functional magnetic resonance imaging (fMRI) can provide millimeter-level high spatial resolution information on brain region blood oxygen metabolism and functional connectivity, accurately reflecting brain region activation patterns and structural association characteristics. Blood test data, containing peripheral physiological information such as inflammatory factors, biochemical metabolic indicators, and protein biomarkers, serves as routine medical evidence reflecting the overall physiological state and metabolic level. These three technologies are highly complementary in terms of information dimensions. Achieving effective fusion of multimodal signals through BCI technology is a core direction for overcoming the limitations of single-modal information representation and constructing a comprehensive and complete medical information system. However, existing brain-computer interface multimodal signal fusion methods suffer from strong modal heterogeneity, low spatiotemporal alignment accuracy, poor fusion efficiency, lack of physiological correlation modeling, and weak generalization ability for small samples in EEG, fMRI, and blood test data.

[0003] It is evident that there is an urgent need for a brain-computer interface multimodal signal fusion method that integrates high efficiency, accuracy, and adaptability. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a brain-computer interface multimodal signal fusion method, which at least partially solves the problems of poor fusion efficiency, accuracy and adaptability in the prior art.

[0005] In a first aspect, embodiments of the present invention provide a brain-computer interface multimodal signal fusion method, comprising: Step 1: Acquire the EEG signals, functional magnetic resonance imaging (fMRI) signals, and blood test data of the target object, and based on a unified global time reference, accurately align the EEG signals, fMRI signals, and blood test data in the time dimension to obtain time-aligned multimodal data. Step 2: Modality-specific standardized preprocessing is performed on the EEG signals, functional magnetic resonance imaging signals, and blood test data in the time-aligned multimodal data, and a multi-dimensional data quality scoring system is constructed to assess and control the quality of each modality of data. Step 3: Based on the brain's anatomical structure and physiological functional regions, a five-level brain topology hierarchy is constructed, and a mapping relationship and physiological weight system between each level are established. Based on this, the electrode space of EEG signals, the brain region space of functional magnetic resonance imaging signals, and the biochemical indicators of blood test data are aligned in the spatial-physiological dimension. Step 4: Based on the five-level brain topology hierarchy, deep feature extraction is performed on the preprocessed EEG signals, functional magnetic resonance imaging signals, and blood test data to obtain EEG multi-scale features, functional magnetic resonance imaging multi-scale features, and blood test multi-scale features that are aligned with the five-level brain topology hierarchy. Step 5: Project the multi-scale features of EEG, multi-scale features of functional magnetic resonance imaging, and multi-scale features of blood tests into a unified latent semantic space. Through cross-modal comparative learning of physiological perception, minimize the distance between features of different modalities under the same physiological state, while maximizing the distance between features under different physiological states, to achieve semantic alignment of cross-modal features. Step 6: Construct a physiologically guided cross-modal hierarchical fusion network. Based on a five-level brain topology hierarchy, cross-modal collaborative fusion of semantically aligned EEG features, functional magnetic resonance imaging features, and blood test features is performed layer by layer from coarse to fine scale to obtain multi-scale fusion features. The multi-scale fusion features are then globally aggregated based on the physiological weights of each scale to generate the final multi-modal fusion features.

[0006] The brain-computer interface multimodal signal fusion scheme in this embodiment of the invention includes: Step 1.1: Using the hardware clock of the acquisition system as the global unified time reference, mark each sampling point of the EEG signal, each time frame of the functional magnetic resonance imaging signal, and the blood collection time point of the blood test data with a globally unique timestamp. Step 1.2: Based on the timestamp, the low temporal resolution sequence of the functional magnetic resonance imaging (fMRI) signal is upsampled to the same time step as the EEG signal using linear interpolation. The expression for the linear interpolation method is:

[0007] in , The timestamps are the timestamps of two adjacent time frames of a functional magnetic resonance imaging (fMRI) signal. The time point corresponding to the EEG signal. This indicates the blood oxygenation level dependent on the signal value in fMRI; Step 1.3: Using the blood collection time point of the blood test data as the anchor point, the corresponding blood test feature vector is used as the global physiological feature and broadcast on the entire time series of EEG signals and functional magnetic resonance imaging signals to realize the temporal dimension correlation between biochemical indicators and brain signal temporal features.

[0008] According to a specific implementation of an embodiment of the present invention, step 2 specifically includes: Step 2.1: Modality-specific standardization preprocessing is performed on the EEG signals, functional magnetic resonance imaging signals, and blood test data in the time-aligned multimodal data. Step 2.2: Calculate the quality scores for the electroencephalogram (EEG) signal, functional magnetic resonance imaging (fMRI) signal, and blood test data, respectively. The expression for the quality score is as follows:

[0009] in, For modality Quality rating This is the normalized signal-to-noise ratio value. The percentage of artifacts / outliers. For data missing rate, These are the weighting coefficients; Step 2.3: Based on the calculated quality score, classify the multimodal data, remove invalid samples with a comprehensive quality score lower than a preset threshold, and set a modality missing label for samples with substandard single-modal quality.

[0010] According to a specific implementation of an embodiment of the present invention, step 4 specifically includes: Step 4.1: Extract multi-scale temporal topological features of EEG signals at each scale of the five-level brain topological hierarchy using a multi-scale temporal convolutional network, where the first... The formula for extracting EEG features at different scales is:

[0011] in, For the first EEG channel grouping characteristics at different scales For spatial regional attention mechanisms, This represents the physiological weight at this scale. The kernel weights at this scale, For the first Scale-based EEG features To unify feature dimensions; Step 4.2: Using a multi-scale graph convolutional network with a five-level brain topology as the graph structure, extract the multi-scale spatial connectivity features of functional magnetic resonance imaging (fMRI) signals at each scale. The formula for updating the features of functional magnetic resonance imaging at different scales is:

[0012] in, For the first The scale of the subgraph adjacency matrix. This is the corresponding degree matrix. , For the first Layer Scale-based graph convolution weights and biases It is the ReLU activation function. For the first fMRI features at scale; Step 4.3: Using a gated attention multilayer perceptron and combining it with a physiological correlation matrix, extract multiscale physiological and biochemical features of blood test data at various scales, where the first... The formula for extracting blood test features based on scale is:

[0013]

[0014] in, For the first A blood test indicator, For this indicator in the 1st Attention weights based on scale For physiological correlation strength, For learnable parameters, For the first Blood test characteristics of scale.

[0015] According to a specific implementation of an embodiment of the present invention, step 5 specifically includes: Step 5.1: Design a feature projection head with a shared structure for EEG signals, functional magnetic resonance imaging signals and blood test data respectively, and project the multi-scale features of each modality onto a unified latent semantic space to obtain the projected EEG features, functional magnetic resonance imaging features and blood test features. Step 5.2: Using each scale of the five-level brain topology hierarchy as a unit, construct a contrastive loss function for physiological perception. Different modal projection features at the same time step and under the same physiological state are used as positive sample pairs, and projection features at different physiological states or different time steps are used as negative sample pairs. Minimize the distance between positive sample pairs while maximizing the distance between negative sample pairs. The contrastive loss function for physiological perception includes the... Scale contrast loss and total alignment loss :

[0016]

[0017] in, For the sample size, Let be the modal combination number. The cosine similarity function is used. For temperature coefficient, For the first The first sample Modal first Projection characteristics of scale; Step 5.3: Based on the contrastive loss function, optimize the parameters of the feature projection head and the feature extraction network of each modality until the projected features of different modalities achieve semantic alignment at the physiological state level in the latent semantic space.

[0018] According to a specific implementation of an embodiment of the present invention, the step of performing cross-modal collaborative fusion of semantically aligned EEG features, functional magnetic resonance imaging features, and blood test features layer by layer from coarse to fine scale includes: For the current scale Through a cross-modal collaborative attention mechanism, it adaptively learns the complementary correlations among EEG features, functional magnetic resonance imaging features, and blood test features, and combines them with physiological weights at the current scale. This generates intra-layer fusion features at this scale, where the formula for calculating collaborative attention features is:

[0019]

[0020]

[0021] in, The projection matrix is ​​learnable. For modality For modes Cross-modal attention features This represents the collaborative attention fusion feature at this scale; The previous coarse-scale fusion features Through hierarchical mapping matrix After dimensional transformation, the intra-layer fusion features are combined with the current fine-scale features. Residual connections are performed to transfer coarse-scale global physiological information to fine-scale local features. The fusion formula is as follows:

[0022] in, This is a bias term.

[0023] According to a specific implementation of an embodiment of the present invention, a pre-training step is further included: Construct a pre-trained dataset containing unlabeled or weakly labeled data with multiple centers and multiple modalities; The design incorporates a multi-task pre-training objective that includes cross-modal alignment loss, modal reconstruction loss, and mask modeling loss. The total pre-training loss function is:

[0024] in, For cross-modal alignment loss, For modal reconstruction loss, For mask modeling loss, , , These are the weighting coefficients; Based on the pre-trained dataset, the total pre-training loss function is minimized, and the feature extraction network, cross-modal alignment module, and cross-modal hierarchical fusion network involved in steps 4 to 6 are jointly pre-trained to learn a general representation of brain physiological states.

[0025] The brain-computer interface multimodal signal fusion scheme in this embodiment of the invention includes: Step 1, acquiring the target object's EEG signals, functional magnetic resonance imaging (fMRI) signals, and blood test data, and based on a unified global time reference, precisely aligning the EEG signals, fMRI signals, and blood test data in the time dimension to obtain time-aligned multimodal data; Step 2, performing modality-specific standardized preprocessing on the EEG signals, fMRI signals, and blood test data in the time-aligned multimodal data, and constructing a multidimensional data quality scoring system to assess and control the quality of each modality of data; Step 3, constructing a five-level brain topology hierarchy based on brain anatomical structure and physiological functional regions, and establishing mapping relationships and physiological weight systems between each level, thereby aligning the electrode space of the EEG signals, the brain region space of the fMRI signals, and the biochemical indicators of the blood test data in the spatial-physiological dimension; Step 4, based on the five-level brain topology hierarchy, performing preprocessing on the EEG signals, fMRI signals, and blood test data in the time dimension, and aligning the time dimension of the target object's EEG signals, fMRI signals, and blood test data in the time dimension; Deep feature extraction is performed on the processed EEG signals, functional magnetic resonance imaging (fMRI) signals, and blood test data to obtain multi-scale features of EEG, fMRI, and blood tests that are aligned with the five-level brain topology. Step 5: The multi-scale features of EEG, fMRI, and blood tests are projected into a unified latent semantic space. Through cross-modal comparative learning of physiological perception, the distance between features of different modalities under the same physiological state is minimized, while the distance between features under different physiological states is maximized, to achieve semantic alignment of cross-modal features. Step 6: A physiologically guided cross-modal hierarchical fusion network is constructed. Based on the five-level brain topology, the semantically aligned EEG features, fMRI features, and blood test features are fused layer by layer from coarse to fine scale to obtain multi-scale fused features. The multi-scale fused features are then globally aggregated based on the physiological weights of each scale to generate the final multi-modal fused features.

[0026] The beneficial effects of the embodiments of the present invention are as follows: Through the solution of the present invention, a five-level brain topological hierarchy is constructed based on the prior knowledge of brain physiology, realizing the spatial-physiological dual precise alignment of EEG, fMRI, and blood test data. Then, through modality-specific feature extraction, cross-modal semantic space unification, and physiologically guided hierarchical fusion, the deep coupling of temporal-spatial-biochemical information is completed. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating a brain-computer interface multimodal signal fusion method provided in an embodiment of the present invention. Detailed Implementation

[0029] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0030] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0031] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this invention, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0032] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0033] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0034] This invention provides a brain-computer interface multimodal signal fusion method, which can be applied to medical information data processing in medical scenarios.

[0035] See Figure 1 This is a flowchart illustrating a brain-computer interface multimodal signal fusion method provided in an embodiment of the present invention. Figure 1 As shown, the method mainly includes the following steps: Step 1: Acquire the EEG signals, functional magnetic resonance imaging (fMRI) signals, and blood test data of the target object, and based on a unified global time reference, accurately align the EEG signals, fMRI signals, and blood test data in the time dimension to obtain time-aligned multimodal data. In practice, the brain-computer interface multimodal acquisition system is used to simultaneously acquire the target object's EEG signals, fMRI signals, and blood test data, establish a unified time reference system, and achieve precise alignment of the time dimensions of the three modal signals, thus eliminating the information misalignment problem caused by time sequence offset from the source.

[0036] Multimodal signal synchronous acquisition: The acquisition system adopts a unified hardware triggering and timestamp mechanism. EEG signals are acquired through a 64-channel non-invasive EEG electrode cap, with a sampling frequency set to 1000Hz; fMRI signals are acquired through a 3.0T MRI scanner, using gradient echo planar imaging (EPI) sequences, with a repetition time of [not specified]. echo time Layer thickness Interlayer spacing Blood test data collection was completed simultaneously with EEG and fMRI scans. Fasting venous blood samples were collected, and biochemical indicators, inflammatory factors, and protein biomarkers were detected. The precise blood collection time points were recorded.

[0037] Global time reference establishment: The hardware clock of the acquisition system is used as the global unified time reference. Each sampling point of the EEG signal, each time frame of the fMRI signal, and the blood collection time point of the blood test are marked with a globally unique timestamp. The timestamp accuracy reaches the millisecond level, eliminating the clock offset of different acquisition devices.

[0038] Precise cross-modal timing alignment: For EEG and fMRI signals, based on global timestamps, a linear interpolation method is used to upsample the low temporal resolution BOLD signal sequence of fMRI to the same time step as the EEG signal. The interpolation formula is as follows:

[0039] in, , The timestamps of two adjacent time frames in fMRI. For the time point corresponding to EEG, This represents the interpolated fMRI signal value. This method achieves complete time step synchronization between EEG and fMRI signals, controlling the timing synchronization error within a specified range. Within.

[0040] For blood test data, the global timestamp corresponding to the blood collection time point is used as the anchor point, and the blood test feature vector is used as the global physiological feature. It is broadcast on the full time series of EEG and fMRI to realize the temporal dimension correlation between biochemical indicators and brain signal temporal features, and complete the unification of the time reference of the three modalities.

[0041] Step 2: Modality-specific standardized preprocessing is performed on the EEG signals, functional magnetic resonance imaging signals, and blood test data in the time-aligned multimodal data, and a multi-dimensional data quality scoring system is constructed to assess and control the quality of each modality of data. In practice, a modality-specific standardized preprocessing process is designed to address the noise characteristics and data type differences of EEG, fMRI, and blood test data. At the same time, a multimodal data quality quantification evaluation system is constructed to achieve artifact removal, missing value completion, outlier removal, and quality grading, filtering out interference from low-quality data in the subsequent fusion process.

[0042] EEG signal normalization preprocessing: Preprocessing procedure: First, the raw EEG signal is processed... Bandpass Butterworth filtering is used to remove low-frequency baseline drift and high-frequency electromyography noise; Notch filtering eliminates power frequency interference; independent component analysis (ICA) decomposes the signal and, combined with an automatic artifact recognition algorithm, removes artifacts related to electrooculography, electrocardiography, and head movement; and spherical interpolation based on brain topology space is used to complete the bad channel electrodes, eliminating amplitude differences between individuals.

[0043] fMRI signal normalization preprocessing: Preprocessing workflow: First, time-to-time correction is performed on fMRI functional images to eliminate acquisition time differences between different slices; then head motion correction is performed to remove images with head translation exceeding [a certain threshold]. Rotation exceeds Invalid samples were removed; functional images were registered to the MNI152 standard space to achieve spatial alignment of brain regions from different individuals; Spatial smoothing was performed using a full-width half-height (FWHM) Gaussian kernel to reduce spatial noise; irrelevant covariates such as head motion parameters, white matter signal, and cerebrospinal fluid signal were regressed to remove physiological noise; finally, based on the AAL3 brain anatomy template, 90 cortical and subcortical brain regions were segmented, and the mean BOLD time series and functional connectivity matrix of each brain region were extracted to obtain a standardized fMRI feature tensor. ,in For the number of brain regions, For time step, For feature dimensions.

[0044] Standardized preprocessing of blood test data: Preprocessing procedure: First, adopt The criteria identified and removed outliers, replacing them with the mean of adjacent healthy individuals. Multiple interpolation using chain equations (MICE) was employed to imputate missing indicators, ensuring data integrity. For biochemical indicators of different dimensions, Z-score normalization was used for normalization. Multimodal data quality quantitative control: A multi-dimensional data quality scoring system was constructed, and quality scores were calculated for each modality: EEG quality score was calculated based on artifact percentage, signal-to-noise ratio, and number of bad sectors; fMRI quality score was calculated based on head movement amplitude, signal-to-noise ratio, and brain region registration accuracy; and blood test data quality score was calculated based on missing rate and outlier percentage. The quality scoring formula is as follows: in, For modality Quality rating This is the normalized signal-to-noise ratio value. The percentage of artifacts / outliers. For data missing rate, These are the weighting coefficients.

[0045] Data is graded based on quality scores, and invalid samples with overall quality scores below a preset threshold are removed. For samples with substandard single-modal quality, modality missing markers are set to provide a basis for robust fusion in subsequent missing scenarios.

[0046] Step 3: Based on the brain's anatomical structure and physiological functional regions, a five-level brain topology hierarchy is constructed, and a mapping relationship and physiological weight system between each level are established. Based on this, the electrode space of EEG signals, the brain region space of functional magnetic resonance imaging signals, and the biochemical indicators of blood test data are aligned in the spatial-physiological dimension. In practice, based on the brain's anatomical structure and physiological functional divisions, a five-level physiological brain topology (BTH) is constructed, consisting of "whole brain - functional brain region - sub-functional region - electrode cluster / anatomical division - single electrode / monocyte". A mapping relationship and physiological weight system between the levels are established to achieve spatial-physiological dual alignment of EEG electrode space, fMRI brain region space, and blood biochemical indicators, providing a physiological basis for cross-modal fusion.

[0047] Physiological definition of the five-level brain topological hierarchy: The brain topological hierarchy of this invention is defined as follows: The scale is refined step by step from coarse to fine, and the definitions and physiological relationships of each level are as follows: (Whole-brain scale): The coarsest level, including all EEG electrodes and fMRI brain regions of the whole brain, characterizing the global brain activity state and overall physiological characteristics, corresponding to the full-channel characteristics of EEG, the full-brain region characteristics of fMRI, and the global characteristics of all indicators of blood tests; (Functional brain region scale): Based on the physiological functional division of the brain, core functional brain regions are divided (such as the prefrontal cortex related to cognitive function, the hippocampus related to memory function, the occipital lobe related to visual function, etc.). Each functional brain region corresponds to the electrode grouping of EEG, the anatomical division of fMRI, and the biochemical indicators related to the functional regulation of the brain region in blood tests. (Sub-functional area scale): The functional brain regions are divided into multiple sub-functional regions to achieve more refined spatial division, corresponding to adjacent electrode clusters of EEG and subdivided anatomical regions of fMRI, capturing local functional features within the functional brain regions. (Electrode / regional cluster scale): Further refined into spatially adjacent EEG electrode clusters and fMRI voxel clusters, capturing the microcircuit functional characteristics of local brain regions, serving as an intermediate level connecting global brain regions with single electrodes / voxels. (Single electrode / voxel scale): The finest granular level, corresponding to the raw temporal signal of a single EEG electrode and the features of a single brain region / voxel in fMRI, to achieve the most refined capture of physiological features.

[0048] Construction of inter-level mapping matrix: Construct a spatial mapping matrix for adjacent levels of BTH. ,in For the first The number of nodes at the scale. For the first The number of nodes at each scale. The mapping matrix is ​​constructed based on the spatial proximity and functional connectivity strength of electrodes / brain regions, enabling feature transfer and spatial association from coarse to fine scales, ensuring accurate spatial alignment between the electrode space of EEG and the brain region space of fMRI within the BTH framework.

[0049] Physiological weight assignment: Based on the importance of brain physiological functions, physiological weights are assigned to each level and each functional brain region group of BTH. Core functional brain regions are assigned higher weight values ​​to their corresponding levels and groups, while non-core functional regions are assigned lower weight values. The weight values ​​range from [value missing]. Physiological weights will enhance the contribution of core function-related features and suppress redundant noise in irrelevant areas during subsequent feature extraction and fusion processes, making the fusion process conform to the logic of brain physiological functions.

[0050] Physiological hierarchy mapping of blood test indicators: Constructing a physiological correlation matrix ,in For the number of blood test indicators, Number of BTH levels, matrix elements Indicates the first The blood test indicators and the first The physiological correlation strength of hierarchical brain regions is determined based on physiological mechanism studies and statistical correlation analysis. This matrix maps non-spatial blood biochemical indicators to the corresponding levels of BTH, aligning blood test data with the physiological dimensions of brain topology, ultimately achieving spatial-physiological dual unification of the three modalities.

[0051] Step 4: Based on the five-level brain topology hierarchy, deep feature extraction is performed on the preprocessed EEG signals, functional magnetic resonance imaging signals, and blood test data to obtain EEG multi-scale features, functional magnetic resonance imaging multi-scale features, and blood test multi-scale features that are aligned with the five-level brain topology hierarchy. In practice, to address the differences in feature types among EEG, fMRI, and blood test data, a multi-branch modality-specific feature extraction network that is fully aligned with the BTH hierarchy is designed. This network extracts the depth physiological features of each modality at different BTH scales, ensuring complete spatial hierarchical matching of features across modalities and laying the foundation for subsequent cross-modal fusion.

[0052] EEG Multi-Scale Temporal Topological Feature Extraction Branch: A multi-scale temporal convolutional network based on the BTH layer is designed to extract spatiotemporal topological features of EEG at different scales. The network input is standardized EEG temporal samples. First, basic temporal features are extracted through shared basic convolutional layers. Then, according to the hierarchical division of BTH, the EEG channels are grouped to the corresponding levels. Through level-specific temporal convolutional blocks and spatial region attention mechanisms, spatiotemporal features at that scale are extracted.

[0053] No. The formula for EEG feature extraction at scale is:

[0054] in, For the first EEG channel grouping characteristics at different scales For spatial regional attention mechanisms, This represents the physiological weight at this scale. The kernel weights at this scale, For the first Scale-based EEG features To unify feature dimensions.

[0055] The final output is an EEG multi-scale feature tensor aligned with the BTH level. ,in This is the BTH scale number.

[0056] fMRI multi-scale spatial connectivity feature extraction branch: Based on a multi-scale graph convolutional network (GCN) architecture, using the BTH level as the graph structure, brain regions as nodes, and functional connections between brain regions as edges, this method extracts spatial topological and functional connectivity features from fMRI at different scales. The network input is the functional connectivity matrix of brain regions from fMRI. Based on the hierarchical division of BTH, subgraph structures of different scales are constructed using brain region temporal feature tensors. Spatial features of brain regions at each scale are extracted through hierarchical graph convolutional blocks.

[0057] No. The formula for updating fMRI features at different scales is:

[0058] in, For the first The scale of the subgraph adjacency matrix. This is the corresponding degree matrix. , For the first Layer Scale-based graph convolution weights and biases It is the ReLU activation function. For the first fMRI features of scale.

[0059] The final output is an fMRI multiscale feature tensor aligned with the BTH level. It is completely consistent with the dimensions of EEG features.

[0060] Multiscale physiological and biochemical feature extraction branch of blood tests: Based on a gated attention multilayer perceptron (MLP) architecture and combined with the physiological correlation matrix of BTH, this study extracts physiological and biochemical features of blood test data at different scales. The network input is a standardized blood test feature vector. First, basic biochemical features are extracted through a shared fully connected layer, and then based on the physiological correlation matrix... The blood indicators are mapped to the corresponding levels of BTH. Through the level-specific gating attention module, the contribution weight of each indicator at the corresponding level is calculated, and the biochemical features at that scale are extracted.

[0061] No. The formula for extracting blood test features based on scale is:

[0062]

[0063] in, For the first A blood test indicator, For this indicator in the 1st Attention weights based on scale For physiological correlation strength, For learnable parameters, For the first Blood test characteristics of scale.

[0064] Finally, through time-dimension broadcasting, the features at each scale are expanded into tensors with the same feature dimensions as EEG and fMRI. This completes the unification of the hierarchy and dimensions of the three modal features.

[0065] Step 5: Project the multi-scale features of EEG, multi-scale features of functional magnetic resonance imaging, and multi-scale features of blood tests into a unified latent semantic space. In practical implementation, a cross-modal contrastive learning module for physiological perception is designed to map multi-scale features from three modalities—EEG, fMRI, and blood tests—to the same semantic feature space. This minimizes the cross-modal distance between homologous features at the same BTH scale and under the same physiological state, while maximizing the feature distance between different physiological states, thereby achieving semantic unification and physiological association enhancement of cross-modal features.

[0066] Feature Projector Head Design: A shared feature projector head is designed for each modality. The projector head consists of two fully connected layers and a ReLU activation function, which projects the original features of each modality. , , Projecting onto a unified latent semantic space yields the projected features. , , This ensures that the projection feature dimensions of the three modalities are completely consistent.

[0067] Construction of the contrastive loss function for physiological perception: Using the BTH level as a unit, a contrastive loss is constructed for each scale. Different modal features at the same time step and under the same physiological state are used as positive sample pairs, and features at different physiological states and different time steps are used as negative sample pairs. A contrastive loss function for physiological perception is designed as follows:

[0068] in, For the sample size, Let be the modal combination number. The cosine similarity function is used. For temperature coefficient, For the first The first sample Modal first Scale projection characteristics.

[0069] The total contrast loss across all scales is the weighted sum of the losses at each scale, with the weights being the physiological weights for the corresponding scales. :

[0070] Cross-modal semantic alignment optimization: Based on large-scale unlabeled multimodal data, by minimizing contrastive loss The parameters of the feature projection head and the feature extraction network of each modality are optimized to eliminate the semantic gap between modalities and provide a unified feature basis for subsequent cross-modal fusion.

[0071] Step 6: Construct a physiologically guided cross-modal hierarchical fusion network. Based on a five-level brain topology hierarchy, cross-modal collaborative fusion of semantically aligned EEG features, functional magnetic resonance imaging features, and blood test features is performed layer by layer from coarse to fine scale to obtain multi-scale fusion features.

[0072] In practice, a Physiology-guided Cross-modal Hierarchical Fusion Network (PCHF-Net) is designed. Based on the BTH hierarchy, cross-modal feature fusion is completed layer by layer from coarse to fine scale, ultimately achieving deep coupling of EEG temporal features, fMRI spatial features, and blood biochemical features.

[0073] The fusion process is divided into three stages: intra-layer cross-modal collaborative fusion, inter-layer scale feature transfer, and full-scale global feature aggregation. arrive The process is executed layer by layer, and the specific technical solution is as follows: Intra-layer cross-modal collaborative fusion: Regarding BTH's first Scale, input the three aligned modal features at this scale. , , A cross-modal collaborative attention mechanism was designed to adaptively learn the complementary correlations of features from three modalities, while also incorporating physiological weights. Strengthen the fusion weights of core physiological features to generate intra-layer fusion features at this scale.

[0074] First, calculate the cross-modal collaborative attention weights:

[0075]

[0076]

[0077] in, The projection matrix is ​​learnable. For modality For modes Cross-modal attention features This represents the collaborative attention fusion feature at this scale.

[0078] Finally, the original modal features and collaborative attention features are fused through a gated fusion unit to obtain the intra-layer fused features at this scale:

[0079] in, For feature splicing operations, It is a gated linear unit that adaptively controls the fusion ratio of the original features and the attention features.

[0080] Interlayer scale feature transfer: The previous coarse scale ( ) fusion features Through hierarchical mapping matrix Perform upsampling and dimensionality transformation, and compare with the current fine scale ( ) fusion features Residual connections are performed to transfer coarse-scale global physiological information to fine-scale local features, ensuring the hierarchical continuity of the fused features. The formula is as follows:

[0081] in, For bias terms, It is a one-dimensional convolutional layer used for feature dimension matching.

[0082] Full-scale global feature aggregation: Finish arrive After layer-by-layer fusion, a full-scale cross-modal fusion feature tensor is obtained. The multi-scale adaptive pooling module performs weighted aggregation of fused features at different scales, with the weights being the physiological weights of each scale. Finally, a globally unified multimodal fusion feature vector is obtained. The formula is:

[0083] in, For average pooling, For max pooling, This is a feature splicing operation. The fused feature comprehensively integrates temporal, spatial, and biochemical physiological information from three modalities, forming a complete multi-dimensional medical information system.

[0084] The brain-computer interface multimodal signal fusion method provided in this embodiment achieves precise spatial-physiological alignment of heterogeneous modalities through a multimodal fusion framework guided by full-process physiological priors. Existing methods can only achieve pure data-driven, indiscriminate fusion, completely ignoring the physiological structure and functional logic of the brain, and cannot solve the problem of spatiotemporal heterogeneity among EEG, fMRI, and blood test modalities. This invention constructs a five-level brain topological hierarchy (BTH) based on brain physiological priors, and for the first time achieves spatial-physiological alignment of EEG electrode space, fMRI brain region space, and blood biochemical indicators, maximizing the preservation of effective physiological information of each modality. Existing methods employ a uniform feature extraction and processing flow, which cannot adapt to the signal characteristics of different modalities, resulting in the loss of a large amount of effective physiological information. This invention designs a dedicated technical solution for each modality, from acquisition synchronization, preprocessing, feature extraction to semantic alignment, specifically addressing the noise interference, feature extraction, and scale matching problems of each modality. At the same time, it achieves complete alignment of features of each modality through the BTH layer, maximizing the preservation of effective physiological information of each modality before fusion, laying a solid foundation for subsequent deep fusion. A physiologically guided cross-modal hierarchical fusion network is designed to achieve deep coupling modeling of temporal-spatial-biochemical information. Existing fusion methods can only achieve shallow feature splicing or decision fusion, failing to capture the intrinsic physiological connections between the three modalities. This invention achieves cross-modal fusion layer by layer from coarse to fine through intra-layer cross-modal collaborative attention and inter-layer scale feature transfer mechanisms. Simultaneously, it strengthens core physiological features through physiological weights, suppresses redundant noise, and fully explores the physiological coupling relationships between EEG dynamic temporal information, fMRI spatial topological information, and blood biochemical information, forming a comprehensive, complete, and high-value multi-dimensional medical information system. Furthermore, it proposes a semi-supervised pre-training + federated learning training system, simultaneously solving the two core challenges of small-sample generalization and privacy protection.Existing methods rely on large-scale centralized paired data for training, but such data is extremely scarce in clinical settings, and medical data is subject to strict privacy protection requirements. This invention utilizes semi-supervised contrastive pre-training to fully leverage large-scale unlabeled data to learn general physiological representations, maintaining stable model performance even in small sample scenarios with only 10% labeled data. Existing methods have extremely high data quality requirements, and their performance drops significantly in scenarios with single-modal missing data, low signal-to-noise ratio, and artifact interference. This invention, through a data quality control system, mask pre-training strategy, and adaptive fusion design for modality missing data, allows the model to perform well in scenarios with single-modal / ... Even in scenarios with missing or low-quality bimodal data, this invention maintains stable medical information fusion performance. Its modular architecture supports flexible adaptation to different modality combinations and application scenarios, allowing for rapid migration to various scenarios such as brain science research and physiological state monitoring, demonstrating far superior adaptability compared to existing methods. Existing deep learning fusion models lack physiological basis in their fusion process, resulting in difficult-to-interpret output information. This invention's fusion process is entirely based on prior knowledge of brain physiology, accurately quantifying the contribution of different modalities, brain regions, and biochemical indicators to the fused information. The fused medical information is directly correlated with clinical physiological indicators, possessing clear physiological significance, and providing interpretable and high-value information support for brain science research and clinical applications.

[0085] Based on the above embodiments, a pre-training step is also included: Construct a pre-trained dataset containing unlabeled or weakly labeled data with multiple centers and multiple modalities; The design incorporates a multi-task pre-training objective that includes cross-modal alignment loss, modal reconstruction loss, and mask modeling loss. The total pre-training loss function is:

[0086] in, For cross-modal alignment loss, For modal reconstruction loss, For mask modeling loss, , , These are the weighting coefficients; Based on the pre-trained dataset, the total pre-training loss function is minimized, and the feature extraction network, cross-modal alignment module, and cross-modal hierarchical fusion network involved in steps 4 to 6 are jointly pre-trained to learn a general representation of brain physiological states.

[0087] In practice, to address the scarcity of clinical multimodal paired data, a semi-supervised contrastive pre-training framework was designed. Based on large-scale unlabeled / weakly labeled multimodal data, the model learns general representations of brain physiological states, significantly reducing the model's dependence on large-scale paired data and improving the model's generalization ability in small sample scenarios.

[0088] Construction of pre-training dataset: Collect large-scale, multi-center, unlabeled multimodal data, including fully paired EEG+fMRI+blood test data and incomplete unimodal / bimodal data, to construct a pre-training dataset with a total of no less than 100,000 data points, covering people of different ages, genders, and physiological states, to ensure the diversity and representativeness of the data.

[0089] Multi-task pre-training objective design: The pre-training process employs multi-task joint optimization, and the overall pre-training loss function is a weighted sum of the losses from the three sub-tasks:

[0090] in, The weights are as follows: The losses for the three sub-tasks are: Cross-modal alignment loss This refers to the physiological perception contrast loss in step five, which is used to achieve semantic alignment of cross-modal features; Modal reconstruction loss Based on fusion features, the original input features of each modality are reconstructed through a decoder. The fusion features are constrained to retain complete and effective physiological information. The loss value is the mean squared error between the reconstructed features and the original features.

[0091] Mask modeling loss The random mask uses partial modality / partial scale features, allowing the model to predict the masked content based on the remaining features, thus improving the model's robustness to scenarios with missing single modality data. Cross-entropy loss is used to calculate the mask prediction error.

[0092] Pre-training process optimization: Based on a large-scale unlabeled dataset, the AdamW optimizer is used to minimize the total pre-training loss. The pre-training of the feature extraction network, cross-modal alignment module, and fusion network was completed to learn general representations of brain physiological states. After pre-training, the backbone parameters of the feature extraction and fusion networks were frozen, and only the subsequent information processing layers were retained for training, laying the foundation for subsequent in-depth analysis of medical information.

[0093] The feasibility, effectiveness, robustness, and clinical suitability of the method and system have been fully demonstrated through large-scale simulation experiments, multi-center dataset validation, and small-sample clinical trials. The experiment focused on assessing brain physiological status, extracting brain functional features, and integrating and analyzing multi-dimensional medical information as core validation objectives. It selected the ADNI Alzheimer's disease dataset, the TUAB / TUEV epilepsy dataset, the REST-meta-MCI cognitive impairment dataset, and a small-sample multimodal dataset from domestic hospitals (a total of 240 samples, including healthy and abnormal individuals with different physiological states; all samples included paired EEG, fMRI, and blood test data). The method of this invention was comprehensively compared with existing mainstream methods such as EEGNet, NeuroLM, EEGPT, and BIOT. Feature reconstruction accuracy (Recon-ACC), cross-modal information matching (CMI), and physiological feature extraction completeness (PFI) were used as core evaluation indicators. Simultaneously, specific validations were conducted on small-sample performance, modality missing robustness, and cross-center generalization performance. In the three core validation objectives, the overall performance of the method of this invention comprehensively surpassed existing mainstream methods. The core experimental results are shown in Table 1.

[0094] Table 1

[0095] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof.

[0096] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for multimodal signal fusion in a brain-computer interface, characterized in that, include: Step 1: Acquire the EEG signals, functional magnetic resonance imaging (fMRI) signals, and blood test data of the target object, and based on a unified global time reference, accurately align the EEG signals, fMRI signals, and blood test data in the time dimension to obtain time-aligned multimodal data. Step 2: Modality-specific standardized preprocessing is performed on the EEG signals, functional magnetic resonance imaging signals, and blood test data in the time-aligned multimodal data, and a multi-dimensional data quality scoring system is constructed to assess and control the quality of each modality of data. Step 3: Based on the brain's anatomical structure and physiological functional regions, a five-level brain topology hierarchy is constructed, and a mapping relationship and physiological weight system between each level are established. Based on this, the electrode space of EEG signals, the brain region space of functional magnetic resonance imaging signals, and the biochemical indicators of blood test data are aligned in the spatial-physiological dimension. Step 4: Based on the five-level brain topology hierarchy, deep feature extraction is performed on the preprocessed EEG signals, functional magnetic resonance imaging signals, and blood test data to obtain EEG multi-scale features, functional magnetic resonance imaging multi-scale features, and blood test multi-scale features that are aligned with the five-level brain topology hierarchy. Step 5: Project the multi-scale features of EEG, multi-scale features of functional magnetic resonance imaging, and multi-scale features of blood tests into a unified latent semantic space. Through cross-modal comparative learning of physiological perception, minimize the distance between features of different modalities under the same physiological state, while maximizing the distance between features under different physiological states, to achieve semantic alignment of cross-modal features. Step 6: Construct a physiologically guided cross-modal hierarchical fusion network. Based on a five-level brain topology hierarchy, cross-modal collaborative fusion of semantically aligned EEG features, functional magnetic resonance imaging features, and blood test features is performed layer by layer from coarse to fine scale to obtain multi-scale fusion features. The multi-scale fusion features are then globally aggregated based on the physiological weights of each scale to generate the final multi-modal fusion features.

2. The method according to claim 1, characterized in that, The step of precisely aligning EEG signals, functional magnetic resonance imaging signals, and blood test data in the time dimension based on a unified global time reference to obtain time-aligned multimodal data includes: Step 1.1: Using the hardware clock of the acquisition system as the global unified time reference, mark each sampling point of the EEG signal, each time frame of the functional magnetic resonance imaging signal, and the blood collection time point of the blood test data with a globally unique timestamp. Step 1.2: Based on the timestamp, the low temporal resolution sequence of the functional magnetic resonance imaging (fMRI) signal is upsampled to the same time step as the EEG signal using linear interpolation. The expression for the linear interpolation method is: in , The timestamps are the timestamps of two adjacent time frames of a functional magnetic resonance imaging (fMRI) signal. The time point corresponding to the EEG signal. This indicates the blood oxygenation level dependent on the signal value in fMRI; Step 1.3: Using the blood collection time point of the blood test data as the anchor point, the corresponding blood test feature vector is used as the global physiological feature and broadcast on the entire time series of EEG signals and functional magnetic resonance imaging signals to realize the temporal dimension correlation between biochemical indicators and brain signal temporal features.

3. The method according to claim 1, characterized in that, Step 2 specifically includes: Step 2.1: Modality-specific normalization preprocessing is performed on the EEG signals, functional magnetic resonance imaging (fMRI) signals, and blood test data in the time-aligned multimodal data. The modality-specific normalization preprocessing for EEG signals includes at least one of bandpass filtering, notch filtering, and independent component analysis for artifact removal. The modality-specific normalization preprocessing for fMRI signals includes at least one of temporal correction, head motion correction, spatial registration, and spatial smoothing. The Z-score normalization method is used to normalize the blood test data. Step 2.2: Calculate the quality scores for the electroencephalogram (EEG) signal, functional magnetic resonance imaging (fMRI) signal, and blood test data, respectively. The expression for the quality score is as follows: in, For modality Quality rating This is the normalized signal-to-noise ratio value. The percentage of artifacts / outliers. For data missing rate, These are the weighting coefficients; Step 2.3: Based on the calculated quality score, classify the multimodal data, remove invalid samples with a comprehensive quality score lower than a preset threshold, and set a modality missing label for samples with substandard single-modal quality.

4. The method according to claim 1, characterized in that, The five-level brain topology hierarchy includes whole-brain scale, functional brain region scale, sub-functional region scale, electrode or regional cluster scale, and single electrode or single element scale. Adjacent levels are connected by a spatial mapping matrix. To achieve feature transfer and spatial association, where For the first The number of nodes at the scale. For the first The number of nodes at different scales, and assigning physiological weights to each level and functional brain region. .

5. The method according to claim 4, characterized in that, The step of aligning the electrode space of EEG signals, the brain region space of functional magnetic resonance imaging signals, and the biochemical indicators of blood test data in a spatial-physiological dimension includes: The electrode space of EEG signals and the brain region space of functional magnetic resonance imaging signals are aligned within a five-level brain topological hierarchy framework using a spatial mapping matrix. Constructing a physiological correlation matrix The biochemical indicators of blood test data are mapped to the corresponding levels of the five-level brain topological hierarchy, among which... For the number of blood test indicators, The number of levels in the five-level brain topology hierarchy, matrix elements Indicates the first The blood test indicators and the first The strength of physiological connections between hierarchical brain regions.

6. The method according to claim 1, characterized in that, Step 4 specifically includes: Step 4.1: Extract multi-scale temporal topological features of EEG signals at each scale of the five-level brain topological hierarchy using a multi-scale temporal convolutional network, where the first... The formula for extracting EEG features at different scales is: in, For the first EEG channel grouping characteristics at different scales For spatial regional attention mechanisms, This represents the physiological weight at this scale. The kernel weights at this scale, For the first Scale-based EEG features To unify feature dimensions; Step 4.2: Using a multi-scale graph convolutional network with a five-level brain topology as the graph structure, extract the multi-scale spatial connectivity features of functional magnetic resonance imaging (fMRI) signals at each scale. The formula for updating the features of functional magnetic resonance imaging at different scales is: in, For the first The scale of the subgraph adjacency matrix. This is the corresponding degree matrix. , For the first Layer Scale-based graph convolution weights and biases It is the ReLU activation function. For the first fMRI features at scale; Step 4.3: Using a gated attention multilayer perceptron and combining it with a physiological correlation matrix, extract multiscale physiological and biochemical features of blood test data at various scales, where the first... The formula for extracting blood test features based on scale is: in, For the first A blood test indicator, For this indicator in the 1st Attention weights based on scale For physiological correlation strength, For learnable parameters, For the first Scale of blood test characteristics.

7. The method according to claim 1, characterized in that, Step 5 specifically includes: Step 5.1: Design a feature projection head with a shared structure for EEG signals, functional magnetic resonance imaging signals and blood test data respectively, and project the multi-scale features of each modality onto a unified latent semantic space to obtain the projected EEG features, functional magnetic resonance imaging features and blood test features. Step 5.2: Using each scale of the five-level brain topology hierarchy as a unit, construct a contrastive loss function for physiological perception. Different modal projection features at the same time step and under the same physiological state are used as positive sample pairs, and projection features at different physiological states or different time steps are used as negative sample pairs. Minimize the distance between positive sample pairs while maximizing the distance between negative sample pairs. The contrastive loss function for physiological perception includes the... Scale contrast loss and total alignment loss : in, For the sample size, Let be the modal combination number. The cosine similarity function is used. For temperature coefficient, For the first The first sample Modal first Projection characteristics of scale; Step 5.3: Based on the contrastive loss function, optimize the parameters of the feature projection head and the feature extraction network of each modality until the projected features of different modalities achieve semantic alignment at the physiological state level in the latent semantic space.

8. The method according to claim 1, characterized in that, The step of performing cross-modal collaborative fusion of semantically aligned EEG features, functional magnetic resonance imaging features, and blood test features layer by layer from coarse to fine scale includes: For the current scale Through a cross-modal collaborative attention mechanism, it adaptively learns the complementary correlations among EEG features, functional magnetic resonance imaging features, and blood test features, and combines them with physiological weights at the current scale. This generates intra-layer fusion features at this scale, where the formula for calculating collaborative attention features is: in, The projection matrix is ​​learnable. For modality For modes Cross-modal attention features This represents the collaborative attention fusion feature at this scale; The previous coarse-scale fusion features Through the hierarchical mapping matrix After dimensional transformation, the intra-layer fusion features are combined with the current fine-scale features. Residual connections are performed to transfer coarse-scale global physiological information to fine-scale local features. The fusion formula is as follows: in, This is a bias term.

9. The method according to claim 1, characterized in that, It also includes a pre-training step: Construct a pre-trained dataset containing unlabeled or weakly labeled data with multiple centers and multiple modalities; The design incorporates a multi-task pre-training objective that includes cross-modal alignment loss, modal reconstruction loss, and mask modeling loss. The total pre-training loss function is: in, For cross-modal alignment loss, For modal reconstruction loss, For mask modeling loss, , , These are the weighting coefficients; Based on the pre-trained dataset, the total pre-training loss function is minimized, and the feature extraction network, cross-modal alignment module, and cross-modal hierarchical fusion network involved in steps 4 to 6 are jointly pre-trained to learn a general representation of brain physiological states.

Images