A federated self-supervised training method for EEG-fNIRS fusion

Abstract

The application discloses a kind of federated self-supervised training methods for EEG-fNIRS fusion, it is related to artificial intelligence technical field.The method is based on the distributed system of client and federal server, data preprocessing is completed locally by client, HRF individualization time lag estimation and alignment, cross-modal self-supervised feature learning, projection embedding mean, sparse cross-correlation matrix such as low-dimensional statistics are generated, and are uploaded after differential privacy processing;Server constructs proxy loss function containing diagonal correlation enhancement, non-diagonal inhibition, mean alignment and uncertainty weight adjustment by securely aggregating statistics, optimizes and updates global model and issues, and client iteratively trains until model convergence.The present application does not need to concentrate original sensitive data, and considers privacy security and communication efficiency, and through individualized HRF alignment and adaptive contrast learning, the robustness and generalization ability of cross-modal joint representation are improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a federated self-supervised training method for EEG-fNIRS fusion. Background Technology

[0002] Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) are two widely used non-invasive brain imaging techniques. EEG primarily records neuronal potential changes using scalp electrodes. This technique boasts high temporal resolution down to the millisecond level, directly reflecting rapid neural information processing. However, its spatial resolution is relatively low, and the signal is susceptible to artifacts. fNIRS, on the other hand, estimates changes in blood oxygen metabolism in the cerebral cortex through near-infrared light illumination and reflection. It offers high spatial resolution, wearability, and is relatively robust to motion artifacts. However, it reflects the indirect hemodynamic response induced by neural activity, resulting in a time delay of several seconds and lower temporal resolution. Due to their natural complementarity in spatiotemporal resolution, their fusion has become an important research direction in the fields of multimodal brain-computer interfaces (BCI) and neuroimaging in recent years.

[0003] However, most existing EEG-fNIRS fusion methods are based on a centralized supervised learning paradigm. Whether it is the early feature-level / decision-level fusion, or the dual-stream or multi-stream (e.g., CNN / RNN / Transformer architecture) convolutional networks and multi-layer progressive / fusion deep learning frameworks adopted in recent years, all require the original datasets collected from multiple centers to be centralized on a single server for unified training. This model has significant drawbacks: First, it seriously violates data privacy and security regulations. EEG and blood oxygenation data are sensitive personal physiological information, and their centralized storage and transmission face compliance obstacles in multi-institutional collaboration scenarios, making practical implementation difficult. Second, it ignores the physiological basis of cross-modal temporal alignment. Existing methods mostly use fixed HRF (Hemodynamic Response Function) templates or implicit network learning alignment, lacking a mechanism to explicitly estimate HRF and its time delay parameters at the individual level and incorporate them into the training objective. This results in insufficient alignment under cross-subject and cross-center data conditions. Third, it heavily relies on large-scale labeled data, but the labeling cost of synchronous EEG-fNIRS data is high, and the distribution of subjects varies greatly, limiting the model's generalization ability.

[0004] In recent years, to reduce reliance on labeled data and alleviate the problems of expensive EEG annotation and large differences in subject distribution, self-supervised learning has been widely used in EEG single-modal analysis. It can learn transferable temporal representations from unlabeled data through tasks such as contrastive learning and mask prediction. However, existing EEG self-supervised methods are mostly centralized training methods, failing to consider the practical situation of data isolation across multiple institutions. Existing centralized self-supervised methods for EEG-fNIRS generally fail to explicitly model the physiological priors of electro-blood flow coupling and do not design federated self-supervised training mechanisms for multi-center data silo environments. Furthermore, with increasing data security regulations and privacy protection requirements, federated learning (FL) has attracted widespread attention in the fields of medical imaging and EEG. It further combines the advantages of unlabeled data utilization and privacy protection, enabling collaborative model training using distributed data while protecting data privacy. However, existing federated learning and federated self-supervised learning schemes are mainly geared towards single-modal EEG or medical imaging. Their technical measures do not consider the unique cross-modal timescale differences, HRF delays, and joint representation requirements of EEG-fNIRS, making them difficult to directly apply to high-quality joint modeling of EEG-fNIRS in a multi-institutional distributed environment. Therefore, under current technological conditions, how to conduct federated self-supervised training using unlabeled EEG-fNIRS data from multiple centers without centralizing the original EEG-fNIRS data, and how to combine this with individualized HRF delay alignment to construct robust cross-modal joint representations, remains an unresolved technical problem.

[0005] In view of this, the present invention is proposed. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a federated self-supervised training method for EEG-fNIRS fusion, thereby solving at least one of the aforementioned technical problems.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] A federated self-supervised training method for EEG-fNIRS fusion, implemented using a system consisting of client and cloud server layers, includes the following steps:

[0009] Step S1, Data Preprocessing and Fragment Generation:

[0010] On the client side, the data acquisition and preprocessing module performs normalization processing on the raw EEG signal and fNIRS signal stored locally on the client side to obtain standardized data fragments;

[0011] Step S2, Individualized Delay Estimation and Alignment:

[0012] The client explicitly corrects the time-scale differences between EEG and fNIRS signals through physiological modeling and dynamic estimation, generates physiologically reliable cross-modal positive sample pairs, and outputs the alignment uncertainty.

[0013] Step S3, Cross-modal self-supervised feature learning:

[0014] The client constructs a cross-modal contrast task based on aligned segments, enabling the encoder to learn the joint representation space of EEG-fNIRS without relying on manual annotation;

[0015] Step S4: The client completes the generation of low-dimensional statistics, differential privacy processing, and uploading.

[0016] After completing cross-modal self-supervised feature learning, the client only extracts low-dimensional statistics that support the server's global update, and uploads them to the federated server after privacy protection processing.

[0017] Step S5, Server Security Aggregation and Global Approximate Update:

[0018] After the client completes the generation of statistics and uploads the differential privacy perturbation, the server enters the global aggregation and model update stage. That is, without obtaining any original EEG-fNIRS data and fragment-level high-dimensional features, the low-dimensional statistics uploaded by the client are used to approximately construct the global optimization direction of cross-modal contrastive learning.

[0019] Step S6: Send the updated globally shared model parameters to the client and start the next round of federated training iteration;

[0020] Step S7: The client uses the new global model as the initial value and re-executes steps S1 to S6 to enter the next round of federated training until the model converges.

[0021] Compared with the prior art, the present invention has at least the following beneficial effects:

[0022] 1. This invention adopts a federated learning architecture. The client only uploads low-dimensional aggregated statistics such as projection embedding mean and sparse cross-correlation matrix, without transmitting the original EEG-fNIRS data and high-dimensional features. Combined with the differential privacy mechanism of L2 norm pruning and Gaussian noise injection, it avoids the leakage of sensitive physiological information from the root, fully complies with medical data privacy protection regulations, and solves the compliance problem of centralized storage and transmission of data in multi-center collaboration.

[0023] 2. An innovative individualized dual-gamma HRF model is introduced, which estimates the time delay parameters through state-space modeling and Kalman filtering, combined with dynamic time warping fine-tuning, to explicitly correct the physiological time scale differences between EEG and fNIRS, while outputting the alignment uncertainty. This overcomes the limitations of traditional fixed templates or implicit learning alignment and improves the alignment reliability of cross-subject and cross-center data.

[0024] 3. Based on the self-supervised contrastive learning paradigm, without the need for large-scale labeled data, physiologically reliable positive and negative sample pairs are constructed through HRF alignment, and the alignment uncertainty is used to adaptively adjust the InfoNCE loss temperature, enabling the model to learn robust cross-modal joint representations from multi-center unlabeled data, effectively alleviating the problems of high labeling costs and large data distribution differences.

[0025] 4. The "secure aggregation and global approximate update" mechanism proposed in this invention enables the server to still obtain the global optimization capabilities required for contrastive learning without touching the original EEG-fNIRS data and sample-level embeddings. This overcomes the structural limitation of existing federated contrastive learning, which cannot perform InfoNCE under "statistical federation" conditions. By utilizing only low-dimensional statistics (mean, cross-modal correlation structure, and time-delay uncertainty, etc.) uploaded by the client and protected by differential privacy, the server can construct a computable surrogate optimization objective, ensuring its update direction is consistent with the theoretical optimal solution of InfoNCE. This allows for effective approximate optimization of contrastive learning even in scenarios without samples. This mechanism not only has a natural advantage in privacy protection, avoiding the privacy leakage risks associated with uploading high-dimensional embeddings or gradients, but also naturally achieves consistent approximation of the cross-center contrastive structure through statistical aggregation. This enables the global model to steadily converge to the client's true InfoNCE learning distribution, thereby significantly improving cross-center generalization performance.

[0026] 5. This invention utilizes the time delay uncertainty of the upload to adjust the weights of the global update, enabling the server to automatically reduce the update intensity when alignment is unreliable and enhance cross-modal consistency when alignment is stable, forming a robust end-to-end optimized link. Simultaneously, since the uploaded data volume is only in the KB range, an optimal trade-off is achieved between communication cost and privacy protection, enabling high-performance federated self-supervised pre-training to be conducted even in real-world scenarios with multiple centers, low bandwidth, and strong privacy constraints. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0028] Figure 1 This is a flowchart of the federated self-supervised training method of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments. The illustrative embodiments and descriptions of this invention are only used to explain this invention and are not intended to limit this invention.

[0030] First, it should be noted that the method of this invention is implemented using a system consisting of two layers: a client and a cloud server. The client is deployed at the data holding institution (e.g., a hospital, laboratory), where it locally collects and holds the raw EEG-fNIRS data, performs data preprocessing, HRF individualized alignment, self-supervised encoding and loss calculation, statistic generation, differential privacy perturbation, and uploads the data. The federated server does not access the raw data; it securely aggregates low-dimensional statistics or low-rank parameter updates from each client, performs global approximate optimization of the self-supervised objective and model version management, and distributes the global encoder.

[0031] The relevant terms and symbols are as follows:

[0032] (1) EEG segment, which is a single-window EEG signal segment obtained after preprocessing, and its mathematical expression is denoted as: In the formula, C e The number of channels for the EEG signal (determined by the electrode layout of the acquisition device, such as 64 channels or 128 channels). L e The window length for a single EEG segment (unit: sampling points; calculated from the sampling rate and time window length, for example, a 10-second window at a 100Hz sampling rate corresponds to...). L e =1000); i is the fragment index, used to distinguish EEG data from different time windows within the same client; This represents the i-th EEG segment. The dimension is C e Line × L e A matrix of real numbers in columns;

[0033] (2) The fNIRS fragment, a single-window functional near-infrared spectral signal fragment obtained after preprocessing, is mathematically expressed as: In the formula, C n The number of channels / optical paths for the fNIRS signal (determined by the optical pole pair layout, e.g., 48 optical paths); L n The window length for a single fNIRS segment (unit: sample points; determined by the resampled timing reference, e.g., a 10-second window at a 10Hz sampling rate). Ln =100); i is the fragment index, and One-to-one correspondence, representing cross-modal data pairs within the same time window; This represents the i-th fNIRS segment. The dimension is C n Line × L n A matrix of real numbers in columns;

[0034] (3) HRF parameter θ, in this embodiment, a dual-gamma HRF model is preferred, the parameters of which include positive peak response parameters and negative peak response parameters. The positive peak response parameters include the first shape parameter α1, the first scale parameter β1 and the first amplitude parameter A1, and the negative peak response parameters include the second shape parameter α2, the second scale parameter β2 and the second amplitude parameter A2.

[0035] (4) Time delay That is, the delay from EEG triggering to fNIRS response, in seconds;

[0036] (5) The embedding / projection relationship satisfies: ;in, Indicates that the EEG encoder is used for The high-dimensional feature vector obtained after processing; Indicates that the fNIRS encoder is used for The high-dimensional feature vector obtained after processing; Indicates will via EEG projection head (g) e The low-dimensional embedding vector obtained after mapping, Indicates will via fNIRS projection head (g n The low-dimensional embedding vector obtained after mapping, Represents the d-dimensional real space;

[0037] (6) InfoNCE temperature T (Note: This temperature is not the physical temperature, but a hyperparameter in machine learning used to adjust the "sharpness" of the similarity distribution). In this embodiment, the temperature T is determined by... Adaptive adjustment; InfoNCE (Information Noise-Contrastive Estimation) is the core loss function of contrastive learning, which aims to learn data representation by "bringing the similarity of positive sample pairs closer and pushing the similarity of negative sample pairs further apart"; It is the variance of the multi-window time delay estimate, which directly reflects the uncertainty of HRF time delay alignment;

[0038] (7) Low-dimensional statistics, including mean, variance, cross-correlation matrix, similarity histogram, alignment parameter statistics, etc.

[0039] Next, the federated self-supervised training method of the present invention will be described in detail:

[0040] like Figure 1 As shown, this invention provides a federated self-supervised training method for EEG-fNIRS fusion, which includes the following steps:

[0041] Step S1, Data Preprocessing and Fragment Generation:

[0042] On the client side, the data acquisition and preprocessing module first normalizes the locally stored raw EEG and fNIRS signals to obtain normalized data fragments that can be used for subsequent HRF alignment and cross-modal self-supervised learning. Specifically, this includes the following steps:

[0043] S11. Signal cleaning and bandpass filtering:

[0044] To address the differences in signal characteristics between EEG and fNIRS, a differentiated filtering strategy is employed to retain the effective components related to neural activity. Specifically, for EEG signals, a bandpass filter of 0.5–45Hz (or 1–40Hz) is used to remove power line interference and high-frequency noise, with a 50 / 60Hz notch filter added when necessary. For fNIRS signals, bandpass filters of 0.01–0.2Hz are applied to the HbO (oxyhemoglobin) and HbR (deoxyhemoglobin) channels respectively to suppress low-frequency drift and high-frequency artifacts, ensuring that the output signal mainly contains the effective components related to neural blood flow coupling.

[0045] S12. Artifact Detection and Correction:

[0046] Artifacts are a major source of interference in brain signals, requiring differentiated processing for EEG and fNIRS signals based on their artifact types. Specifically, for EEG signals, independent component analysis (ICA), signal spatial projection (SSP), or reference channel-based regression methods are used to remove artifacts related to electrooculography, electrocardiography, and electromyography. For fNIRS signals, motion artifacts are identified through threshold detection assisted by accelerometers or sliding window outlier analysis, and corrected using multinomial regression, short-channel regression, or waveform interpolation to ensure the quality of the blood oxygen time series required for subsequent HRF modeling.

[0047] S13. Sampling rate unification and time axis synchronization:

[0048] Since EEG and fNIRS have different sampling frequencies (e.g., EEG is 128-512Hz, fNIRS is 5-20Hz), this embodiment unifies the two signals to a preset timing reference through resampling (e.g., downsampling EEG to 100Hz and upsampling fNIRS to 10Hz or aligning by frequency multiples). Then, combined with the device acquisition timestamp, event trigger flag, or synchronization pulse, the two modal signals are time-axis corrected to ensure that they have a consistent reference time at the window level.

[0049] S14. Windowing and Fragment Generation:

[0050] The time-synchronized EEG and fNIRS signals are processed by sliding segmentation according to a fixed window length L (preferably 8-12 seconds, e.g., the number of sampling points under a typical value of 10 seconds) and a step size s (0.5–2 seconds) to obtain paired EEG segments. With fNIRS fragments During the windowing process, the corresponding session number, subject number, and task event type are recorded to provide auxiliary information for subsequent HRF alignment and positive sample construction.

[0051] Through the above preprocessing steps, each client generates a set of reliable, time-synchronized, and structurally normalized multimodal data fragments locally. This set serves as the input basis for subsequent HRF individualized time delay estimation and cross-modal self-supervised feature learning, thereby ensuring the stability and comparability of the entire federated training process.

[0052] Step S2, HRF Individualized Delay Estimation and Alignment (Client):

[0053] The core objective of this step is to explicitly correct the temporal differences between EEG and fNIRS signals through physiological modeling and dynamic estimation, generate physiologically reliable cross-modal positive sample pairs, and output alignment uncertainty, thus providing a foundation for subsequent self-supervised training. The specific steps are as follows:

[0054] S21, HRF model construction:

[0055] To accurately describe the physiological mapping relationship between neural activity and fNIRS blood oxygenation signal, a dual-gamma HRF model with strong physiological interpretability was selected, adapted to individual differences in neurovascular coupling. The model formula is as follows:

[0056] ;

[0057] In the formula, h(t) is the hemodynamic response function, representing the standardized blood oxygen signal change at time t under unit neural activation; α1, β1, and A1 represent the shape, scale, and amplitude parameters of the positive peak response; α2, β2, and A2 represent the shape, scale, and amplitude parameters of the negative peak response; Γ() represents the gamma function; where the parameters of HRF are allowed to vary within a reasonable physical range to accommodate individual differences, for example, α1∈[4,8], β1∈[0.5,1.5], A1∈[0.5,2.0]; α2∈[8,16], β2∈[0.5,2.0], A2∈[0.1,1.0];

[0058] S22, Neural drive structure:

[0059] The standardized EEG fragment output from step S1 In this process, driving signals reflecting neural activation intensity are extracted and used as convolution inputs to the HRF model to generate the desired fNIRS waveform. Specifically, multi-band power, principal component analysis (PCA), or Hilbert envelope extraction methods are preferred for extraction.

[0060] S23. State-space modeling and recursive estimation:

[0061] To dynamically estimate HRF parameters and time delays, this step uses a state-space model combined with extended Kalman filtering (EKF) or unscented Kalman filtering (UKF) for recursive estimation on the client side.

[0062] State-space modeling includes:

[0063] Define state vector In the formula, i k This represents the estimates of each parameter of the HRF in the k-th time window. This indicates the time lag between EEG-driven blood oxygenation response and fNIRS response;

[0064] The state equations are constructed as follows: In the formula, Let Q be the process noise at the k-th time window, and let Q be the process noise covariance matrix, set based on sensor noise levels or empirical statistics. This indicates that the process noise follows a normal Gaussian distribution with a mean of 0 and a covariance matrix of Q;

[0065] The observation equations are constructed as follows:

[0066] ;

[0067] In the formula, This represents the actual change in blood oxygen concentration measured by the fNIRS device in the k-th window, i.e., the actual observed value. This represents the predicted fNIRS blood oxygenation signal value calculated by the HRF model from the neural driving signal in the k-th window. represents the observation noise for the k-th time window, used to describe random errors (such as optical path interference, ambient light noise, etc.) during the fNIRS measurement process; R is the observation noise covariance matrix, set according to the sensor noise level or empirical statistics; * represents the time-domain convolution operator. This indicates that the observation noise follows a normal Gaussian distribution with a mean of 0 and a covariance matrix of R; The driving signal that reflects the intensity of neural activation is extracted for the k-th time window and is calculated from the preprocessed EEG fragment; The physiological mapping function value (obtained from the dual-gamma HRF model) represents the HRF parameters of the k-th window. i k The defined double gamma function is used to characterize the dynamic law of "neural activation → blood oxygen response"; The EEG-fNIRS lag (in seconds) for the kth time window represents the time delay from the triggering of the neural drive signal to the generation of the fNIRS blood oxygenation response, and is used to correct the time offset between the predicted signal and the observed signal.

[0068] After state-space modeling is completed, the next step is to perform recursive estimation using EKF or UKF, which is mainly divided into two stages: "prediction - new loop". Specifically, in the prediction stage, based on the estimation results of the previous window, the prior state estimate for the k-th window is obtained first. , (in, The initial estimate is determined by calculating the prior error covariance (for the (k-1)th window's posterior state estimate) and then the prior error covariance. In the update phase, the current observations are fused to optimize the estimation. First, the Kalman gain is calculated based on the prior error covariance, observation noise covariance, and observation matrix (EKF requires linearization of the observation equation to calculate the Jacobian matrix, while UKF avoids the linearization step through unscented transformation). Then, this gain is used to fuse the prior estimate and observation residuals to obtain the posterior state estimate for the k-th window. , 【in, [These are weighting coefficients, whose values ​​are jointly determined by the prediction error covariance matrix and the measurement error covariance matrix]. Finally, the posterior error covariance matrix is ​​updated. Through the above EKF / UKF prediction-new loop, the HRF parameters can be obtained for each sliding window k. Time delay The corresponding uncertainties (e.g., those from the diagonal elements of the filter covariance matrix) provide the core quantitative basis for subsequent cross-modal signal alignment and temperature regulation through self-supervised learning;

[0069] S24, Dynamic Time Warping (DTW):

[0070] For scenarios involving fast artifact interference or individual deviations in HRF shape, this step, based on HRF recursive estimation, achieves more accurate temporal fine-tuning through local dynamic time warping. The specific process is as follows:

[0071] S241. Low-dimensional feature extraction: Extract low-dimensional time series features (such as PCA principal components, signal energy envelope, etc.) from the EEG and fNIRS segments of the current window, respectively, to reduce computational complexity while preserving core time series information.

[0072] S242. Perform dynamic time warping within a limited window width (e.g., within 10% of the window length) to search for the optimal time matching path between the EEG and fNIRS low-dimensional feature sequences.

[0073] S243. Calculation of Time Delay Correction Term: The time offset corresponding to the optimal matching path is used as the time delay correction term. ;

[0074] S244. Final Delay Update: Combining the delay results estimated by HRF recursion, the corrected final delay is obtained, and the final alignment is performed accordingly; the final delay is calculated using the following formula:

[0075] ;

[0076] In the formula, This is the corrected final time delay. The time delay corresponding to each sliding window k obtained during the recursive estimation in step S23;

[0077] S25. Time alignment and sample pair construction across modal segments:

[0078] Based on the above time delay estimation results (corrected final time delay) ), to complete the temporal matching of cross-modal segments and construct positive sample pairs, including:

[0079] For each data window i, the original EEG fragment With final time delay Corrected fNIRS fragment Perform timeline synchronization, and then... (The sentence is incomplete and requires more context to translate accurately.) With the corrected fNIRS fragment Combined, forming cross-modal positive sample pairs ( , This is used as the standard input for subsequent self-supervised training. During this process, if the alignment uncertainty of the current window (such as time-delay variance or DTW alignment residual) is higher than the preset threshold, the window is marked as a "weak alignment window" to increase redundancy in subsequent self-supervised loss calculation and avoid amplifying training bias due to misalignment.

[0080] To support the subsequent self-supervised temperature adaptive adjustment, this step needs to record and output the alignment uncertainty for each window. Specifically, for each window, at least one of the following indicators should be recorded as the alignment uncertainty (this uncertainty should be output as an independent statistic to provide the necessary basis for the self-supervised temperature adjustment in the next step):

[0081] (1) Variance of time delay estimation (From the diagonal elements corresponding to τ in the EKF / UKF posterior covariance matrix Pk); or,

[0082] (2) Alternative indicators such as DTW alignment residuals and cross-correlation peak width.

[0083] It should be noted that through the above step S2, not only can the "individualized, dynamic HRF parameters" and "time offset" of each EEG-fNIRS fragment be obtained on the client side, realizing explicit correction of the physiological time scale difference between EEG and fNIRS, but also by aligning the output of uncertainty, it can further provide a quantitative basis for subsequent temperature adaptive contrast loss, forming a core technical link that combines physiological prior and self-supervised learning.

[0084] Step S3, Cross-modal self-supervised feature learning (client):

[0085] After completing individualized HRF time-delay estimation and alignment of the EEG-fNIRS signal, the client enters the cross-modal self-supervised feature learning stage. This stage involves constructing a cross-modal contrastive task based on aligned segments, enabling the encoder to learn the joint representation space of EEG-fNIRS without relying on manual annotation. This process includes cross-modal encoder design, projection space construction, positive and negative sample set generation, contrastive loss calculation, and temperature adaptive adjustment. The specific steps are as follows:

[0086] S31, Design of a cross-modal encoder:

[0087] The client contains two independent feature encoding sub-networks, used to process EEG and fNIRS segments respectively. The EEG encoder extracts the time-frequency features of the temporal neural electrical signals, preferably using the following lightweight encoder architecture (taking L=10s as an example): the input is an EEG segment preprocessed by S1 and aligned by S2; the structure adopts a depthwise separable 1D-CNN architecture with the following parameters: 1×64 convolutional kernels (stride 1×8, 32 kernels, GeLU activation), 2 depthwise separable convolutional blocks (1×16 depthwise convolution + 1×1 pointwise convolution + PReLU + BatchNorm1d + Dropout(0.2)), using global average pooling + linear layer output EEG feature embedding. High-dimensional feature vectors (e.g., de=128). The fNIRS encoder uses the S2-corrected fNIRS segment. As input, the structure employs a 1D-CNN + Graph Convolutional Network (GCN) with the following parameters: 1×7 convolutional kernels (1×2 stride, 32 kernels, ReLU activation), and a GCN spatial modeling layer, i.e., a predefined adjacency matrix A (e.g., D) based on the physical arrangement of the fNIRS light poles. −1 / 2 AD −1 / 2 ), perform GCN operation Aggregate spatially adjacent channel information, among which... This represents the updated feature matrix of the channel after passing through one layer of GCN. This represents the normalized adjacency matrix. This represents the input feature matrix. This represents the learnable weight matrix for graph convolution, with ReLU() as the activation function. Global average pooling is used for the output. High-dimensional feature vectors (e.g., dn=128); preferably, the output dimension of the EEG encoder is the same as that of the fNIRS encoder.

[0088] S32, Construction of cross-modal projection space:

[0089] The feature encodings of EEG and fNIRS are mapped to the same latent representation space, specifically by using two projection heads. The projection expression is:

[0090] ;

[0091] This design aligns the two modal representations within a unified contrastive learning space; whereby, The high-dimensional feature vector extracted by the EEG encoder. The high-dimensional feature vector extracted by the fNIRS encoder.

[0092] To be The low-dimensional embedding vector obtained after mapping by the EEG projection head; To be The low-dimensional embedding vector obtained after mapping by the fNIRS projection head is shown in Table i below, where i represents the corresponding fragment index.

[0093] g e (), g n () represents the projection function of the response;

[0094] S33. Construct cross-modal positive and negative sample pairs based on HRF alignment to generate a positive and negative sample set:

[0095] Positive samples: Obtain alignment delay in step S2 After (corrected final delay), EEG fragments within the same window With Corrected fNIRS fragment Form a pair of cross-modal positive samples These are considered as joint observations representing the same neural event. This construction ensures that positive sample pairs satisfy the true physiological coupling relationship, which is key to realizing the self-supervised mechanism;

[0096] Negative samples: Negative sample pairs can come from other samples in this batch, fragments from different subjects, and cross-session / cross-center samples (weighted sampling).

[0097] During the construction process, data augmentation is performed on the fragments with temporal occlusion of ≤10% window length and small time shift of ≤0.5s (to ensure that the HRF timing relationship is not disrupted).

[0098] S34. Adaptive Temperature InfoNCE Loss Calculation: Specifically, cross-modal InfoNCE loss is used to bring positive sample pairs closer together in the representation space and negative sample pairs further apart. The total loss is calculated using the following formula: , This represents the loss in the EEG→fNIRS direction. For the loss in the mirror direction fNIRS→EEG, taking the EEG→fNIRS direction as an example, the loss function is constructed as follows:

[0099] ;

[0100] In the formula, sin() represents the cosine similarity, and T i To adapt to temperature, the denominator includes all negative samples; This indicates summing over all samples j, including both positive and negative samples;

[0101] In this process, to improve robustness under alignment error conditions, a dynamic temperature is assigned to each sample based on the uncertainty of the HRF time delay estimation:

[0102] ;

[0103] Where T0 is the base temperature, and α is the adjustment coefficient. Uncertainty estimates from EKF / UKF or DTW. When the time lag estimate is unstable ( When the temperature T increases, the contrast constraint relaxes, tolerating "false" positive samples and avoiding the damage of "mismatched positive samples" to the model; when the alignment determinism is strong ( When the temperature T decreases (in hours), the constraints are strengthened, thereby improving the robustness of the model. This mechanism enables self-supervised learning to be robust across sessions, devices, and centers.

[0104] Step S4: Complete low-dimensional statistics generation, differential privacy, and uploading on the client side (client-side):

[0105] After completing the cross-modal self-supervised task, the client needs to upload the necessary information to the federated server to support global model updates on the server side. To ensure that the original EEG-fNIRS data and its high-dimensional representation are not leaked, and to significantly reduce the communication load, this embodiment designs a privacy-preserving upload process based on low-dimensional statistics and differential privacy perturbations, which includes the following steps:

[0106] S41. Statistical Generation:

[0107] On the client side, no raw signals, fragment-level embeddings, model gradients, or invertible features are uploaded; only a number of rigorously constrained low-dimensional statistics are uploaded. These low-dimensional statistics can be used to infer the approximate gradient direction of the global contrast target, thereby supporting model updates on the server side. Low-dimensional statistics include, but are not limited to, the following categories:

[0108] 1) Projected embedding mean, which is the expectation calculated for the EEG projected representation and fNIRS projected representation of a batch of samples respectively:

[0109] ;

[0110] In the formula, , E[] represents the EEG projection representation and the fNIRS projection representation, respectively, and E[] represents the expectation value calculation; these means are used for mean alignment constraints on the server side to improve the consistency of the shared representation of the two modalities;

[0111] 2) Sparse cross-correlation matrix:

[0112] To characterize the correspondence between EEG representations and fNIRS representations in the shared space, the following sparse cross-correlation matrix is ​​defined:

[0113] ;

[0114] In the formula, C represents the cross-correlation matrix of EEG and fNIRS projection embedding. Indicates to transpose, Represents a real matrix space of dimension d×d;

[0115] In a preferred embodiment, to prevent excessive parameter dimensionality from leading to privacy leaks and communication overhead, when using the above sparse cross-correlation matrix, only one of the following sparse forms is uploaded: diagonal + m neighboring band structure, random projection compressed cross-correlation matrix, or low-rank approximation (such as the first r feature pairs of SVD). For example, the original 5×5 cross-correlation matrix is:

[0116] ,

[0117] Then, after sparsification using "diagonal + 1 neighboring bands", the corresponding sparse correlation matrix is ​​obtained. :

[0118] ;

[0119] 3) Alignment parameter statistics:

[0120] The low-dimensional statistical information relevant to subsequent tasks extracted from the HRF parameter and time delay estimates in step S2 includes the mean and variance of the HRF parameters, the mean and variance of the time delays, and especially the uncertainty estimates from EKF / UKF or DTW. It is uploaded to the server as an alignment uncertainty evaluation index to adjust the loss weight in the server-side proxy target;

[0121] The general characteristics of low-dimensional statistics are: all statistics are low-dimensional and aggregated; they cannot be inverted back to a single sample; and their dimension is usually O(d) or O(r), which is much smaller than any model gradient or original embedding.

[0122] S42. Differential Privacy (DP):

[0123] To better achieve the objectives of this invention, a differential privacy mechanism is employed to further reduce the sensitivity of statistics to data from a single subject. The differential privacy process includes the following two steps:

[0124] 1) L2 norm pruning: For each statistical vector s to be uploaded, perform:

[0125] ;

[0126] Where s is the statistical vector to be uploaded (such as the projected embedding mean μ). e μ n c is a preset pruning threshold (e.g., 1.0), used to limit the maximum amount of information; is the clipping factor. If the L2 norm of s exceeds the threshold c, then s is reduced proportionally; otherwise, s remains unchanged.

[0127] 2) Gaussian noise injection:

[0128] After cropping, Gaussian noise is added to the statistics to obtain the final uploaded privacy-protected statistics:

[0129] ;

[0130] in, This is the final statistical vector after pruning and noise injection. This indicates that the expression follows a pattern with a mean of 0 and a covariance matrix of... Multivariate normal (Gaussian) distributed noise, noise scale Based on DP budget parameters (Privacy loss cap) and δ (Privacy loss exceeding) The probability of each subject is determined by the sum of its probabilities, which is calculated through cumulative privacy loss; I is an identity matrix with dimensions consistent with the statistical vector s, ensuring that noise is independent in each dimension. This process ensures that the contribution of an individual subject is not discernible in the uploaded statistics, thereby meeting the privacy requirements of multi-center medical scenarios;

[0131] S43, Low-dimensional structure upload:

[0132] During the upload phase, only the low-dimensional statistical structure, after sparsification and aggregation, is uploaded. This ensures that the federated server can complete the global model update while achieving the dual goals of privacy protection and reduced communication load. Examples of the uploaded low-dimensional structure parameters are as follows: , and (The scale is approximately 2–5 days) and (2–10 scalars). Typically, the upload size per client per round is approximately 2.6–3.0 KB, which is 3–4 orders of magnitude lower than the upload model gradient (MB level).

[0133] Step S5, Server Security Aggregation and Global Approximate Update (Server):

[0134] After the client completes the generation of statistics and uploads the differential privacy perturbations, the server enters the global aggregation and model update phase. The design goal of this step is to approximate the global optimization direction for cross-modal contrastive learning using the low-dimensional statistics uploaded by the client, without acquiring any raw EEG-fNIRS data or fragment-level high-dimensional features, thereby achieving secure and effective global encoder updates. The server-side processing flow includes steps such as statistics verification, secure aggregation and summation, surrogate contrast target construction, and global encoder parameter updates, as detailed below:

[0135] S51. Receiving secure aggregation results:

[0136] The server first receives the summation and / or weighted average of the noisy statistics from all clients from the security aggregation module, including:

[0137] Global EEG mean embedding (EEG projection embedding mean μ for all clients) e (weighted average)

[0138] Global fNIRS mean embedding (mean μ of fNIRS projected embedding for all clients) n (weighted average)

[0139] Global cross-modal correlation structure (Weighted average of the sparse cross-correlation matrix C for all clients).

[0140] Global time delay mean (Time-delay weighted average of all clients)

[0141] Global alignment uncertainty (Weighted average of uncertainties across all clients);

[0142] These global alignments are a secure form on the client side after L2 pruning and differential privacy noise injection, from which the server cannot reconstruct any single-end data;

[0143] S52. The server performs the following consistency checks on the aggregate statistics to ensure the stability of subsequent model updates:

[0144] (1) Verify whether the dimensions of each statistic are consistent with the system's preset projection space dimension d (or compression dimension r). , If the dimension deviates from d, then the batch aggregation result is rejected; if If the dimension deviates from d×d (or the compressed dimension after sparsification), the re-aggregation process is triggered;

[0145] (2) Apply amplitude limiting to the aggregate value: , , The element values ​​are restricted to a preset range (e.g., [−5,5]) to avoid numerical overflow caused by noise superposition and to ensure the optimizability of the subsequent objective function;

[0146] (3) An outlier-resistant aggregation method is adopted to suppress malicious / erroneous data that may exist in the aggregation result. For example, the mean is trimmed (the extreme values ​​of the first 5% and the last 5% in the aggregation result are removed and then averaged), the median of the coordinates is taken (the median of the aggregation value of each dimension is taken separately to weaken the impact of outliers on a single client), and the amplitude is clipped (values ​​that exceed the reasonable range are forcibly truncated to the interval boundary to avoid outliers dominating the global result).

[0147] S53. Construct server-side proxy target matching:

[0148] Since the server does not access sample-level embeddings, it cannot directly calculate the InfoNCE loss. To address this technical bottleneck, this step designs a surrogate update mechanism based on low-dimensional statistical matching. Its fitting direction aligns with the optimal solution of InfoNCE, enabling robust model updates even without the original data. Specifically, this includes the following:

[0149] 1) Set a diagonal correlation enhancement term, that is, ensure that EEG and fNIRS representations show a stronger positive correlation on the same dimension through the following formula, thereby improving cross-modal consistency:

[0150] ;

[0151] In the formula, This is the global cross-correlation matrix; Indicates taking The diagonal elements (corresponding to the cross-modal dimensional correlation of the same sample pair); 1 represents an all-1 vector (the expected dimensional correlation of positive sample pairs is close to 1); It is the square of the L2 norm, used to quantify the difference between the diagonal elements and the ideal value;

[0152] 2) Set an off-diagonal suppression term, which reduces redundancy and collinearity between different dimensions and improves the decorrelation capability of the representation through the following formula:

[0153] ;

[0154] In the formula, λ is the adjustment coefficient. express The off-diagonal elements (corresponding to the cross-modal dimensional correlation of different sample pairs); The square of the Frobenius norm is used to quantify the overall strength of off-diagonal elements;

[0155] 3) Set a modal mean alignment term, that is, make the global mean embeddings of EEG and fNIRS consistent through the following formula, thereby improving the overlap of the joint feature distribution:

[0156] ;

[0157] In the formula, λ is the adjustment coefficient (and... (Shared or independent settings) , These are global EEG and fNIRS mean embeddings, respectively. The square of the L2 norm, Used to quantify the difference in mean distribution between EEG and fNIRS modes;

[0158] 4) Weight adjustment based on alignment uncertainty: Specifically, the strength of the surrogate target is dynamically adjusted using the following formula, based on the obtained global time-delay uncertainty, to adapt to fluctuations in alignment quality:

[0159] ;

[0160] In the formula, The global time delay uncertainty is represented by γ, which is a correction coefficient (typically 1-5) used to control the influence of the global time delay uncertainty on the weighting coefficient; ω is the weighting coefficient: when cross-client alignment is unstable ( When the value is larger, ω decreases; when the alignment is stable, ω increases. Setting this parameter helps to avoid low-quality alignment dominating model updates.

[0161] 5) Construct the final proxy loss function on the server side, as shown in the following formula:

[0162] ;

[0163] In the formula, Here, ω represents the final proxy loss function on the server side, and ω is the weight coefficient. For diagonal correlation enhancement terms, This is an off-diagonal suppression term. This is the modal mean alignment term. The advantage of this final proxy loss function is that it makes the server tend to update conservatively during the "cross-client alignment instability" stage, avoiding low-quality alignment from interfering with the model; and when the alignment is stable, it increases the cross-modal consistency constraint and strengthens feature association.

[0164] S54. Based on the final proxy loss function constructed in S53, perform global encoder parameter updates:

[0165] To ensure the global encoder parameter updates are implemented, the server maintains a globally shared set of model parameters. ,in, For EEG encoder parameters, For fNIRS encoder parameters, g e g n These are the projection head parameters corresponding to EEG and fNIRS, respectively;

[0166] The server updates the model using the following steps:

[0167] S541. Calculate the approximate gradient direction:

[0168] For the final proxy loss function Perform backpropagation to obtain the loss function for the globally shared model parameters. gradient (i.e., the approximate direction of parameter updates);

[0169] S542, Gradient Descent Parameter Update

[0170] Use the preset server-side learning rate or server Update parameters in the opposite direction of the gradient:

[0171] ;

[0172] In the formula, or server The typical value is 10. -4 ~10 -3 The step size for controlling parameter updates;

[0173] S543, Parameter stability processing:

[0174] Perform weight pruning or regularization (such as L2 regularization or maximum norm pruning) on ​​the updated parameters to obtain the updated global shared model parameters. This step is used to avoid excessive parameter fluctuations and ensure the stability of the training process.

[0175] The server updates the model using the above method. This process does not involve any original EEG-fNIRS data or sample-level high-dimensional embeddings. The optimization is completed entirely based on low-dimensional statistics uploaded by the client. In this way, the privacy risks of cross-center transmission of original data are avoided, and the core information of cross-modal associations is preserved through the aggregation of statistics. This ensures privacy and security, the effectiveness of model updates, and the stability of the training process in cross-center scenarios.

[0176] S6. Issue the global encoder and proceed to the next round:

[0177] The core task of this step is to distribute the updated globally shared model parameters to the client, initiate the next round of federated training iterations, and continuously optimize the generalization ability of the cross-modal encoder. Specifically, this includes:

[0178] S61. The server will update the global shared model parameters (including EEG encoder parameters). fNIRS encoder parameters Projector head g e / g n The data is packaged into a lightweight parameter package, for example, containing only learnable weights (excluding intermediate training states, such as optimizer momentum), further compressing the amount of data sent out.

[0179] S62. Distribute the encapsulated lightweight parameter packet to all clients participating in federated training via an encrypted communication channel (such as TLS 1.3) to prevent it from being stolen or tampered with during transmission.

[0180] S63. After receiving the lightweight parameter package, the client replaces the parameters of the local encoder with the updated global shared model parameters as the initial model for the next round of training. At the same time, it retains the auxiliary state of local training (such as the negative sample queue of self-supervised contrastive learning and optimizer momentum) to avoid the decrease in convergence efficiency caused by training from scratch.

[0181] S7. The client, using the new global model as the initial value, re-executes steps S1 (data preprocessing and fragment generation) to S6 (server secure aggregation and global approximate update) to enter the next round of federated training loop until the model converges. This step realizes continuous iteration of "global model - local training," gradually improving the generalization ability of the cross-modal encoder to data from different centers through multiple rounds of collaborative optimization, while always maintaining the privacy and communication advantages of "only using traditional econometrics."

[0182] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A federated self-supervised training method for EEG-fNIRS fusion, which is implemented using a system consisting of two layers: a client and a cloud server, characterized in that... The method includes the following steps: Step S1, Data Preprocessing and Fragment Generation: On the client side, the data acquisition and preprocessing module performs normalization processing on the raw EEG signal and fNIRS signal stored locally on the client side to obtain standardized data fragments; Step S2, Individualized Delay Estimation and Alignment: Based on the standardized data fragments obtained in step S1, the client explicitly corrects the differences between the EEG and fNIRS two-modal signals on the time scale through physiological modeling and dynamic estimation, generates physiologically reliable cross-modal positive sample pairs, and outputs the alignment uncertainty. Step S3, Cross-modal self-supervised feature learning: The client constructs a cross-modal contrast task based on aligned segments, enabling the encoder to learn the joint representation space of EEG-fNIRS without relying on manual annotation; Step S4: The client completes the generation of low-dimensional statistics, differential privacy processing, and uploading. After completing cross-modal self-supervised feature learning, the client only extracts low-dimensional statistics that support the server's global update, and uploads them to the federated server after privacy protection processing. Step S5, Server Security Aggregation and Global Approximate Update: After the client completes the generation of statistics and uploads the differential privacy perturbation, the server enters the global aggregation and model update stage. That is, without obtaining any original EEG-fNIRS data and fragment-level high-dimensional features, the low-dimensional statistics uploaded by the client are used to approximately construct the global optimization direction of cross-modal contrastive learning. Step S6: Send the updated globally shared model parameters to the client and start the next round of federated training iteration; Step S7: The client uses the new global model as the initial value and re-executes steps S1 to S6 to enter the next round of federated training until the model converges. Step S2 includes the following sub-steps: S21. Construct a dual-gamma HRF model, the model formula of which is: ； In the formula, h(t) is the hemodynamic response function, representing the standardized blood oxygen signal change generated at time t under unit neural activation; α1, β1, and A1 represent the shape, scale, and amplitude parameters of the positive peak response; α2, β2, and A2 represent the shape, scale, and amplitude parameters of the negative peak response; Γ() represents the gamma function; S22. Extract the driving signal that reflects the intensity of neural activation from the standardized EEG segment output in step S1, and use it as the convolution input of the HRF model to generate the desired fNIRS waveform. S23. On the client side, a state-space model is used in combination with extended Kalman filtering or unscented Kalman filtering to perform recursion and dynamically estimate HRF parameters and time delays. S24. Obtain the corrected final time delay through local dynamic time warping. ; S25, Based on the corrected final time delay Complete the temporal matching of cross-modal segments and construct positive sample pairs; Step S3 includes the following sub-steps: S31. Design of cross-modal encoder: Two independent feature encoding sub-networks are set up on the client side to process EEG and fNIRS segments respectively. The output dimensions of the EEG encoder and the fNIRS encoder are the same. S32. Construction of cross-modal projection space: mapping the feature encodings of EEG and fNIRS to the same latent representation space; S33. Construct cross-modal positive and negative sample pairs based on HRF alignment to generate a positive and negative sample set; S34, Based on adaptive temperature T i Perform InfoNCE loss calculation, where the adaptive temperature T i Dynamic temperature is assigned to each sample based on the uncertainty estimated by HRF time delay.

2. The federated self-supervised training method for EEG-fNIRS fusion as described in claim 1, characterized in that, Adaptive temperature T i The following formula is used to calculate: ； Where T0 is the base temperature and α is the adjustment coefficient. Uncertainty estimates obtained from recursion or dynamic time warping of extended Kalman filtering or unscented Kalman filtering.

3. The federated self-supervised training method for EEG-fNIRS fusion as described in claim 1, characterized in that, Step S4 includes: S41. On the client side, only low-dimensional statistics constructed with strict constraints are uploaded; S42. Further reduce the sensitivity of low-dimensional statistics to single-subject data by using L2 norm pruning and / or Gaussian noise injection. S43. Upload the low-dimensional statistical structure, which has undergone sparsification and aggregation, to the federated server.

4. The federated self-supervised training method for EEG-fNIRS fusion as described in claim 1, characterized in that, Step S5 includes: S51. The server receives the summation / weighted average of the noisy statistics from all clients, including the global EEG mean embedding. Global fNIRS mean embedding Global cross-modal correlation structure Global time delay mean and global alignment uncertainty ; S52. The server performs the following consistency check on the summation / weighted average of the received noisy statistics to ensure the stability of subsequent model updates. S53. Construct the proxy target matching on the server side, including setting diagonal correlation enhancement terms, setting off-diagonal suppression terms, setting modal mean alignment terms, and determining adjustment weight coefficients based on alignment uncertainty, and constructing the final proxy loss function on the server side. S54. Based on the final proxy loss function constructed in S53, the global encoder parameters are updated.

5. A federated self-supervised training method for EEG-fNIRS fusion as described in claim 4, characterized in that, In step S53, the final proxy loss function constructed is: ； In the formula, Here, ω represents the final proxy loss function on the server side, and ω is the weight coefficient. For diagonal correlation enhancement terms, This is an off-diagonal suppression term. This is the modal mean alignment term.

6. A federated self-supervised training method for EEG-fNIRS fusion as described in claim 5, characterized in that, In step S54, to ensure the global encoder parameter update, the server stores a globally shared set of model parameters. ,in, For EEG encoder parameters, For fNIRS encoder parameters, g e g n These are the projection head parameters corresponding to EEG and fNIRS, respectively.

7. A federated self-supervised training method for EEG-fNIRS fusion as described in claim 6, characterized in that, In step S54, the server updates the model through the following steps: S541. Calculate the approximate gradient direction: For the final proxy loss function Perform backpropagation to obtain the loss function for the globally shared model parameters. gradient ; S542. Calculate the approximate gradient direction: Using the preset server-side learning rate η server Update parameters in the opposite direction of the gradient: ； S543. Perform weight pruning or regularization on the updated parameters to obtain the updated global shared model parameters.

Images