A multi-modal biosignal fusion method and system

Abstract

The present application relates to the technical field of multispectral image fusion, and particularly relates to a multi-modal biological signal fusion method and system, multi-modal biological signal acquisition: at least two different types of biological signals are collected; time synchronization processing: a unified time axis T_ref=[0, 10ms, 20ms, …, T_end] with a resolution of 10ms is constructed, and alignment of each modal signal with the unified time axis is realized through a three-level synchronization strategy; cross-device standardization processing: the biological signals after time synchronization are subjected to standardization processing, and characteristic differences among different devices are eliminated; signal quality calculation: the signal quality Q(t) of each modal signal at time t is calculated in real time; dynamic weight updating: based on the real-time quality Q(t) of each modal signal, the fusion weight of each modal is dynamically updated; the present application has the advantages that: through a three-level mixed time synchronization algorithm, a unified time axis is constructed, and precise alignment of multi-modal signals with different sampling rates and different timing characteristics is realized.

Description

Technical Field

[0001] This invention relates to the field of multispectral image fusion technology, specifically to a multimodal biological signal fusion method and system. Background Technology

[0002] With the rapid development of wearable devices and mobile healthcare, biosignal acquisition methods are becoming increasingly diversified and convenient, and the types of biosignals that can be acquired are becoming increasingly rich, mainly including electrocardiogram (ECG), electroencephalogram (EEG), photoplethysmography (PPG), electrical skin analysis (EDA), biochemical / immunological quantitative signals, optical absorption / scattering signals, acceleration / attitude signals, etc. These multimodal biosignals reflect the physiological health status of the human body from different dimensions. Effectively fusing them can improve the accuracy and reliability of health monitoring and disease diagnosis, and is currently a research hotspot in the field of biomedical engineering.

[0003] However, in practical applications, the fusion of multimodal biological signals faces many technical challenges, mainly due to the significant differences between the multimodal signals themselves and the limitations of traditional fusion algorithms, as specifically as follows:

[0004] Significant differences in sampling rates: Different types of biological signals have vastly different sampling rates. For example, ECG has a sampling rate of 256–500 Hz, EEG has a sampling rate of 128–512 Hz, PPG has a sampling rate of 25–100 Hz, and EDA has a sampling rate of 4–10 Hz. Since biochemical / immunoassay signals are non-time-series discrete point signals, this huge difference in sampling rates makes it impossible to directly perform time-series alignment and fusion of signals from different modalities.

[0005] Significant differences exist between devices: Signal acquisition devices from different manufacturers and of different models have significant differences in hardware characteristics, including different LED light source intensities, noise levels, and photodetector response curves. At the same time, the ambient light interference encountered during the acquisition process also varies. These differences make the signal values ​​of the same biological signal acquired on different devices incomparable, which seriously affects the fusion accuracy.

[0006] Lack of a unified time synchronization mechanism: The clock precision of the acquisition devices for different modal signals is different, which leads to differences in the timestamp precision of each modal signal. At the same time, the sampling points of each modal signal cannot be accurately aligned, and some modal signals may also have missing segments. These timing inconsistencies make it impossible to directly fuse multimodal signals.

[0007] Traditional multimodal fusion algorithms have serious limitations: existing fusion algorithms often employ simple feature concatenation, which easily introduces noise and reduces fusion quality; they use fixed weights for fusion, making it impossible to adjust weight allocation according to the dynamic fluctuations in the quality of each modality signal, resulting in poor fusion stability; they ignore the differences in optical characteristics between different devices, leading to poor cross-device consistency of the fusion results; and they lack modeling of deep cross-modal correlations (such as mutual information and temporal correlation), failing to fully explore the complementary information of multimodal signals and resulting in poor fusion performance. To address these limitations, we propose a multimodal biosignal fusion method and system. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a multimodal biological signal fusion method, comprising: S1, multimodal biological signal acquisition: acquiring at least two different types of biological signals;

[0009] S2. Time synchronization processing: Construct a unified timeline with a resolution of 10ms.

[0010] T_ref=[0,10ms,20ms,…,T_end], uses a three-level synchronization strategy to align each modal signal with a unified time axis;

[0011] S3, Cross-device standardization processing: Standardize the time-synchronized biological signals to eliminate characteristic differences between different devices;

[0012] S4. Signal quality calculation: Real-time calculation of the signal quality Q(t) of each modal signal at time t.

[0013] S5. Dynamic weight update: Based on the real-time quality Q(t) of each modal signal, dynamically update the fusion weight α_i(t) of each modality;

[0014] S6. Deep Fusion and Robustness Enhancement: A deep fusion model of Transformer + Cross Attention + Mutual Information Loss (MILoss) is adopted to deeply fuse the signals of each modality after standardization and dynamic weighting; at the same time, missing modalities are filled in through noise modeling, attention-based feature reconstruction, and Mask fusion strategy, thereby improving the fusion robustness.

[0015] S7. Output fusion results: Organize the deeply fused signals and output the final multimodal biological signal fusion results.

[0016] Furthermore, the biosignals include any combination of electrocardiogram (ECG), electroencephalogram (EEG), photoplethysmography (PPG), electrical skin conductance (EDA), optical signals, quantitative immunoassay signals, and motion acceleration signals.

[0017] Furthermore, the three-level synchronization strategy for aligning each modal signal with a unified time axis includes:

[0018] Coarse synchronization: For modal signals with sampling rates lower than the sampling rate corresponding to a unified time axis, linear interpolation is performed. The linear interpolation formula is as follows: ,in , For two adjacent original sampling times of the target time t, , They are respectively , The original signal value at that moment;

[0019] Fine synchronization: Local sequence alignment is performed using the Dynamic Time Warping (DTW) algorithm combined with a bandwidth-constrained strategy. The cost matrix of the DTW algorithm is as follows: The cumulative cost matrix is The bandwidth is limited by Sakoe–Chiba constraints, with a constraint range of ±12%, and no more than 2 jumps are allowed.

[0020] Micro-synchronization: The time error in the DTW backtracking path is corrected by local quadratic interpolation. The quadratic interpolation formula is x(t)=at²+bt+c, where a, b, and c are calculated based on the three nearest sampling points to the target time t.

[0021] Furthermore, the formula for calculating the signal quality Q(t) is as follows:

[0022] Where SNR(t) is the signal-to-noise ratio at time t, R(t) is the signal correlation at time t, and C(t) is the signal continuity at time t. , , The weighting coefficients for SNR(t), R(t), and C(t) are respectively, and .

[0023] Furthermore, the formula for the fusion weight α_i(t) is:

[0024] α_i(t) = Q_i(t) / ∑Q_j(t), where Q_i(t) is the signal quality of the i-th mode at time t, and ∑Q_j(t) is the sum of the signal qualities of all modes at time t.

[0025] Furthermore, the LED light source intensity normalization adopts the peak normalization method to uniformly normalize the LED light source intensity of different devices to a preset standard intensity range; the dark current / ambient light compensation adopts the dark box calibration method, which collects the signal baseline in a dark environment and subtracts the dark current and ambient light interference components; the three-wavelength response curve calibration adopts the standard sample calibration method, which uniformly corrects the photodetector response curves of different devices based on the optical response curve of a preset standard sample.

[0026] A multimodal biological signal fusion system, comprising:

[0027] Signal acquisition module: used to acquire at least two different types of multimodal biological signals and transmit the acquired signals to the time synchronization module;

[0028] Time synchronization module: Connected to the signal acquisition module, it is used to construct a unified time axis and achieve the alignment of each modal signal with the unified time axis through a three-level synchronization strategy of coarse synchronization, fine synchronization and micro synchronization, and transmit the synchronized signal to the cross-device standardization module.

[0029] Cross-device standardization module: Connected to the time synchronization module, it is used to normalize the LED light source intensity, compensate for dark current / ambient light, and calibrate the three-wavelength response curve of the synchronized signal, eliminating the characteristic differences between different devices, and transmitting the standardized signal to the signal quality calculation module and the deep fusion module.

[0030] Signal quality calculation module: Connected to the cross-device standardization module, it is used to calculate the real-time quality Q(t) of each modal signal in real time and transmit the calculation results to the dynamic weight update module;

[0031] Dynamic weight update module: connected to the signal quality calculation module, used to dynamically update the fusion weight α_i(t) of each mode based on the real-time quality Q(t) of each mode signal, and transmit the weight information to the deep fusion module;

[0032] Deep fusion module: It is connected to the cross-device standardization module and the dynamic weight update module respectively. It adopts the deep fusion model of Transformer+Cross Attention+MI Loss, combined with noise modeling and missing modality completion mechanism, to perform deep fusion of standardized signal and dynamic weight, and transmit the fusion result to the result output module.

[0033] Results output module: Connected to the deep fusion module, it is used to organize and format the fused signals and output the final multimodal biological signal fusion results.

[0034] Furthermore, the signal acquisition module includes multiple signal acquisition units, each corresponding to a type of biological signal. Each signal acquisition unit includes a sensor and a signal preprocessing subunit, which is used to filter and denoise the raw signal acquired by the sensor.

[0035] The time synchronization module, cross-device standardization module, signal quality calculation module, dynamic weight update module, and deep fusion module are all integrated into the processor. The processor is an embedded processor that supports device-side deployment and can adapt to the hardware requirements of wearable devices and smart health monitoring devices.

[0036] The beneficial effects of this invention are reflected in:

[0037] By employing a three-level hybrid time synchronization algorithm, a unified time axis is constructed, achieving precise alignment of multimodal signals with different sampling rates and temporal characteristics. The alignment error is controlled within 10ms, providing a reliable foundation for subsequent fusion. Through a cross-device standardization module, the differences in optical and noise characteristics between different devices are eliminated, ensuring that signals collected by different devices have a unified standard. The dynamic weight self-adjustment mechanism based on signal quality can adapt to fluctuations in the quality of each modal signal in real time, avoiding significant interference from poor-quality modes in the fusion results. This is suitable for long-term health monitoring in complex scenarios. The deep fusion model with mutual information constraints can fully explore the deep correlations of multimodal signals. Combined with noise modeling and missing mode completion mechanisms, it not only improves fusion accuracy but also solves the problem of insufficient fusion robustness in the case of missing modes, adapting to the application needs of different users and different scenarios. Attached Figure Description

[0038] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. The elements or parts in the drawings are not necessarily drawn to scale.

[0039] Figure 1 This is a schematic flowchart of the method of the present invention;

[0040] Figure 2 This is a schematic diagram of the system flow of the present invention. Detailed Implementation

[0041] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0042] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0043] Step 1: Multimodal biosignal acquisition

[0044] At least two different types of biological signals are collected, covering any combination of electrocardiogram (ECG), electroencephalogram (EEG), photoplethysmography (PPG), electrical conductance analysis (EDA), optical absorption / scattering signals, quantitative immunoassay signals, and motion acceleration signals (ACC). The sampling rate range of each modality strictly follows industry standards, as follows: ECG 256–500Hz, EEG 128–512Hz, PPG 25–100Hz, EDA 4–10Hz, Motion (ACC) 25–100Hz, and quantitative immunoassay signals 0–1Hz (non-time-series discrete point signals).

[0045] During the acquisition process, the raw signal is acquired through the corresponding sensor. At the same time, the raw signal is pre-processed by filtering and noise reduction to remove high-frequency interference and baseline drift, ensuring the validity of the raw signal. The pre-processed signal is then transmitted to the subsequent time synchronization module.

[0046] Step 2: Time Synchronization Processing

[0047] The core objective of this step is to construct a unified timeline, achieve precise timing alignment of signals across different modes, and resolve issues related to sampling rate differences and timing inconsistencies. This is specifically divided into three stages: coarse synchronization, fine synchronization, and micro synchronization. Figure 2 (Flowchart of Hybrid Synchronization Algorithm) Detailed Explanation:

[0048] Phase 1: Building a Unified Timeline

[0049] A unified time axis with a resolution of Δt = 10ms is constructed, with the time axis expression T_ref = [0, 10ms, 20ms, ..., T_end], where T_end is the end time of signal acquisition. The construction of the unified time axis aims to provide a consistent time reference for all modal signals, ensuring the consistency of subsequent fusion calculations. The 10ms resolution balances fusion accuracy and computational efficiency, adapting to the deployment requirements of the device.

[0050] Phase Two: Coarse Synchronization

[0051] Linear interpolation is performed on modal signals (mainly including PPG, EDA, immunoquantitative signals, and some motion signals) with sampling rates lower than the corresponding sampling rate (100Hz) of the unified time axis to fill the sampling point gaps and make the sampling frequency of each modal signal initially match the unified time axis.

[0052] The linear interpolation formula is: ;

[0053] Where t is the target time on the unified time axis. , For the two raw sampling times adjacent to the target time t , , They are respectively , The original signal value at time t, and x(t) is the interpolation result at time t. Linear interpolation has the advantages of low computational cost and strong real-time performance, and is suitable for fast alignment in the coarse synchronization stage.

[0054] Phase 3: Precise Synchronization

[0055] After coarse synchronization, the sampling frequencies of each modal signal have been matched with a unified time axis, but there are still slight timing deviations (such as sampling point offsets caused by device clock deviations). In this stage, the Dynamic Time Warping (DTW) algorithm combined with a bandwidth limiting strategy is used to achieve accurate alignment of local sequences and avoid global distortion.

[0056] Specific implementation: First, calculate the cost matrix and cumulative cost matrix of two modal signals (ECG as the reference mode and other modes as the modes to be aligned). The cost matrix is ​​used to measure the similarity between two sampling points, and the cumulative cost matrix is ​​used to find the optimal alignment path.

[0057] Cost matrix formula: , where x_A(i) is the signal value of the i-th sampling point of the reference mode (ECG), x_B(j) is the signal value of the j-th sampling point of the mode to be aligned, and D(i,j) is the absolute value of the difference between the two sampling points. The smaller the difference, the higher the similarity.

[0058] Cumulative cost matrix formula: Where C(i,j) is the cumulative cost. This indicates that the minimum value of the previous cumulative cost is selected to ensure the optimality of the alignment path.

[0059] To avoid global distortion in the DTW algorithm, a Sakoe-Chiba bandwidth limiting strategy is adopted, with a constraint range of ±12%, meaning the alignment path can only fluctuate within a 12% range on both sides of the main diagonal. At the same time, a jump constraint is set, which does not allow jumps to exceed 2 points, to ensure the rationality of the alignment path and the consistency of timing.

[0060] Phase Four: Micro-Synchronization

[0061] After fine synchronization, the timing deviation of each mode signal has been greatly reduced, but there may still be a small time error in the DTW backtracking path (usually within 10ms). In this stage, the alignment accuracy is further improved by local quadratic interpolation correction.

[0062] The quadratic interpolation formula is: x(t) = at² + bt + c, where a, b, and c are interpolation coefficients, calculated based on the signal values ​​of the three nearest sampling points (t-10ms, t, t+10ms) at the target time t. Quadratic interpolation can better fit the signal's changing trend, and compared to linear interpolation, it has higher correction accuracy, controlling timing deviations within 1ms.

[0063] At this point, the precise alignment of all multimodal signals with the unified time axis is complete, and the timing inconsistency problem is completely resolved.

[0064] Step 3: Cross-device standardization processing

[0065] To address the differences in optical and noise characteristics of signals acquired by different devices, this step uses a cross-device standardization system to standardize the time-synchronized signals, eliminating system errors between devices and making the same type of signals acquired by different devices comparable. This process includes three sub-steps:

[0066] LED light source intensity normalization

[0067] For signals acquired using LED light sources (such as PPG and optical absorption / scattering signals), the intensity of the LED light sources varies between different devices, resulting in significant differences in signal amplitude. A maximum-minimum normalization method is used to map the LED light source intensities of different devices to the [0,1] interval. The normalization formula is as follows: Where I is the original light source intensity, I_min is the minimum light source intensity of the device, I_max is the maximum light source intensity of the device, and I_norm is the normalized light source intensity.

[0068] Dark current / ambient light compensation

[0069] During signal acquisition, the dark current (output current when there is no light source) of the photodetector and ambient light can interfere with the signal, causing signal baseline shift. A background subtraction algorithm is used to acquire the dark current and ambient light interference signals (background signal) when there is no signal input. The actual acquired signal is subtracted from the background signal to compensate for the dark current and ambient light interference. The compensation formula is: S_comp = S_actual − S_background, where S_actual is the actual acquired signal, S_background is the background signal, and S_comp is the compensated signal.

[0070] Three-wavelength response curve calibration

[0071] The response curves of photodetectors from different devices differ for light of different wavelengths, leading to inconsistent acquisition results for the same optical signal on different devices. A polynomial fitting method is used, with the response curve of a standard photodetector as a benchmark, to calibrate the response curves of the photodetectors from each device. The fitting polynomial is as follows: Where λ is the wavelength of the light and y is the detector response value. The fitting coefficients are obtained through calibration using standard samples. After calibration, the deviation between the detector response curves of each device and the standard curve is controlled within 5%.

[0072] Step 4: Real-time calculation of signal quality

[0073] To achieve dynamic weight adjustment, it is necessary to calculate the signal quality Q(t) of each modal signal at each time t in real time. The signal quality Q(t) comprehensively considers three indicators: signal-to-noise ratio (SNR), signal correlation (R), and signal continuity (C), and is calculated using a weighted summation method, as follows:

[0074] Calculation formula: ;

[0075] Where: SNR(t): Signal-to-noise ratio at time t, reflecting the ratio of effective signal to noise in the signal, SNR(t) = 10lg(P_signal / P_noise), where P_signal is the effective signal power and P_noise is the noise power. The larger the SNR(t), the better the signal quality.

[0076] R(t): Signal correlation at time t, reflecting the correlation between the current sampling point and its neighboring sampling points. It is calculated using the Pearson correlation coefficient. The value range of R(t) is [-1, 1]. The closer the absolute value is to 1, the stronger the signal correlation and the better the quality.

[0077] C(t): Signal continuity at time t, reflecting whether there are any anomalies such as missing signals or sudden changes. Where N_missing is the number of missing sampling points in the adjacent window at the current time, and N_total is the total number of sampling points in the window. The closer C(t) is to 1, the better the signal continuity.

[0078] , , : The weighting coefficients of SNR(t), R(t), and C(t), respectively, satisfying The weighting coefficients can be dynamically adjusted according to the actual application scenario; the default value is [value to be filled in]. , , Prioritize ensuring the impact on signal-to-noise ratio.

[0079] Step 5: Dynamically update the fusion weights

[0080] Based on the real-time quality Q(t) of each modal signal calculated in step 4, the fusion weight α_i(t) of each modal signal is dynamically updated, following the principle that "the higher the signal quality, the greater the fusion weight," to ensure that the mode with better signal quality plays a greater role in the fusion process. The weight update formula is as follows:

[0081] α_i(t) = Q_i(t) / ∑Q_j(t)

[0082] Where α_i(t) is the fusion weight of the i-th modal signal at time t, Q_i(t) is the signal quality of the i-th modal signal at time t, and ∑Q_j(t) is the sum of the signal qualities of all modal signals participating in the fusion at time t. This formula ensures that the sum of the fusion weights of all modal signals is 1, guaranteeing the rationality of the fusion calculation.

[0083] For example, if the modes currently participating in the fusion are ECG, PPG, and EDA, their signal qualities are respectively , , The corresponding fusion weights are respectively , , The ECG signal, being of the highest quality, receives the highest fusion weight.

[0084] Step 6: Deep Fusion and Robustness Enhancement

[0085] This step is the core of the fusion process. It employs a deep fusion model of Transformer + Cross Attention + MI Loss to deeply fuse the standardized and dynamically weighted signals of each modality. Simultaneously, it enhances the robustness of the fusion through noise modeling and missing modality completion mechanisms. The specific implementation is as follows:

[0086] Deep fusion model construction

[0087] The deep fusion model consists of three sub-modules: feature extraction, cross-modal attention interaction, and mutual information constraint fusion.

[0088] Feature extraction submodule: Uses convolutional neural network (CNN) to extract features from each modality of signal, converting one-dimensional signal into high-dimensional feature vector. The extracted features include time-domain features (such as peaks, valleys, and periods) and frequency-domain features (such as spectral peaks and frequency distribution), ensuring the comprehensiveness of features;

[0089] Cross-modal attention interaction submodule: Employs the Cross Attention mechanism to implement weight allocation for interactions between features of different modalities, enabling the model to focus on key information in each modal feature and strengthen the coupling between modalities. Specifically, it uses a feature of one modality as a query and a feature of another modality as a key and value, calculates attention weights, and realizes the interaction and fusion of features.

[0090] The mutual information constraint fusion submodule introduces mutual information loss (MI Loss) to maximize the mutual information between features of different modalities. This ensures that the fused features fully retain the complementary information of each modality, reduces information redundancy, and improves fusion accuracy. The mutual information loss is calculated based on the probability distributions of the two modal features, using KL divergence to measure the distribution difference and achieve the constraint of maximizing mutual information.

[0091] Noise modeling

[0092] To address interference such as Gaussian noise and impulse noise during signal acquisition, a Gaussian Mixture Model (GMM) is used to model the noise. The distribution parameters (mean and variance) of the noise are estimated using the EM algorithm. The noise model is then integrated into the training process of the deep fusion model, enabling the model to adaptively identify and suppress noise, thereby reducing the impact of noise on the fusion results.

[0093] Missing modal completion

[0094] When a certain modal signal has missing segments (such as sampling interruption caused by equipment failure or signal occlusion), an attention-based feature reconstruction and mask fusion strategy is used to complete the missing modal.

[0095] Feature reconstruction: Based on the features of other normal modalities, the association between normal modalities and missing modalities is mined through an attention mechanism to reconstruct the feature vector of the missing modalities;

[0096] Mask fusion strategy: Assign lower confidence weights (Mask weights) to the reconstructed missing modal features and higher confidence weights to the normal modal features, integrating both into the fusion process to avoid reconstruction errors causing significant interference to the overall fusion result.

[0097] Step 7: Output the fusion result

[0098] The deeply fused feature vectors are decoded and converted into interpretable biosignal fusion results. The fusion results are then organized and formatted (e.g., arranged according to a uniform time axis and labeled with the fusion amplitude at each moment). Finally, they are output to smart terminals, cloud-based health analysis systems, medical auxiliary diagnostic devices, etc., via wired or wireless means for subsequent health assessment and disease diagnosis.

[0099] A multimodal biological signal fusion system,

[0100] This system is used to execute the aforementioned quantitative algorithm, integrating signal acquisition, processing, analysis, and result output. It includes a multispectral acquisition module, an image acquisition module, a fusion processing module, and a display module. Additional modules such as a temperature sensor, data transmission module, cloud server, and power supply module can be added as needed. The structure and function of each module are as follows:

[0101] Signal acquisition module

[0102] Function: Used to collect at least two different types of multimodal biological signals and perform preliminary preprocessing on the raw signals.

[0103] Structure: It includes multiple signal acquisition units, each corresponding to a type of biological signal. Each signal acquisition unit consists of a sensor and a signal preprocessing subunit.

[0104] Sensors: Select the appropriate type based on the type of biological signal being acquired, such as ECG sensor, EEG sensor, PPG sensor, EDA sensor, accelerometer, immunoassay sensor, etc., to ensure the accuracy of signal acquisition;

[0105] Signal preprocessing subunit: Using filtering circuits and simple algorithms, the raw signals acquired by the sensors are filtered, denoised, and baseline corrected to remove high-frequency interference and baseline drift, ensuring the validity of the raw signals. The preprocessed signals are then transmitted to the time synchronization module.

[0106] Time synchronization module

[0107] Function: Construct a unified time axis and achieve precise alignment of each modal signal with the unified time axis through a three-level strategy of coarse synchronization, fine synchronization and micro synchronization.

[0108] Structure: Integrated into the processor, it includes a unified time axis construction subunit, a coarse synchronization subunit, a fine synchronization subunit, and a micro synchronization subunit, which correspond to the four stages of time synchronization processing. Each subunit cooperates with the other to complete timing alignment, and the aligned signal is transmitted to the cross-device standardization module.

[0109] Cross-device standardized modules

[0110] Function: Eliminates differences in optical and noise characteristics between different devices, making signals collected by different devices comparable.

[0111] Structure: Integrated into the processor, it includes an LED light source intensity normalization subunit, a dark current / ambient light compensation subunit, and a three-wavelength response curve calibration subunit, which correspond to the three sub-steps of cross-device standardization processing, respectively. The standardized signal is then transmitted to the signal quality calculation module and the deep fusion module.

[0112] Signal quality calculation module

[0113] Function: Calculates the real-time quality Q(t) of each modal signal in real time, providing data support for dynamic weight updates.

[0114] Structure: Integrated into the processor, it includes a signal-to-noise ratio calculation subunit, a correlation calculation subunit, a continuity calculation subunit, and a weighted summation subunit, which calculate SNR(t), R(t), and C(t) respectively, and obtain Q(t) through weighted summation. The calculation results are transmitted to the dynamic weight update module.

[0115] Dynamic weight update module

[0116] Function: Based on the real-time quality Q(t) of each modal signal, dynamically update the fusion weight α_i(t) of each modality.

[0117] Structure: Integrated into the processor, it includes a weight calculation subunit and a weight update subunit. The weight calculation subunit calculates the fusion weight of each mode according to α_i(t)=Q_i(t) / ∑Q_j(t). The weight update subunit updates the weight information in real time and transmits the weight information to the deep fusion module.

[0118] Deep fusion module

[0119] Function: Employs a deep fusion model to deeply fuse standardized signals and dynamic weights, combining noise modeling and missing mode completion mechanisms to improve fusion accuracy and robustness.

[0120] Structure: Integrated into the processor, it includes a feature extraction submodule, a cross-modal attention interaction submodule, a mutual information constraint fusion submodule, a noise modeling submodule, and a missing modality completion submodule. Each submodule works together to complete deep fusion, and the fusion result is transmitted to the result output module.

[0121] Result Output Module

[0122] Function: Organize and format the fused signals and output them to the target device or system.

[0123] Structure: It includes a signal decoding subunit, a formatting subunit, and a transmission subunit. The signal decoding subunit decodes the fused feature vector into an interpretable biological signal. The formatting subunit formats the signal. The transmission subunit transmits the fusion result to a smart terminal, a cloud-based health analysis system, or a medical auxiliary diagnostic device via wired (e.g., USB) or wireless (e.g., Bluetooth, Wi-Fi, NFC) methods.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method for multimodal biological signal fusion, characterized in that, include, S1. Multimodal biosignal acquisition: Acquire at least two different types of biosignals; S2. Time synchronization processing: Construct a unified timeline with a resolution of 10ms. T_ref=[0,10ms,20ms,…,T_end], uses a three-level synchronization strategy to align each modal signal with a unified time axis; S3, Cross-device standardization processing: Standardize the time-synchronized biological signals to eliminate characteristic differences between different devices; S4. Signal quality calculation: Real-time calculation of the signal quality Q(t) of each modal signal at time t. S5. Dynamic weight update: Based on the real-time quality Q(t) of each modal signal, dynamically update the fusion weight α_i(t) of each modality; S6. Deep Fusion and Robustness Enhancement: A deep fusion model of Transformer + Cross Attention + Mutual Information Loss (MILoss) is adopted to deeply fuse the signals of each modality after standardization and dynamic weighting; at the same time, missing modalities are filled in through noise modeling, attention-based feature reconstruction, and Mask fusion strategy, thereby improving the fusion robustness. S7. Output fusion results: Organize the deeply fused signals and output the final multimodal biological signal fusion results.

2. The multimodal biological signal fusion method according to claim 1, characterized in that: The biosignals include any combination of electrocardiogram (ECG), electroencephalogram (EEG), photoplethysmography (PPG), electrical skin conductance (EDA), optical signals, quantitative immunoassay signals, and motion acceleration signals.

3. The multimodal biological signal fusion method according to claim 1, Its features are: The three-level synchronization strategy achieves alignment of each modal signal with a unified time axis by including: Coarse synchronization: For modal signals with sampling rates lower than the sampling rate corresponding to a unified time axis, linear interpolation is performed. The linear interpolation formula is as follows: ,in , For two adjacent original sampling times of the target time t, , They are respectively , The original signal value at that moment; Fine synchronization: Local sequence alignment is performed using the Dynamic Time Warping (DTW) algorithm combined with a bandwidth-constrained strategy. The cost matrix of the DTW algorithm is as follows: The cumulative cost matrix is The bandwidth is limited by Sakoe–Chiba constraints, with a constraint range of ±12%, and no more than 2 jumps are allowed. Micro-synchronization: The time error in the DTW backtracking path is corrected by local quadratic interpolation. The quadratic interpolation formula is x(t)=at²+bt+c, where a, b, and c are calculated based on the three nearest sampling points to the target time t.

4. The multimodal biological signal fusion method according to claim 1, characterized in that: The formula for calculating the signal quality Q(t) is: Where SNR(t) is the signal-to-noise ratio at time t, R(t) is the signal correlation at time t, and C(t) is the signal continuity at time t. , , The weighting coefficients for SNR(t), R(t), and C(t) are respectively, and .

5. The multimodal biological signal fusion method according to claim 1, characterized in that: The formula for the fusion weight α_i(t) is: α_i(t) = Q_i(t) / ∑Q_j(t), where Q_i(t) is the signal quality of the i-th mode at time t, and ∑Q_j(t) is the sum of the signal qualities of all modes at time t.

6. The multimodal biological signal fusion method according to claim 1, characterized in that: The LED light source intensity normalization adopts the peak normalization method, which uniformly normalizes the LED light source intensity of different devices to a preset standard intensity range; the dark current / ambient light compensation adopts the dark box calibration method, which collects the signal baseline in a dark environment and deducts the dark current and ambient light interference components; the three-wavelength response curve calibration adopts the standard sample calibration method, which uniformly corrects the photodetector response curves of different devices based on the optical response curve of a preset standard sample.

7. A multimodal biological signal fusion system, based on the multimodal biological signal fusion method described in claims 1-6, characterized in that: include: Signal acquisition module: used to acquire at least two different types of multimodal biological signals and transmit the acquired signals to the time synchronization module; Time synchronization module: Connected to the signal acquisition module, it is used to construct a unified time axis and achieve the alignment of each modal signal with the unified time axis through a three-level synchronization strategy of coarse synchronization, fine synchronization and micro synchronization, and transmit the synchronized signal to the cross-device standardization module. Cross-device standardization module: Connected to the time synchronization module, it is used to normalize the LED light source intensity, compensate for dark current / ambient light, and calibrate the three-wavelength response curve of the synchronized signal, eliminating the characteristic differences between different devices, and transmitting the standardized signal to the signal quality calculation module and the deep fusion module. Signal quality calculation module: Connected to the cross-device standardization module, it is used to calculate the real-time quality Q(t) of each modal signal and transmit the calculation results to the dynamic weight update module; Dynamic weight update module: connected to the signal quality calculation module, used to dynamically update the fusion weight α_i(t) of each mode based on the real-time quality Q(t) of each mode signal, and transmit the weight information to the deep fusion module; Deep fusion module: It is connected to the cross-device standardization module and the dynamic weight update module respectively. It adopts the deep fusion model of Transformer+Cross Attention+MI Loss, combined with noise modeling and missing modality completion mechanism, to perform deep fusion of standardized signal and dynamic weight, and transmit the fusion result to the result output module. Results output module: Connected to the deep fusion module, it is used to organize and format the fused signals and output the final multimodal biological signal fusion results.

8. A multimodal biosignal fusion system according to claim 7, characterized in that: The signal acquisition module includes multiple signal acquisition units, each corresponding to a type of biological signal. Each signal acquisition unit includes a sensor and a signal preprocessing subunit. The signal preprocessing subunit is used to filter and denoise the raw signal acquired by the sensor. The time synchronization module, cross-device standardization module, signal quality calculation module, dynamic weight update module, and deep fusion module are all integrated into the processor. The processor is an embedded processor that supports device-side deployment and can adapt to the hardware requirements of wearable devices and smart health monitoring devices.

Images