A motion-guided electroencephalogram-electrocardiogram dynamic fusion method and system

Abstract

The present application relates to the technical fields of biomedical signal processing, multi-modal physiological signal fusion and intelligent health monitoring, and particularly relates to a motion-guided ear-brain electroencephalogram-electrocardiogram dynamic fusion method and system. The present application performs time synchronization and preprocessing on ear-brain electroencephalogram signals, electrocardiogram signals and motion signals, extracts modal features from the ear-brain electroencephalogram signals and the electrocardiogram signals respectively to obtain ear-brain electroencephalogram features and electrocardiogram features; extracts motion intensity features according to the motion signals, generates dynamic gating weights in combination with signal quality features, performs dynamic fusion on the ear-brain electroencephalogram features and the electrocardiogram features based on the dynamic gating weights to obtain fused physiological features; and inputs the fused physiological features into a downstream prediction or recognition model to obtain a target physiological state, disease risk or early warning probability. The present application promotes the motion signals from an auxiliary input to a fusion control variable, thereby improving the stability of multi-modal physiological signal recognition and prediction.

Description

Technical Field

[0001] This invention relates to the fields of biomedical signal processing, multimodal physiological signal fusion, and intelligent health monitoring, and particularly to a motion-guided dynamic fusion method and system for ear-brain-electrocardiogram. Background Technology

[0002] EEG (electroencephalography) is a brainwave acquisition method suitable for long-term wear and daily monitoring. It can acquire electrophysiological signals related to brain activity at or around the ear, offering advantages over traditional scalp multichannel EEG, such as discreet wearing, high comfort, and convenient deployment. ECG signals can reflect physiological changes such as autonomic nervous activity and heart rate variability, and have complementary value to EEG signals in tasks such as epilepsy prediction, fatigue detection, sleep state recognition, and emotional state assessment.

[0003] However, EEG signals have lower amplitude and fewer channels, and are easily affected by motion artifacts, changes in electrode contact, electromyography (EMG), and environmental noise. The reliability of EEG signals decreases significantly when the subject is walking, shaking their head, or wearing loose earpieces; conversely, EEG can provide more direct information about neural activity at rest or during minimal movement. ECG signals exhibit higher stability than EEG signals in certain motion scenarios, but their ability to directly represent abnormal brain activity is limited.

[0004] Existing multimodal physiological signal fusion methods typically employ feature splicing, fixed-weight averaging, or simple attentional fusion. These methods do not adequately consider the dynamic impact of motion on the reliability of EEG signals and struggle to adaptively adjust the contributions of EEG and ECG under different time windows, individual conditions, and motion intensities. When motion artifacts are enhanced, fixed fusion strategies may still overly rely on contaminated EEG features; conversely, when motion is stable, they may not fully utilize the effective neural information in EEG signals.

[0005] Therefore, there is an urgent need for a fusion method that can dynamically adjust the features of ear, brain, and heart using motion signals, so that the model can adaptively select more reliable modal information based on the current motion state and signal quality, thereby improving the robustness and interpretability of multimodal prediction or recognition tasks in wearable long-term monitoring scenarios. Summary of the Invention

[0006] The problem this invention aims to solve is to overcome the shortcomings of the prior art and provide a motion-guided dynamic fusion method and system for ear-brain-electrocardiography.

[0007] The first aspect of this invention provides a motion-guided dynamic fusion method of ear-brain-electrocardiogram, comprising the following steps: S1: Acquire synchronously collected EEG, ECG, and motion signals, and synchronize the EEG, ECG, and motion signals in time; S2: Preprocess the EEG, ECG, and motion signals to remove noise and baseline drift, and construct multimodal time window samples according to a fixed time window; S3: Modal features are extracted from the electroencephalogram (EEG) and electrocardiogram (ECG) signals in the multimodal time window samples to obtain the EEG feature Xeeg and the ECG feature Xecg. S4: Extract motion intensity features from motion signals in multimodal time window samples, map the motion intensity features to a motion embedding vector Xm, and generate dynamic gating weights through the motion embedding vector Xm: alpha=sigmoid(W·Xm+b), where W and b are learnable parameters, and sigmoid is the Sigmoid activation function; or extract motion intensity features from motion signals in multimodal time window samples and standardize them, extract signal quality features from EEG and ECG signals respectively, concatenate the standardized motion intensity features and signal quality features into a gating input vector g, and input the gating input vector g into a multilayer perceptron to obtain dynamic gating weights; S5: Dynamically fuse the electroencephalogram (EEG) feature Xeeg and electrocardiogram (ECG) feature Xecg based on dynamic gating weights to obtain the fused physiological feature: Xfused=(1-alpha)·Xeeg+alpha·Xecg; or concatenate (1-alpha)·Xeeg, alpha·Xecg, and alpha and input them into a linear layer or multilayer perceptron to obtain the fused physiological feature. S6: Input the fused physiological features into the downstream prediction or identification model to obtain the target physiological state, disease risk, or early warning probability.

[0008] Furthermore, the dynamic gating weight can be a time window level weight, a feature dimension level weight, or a channel level weight; when the dynamic gating weight is a feature dimension level weight, its dimension is the same as that of the EEG and ECG features.

[0009] Furthermore, in S4, the motion embedding vector Xm is constructed as follows: S41: Within each fixed time window, read the triaxial acceleration signal and the triaxial angular velocity signal, calculate the composite acceleration magnitude and the composite angular velocity magnitude respectively, and then calculate the statistics of the composite acceleration magnitude and the composite angular velocity magnitude within the same time window. Concatenate the statistics in a fixed order to obtain the original motion feature vector F_m. Standardize the original motion feature vector F_m to obtain the normalized motion feature vector F'_m. S42: Directly use the normalized motion feature vector F'_m as the motion embedding vector Xm, or input F'_m into the motion embedding layer to obtain the motion embedding vector Xm, Xm=ReLU(W_m·F'_m+b_m), where W_m and b_m are the learnable parameters of the motion embedding layer, and ReLU is a non-linear activation function.

[0010] Furthermore, in step S4, the learnable parameters W, b and / or W_m, b_m are all learned through backpropagation during model training. During model training, the obtained fused physiological features are input into the downstream prediction or recognition model to obtain the prediction result. The task loss between the prediction result and the real label is calculated, and then backpropagation is performed based on the task loss to update the learnable parameters. Each time the learnable parameters are updated, the optimizer calculates the gradient of W, b and / or W_m, b_m based on the task loss, and updates the learnable parameters according to the gradient descent or Adam optimization algorithm.

[0011] Furthermore, a gated smoothing regularization term and / or a gated prior constraint term are added to the task loss during model training, and a modal random deactivation strategy is adopted during model training.

[0012] Furthermore, in step S6, the downstream prediction or identification model is a state-space model, a recurrent neural network, a convolutional neural network, a Transformer model, a classifier, or a regressor.

[0013] A second aspect of the present invention provides a motion-guided dynamic fusion system of ear-brain-electrocardiogram, comprising: The multimodal acquisition module is used to acquire synchronously acquired electroencephalogram (EEG), electrocardiogram (ECG), and motion signals, and to synchronize the EEG, ECG, and motion signals in time. The preprocessing module is used to preprocess the EEG, ECG, and motion signals. It performs bandpass filtering, notch filtering, and standardization on the EEG and ECG signals, detrending or amplitude normalization on the motion signals, and divides each modal signal into fixed time windows.

[0014] The modal feature extraction module is used to extract ear-brain electroencephalogram (EEG) features and electrocardiogram (ECG) features from ear-brain electroencephalogram (EEG) signals and ECG signals, respectively.

[0015] The motion gating generation module is used to generate dynamic gating weights based on motion intensity characteristics and / or signal quality characteristics.

[0016] The dynamic fusion module is used to dynamically fuse ear, brain, and electrocardiogram features based on dynamic gating weights to obtain fused physiological features.

[0017] The task output module is used to output the target's physiological state, disease risk, or early warning probability based on the fused physiological characteristics.

[0018] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the motion-guided dynamic fusion method of electroencephalography and electrocardiography as described in the first aspect of the present invention.

[0019] A fourth aspect of the present invention provides an apparatus comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the motion-guided dynamic fusion method of ear-brain-electrocardiogram as described in the first aspect of the present invention; the apparatus may be an ear wearable monitoring device, a mobile terminal, an edge computing device, or a server.

[0020] Beneficial effects of this invention: This invention utilizes motion signals to generate dynamic gating weights, so that motion signals are no longer just passive inputs, but participate in controlling the fusion ratio of two types of physiological features: electroencephalography (EEG) and electrocardiography (ECG). It can adaptively adjust modal contributions according to the wearer's motion state.

[0021] This invention reduces the weight of EEG features when motion artifacts are strong or EEG quality is low, and increases the weight of EEG features when motion is stable or EEG quality is good, thereby enhancing the robustness of wearable long-term monitoring.

[0022] The dynamic gating weights of this invention can output time window-level or feature dimension-level explanatory information, which facilitates the analysis of the model's dependence on EEG and ECG at different times and improves the interpretability of multimodal fusion results.

[0023] This invention can be combined with various downstream tasks such as epilepsy prediction, fatigue detection, and sleep monitoring, and has strong versatility and scalability.

[0024] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0025] Figure 1 This is an overall flowchart of the method in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of one of the dynamic gating weight generation processes in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of one type of dynamic fusion in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the dynamic fusion model training and output process in Embodiment 1 of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0027] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0028] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0029] Example 1: Embodiment 1 of the present invention provides a motion-guided dynamic fusion method of ear-brain-electrocardiogram (EEG) and electrocardiogram (ECG). This method synchronously collects EEG, ECG, and motion signals, uses motion signals to characterize the current wearing state and motion artifact intensity, and generates dynamic gating weights accordingly to control the fusion ratio of EEG and ECG features.

[0030] In actual deployment, the system performs online calculations using fixed time windows as the smallest processing unit. Each time window contains segments of electroencephalogram (EEG), electrocardiogram (ECG), and motor data, and corresponds to the same label or the same prediction target. Adjacent time windows can overlap to improve the temporal resolution of early warning or status recognition tasks.

[0031] This invention discloses a motion-guided dynamic fusion method of ear-brain-electrocardiogram, specifically comprising the following steps: S1: Acquire synchronously collected ear, brain, electrocardiogram, and motion signals.

[0032] The system simultaneously acquires ear, brain, electrocardiogram (ECG), and motion signals. It synchronizes these signals in time and uses a unified timestamp or sampling point alignment to map subsequent features to the same fixed time window.

[0033] In one specific implementation, the electroencephalogram (EEG) signals are collected by electrodes placed on or around the ear. Specifically, the EEG collection electrodes can be positioned in the concha, ear canal, mastoid process behind the ear, or periauricular skin. The collection channel can be a single-channel, dual-channel, or multi-channel configuration for one ear. The electrocardiogram (ECG) signals are collected by ECG electrodes or wearable ECG sensors. ECG signals can be collected using chest patch, wristband, ear clip, or clothing-integrated electrodes. Motion signals are collected by accelerometers, gyroscopes, or inertial measurement units (IMUs). Preferably, a combination of a three-axis accelerometer and a three-axis gyroscope is used to collect motion signals. A magnetometer, pressure sensor, or wearable looseness detection sensor may also be added.

[0034] The sampling rates for different modal signals can be different. For example, the sampling rate for EEG signals can be set from 125Hz to 1000Hz, the sampling rate for ECG signals can be set from 125Hz to 500Hz, and the sampling rate for motion signals can be set from 25Hz to 200Hz. The acquisition end writes a unified clock timestamp to each sampling point or each data packet; if different sensors use independent clocks, clock deviation correction is performed by starting a synchronization pulse, Bluetooth time synchronization packet, or software interpolation.

[0035] When short-term data loss occurs, the system records the location and duration of the loss. If the duration is less than a preset threshold, it is filled in using linear interpolation, spline interpolation, or forward hold. If the duration exceeds the preset threshold, the time window is marked as a low-quality sample, and the reliability of the corresponding modality is reduced in subsequent gating generation.

[0036] S2: Preprocess the EEG, ECG, and motion signals and construct multimodal time window samples according to a fixed time window.

[0037] Preprocessing is performed on the EEG, ECG, and motion signals to remove noise and baseline drift, thereby improving signal quality. In one specific implementation, bandpass filtering, notch filtering, and normalization are applied to the EEG and ECG signals, while detrending or amplitude normalization is performed on the motion signals. Subsequently, each modal signal is divided into fixed time windows, such as 5-second windows, with each time window containing a corresponding EEG waveform segment, ECG waveform segment, and motion segment.

[0038] Specifically, preprocessing of EEG signals may include bandpass filtering from 0.5Hz to 45Hz, notch filtering at 50Hz or 60Hz, baseline drift removal, abnormal amplitude truncation, independent component analysis, or adaptive filtering. For ear-based acquisition scenarios, motion artifacts can be identified based on the correlation between motion signals and low-frequency components of EEG signals, and segments with high correlation can be downweighted or removed.

[0039] ECG signal preprocessing may include operations such as 0.5Hz to 40Hz bandpass filtering, power frequency notch filtering, R-peak detection, abnormal heartbeat removal, RR interval interpolation, and heart rate variability sequence construction.

[0040] Motion signal preprocessing may include operations such as zero bias calibration, gravity component estimation, triaxial composite magnitude calculation, and moving mean smoothing.

[0041] The fixed time window length can be set from 1 second to 30 seconds depending on the task, preferably 2 seconds, 5 seconds, 10 seconds, or 30 seconds. For tasks requiring longer context, such as epileptic seizure prediction, multiple consecutive time windows can be combined into a sequence and input into the downstream model; for fatigue detection or emotion recognition tasks, a longer window can be used to obtain stable frequency domain and heart rate variability features.

[0042] When constructing multimodal time window samples, the system uses the start and end times of the window as a reference to extract corresponding segments from the three types of signals and resamples each mode to the length required by the model. If the effective sampling ratio of a certain mode within the window is lower than a preset ratio, such as 80%, a quality mask is generated for that mode for use by the motion gating generation module and downstream models.

[0043] S3: Modal feature extraction is performed on the EEG and ECG signals in the multimodal time window samples to obtain EEG features and ECG features respectively. In one specific implementation, a modality-independent neural network feature extractor is used to extract ear-brain electroencephalogram (EEG) features and electrocardiogram (ECG) features, respectively. The neural network feature extractor includes a signal segmenter and an encoder; wherein the signal segmenter is used to divide or map continuous ear-brain electroencephalogram (EEG) signals or ECG signals into segment representations corresponding to multiple time segments, spectral segments, or heartbeat segments; and the encoder is used to perform temporal modeling and feature abstraction on the segment representations to obtain the ear-brain electroencephalogram (EEG) features and ECG features, denoted as Xeeg and Xecg, respectively.

[0044] EEG characteristics can include time-domain statistics, band energy, relative band power, spectral entropy, sample entropy, Hjorth parameter, time-frequency plot features, and neural network coding features. Band energy can be calculated according to the δ, θ, α, β, and γ frequency bands, or individualized bands can be set according to the characteristics of EEG signals.

[0045] ECG features can include heart rate, mean RR interval, SDNN, RMSSD, pNN50, low-frequency power, high-frequency power, low-to-high frequency ratio, R-peak morphology features, and neural network encoding features. For wearable scenarios where only a single-lead ECG can be obtained, R-peak time series and heartbeat morphology segments can be used as inputs preferentially.

[0046] If a neural network encoder is used, the EEG encoder and ECG encoder can each consist of one or more of the following: one-dimensional convolutional layers, batch normalization layers, activation layers, pooling layers, and state-space modeling layers. The EEG encoder and ECG encoder can use the same structure or different structures depending on their respective signal characteristics. The feature dimensions of the two encoder outputs can be unified to d-dimensionality (e.g., 64, 128, or 256 dimensions) through a linear projection layer, enabling subsequent dynamic fusion layers to perform dimension-wise weighting.

[0047] To improve individual adaptability, the system can incorporate subject identifier embedding, device identifier embedding, or electrode contact quality embedding at the model input; it can also fine-tune the normalization parameters, projection layers, or gating layers using a small amount of individual calibration data after training.

[0048] S4: Extract motion intensity features from motion signals in multimodal time window samples, extract signal quality features from preprocessed EEG and ECG signals, map motion intensity features to motion embedding vectors, generate dynamic gating weights through motion embedding vectors, or concatenate motion intensity features and signal quality features and input them into a multilayer perceptron to obtain dynamic gating weights.

[0049] First, motion intensity features are extracted from the preprocessed motion signal, including but not limited to variance within the window, energy, mean, peak-to-peak value, rate of change of acceleration, or rate of change of angular velocity. Second, signal quality features are extracted from the preprocessed EEG and ECG signals, including amplitude saturation ratio, drift intensity, channel variance, power frequency residual intensity, and motion artifact correlation.

[0050] In one specific implementation, the motion intensity features are mapped to a motion embedding vector Xm, and the dynamic gating weight alpha is generated as follows: alpha = sigmoid(W·Xm+b), where the linear gating layer z = W·Xm+b, W and b are learnable parameters, and sigmoid is the Sigmoid activation function.

[0051] Specifically, the motion embedding vector Xm can be constructed as follows: Within each fixed time window, the triaxial acceleration signals a_x, a_y, a_z and the triaxial angular velocity signals g_x, g_y, g_z are first read, and the composite acceleration magnitude a(t) = sqrt(a_x(t)^2 + a_y(t)^2 + a_z(t)^2) and the composite angular velocity magnitude g(t) = sqrt(g_x(t)^2 + g_y(t)^2 + g_z(t)^2) are calculated respectively. Subsequently, within the same time window, the mean, variance, energy, peak-to-peak value, first-order difference mean, first-order difference mean square value, and maximum value of a(t) and g(t) are calculated respectively, and the above statistics are concatenated in a fixed order to obtain the original motion feature vector F_m. The above fixed order can be to first arrange all the statistics of a(t) completely, and then arrange all the statistics of g(t) completely. For example: F_m = [acceleration mean, acceleration variance, acceleration energy, acceleration peak value, acceleration first difference mean, acceleration first difference mean square, acceleration maximum value, angular velocity mean, angular velocity variance, angular velocity energy, angular velocity peak value, angular velocity first difference mean, angular velocity first difference mean square, angular velocity maximum value].

[0052] To eliminate the influence of differences in sensor dimensions and individual wearing methods, the original motion feature vector F_m is standardized to obtain the normalized motion feature vector F'_m=(F_m-mu_m) / sigma_m, where mu_m and sigma_m (σm) are the mean and standard deviation of the motion feature, respectively.

[0053] In one executable implementation, the normalized motion feature vector F'_m can be directly used as the motion embedding vector Xm; in another executable implementation, F'_m is input into the motion embedding layer to obtain Xm, i.e., Xm = ReLU(W_m·F'_m + b_m), where W_m and b_m are learnable parameters of the motion embedding layer, and ReLU is a non-linear activation function. The motion embedding layer is a neural network layer that generates the motion embedding vector Xm. Through this motion embedding layer, the system can map low-dimensional artificial statistical features into a high-dimensional motion state representation that is more suitable for gating computation.

[0054] The parameters W and b are used to map the motion embedding vector Xm to the dynamic gating weight alpha. When the dynamic gating weight is a time window level weight, Xm is a d-dimensional vector, W is a 1×d-dimensional weight vector, b is a scalar, and the output alpha is a scalar between 0 and 1. When the dynamic gating weight is a feature dimension level weight, W is a D×d-dimensional weight matrix, b is a D-dimensional bias vector, and the output alpha is a D-dimensional vector, where D is the same as the unified feature dimension of EEG and ECG features. When the dynamic gating weight is a channel level weight, D can also be set to the number of EEG channels or the number of feature channels.

[0055] W, b, and optional W_m, b_m are all learned through backpropagation during model training. During model training, the EEG signal is processed by an EEG encoder to obtain Xeeg, the ECG signal by an ECG encoder to obtain Xecg, and the motion signal by a motion intensity feature construction and motion embedding layer to obtain Xm. Dynamic gating weights are then obtained using alpha = sigmoid(W·Xm + b), and fused physiological features are obtained according to Xfused = (1-alpha)·Xeeg + alpha·Xecg. The fused physiological features are input into the downstream prediction or recognition model to obtain the prediction result. The task loss between the prediction result and the true label is calculated. This task loss can be cross-entropy loss, binary classification cross-entropy loss, or mean squared error loss. Then, backpropagation is performed based on the task loss to update the learnable parameters (W, b, and optional W_m, b_m). At each update of the learnable parameters, the optimizer calculates the gradient of W, b, and / or W_m, b_m based on the task loss and updates the learnable parameters according to gradient descent, Adam, or other optimization algorithms. For example, , , , .

[0056] Where L is the training loss and η is the learning rate.

[0057] After multiple rounds of training, the model can automatically learn the correspondence between motion state and modal credibility, so that when the motion intensity increases or the EEG quality decreases, alpha tends to increase, thereby improving the contribution of ECG features; when the motion is stable and the EEG quality is good, alpha tends to decrease, thereby improving the contribution of EEG features.

[0058] Furthermore, to avoid abrupt changes in the dynamic gating weights of adjacent time windows and reduce jitter, a gating smoothing regularization term can be added to the task loss during training. The gating smoothing regularization term L_s can be expressed as:

[0059] Where T represents the number of consecutive time windows, D represents the dimension of the dynamic gating weights, alpha(t,j) represents the j-th dimension of the dynamic gating weight in the t-th time window, and alpha(t-1,j) represents the j-th dimension of the dynamic gating weight in the (t-1)-th time window. When the dynamic gating weights are time window-level scalar weights, D = 1. By adding L_s to the total training loss, the smooth changes in the dynamic gating weights of adjacent time windows can be constrained, reducing the abrupt changes in fused weights caused by noise in a single time window.

[0060] Furthermore, a gating prior constraint term can be added to the task loss to ensure that the alpha corresponding to high motion intensity samples is not lower than a preset lower limit, and the alpha corresponding to low motion intensity samples with high EEG quality is not higher than a preset upper limit. Through joint optimization of the task loss, gating smoothing regularization term, and gating prior constraint term, dynamic gating weights that conform to both physiological intuition and actual data distribution can be obtained.

[0061] Furthermore, a modal random deactivation strategy is employed during model training to achieve multimodal adaptive training enhancement. This involves randomly masking one or more of the following: EEG features, ECG features, or motion features. This allows the model to still output stable fused features even when a particular modality is missing or its quality degrades. The proportion of modal random deactivation can be dynamically adjusted according to the training stage. For example, in the early stages of training, a single modality can be masked with a low probability to ensure model convergence. In the later stages of training, the masking probability can be increased, enabling the model to still output stable results even under conditions of EEG contamination, ECG loss, or abnormal motion signals.

[0062] In another specific implementation, motion intensity features and signal quality features are concatenated and input into a multilayer perceptron to obtain dynamic gating weights at the time window level, channel level, or feature dimension level. The dynamic gating weights range from 0 to 1 and are used to characterize the contribution ratio of the electroencephalogram (EEG) features to the electroencephalogram (EEG) features in the current time window.

[0063] The motion intensity characteristics can be calculated as follows: For the triaxial acceleration signals a_x, a_y, and a_z, calculate the composite acceleration magnitude a(t) = sqrt(a_x(t)^2 + a_y(t)^2 + a_z(t)^2), and then calculate the mean, variance, energy, peak-to-peak value, first-order difference mean, first-order difference mean square value, and maximum value of a(t) within each time window. For the triaxial angular velocity signals, calculate the composite angular velocity magnitude g(t) = sqrt(g_x(t)^2 + g_y(t)^2 + g_z(t)^2), and similarly calculate the mean, variance, energy, peak-to-peak value, first-order difference mean, first-order difference mean square value, and maximum value of g(t) within each time window. After standardization, the above statistics yield the motion feature vector.

[0064] Signal quality characteristics of electroencephalogram (EEG) signals can include the proportion of samples exceeding the amplitude threshold within a window, the proportion of low-frequency energy, the proportion of power frequency energy, outliers in inter-channel correlation, the correlation coefficient with acceleration modulus, and the electrode contact impedance level. Signal quality characteristics of electrocardiogram (ECG) signals can include the confidence level of R-peak detection, the proportion of abnormal adjacent RR intervals, QRS waveform similarity, and the proportion of ECG amplitude saturation.

[0065] Dynamic gating weights can be generated by combining rule initialization with a learnable model. Rule initialization ensures that the model aligns with physiological intuition during the initial training phase; for example, the higher the exercise intensity and the lower the EEG quality, the greater the ECG weight. The learnable model automatically adjusts the importance of each indicator based on the specific data task.

[0066] In one executable implementation, motion feature vectors (motion intensity features), EEG signal quality features, and ECG signal quality features are concatenated into a gated input vector g. This gated input vector g is then fed into a multilayer perceptron and sequentially passed through a fully connected layer, a ReLU activation layer, a Dropout layer, and an output fully connected layer with Sigmoid activation to obtain dynamic gated weights alpha. If the output is a time window-level weight, alpha is a scalar; if the output is a feature dimension-level weight, alpha is a d-dimensional vector; if the output is a channel-level weight, the dimension of alpha is consistent with the number of EEG channels or the number of feature channels.

[0067] Furthermore, the dynamic gating weight can be a time window level weight, a feature dimension level weight, or a channel level weight; when the dynamic gating weight is a feature dimension level weight, its dimension is the same as that of the EEG and ECG features.

[0068] S5: Dynamically fuse ear, brain, and electrocardiogram features based on dynamic gating weights to obtain fused physiological features.

[0069] The dynamic fusion layer calculates the fused physiological features based on the dynamic gating weight alpha: Xfused = (1-alpha)·Xeeg + alpha·Xecg. When alpha is close to 0, the fused physiological features are mainly contributed by electroencephalogram (EEG) features; when alpha is close to 1, the fused physiological features are mainly contributed by electrocardiogram (ECG) features.

[0070] In one specific implementation, when the exercise intensity is high, the amplitude of the electroencephalogram (EEG) fluctuates abnormally, or the quality of the EEG signal decreases, the dynamic gating weight alpha increases, thereby increasing the contribution of the electrocardiogram (ECG) features; when the exercise intensity is low and the quality of the EEG signal is good, the dynamic gating weight alpha decreases, thereby increasing the contribution of the EEG features.

[0071] In another specific implementation, the dynamic gating weights are feature dimension-level weights. The model can use different fusion ratios for different feature dimensions within the same time window, so that some dimensions are biased towards EEG information and some dimensions are biased towards ECG information.

[0072] To avoid fusion bias caused by inconsistent feature scales across different modalities, layer normalization, batch normalization, or L2 normalization can be performed on Xeeg and Xecg respectively before dynamic fusion. If the EEG and ECG features have different dimensions, they can be mapped to the same dimension first through a linear projection layer.

[0073] Besides the weighted summation method, dynamic fusion can also employ a splicing-projection approach, where (1-alpha)·Xeeg, alpha·Xecg, and alpha itself are spliced ​​together and input into a linear or multilayer perceptron to obtain fused physiological features. This method preserves gating weight information, allowing downstream models to simultaneously perceive both the fusion result and the fusion basis.

[0074] Physiological features can be directly input into a single-window classifier, or stacked sequentially into a state-space model, recurrent network, or Transformer. For early warning tasks, the output can employ a moving average, exponential smoothing, or continuous multi-window confirmation strategy to reduce false alarms from a single window.

[0075] S6: Input the fused physiological features into the downstream prediction or identification model to obtain the target physiological state, disease risk, or early warning probability.

[0076] The downstream prediction or recognition model can be any one of a state-space model, recurrent neural network, convolutional neural network, Transformer model, classifier, or regressor. The fused physiological features are input into the state-space model, recurrent neural network, convolutional neural network, Transformer model, classifier, or regressor, and the output is the target physiological state, disease risk, or warning probability. Downstream tasks can include epileptic seizure prediction, fatigue detection, sleep state recognition, emotional state recognition, or cardio-brain coordination abnormality detection. The target physiological state includes the pre-seizure state, fatigue state, sleep state, emotional state, or cardio-brain coordination abnormality state.

[0077] Taking the epileptic seizure prediction task as an example, the system integrates physiological features to form a long-term sequence in chronological order, inputs it into the Mamba state-space temporal modeling layer, and outputs the risk probability before the seizure. The system can also output a dynamic gating weight sequence as an explanation of the model's dependence on EEG and ECG at different time windows.

[0078] In the epilepsy seizure prediction task, a preset time period before a seizure can be marked as the pre-seizure state, and the time period far from the seizure can be marked as the interictal state. The system performs temporal modeling on continuous multi-window fusion physiological features and outputs the seizure risk probability corresponding to each time window; when the risk probability exceeds the warning threshold within K consecutive time windows, an early warning is triggered.

[0079] In fatigue detection tasks, dynamically fused physiological features can be input into a classifier to output levels of alertness, mild fatigue, moderate fatigue, or severe fatigue, or a continuous fatigue score can be output. Motion gating weights can be used to distinguish fatigue features caused by genuine neurophysiological changes from signal abnormalities caused by motion artifacts.

[0080] In sleep or mood monitoring tasks, the system can adjust the window length, frequency band characteristics, and downstream model structure according to the characteristics of the task, but the core processes of motion signal-guided gating, quality assessment, and dynamic fusion remain unchanged.

[0081] Example 2: Embodiment 2 of the present invention provides a motion-guided dynamic fusion system of ear-brain-electrocardiogram, including a multimodal acquisition module, a preprocessing module, a modal feature extraction module, a motion gating generation module, a dynamic fusion module, and a task output module.

[0082] The multimodal acquisition module is configured to acquire synchronously acquired electroencephalogram (EEG), electrocardiogram (ECG), and motion signals, and to synchronize the EEG, ECG, and motion signals in time.

[0083] The preprocessing module is configured to preprocess EEG signals, ECG signals, and motion signals, perform bandpass filtering, notch filtering, and standardization on EEG and ECG signals, perform detrending or amplitude normalization on motion signals, and divide each modal signal into fixed time windows.

[0084] The modal feature extraction module is configured to extract EEG features and ECG features from EEG signals and ECG signals, respectively.

[0085] The motion gating generation module is configured to generate dynamic gating weights based on motion intensity characteristics and / or signal quality characteristics.

[0086] The dynamic fusion module is configured to dynamically fuse ear, brain, and electrocardiogram features based on dynamic gating weights to obtain fused physiological features.

[0087] The task output module is configured to output the target's physiological state, disease risk, or early warning probability based on the fused physiological characteristics.

[0088] In one specific implementation, the multimodal acquisition module also includes a sensor state detection unit for detecting electrode contact impedance, battery power, wireless connection status, and looseness of wear; the preprocessing module also includes a data integrity detection unit for generating an effective sampling ratio and quality mask; the motion gating generation module also includes a rule constraint unit for applying upper and lower limit constraints to the dynamic gating weights based on motion intensity characteristics and signal quality characteristics.

[0089] The dynamic fusion module can be deployed on mobile terminals or edge computing devices to reduce data upload volume and response latency; the task output module can be deployed on a local processor or cloud server depending on the task complexity. For real-time early warning scenarios, local inference is preferred, and only early warning results, gating explanation information, and necessary low-dimensional features are uploaded.

[0090] Example 3: Embodiment 3 of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the motion-guided dynamic fusion method of electroencephalography and electrocardiography as described in Embodiment 1.

[0091] Example 4: Embodiment 4 of the present invention provides a device including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the motion-guided ear-brain-electrocardiogram dynamic fusion method as described in Embodiment 1. The device can be a wearable ear monitoring device, a mobile terminal, an edge computing device, or a server.

[0092] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0093] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; those skilled in the art should understand that modifications can still be made to the technical solutions described in the above embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A motion-guided dynamic fusion method of ear-brain-electrocardiogram, characterized in that: Includes the following steps: S1: Acquire synchronously collected EEG, ECG, and motion signals, and synchronize the EEG, ECG, and motion signals in time; S2: Preprocess the EEG, ECG, and motion signals to remove noise and baseline drift, and construct multimodal time window samples according to a fixed time window; S3: Modal features are extracted from the electroencephalogram (EEG) and electrocardiogram (ECG) signals in the multimodal time window samples to obtain the EEG feature Xeeg and the ECG feature Xecg. S4: Extract motion intensity features from motion signals in multimodal time window samples, map the motion intensity features to a motion embedding vector Xm, and generate dynamic gating weights through the motion embedding vector Xm: alpha=sigmoid(W·Xm+b), where W and b are learnable parameters, and sigmoid is the Sigmoid activation function; or extract motion intensity features from motion signals in multimodal time window samples and standardize them, extract signal quality features from EEG and ECG signals respectively, concatenate the standardized motion intensity features and signal quality features into a gating input vector g, and input the gating input vector g into a multilayer perceptron to obtain dynamic gating weights; S5: Dynamically fuse the electroencephalogram (EEG) feature Xeeg and electrocardiogram (ECG) feature Xecg based on dynamic gating weights to obtain the fused physiological feature: Xfused=(1-alpha)·Xeeg+alpha·Xecg; or concatenate (1-alpha)·Xeeg, alpha·Xecg, and alpha and input them into a linear layer or multilayer perceptron to obtain the fused physiological feature. S6: Input the fused physiological features into the downstream prediction or identification model to obtain the target physiological state, disease risk, or early warning probability.

2. The motion-guided dynamic fusion method of ear-brain-electrocardiogram according to claim 1, characterized in that: The dynamic gating weight can be a time window level weight, a feature dimension level weight, or a channel level weight; when the dynamic gating weight is a feature dimension level weight, its dimension is the same as that of the EEG and ECG features.

3. The motion-guided dynamic fusion method of ear-brain-electrocardiogram according to claim 1, characterized in that: In S4, the motion embedding vector Xm is constructed as follows: S41: Within each fixed time window, read the triaxial acceleration signal and the triaxial angular velocity signal, calculate the composite acceleration magnitude and the composite angular velocity magnitude respectively, and then calculate the statistics of the composite acceleration magnitude and the composite angular velocity magnitude within the same time window. Concatenate the statistics in a fixed order to obtain the original motion feature vector F_m. Standardize the original motion feature vector F_m to obtain the normalized motion feature vector F'_m. S42: Directly use the normalized motion feature vector F'_m as the motion embedding vector Xm, or input F'_m into the motion embedding layer to obtain the motion embedding vector Xm, Xm=ReLU(W_m·F'_m+b_m), where W_m and b_m are the learnable parameters of the motion embedding layer, and ReLU is a non-linear activation function.

4. The motion-guided dynamic fusion method of ear-brain-electrocardiogram according to claim 1, characterized in that: In step S4, the learnable parameters W, b and / or W_m, b_m are all learned through backpropagation during model training. During model training, the obtained fused physiological features are input into the downstream prediction or recognition model to obtain the prediction result. The task loss between the prediction result and the real label is calculated, and then backpropagation is performed based on the task loss to update the learnable parameters. Each time the learnable parameters are updated, the optimizer calculates the gradient of W, b and / or W_m, b_m based on the task loss, and updates the learnable parameters according to the gradient descent or Adam optimization algorithm.

5. The motion-guided dynamic fusion method of ear-brain-electrocardiogram according to claim 4, characterized in that: The task loss is supplemented with gated smoothing regularization terms and / or gated prior constraints.

6. The motion-guided dynamic fusion method of ear-brain-electrocardiogram according to claim 4, characterized in that: The model training process employs a modal random deactivation strategy.

7. The motion-guided dynamic fusion method of ear-brain-electrocardiogram according to claim 1, characterized in that: In step S6, the downstream prediction or identification model is a state-space model, a recurrent neural network, a convolutional neural network, a Transformer model, a classifier, or a regressor.

8. A motion-guided dynamic fusion system of electroencephalography and electrocardiography, used to implement the method described in any one of claims 1-7, characterized in that: include: The multimodal acquisition module is used to acquire synchronously acquired electroencephalogram (EEG), electrocardiogram (ECG), and motion signals, and to synchronize the EEG, ECG, and motion signals in time. The preprocessing module is used to preprocess the EEG, ECG and motion signals. It performs bandpass filtering, notch filtering and standardization on the EEG and ECG signals, detrending or amplitude normalization on the motion signals, and divides each modal signal into fixed time windows. The modal feature extraction module is used to extract electroencephalogram (EEG) features and electrocardiogram (ECG) features from EEG signals and ECG signals, respectively. A motion gating generation module is used to generate dynamic gating weights based on motion intensity characteristics and / or signal quality characteristics; The dynamic fusion module is used to dynamically fuse ear, brain, and electrocardiogram features based on dynamic gating weights to obtain fused physiological features. The task output module is used to output the target's physiological state, disease risk, or early warning probability based on the fused physiological characteristics.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by the processor, it implements the method of any one of claims 1-7.

10. An apparatus comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the method of any one of claims 1-7; the device may be an ear wearable monitoring device, a mobile terminal, an edge computing device, or a server.

Images