Portable state analysis method and system based on multi-mode electroencephalogram and electrocardio
By simultaneously acquiring EEG and ECG signals using portable devices, constructing difference and ratio features, and using the TabPFN model for feature fusion, the problem of insufficient synchronization and generalization ability in multimodal physiological signal analysis is solved, realizing portable and stable physiological state assessment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU SEVENTH PEOPLES HOSPITAL
- Filing Date
- 2026-01-13
- Publication Date
- 2026-04-17
AI Technical Summary
Existing technologies for the synchronous acquisition and analysis of multimodal physiological signals suffer from problems such as difficulty in ensuring synchronization, lack of systematic feature fusion, insufficient model generalization ability, and insufficient portability, which limit their application in various scenarios.
Portable devices are used to simultaneously acquire EEG and ECG signals. Difference and ratio features are constructed through preprocessing and feature extraction. Feature fusion is then performed using a tabular data processing model (TabPFN) to achieve end-to-end state analysis.
It improves feature representation capabilities, enhances the model's generalization performance, achieves stable classification and regression prediction under limited sample conditions, lowers the barrier to entry, and provides reliable technical support for physiological state assessment in multiple scenarios.
Smart Images
Figure CN121881002A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of state analysis, and in particular relates to a portable state analysis method and system based on multimodal electroencephalography and electrocardiography. Background Technology
[0002] With the development of wearable devices and signal processing technology, state recognition and analysis based on multimodal physiological signals (such as EEG and ECG) has become a research hotspot. Existing technologies typically acquire multi-channel signals from EEG and ECG simultaneously, extracting the time-domain and frequency-domain features of each signal (such as power spectral density and heart rate variability), and employing feature-layer fusion strategies combined with machine learning models to classify and quantify the specific physiological or cognitive states of subjects. These methods aim to utilize complementary information between different physiological signals to improve the objectivity and stability of state recognition.
[0003] However, existing technical solutions still have several limitations in practical applications. First, the synchronous acquisition and high-precision time alignment of multimodal signals are difficult to guarantee in practice, affecting the reliability of subsequent fusion analysis. Second, feature fusion often relies on simple splicing, lacking a feature construction method that systematically represents the dynamic changes between different physiological states. Furthermore, most analysis models depend on large amounts of labeled data, resulting in insufficient generalization ability and robustness in scenarios with limited samples. Finally, existing systems are often based on large-scale equipment in laboratory environments, lacking in portability, standardized acquisition processes, and end-to-end automated processing, limiting their deployment and application in diverse scenarios. Summary of the Invention
[0004] To address the aforementioned technical problems, this invention provides a portable state analysis method based on multimodal electroencephalography (EEG) and electrocardiography (ECG), comprising:
[0005] Simultaneously collect the subject's electroencephalogram (EEG) and electrocardiogram (ECG) signals;
[0006] The electroencephalogram (EEG) signal and the electrocardiogram (ECG) signal are preprocessed and feature extracted respectively to obtain EEG features and ECG features;
[0007] Based on the values of the electrocardiogram characteristics under different physiological states, new features representing the differences between states are constructed.
[0008] The EEG features, ECG features, and new features are fused to obtain multimodal fusion features;
[0009] The multimodal fusion features are input into a pre-trained tabular data processing model to obtain classification results or quantitative prediction values that characterize the subject's state.
[0010] Optionally, the simultaneous acquisition of the subject's electroencephalogram (EEG) and electrocardiogram (ECG) signals includes:
[0011] The subject was guided to sequentially experience at least two resting states and at least two task states;
[0012] During the subjects' various states, time-aligned EEG and ECG signal sequences were synchronously acquired using portable EEG and ECG acquisition devices.
[0013] Optionally, constructing new features representing differences between states based on the values of the electrocardiogram characteristics under different physiological states includes:
[0014] For each of the at least two resting states, a state comparison pair is formed with each of the at least two task states;
[0015] For each ECG characteristic index, the difference and ratio characteristics between its value in the task state and its value in the resting state are calculated in each state comparison pair to obtain the new characteristic.
[0016] Optionally, the preprocessing and feature extraction of the EEG signal and the ECG signal to obtain EEG features and ECG features respectively includes:
[0017] After filtering and denoising the EEG signal, its power spectral density is extracted as the EEG feature;
[0018] The electrocardiogram (ECG) signal is processed to extract heart rate and heart rate variability indices as ECG features.
[0019] Optionally, inputting the multimodal fusion features into a pre-trained tabular data processing model includes: inputting the multimodal fusion features into a tabular prior data fitting network model.
[0020] Optionally, the multimodal fusion features are input into a network model that fits the tabular prior data to obtain classification results characterizing the subject's state, including:
[0021] The features of the training set samples containing labels are encoded with the multimodal fusion features and concatenated into an input sequence;
[0022] The input sequence is fed into the prior data of the table to fit the network model, and the output sequence is obtained after one forward propagation.
[0023] The classification result is determined based on the decoding result corresponding to the multimodal fusion feature in the output sequence.
[0024] Optionally, inputting the multimodal fusion features into a pre-trained tabular data processing model to obtain a quantitative prediction value for characterizing the subject's state includes: inputting the multimodal fusion features into a parameter-tuned tabular prior data fitting network regressor model to obtain the quantitative prediction value.
[0025] On the other hand, the present invention also provides a portable state analysis system based on multimodal electroencephalography and electrocardiography, comprising:
[0026] The EEG acquisition module uses a portable single-channel forehead-mounted EEG device to acquire the user's EEG signals in real time.
[0027] The ECG acquisition module uses portable ECG electrode patches to synchronously acquire the user's ECG signals;
[0028] The questionnaire assessment module includes self-rating and other-rating scales, used to collect subjective and other-rating symptom data from the participants;
[0029] The emotion stimulation module uses a standardized emotional image library to induce emotional responses and record corresponding EEG and ECG changes.
[0030] The data synchronization and management module coordinates the synchronous collection, annotation, and management of data from various modules to ensure time alignment and structural uniformity of multi-source data.
[0031] The feature extraction and fusion analysis module performs data cleaning, filtering, noise reduction, and feature extraction on the raw data of each modality. It also achieves feature fusion through standardization and normalization. Based on the fusion, new features are constructed through feature engineering methods to capture potential information.
[0032] The discrimination and output module contains a pre-trained tabular data processing model, which outputs classification results or quantitative predictions that characterize the subject's state based on the input multimodal fusion features.
[0033] On the other hand, the present invention also provides an electronic device including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.
[0034] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.
[0035] Compared with the prior art, the present invention has the following advantages and technical effects:
[0036] This invention enhances the representational ability of features by constructing difference and ratio features of electrocardiogram characteristics between different physiological states, automatically capturing dynamic patterns under state transitions. Employing a meta-learning pre-trained tabular data processing model (TabPFN), it achieves stable and efficient classification and regression prediction under limited sample conditions, improving the model's generalization performance and practicality. The entire solution is integrated into portable hardware and automated software workflows, realizing end-to-end standardized operations from signal acquisition, processing, feature fusion to result output, significantly reducing the barrier to entry and dependence on professional environments, and providing reliable technical support for rapid and objective physiological state assessment in multiple scenarios. Attached Figure Description
[0037] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0038] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;
[0039] Figure 2 This is a scatter plot of the predicted and actual values of depression severity from the TabPFN regression model in this embodiment of the invention. Detailed Implementation
[0040] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0041] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0042] Example 1
[0043] like Figure 1 As shown, this embodiment provides a portable state analysis method based on multimodal EEG and ECG, including:
[0044] Simultaneously collect the subject's electroencephalogram (EEG) and electrocardiogram (ECG) signals;
[0045] The electroencephalogram (EEG) signal and the electrocardiogram (ECG) signal are preprocessed and feature extracted respectively to obtain EEG features and ECG features;
[0046] Based on the values of the electrocardiogram characteristics under different physiological states, new features representing the differences between states are constructed.
[0047] The EEG features, ECG features, and new features are fused to obtain multimodal fusion features;
[0048] The multimodal fusion features are input into a pre-trained tabular data processing model to obtain classification results or quantitative prediction values that characterize the subject's state.
[0049] The specific process includes:
[0050] 1. Data Acquisition and Synchronization:
[0051] Subjects wore EEG and ECG electrodes and underwent four task states in a quiet environment: RestEC with eyes closed, RestEO with eyes open, viewing negative emotion images (Neg), and an interview. The system, through a data synchronization module, achieved real-time synchronous acquisition of multi-source data (EEG, ECG, questionnaire, image-induced responses) under different physiological states, ensuring consistent timestamps for subsequent analysis.
[0052] 2. Data preprocessing and feature extraction:
[0053] After filtering and denoising, frequency domain features such as power spectral density (PSD) were extracted from the EEG signals; features such as heart rate (BPM) and heart rate variability (HRV) were extracted from the ECG signals; and the questionnaire module automatically recorded the scores for each item. Each module transmitted the raw data to the multimodal fusion analysis module. The multimodal fusion analysis module received a total of 36 features, including 4 EEG features and 32 ECG features.
[0054] 3. Multimodal fusion and feature engineering:
[0055] The system employs fusion strategies such as feature splicing to integrate standardized multimodal features from EEG, ECG, and other data, fully utilizing complementary information from each modality. Building upon this foundation, the system constructs new features using specific feature engineering methods, thereby accurately capturing latent patterns in the data and improving prediction accuracy and generalization ability.
[0056] The system's feature engineering method accomplishes this by comparing differences in ECG feature changes under different physiological states. The set of four physiological states experienced by the subjects in the experiment is represented as follows: ,in Represents a resting state (including resting with eyes open and resting with eyes closed). Representing task states (including viewing images with negative emotions and conducting voice interviews). Defining state comparison pairs. For each state pair and each ECG indicator Calculate the difference characteristics;
[0057] ;
[0058] Characteristics of ratios:
[0059] ;
[0060] in, Represents the sample sequence number. It is a tiny amount (to avoid division by zero, the actual value is taken as follows). )
[0061] In this way, each ECG indicator can generate 8 new features. Since there are 8 ECG indicators in total, 64 new features are constructed after feature engineering. After fusing with the 36 original EEG and ECG features, a total of 100 features are obtained.
[0062] 4. Judgment Process and Output
[0063] The discrimination module accepts fused features as input and uses the TabPFN (Tab Prior Data Fitting Network) machine learning algorithm to train a model to assess the depression risk of subjects. Based on the model's discrimination results, the system classifies subjects into two categories, outputs the final screening level (high / low risk) and subsequent suggestions, thereby automating and standardizing the screening process.
[0064] TabPFN is a foundational machine learning model designed for tabular data and can be applied to classification and regression tasks. Its implementation is based on a Transformer network architecture and a self-attention mechanism, with most parameters determined during pre-training.
[0065] TabPFN employs a meta-learning approach for training. This training process generates millions of synthetic tasks with varying data distributions, enabling the model to learn Bayesian priors for different types of tabular problems. During the inference phase, TabPFN utilizes in-context learning (ICL) to merge the tabular data from the training and test sets into a single, continuous sequence, which is then processed as context. The predicted posterior probability can be output with a single forward propagation, eliminating the need for fine-tuning or retraining.
[0066] This prediction process can be summarized as follows:
[0067] (1) Data serialization and embedding;
[0068] Suppose the training set contains Sample ,in yes 3D feature vectors It's a label. The test sample is... Unlabeled. The encoder encodes the features and labels of each sample in the training and test sets into vectors, which are then concatenated to obtain the input sequence. .
[0069] ;
[0070] in This is the encoding function, which transforms the input features X and labels y into embedding vectors that the model can process. This indicates that the label positions in the test set are empty.
[0071] (2) Forward propagation;
[0072] spliced sequence Input a pre-trained Transformer model and obtain the output through one forward propagation. For each position in the sequence... (Corresponding to the features or labels of a certain sample), the model calculates its relationship with all locations. attention weights
[0073] ;
[0074] These are query, key, and value matrices, respectively, derived from the input sequence. It is obtained through linear transformation. It is the dimension of the key vector, used to scale the attention weights.
[0075] After stacking multiple Transformer structures, the output sequence is obtained. .
[0076] (3) Decoding and prediction;
[0077] From the output sequence The label positions corresponding to the test samples are extracted and embedded, and the prediction results can be obtained by mapping through a linear layer and a softmax activation function.
[0078] ;
[0079] in It is the embedding vector of the test sample in the output sequence. These are the pre-trained output layer parameters. This is the predicted value of the test sample label.
[0080] Existing research has shown that TabPFN outperforms traditional machine learning algorithms in classification and regression tasks on small sample tabular datasets. Its prior knowledge obtained through meta-learning can achieve effective and robust generalization and uncertainty estimation even with limited data, making it suitable for the experimental scenario of this invention.
[0081] Furthermore, for patients, this system also constructs a regression prediction model for depression severity based on the Multimodal Physiological Signals and Clinical Scale (HAMD-24) score. Specifically, the physiological signal characteristics of the patient are first extracted through the aforementioned three-step process, and then these physiological signal characteristics are input into the parameter-tuned TabPFN regressor model. Through the calculation of the TabPFN regressor model, a quantitative prediction result representing the patient's depression severity can be output, achieving accurate assessment and prediction of the patient's depression severity.
[0082] Table 1 shows the results obtained using three feature sets: EEG single modality, ECG single modality, and multimodal fusion. It can be found that under the same TabPFN model, the classification performance of using EEG features alone is lower than that of using ECG features alone. However, when using multimodal fusion features for classification, all indicators are better than those of single modality. This indicates that multimodal brain-heart feature fusion helps to improve the objectivity and robustness of state analysis.
[0083] Table 1
[0084]
[0085] Table 2 shows the accuracy metrics obtained by different machine learning classification models using different feature sets. It can be noted that the TabPFN model performs significantly better than traditional machine learning models on multimodal feature sets.
[0086] Table 2
[0087]
[0088] For the prediction task, this invention further employs the TabPFN regressor, establishing a quantitative prediction model based on EEG-ECG multimodal physiological signals and the Hamilton Depression Rating Scale (HAMD-24) score. In the experiment, for 26 patients with depression, the system was evaluated by dividing the training and test sets using leave-one-out cross-validation. To achieve better regression prediction results, the TabPFN regressor model underwent random parameter tuning optimization. The optimized model parameters are shown in Table 3.
[0089] Table 3
[0090]
[0091] In this experimental setting, the model achieved a high positive correlation between the predicted score and the actual score (correlation coefficient r = 0.659, p < 0.001, root mean square error RMSE = 3.617, coefficient of determination R² = 0.352). Figure 2 The scatter plot of predicted and actual values of the Hamilton Rating Scale and various statistical indicators are presented. This result demonstrates that the proposed solution can quantify the state using objective physiological characteristics, support data-driven clinical auxiliary assessment methods, and help compensate for the shortcomings of traditional subjective scale assessments.
[0092] In this experiment, Python was used as the programming language for the machine learning prediction part, Scikit-learn was used as the machine learning framework, and PyTorch was used as the deep learning framework for running the TabPFN model. Table 4 lists the relevant software and hardware environment configurations.
[0093] Table 4
[0094]
[0095] The technical effect of this embodiment is as follows: It constructs a portable, rapid, and intelligent objective quantification and auxiliary analysis scheme for depression risk based on multimodal EEG and ECG signals. Through integrated portable hardware, it achieves synchronous acquisition and transmission of EEG and ECG signals, and combines multimodal features to construct and classify models to output depression-related risk levels or risk indices. Thus, without directly providing clinical diagnostic conclusions, it provides technical support for the objective assessment of depression-related mental health status, possessing good prospects for promotion and application value.
[0096] This invention integrates the steps of multimodal physiological signal data synchronization, standardized preprocessing, feature extraction, feature fusion, and intelligent classification analysis into a unified automated processing flow, realizing standardized and automated processing from signal acquisition to risk classification results or risk index output. This end-to-end process effectively reduces reliance on professional operation and manual interpretation, improves the efficiency and consistency of status assessment, and is conducive to large-scale deployment and promotion in fields such as primary healthcare services, school and corporate mental health management, and community public health.
[0097] In summary, the multimodal portable state analysis method and system proposed in this invention have outstanding beneficial effects in terms of technical performance, ease of application, and promotional value, and possess significant innovation and practicality compared to existing technologies. This system, through the objective acquisition and automatic analysis of multimodal physiological signals from electroencephalography (EEG) and electrocardiography (ECG), provides a scientific, effective, and easily scalable technical solution for early risk identification and stratified management of mood disorders without directly providing clinical diagnostic conclusions.
[0098] On the other hand, this embodiment also provides a portable state analysis system based on multimodal EEG and ECG, including:
[0099] The EEG acquisition module uses a portable single-channel forehead-mounted EEG device to acquire the user's EEG signals in real time.
[0100] The ECG acquisition module uses portable ECG electrode patches to synchronously acquire the user's ECG signals;
[0101] The questionnaire assessment module includes self-rating and other-rating scales, used to collect subjective and other-rating symptom data from the participants;
[0102] The emotion stimulation module uses a standardized emotional image library to induce emotional responses and record corresponding EEG and ECG changes.
[0103] The data synchronization and management module coordinates the synchronous collection, annotation, and management of data from various modules to ensure time alignment and structural uniformity of multi-source data.
[0104] The feature extraction and fusion analysis module performs data cleaning, filtering, noise reduction, and feature extraction on the raw data of each modality. It also achieves feature fusion through standardization and normalization. Based on the fusion, new features are constructed through feature engineering methods to capture potential information.
[0105] The discrimination and output module contains a pre-trained tabular data processing model, which outputs classification results or quantitative predictions that characterize the subject's state based on the input multimodal fusion features.
[0106] On the other hand, this embodiment also provides an electronic device, including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.
[0107] On the other hand, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.
[0108] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A portable state analysis method based on multi-modal electroencephalogram and electrocardiogram, characterized in that, include: Simultaneously collect the subject's electroencephalogram (EEG) and electrocardiogram (ECG) signals; The electroencephalogram (EEG) signal and the electrocardiogram (ECG) signal are preprocessed and feature extracted respectively to obtain EEG features and ECG features; Based on the values of the electrocardiogram characteristics under different physiological states, new features representing the differences between states are constructed. The EEG features, ECG features, and new features are fused to obtain multimodal fusion features; The multimodal fusion features are input into a pre-trained tabular data processing model to obtain classification results or quantitative prediction values that characterize the subject's state.
2. The method of claim 1, wherein, The simultaneous acquisition of the subject's electroencephalogram (EEG) and electrocardiogram (ECG) signals includes: The subject was guided to sequentially experience at least two resting states and at least two task states; During the subjects' various states, time-aligned EEG and ECG signal sequences were synchronously acquired using portable EEG and ECG acquisition devices.
3. The method according to claim 2, characterized in that, The construction of new features characterizing the differences between states based on the values of the electrocardiogram characteristics under different physiological states includes: For each of the at least two resting states, a state comparison pair is formed with each of the at least two task states; For each ECG characteristic index, the difference and ratio characteristics between its value in the task state and its value in the resting state are calculated in each state comparison pair to obtain the new characteristic.
4. The method according to claim 1, characterized in that, The preprocessing and feature extraction of the EEG and ECG signals to obtain EEG features and ECG features respectively include: After filtering and denoising the EEG signal, its power spectral density is extracted as the EEG feature; The electrocardiogram (ECG) signal is processed to extract heart rate and heart rate variability indices as ECG features.
5. The method according to claim 1, characterized in that, Inputting the multimodal fusion features into a pre-trained tabular data processing model includes: inputting the multimodal fusion features into a tabular prior data fitting network model.
6. The method according to claim 5, characterized in that, The multimodal fusion features are input into a network model that fits the tabular prior data to obtain classification results characterizing the subject's state, including: The features of the training set samples containing labels are encoded with the multimodal fusion features and concatenated into an input sequence; The input sequence is fed into the prior data of the table to fit the network model, and the output sequence is obtained after one forward propagation. The classification result is determined based on the decoding result corresponding to the multimodal fusion feature in the output sequence.
7. The method according to claim 1, characterized in that, The process of inputting the multimodal fusion features into a pre-trained tabular data processing model to obtain a quantitative prediction value that characterizes the subject's state includes: inputting the multimodal fusion features into a parameter-tuned tabular prior data fitting network regressor model to obtain the quantitative prediction value.
8. A portable state analysis system based on multimodal electroencephalography (EEG) and electrocardiography (ECG), characterized in that, include: The EEG acquisition module uses a portable single-channel forehead-mounted EEG device to acquire the user's EEG signals in real time. The ECG acquisition module uses portable ECG electrode patches to synchronously acquire the user's ECG signals; The questionnaire assessment module includes self-rating and other-rating scales, used to collect subjective and other-rating symptom data from the participants; The emotion stimulation module uses a standardized emotional image library to induce emotional responses and record corresponding EEG and ECG changes. The data synchronization and management module coordinates the synchronous collection, annotation, and management of data from various modules to ensure time alignment and structural uniformity of multi-source data. The feature extraction and fusion analysis module performs data cleaning, filtering, noise reduction, and feature extraction on the raw data of each modality. It also achieves feature fusion through standardization and normalization. Based on the fusion, new features are constructed through feature engineering methods to capture potential information. The discrimination and output module contains a pre-trained tabular data processing model, which outputs classification results or quantitative predictions that characterize the subject's state based on the input multimodal fusion features.
9. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computing program, it implements the method of any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-7.