Wearable emotion monitoring and evaluating system based on brain-heart-intestine nervous system electric signals

Abstract

The invention discloses a wearable emotion monitoring and evaluating system based on brain-heart-enteric nervous system electric signals, and belongs to the field of bio-electricity signal processing and recognition. According to the system, electroencephalogram (EEG) and electrocardio (ECG) are integrated, and an EGG signal closely related to nervous system activity is creatively integrated, so that the system can provide comprehensive emotion evaluation. The signals respectively represent different emotional response mechanisms, the emotion change can be more accurately captured through the fusion of the multi-modal signals, and the accuracy and reliability of emotion detection are improved. The system can collect and analyze physiological signals of a user in real time, provide instant emotion feedback, help the user to recognize emotion fluctuation in an early stage and take corresponding measures, and avoid emotion out-of-control or long-term emotion lowering.

Description

Technical Field

[0001] The present invention relates to the field of bioelectric signal processing and identification, in particular to the field of biomedical engineering and wearable technology. Background Art

[0002] As people pay more and more attention to health and emotional management, they are increasingly aware of the importance of emotions to mental health. However, traditional wearable devices usually focus on monitoring physiological indicators, such as heart rate, number of steps, etc., while the monitoring and management of emotions have not received sufficient attention. Emotions, as an important indicator of human mental health, are closely related to the brain-heart-gastrointestinal nervous system. The brain is responsible for rational thinking and emotional processing, while the heart and gastrointestinal nervous system play an important role in emotions, representing emotional and intuitive reactions respectively.

[0003] Although wearable devices on the market generally have certain physiological monitoring functions, they are not specifically optimized for emotion monitoring. Some smart bracelets and smart watches may integrate some emotion-related sensors, such as heart rate monitoring and skin electrical activity sensors, but their emotion recognition accuracy and practicality are limited. These traditional devices often fail to fully consider the synergy of the brain, heart, and gastrointestinal nervous system in emotional expression, and therefore cannot fully capture the complexity of an individual's emotional state. Summary of the invention

[0004] The present invention proposes a wearable human emotion monitoring system based on multimodal physiological signals, aiming to overcome the limitations of the prior art. By combining the information responses of the brain, heart and gastrointestinal nervous system, our system can more comprehensively assess the emotional state of individuals and provide users with more accurate and comprehensive emotion management services. The present invention considers emotion-related signals obtained from wearable devices, such as EEG (electroencephalogram), ECG (electrocardiogram), and EGG (electrogastric signal), and integrates them to form an emotion monitoring and evaluation system, so that emotions can be detected and feedback can be given.

[0005] The technical solution of the present invention is a wearable emotion monitoring and evaluation system based on the electrical signals of the brain-heart-enteric nervous system, the system comprising: a signal acquisition and preprocessing unit, a feature extraction unit, a feature fusion unit, an emotion recognition unit, and an emotion evaluation unit; the signal acquisition and preprocessing unit comprises: the acquisition and preprocessing of electroencephalogram (EEG), electrocardiogram (ECG), and electrogastric (EGG) signals, and then transmits the preprocessed signals to the feature extraction unit; the feature extraction unit extracts features from the received electroencephalogram, electrocardiogram, and electrogastric signals, and transmits the extracted features to the feature fusion unit; the features extracted by the feature extraction unit comprise: electroencephalogram features, electrocardiogram features, and electrogastric features;

[0006] The EEG features include: EEG time domain features, EEG frequency domain features, and EEG nonlinear features;

[0007] The ECG characteristics include: ECG time domain characteristics, ECG frequency domain characteristics, and ECG nonlinear characteristics;

[0008] The gastric electrical characteristics include: gastric electrical time domain characteristics, gastric electrical frequency domain characteristics, and gastric electrical nonlinear characteristics;

[0009] The feature fusion unit is a multi-input deep learning network that fuses all features and then inputs the fused features into the emotion recognition unit and the emotion assessment unit;

[0010] The emotion recognition unit recognizes the type of emotion, and the emotion evaluation unit recognizes the intensity of emotion.

[0011] Furthermore, the EEG time domain features include:

[0012] Mean potential: the average level of brain electrical activity during a set period of time.

[0013] Peak potential: the potential value at the highest and lowest points,

[0014] Waveform area: the area of ​​a signal waveform within a certain period of time.

[0015] EEG amplitude: the difference between the maximum and minimum amplitude of a signal;

[0016] The EEG frequency domain features include:

[0017] Power spectral density (PSD): The power distribution in each frequency band, commonly used for the energy characteristics of alpha waves, beta waves, and theta waves. Alpha waves are 8-12Hz, beta waves are 13-30Hz, and theta waves are 4-7Hz.

[0018] Spectrum center of mass: The center of mass position of the spectrum, indicating the center frequency of the spectrum.

[0019] Frequency bandwidth: the frequency range of alpha waves and beta waves;

[0020] The EEG nonlinear characteristics include:

[0021] Approximate entropy (ApEn): measures signal complexity and irregularity,

[0022] Sample entropy (SampEn): Similar to approximate entropy, but the calculation is more stable.

[0023] Fractal dimension: self-similarity and complexity characteristics of signals.

[0024] Furthermore, the ECG time domain features include:

[0025] RR interval: the time interval between two adjacent R wave peaks.

[0026] HRV time domain indicators: including standard deviation (SDNN), mean RR interval (mRR), standard deviation of adjacent RR intervals (SDSD), QT interval: the time interval from the start of the Q wave to the end of the T wave;

[0027] The ECG frequency domain features include:

[0028] HRV frequency domain indicators: including low frequency LF of 0.04-0.15Hz, high frequency HF of 0.15-0.4Hz and ratio LF / HF;

[0029] Spectrum centroid: The centroid position of the ECG signal spectrum.

[0030] Band energy: energy within the frequency bands LF and HF;

[0031] The ECG nonlinear characteristics include:

[0032] Approximate entropy (ApEn): measures the complexity of ECG signals,

[0033] Sample entropy (SampEn): irregularity of ECG signal.

[0034] Furthermore, the gastric electrical time domain features include:

[0035] Basic frequency: The main frequency component of the gastric electrical signal, generally 3 times / minute (0.05Hz),

[0036] Waveform amplitude: the maximum and minimum amplitude of the gastric electrical signal,

[0037] Slow wave period: the periodic characteristics of slow waves;

[0038] The gastroelectric frequency domain features include:

[0039] Power spectral density (PSD): the energy distribution of gastric electrical signals in the frequency domain.

[0040] Spectrum peak: the main frequency peak of the gastric electrical signal,

[0041] Frequency bandwidth: set the frequency range within the frequency band;

[0042] The gastric electrical nonlinear characteristics include:

[0043] Approximate entropy (ApEn): the complexity of gastric electroencephalogram signals,

[0044] Sample entropy (SampEn): irregularity of gastric electroencephalogram signal,

[0045] Fractal dimension: self-similarity of gastric electroencephalogram signals;

[0046] Furthermore, the multi-input deep learning network of the feature fusion unit consists of a convolutional neural network and a long short-term memory network.

[0047] Furthermore, the emotion recognition unit is a deep learning network, and the emotion assessment unit is a machine learning network.

[0048] Furthermore, the deep learning network of the emotion recognition unit is a fully connected layer and an activation function; and the machine learning network of the emotion assessment unit is a support vector machine SVM.

[0049] By integrating EEG, ECG, and innovatively incorporating electrogastric (EGG) signals that are closely related to nervous system activity, the system can provide a comprehensive emotional assessment. Each of these signals represents a different emotional response mechanism, and the fusion of multimodal signals can more accurately capture emotional changes and improve the accuracy and reliability of emotion detection. The system can collect and analyze the user's physiological signals in real time, provide instant emotional feedback, and help users identify emotional fluctuations at an early stage and take corresponding measures to avoid emotional out-of-control or long-term depression. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flowchart of the workflow.

[0051] Figure 2 This is a diagram of the working principle of using CNN-LSTM to recognize emotions. DETAILED DESCRIPTION

[0052] To achieve the above objectives, the present invention adopts a wearable EEG, ECG, and GEG acquisition module to measure electrical signals of the human nervous system, obtain EEG, ECG, and GEG signals, and measure characteristic parameters related to emotions in the signals; then a multi-input deep learning algorithm is used for feature fusion, and finally machine learning (ML) and deep learning (DL) algorithms are used for emotion classification and evaluation.

[0053] The features extracted from the signal include the mean EEG potential: the average level of EEG activity over a certain period of time.

[0054] The features extracted from the signal include EEG peak potential: the potential values ​​at the highest and lowest points.

[0055] The features extracted from the signal include the area of ​​the EEG waveform: the area of ​​the signal waveform within a certain period of time.

[0056] Features extracted from the signal include EEG wave amplitude: the difference between the maximum and minimum amplitudes of the signal.

[0057] The features extracted from the signal include EEG power spectral density (PSD): power distribution in each frequency band, commonly used for energy characteristics of alpha waves (8-12 Hz), beta waves (13-30 Hz), and theta waves (4-7 Hz).

[0058] The features extracted from the signal include the centroid of the EEG spectrum: the centroid position of the spectrum, indicating the center frequency of the spectrum.

[0059] The features extracted from the signal include EEG frequency bandwidth: the frequency range within a specific frequency band, such as the bandwidth of alpha waves and beta waves.

[0060] Features extracted from the signal include EEG approximate entropy (ApEn): a measure of signal complexity and irregularity.

[0061] The features extracted from the signal include EEG sample entropy (SampEn): similar to approximate entropy, but more stable in calculation.

[0062] The features extracted from the signal include EEG fractal dimension: self-similarity and complexity characteristics of the signal.

[0063] The features extracted from the signal include the ECG RR interval: the time interval between two adjacent R wave peaks.

[0064] The features extracted from the signal include ECG HRV time domain indicators: including standard deviation (SDNN), mean RR interval (mRR), standard deviation of adjacent RR interval differences (SDSD), etc.

[0065] The features extracted from the signal include the electrocardiogram QT interval: the time interval from the start of the Q wave to the end of the T wave.

[0066] The features extracted from the signal include ECG HRV frequency domain indicators: including low frequency (LF, 0.04-0.15 Hz), high frequency (HF, 0.15-0.4 Hz) and LF / HF ratio.

[0067] The features extracted from the signal include the centroid of the ECG spectrum: the centroid position of the ECG signal spectrum.

[0068] The features extracted from the signal include ECG frequency band energy: energy within a specific frequency band (eg, LF, HF).

[0069] Features extracted from the signal include ECG Approximate Entropy (ApEn): a measure of the complexity of the ECG signal.

[0070] The features extracted from the signal include ECG sample entropy (SampEn): irregularity of the ECG signal.

[0071] The features extracted from the signal include the basic gastric electroencephalogram frequency: the main frequency component of the gastric electroencephalogram signal, generally 3 times / minute (0.05 Hz).

[0072] The features extracted from the signal include the electrogastric waveform amplitude: the maximum and minimum amplitudes of the electrogastric signal.

[0073] The features extracted from the signal include the period of the slow wave of gastric electricity: the periodic characteristics of the slow wave.

[0074] The features extracted from the signal include gastric electrical power spectral density (PSD): energy distribution of gastric electrical signals in the frequency domain.

[0075] The features extracted from the signal include the electrogastric spectrum peak: the main frequency peak of the electrogastric signal.

[0076] Features extracted from the signal include electrogastric frequency bandwidth: the range of frequencies within a specific frequency band.

[0077] The features extracted from the signal include gastric electroencephalogram (ApEn): the complexity of the gastric electroencephalogram (GEG) signal.

[0078] The features extracted from the signal include gastric electroencephalogram (SampEn): irregularity of gastric electroencephalogram (GEG) signals.

[0079] The features extracted from the signal include gastric electrographic fractal dimension: the self-similarity of gastric electrographic signals.

[0080] The CNN-LSTM algorithm is used for feature fusion: (CNN part) Spatial feature extraction: The preprocessed physiological signals are converted into a two-dimensional form suitable for CNN processing (such as time-frequency diagram). CNN is used to extract spatial features such as local patterns, spectral features, etc. from these two-dimensional representations. Convolutional layer: Use convolutional layers and pooling layers to extract and compress features to capture important patterns and features in the signal.

[0081] Time Series Modeling (LSTM part) Sequence Modeling: Input the feature sequences obtained from the CNN part into the LSTM network. LSTM processes the temporal dependencies in these sequences and captures the dynamic characteristics of emotional states changing over time. Long-term dependencies: LSTM is able to remember and update long-term dependent information, which is very important for capturing changes in emotional states.

[0082] Emotion classification: Classification layer: After the LSTM layer, a fully connected layer and an activation function are usually added to map the extracted features to specific emotion categories. Prediction: Emotion prediction is performed through the trained model, and the emotion category corresponding to the physiological signal (such as happiness, sadness, anger, etc.) is output.

[0083] Use ML and DL algorithms (such as SVM) for emotion assessment. SVM is a supervised learning algorithm that is particularly suitable for classification tasks. After selecting the features, choose the kernel function. SVM can use different kernel functions (such as linear kernel, polynomial kernel, radial basis kernel (RBF) etc.). Choosing a kernel function that fits the data can improve classification performance. Training model: Use labeled emotion data (physiological signals with known emotion labels) to train the SVM model. The model will learn how to predict the intensity of emotion based on the input features.

[0084] Emotion recognition uses machine learning models (ML) and deep learning (DL) models. Machine learning models: such as support vector machines (SVM) classify emotions based on comprehensive features. Deep learning models: such as convolutional neural networks (CNN) and recurrent neural networks (RNN), etc., perform emotion recognition. Clinical psychological assessment methods (such as psychological scales POMS and SCL90) are used to test the credibility of emotion recognition. Feedback can be provided to users when abnormal or negative emotions and psychological states are detected.

[0085] Parameter tuning: Adjust the hyperparameters of SVM (such as C parameters and kernel function parameters) to optimize model performance. Cross-validation is usually used to select the best parameter combination. Use the trained SVM model to predict emotions on new physiological signal data. Input the feature vector and the model outputs the predicted emotion category.

Claims

1. A wearable emotion monitoring and evaluation system based on brain-heart-enteric nervous system electrical signals, the system comprising: a signal acquisition and preprocessing unit, a feature extraction unit, a feature fusion unit, an emotion recognition unit, and an emotion evaluation unit; The signal acquisition and preprocessing unit includes: Collect and preprocess the EEG, ECG and EG signals, and then transmit the preprocessed signals to the feature extraction unit; The feature extraction unit extracts features from the received EEG, ECG and EGG signals, and transmits the extracted features to the feature fusion unit; The features extracted by the feature extraction unit include: EEG features, ECG features, and GAST features; The EEG features include: EEG time domain features, EEG frequency domain features, and EEG nonlinear features; The ECG characteristics include: ECG time domain characteristics, ECG frequency domain characteristics, and ECG nonlinear characteristics; The gastric electrical characteristics include: gastric electrical time domain characteristics, gastric electrical frequency domain characteristics, and gastric electrical nonlinear characteristics; The feature fusion unit is a multi-input deep learning network that fuses all features and then inputs the fused features into the emotion recognition unit and the emotion assessment unit; The emotion recognition unit recognizes the type of emotion, and the emotion evaluation unit recognizes the intensity of emotion.

2. A wearable emotion monitoring and evaluation system based on brain-heart-enteric nervous system electrical signals as claimed in claim 1, characterized in that: The EEG time domain features include: Mean potential: the average level of brain electrical activity during a set period of time. Peak potential: the potential value at the highest and lowest points, Waveform area: the area of ​​a signal waveform within a certain period of time. EEG amplitude: the difference between the maximum and minimum amplitude of a signal; The EEG frequency domain features include: Power spectral density: The power distribution in each frequency band, commonly used for the energy characteristics of alpha waves, beta waves, and theta waves. Alpha waves are 8-12Hz, beta waves are 13-30Hz, and theta waves are 4-7Hz. Spectrum center of mass: The center of mass position of the spectrum, indicating the center frequency of the spectrum. Frequency bandwidth: the frequency range of alpha waves and beta waves; The nonlinear characteristics of EEG include: approximate entropy, sample entropy, and fractal dimension.

3. A wearable emotion monitoring and evaluation system based on brain-heart-enteric nervous system electrical signals as claimed in claim 1, characterized in that: The ECG time domain features include: RR interval: the time interval between two adjacent R wave peaks. HRV time domain indicators: including standard deviation, average RR interval, standard deviation of the difference between adjacent RR intervals, QT interval: the time interval from the beginning of the Q wave to the end of the T wave; The ECG frequency domain features include: HRV frequency domain indicators: including low frequency LF of 0.04-0.15Hz, high frequency HF of 0.15-0.4Hz and ratio LF / HF; Spectrum centroid: The centroid position of the ECG signal spectrum. Band energy: energy within the frequency bands LF and HF; The ECG nonlinear characteristics include: approximate entropy and sample entropy.

4. A wearable emotion monitoring and evaluation system based on brain-heart-enteric nervous system electrical signals as claimed in claim 1, characterized in that: The gastric electrical time domain characteristics include: basic frequency, waveform amplitude, and slow wave period; The gastroelectric frequency domain characteristics include: power spectrum density, spectrum peak, and frequency bandwidth; The gastric electrical nonlinear characteristics include: approximate entropy, sample entropy, and fractal dimension.

5. A wearable emotion monitoring and evaluation system based on brain-heart-enteric nervous system electrical signals as claimed in claim 1, characterized in that: The multi-input deep learning network of the feature fusion unit consists of a convolutional neural network and a long short-term memory network.

6. A wearable emotion monitoring and evaluation system based on brain-heart-enteric nervous system electrical signals as claimed in claim 1, characterized in that: The emotion recognition unit is a deep learning network, and the emotion assessment unit is a machine learning network.

7. A wearable emotion monitoring and evaluation system based on brain-heart-enteric nervous system electrical signals as claimed in claim 1, characterized in that: The deep learning network of the emotion recognition unit is a fully connected layer and an activation function; the machine learning network of the emotion assessment unit is a support vector machine SVM.

Images