An emotion recognition system fusing heart rate variability and prefrontal electroencephalogram signals

Abstract

A system for emotion recognition integrating heart rate variability (HRV) and prefrontal EEG signals is disclosed. The system comprises: an information acquisition module, including an HRV signal acquisition unit and a prefrontal EEG signal acquisition unit. The HRV signal acquisition unit acquires HRV signals via an acquisition device, and the prefrontal EEG signal acquisition unit acquires EEG signal 1 (F3) and EEG signal 2 (F4) via an EEG acquisition device; and a preprocessing module connected to the information acquisition module, which receives the HRV signals and performs preprocessing on the EEG signal 1 and EEG signal 2 respectively, generating a preprocessed HRV signal, preprocessed EEG signal 1, and preprocessed EEG signal 2, which are then fed back to the feature extraction module. The system fully covers five frequency bands of EEG from 0.5 to 100 Hz, providing more comprehensive emotion features.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to an emotion recognition system that integrates heart rate variability and prefrontal cortex electroencephalogram (EEG) signals. Background Technology

[0002] Emotional state recognition is a crucial foundational capability in applications such as mental health monitoring, human-computer interaction, intelligent companionship devices, and mental health screening. Traditional emotion recognition methods primarily rely on three types of information: facial expression recognition (FER), speech acoustic feature recognition, and physiological signal recognition (such as heart rate and skin conductance). In recent years, increasing research has shown that: Heart rate variability (HRV) reflects autonomic nervous system activity and is closely related to stress, anxiety, and mood fluctuations; prefrontal cortex EEG (F3 and F4 regions, 0.5-150 Hz frequency band) is closely related to emotion regulation, emotional bias, and willpower control, among which... Delta (0.5–4Hz): arousal level, deep emotional response; Theta (4–8Hz): Emotion regulation, attention; Alpha (8–12 Hz): Emotional approach-avoidance (lateralization feature), alpha asymmetry (FAA) is a well-established indicator in emotion neuroscience; Beta (12–30Hz): Tension, anxiety, mental overload; Gamma (30–150Hz): Cognitive integration, emotional activation.

[0003] Single-modal feature discrimination is limited; HRV is easily affected by movement and breathing; EEG is particularly susceptible to noise; accuracy when used alone is often only 70%–85%. EEG uses a limited frequency band, mostly only using alpha waves or alpha / beta bands, failing to cover the full Delta–Gamma band information of 0.5–150Hz, resulting in incomplete feature representation; lack of neuroscience-based weighting mechanisms; simple feature splicing cannot reflect the connection mechanism between prefrontal lateralization and the autonomic nervous system; insufficient multimodal fusion; mostly using simple splicing or early fusion, failing to deeply utilize the F3 / F4 emotion lateralization mechanism, and failing to build a deep fusion model that can simultaneously process the multidimensional spatial structure of HRV temporal sequence and EEG frequency bands; weak generalization ability of algorithm models; lack of modeling for the prefrontal emotion network (left-side approach emotion, right-side avoidance emotion), making the recognition results susceptible to individual differences; overall accuracy is insufficient; the general accuracy of existing publicly available solutions is difficult to exceed 90%, which is insufficient to meet the requirements of mental health devices, child companion devices, and mental state monitoring devices. Summary of the Invention

[0004] The purpose of this invention is to provide an emotion recognition system that integrates heart rate variability and prefrontal cortex electroencephalogram (EEG) signals to address the problem of insufficient overall accuracy mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: An emotion recognition system integrating heart rate variability and prefrontal cortex electroencephalogram (EEG) signals, characterized by comprising: The information acquisition module includes an HRV signal acquisition unit and a prefrontal EEG signal acquisition unit. The HRV signal acquisition unit acquires HRV signals through an acquisition device, and the prefrontal EEG signal acquisition unit acquires EEG signal one of F3 and EEG signal two of F4 through an EEG acquisition device. The preprocessing module, connected to the information acquisition module, is used to receive the HRV signal and the first and second brainwave signals respectively, perform preprocessing, and generate the preprocessed HRV signal, the preprocessed first brainwave signal and the preprocessed second brainwave signal, which are then fed back to the feature extraction module. The feature extraction module, connected to the preprocessing module, is used to receive preprocessed HRV signals, preprocessed EEG signal 1, and preprocessed EEG signal 2, respectively, and perform feature extraction to obtain feature vector 1, feature 2, and feature 3, which are then fed back to the multimodal fusion module. Specifically, feature vector 1 is extracted from the preprocessed HRV signals, feature 2 is extracted from the preprocessed EEG signal 1, and feature 3 is extracted from the preprocessed EEG signal 2. The multimodal fusion module is connected to the feature extraction module. The multimodal fusion module converts feature 2 and feature 3 and feature vector 1 into a feature token sequence, and obtains a full-modal emotion representation vector through Transformer encoding, which is then fed back to the emotion classification module. The emotion classification module, connected to the multimodal fusion module, receives the full-modal emotion representation vector, classifies it, and outputs emotion labels, which are then fed back to the output module. The output module, connected to the emotion classification module, is used to receive emotion tags and their corresponding emotion intensities.

[0006] Preferably, the acquisition device includes at least one of ECG or PPG.

[0007] Preferably, the HRV signal acquisition unit includes: The target object's continuous RR interval sequence is acquired by ECG or PPG, and the HRV signal is output.

[0008] Preferably, the prefrontal EEG signal acquisition unit: The EEG signal 1 for F3 and the EEG signal 2 for F4 were collected using an EEG acquisition device, with a signal frequency range of 0.5-150Hz.

[0009] Preferably, the preprocessing module: The HRV signal is sequentially subjected to filtering, baseline drift removal, peak detection, RR / IBIS sequence reconstruction, and resampling / interpolation to obtain a preprocessed HRV signal; The EEG signal 1 and EEG signal 2 are sequentially subjected to raw examination, bandpass filtering, artifact detection and removal, independent component analysis for artifact removal, short-time segmentation / windowing, power spectral density calculation, and time-series / statistical feature aggregation to obtain preprocessed EEG signal 1 and preprocessed EEG signal 2. The preprocessed EEG signal 1 and preprocessed EEG signal 2 respectively include Delta, Theta, Alpha, Beta, and Gamma.

[0010] Preferably, the first feature vector includes time-domain features, frequency-domain features, and nonlinear features. The time-domain features include SDNN and RMSSD, the frequency-domain features include LF, HF, and LF / HF, and the nonlinear features include SD1 and SD2. The second and third features include five-band absolute power, five-band relative power, left-right lateralization index, selectable frequency band ratio, and statistical features.

[0011] Preferably, the multimodal fusion module includes five-band EEG energy feature extraction, construction of multi-band side-modulation parameters, adaptive fusion weight model, final fusion feature and Transformer multimodal coding. The specific implementation steps of the multimodal fusion module are as follows: S1. Five-band EEG energy feature extraction: Extract the energy of feature two in the five frequency bands Delta, Theta, Alpha, Beta, and Gamma to obtain the first energy feature vector of feature two. Also, extract the energy of feature three in the five frequency bands Delta, Theta, Alpha, Beta, and Gamma to obtain the second energy feature vector of feature two. S2. Construct multi-band side-shifting parameters: Combine the energy feature vector one and the energy feature vector two from step S1 with the side-shifting exponent formula to form side-shifting vector one; S3, Adaptive Fusion Weight Model: The fusion weight one is calculated by using the sigmoid function to calculate the fusion weight one of the side-shifted vector one in step S2, and the fusion weight two of the feature vector one is 1 minus the fusion weight one. S4. Final fusion features: The final fusion features are obtained by calculating the energy feature vector 1, feature vector 2, side-shifting vector 1, fusion weight 1, fusion weight 2, and feature vector 1. S5, Transformer Multimodal Encoding: Input feature vector 1, energy feature vector 1, energy feature vector 2, side-shifting vector 1, and the final fused feature into Transformer encoding, and finally output a full-modal sentiment representation vector.

[0012] The beneficial effects of this invention are: 1. This invention system acquires EEG signals from the prefrontal F3 and F4 channels in five frequency bands: Delta (0.5–4Hz), Theta (4–8Hz), Alpha (8–12Hz), Beta (12–30Hz), and Gamma (30–150Hz), comprehensively covering all frequency bands related to emotional states, overcoming the limitations of existing technologies that only use alpha waves or limited frequency bands. By extracting the absolute power, relative power, and statistical characteristics of each frequency band, it significantly enhances the ability to express multidimensional emotional states.

[0013] 2. This invention is based on the prefrontal lateralization theory. It dynamically calculates the adaptive fusion weights between HRV and EEG features using a lateralization index and a sigmoid function, enabling the model to automatically adjust the contribution ratio of HRV and EEG according to the degree of lateralization in different frequency bands. This mechanism fully reflects the neural coupling relationship between the autonomic nervous system and the prefrontal emotion regulation network, enhancing the model's adaptability to individual differences.

[0014] 3. This invention employs a Transformer to jointly encode HRV features, EEG five-band energy features, side-shifting vectors, and adaptive weighted fusion features, achieving deep fusion of HRV temporal information and EEG frequency band spatial information. Actual testing shows that the overall emotion recognition accuracy of this system reaches 96.4%, significantly outperforming single-modality (HRV: 81.1%; EEG: 88.6%) and traditional simple splicing methods.

[0015] 4. The system of this invention exhibits stable recognition performance in patients with depressive disorders, bipolar disorder, individuals with high anxiety, and normal controls. Quantifiable and significant differences are found between the clinically diagnosed group and the normal group in dimensions such as depressive mood, mental fatigue, and psychological anxiety. The system demonstrates good resistance to individual differences, signal noise, and changes in the acquisition environment, and exhibits excellent cross-population generalization performance.

[0016] 5. The system of this invention supports ECG or PPG acquisition of HRV signals, is compatible with a variety of lightweight sensors, has an average recognition latency of only 0.8 seconds, and a recognition accuracy fluctuation of less than 1% after 24 hours of continuous operation. These low latency, low power consumption, and high stability features make it suitable for deployment in wearable devices, children's psychological companionship devices, and other scenarios. Attached Figure Description

[0017] Figure 1 This is a system diagram of an embodiment of the present invention; Figure 2 This is a comparison chart of the accuracy of prior art and the present invention in the embodiments of the present invention; Figure 3 This is a schematic diagram of the synchronous acquisition of F3 / F4 EEG and HRV features according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figures 1-3 This invention provides an emotion recognition system that integrates heart rate variability and prefrontal cortex electroencephalogram (EEG) signals, characterized by comprising: The information acquisition module includes an HRV signal acquisition unit and a prefrontal EEG signal acquisition unit. The HRV signal acquisition unit acquires HRV signals through an acquisition device, and the prefrontal EEG signal acquisition unit acquires EEG signal one of F3 and EEG signal two of F4 through an EEG acquisition device. The preprocessing module, connected to the information acquisition module, is used to receive the HRV signal and the first and second brainwave signals respectively, perform preprocessing, and generate the preprocessed HRV signal, the preprocessed first brainwave signal and the preprocessed second brainwave signal, which are then fed back to the feature extraction module. The feature extraction module, connected to the preprocessing module, is used to receive preprocessed HRV signals, preprocessed EEG signal 1, and preprocessed EEG signal 2, respectively, and perform feature extraction to obtain feature vector 1, feature 2, and feature 3, which are then fed back to the multimodal fusion module. Specifically, feature vector 1 is extracted from the preprocessed HRV signals, feature 2 is extracted from the preprocessed EEG signal 1, and feature 3 is extracted from the preprocessed EEG signal 2. The multimodal fusion module is connected to the feature extraction module. The multimodal fusion module converts feature 2 and feature 3 and feature vector 1 into a feature token sequence, and obtains a full-modal emotion representation vector through Transformer encoding, which is then fed back to the emotion classification module. The emotion classification module, connected to the multimodal fusion module, receives the full-modal emotion representation vector, classifies it, and outputs emotion labels, which are then fed back to the output module. The output module, connected to the emotion classification module, is used to receive emotion tags and their corresponding emotion intensities.

[0020] Specifically, the feature is that the acquisition device includes at least one of ECG or PPG.

[0021] Specifically, the HRV signal acquisition unit includes: The target object's continuous RR interval sequence is acquired by ECG or PPG, and the HRV signal is output.

[0022] Specifically, the prefrontal EEG signal acquisition unit: The EEG signal 1 for F3 and the EEG signal 2 for F4 were collected using an EEG acquisition device, with a signal frequency range of 0.5-150Hz.

[0023] Specifically, the preprocessing module: The HRV signal is sequentially subjected to filtering, baseline drift removal, peak detection, RR / IBIS sequence reconstruction, and resampling / interpolation to obtain a preprocessed HRV signal; The EEG signal 1 and EEG signal 2 are sequentially subjected to raw examination, bandpass filtering, artifact detection and removal, independent component analysis for artifact removal, short-time segmentation / windowing, power spectral density calculation, and time-series / statistical feature aggregation to obtain preprocessed EEG signal 1 and preprocessed EEG signal 2. The preprocessed EEG signal 1 and preprocessed EEG signal 2 respectively include Delta, Theta, Alpha, Beta, and Gamma.

[0024] The detailed process of the preprocessing module for brainwave signal one and brainwave signal two is as follows: 1. Original examination: Check the sampling rate, missing segments, and channel impedance records.

[0025] If the channel is disconnected or a large amount of data is lost, it is marked as BAD.

[0026] 2. Bandpass filtering: Design: 0.5–100Hz bandpass filter (to preserve Delta–Gamma).

[0027] Recommended filters: 4th order Butterworth IIR (bidirectional zero-phase FIR FIR implementation), or FIR (linear phase) window function design (e.g., Hamming).

[0028] Simultaneously perform 50 / 60Hz power frequency notch filtering (IIRnotch, Q=30–35) or multi-order notch filtering to remove power frequency and its harmonics (50 / 60, 100 / 120Hz).

[0029] 3. Artifact Detection and Removal (Coarse-grained): Time domain threshold: If the absolute voltage is >100–200 μV, it is marked as a possible artifact segment (the threshold for typical electrooculography or motion artifacts can be set to 100 μV).

[0030] Short-term translation / marking: Interpolation can be performed for short-term jumps (<200ms); for longer artifact segments, they are directly marked as missing, and subsequent windows skip or interpolate them.

[0031] 4. Independent Component Analysis (ICA) artifact removal: Use FastICA or InfomaxICA to run on each segment of clean data (typically ICA is performed on an entire session or several minutes of data).

[0032] Identifying artifact components: ICA components that are highly correlated with EOG channels / electromyogenic channels; Component time-domain / frequency-domain characteristics (high amplitude low frequency is blinking, and a large amount of energy in 20–80Hz is electromyography); Kurtosis or manual / semi-automatic labels (such as SASICA, ICLabel) can be used to assist in identification.

[0033] Reconstruct the EEG after removing artifact components.

[0034] If ICA is not available (lightweight equipment), linear regression can be used to remove artifacts based on EOG or waveform thresholding.

[0035] 5. Short-time segmentation / window partitioning: The reconstructed EEG is cut into short windows (e.g., 2s, 50% overlap) for PSD calculation.

[0036] Each short window must have a low artifact percentage (e.g., <20%), otherwise the window will be discarded or marked as low quality.

[0037] 6. Power Spectral Density (PSD) Calculation: Recommended method: Welch method (e.g., window length 2s, Hamming window, 50% overlap, FFT length 512 / 1024 depending on the sampling rate).

[0038] Calculate the PSD of each channel on each window, and integrate to obtain the energy of each frequency band: Delta: 0.5–4Hz Theta: 4–8Hz Alpha: 8–12Hz Beta: 12–30Hz Gamma: 30–100Hz Frequency band energy.

[0039] It can calculate absolute power (μV²) and relative power (bandpower / full band 0.5–100Hz).

[0040] 7. Time series / statistical feature aggregation (corresponding to HRV window): Within an HRV window (e.g., 30 seconds), perform statistical analysis on the PSDs of all short windows contained within that window: mean, variance, skewness, peak value, etc.

[0041] Calculate the left and right lateralization index (perband): where (i) is the frequency band (D,T,A,B,G).

[0042] 8. Other optional features: Inter-band ratio (e.g., Beta / Alpha, Theta / Alpha); Phase lock value, power spectrum slope (1 / f exponential estimate); Time-frequency characteristics (instantaneous energy of short-time wavelet transform); Mutual information / coherence (F3-F4 coherence); Output (EEG features, units, and format); For each HRV time window (e.g., 30 seconds), output the following vector (example): Basic energy (absolute power): F3_Delta_abs,F3_Theta_abs,F3_Alpha_abs,F3_Beta_abs,F3_Gamma_abs(μV²) F4 - Same as above (μV²) Relative power: F3_Delta_rel…F4_Gamma_rel (percentage 0–1) Side-shifting indicators (5): L_D, L_T, L_A, L_B, L_G (unitless, -1..1) Bandwidth ratio (optional): Beta / Alpha(F3,F4),Theta / Alpha, etc. Statistical summary (per frequency band): mean, std, skew (numerical value) Overall EEG feature dimension estimation (example): Absolute power: 2 (channels) × 5 (bands) = 10 Relative power: 10 Side-channeling index: 5 Ratio / statistics: approximately 10–15 => ~35–40 dimensions per window of EEG feature vectors (scalable).

[0043] The detailed processing flow of the HRV signal by the preprocessing module is as follows: 1. Filtering: ECG: Bandpass 0.5–40Hz (4th order Butterworth bidirectional), while removing power frequency (50 / 60Hz notch).

[0044] PPG: Low-pass filtering removes high-frequency noise (e.g., cutoff ~8Hz), followed by smoothing.

[0045] 2. Baseline drift removal: Use a high-pass filter (0.5Hz) or morphological filter / polynomial detrending process.

[0046] 3. Peak detection (R-peak / PPG peak): ECG recommended methods: Pan–Tompkins algorithm (including differential-squared-integral window-threshold detection), or detection based on continuous wavelet transform (CWT) is more robust.

[0047] PPG peak detection: First, smooth the peaks by taking the first derivative / second derivative plus a threshold / minimum peak spacing (to prevent false detections).

[0048] Example parameters: minimum heart rate interval 300ms (protection for maximum HR=200bpm), minimum amplitude threshold set according to signal-to-noise ratio (or use adaptive threshold).

[0049] 4. RR / IBIS sequence reconstruction: Extract the time difference between adjacent peaks: RR_i = t_i + 1 - t_i (unit: s or ms).

[0050] Quality control: Remove abnormal intervals (<300ms or >2000ms), or use median filtering / local replacement.

[0051] For missing / outlier points, linear or spline interpolation can be used to fill in the gaps (for short missing points), but the interpolation segments should be labeled.

[0052] 5. Resampling / Interpolation (for frequency domain analysis or EEG alignment): To perform frequency domain HRV (LF / HF), it is common practice to first establish an interpolated heart rate time series (RR to evenly sampled series), for example, interpolated to 4Hz or 10Hz, and then perform PSD.

[0053] Recommendation: Interpolate to 4Hz (for short-time spectrum analysis).

[0054] 6. HRV time-domain characteristics (commonly used): SDNN (Standard Deviation): RMSSD: opNN50: Percentage of adjacent differences > 50ms (%) oMeanRR / MeanHR (ms / bpm) 7. HRV frequency domain characteristics: Calculate the PSD (Welch) and integrate to obtain the LF and HF energies: LF: 0.04–0.15Hz HF: 0.15–0.4Hz LF_power, HF_power, LF / HF. Units: ms² or normalized units.

[0055] Note: For short windows (such as 30s), LF band estimation is unstable. A 60s window is recommended to improve LF stability. If a 30s window is used, the uncertainty of the results should be marked.

[0056] 8. Nonlinearity and complexity characteristics: Poincaréplot: SD1, SD2 (ms) SampleEntropy(SampEn),ApproximateEntropy(ApEn) DetrendedFluctuationAnalysis(DFAα1 / α2) 9. Quality control and noise labeling: Calculate the peak jitter / distortion rate (e.g., peak correlation coefficient) within the narrow window; if the quality is low, discard the entire window or reduce the label weight.

[0057] Output (HRV characteristics, units and format) Output vector for each HRV window (e.g., 30s / 60s) (example): Time domain: MeanRR (ms), SDNN (ms), RMSSD (ms), pNN50 (%) Frequency domain: LF(ms²),HF(ms²),LF_norm,HF_norm,LF / HF Nonlinear: SD1(ms), SD2(ms), SampEn, DFA_alpha1 Other: MeanHR(bpm), RR_count Overall HRV feature dimension estimation: approximately 10–15 dimensions.

[0058] Specifically, the first feature vector includes time-domain features, frequency-domain features, and nonlinear features. The time-domain features include SDNN and RMSSD, the frequency-domain features include LF, HF, and LF / HF, and the nonlinear features include SD1 and SD2. The second and third features include five-band absolute power, five-band relative power, left-right lateralization index, selectable band ratio, and statistical features.

[0059] Wherein, the SDNN (RR interval standard deviation) Objective: To determine the overall heart rate variability index, reflecting the combined activity capacity of the sympathetic and parasympathetic nervous systems.

[0060] Calculation method: Unit: ms Input: The actual RR sequence in the current window Features: Sensitive to sequence length, therefore a fixed window length (e.g., 30 / 60s) is very important.

[0061] RMSSD (Root Mean Square Difference of Adjacent RR Differences) Objective: To reflect parasympathetic (vagus nerve) activity, which is closely related to short-term mood fluctuations.

[0062] Calculation method: Unit: ms. Sensitive to high-frequency HRV; suitable for short time windows (>30 seconds). 1.3 Frequency Domain HRV: LF, HF, LF / HF Interpolation and PSD To obtain LF / HF, the RR sequence needs to be converted into an equally spaced sequence (e.g., 4 Hz interpolation) before calculating the PSD.

[0063] method: Welch method (Hamming window recommended, 50% overlap) The number of FFT points is automatically selected based on the interpolation sampling rate (typically 256–512). Frequency band definition: LF: 0.04 – 0.15 Hz (low frequency) HF: 0.15 – 0.4 Hz (high frequency) calculate: Force ratio LF / HF: Reflects the balance between sympathetic and parasympathetic nervous systems (usually elevated when sympathetic levels are high). Optional: Output normalization LF_norm / HF_norm 1.4 Nonlinear HRV: SD1, SD2 (Poincaré Plot) The Poincaré scatter plot is constructed with points (RR_i, RR_{i+1}).

[0064] set up: but: significance: SD1: Reflects short-term parasympathetic regulation SD2: Reflects long-term variability (a combined sympathetic and parasympathetic index). 2. EEG (F3 / F4) Five-Band Feature Extraction Method Based on the clean EEG signal obtained in the preprocessing stage, short window segmentation (e.g., 2 s, 50% overlap) is first performed, then the power spectral density (PSD) of each window is calculated, and finally all windows are statistically summarized.

[0065] 2.1 Power Spectral Density (PSD) Calculation Logic Window configuration: Short window length: 1–4 s (recommended: 2 s) Overlap: 50% FFT: 512 or dynamically adjusted according to the sampling rate Method: Welch(Hamming window) Frequency band definition: Single-window frequency band energy calculation: Calculate F3 and F4 separately.

[0066] 2.2 Feature 1: Five-band absolute power The energy of all short windows within a large window (e.g., 30 seconds) is averaged: Output: F3_Delta_abs F3_Theta_abs F3_Alpha_abs F3_Beta_abs F3_Gamma_abs F4 corresponds to five frequency bands Total: 10 features Unit: μV² 2.3 Feature 2: Five-band relative power Each channel outputs five relative power metrics (0–1).

[0067] Output: F3_Delta_rel … F3_Gamma_rel F4_Delta_rel … F4_Gamma_rel, a total of 10 features. 2.4 Feature 3: Lateralization Index It is used to reflect the bias differences between the left and right frontal lobes (key areas for emotion regulation) in different frequency bands.

[0068] formula: Value range: -1 to +1 Positive value: Stronger activation on the right side Negative values: Stronger activation on the left side Output: L_Delta; L_Theta; L_Alpha; L_Beta; L_Gamma; A total of 5 features Value: In particular, the left-right lateralization of alpha (FAA) is highly correlated with the emotional approach / avoidance model; Commonly used in clinical and emotion computing.

[0069] 2.5 Feature 4: Band Ratios (Optional) Common combination: Theta / Alpha: fatigue level, arousal level Beta / Alpha: Anxiety and Tension Gamma / Beta: Higher-order cognitive processing Formula example: If needed, I can add them to the final feature table.

[0070] 2.6 Feature 5: Statistical Features (Floating Values ​​within the Window) For example, calculating for all short windows: Mean (already used for absolute power) Standard deviation Kurtosis Coefficient of variation CV = std / mean It reflects the stability of the state.

[0071] For example: It is usually used to enrich model inputs, but is not necessary.

[0072] 3. Feature Output Format (Final Feature Vector) Output one line of features for each time window (e.g., 30 seconds), including: HRV (7 units) SDNN, RMSSD, LF, HF, LF / HF, SD1, SD2 EEG (F3 / F4) five-band (25–30 bands) F3 / F4: Absolute power across five frequency bands (10 bands) F3 / F4: Relative power across five frequency bands (10 bands) Lateralization index (5 items) Optional: Band ratios (2–6) Optional: Statistics (if enabled) The final multimodal vector typically has a length of about 35–45 dimensions, depending on whether ratios / statistics are enabled.

[0073] Specifically, the preprocessing for alignment and fusion of EEG and HRV. 1. Time alignment: Use the HRV window as the main time axis (e.g., every 30 seconds). Aggregate EEG short-window features (mean, variance, lateralization indices, etc.) within this window.

[0074] Ensure that the timestamp format is consistent (UTCms) and correct for delays (e.g., sensor time offset between PPG and EEG).

[0075] 2. Handling missing values: If a window has insufficient EEG / HRV data (e.g., artifacts account for >50%), it is marked as low quality and can be removed or imputed. However, the quality is reflected by sample weights during training.

[0076] 3. Normalization / Standardization: Two levels: Individual standardization (commonly used): For devices that require real-time adaptation, z-score standardization is performed using the subject baseline (quiet for 1–2 minutes).

[0077] Global standardization: Perform global z-score on the training set using computemean / std.

[0078] For most features, it is recommended to perform a log transformation first (energy / power is often right-skewed), such as ((E+)), and then perform a z-score.

[0079] 4. Feature splicing format: Generate a one-dimensional feature vector for each window: [HRV(≈12)||EEG(≈35)||meta(quality, timestamp)] → final dimension ≈47–60.

[0080] Saving as a CSV (one window per row) or a NumPy binary file (shape: [n_windows, n_features]) facilitates model training. It can also be saved as a protobuf or TFRecord.

[0081] Final data features Examples in output vector order (per window): 1. Timestamp start_time (ISO / epochms) 2. duration(s) 3. HRV characteristics (12): mean_RR_ms,mean_HR_bpm,SDNN_ms,RMSSD_ms,pNN50_pct,LF_ms2,HF_ms2,LF_norm,HF_norm,LF_HF_ratio,SD1_ms,SD2_ms 4. Absolute power of EEGF3 (5): F3_D_abs…F3_G_abs (μV²) 5. EEGF4 Absolute Power (5) 6. Relative power of EEGF3 (5) 7. Relative power of EEGF4 (5) 8. Side-shifting indices (5): L_D, L_T, L_A, L_B, L_G 9. EEG band ratios / statistics (6–10, optional): e.g., F3_Beta_Alpha, F4_Beta_Alpha, F3_Theta_Alpha, F4_Theta_Alpha, F3_power_std, F4_power_std 10. Signal quality score (HRV_quality, EEG_quality, 0–1) Example of total dimensions: 2+12+(5+5+5+5)+5+8+2≈52 features.

[0082] Specifically, the multimodal fusion module includes five-band EEG energy feature extraction, construction of multi-band side-modulation parameters, adaptive fusion weight model, final fusion features, and Transformer multimodal coding. The specific implementation steps of the multimodal fusion module are as follows: S1. Five-band EEG energy feature extraction: The energy of feature two in the five frequency bands Delta, Theta, Alpha, Beta, and Gamma is used to obtain the energy feature vector one of feature two. The energy of feature three in the five frequency bands Delta, Theta, Alpha, Beta, and Gamma is extracted to obtain the energy feature vector two of feature two. S2. Construct multi-band side-shifting parameters: Combine the energy feature vector one and the energy feature vector two from step S1 with the side-shifting exponent formula to form side-shifting vector one; S3, Adaptive Fusion Weight Model: The fusion weight one is calculated by using the sigmoid function to calculate the fusion weight one of the side-shifted vector one in step S2, and the fusion weight two of the feature vector one is 1 minus the fusion weight one. S4. Final fusion features: The final fusion features are obtained by calculating the energy feature vector 1, feature vector 2, side-shifting vector 1, fusion weight 1, fusion weight 2, and feature vector 1. S5, Transformer Multimodal Encoding: Input feature vector 1, energy feature vector 1, energy feature vector 2, side-shifting vector 1, and the final fused feature into Transformer encoding, and finally output a full-modal sentiment representation vector.

[0083] Based on the above implementation method, the specific operations are as follows: Example 1 Signal range: EEG: 0.5–100 Hz, five-band or HRV: ECG / PPG Subjects: 60 Stimulation methods: IAPS (Intra-Action Picture and Music) triggers Model: 6-layer multimodal Transformer.

[0084] result: HRV single-modal: 81.1%, EEG five-band single-mode: 88.6%, The fusion model of this invention has a success rate of 96.4%.

[0085] Product performance test data Average recognition latency: 0.8s; Stability after 24 hours of continuous operation: recognition accuracy fluctuation is less than 1%.

[0086] Example 2 1. Test environment and test subject information Test environment: Indoor environment with normal lighting and no interference; Subject: Adult test subject Fei, diagnosed with depressive disorder by the outpatient department of a mental health center, who was hospitalized for systematic treatment due to the severity of her condition and is currently in the maintenance medication period; System status: The brainwave emotion detection system of this invention is fully powered on and running, with prefrontal cortex F3 / F4 dual-channel EEG acquisition and multi-dimensional emotion weight model full module calculation enabled.

[0087] 2. Actual testing process Participants wore a prefrontal cortex electroencephalogram (EEG) acquisition device, remained seated and relaxed, and had their EEG signals continuously collected for 180 seconds. The system extracted energy distribution characteristics across eight frequency bands: delta, theta, lowAlpha, highAlpha, lowBeta, highBeta, lowGamma, and highGamma. Simultaneously, it collected three physiologically related indicators: relaxation level, focus level, and heart rate. The system used a built-in multi-dimensional emotion weighting model to perform real-time calculations, extracting 17 feature parameters such as theta-alpha ratio, beta-alpha ratio, gamma percentage, focus-relaxation difference, and median heart rate via a second-by-second sliding window. These parameters were then weighted and scored to output quantitative scores across nine psychological and emotional dimensions using a weighted scoring model.

[0088] 3. Real-world test results System recognition results: Sleep-related dimensions: | Dimension | Score | Explanation | |——|——|——| | Difficulty falling asleep | 2 | Time from lying in bed to falling asleep within 30 minutes | | Frequent dreams | 2 | Good quality deep sleep | | Feeling tired upon waking | 2 | Energy largely restored, feeling refreshed | Emotion-Related Dimensions: | Dimension | Score | Explanation | |——|——|——| | Psychological Anxiety | 1 | Can feel anxiety, but has not yet formed a fixed neural circuit at the emotional level, and is capable of adjusting and calming down | | Physical Anxiety | 0 | No anxiety events exist, and the body is healthy | | Obsessive-Compulsive Tendency | 0 | No compulsive behaviors | | Depressive Mood | 2 | Can clearly perceive depressive mood, manifested as dullness, lack of interest, and insufficient energy | | Mental Fatigue | 2 | Significant mental fatigue, slow thinking, insufficient energy, and unstable emotions | | Inner Resilience | 3 | Can digest setbacks on their own within about half a day and restore a calm and positive mindset | Overall assessment: Mild to moderate depressive mood (level 2 / 3) was detected, accompanied by significant mental fatigue (level 2 / 4), and mild anxiety (level 1 / 3). The system identification results correspond to the clinical diagnosis of depressive disorder in the two dimensions of depressive mood and mental fatigue, but the overall severity rating is lower than the clinical manifestations.

[0089] 4. Technical Effects This system can detect core characteristics of depressive disorders, such as low mood, lack of interest, and mental fatigue, and the depressive mood dimension shows a distinguishable difference from that of non-depressive individuals. It should be noted that in this embodiment, the subjects were in a medication maintenance period, and the moderating effect of the medication on EEG characteristics may cause scores in some dimensions to approach the normal range.

[0090] Example 3 (Actual Test Implementation in Bipolar Disorder Patients: Verification of Emotional Instability) 1. Test environment and test subject information Test environment: Indoor environment with normal lighting and no interference; Test subject: Fan, an adult test subject, diagnosed with bipolar disorder by a psychiatric outpatient clinic, who was hospitalized for about a week due to a severe depressive episode accompanied by suicidal ideation and behavior, and is currently taking psychiatric medication; System status: The brainwave emotion detection system of this invention is fully powered on and running, with prefrontal cortex F3 / F4 dual-channel EEG acquisition enabled and multi-dimensional emotion weight model full module calculation.

[0091] 2. Actual testing process Participants wore a prefrontal cortex electroencephalogram (EEG) acquisition device, remained seated and relaxed, and had their EEG signals continuously collected for 180 seconds. The system extracted the energy distribution characteristics of eight frequency bands and three physiological correlation indicators in real time, and calculated them in real time through a built-in multi-dimensional emotion weighting model, outputting quantitative scores for nine psychological and emotional dimensions.

[0092] 3. Real-world test results System recognition results: Overall assessment: Mild to moderate depressive mood (level 2 / 3) was detected, accompanied by mental fatigue (level 2 / 4), and low psychological resilience (level 2 / 5), indicating insufficient psychological resilience. The system successfully identified three abnormal indicators: depressive mood, mental fatigue, and low psychological resilience, which partially correspond to the clinical manifestations of the depressive phase of bipolar disorder.

[0093] 4. Technical Effects This system can detect multiple abnormalities in bipolar disorder patients, including depressive mood, mental fatigue, and decreased psychological resilience, in a single resting-state data collection. The psychological resilience dimension (2 / 5) reflects a longer recovery period (half a day to one day) after the subject experiences setbacks, which is consistent with the difficulty in mood regulation characteristic of bipolar disorder.

[0094] Example 4 (Experimental Test of Suspected Depressive and Anxious State: Validation Example of High SCL-90 Factor Score) 1. Test environment and test subject information Test environment: Indoor environment with normal lighting and quiet, undisturbed surroundings; Test subject: Adult test subject Han, who had not sought professional clinical diagnosis, but whose SCL-90 self-assessment scale results indicated significantly high scores on depression and anxiety factors, and who reported long-term low mood, difficulty falling asleep, and anxiety-induced insomnia; System status: The brainwave emotion detection system of this invention is fully powered on and running, with prefrontal cortex F3 / F4 dual-channel EEG acquisition enabled and multi-dimensional emotion weight model full-module calculations performed.

[0095] 2. Actual testing process Participants wore a prefrontal cortex electroencephalogram (EEG) acquisition device, remained seated and relaxed, and had their EEG signals continuously collected for 180 seconds. The system extracted the energy distribution characteristics of eight frequency bands and three physiological correlation indicators in real time, and calculated them in real time through a built-in multi-dimensional emotion weighting model, outputting quantitative scores for nine psychological and emotional dimensions.

[0096] 3. Real-world test results System recognition results: Overall assessment: Moderate psychological anxiety (level 2 / 3) and physical anxiety (level 2 / 3) were detected, with the anxiety dimension signal significantly higher than the depression dimension (level 1 / 3), presenting a typical "high anxiety-low depression" EEG pattern. This result differs from the pattern reported by the individual on the SCL-90 self-report scale, where both anxiety and depression scores were elevated.

[0097] 4. Technical Effects This system successfully detected anxiety in individuals who had not sought medical attention. Both psychological anxiety (level 2) and physical anxiety (level 2) reached moderate levels, confirming the system's sensitivity in identifying subclinical anxiety states. The anxiety detection sensitivity was superior to the depression detection sensitivity.

[0098] Example 5 (Measurement of Normal Control Subjects - Baseline Validation Example) 1. Test environment and test subject information Test environment: Indoor environment with normal lighting and quiet, undisturbed environment; Test subject: Adult test subject Chen, with normal physical and mental condition, no history of psychiatric treatment, good daily social functioning, and normal performance in all aspects; System status: The brainwave emotion detection system of this invention is fully powered on and running, with prefrontal cortex F3 / F4 dual-channel EEG acquisition enabled and multi-dimensional emotion weight model full module calculation enabled.

[0099] 2. Actual testing process Participants wore a prefrontal cortex electroencephalogram (EEG) acquisition device, remained seated and relaxed, and had their EEG signals continuously collected for 180 seconds. The system extracted the energy distribution characteristics of eight frequency bands and three physiological correlation indicators in real time, and calculated them in real time through a built-in multi-dimensional emotion weighting model, outputting quantitative scores for nine psychological and emotional dimensions.

[0100] 3. Real-world test results System recognition results: Overall assessment: All dimensions are within the normal range. Levels 1 in psychological anxiety, 1 in depressive mood, and 1 in mental fatigue are all within the normal daily fluctuation range for most people. Level 0 in physical anxiety confirms good physical health and the absence of somatization. Overall emotional state is assessed as normal.

[0101] 4. Technical Effects The system's output results for normal subjects all fell within the non-clinical range, with no false high alarms, and its identification of the normal emotional baseline was stable and reliable. A quantifiable difference of level 1 vs. level 2 was found between normal subjects and clinically diagnosed patients on the depressive mood dimension, confirming the system's basic ability to distinguish between normal and pathological states.

[0102] Comprehensive Comparison Table of Four Examples Key findings: 1. The clinically diagnosed group (Fei and Fan) consistently scored higher than the normal group (Chen, Level 1) on both the depressive mood (Level 2) and mental fatigue (Level 2) dimensions, forming a distinguishable difference. 2. The suspected anxiety group (Han) scored significantly higher than the other three groups on both the psychological anxiety (Level 2) and physical anxiety (Level 2) dimensions, demonstrating good sensitivity in anxiety detection. 3. There were no significant differences among the four groups in the three sleep-related dimensions (slowness of falling asleep, vivid dreams, and fatigue upon waking), indicating limited distinguishability. 4. The patient with bipolar disorder (Fan) scored the lowest (Level 2) in terms of mental resilience, which may reflect the long-term damage of mood disorders to psychological resilience.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An emotion recognition system integrating heart rate variability and prefrontal cortex electroencephalogram (EEG) signals, characterized in that, include: The information acquisition module includes an HRV signal acquisition unit and a prefrontal EEG signal acquisition unit. The HRV signal acquisition unit acquires HRV signals through an acquisition device, and the prefrontal EEG signal acquisition unit acquires EEG signal one of F3 and EEG signal two of F4 through an EEG acquisition device. The preprocessing module, connected to the information acquisition module, is used to receive the HRV signal and the first and second brainwave signals respectively, perform preprocessing, and generate the preprocessed HRV signal, the preprocessed first brainwave signal and the preprocessed second brainwave signal, which are then fed back to the feature extraction module. The feature extraction module, connected to the preprocessing module, is used to receive preprocessed HRV signals, preprocessed EEG signal 1, and preprocessed EEG signal 2, respectively, and perform feature extraction to obtain feature vector 1, feature 2, and feature 3, which are then fed back to the multimodal fusion module. Specifically, feature vector 1 is extracted from the preprocessed HRV signals, feature 2 is extracted from the preprocessed EEG signal 1, and feature 3 is extracted from the preprocessed EEG signal 2. The multimodal fusion module is connected to the feature extraction module. The multimodal fusion module converts feature 2 and feature 3 and feature vector 1 into a feature token sequence, and obtains the full-modal sentiment representation vector through Transformer encoding. The full-modal sentiment representation vector is then fed back to the sentiment classification module. The emotion classification module, connected to the multimodal fusion module, receives the full-modal emotion representation vector, classifies it, and outputs emotion labels, which are then fed back to the output module. The output module, connected to the emotion classification module, is used to receive emotion tags and their corresponding emotion intensities.

2. The emotion recognition system integrating heart rate variability and prefrontal cortex electroencephalogram (EEG) signals according to claim 1, characterized in that: The acquisition device includes at least one of ECG or PPG.

3. The emotion recognition system integrating heart rate variability and prefrontal cortex electroencephalogram (EEG) signals according to claim 1, characterized in that: The HRV signal acquisition unit includes: The target object's continuous RR interval sequence is acquired by ECG or PPG, and the HRV signal is output.

4. The emotion recognition system integrating heart rate variability and prefrontal cortex electroencephalogram (EEG) signals according to claim 1, characterized in that: The prefrontal EEG signal acquisition unit: The EEG signal 1 for F3 and the EEG signal 2 for F4 were collected using an EEG acquisition device, with a signal frequency range of 0.5-150Hz.

5. The emotion recognition system integrating heart rate variability and prefrontal cortex electroencephalogram (EEG) signals according to claim 1, characterized in that: The preprocessing module: The HRV signal is sequentially subjected to filtering, baseline drift removal, peak detection, RR / IBIS sequence reconstruction, and resampling / interpolation to obtain a preprocessed HRV signal; The EEG signal 1 and EEG signal 2 are sequentially subjected to raw examination, bandpass filtering, artifact detection and removal, independent component analysis for artifact removal, short-time segmentation / windowing, power spectral density calculation, and time-series / statistical feature aggregation to obtain preprocessed EEG signal 1 and preprocessed EEG signal 2. The preprocessed EEG signal 1 and preprocessed EEG signal 2 respectively include Delta, Theta, Alpha, Beta, and Gamma.

6. The emotion recognition system integrating heart rate variability and prefrontal cortex electroencephalogram (EEG) signals according to claim 1, characterized in that: The first feature vector includes time-domain features, frequency-domain features, and nonlinear features. The time-domain features include SDNN and RMSSD, the frequency-domain features include LF, HF, and LF / HF, and the nonlinear features include SD1 and SD2. The second and third features include five-band absolute power, five-band relative power, left-right lateralization index, selectable band ratio, and statistical features.

7. The emotion recognition system integrating heart rate variability and prefrontal cortex electroencephalogram (EEG) signals according to claim 1, characterized in that: The multimodal fusion module includes five-band EEG energy feature extraction, construction of multi-band side-modulation parameters, adaptive fusion weight model, final fusion features, and Transformer multimodal coding. The specific implementation steps of the multimodal fusion module are as follows: S1. Five-band EEG energy feature extraction: Extract the energy of feature two in the five frequency bands Delta, Theta, Alpha, Beta, and Gamma to obtain the first energy feature vector of feature two. Also, extract the energy of feature three in the five frequency bands Delta, Theta, Alpha, Beta, and Gamma to obtain the second energy feature vector of feature two. S2. Construct multi-band side-shifting parameters: Combine the energy feature vector one and the energy feature vector two from step S1 with the side-shifting exponent formula to form side-shifting vector one; S3, Adaptive Fusion Weight Model: The fusion weight one is calculated by using the sigmoid function to calculate the fusion weight one of the side-shifted vector one in step S2, and the fusion weight two of the feature vector one is 1 minus the fusion weight one. S4. Final fusion features: The final fusion features are obtained by calculating the energy feature vector 1, feature vector 2, side-shifting vector 1, fusion weight 1, fusion weight 2, and feature vector 1. S5, Transformer Multimodal Encoding: Input feature vector 1, energy feature vector 1, energy feature vector 2, side-shifting vector 1, and the final fused feature into Transformer encoding, and finally output a full-modal sentiment representation vector.

Images