Dream monitoring and visualization method and system based on multi-modal physiological signal fusion

Abstract

The invention relates to the technical field of biological signal processing and intelligent health equipment, in particular to a dream monitoring and visualization method and system based on multi-modal physiological signal fusion. The intelligent monitoring terminal is integrated in the portable sleep eyeshade and comprises a flexible dry electrode array (used for EEG acquisition), an integrated photoelectric sensor (used for PPG signal acquisition), a miniature millimeter wave radar chip (used for non-contact EOG acquisition), a signal processing module, a wireless transmission module, a semiconductor temperature control module and a double-beat audio module; and the user terminal APP is used for receiving, storing and displaying sleep data (a sleep staging chart and REM period information), a dream intensity index (DII) and a dream cloud chart, and generating a sleep health report and trend analysis. The method has the advantages that high-precision recognition is achieved, the problems that a single sensor is prone to interference and high in misjudgment rate are solved through EEG, EOG and PPG three-mode signal fusion judgment, and the REM period recognition accuracy is improved by 40% or above compared with single EEG monitoring.

Description

Technical Field

[0001] This invention relates to the field of biosignal processing and intelligent health device technology, specifically to a dream monitoring and visualization method and system based on multimodal physiological signal fusion. Background Technology

[0002] Sleep health monitoring has become one of the core needs of urban residents, and the monitoring and analysis of dreams, as a typical feature of REM sleep, is of great significance for sleep quality assessment. Current sleep monitoring technologies have significant limitations: consumer-grade devices mostly rely on body movement or a single heart rate signal to determine sleep state, failing to accurately distinguish between sleep stages such as wakefulness, light sleep, deep sleep, and REM sleep, and even more so, accurately identifying the dream stage; while professional polysomnography (PSG) devices can achieve high-precision monitoring through multimodal signals, they require operation by professionals in hospitals, are expensive, and require the use of wet electrodes and patch sensors, resulting in extremely poor comfort and failing to meet the home needs of ordinary users.

[0003] Currently, the market lacks a dream monitoring solution that balances non-invasive comfort, home convenience, and monitoring accuracy. It cannot transform abstract dream experiences into quantifiable health data, making it difficult to meet users' needs for in-depth insights into sleep quality and personalized improvement. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a comfortable, convenient, and home-use dream monitoring method and system that enables precise division of sleep stages, accurate identification of the dream activity period (REM period), and presents dream intensity to users in a quantitative and visual form.

[0005] 1. System composition,

[0006] The system described in this invention includes an intelligent monitoring terminal and a user terminal APP, which interact with each other via wireless communication (Bluetooth, WiFi, etc.).

[0007] Intelligent monitoring terminal: integrated into a lightweight sleep mask, including a flexible dry electrode array (for EEG acquisition), an integrated photoelectric sensor (for PPG signal acquisition), a miniature millimeter-wave radar chip (for non-contact EEG acquisition), a signal processing module, a wireless transmission module, a semiconductor temperature control module, and a dual-beat audio module;

[0008] User terminal APP: Used to receive, store, and display sleep data (sleep stage chart, REM stage information), dream intensity index (DII) and "dream cloud map", and generate sleep health reports and trend analysis.

[0009] 2. Monitoring and visualization methods,

[0010] The method includes the following steps:

[0011] S1: Signal acquisition and preprocessing,

[0012] Signal acquisition: EEG signals from the user's forehead are acquired through a flexible dry electrode array on the forehead of the smart goggles; HR and HRV signals are acquired through a photoelectric sensor integrated on the nose side of the goggles; EOG signals are acquired in a non-contact manner through a miniature millimeter-wave radar chip in the periorbital area of ​​the goggles.

[0013] Preprocessing: The acquired EEG, PPG, and EOG signals are sequentially filtered (EEG uses 0.5-100Hz bandpass filtering, PPG uses 5-30Hz bandpass filtering, and EOG uses 1-50Hz bandpass filtering), denoised (interference removal based on wavelet transform), and normalized (signal amplitude is normalized to the [-1,1] interval) to eliminate power frequency interference, electromyographic interference, and motion artifacts.

[0014] S2: Preliminary assessment of sleep stages and REM sleep.

[0015] Feature extraction: Fast Fourier Transform (FFT) is performed on the preprocessed EEG signal to extract the power spectral density features of delta waves (0.5-4Hz), theta waves (4-8Hz), alpha waves (8-13Hz), beta waves (13-30Hz), and gamma waves (30-100Hz).

[0016] Sleep staging and initial judgment: Input the above features into a rule-based threshold classifier or a lightweight machine learning model (such as lightweight random forest or support vector machine), and output the current sleep stage (awake, light sleep, deep sleep, potential REM sleep stage), and identify the potential REM sleep stage that meets the "EEG low amplitude mixed frequency" feature for the first time.

[0017] S3: Multimodal signal fusion and confirmation of the dream phase.

[0018] A multimodal signal fusion decision model is established, and the potential REM periods identified in S2 are cross-validated. The time period is confirmed as a dream activity period if and only if the following three conditions are met simultaneously:

[0019] Condition 1: The EEG signal continuously shows low-amplitude mixed frequency waves (characteristics similar to those of awake EEG, with delta waves accounting for <10% and beta and gamma waves accounting for >40% combined).

[0020] Condition 2: Periodic rapid eye movements (eye movement velocity > 5° / s, movement period 0.2-2s) are detected by EOG signal;

[0021] Condition 3: The PPG signal shows that the heart rate increases by 10% - 30% compared to the adjacent non-REM period, and the time-domain index SDNN (standard deviation) of heart rate variability (HRV) increases by > 20%.

[0022] S4: Dream intensity index calculation and visualization

[0023] Calculation of the dream intensity index (DII): Deeply analyze the physiological signals during the confirmed dream-active period, and calculate DII based on the following formula:

[0024] DII = k1×(normalized γ-wave power) + k2×(normalized HRV)

[0025] Where k1 and k2 are weighting coefficients (0 < k1, k2 < 1, and k1 + k2 ≤ 1), determined through training with a large number of samples (preferably k1 = 0.6, k2 = 0.4); the normalized γ-wave power value is determined by the ratio of the γ-wave power in this period to the maximum γ-wave power in the entire sleep cycle, and the normalized HRV value is determined by the ratio of the HRV index SDNN in this period to the maximum SDNN value in the entire sleep cycle;

[0026] Generation of the dream cloud map: Dynamically generate an abstract visual "dream cloud map" according to the DII value - with time as the horizontal axis and the DII value as the vertical axis, map the dream intensity change through color flow (low DII is cold color such as blue, high DII is warm color such as red) and particle density (low DII has sparse particles, high DII has dense particles). The cloud map does not represent specific dream content, but only realizes an intuitive expression of intensity.

[0027] S5: Data output and feedback

[0028] Send the sleep staging results (including the duration ratio of each stage), the start time / duration of the dream-active period, the corresponding DII time-series data, and the "dream cloud map" to the user terminal APP through the wireless transmission module; the APP generates a concise and easy-to-understand sleep health report, including the REM period ratio, dream intensity distribution, sleep quality score, and personalized improvement suggestions (such as the timing of audio guidance based on sleep stages, temperature control parameter adjustment).

[0029] The beneficial effects of the present invention are:

[0030] High-precision identification: Through the fusion decision of EEG, EOG, and PPG triple-modal signals, the problem of single sensor being vulnerable to interference and high misjudgment rate is solved. The REM period identification accuracy is improved by more than 40% compared to single EEG monitoring, reaching clinical-level accuracy;

[0031] High wearing comfort: Flexible dry electrodes are used instead of traditional wet electrodes, and non-contact eye movement monitoring is achieved through millimeter-wave radar, avoiding skin irritation from patch sensors. The overall weight of the eye mask is less than 50g, making it suitable for long-term home wear.

[0032] Innovative quantification and visualization: The first concept of "Dream Intensity Index (DII)" transforms subjective dream experiences into objective and quantifiable data, and combines it with "Dream Cloud Map" to achieve artistic visualization, providing users with a brand-new dimension of sleep health insights;

[0033] Strong practicality and productization potential: The entire system is integrated into a lightweight sleep mask, which combines sleep analysis, dream monitoring, and intelligent sleep aid (dual-beat audio, semiconductor temperature control) functions, realizing the consumer-grade transformation of laboratory-level technology, and has broad market application prospects.

[0034] Obviously, based on the above description of the present invention, and according to common technical knowledge and conventional methods in the field, various other modifications, substitutions, or alterations can be made without departing from the basic technical concept of the present invention.

[0035] The following detailed embodiments further illustrate the above-described content of the present invention. However, this should not be construed as limiting the scope of the present invention to the following examples. All technologies implemented based on the above-described content of the present invention fall within the scope of the present invention. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the process structure of the present invention. Detailed Implementation

[0037] The present invention is illustrated below with specific embodiments, which are not intended to limit the scope of the invention.

[0038] like Figure 1 As shown, a dream monitoring and visualization system based on multimodal physiological signal fusion is characterized by including an intelligent monitoring terminal and a user terminal APP, which realize data interaction through wireless communication.

[0039] The intelligent monitoring terminal is integrated into a lightweight sleep mask and includes a flexible dry electrode array for EEG acquisition, an integrated photoelectric sensor for PPG signal acquisition, a miniature millimeter-wave radar chip for non-contact EOG acquisition, a signal processing module, a wireless transmission module, a semiconductor temperature control module, and a dual-beat audio module.

[0040] The flexible dry electrode is a 4-channel array with a spacing of 2cm and a sampling rate of 256Hz; the integrated photoelectric sensor is a reflective type with a wavelength of 660nm and a sampling rate of 100Hz; the miniature millimeter-wave radar chip operates at a frequency of 60GHz, has a ranging accuracy of 0.1mm, and one chip is deployed on each side of the eye area of ​​the goggles.

[0041] The signal processing module uses a low-power MCU with a built-in FFT operation unit; the semiconductor temperature control module can achieve 38-42℃ hot compress and 18-22℃ cold compress adjustment; the dual-beat audio module supports 8-12Hz alpha wave audio output.

[0042] The user terminal APP is used to receive, store, and display sleep data, dream intensity index (DII), and "dream cloud map," and generate sleep health reports and trend analyses. The sleep data includes sleep stage maps and REM stage information.

[0043] The user terminal APP uses the Canvas drawing engine to generate the "Dream Cloud Map".

[0044] A dream monitoring and visualization method based on multimodal physiological signal fusion, characterized by the following steps:

[0045] S1: Signal Acquisition and Preprocessing. EEG, HR, HRV, and EOG signals are acquired. The acquired signals are sequentially filtered, denoised, and normalized. Filtering parameters are as follows: 0.5-100Hz bandpass filtering for EEG, 5-30Hz bandpass filtering for PPG, and 1-50Hz bandpass filtering for EOG. Noise reduction employs wavelet transform-based interference removal, including DB4 wavelet transform-based removal of electromyography artifacts. Normalization normalizes the signal amplitude to the [-1,1] interval using the Z-score method. Preprocessing also includes a 50Hz notch filter to eliminate power frequency interference.

[0046] S2: Initial determination of sleep stages and REM sleep periods. A Fast Fourier Transform is performed on the preprocessed EEG signal to extract the power spectral density features of delta waves, theta waves, alpha waves, beta waves, and gamma waves. These features are input into a classifier or a lightweight machine learning model, which outputs the sleep stage and identifies potential REM sleep periods. The lightweight machine learning model is a lightweight random forest or support vector machine. The sleep stage determination rules are: delta wave proportion > 50% indicates deep sleep, theta wave proportion > 40% indicates light sleep, and alpha wave proportion > 30% indicates wakefulness.

[0047] S3: Multimodal signal fusion and dream period confirmation. Establish a multimodal signal fusion decision model and cross-validate potential REM periods. When EEG, EOG, and PPG signals all meet the preset conditions, the time period is confirmed as the dream activity period.

[0048] The preset conditions include:

[0049] ① The EEG signal continuously presents low-amplitude mixed-frequency waves, with the proportion of δ waves < 10% and the combined proportion of β waves and γ waves > 40%;

[0050] ② The EOG signal detects periodic rapid eye movements, with an eye movement speed > 5° / s and a movement period of 0.2 - 2 s;

[0051] ③ The PPG signal shows that the heart rate increases by 10% - 30% compared to the adjacent non-REM period, and the SDNN, a time-domain index of HRV, increases by > 20%; The multi-modal signal fusion decision model verifies the three conditions through a logical AND gate circuit.

[0052] S4: Dream intensity index calculation and visualization. Calculate the DII based on the normalized γ-wave power value and the normalized HRV value, and dynamically generate a "dream cloud map" according to the DII value. The "dream cloud map" uses time as the horizontal axis and the DII value as the vertical axis, and maps the dream intensity changes through color flow and particle density (low DII is cold-colored and the particles are sparse, high DII is warm-colored and the particles are dense), and the image is refreshed once per second. The calculation formula of DII is: DII = k1×(normalized γ-wave power value) + k2×(normalized HRV value), where 0 < k1, k2 < 1, and k1 + k2 ≤ 1. Preferably, k1 = 0.6 and k2 = 0.4;

[0053] S5: Data output and feedback. Send the sleep stage results, dream active period information, DII time-series data, and "dream cloud map" to the user terminal APP, and the APP generates a sleep health report.

[0054] The following details the implementation manners of the present invention in combination with specific scenarios:

[0055] 1. Hardware integration

[0056] The intelligent eye mask is made of skin-friendly and breathable memory cotton material. A 4-channel flexible dry electrode array (spacing 2 cm, sampling rate 256 Hz) is embedded in the forehead, a reflective photoelectric sensor (wavelength 660 nm, sampling rate 100 Hz) is integrated on the nasal side, and one micro-millimeter wave radar chip (operating frequency 60 GHz, ranging accuracy 0.1 mm) is deployed on each side of the eye socket; The signal processing module uses a low-power MCU (such as the STM32L4 series) with an internal FFT operation unit; The semiconductor temperature control module can achieve hot compress adjustment at 38 - 42 °C and cold compress adjustment at 18 - 22 °C, and the audio module supports the output of 8 - 12 Hz α-wave binaural beats.

[0057] 2. Software algorithm implementation

[0058] Preprocessing algorithm: 50Hz Notch filter is used to eliminate power frequency interference, db4 wavelet transform is used to remove electromyography artifacts, and Z-score method is used to standardize the signal;

[0059] Sleep staging model: A hybrid model combining regular thresholds and lightweight SVM is used—deep sleep is determined by the proportion of delta waves > 50%, light sleep by the proportion of theta waves > 40%, and wakefulness by the proportion of alpha waves > 30%, which preliminarily screens potential REM sleep stages;

[0060] Fusion decision model: Three-condition verification is implemented through logic AND gate circuits. When all three conditions are met, a REM period confirmation signal is triggered.

[0061] DII Calculation and Cloud Map Generation: The APP uses the Canvas drawing engine to dynamically update the color flow and particle density based on DII time-series data, refreshing the image once per second.

[0062] 3. Practical application process,

[0063] Users wear a smart eye mask before bed, and the device automatically enters signal acquisition mode after being turned on. During sleep, the system executes steps S1-S4 in real time and stores data. When the user wakes up the next day, the APP automatically synchronizes the data and generates a report, showing "2.5 hours of deep sleep, 3 hours of light sleep, 1.5 hours of REM sleep, and a maximum DII value of 0.8 (during the period from 3:00 to 3:20 am, the cloud map shows dense orange-red particles)". It also recommends "Based on the REM sleep distribution, it is recommended to fall asleep before 11:00 pm and turn on a 30-minute hot compress to help with sleep".

[0064] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0065] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

[0066] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A dream monitoring and visualization system based on multimodal physiological signal fusion, characterized in that, This includes intelligent monitoring terminals and user terminal apps, which interact with each other via wireless communication; The intelligent monitoring terminal is integrated into a lightweight sleep mask and includes a flexible dry electrode array for EEG acquisition, an integrated photoelectric sensor for PPG signal acquisition, a miniature millimeter-wave radar chip for non-contact EOG acquisition, a signal processing module, a wireless transmission module, a semiconductor temperature control module, and a dual-beat audio module. The user terminal APP is used to receive, store, and display sleep data, dream intensity index (DII), and "dream cloud map," and generate sleep health reports and trend analyses. The sleep data includes sleep stage maps and REM stage information.

2. The system according to claim 1, characterized in that, The flexible dry electrode is a 4-channel array with a spacing of 2cm and a sampling rate of 256Hz; the integrated photoelectric sensor is a reflective type with a wavelength of 660nm and a sampling rate of 100Hz; the miniature millimeter-wave radar chip operates at a frequency of 60GHz, has a ranging accuracy of 0.1mm, and one chip is deployed on each side of the eye area of ​​the goggles.

3. The system according to claim 1, characterized in that, The signal processing module uses a low-power MCU with a built-in FFT operation unit; the semiconductor temperature control module can achieve 38-42℃ hot compress and 18-22℃ cold compress adjustment; the dual-beat audio module supports 8-12Hz alpha wave audio output.

4. The system according to claim 1, characterized in that, The user terminal APP uses the Canvas drawing engine to generate "Dream Cloud Map".

5. A method for dream monitoring and visualization based on multimodal physiological signal fusion, characterized in that, Includes the following steps: S1: Signal acquisition and preprocessing, acquiring EEG signals, HR and HRV signals, and EOG signals, and performing filtering, noise reduction and standardization on the acquired signals in sequence; S2: Initial determination of sleep stages and REM sleep stages. The preprocessed EEG signal is subjected to fast Fourier transform to extract the power spectral density features of delta, theta, alpha, beta and gamma waves. The features are input into a classifier or lightweight machine learning model to output the sleep stage and identify the potential REM sleep stage. S3: Multimodal signal fusion and dream period confirmation. Establish a multimodal signal fusion decision model and cross-validate potential REM periods. When EEG, EOG, and PPG signals all meet the preset conditions, the time period is confirmed as the dream activity period. S4: Dream Intensity Index Calculation and Visualization. The DII is calculated based on the normalized values ​​of gamma wave power and HRV, and a "dream cloud map" is dynamically generated based on the DII values. S5: Data output and feedback, sending sleep stage results, dream activity period information, DII time series data and "dream cloud map" to the user's terminal APP, and the APP generates a sleep health report.

6. The method according to claim 5, characterized in that, The filtering parameters in S1 are as follows: EEG uses a 0.5-100Hz bandpass filter, PPG uses a 5-30Hz bandpass filter, and EOG uses a 1-50Hz bandpass filter; noise reduction uses interference removal based on wavelet transform, including removal of electromyography artifacts based on db4 wavelet transform; normalization processing normalizes the signal amplitude to the [-1,1] interval, and is achieved by the Z-score method; preprocessing also includes using a 50Hz notch filter to eliminate power frequency interference.

7. The method according to claim 5, characterized in that, The lightweight machine learning model described in S2 is a lightweight random forest or a support vector machine; the sleep stage determination rule is: when the proportion of δ waves > 50%, it is determined as the deep sleep stage, when the proportion of θ waves > 40%, it is determined as the light sleep stage, and when the proportion of α waves > 30%, it is determined as the waking stage.

8. The method according to claim 5, characterized in that, The preset conditions described in S3 include: ① The EEG signal continuously shows low-amplitude mixed-frequency waves, the proportion of δ waves < 10%, and the combined proportion of β waves and γ waves > 40%; ② The EOG signal detects periodic rapid eye movements, the eye movement speed > 5° / s, and the movement period is 0.2 - 2 s; ③ The PPG signal shows that the heart rate increases by 10% - 30% compared with the adjacent non-REM period, and the HRV time-domain index SDNN increases > 20%; the multi-modal signal fusion decision model realizes the verification of the three conditions through a logical AND gate circuit.

9. The method according to claim 5, characterized in that, The calculation formula of DII in S4 is: DII = k1×(normalized value of γ wave power) + k2×(normalized value of HRV), where 0 < k1, k2 < 1, and k1 + k2 ≤ 1. Preferably, k1 = 0.6 and k2 = 0.

4.

10. The method according to claim 5, characterized in that, In S4, the "dream cloud map" uses time as the horizontal axis and the DII value as the vertical axis, and maps the dream intensity change through color flow and particle density (low DII is cold-colored and the particles are sparse, high DII is warm-colored and the particles are dense), and the image is refreshed once per second.

Images