Non-contact eye movement monitoring method and system based on millimeter wave perception
By analyzing the minute movements of the eyelids and surrounding muscles using millimeter-wave radar, and combining convolutional neural networks and long short-term memory networks, eye movement monitoring in the closed-eye state was achieved. This solves the problem of low usability of traditional methods during sleep and provides high-precision sleep stage inference and comfort monitoring.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SICHUAN PROVINCIAL INSTITUTE OF ARTIFICIAL INTELLIGENCE
- Filing Date
- 2025-11-28
- Publication Date
- 2026-04-21
AI Technical Summary
Existing millimeter-wave technology has limited applications in eye movement monitoring with eyes closed, especially when the subject's position is not fixed during sleep. Traditional methods have low usability and pose privacy and comfort issues.
By employing a non-contact eye movement monitoring method based on millimeter-wave sensing, the method utilizes the minute movement changes of the eyelids and periocular muscles, transmits continuous Chirp signals via millimeter-wave radar, performs amplitude and phase dual-channel temporal feature analysis, and combines models such as convolutional neural networks and long short-term memory networks to achieve eye movement monitoring in the closed eye state.
It achieves accurate eye movement recognition even with eyes closed, improving the comfort and privacy of sleep monitoring. It can work stably in low-light environments, resists eye movement interference, is suitable for long-term home use, and provides high-precision sleep stage inference.
Smart Images
Figure CN121890976A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of eye movement monitoring technology, specifically to a non-contact eye movement monitoring method and system based on millimeter-wave sensing. Background Technology
[0002] The distinction between rapid eye movement (REM) and non-rapid eye movement (NREM) stages is one of the core indicators of sleep staging. Traditional eye movement monitoring mainly falls into two categories: Electrooculography (EOG): Electrodes are attached around the eye to monitor changes in electrical potential to reflect eye movements; Camera / optical imaging (visible light / infrared): Inferring eye movement by tracking pupil / iris / motion or eyeball features.
[0003] The aforementioned methods have significant limitations in terms of closed eyes, low light, privacy, and long-term comfort. In recent years, millimeter-wave non-contact sensing has been used for human micro-motion detection and physiological monitoring due to its light-independent and privacy-friendly characteristics. However, there is a gap in the application of millimeter-wave sensing for sleep monitoring through closed-eye eye movement monitoring.
[0004] Current millimeter-wave eye-tracking research primarily focuses on "open-eye" scenarios, such as gaze tracking, human-computer interaction, and applications related to fatigued driving. In these scenarios, millimeter-wave technology mainly senses changes in eye movement and pupil position, typically requiring the subject to have their eyes open and face the radar directly. The application scenarios for this type of method in the closed-eye state are relatively limited, and it is also affected by the subject's unpredictable posture; therefore, the usability of open-eye methods in sleep monitoring is low. Summary of the Invention
[0005] To address the aforementioned shortcomings of existing technologies, this invention provides a non-contact eye movement monitoring method and system based on millimeter-wave sensing, focusing on sensing minute changes in eyelid movement and periocular muscle movement. In the closed-eye state, eyeball rotation and pupil position changes cannot be effectively sensed, making the minute movements of the eyelids and surrounding muscles crucial for perception. This invention achieves eye movement monitoring in the closed-eye state by performing detailed analysis of the amplitude and phase changes of the millimeter-wave echoes caused by these minute movements.
[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: In a first aspect, the present invention proposes a non-contact eye-tracking monitoring method based on millimeter-wave sensing, comprising the following steps: Raw data is obtained by transmitting continuous Chirp signals and receiving echoes using millimeter-wave radar. The raw data is processed to construct amplitude and phase dual-channel temporal features for characterizing micro-movements of the eyelids and periocular region in a closed-eye state. The processing includes: splicing together data corresponding to several consecutive Chirp signals; performing frequency domain analysis on the spliced data to obtain a distance spectrum containing multiple distance unit bins; selecting one or more bins with the highest signal intensity within a preset facial distance range as target bins representing the eye; and extracting amplitude and phase information from multiple bins in the neighborhood of the target bin to form amplitude temporal data and phase temporal data, respectively. The amplitude and phase dual-channel timing features are divided into second-level time windows. Based on the dual-channel timing features within each time window, it is determined whether the window belongs to a state of no eye movement, slow eye movement, or intense eye movement.
[0007] Furthermore, the concatenation of data corresponding to several consecutive Chirp signals specifically involves: Perform IQ deDC removal on each of the Nt consecutive Chirps; The fast-time sampling sequences of Nt Chirps are concatenated end-to-end in the time domain in chronological order to form a complex IQ sequence of length Nt×N; where Nt is an integer greater than 1 and N is the number of fast-time sampling points.
[0008] Furthermore, the step of selecting one or more bins with the highest signal intensity within a preset facial distance range as target bins representing the eye specifically includes: The distance range is set based on the relative distance between the radar and the face during sleep. and map it to the range of bin. ; The range of bin Within the bin, find the bin with the largest average amplitude and determine it as the target bin.
[0009] Furthermore, after selecting the target bin, the process also includes phase anomaly detection and bin reselection steps: The phase sequence of the target bin is unwrapped and differentially calculated. If the absolute value of the phase difference exceeds a preset threshold within a specific time window, it is determined that there is body motion interference in the current window. In response to the determination of motion interference, discard the current window data and within the range of the bin. Select the target bin again.
[0010] Furthermore, the determination of eye movement state based on dual-channel temporal features within each time window is achieved by inputting the dual-channel temporal features into a trained classification model.
[0011] Furthermore, the classification model includes: The spatial feature extraction module is used to extract spatial features from amplitude time-series features and phase time-series features, respectively. The temporal modeling module is used to model the temporal dependencies of features after spatial feature extraction. The weighted fusion module is used to perform weighted fusion of features from the amplitude channel and the phase channel, where the weight of the amplitude channel is greater than that of the phase channel. The classifier module is used to output the classification result of eye movement state based on the weighted fused features.
[0012] Furthermore, the spatial feature extraction module employs a convolutional neural network or a graph convolutional network; the temporal modeling module employs a long short-term memory network, a gated recurrent unit, or a self-attention mechanism model.
[0013] Furthermore, it also includes: Based on the eye movement status determination results within each second-level time window of the minute-level time period, the proportion and / or degree of eye movement are statistically analyzed, and the sleep stage of the target is inferred based on the statistical results.
[0014] Furthermore, the eye movement percentage is the percentage of the duration of the second-level window that is determined to have eye movement within a minute-level time period; the eye movement intensity is the percentage of the duration of the second-level window that is determined to have intense eye movement within a minute-level time period and / or the number of eye movements.
[0015] Secondly, the present invention proposes a non-contact eye-tracking monitoring system based on millimeter-wave sensing, used to implement the above-mentioned method, the system comprising: Millimeter-wave radar module, used to transmit continuous chirp signals and receive echoes; The data processing module is configured as follows: The radar echo data is processed by splicing together the data corresponding to several consecutive Chirp signals. Frequency domain analysis was performed on the stitched data to obtain the distance spectrum; Within a preset facial distance range, a target bin representing the eye is selected, and the amplitude and phase information of multiple bins in the neighborhood are extracted with the target bin as the center to construct amplitude and phase dual-channel temporal features; The eye movement state determination module is configured to divide the dual-channel timing features into second-level time windows and determine the eye movement state within each window.
[0016] The present invention has the following beneficial effects: Unlike existing research that primarily focuses on eye movement monitoring during sleep with eyes open, the core innovation of this invention lies in its focus on monitoring sleep with eyes closed, particularly when the subject's position changes and their eyes are closed. This non-contact millimeter-wave sensing method can accurately identify eye movements with eyes closed, thereby inferring different sleep stages, offering high comfort and privacy protection. Attached Figure Description
[0017] Figure 1 This is a schematic diagram illustrating a non-contact eye-tracking monitoring method based on millimeter-wave sensing according to the present invention. Figures 2(a) to 2(c) are schematic diagrams comparing the single rangebin phase time series data of the eyes under the conditions of no eye movement with eyes closed, slow eye movement with eyes closed, and violent eye movement with eyes closed; Figures 3(a) to 3(c) are schematic diagrams comparing the amplitude time series data of no eye movement with eyes closed, slow eye movement with eyes closed, and intense eye movement with eyes closed. Detailed Implementation
[0018] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0019] like Figure 1 As shown, an embodiment of the present invention provides a non-contact eye-tracking monitoring method based on millimeter-wave sensing, comprising the following steps S1 to S3: S1. Transmit continuous Chirp signals through millimeter-wave radar and receive the echoes to obtain raw data; In this embodiment, the radar configuration and data acquisition involved in step S1 includes the following parts: Operating frequency band and bandwidth: set according to device specifications (e.g., starting frequency can be 77GHz, 81GHz, 66GHz, etc., and bandwidth can be set to 1-4GHz); the specific frequency band and bandwidth can be adjusted according to actual application to ensure optimal performance in different scenarios.
[0020] Chirp structure: It uses a continuous chirp stream, splicing together several consecutive chirs to improve distance resolution and data accuracy.
[0021] Data collection strategy (application): The data collection segments should be adjusted according to the needs. It is recommended to use several minute-level data segments to determine the changes in the subject's sleep state at night.
[0022] S2. The raw data is processed to construct amplitude and phase dual-channel temporal features for characterizing micro-movements of the eyelids and periocular region in a closed-eye state. The processing includes: splicing data corresponding to several consecutive Chirp signals; performing frequency domain analysis on the spliced data to obtain a distance spectrum containing multiple distance unit bins; selecting one or more bins with the highest signal intensity within a preset facial distance range as target bins representing the eye; and extracting amplitude and phase information of multiple bins in the neighborhood of the target bin to form amplitude temporal data and phase temporal data, respectively. In this embodiment, step S2 concatenates the data corresponding to several consecutive Chirp signals, specifically as follows: Perform IQ deDC removal on each of the Nt consecutive Chirps; The fast-time sampling sequences of Nt Chirps are concatenated end-to-end in the time domain in chronological order to form a complex IQ sequence of length Nt×N; where Nt is an integer greater than 1 and N is the number of fast-time sampling points.
[0023] Step S2 selects one or more bins with the highest signal intensity within a preset facial distance range as target bins representing the eye, specifically including: The distance range is set based on the relative distance between the radar and the face during sleep. and map it to the range of bin. ; The range of bin Within the bin, find the bin with the largest average amplitude and determine it as the target bin.
[0024] Step S2, after selecting the target bin, also includes a phase anomaly detection and bin reselection step: The phase sequence of the target bin is unwrapped and differentially calculated. If the absolute value of the phase difference exceeds a preset threshold within a specific time window, it is determined that there is body motion interference in the current window. In response to the determination of motion interference, discard the current window data and within the range of the bin. Select the target bin again.
[0025] The distance processing and eye-bin selection involved in this embodiment include the following parts: Data processing: Frequency domain analysis is performed on the data after splicing several consecutive chirps to obtain the range spectrum, thereby determining the relative position of the target.
[0026] Eye range bin selection: Within the estimated range of the face, by analyzing the echo intensity, the bin containing the largest echo is selected as the "eye bin" reference interval; this area can be flexibly set according to specific equipment and monitoring needs to ensure effective capture of subtle eye movements.
[0027] Phase anomaly detection: The phase sequence of the reference bin is analyzed. If a significant abnormal phase jump is found (e.g., exceeding a set threshold or an adaptively set threshold), it is determined to be non-eye-related interference (e.g., the subject turning over or turning their head). At this time, the eye bin is reselected or the data of the current time period is discarded to ensure the accuracy of the monitoring results.
[0028] Specifically, this embodiment splices together data corresponding to several consecutive Chirp signals, including: definition: The data contains Nc chirps; each chirp contains N fast-time sampling points.
[0029] Length of each splice group: Nt chirps.
[0030] Grouping: This yields a concatenated sequence of G = [Nc / Nt] segments (if Nc is not an integer multiple of Nt, the chirp at the end is discarded if it is insufficient).
[0031] Steps (executed for each group of Nt "time-continuous" chirps): 1) Chirp preprocessing: IQ is de-DC (mean is reduced).
[0032] 2) Time-domain concatenation: The fast time samples of Nt chirp are concatenated end to end in chronological order to obtain a complex IQ sequence of length Nt×N.
[0033] Output format: The above G spliced sequences are grouped and denoted as L[g,m]: g=1,...,G represents the g-th group; m=0,...,Nt×N-1 represents the m-th fast-time sampling point in this group.
[0034] This embodiment performs frequency domain analysis on the spliced data, including: Input: A two-dimensional sequence of complex numbers L[g,m] concatenated from beginning to end in the time domain (where g=1,...,G, m=0,...,Nt*N-1).
[0035] step: 1) Windowing to suppress spectral leakage; 2) Perform Range-FFT on L[g,m], (FFT points) Obtain the distance spectrum ; 3) A[g,r]=|X[g,r]| yields the amplitude spectrum A[g,r]; Phase spectrum obtained .
[0036] In this embodiment, within a preset facial distance range, one or more bins with the highest signal strength are selected as target bins representing the eye, including: 1) Set "Face Prediction Range": Based on the approximate relative distance between the radar and the face during sleep. (e.g., 0.4-1.2m). And convert the distance to the bin range using a distance-to-bin mapping. ; 2) Temporal amplitude spectrum integration: Given a time window W, the mean value of the amplitudes of the same bin along the time dimension (g dimension) is obtained to obtain A[r]; 3) Peak detection determines the "eye bin" (choose one): Maximum value method: within the bin range Let r0 = argmaxA[r], that is, select the bin with the strongest amplitude from the bin range as the "eye bin"; CFAR method: Perform CA-CFAR on each candidate bin within the bin range, and take the maximum amplitude above the threshold. "eyes bin".
[0037] This embodiment performs phase anomaly detection and eye bin reselection, including: Input: Eye bin phase sequence (For "eye bin"), time window W and threshold Depending on the wavelength, time window length, and radar parameters, it is usually a pure eye-movement phase change. Several times (of)
[0038] step: 1) Perform phase unwrapping and adjacent difference to obtain ; 2) If the phase difference within this time window is greater than or equal to the threshold, i.e. The current time window is marked as abnormal, and the amplitude / phase data for that period is discarded; 3) Within the bin range Re-perform peak detection to determine the "eye bin", prioritizing bins that are closer to but different from the previous time period.
[0039] 4) Continue to perform rolling detection and reselection according to the above process until no abnormality is triggered for several consecutive time windows, and normal output is restored.
[0040] The amplitude / phase dual-channel feature construction involved in this embodiment includes the following parts: Amplitude channel (primary): Centered on the "eye bin", the amplitude time series data of the first and last several bins are extracted as the primary features (the specific number of bins can be adjusted according to needs) to reflect the intensity changes of eye movements.
[0041] Phase channel (secondary): Centered on the "eye bin", extract the phase time series data of the first and last several bins as auxiliary features (the specific number of bins can be adjusted according to needs) to capture changes in eye movement and as a basis for body movement detection.
[0042] Data preprocessing: After extraction, the data from both channels undergo a denoising step to eliminate external interference, improve the signal-to-noise ratio, and ensure stable and reliable eye-tracking detection results.
[0043] Specifically, the acquisition of phase timing data and amplitude timing data in this embodiment includes: step: 1) Neighborhood selection: by Centered on the "eye bin", take M+N+1 bins from the neighborhood before and after the "eye bin"; Neighborhood index set: O={-M,...,-1,0,+1,...,+N}, where the sizes of M and N are determined based on the approximate distance from the face to the radar.
[0044] 2) Boundary handling: If a bin exceeds the distance spectrum boundary, that bin will be pruned and discarded.
[0045] 3) Read amplitude and phase: Amplitude: Extract amplitude time series data from the selected neighborhood in the amplitude spectrum. ; Phase: Extract the phase time series data of the selected neighborhood from the phase spectrum. .
[0046] S3. Divide the amplitude and phase dual-channel timing features into second-level time windows. Based on the dual-channel timing features within each time window, determine whether the window belongs to a state of no eye movement, slow eye movement, or intense eye movement.
[0047] In this embodiment, the windowing determination and statistical calculation involved include the following: Windowing processing: The acquired data is divided into windows on a second-by-second basis (e.g., adjustable windows of 1 second, 2 seconds, 3 seconds, etc.), and slow eye movement, intense eye movement, and no eye movement states are determined window by window (or eye movement and no eye movement states are determined). The window duration can be flexibly selected according to actual needs to adapt to different accuracy requirements.
[0048] Statistical calculation: For each minute-level data collection segment (the specific time window can be set according to actual needs), the proportion of eye movement and the degree of eye movement are statistically analyzed, which are used as indicators for stage-by-stage sleep monitoring.
[0049] Specifically, the windowing determination in this embodiment includes: step: 1) Window Partitioning: Divide the obtained amplitude and phase time series data into windows of length Wsec (on a second scale) (it is recommended to match the window length for phase anomaly detection). Time window .
[0050] 2) Window-level determination: Input the “amplitude main branch + phase auxiliary branch” data of each time window into the trained model / classifier to obtain the classification result yi∈{0: no eye movement state; 1: slow eye movement; 2: violent eye movement}.
[0051] The statistical calculations performed in this embodiment include: step: 1) Let the length of the minute-level statistical window be Mmin∈{5,10,20} minutes, and the step size be a sliding window of 1-5 minutes. Map all second-level window decisions yi to this minute segment.
[0052] 2) Based on the second-level window, calculate the proportion of eye movements MR=∑(yi=1、2)*Wsec / Mmin, the proportion of intense eye movements FR=∑(yi=2)*Wsec / Mmin, the number of eye movements Nm=∑(yi=1、2), etc.
[0053] Step S3 determines the eye movement state based on the dual-channel temporal features within each time window by inputting the dual-channel temporal features into a trained classification model.
[0054] The classification models include: The spatial feature extraction module is used to extract spatial features from amplitude time-series features and phase time-series features, respectively. The temporal modeling module is used to model the temporal dependencies of features after spatial feature extraction. The weighted fusion module is used to perform weighted fusion of features from the amplitude channel and the phase channel, where the weight of the amplitude channel is greater than that of the phase channel. The classifier module is used to output the classification result of eye movement state based on the weighted fused features.
[0055] The spatial feature extraction module employs a convolutional neural network or a graph convolutional network; the temporal modeling module employs a long short-term memory network, a gated recurrent unit, or a self-attention mechanism model.
[0056] The model structure involved in this embodiment includes the following: Dual-branch input: The model employs a dual-branch input structure, with amplitude and phase features input separately. Each branch is processed by a suitable spatial feature extraction module to capture local spatiotemporal texture information. The spatial feature extraction module can use any model suitable for the task requirements, not limited to traditional convolutional neural networks; for example, deep neural networks, graph convolutional networks, and other spatial feature extraction methods can also be used.
[0057] Temporal Modeling: After spatial feature extraction, the outputs of each branch are concatenated along the time dimension and fed into a suitable temporal modeling module. This module captures time-series dependencies and dynamic changes across windows. It is not limited to traditional Long Short-Term Memory (LSTM) networks; other temporal modeling methods, such as gated recurrent units, self-attention-based models, or any structure suitable for time-series data can be used.
[0058] Weighted fusion: Learnable or configurable weights are introduced into the fusion layer or loss layer to highlight the dominant role of amplitude features while combining the auxiliary role of phase features. This weighted fusion mechanism can be implemented at different levels of the model, whether before or after time series modeling, and can even be optimized through other fusion techniques, such as weighted averaging and attention mechanisms, to improve the integration of amplitude and phase features.
[0059] Classification Head: The design of the classification part is not limited to traditional fully connected layers and Softmax structures, but adopts flexible classification methods. The classification head can use any classifier structure suitable for the task, such as support vector machines, attention mechanisms, or other non-fully connected feature aggregation methods, thereby adapting to different task requirements, such as binary classification or multi-class classification.
[0060] Data augmentation: To improve the model's generalization ability, data augmentation methods such as random gain and phase jitter are applied to the amplitude and phase features during training. Data augmentation strategies can be further designed flexibly, including but not limited to noise injection, temporal perturbation, and spatial transformation, to improve the model's robustness under various environments and conditions.
[0061] In an optional embodiment of the present invention, the method further includes: Based on the eye movement status determination results within each second-level time window of the minute-level time period, the proportion and / or degree of eye movement are statistically analyzed, and the sleep stage of the target is inferred based on the statistical results.
[0062] The eye movement percentage is the percentage of time within a minute-level time period where eye movement is identified as occurring within a second-level window; the eye movement intensity is the percentage of time within a minute-level time period where intense eye movement is identified within a second-level window and / or the number of eye movements.
[0063] The sleep stage inference involved in this invention includes the following: Stage inference method: Combining the proportion of eye movements, intensity of eye movements, and frequency of eye movements in the closed-eye state, the sleep stage of each time window is inferred using known clinical sleep staging standards (e.g., REM / N1 / N2 / N3). The threshold and mapping strategy can be adjusted according to specific applications (e.g., using custom sleep stage classification standards) to adapt to different monitoring needs.
[0064] Output results: Outputs the sleep stage sequence, eye movement percentage curve, eye movement intensity, and corresponding confidence interval for each time period, helping to conduct more accurate sleep monitoring and analysis.
[0065] The anomaly and robustness handling involved in this invention includes the following: Body movement / turning over detection: Eye bins are promptly removed or repositioned through phase anomaly detection to avoid the influence of external interference such as body movement or turning over on the monitoring results. If a phase jump is detected and determined to be body movement or turning over, the system will automatically adjust the eye movement monitoring range or discard the data for that period to ensure the stability and reliability of the monitoring.
[0066] This invention provides a non-contact eye-tracking monitoring system based on millimeter-wave sensing to implement the above-described method. The system includes: Millimeter-wave radar module, used to transmit continuous chirp signals and receive echoes; The data processing module is configured as follows: The radar echo data is processed by splicing together the data corresponding to several consecutive Chirp signals. Frequency domain analysis was performed on the stitched data to obtain the distance spectrum; Within a preset facial distance range, a target bin representing the eye is selected, and the amplitude and phase information of multiple bins in the neighborhood are extracted with the target bin as the center to construct amplitude and phase dual-channel temporal features; The eye movement state determination module is configured to divide the dual-channel timing features into second-level time windows and determine the eye movement state within each window.
[0067] The feasibility analysis of the method and system of the present invention will be explained below with specific examples.
[0068] The example uses a single device, bedside installation scenario: 1. Install the mmWave radar on the side of the bed, 0.5-1.0m from the head, at a height slightly higher than the pillow surface; 2. Automatic nighttime data collection: Collect one segment every 5 minutes (to determine the sleep stage once); 3. For each data segment: The data obtained by splicing consecutive chirps (such as 3 chirps) undergoes frequency domain analysis to obtain a range spectrum, thereby determining the relative position of the target.
[0069] Within the facial estimation range, by analyzing the echo intensity, the bin containing the largest echo is selected as the "eye bin" reference interval, and phase anomaly detection is performed; The amplitude and phase time series data of the first few bins (e.g., the first 4 bins) and the last few bins (e.g., the last 10 bins) are extracted, preprocessed, and denoised before being used as features. The collected data is divided into second-level windows (e.g., 2-second windows). The segmented data is input into the model and classifier to determine slow eye movement, intense eye movement, and no eye movement. Eye movement percentage and degree are statistically analyzed for minute-level data (such as 10-minute sleep stage assessments, with the specific time window set according to actual needs) to serve as a basis for staged sleep monitoring.
[0070] Two subjects sat facing the radar and closed their eyes to simulate no eye movement during sleep, slow eye movement during sleep, and violent eye movement during sleep. Subjects were photographed multiple times at different locations to simulate the non-fixed head position during sleep: As shown in Figures 2(a) to 2(c), the differences in single rangebin phase time series data of the eyes under conditions of no eye movement with eyes closed, slow eye movement with eyes closed, and intense eye movement with eyes closed are as follows: The X-axis represents time (chirp count), and the Y-axis represents phase. The flatter the curve, the smaller the phase timing change; the more tortuous the curve, the larger the phase timing change.
[0071] As shown in Figures 3(a) to 3(c), the differences in amplitude time series data under the conditions of no eye movement with eyes closed, slow eye movement with eyes closed, and violent eye movement with eyes closed are as follows: The X-axis represents time (chirp count), and the Y-axis represents the rangebin (straight-line distance from the radar). In the heatmap, brighter colors represent larger amplitudes, and darker colors represent smaller amplitudes.
[0072] If the color of the heatmap does not change significantly with the increase of the y-axis, then there is no significant change in the amplitude over time. If the color of the heatmap changes significantly as the y-axis increases, then there is a significant change in the amplitude over time.
[0073] In summary, the beneficial effects of the present invention are as follows: Detectable with eyes closed: Unlike traditional millimeter-wave eye movement monitoring schemes that rely on pupil and iris reflection signals, the method of this invention can effectively monitor eye movements in a closed-eye state. Traditional schemes usually require the eyes to be open and facing the radar, while the method of this invention is designed specifically for a closed-eye state. By detecting eyelid micro-movements and periocular muscle movements, it does not require light or infrared sources, making it suitable for closed-eye scenarios such as sleep, thus improving privacy and comfort; Low-light robustness: The method of this invention is not sensitive to lighting conditions and can work stably in completely dark or low-light environments, ensuring reliable monitoring under various lighting conditions.
[0074] Non-contact high compliance: The method of this invention adopts non-contact detection technology, which does not require patches or cables, and is suitable for long-term, comfortable home use, thereby improving user compliance and user experience.
[0075] Balancing resolution and real-time performance: The method of this invention processes continuous data streams, splicing together several consecutive data segments (e.g., consecutive chirp streams) and then processing them through frequency domain analysis. This can effectively improve the spatial resolution of the monitoring system while also meeting real-time requirements, making it suitable for dynamic monitoring scenarios.
[0076] Anti-eye movement interference: The method of this invention uses a phase jump detection mechanism to determine whether there are non-eye interference factors (such as body movement or turning over), and promptly relocates or discards the interfered data in the eye movement monitoring area (bin), thereby significantly reducing the impact of body movement on eye movement monitoring results and improving data quality.
[0077] End-to-end recognition: The method of this invention combines a spatial feature extraction module and a temporal modeling module for dual-branch weighted fusion, with amplitude features as the main feature and phase features as the supplement. This can effectively improve the recognition accuracy of minute movements with eyes closed and ensure high accuracy of eye movement detection in the closed eye state. Easy to apply: The method of this invention, which is based on the stage inference method of minute-level eye movement proportion and eye movement degree, is highly consistent with the clinical sleep staging process, making it easy to integrate with existing clinical testing processes and have good applicability.
[0078] The differences and advantages of this invention compared to the millimeter-wave "eye-opening and eye-tracking" solution are as follows: Different scenario positioning: Existing millimeter-wave eye-tracking technology is mainly used in awake (eyes-open) scenarios, typically applicable to gaze tracking, human-computer interaction, and fatigue driving. However, during sleep, the usability of traditional open-eye methods is extremely low because the subject's eyes are closed and body position is variable. This invention is specifically designed for sleep with eyes closed, providing more accurate and stable eye-tracking monitoring regardless of the subject's position or eye-closed state.
[0079] Different sensory mechanisms: Existing open-eye millimeter-wave technology primarily relies on strong reflected signals from the eye, focusing on eye movements and changes in pupil and iris position. With eyes open, these strong reflected signals can clearly capture eye movements and pupil position changes. However, with eyes closed, these strong reflected signals are significantly weakened, making it difficult to effectively detect eye movements.
[0080] This invention is based on the subtle movements of the eyelids and the minute changes in the muscles around the eyes when the eyes are closed, and detects these movements through weak amplitude variations and phase perturbations in millimeter-wave signals. Continuous signal processing (such as time stitching and frequency domain analysis) improves distance resolution, thus enabling stable eye movement detection even with the eyes closed. Unlike techniques using open-eye techniques, this invention focuses on subtle changes in eye movement, especially the minute movements of the eyelids and muscles around the eyes, which is particularly important in sleep monitoring.
[0081] The target and time granularity are different: Existing eye-opening protocols typically focus on monitoring short-lived events, such as rapid eye movements (REMs) like saccades or blinks, with a relatively short time granularity. In contrast, this invention uses a second-level window to determine "slow eye movement," "rapid eye movement," or "no eye movement" states, and uses the percentage of eye movements and the intensity of eye movements within a minute-level timeframe as statistical indicators for sleep staging. This time granularity more closely aligns with the statistical characteristics of clinical sleep staging, better reflects the periodic changes in eye movements during sleep, and has higher compatibility with traditional sleep monitoring methods. Data processing differs from modeling: Existing eye-opening technologies often rely on strong reflective signals from the eyeball and iris, as well as pupil position changes, to track eye movements or determine blinks, and then use this as a basis for data processing and model selection. However, these data processing and models are not suitable for eye movement monitoring in the closed-eye state or for positional changes during sleep. Therefore, this invention selects phase and amplitude data to construct a dual-channel feature model. After data extraction, abnormal data is promptly removed and the eye position is repositioned through phase anomaly detection to avoid the influence of external interference such as body movement or turning over on the monitoring results. In terms of modeling, this invention constructs a dual-channel temporal feature model including amplitude and phase, focusing on the micro-motion characteristics in the closed-eye state. Amplitude and phase features are distinguished at a fine-grained level using several bins. This feature model fully utilizes the signal-to-noise characteristics and temporal dependencies in the closed-eye state through weighted fusion of spatial feature extraction and temporal modeling. Furthermore, the model structure exhibits high robustness to positional changes.
[0082] In this invention, the application of the model is not limited to spatial feature extraction and temporal modeling. In fact, different types of models can be flexibly selected according to different needs. Other models, such as convolutional networks, self-attention mechanisms, statistical models, and hybrid models, can all adapt to different data characteristics and task requirements. This flexibility and scalability of the model enable this invention to adapt to eye movement monitoring needs in various eye-closed states, providing higher detection accuracy and robustness.
[0083] The essential differences between this invention and the prior art are as follows: This invention is not a simple transfer of the "open-eye millimeter-wave eye movement" scheme to a sleep scenario, but rather a systematic innovation in the target object, signal processing chain, and statistical staging strategy: Target group: Individuals with eyes closed who are prone to changing body position. Signal processing chain: continuous signal processing (such as time splicing and frequency domain analysis) → "eye-bin" selection → phase jump elimination and eye retargeting → amplitude main channel / phase auxiliary channel construction → second-level window determination → minute-level statistical period determination; Model structure: A classifier is used to determine the classifier after a weighted fusion of spatial feature extraction and temporal modeling. Different types of models can be flexibly selected according to different needs. Other models include convolutional networks, self-attention mechanisms, statistical models, and hybrid models. Statistical staging strategy: Eye movement judgment at the second level + eye movement statistics over a time period at the minute level Application compatibility: It can still operate stably under conditions of closed eyes, dim light, and changes in body position, solving the problem of low usability of the open-eye method under sleep conditions.
[0084] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0086] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0087] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.
[0088] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.
Claims
1. A non-contact eye-tracking monitoring method based on millimeter-wave sensing, characterized in that, Includes the following steps: Raw data is obtained by transmitting continuous Chirp signals and receiving echoes using millimeter-wave radar. The raw data is processed to construct amplitude and phase dual-channel temporal features for characterizing micro-movements of the eyelids and periocular region in a closed-eye state. The processing includes: splicing together data corresponding to several consecutive Chirp signals; performing frequency domain analysis on the spliced data to obtain a distance spectrum containing multiple distance unit bins; selecting one or more bins with the highest signal intensity within a preset facial distance range as target bins representing the eye; and extracting amplitude and phase information from multiple bins in the neighborhood of the target bin to form amplitude temporal data and phase temporal data, respectively. The amplitude and phase dual-channel timing features are divided into second-level time windows. Based on the dual-channel timing features within each time window, it is determined whether the window belongs to a state of no eye movement, slow eye movement, or intense eye movement.
2. The non-contact eye-tracking monitoring method based on millimeter-wave sensing according to claim 1, characterized in that, The process of concatenating data corresponding to several consecutive Chirp signals specifically involves: Perform IQ deDC removal on each of the Nt consecutive Chirps; The fast-time sampling sequences of Nt Chirps are concatenated end-to-end in the time domain in chronological order to form a complex IQ sequence of length Nt×N; where Nt is an integer greater than 1 and N is the number of fast-time sampling points.
3. The non-contact eye-tracking monitoring method based on millimeter-wave sensing according to claim 1, characterized in that, Within a preset facial distance range, selecting one or more bins with the highest signal strength as target bins representing the eye specifically includes: The distance range is set based on the relative distance between the radar and the face during sleep. and map it to the range of bin. ； The range of bin Within the bin, find the bin with the largest average amplitude and determine it as the target bin.
4. The non-contact eye-tracking monitoring method based on millimeter-wave sensing according to claim 3, characterized in that, After selecting the target bin, the process also includes phase anomaly detection and bin reselection steps: The phase sequence of the target bin is unwrapped and differentially calculated. If the absolute value of the phase difference exceeds a preset threshold within a specific time window, it is determined that there is body motion interference in the current window. In response to the determination of motion interference, discard the current window data and within the range of the bin. Select the target bin again.
5. The non-contact eye-tracking monitoring method based on millimeter-wave sensing according to claim 1, characterized in that, The eye movement state determination based on dual-channel temporal features within each time window is achieved by inputting the dual-channel temporal features into a trained classification model.
6. The non-contact eye-tracking monitoring method based on millimeter-wave sensing according to claim 5, characterized in that, The classification model includes: The spatial feature extraction module is used to extract spatial features from amplitude time-series features and phase time-series features, respectively. The temporal modeling module is used to model the temporal dependencies of features after spatial feature extraction. The weighted fusion module is used to perform weighted fusion of features from the amplitude channel and the phase channel, where the weight of the amplitude channel is greater than that of the phase channel. The classifier module is used to output the classification result of eye movement state based on the weighted fused features.
7. The non-contact eye-tracking monitoring method based on millimeter-wave sensing according to claim 6, characterized in that, The spatial feature extraction module employs a convolutional neural network or a graph convolutional network; the temporal modeling module employs a long short-term memory network, a gated recurrent unit, or a self-attention mechanism model.
8. The non-contact eye-tracking monitoring method based on millimeter-wave sensing according to claim 1, characterized in that, Also includes: Based on the eye movement status determination results within each second-level time window of the minute-level time period, the proportion and / or degree of eye movement are statistically analyzed, and the sleep stage of the target is inferred based on the statistical results.
9. A non-contact eye-tracking monitoring method based on millimeter-wave sensing according to claim 8, characterized in that, The eye movement percentage is the percentage of time within a minute-level time period where eye movement is identified as occurring within a second-level window; the eye movement intensity is the percentage of time within a minute-level time period where intense eye movement is identified within a second-level window and / or the number of eye movements.
10. A non-contact eye-tracking monitoring system based on millimeter-wave sensing, characterized in that, The system for implementing the method as described in any one of claims 1 to 9 comprises: Millimeter-wave radar module, used to transmit continuous chirp signals and receive echoes; The data processing module is configured as follows: The radar echo data is processed by splicing together the data corresponding to several consecutive Chirp signals. Frequency domain analysis was performed on the stitched data to obtain the distance spectrum; Within a preset facial distance range, a target bin representing the eye is selected, and the amplitude and phase information of multiple bins in the neighborhood are extracted with the target bin as the center to construct amplitude and phase dual-channel temporal features; The eye movement state determination module is configured to divide the dual-channel timing features into second-level time windows and determine the eye movement state within each window.