Driver dry eye risk assessment method, device and electronic equipment

Abstract

The application provides a driver dry eye risk assessment method and device and electronic equipment. First, key point positioning is performed on the facial image data of the driver collected by the driver monitoring system, and then blink detection, eye rubbing detection and squeeze blinking detection are performed based on the key point positioning results. Then, core basic feature extraction is performed based on the blink detection results, eye rubbing detection results and squeeze blinking detection results, and a time sequence is constructed based on the core basic feature extraction results. Then, based on the time sequence, a dry eye risk score prediction is performed through a pre-trained deep learning model. Finally, based on the dry eye risk score prediction result, a dry eye risk level determination is performed according to a preset risk level determination rule. The driver dry eye disease risk can be quantitatively and accurately evaluated by using the application, thereby ensuring driving safety.

Description

Technical Field

[0001] This invention relates to the field of driving state recognition technology, and in particular to a method, device and electronic device for assessing the risk of dry eye syndrome in drivers. Background Technology

[0002] Currently, the mainstream solutions for non-contact driver monitoring systems (DMS) based on vehicle cameras generally use near-infrared cameras to continuously capture facial images and extract biological features such as eyelid opening and closing, blinking frequency, gaze direction, and pupil light reflection through deep learning models to provide real-time warnings of driving status (such as fatigue, distraction, drunk driving, driving under the influence of drugs, etc.).

[0003] Drivers are generally at a much higher risk of developing dry eye syndrome than other populations; however, existing DMS (Dry Eye Monitoring System) cannot quantify the risk of chronic eye diseases (especially dry eye syndrome) in drivers. Furthermore, traditional dry eye detection methods heavily rely on near-field eye imaging devices, which are too large to be used in vehicles, while in-vehicle cameras have limited accuracy for long-distance detection. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, device and electronic device for assessing the risk of dry eye syndrome in drivers, so as to alleviate the above-mentioned problems existing in the related art.

[0005] In a first aspect, embodiments of the present invention provide a method for assessing the risk of dry eye syndrome in drivers, comprising: locating key points in facial image data of a driver collected by a driver monitoring system, and performing blink detection, eye rubbing detection, and squeeze blink detection based on the key point location results; extracting core basic features based on the blink detection results, eye rubbing detection results, and squeeze blink detection results, and constructing a time series based on the core basic feature extraction results; predicting a dry eye syndrome risk score using a pre-trained deep learning model based on the time series; and determining the dry eye syndrome risk level according to a preset risk level determination rule based on the predicted dry eye syndrome risk score.

[0006] Secondly, embodiments of the present invention also provide a driver dry eye risk assessment device, comprising: a detection module for locating key points in facial image data of a driver collected by a driver monitoring system, and performing blink detection, eye rubbing detection, and squeeze blink detection based on the key point location results; a construction module for extracting core basic features based on the blink detection results, eye rubbing detection results, and squeeze blink detection results, and constructing a time series based on the core basic feature extraction results; a prediction module for predicting a dry eye risk score based on the time series using a pre-trained deep learning model; and a judgment and warning module for determining the dry eye risk level based on the predicted dry eye risk score and according to preset risk level judgment rules.

[0007] Thirdly, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the driver dry eye risk assessment method described in the first aspect above.

[0008] This invention provides a method, device, and electronic device for assessing the risk of dry eye syndrome in drivers. First, key point localization is performed on facial image data of the driver collected by a driver monitoring system. Based on the key point localization results, blink detection, eye rubbing detection, and squeeze blink detection are performed. Then, core basic features are extracted based on the blink, eye rubbing, and squeeze blink detection results, and a time series is constructed based on the extracted core basic features. Next, based on the time series, a pre-trained deep learning model is used to predict a dry eye syndrome risk score. Finally, based on the predicted dry eye syndrome risk score, a dry eye syndrome risk level is determined according to preset risk level judgment rules. Using the above technology, facial image data of the driver can be collected by a driver monitoring system. Key point localization, blink detection, eye rubbing detection, squeeze blink detection, and core basic feature extraction are performed on the collected facial image data. A time series is constructed, and the dry eye syndrome risk level of the driver is identified using a deep learning model and preset risk level judgment rules. This allows for a relatively accurate quantitative assessment of the driver's risk of developing dry eye syndrome, thereby helping to ensure driving safety.

[0009] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0010] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0011] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a method for assessing the risk of dry eye syndrome in drivers, as described in an embodiment of the present invention. Figure 2 This is a flowchart of the blink detection algorithm in an embodiment of the present invention; Figure 3 This is a flowchart of the eye rubbing detection algorithm in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the driver dry eye risk assessment method in an embodiment of the present invention. Figure 5 This is a schematic diagram of the structure of a driver dry eye risk assessment device according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Currently, mainstream non-contact DMS solutions based on vehicle cameras generally use near-infrared cameras to continuously capture facial images and extract biometric features such as eyelid opening and closing, blinking frequency, gaze direction, and pupillary light reflex through deep learning models to provide real-time warnings of driving status (such as fatigue, distraction, drunk driving, and driving under the influence of medication). However, existing DMS systems cannot quantitatively assess the risk of dry eye syndrome. Furthermore, traditional dry eye detection solutions heavily rely on near-field eye imaging devices that are too large to be used in vehicles, and the long-range detection accuracy of vehicle cameras is limited.

[0015] Based on this, the present invention provides a driver dry eye risk assessment method, device, and electronic device, which can alleviate the above-mentioned problems existing in related technologies.

[0016] To facilitate understanding of this embodiment, a method for assessing the risk of dry eye syndrome in drivers disclosed in this invention will first be described in detail. (See [link to relevant documentation]). Figure 1 As shown, the method may include the following steps: Step S102: Perform key point localization on the driver's facial image data collected by the driver monitoring system, and perform blink detection, eye rubbing detection and squeeze blink detection based on the key point localization results.

[0017] Step S104: Extract core basic features based on blink detection results, eye rubbing detection results, and squeeze blink detection results, and construct a time series based on the core basic feature extraction results.

[0018] Step S106: Based on the time series, predict the risk score of dry eye disease using a pre-trained deep learning model.

[0019] Step S108: Based on the dry eye risk score prediction results, determine the dry eye risk level according to the preset risk level determination rules.

[0020] This invention provides a method for assessing the risk of dry eye syndrome in drivers. First, key point localization is performed on facial image data of the driver collected by a driver monitoring system. Based on the key point localization results, blink detection, eye rubbing detection, and squeeze blink detection are performed. Then, core basic features are extracted based on the blink, eye rubbing, and squeeze blink detection results, and a time series is constructed based on the extracted core basic features. Next, a dry eye syndrome risk score is predicted using a pre-trained deep learning model based on the time series. Finally, the dry eye syndrome risk level is determined according to a preset risk level judgment rule based on the predicted dry eye syndrome risk score. Using this technology, facial image data of the driver can be collected by a driver monitoring system. Key point localization, blink detection, eye rubbing detection, squeeze blink detection, and core basic feature extraction are performed on the collected facial image data. A time series is constructed, and the driver's dry eye syndrome risk level is identified using a deep learning model and preset risk level judgment rules. This method can accurately quantify and assess the risk of dry eye syndrome in drivers, thereby helping to ensure driving safety.

[0021] As one possible implementation, the key point localization result includes eye key point localization information; based on this, the blink detection, eye rubbing detection, and squeeze blink detection performed based on the key point localization result in step S102 above may include: Step A1: Calculate the eye aspect ratio (EAR) based on the eye key point positioning information to obtain the driver's real-time EAR time-series data.

[0022] Step A2: Perform blink waveform detection on the real-time EAR time series data, and determine the blink detection result based on the blink waveform detection result and the first threshold information.

[0023] For example, the blink detection result may include the number of complete blinks, the number of incomplete blinks, and the blink completion rate. The first threshold information may include a first EAR threshold, a second EAR threshold, and a micro-flicker threshold, wherein the first EAR threshold is less than the second EAR threshold. The step A2 above, which determines the blink detection result based on the blink waveform detection result and the first threshold information, may include: if the blink waveform detection result contains detected complete blink waveforms, then extracting the minimum EAR value in each complete blink waveform; counting the number of complete blink waveforms whose minimum EAR value is not greater than the first EAR threshold and whose duration is greater than the micro-flicker threshold as the number of complete blinks; counting the number of complete blink waveforms whose minimum EAR value is greater than the first threshold and not greater than the second EAR threshold and whose duration is greater than the micro-flicker threshold as the number of incomplete blinks; and determining the blink completion rate (BCR) based on the number of complete blinks and the number of incomplete blinks.

[0024] The driver's facial image data (which can be a sequence of video frames) collected by the DMS camera can be used to independently monitor complete and incomplete blinking behaviors. Then, a blink detection algorithm (including key point localization of the driver's facial image data and blink detection based on the key point localization results) can be used to achieve blink detection. Figure 2 As shown, the specific implementation steps of the blink detection algorithm are as follows: Step S11: Localization of key eye points and calculation of EAR.

[0025] In the images acquired by DMS (i.e., real-time camera data), the system uses facial landmark detection algorithms (such as PIPNet) to locate the contours of the left and right eyelids and detect key eye landmarks. For a single eye, six key feature points are defined: the inner corner of the eye... outer corner of the eye Feature points along the upper eyelid , Characteristic points along the lower eyelid line , .

[0026] Following the previous example, to quantify the degree of eyelid closure, the system uses the eye aspect ratio (EAR) as the evaluation index. The formula for calculating the EAR value of a single eye is as follows:

[0027] in, This represents the Euclidean distance between two feature points. Characterizes the distance the eyes open and close in the vertical direction. Characterizes the width of the eye in the horizontal direction (used for normalization to eliminate the influence of variations in face distance). Final real-time eye aspect ratio. Take left and right eyes The arithmetic mean of the calculated values.

[0028] Step S12, threshold construction.

[0029] Following the previous example, in order to accommodate the different eye physiological characteristics of drivers (such as eye size and palpebral fissure width), the system collects data during the initial startup phase (which can be considered the calibration phase). The driver's eye movement (EAR) data while awake (e.g., collecting EAR data for 5 seconds) is used. The most frequently occurring EAR value in the EAR data is used as the baseline value B for normal eye opening. As the threshold for closing the eyes, take As an incomplete blink threshold, the system can save the set threshold locally.

[0030] Step S13, complete blinking is counted independently.

[0031] Following the previous example, the system continues to monitor... The temporal waveform is calculated. Since the eye's EAR value begins to decrease after a blink, and continues to decrease until it falls below a certain threshold (e.g., the eye-closing threshold), the eye is considered closed. Then, the EAR value begins to rise until it exceeds a corresponding threshold (e.g., the incomplete blink threshold), at which point the eye is considered open. This process completes one blink. Therefore, the moment when the EAR value drops below the corresponding threshold can be considered the start of a blink, and the moment when the EAR value rises above the corresponding threshold can be considered the end of a blink. When a complete blink waveform is detected (i.e., the EAR value drops below the eye-closing threshold and then rises back above the incomplete blink threshold), the minimum value in this blink waveform is extracted. If the conditions are met And the duration of the blink waveform (equivalent to one blink action) If the blink exceeds the micro-flicker threshold (e.g., 200ms, which can be determined based on the average blink time), the system determines the blink as a valid "complete blink." This type of blink effectively remodels the tear film. The system records the time of the blink and accumulates the number of complete blinks. .

[0032] Step S14: Independent statistics on incomplete blinking.

[0033] Continuing from the previous example, in parallel with step S13, the system performs incomplete closure analysis on the same waveform (i.e., the detected complete blink waveform): if the minimum value in the detected complete blink waveform... Meet the conditions And the duration of that blinking action If the blink exceeds the micro-flicker threshold (e.g., 200ms), it is considered not noise, and the system classifies the blink as an "incomplete blink." This type of blink leads to prolonged exposure of the lower cornea, a core indicator of evaporative dry eye syndrome. In this case, the incomplete blink counter... Add 1 to the number of incomplete blinks. Accumulated.

[0034] Step S15: Calculate the blink completion rate.

[0035] Continuing from the previous example, after obtaining the complete number of blinks... and the number of incomplete blinks Next, the system further calculates the blink completion rate (BCR). BCR is the ratio of the number of complete blinks to the total number of complete blinks (i.e., full blinks) and incomplete blinks (i.e., incomplete blinks). The calculation formula is as follows:

[0036] Step S16: Dry eye feature output.

[0037] Following the previous example, the system outputs according to a preset cycle. , As a characteristic of dry eye. Furthermore, the system outputs... , It can also output the corresponding blink completion rate (BCR).

[0038] As one possible implementation, step S102 above, which involves performing blink detection, eye rubbing detection, and squeeze-type blink detection based on the key point localization results, may further include: Step B1: Perform face detection on the facial image data, and obtain the region of interest (ROI) image of the eyes based on the face detection results and eye key point localization information; wherein, the face detection results may include face detection boxes, and the ROI image of the eyes may contain complete information on the periorbital muscle groups and have time point information.

[0039] Step B2: Based on the obtained region of interest (ROI) images, a pre-trained classification model is used to predict the squeezed-eye-closing state. The prediction result includes the classification of whether each ROI image belongs to the squeezed-eye-closing state.

[0040] Step B3: Perform squeeze-type eye-closing state detection based on the squeeze-type eye-closing state prediction results, and determine the squeeze-type blinking detection results based on the squeeze-type eye-closing state detection results and the second threshold information.

[0041] For example, the result of the squeeze-type eye closure state detection may include each detected squeeze-type eye closure state and its duration; the squeeze-type eye closure state detection based on the squeeze-type eye closure state prediction result in step B3 above may include: if there is a first category belonging to the squeeze-type eye closure state, then record the time point information of the target eye region of interest image corresponding to each first category; based on the time point information of all target eye region of interest images, determine each squeeze-type eye closure state and its duration.

[0042] For example, the squeeze blink detection result includes the number of squeeze blinks, and the second threshold information may include a squeeze blink time threshold; the step B3 above, in which the squeeze blink detection result is determined based on the squeeze blink state detection result and the second threshold information, may include: counting the number of squeeze blink states with a duration greater than the squeeze blink time threshold as the number of squeeze blinks.

[0043] It can identify squeeze blinks and count the number of squeeze blinks based on visual detection. The specific process for squeeze blink detection is as follows: Step S21, face detection.

[0044] Following the previous example, for the video frames acquired by DMS, the system first uses a face detection algorithm to locate the driver's face bounding box. After acquiring the face region (located within the face bounding box), it combines the coordinates of the key eye points located in step S11 to extract the region of interest (ROI) image containing the complete periorbital muscle group. The intense contraction of the periorbital muscles and the deepening of wrinkles around the eyes are key visual features distinguishing ordinary blinking from squeeze blinking.

[0045] Step S22: Classification model construction and feature preprocessing.

[0046] Continuing the previous example, to accurately identify squeeze blinks, the system pre-trains and deploys a lightweight convolutional neural network (such as MobileNetV2) as a binary classifier. Before feeding the eye ROI of the current frame into the lightweight convolutional neural network, the system preprocesses the eye ROI image of the current frame. The preprocessing mainly includes: grayscale conversion and normalization of the eye ROI image, and uniform scaling to the standard resolution input image required by the model (e.g., containing...). (pixels) to reduce the impact of different people's features on the model's feature extraction.

[0047] Step S23: Classification and probability output of squeeze-type blinking.

[0048] Continuing from the previous example, the preprocessed eye ROI image is input into the trained classification model (i.e., the binary classifier mentioned above); after forward propagation through convolutional and fully connected layers, the classification model outputs whether the current frame image belongs to the squeezed-eye closed state. When satisfied At that time, the system determines that the driver's eyes are in a squeezed-closed state in the current frame image.

[0049] Step S24: Independently count squeeze-type blinks.

[0050] Continuing from the previous example, the system continuously monitors the classification results of adjacent frames on the timeline. When a continuous "squeezing eye closing" state (which can be considered as a squeezing eye closing action) is detected, its start and end times are recorded. If the duration of this squeezing eye closing action is... If the blinking action exceeds a preset threshold (e.g., 250ms, used to exclude noise such as normal rapid eye closing), the system determines that the complete action is a valid blinking action. This type of action is usually an active compensatory tear squeezing behavior caused by extreme dryness of the ocular surface or severe visual fatigue. The system records the occurrence time of this valid blinking action and accumulates the number of blinking actions. .

[0051] Step S25: Dry eye feature output.

[0052] The system outputs the number of squeeze blinks according to a preset cycle. As a characteristic of dry eye, this feature data, which represents the degree of eye fatigue and dryness, will work in conjunction with other blinking indicators to complete the determination of dry eye condition.

[0053] As one possible implementation, the eye rubbing detection result may include the number of eye rubbings, and the key point localization result may also include key point localization information for the hand, forehead, and nose tip; based on this, the blink detection, eye rubbing detection, and squeeze blink detection based on the key point localization result in step S102 above may further include: Step C1: Based on eye key point positioning information and hand key point positioning information, generate real-time potential eye rubbing time-series data representing the driver's hand-eye proximity state.

[0054] Step C2: If the real-time potential eye-rubbing time-series data includes a second candidate interval whose duration of hand-eye proximity is not less than a preset duration threshold, then the number of second candidate intervals that meet the second condition is determined as the number of eye-rubbing times based on the forehead key point positioning information and the nose tip key point positioning information; wherein, meeting the second condition can characterize that the driver's hand is not significantly higher than the forehead and the hand is not close to the nose tip.

[0055] Eye rubbing detection can be achieved using an algorithm that includes locating key points in the driver's facial image data and detecting eye rubbing based on the key point location results. For example... Figure 3 As shown, the specific implementation steps of the eye rubbing detection algorithm are as follows: Step S31, synchronous detection of key points.

[0056] Continuing from the previous example, the system calls facial landmark detection models (such as PIPNet) and hand detection models (such as YOLOv5n) to extract key points in the eye region (left and right eye contours) and hand key points (fingertips, knuckles, joints, etc.) in real time. The process of extracting key points in the eye region is exactly the same as the eye key point localization and EAR calculation process in step S11 above. The process of extracting key points in the hand region is similar to that of extracting key points in the eye region (the purpose is to locate the key points in the hand to obtain their coordinates), so it will not be described in detail here.

[0057] Step S32, determine the hand-eye distance.

[0058] Continuing from the previous example, the system calculates the two-dimensional Euclidean distance between hand key points and the center point of the eye. To ensure the distance threshold remains effective under different image resolutions and varying distances between the face and the camera, the system normalizes the two-dimensional Euclidean distance between the hand key points and the center point of the eye, obtaining the normalized distance. If the normalized distance between any hand key point (especially the fingertip key point) and the center point of the eye is less than the set distance threshold (e.g., 0.03), it indicates that the driver is in a "hand-eye proximity" state. Then, the video frame (single frame image or a segment containing multiple consecutive frames) corresponding to the driver's "hand-eye proximity" state is marked.

[0059] Step S33: Continuous action confirmation.

[0060] Following the previous example, the system verifies the continuity of the "hand-eye proximity" state through a continuous frame counter. Only when the "hand-eye proximity" state is continuously maintained at or above a set frame rate threshold (for common 30FPS videos, the frame rate threshold is usually set between 6 and 15 frames) is it determined to be a valid eye-rubbing action, thus avoiding momentary interference.

[0061] Step S34, spatial false alarm filtering.

[0062] Following the previous example, the system can use spatial location rules (including the 3D spatial relationship between the face and hands) to eliminate common false alarms (i.e., actions that cannot possibly constitute rubbing the eyes): 1) The hand position is significantly higher than the forehead area (a key hand point was detected that was lower than the forehead reference point).

[0063] The system obtains the lowest point of the upper edge of the eyebrow as the reference point for the forehead (its Y-coordinate value is denoted as...). ), obtain the center point of the hand (its Y-coordinate is marked). ,like Then the state value representing effective eye rubbing will be set to (That is, not to be judged as rubbing the eyes), so as to distinguish between valid rubbing actions and actions that cannot constitute rubbing the eyes based on the status value.

[0064] 2) The hands are extremely close to the tip of the nose and far away from the eyes.

[0065] The system calculates the distance from the hand to the tip of the nose (denoted as...). ) and the distance from the hand to the nearest eye (denoted as ) ),like and Then the state value representing effective eye rubbing will be set to (That is, not to be judged as rubbing the eyes), so as to distinguish between valid rubbing actions and actions that cannot constitute rubbing the eyes based on the status value.

[0066] The system will set the value indicating effective eye rubbing to either of the following two conditions: the hand is positioned significantly higher than the forehead area, or the hand is extremely close to the tip of the nose and far from the eyes. The system only when not triggered Only then is it considered a valid eye-rubbing action.

[0067] Step S35: Sliding window frequency statistics and dry eye feature output.

[0068] Continuing from the previous example, the system can use a sliding window with a half-hour duration (the step size can be set) to count the number of effective eye rubbings within the sliding window in real time, and dynamically output the eye rubbing frequency (i.e., the number of eye rubbings) every half hour as a dry eye feature.

[0069] As one possible implementation, the core basic feature extraction based on the blink detection results, eye rubbing detection results, and squeeze-type blink detection results in step S104 above may include: determining the core basic features from the blink detection results, eye rubbing detection results, and squeeze-type blink detection results, and generating an initial time series corresponding to the core basic features within a preset time period as the core basic feature extraction result. Correspondingly, the construction of the time series based on the core basic feature extraction result in step S104 above may include: performing derived feature extraction based on the initial time series; and constructing a time series based on the initial time series and the derived feature extraction results.

[0070] Continuing from the previous example, the system can periodically record the aforementioned dry eye characteristics in fixed short-period units, forming time series of dry eye characteristics such as the number of complete blinks, the number of squeeze blinks, and the number of eye rubbings (one or more dry eye characteristics can correspond to the same moment). Based on this, highly discriminative derived features (including window statistics features, event sequence features, and physiological simulation features) are constructed using the following methods: (1) Constructing window statistical features: Within the sliding time window, the system calculates the mean, coefficient of variation, linear trend slope, and deviation of basic features from the baseline (obtained by averaging long-term personal historical data) of basic features such as the number of complete blinks, the number of squeeze blinks, and the number of eye rubbings, and obtains window statistical features.

[0071] (2) Constructing event sequence features: The system identifies and quantifies specific behavioral pattern chains (including: incomplete blink burst density, conditional probability and average delay of eye rubbing within a specific time after incomplete blink, and the occurrence rate of squeeze blink after high-frequency blinking) to obtain event sequence features.

[0072] Incomplete blink bursts refer to an abnormally concentrated cluster of incomplete blink events detected by the system over a period of time. This phenomenon usually indicates that the eyes have been subjected to strong stimulation or that the user has entered a state of extreme focus, leading to a loss of blink control. Incomplete blink burst density is an indicator of the degree of clustering, and the specific calculation method is as follows: a1) Define the window: Set a short-period sliding window (e.g., 30 seconds or 1 minute); a2) Slide Count: In this window Inside, count the number of incomplete blinks. ; a3) Density calculation: , In units of time (e.g., seconds); If the density of incomplete blink bursts exceeds a certain percentile of an individual's historical baseline (e.g., more than twice the standard deviation of the individual's historical average incomplete blink burst density, or more than the 95th percentile), then the current moment is considered an incomplete blink burst.

[0073] The calculation of the conditional probability and average delay of eye rubbing occurring within a specific timeframe after an incomplete blink mainly considers the possibility of a series of incomplete blinks occurring before the user rubs their eyes. An example of how to calculate the conditional probability of eye rubbing occurring within a specific timeframe after an incomplete blink is as follows: Define event A as an incomplete blink; define event B as an eye rubbing; statistical analysis is performed over the entire historical data or the current sliding window (e.g., 10 minutes). The formula for calculating the conditional probability is: , This refers to a specific time window (e.g., 5 seconds) after event A occurs. The higher this conditional probability value, the greater the likelihood that incomplete blinking will be followed by eye rubbing. Incomplete blinking followed by eye rubbing is a typical behavioral chain of dry eye syndrome or eye strain.

[0074] The average delay of eye rubbing within a specific timeframe after incomplete blinking reflects the average reaction time from the onset of eye discomfort (manifested as incomplete blinking) to the user's compensatory behavior (eye rubbing). It filters out all instances that meet the above criteria (i.e., after event A occurs). The event pair (i.e., event A and event B) in which event B occurs. The average delay of eye rubbing within a specific time after incomplete blinking can be calculated as follows: for each event pair, record the time difference (e.g., in seconds) from the time of event A to the time of event B, and then take the average of the obtained time differences.

[0075] The occurrence rate of squeeze blinks after high-frequency blinking segments refers to the probability of a squeeze blink occurring within a short period (e.g., within 10 seconds) after the end of a high-frequency blinking segment. It is determined by statistically analyzing the end times of all high-frequency blinking segments and examining the occurrence rate after each end time. The calculation is based on whether at least one squeeze blink occurred within 10 seconds (e.g., 10 seconds). The formula is as follows: .

[0076] (3) Constructing physiological simulation features: Based on the incomplete blinking pattern and interval, a simulated tear film breakage risk index is constructed, and various compensatory behaviors are weighted and accumulated to form a cumulative load of compensatory behaviors.

[0077] A key clinical indicator for assessing dry eye is tear break-up time (TBUT). This time is how long the tear film remains intact after a full blink. A normal tear film break-up time should be greater than 10 seconds (less than 5 seconds suggests dry eye). While it's impossible to directly observe tear film breakage, user behavior can be observed. When the tear film breaks down, the eye feels discomfort, triggering an incomplete blink to attempt repair. Therefore, the length of the incomplete blink interval can inversely infer tear film stability. The simulated tear film breakage risk index is constructed as follows: b1) Input the time series of incomplete blinks; b2) Calculate the average interval between the most recent N incomplete blinks. (Unit: seconds); b3) Set a reference baseline For example, take the average blinking interval of a normal person (about 5 seconds). b4) Risk Index Calculation: ,in Control sensitivity; if Large (rarely incomplete blinking), simulated tear film breakage risk index approaches 0, low risk; if Very small (frequent incomplete blinking) simulates a tear film breakage risk index close to 1, indicating a high risk; when equal to or approximately equal to At that time, the simulated tear film breakage risk index was approximately 0.5, which was considered a moderate risk.

[0078] As one possible implementation, the steps of constructing a time series based on the initial time series and the derived feature extraction results may include: generating an optimal feature set based on the initial time series and the derived feature extraction results, and constructing a time series based on the optimal feature set.

[0079] Continuing from the previous example, to select the feature subset that is most discriminative of dry eye status from the above features (such as basic features, derived features, etc.), the following dual-track verification mechanism can be used: c1) When collecting data, label the data with dry eye and non-dry eye labels, and perform inter-group parameter statistical tests (such as T-tests) on the data labeled with dry eye and the data labeled without dry eye labels, retaining the characteristics that show significant differences in distribution between the two groups; c2) Use a random forest model for pre-training, and quantify the importance of features through SHAP value analysis. Prioritize features that contribute significantly to classification and show clear separation in SHAP value distribution between the two groups (such as dynamic time series pattern features). c3) Finally, the optimal feature subset is determined by combining recursive feature elimination with cross-validation performance. The specific steps include: In the 5-fold hierarchical cross-validation process, the training data is divided into training and validation sets each time (the training data is randomly divided into 5 datasets, and each time one dataset is selected as the test set while the remaining 4 datasets are used as the training set), and recursive feature elimination (RFE) is performed on the training set. The RFE process is as follows: Based on the feature importance ranking of a tree model (such as a random forest model), the least important feature is removed in each iteration, progressively constructing different subsets from those containing one feature to those containing all features, and evaluating the ROC-AUC performance of each subset on the validation set; after completing all folds, the average ROC-AUC for each number of features is calculated, and the number of features corresponding to the highest performance point is selected as the optimal number. Finally, perform a standard RFE again using all the training data until the data is preserved. These features, which are selected as the optimal features, form the optimal feature subset.

[0080] As one possible implementation, step S106 (i.e., predicting the risk score of dry eye disease based on a time series using a pre-trained deep learning model) may include: constructing a time series matrix corresponding to the time series, inputting the time series matrix into a deep learning model, and using the deep learning model to predict the risk score of dry eye disease as the prediction result of the risk score of dry eye disease.

[0081] Continuing from the previous example, we construct and train a deep learning regression model. Its input is a multivariate time series matrix composed of the obtained optimal feature set within a certain sliding time window (representing that each time point in multiple time points corresponds to multiple statistical data). The output of the deep learning regression model is a continuous dry eye risk score.

[0082] The preferred architecture for deep learning regression models is a hybrid model that integrates a Temporal Convolutional Network (TCN) and a Bi-directional Long-Short Term Memory (Bi-LSTM) network. This model has both a TCN module and a Bi-LSTM module. The TCN module is used to efficiently extract local temporal patterns, while the Bi-LSTM module is used to learn long-term contextual dependencies and behavioral sequence logic. An attention mechanism can be introduced to focus on key time steps.

[0083] Deep learning regression model training and validation: Using a video dataset containing health labels, mild dry eye labels, and moderate to severe dry eye labels, the training set, validation set, and test set are strictly divided according to driver ID to prevent data leakage; mean squared error is used as the loss function for model training, validation, testing, and optimization.

[0084] As one possible implementation, step S108 (i.e., determining the risk level of dry eye disease based on the prediction results of the dry eye disease risk score according to the preset risk level determination rules) may include: 1) if several eye disease risk scores are located in the first scoring interval representing low risk, then the dry eye disease risk level is determined to be low risk; 2) if several eye disease risk scores are located in the second scoring interval representing medium risk, then the dry eye disease risk level is determined to be medium risk; 3) if several eye disease risk scores are located in the third scoring interval representing high risk, then the dry eye disease risk level is determined to be high risk.

[0085] As one possible implementation, after step S108 (i.e., determining the risk level of dry eye based on the dry eye risk score prediction result and according to the preset risk level determination rule), an early warning corresponding to the dry eye risk level determination result can also be triggered.

[0086] As one possible implementation, the steps for triggering the warning corresponding to the dry eye risk level determination result may include: if several eye disease risk levels are low, then triggering the display of low-risk information on the in-vehicle human-machine interface; if several eye disease risk levels are medium, then triggering the display of medium-risk information on the in-vehicle human-machine interface; if several eye disease risk levels are high, then triggering the display of high-risk information on the in-vehicle human-machine interface, and simultaneously outputting voice reminder information corresponding to the high-risk information through a voice output device.

[0087] Following the previous example, the system can determine the risk range (e.g., low risk range) of the dry eye risk score predicted by the deep learning regression model. Medium-risk area High-risk areas The corresponding warning will be activated through the in-vehicle human-machine interface: d1) If the dry eye risk score is in the low-risk range, it is judged as low risk, with no prompt or only a status display; d2) If the dry eye risk score is in the medium risk range, it is judged as medium risk and a visual prompt is given (such as the dashboard icon turning yellow). d3) If the dry eye risk score is in the high-risk range, it is considered high-risk. Visual warnings are highlighted, combined with gentle voice reminders, and rest or relief measures are recommended.

[0088] By adopting the above-mentioned operation method of triggering corresponding warnings after identifying the driver's dry eye risk level, timely warnings can be issued based on a relatively accurate quantitative assessment of the driver's dry eye risk, thereby further ensuring driving safety.

[0089] To facilitate understanding, the implementation process of the above-mentioned driver dry eye risk assessment method is described below using a specific application example. All values ​​involved are examples and can be modified according to actual circumstances.

[0090] See Figure 4 As shown, the procedure for assessing the risk of dry eye in drivers is as follows: Step S1: Full blink detection and incomplete blink detection.

[0091] Step S2, squeeze-type blink detection.

[0092] Step S3, eye rubbing detection.

[0093] Step S4, Feature Engineering Construction.

[0094] Step S5: Feature selection based on statistics and models.

[0095] Step S6: Time-series regression prediction of dry eye risk based on deep learning.

[0096] Step S7: Real-time dynamic risk assessment and early warning, and long-term risk prediction.

[0097] The system executes steps S1 to S4 in real time during vehicle operation, updating the dry eye risk prediction value at a frequency of minutes.

[0098] Alternatively, long-term data can be saved, and features can be selected during model design using the operation method in step S5. Suitable features can be used as inputs for the long-term prediction model, and then the long-term dry eye risk can be predicted based on the model designed in step S6.

[0099] To implement the aforementioned method for assessing the risk of dry eye in drivers, the following work was carried out: (a) Detection criteria for dry eye: In medicine, the fluorescein breakup time (FBUT) is usually used to diagnose dry eye by staining the cornea and conjunctiva and by the Schirmer test. Or without surface anesthesia Dry eye can be diagnosed if one of the following subjective symptoms is present: dryness, foreign body sensation, burning sensation, fatigue, discomfort, or fluctuating vision. Or without surface anesthesia Dry eye can be diagnosed when patients experience one of the following subjective symptoms: dryness, foreign body sensation, burning sensation, fatigue, discomfort, or fluctuating vision, along with positive fluorescein staining of the cornea and conjunctiva. Diagnosis of dry eye should include classification and grading to facilitate targeted treatment and assessment of treatment effectiveness.

[0100] (II) Data Collection: Recruit a certain number of dry eye patients and healthy individuals for dry eye diagnosis. Classify the data using labels (e.g., 1 represents dry eye, 0 represents no dry eye; or 5 represents high degree of dry eye, 4 represents moderate degree of dry eye, 3 represents mild degree of dry eye, 2 represents no dry eye requiring intervention, and 1 represents no dry eye requiring no intervention). Under simulated real-world driving conditions, with fixed road, weather, lighting, cabin temperature, cabin humidity, air conditioning fan speed, and driving duration, record facial image data of different groups of people during the driving process.

[0101] (III) Feature extraction: The facial feature data of the subjects is extracted using the algorithm, and the corresponding eye feature data (number of complete blinks, number of incomplete blinks, blink completion rate, frequency of squeeze blinks, frequency of eye rubbing, average duration of a single eye rubbing, eye rubbing intensity index, and the corresponding time series data, etc.) are extracted respectively.

[0102] (iv) Model Training: A dataset integrating frequency data of eye features with corresponding label classifications is used. This dataset is divided into a sample set and a test set according to a certain ratio. Machine learning methods are employed to train the statistical model, resulting in a dry eye recognition model.

[0103] Using the aforementioned method for assessing the risk of dry eye syndrome in drivers, the driver's eye data is captured by an in-vehicle camera. Feature extraction is performed on this data to extract corresponding features (such as the number of complete blinks, the number of incomplete blinks, the degree of blink completion, the number of squeeze blinks, the number of eye rubbing, etc.) to obtain corresponding time-series data. By statistically analyzing the frequency of these features occurring over a certain period of time (e.g., 20 hours of cumulative driving), the risk level of dry eye syndrome in drivers can be directly identified and timely warnings can be issued, thereby ensuring driving safety.

[0104] Based on the above-described method for assessing the risk of dry eye syndrome in drivers, this invention also provides a device for assessing the risk of dry eye syndrome in drivers. (See attached image.) Figure 5 As shown, the device may include the following modules: The detection module 502 is used to locate key points in the facial image data of the driver collected by the driver monitoring system, and to perform blink detection, eye rubbing detection and squeeze blink detection based on the key point location results.

[0105] Module 504 is used to extract core basic features based on blink detection results, eye rubbing detection results, and squeeze blink detection results, and to construct a time series based on the core basic feature extraction results.

[0106] The prediction module 506 is used to predict the risk score of dry eye disease based on the time series using a pre-trained deep learning model.

[0107] The judgment module 508 is used to determine the risk level of dry eye disease based on the prediction results of the dry eye disease risk score and according to the preset risk level judgment rules.

[0108] Using the aforementioned driver dry eye risk assessment device, facial image data of the driver can be collected through the driver monitoring system. The collected facial image data is then used for key point localization, blink detection, eye rubbing detection, squeeze blink detection, and extraction of core basic features. A time series is constructed, and the driver's dry eye risk level is identified by using a deep learning model and preset risk level judgment rules. This can accurately quantify and assess the driver's risk of developing dry eye, thereby helping to ensure driving safety.

[0109] The driver dry eye risk assessment device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned driver dry eye risk assessment method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0110] This invention also provides an electronic device, such as... Figure 6 The diagram shows the structure of the electronic device, which includes a processor 61 and a memory 60. The memory 60 stores computer-executable instructions that can be executed by the processor 61. The processor 61 executes the computer-executable instructions to implement the aforementioned driver dry eye risk assessment method.

[0111] exist Figure 6 In the illustrated embodiment, the electronic device further includes a bus 62 and a communication interface 63, wherein the processor 61, the communication interface 63, and the memory 60 are connected via the bus 62.

[0112] The memory 60 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 63 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 62 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 62 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0113] Processor 61 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the aforementioned driver dry eye risk assessment method can be completed through the integrated logic circuitry in the hardware of processor 61 or through software instructions. Processor 61 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the driver dry eye risk assessment method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory. The processor 61 reads the information in the memory and, in conjunction with its hardware, completes the steps of the driver dry eye risk assessment method of the aforementioned embodiment.

[0114] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0115] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0117] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for assessing the risk of dry eye syndrome in drivers, characterized in that, include: Key point localization is performed on the facial image data of the driver collected by the driver monitoring system, and blink detection, eye rubbing detection and squeeze blink detection are performed based on the key point localization results; Core basic features are extracted based on blink detection results, eye rubbing detection results, and squeeze blink detection results, and a time series is constructed based on the core basic feature extraction results; Based on the time series, a dry eye risk score is predicted using a pre-trained deep learning model. Based on the dry eye risk score prediction results, the risk level of dry eye is determined according to the preset risk level determination rules.

2. The method for assessing the risk of dry eye syndrome in drivers according to claim 1, characterized in that, The key point localization results include eye key point localization information; Based on the key point localization results, blink detection, eye rubbing detection, and squeeze blink detection are performed respectively, including: Based on the eye key point positioning information, the eye aspect ratio EAR is calculated to obtain the driver's real-time EAR time-series data; Blink waveform detection is performed on the real-time EAR time-series data, and the blink detection result is determined based on the blink waveform detection result and the first threshold information.

3. The method for assessing the risk of dry eye syndrome in drivers according to claim 2, characterized in that, The blink detection results include the number of complete blinks, the number of incomplete blinks, and the blink completion rate. The first threshold information includes a first EAR threshold, a second EAR threshold, and a micro-tremor threshold. The first EAR threshold is less than the second EAR threshold. The blink detection result is determined based on the blink waveform detection result and the first threshold information, including: If the blink waveform detection result contains the detected complete blink waveform, then extract the minimum EAR value from each complete blink waveform; The number of complete blink waveforms whose minimum EAR value is not greater than the first EAR threshold and whose duration is greater than the micro-flicker threshold is counted as the number of complete blinks; The number of complete blink waveforms whose duration is greater than the micro-flicker threshold and whose minimum EAR value is greater than the first threshold and not greater than the second EAR threshold is counted as the number of incomplete blinks; The blink completion rate is determined based on the number of complete blinks and the number of incomplete blinks.

4. The method for assessing the risk of dry eye syndrome in drivers according to claim 3, characterized in that, Based on the key point localization results, blink detection, eye rubbing detection, and squeeze blink detection are performed respectively, and also include: Face detection is performed on the facial image data, and an image of the region of interest (ROI) of the eye is obtained based on the face detection results and the eye key point localization information; wherein, the face detection results include a face detection box, and the image of the ROI of the eye contains complete information on the periorbital muscle groups and has time point information; Based on the obtained region of interest (ROI) images, a pre-trained classification model is used to predict the squeezed eye-closing state. The prediction result includes the classification of whether each ROI image belongs to the squeezed eye-closing state. Based on the predicted results of the squeeze-type eye-closing state, the squeeze-type eye-closing state is detected, and based on the squeeze-type eye-closing state detection results and the second threshold information, the squeeze-type blink detection results are determined.

5. The method for assessing the risk of dry eye syndrome in drivers according to claim 4, characterized in that, The results of the squeeze-type eye-closing state detection include each detected squeeze-type eye-closing state and its duration. The detection of the squeeze-type eye-closing state based on the prediction results includes: if there is a first category belonging to the squeeze-type eye-closing state, then record the time point information of the target eye region of interest image corresponding to each first category; based on the time point information of all target eye region of interest images, determine each squeeze-type eye-closing state and its duration. The squeeze-type blink detection result includes the number of squeeze-type blinks, and the second threshold information includes a squeeze-type blink time threshold; determining the squeeze-type blink detection result based on the squeeze-type closed state detection result and the second threshold information includes: counting the number of squeeze-type closed states with a duration greater than the squeeze-type blink time threshold as the number of squeeze-type blinks.

6. The method for assessing the risk of dry eye syndrome in drivers according to claim 3, characterized in that, The eye-rubbing detection results include the number of eye-rubbings, and the key point localization results also include key point localization information for the hand, forehead, and nose tip. Based on the key point localization results, blink detection, eye rubbing detection, and squeeze blink detection are performed respectively, and also include: Based on the eye key point positioning information and the hand key point positioning information, real-time potential eye rubbing time-series data representing the driver's hand-eye proximity state is generated; If the real-time potential eye-rubbing time-series data includes a second candidate interval whose duration of hand-eye proximity is not less than a preset duration threshold, then the number of second candidate intervals that meet the second condition is determined based on the forehead key point positioning information and the nose tip key point positioning information as the number of eye-rubbing times; wherein, meeting the second condition indicates that the driver's hand is not significantly higher than the forehead and the hand is not close to the nose tip.

7. The method for assessing the risk of dry eye syndrome in drivers according to claim 1, characterized in that, Based on the blink detection results, eye rubbing detection results, and squeeze blink detection results, core basic features are extracted, including: determining core basic features from the blink detection results, eye rubbing detection results, and squeeze blink detection results, and generating an initial time series corresponding to the core basic features within a preset time period as the core basic feature extraction result; Constructing a time series based on the core basic feature extraction results includes: extracting derived features based on the initial time series; and constructing the time series based on the initial time series and the derived feature extraction results.

8. The method for assessing the risk of dry eye syndrome in drivers according to claim 7, characterized in that, Based on the time series, a dry eye risk score is predicted using a pre-trained deep learning model, including: constructing a time series matrix corresponding to the time series, inputting the time series matrix into the deep learning model, and predicting the dry eye risk score using the deep learning model as the dry eye risk score prediction result. Based on the dry eye risk score prediction results, the dry eye risk level is determined according to the preset risk level determination rules, including: if the dry eye risk score is in the first scoring interval representing low risk, the dry eye risk level is determined to be low risk; if the dry eye risk score is in the second scoring interval representing medium risk, the dry eye risk level is determined to be medium risk; if the dry eye risk score is in the third scoring interval representing high risk, the dry eye risk level is determined to be high risk. After determining the risk level of dry eye based on the dry eye risk score prediction results and according to the preset risk level determination rules, the system also includes: triggering an early warning corresponding to the dry eye risk level determination result.

9. A driver dry eye risk assessment device, characterized in that, include: The detection module is used to locate key points in the facial image data of the driver collected by the driver monitoring system, and to perform blink detection, eye rubbing detection and squeeze blink detection based on the key point location results. The module is used to extract core basic features based on blink detection results, eye rubbing detection results, and squeeze blink detection results, and to construct time series based on the core basic feature extraction results; The prediction module is used to predict the risk score of dry eye disease based on the time series using a pre-trained deep learning model. The judgment module is used to determine the risk level of dry eye disease based on the prediction results of the dry eye disease risk score and according to the preset risk level judgment rules.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the driver dry eye risk assessment method according to any one of claims 1 to 8.

Images