State determination method, device, apparatus and storage medium

Abstract

The application relates to the technical field of driving state recognition, and discloses a state discrimination method, device, equipment and storage medium, which comprises the following steps: determining eyelid opening and closing degrees according to eye images of a user; in the case that the eyelid opening and closing degrees are detected to be less than a preset opening degree threshold, acquiring eye movement parameters of the user within interval frame numbers, the eye movement parameters comprising eyelid closing speed, eyeball downward displacement and eyelid-eyeball movement correlation parameters; and discriminating the state of the user according to the eye movement parameters. When the eyelid opening and closing degrees are detected to be less than a preset threshold, the eye behavior mode of the user is comprehensively described from three dimensions of movement speed, displacement amplitude and eyelid-eyeball correlation based on dynamic movement characteristics, so that two different physiological behaviors of natural eyelid drooping caused by looking down and slow eyelid closing caused by fatigue can be effectively distinguished from the mechanism, and the false positives of looking down and fatigue closing eyes are reduced.

Description

Technical Field

[0001] This application relates to the field of driving state discrimination technology, and in particular to a state discrimination method, device, equipment and storage medium. Background Technology

[0002] Currently, Driver Monitoring (DMS) is widely used in the field of intelligent driving. However, in complex scenarios such as when the driver looks down or closes their eyes due to fatigue, the eyelids are in different drooping or closed states. Existing static geometric feature discrimination methods for single-frame images have a large overlap in these two states, which makes it easy to make misjudgments or omissions. It is impossible to accurately distinguish between the natural drooping of the eyelids caused by looking down and the slow closing of the eyelids due to fatigue. Summary of the Invention

[0003] The main objective of this application is to provide a state discrimination method, apparatus, device, and storage medium, aiming to solve the technical problem that existing static geometric feature discrimination methods cannot accurately distinguish between two behaviors: natural eyelid drooping caused by looking down and slow eyelid closure caused by fatigue.

[0004] To achieve the above objectives, this application proposes a state discrimination method, which includes: Determine eyelid opening based on the user's eye image; When the eyelid opening degree is detected to be less than a preset opening degree threshold, the user's eye movement parameters within the interval frame number are obtained. The eye movement parameters include eyelid closing speed, eyeball downward displacement, and eyelid-eyeball movement correlation parameters. The eyelid-eyeball movement correlation parameters characterize the correlation between changes in eyelid opening degree and eyeball downward displacement. The user's state is determined based on the eye movement parameters.

[0005] Optionally, obtaining the user's eye movement parameters within a frame interval includes: The eyelid closing speed is determined based on the first change in the eyelid opening degree within the specified frame interval; The eye tracking point in the eye image is tracked within the specified interval frame number, and the downward displacement of the eyeball is determined based on the second change of the eye tracking point within the specified interval frame number. Based on the eyelid opening and closing degree and the downward displacement of the eyeball, eyelid-eyeball motion-related parameters are determined.

[0006] Optionally, determining the eyelid-eye movement correlation parameters based on the eyelid opening and closing degree and the downward displacement of the eyeball includes: Determine the covariance between the eyelid opening angle and the downward displacement of the eyeball; Based on the covariance, the first variance of the eyelid opening and closing degree, and the second variance of the downward displacement of the eyeball, the eyelid-eyeball motion correlation parameters are determined.

[0007] Optionally, determining the eyelid opening degree based on the user's eye image includes: Key point detection is performed on the eye image to obtain key points of the upper eyelid and the lower eyelid; The eyelid opening degree is determined based on the positional relationship between the key points of the upper eyelid and the key points of the lower eyelid in the same frame.

[0008] Optionally, before determining the eyelid opening degree based on the user's eye image, the method further includes: Obtain the range of the first eyelid movement velocity and the range of the first eyeball relative displacement when the user is looking down; Obtain the range of second eyelid movement velocity and the range of relative displacement of the second eyeball when the user is in a fatigued state; The first closing speed and the second closing speed are determined based on the user's blinking speed, the minimum speed of the first eyelid movement speed range, and the maximum speed of the second eyelid movement speed range. The first downward rotation displacement and the second downward rotation displacement are determined based on the distance between the user's eyeball center and the lower eyelid, the minimum displacement of the first eyeball relative displacement range, and the maximum displacement of the second eyeball relative displacement range.

[0009] Optionally, determining the user's state based on the eye movement parameters includes: If the downward displacement of the eyeball is greater than the first downward displacement, determine whether the eyelid closing speed is greater than the first closing speed, and whether the eyelid-eyeball motion correlation parameter is greater than the first correlation threshold. If the eyelid closing speed is greater than the first closing speed, and the eyelid-eye movement correlation parameter is greater than the first correlation threshold, then the user's state is determined to be looking down.

[0010] Optionally, determining the user's state based on the eye movement parameters includes: If the downward displacement of the eyeball is less than the second downward displacement, determine whether the eyelid closing speed is less than the second closing speed, and whether the eyelid-eyeball motion correlation parameter is less than the second correlation threshold. If the eyelid closure speed is less than the second closure speed, and the eyelid-eye movement correlation parameter is less than the second correlation threshold, then the user's state is determined to be fatigued.

[0011] Furthermore, to achieve the above objectives, this application also proposes a state discrimination device, which includes: The eyelid opening / closing determination module is used to determine the degree of eyelid opening / closing based on the user's eye image; The parameter acquisition module is used to acquire the user's eye movement parameters within an interval of frames when the eyelid opening degree is detected to be less than a preset opening degree threshold. The eye movement parameters include eyelid closing speed, eyeball downward displacement, and eyelid-eyeball movement correlation parameters. The eyelid-eyeball movement correlation parameters characterize the correlation between changes in eyelid opening degree and eyeball downward displacement. The parameter discrimination module is used to discriminate the user's state based on the eye movement parameters.

[0012] In addition, to achieve the above objectives, this application also proposes a state discrimination device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the state discrimination method as described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the state discrimination method described above.

[0014] One or more technical solutions proposed in this application have at least the following technical effects: When the eyelid opening and closing degree is detected to be less than the preset threshold, the dynamic motion characteristics in the time dimension are obtained. The user's eye behavior pattern is comprehensively characterized from three dimensions: motion speed, displacement amplitude and eyelid-eyeball correlation. This effectively distinguishes between two different physiological behaviors: natural eyelid drooping caused by looking down and slow eyelid closing caused by fatigue, thereby reducing false alarms of looking down and eyelid closing due to fatigue. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the first embodiment of the status determination method of this application; Figure 2 This is a flowchart illustrating the second embodiment of the status determination method of this application; Figure 3 This is a flowchart illustrating the third embodiment of the status determination method of this application; Figure 4 This is a schematic diagram of the module structure of the state discrimination device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the state discrimination method in the embodiments of this application.

[0018] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0020] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0021] It should be noted that the executing entity in this embodiment can be the vehicle itself, or a status identification device installed in the vehicle that can identify the driver's status. The status identification device can be a monitor installed in the vehicle, such as a DMS, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment and the following embodiments, the leakage diagnosis method of this embodiment is described using a status identification device as an example.

[0022] Based on this, embodiments of this application provide a state discrimination method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the status determination method of this application.

[0023] In this embodiment, the state discrimination method includes: Step S10: Determine the eyelid opening degree based on the user's eye image.

[0024] In this embodiment, eyelid opening degree can characterize the degree to which a user's eyes are open. The eyelid opening degree is lower when the user is fatigued compared to a normal state. When obtaining eyelid opening degree, the coordinates of key points related to the eyelids can be extracted based on the facial key point detection results in the eye image, and the eyelid opening degree can be calculated based on the positional relationship between these key points.

[0025] Specifically, after DMS accesses video stream data, it can use detection boxes to detect the user's head in the video stream and initially confirm the head position. Based on the determined head region, it further detects the positions of facial key points such as eyes, nose, mouth, and jawline, and tracks these key points. Subsequently, it confirms the driver's head posture based on the above key points, and extracts the region of interest for the eyes from the facial image based on the head posture. Within the region of interest for the eyes, it identifies key points for the upper eyelid, lower eyelid, and pupil center.

[0026] It should be understood that, in order to ensure the stability and anti-shake capability of key points in consecutive video frames, DMS can use Kalman filtering to perform parallel and stable tracking of the above key points after it is enabled, thereby eliminating the position jump of key points caused by slight head shaking or image noise.

[0027] It should be noted that the calculation process of eyelid opening and closing degree can be executed locally on the vehicle embedded processor, or on a cloud server or a domain controller with computing capabilities. The specific implementation method can be selected according to the vehicle's computing resource allocation, and this embodiment does not limit it.

[0028] In the specific implementation process, key point detection can be performed on the eye image to obtain key points of the upper and lower eyelids, and the eyelid opening degree can be determined based on the positional relationship between the two in the same frame. At this time, only a small number of key points need to be extracted to complete the opening degree calculation, which has low computational overhead and high real-time performance, and is suitable for resource-constrained in-vehicle embedded systems. Based on this, step S10 may include: Key point detection is performed on the eye image to obtain key points of the upper eyelid and the lower eyelid; The eyelid opening degree is determined based on the positional relationship between the key points of the upper eyelid and the key points of the lower eyelid in the same frame.

[0029] It is understandable that when obtaining eyelid opening and closing, key point detection can be performed on the eye image to obtain the key points of the upper and lower eyelids, and the eyelid opening and closing can be determined based on the positional relationship between the key points of the upper and lower eyelids in the same frame.

[0030] Specifically, in an eye image, the upper eyelid keypoint is located at the upper edge of the eyelid region, and the lower eyelid keypoint is located at the lower edge of the eyelid region. The vertical distance between the two can reflect the degree of eye opening in the current frame.

[0031] In one example, the upper eyelid height corresponding to the upper eyelid keypoint at time t is marked as follows: The lower eyelid height corresponding to the key points of the lower eyelid is marked as follows: Eyelid opening and closing degree (i.e., the instantaneous opening degree of the eyelid at time t). It is possible It was calculated that when the eyes are fully open, The value is relatively large; when the eyes are closed, The value approaches zero.

[0032] Optionally, to improve the robustness of single-frame detection, the average position of multiple keypoints can be used to calculate the upper and lower eyelid keypoints. For example, multiple keypoints (such as 5 keypoints evenly distributed along the eyelid edge) can be extracted in the upper eyelid region, and the average of their ordinates can be used as the mean. Multiple key points were extracted in the lower eyelid region, and the average of their ordinates was taken as the mean. This reduces the impact of local noise or individual key point detection errors on the opening degree calculation results.

[0033] Step S20: When the eyelid opening degree is detected to be less than a preset opening degree threshold, the user's eye movement parameters within the interval frame number are obtained. The eye movement parameters include eyelid closing speed, eyeball downward displacement, and eyelid-eyeball movement correlation parameters. The eyelid-eyeball movement correlation parameters characterize the correlation between changes in eyelid opening degree and eyeball downward displacement.

[0034] It should be noted that the preset eyelid opening threshold is used to determine whether the user's eyes are in a closed or nearly closed state. When the calculated eyelid opening degree is less than the preset opening threshold, it means that the user's eyes have entered a closed or semi-closed state. At this time, it is necessary to further obtain the user's eye movement parameters within the interval frame to distinguish the specific type of the closing behavior through dynamic features, such as eyelid closure caused by looking down, or active and slow eyelid closure caused by fatigue.

[0035] It should be understood that the interval frame count is used to define the time window range selected when calculating dynamic features, and can be set according to the frame rate of the video stream and the required discrimination sensitivity. For example, the interval frame count can be within the time window. Reduce the number of data frames n to a preset opening threshold, where n is an integer greater than or equal to 1.

[0036] Understandably, eye movement parameters can include eyelid closure velocity, downward eye movement displacement, and eyelid-eye movement correlation parameters. Eyelid closure velocity characterizes the speed of eyelid closure within a frame interval; a higher velocity indicates faster closure, while a lower velocity indicates slower closure. Downward eye movement displacement characterizes the downward rotation of the eye within a frame interval. Eyelid-eye movement correlation parameters characterize the correlation between changes in eyelid opening / closing and downward eye movement displacement; a higher value indicates stronger synchronization between eyelid and eye movements, while a lower value indicates weaker synchronization.

[0037] Understandably, with DMS activated, the eyelid opening during normal driving can be calculated. At this point, the preset opening threshold can be based on the eyelid opening. Make settings, such as eyelid opening / closing. For 2 seconds less than the threshold At this point, it is determined whether the user's eyes are in a closed or semi-closed state, and further state assessment is required; this involves assessing the degree of eyelid opening and closing. Greater than the threshold If the system determines that the user is driving normally, it will not issue an alarm or take any further action.

[0038] Step S30: Determine the user's state based on the eye movement parameters.

[0039] In this embodiment, eye movement parameters include eyelid closure speed, downward eye movement displacement, and eyelid-eye movement correlation parameters. These three parameters characterize the user's dynamic behavior pattern during eyelid closure from three dimensions: eyelid movement speed, downward eye movement amplitude, and eyelid-eye movement coordination degree. Based on the comprehensive analysis of these three dynamic parameters, the category to which the user's current behavior belongs can be distinguished and determined. Specifically, the eyelid closure speed, downward eye movement displacement, and eyelid-eye movement correlation parameters are compared with their respective preset thresholds, and the user's current state category is determined based on the combination of comparison results.

[0040] In one feasible implementation, each eye movement parameter corresponds to a specific threshold, including a threshold set for determining a downward-looking state and a threshold set for determining a fatigue state. There is a numerical relationship between the thresholds in the downward-looking state threshold set and the thresholds in the fatigue state threshold set. For example, the eyelid closure velocity threshold, eyeball downward displacement threshold, and correlation threshold corresponding to the downward-looking state are all greater than the corresponding thresholds for the fatigue state. By comparing the actually calculated parameters with the two sets of thresholds, the current state is determined to be downward-looking when all parameters simultaneously meet the threshold conditions corresponding to the downward-looking state; the current state is determined to be fatigued when all parameters simultaneously meet the threshold conditions corresponding to the fatigue state. If none of the parameter combinations meet the threshold conditions corresponding to either state, the user can be determined to be in another state, such as normal driving.

[0041] In this embodiment, when the eyelid opening and closing degree is detected to be less than a preset threshold, dynamic motion features in the time dimension are obtained, and the user's eye behavior pattern is comprehensively characterized from three dimensions: motion speed, displacement amplitude, and eyelid-eyeball correlation. This effectively distinguishes two different physiological behaviors from the mechanism: natural eyelid drooping caused by looking down and slow eyelid closing caused by fatigue, thereby reducing false alarms of looking down and fatigue-induced eye closure.

[0042] Reference Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the state determination method of this application. Based on the first embodiment described above, a second embodiment of the state determination method of this application is proposed.

[0043] In the second embodiment, step S20 includes: Step S201: Determine the eyelid closing speed based on the first change in the eyelid opening degree within the specified number of frames.

[0044] It is understandable that when determining the eyelid closure speed, the first change in the eyelid opening and closing degree within the interval frame can be calculated based on the eyelid opening and closing degree, and the eyelid closure speed can be determined according to the ratio of the first change to the corresponding time span.

[0045] Specifically, let the current frame be... The interval frame number is ( (Any integer greater than or equal to 1), frame interval duration is The eyelid opening degree of the current frame is ,interval The next frame (i.e., the first frame) The eyelid opening degree of (frame) is Then the first change in eyelid opening and closing within the interval frame number is At this point, the eyelid closing speed can be . The higher the value, the faster the eyelids close. The smaller the value, the slower the eyelids close.

[0046] Step S202: Track the eye tracking point in the eye image within the specified interval frame number, and determine the downward displacement of the eyeball based on the second change amount of the eye tracking point within the specified interval frame number.

[0047] Understandably, eye-tracking points are feature points in an eye image that represent the spatial position of the eyeball. Changes in the position of eye-tracking points can reflect the direction and amplitude of eyeball rotation. Eye-tracking points can be the center of the pupil, the center of the iris, the geometric center of the eyeball contour, etc.

[0048] Understandably, when determining the downward eye movement, an eye-tracking point can be identified in the eye image and continuously tracked within a certain frame interval. Then, a second change in this tracking point within the frame interval is calculated, and this change is used as the downward eye movement. The eye-tracking point can be the center of the pupil, the center of the iris, the geometric center of the eyeball contour, or other significant feature points within the eyeball region.

[0049] In one example, the eye-tracking point is the center of the pupil, obtained through pupil center recognition. Within the region of interest (ROI) of the eye, the center of the pupil region can be located using image processing algorithms, and the pixel coordinates of this center position are used as the coordinates of the eye-tracking point. Let the current frame be t, and the eye height of the eye-tracking point be... The frame after an interval of n frames (i.e., the first frame) The eye height of the eye tracking point in the frame is Then the second change of the eye-tracking point within the interval frame number. This value represents the downward displacement of the eyeball. The larger the value, the greater the downward movement of the eyeball within the interval of frames; The smaller the value, the smaller the downward movement of the eyeball, or the eyeball remains basically still.

[0050] Step S203: Determine eyelid and eyeball movement correlation parameters based on the eyelid opening and closing degree and the downward displacement of the eyeball.

[0051] It should be understood that when a user looks down, their eyes need to rotate downwards to shift their gaze from the road surface to a target below (such as the dashboard or central control screen). At this time, the upper eyelid passively follows the downward rotation of the eyeball, resulting in a passive closing. The decrease in eyelid opening and closing is highly synchronized with the downward displacement of the eyeball; that is, the greater the downward rotation of the eyeball, the greater the degree of eyelid closure. Therefore, there is a strong correlation between eyelid movement and eyeball movement, with a relatively large correlation parameter. The eyelid closure is primarily a passive following movement driven by the downward rotation of the eyeball.

[0052] It should be understood that when a user closes their eyes due to fatigue, the closure of the eyelids is mainly caused by the active relaxation of the levator palpebrae superioris muscle, while the eyeball is often in a neutral or slightly upward position. At this time, there is no obvious synergistic relationship between the change in the degree of eyelid opening and closing and the downward displacement of the eyeball, and the correlation parameters of eyelid and eyeball movement are small.

[0053] In the specific implementation process, the statistical principles of covariance and variance can be used to quantify the intrinsic coupling degree between eyelid opening and closing and eyeball movement, accurately reproducing the true physiological laws of human organ linkage, and providing a basis for state judgment. At this time, step S203 may include: Determine the covariance between the eyelid opening angle and the downward displacement of the eyeball; Based on the covariance, the first variance of the eyelid opening and closing degree, and the second variance of the downward displacement of the eyeball, the eyelid-eyeball motion correlation parameters are determined.

[0054] Specifically, let the current frame be t, and the interval number of frames be n (n is an integer greater than or equal to 1). From the current frame to the nth frame... Within the time window of a frame, the eyelid opening degree is The eyeball downward displacement sequence is First, calculate the sequence of changes in eyelid opening and closing. With eyeball downward displacement sequence covariance between The covariance is used to characterize the degree of coordinated change between the two sequences. Next, the first variance of the change in instantaneous eyelid opening is calculated for each sequence. and the second variance of the downward displacement of the eyeball Then, the eyelid-eye movement correlation parameters are determined based on the ratio of the covariance to the product of the first and second variances. ,Right now .

[0055] In this embodiment, the change in parameters within a frame interval is used to characterize motion features, which can objectively reflect the dynamic changes of the eyelids and eyeballs in the time dimension, providing reliable data support for accurately distinguishing between the downward looking state and the fatigue state.

[0056] Reference Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the state determination method of this application. Based on the first embodiment described above, a third embodiment of the state determination method of this application is proposed.

[0057] In the third embodiment, before step S10, the method further includes: Step S01: Obtain the range of the first eyelid movement speed and the range of the first eyeball relative displacement when the user is looking down.

[0058] Step S02: Obtain the range of second eyelid movement speed and the range of relative displacement of the second eyeball when the user is in a fatigued state.

[0059] It should be noted that the ranges of the first eyelid movement velocity, the first eyeball relative displacement, the second eyelid movement velocity, and the second eyeball relative displacement can be used to determine the various thresholds required in the discrimination process. These ranges can be obtained by calibrating an offline database when the vehicle is first started, or by collecting user data online and performing personalized calibration when the user first uses the DMS function, or by continuously collecting data and dynamically updating it during user use.

[0060] Understandably, the "looking down" state refers to the behavior of a driver shifting their gaze from the road ahead to a target below (such as the instrument panel, center console screen, or gear shift area). In this state, the eyes turn downwards to align with the target, and the upper eyelids passively follow the downward movement of the eyes, closing accordingly. Therefore, the eyelid movement speed corresponding to the "looking down" state is usually relatively fast (because the downward movement of the eyes is inherently fast, and the eyelids, as passive followers, also close relatively quickly), and the relative displacement of the eyeballs is relatively large (the center of the eyeball shifts significantly downwards relative to its position during normal driving). By collecting data on the eyelid closure speed and eyeball downward displacement when a user performs the "looking down" action multiple times, and performing statistical analysis (such as calculating the mean and standard deviation), the range of the first eyelid movement speed and the range of the first relative eyeball displacement during the "looking down" state can be determined.

[0061] It should be understood that fatigue is a behavioral state in which a user's eyelids voluntarily and slowly close due to drowsiness while driving. In this state, eyelid closure is caused by the voluntary relaxation of the levator palpebrae superioris muscle, and the movement speed is significantly slower than the voluntary blinking or downward gaze, and the eyeballs do not rotate downwards synchronously or the downward rotation is minimal. Therefore, the eyelid movement speed corresponding to fatigue is usually slow (slow eyelid closure), and the relative displacement of the eyeballs is small (the eyeballs basically remain in place or show slight upward rotation). By collecting data on the eyelid closure speed and downward displacement of the eyeballs when the user is fatigued (such as when gradually becoming drowsy in a driving simulator, or simulating fatigue-induced eye closure under laboratory conditions), and performing statistical analysis, the range of the second eyelid movement speed and the range of the second relative displacement of the eyeballs in the user's fatigue state can be determined.

[0062] In one example, parameter calibration can be performed using a large-scale sample database before the vehicle leaves the factory or when the DMS system is first activated. Specifically, a large number of spontaneous blinking samples from different users under normal driving conditions are collected, the blinking velocity of each sample frame is calculated, and the average value is recorded as the average blinking velocity. Simultaneously, a large number of eyelid closure velocity samples were collected from users looking downwards. The velocity distribution range within the 95% confidence interval was taken as the first eyelid movement velocity range, denoted as... ,in This is the minimum value within that range. The maximum value within this range; a large number of eyelid closure velocity samples were collected from users under fatigue conditions, and the velocity distribution range within the 95% confidence interval was taken as the second eyelid movement velocity range, denoted as . ,in This is the minimum value within that range. This is the maximum value within that range.

[0063] In another embodiment, a large number of eyeball downward displacement samples of users in a downward-looking state can be collected, and the displacement distribution range within the 95% confidence interval can be taken as the first relative eyeball displacement range, denoted as . ,in This is the minimum value within that range. The maximum value within this range; a large number of user eyeball downward displacement samples under fatigue conditions were collected, and the displacement distribution range within the 95% confidence interval was taken as the second relative eyeball displacement range, denoted as . ,in This is the minimum value within that range. This represents the maximum value within that range. Additionally, the baseline distance D between the center of the eyeball and the lower eyelid can be obtained under normal driving conditions. This distance can be calculated from the results of key eye point detection under normal eye-opening conditions.

[0064] It should be noted that the above-mentioned database statistical calibration and threshold calculation process can be completed offline on the cloud server and the calibration results can be pre-loaded into the DMS system, or it can be executed online in the vehicle-mounted embedded processor. The specific implementation method can be selected according to the computing resources and data sources, and this embodiment does not limit it.

[0065] Step S03: Determine the first closing speed and the second closing speed based on the user's blinking speed, the minimum speed of the first eyelid movement speed range, and the maximum speed of the second eyelid movement speed range.

[0066] Understandably, the first closing speed and the second closing speed can be used as speed thresholds to distinguish between the downward-looking state and the fatigue state during the discrimination process. Among them, the first closing speed is the lower limit of the speed required to determine the downward-looking state, and its value should be relatively high to match the physiological characteristics of rapid eyelid closure; the second closing speed is the upper limit of the speed required to determine the fatigue state, and its value should be relatively low to match the physiological characteristics of slow eyelid closure.

[0067] Specifically, if the user's blinking speed is V (the baseline average blinking speed is V), then... The minimum speed within the range of first eyelid movement speed is The maximum speed of the second eyelid movement is e. First closing speed. The minimum speed can be determined based on the user's blinking speed V and the range of first eyelid movement speed. The ratio relationship is determined. In one embodiment, the first closing velocity can be set to... Second closing velocity It can be based on the user's blinking speed V and the maximum speed range of the second eyelid movement. The ratio relationship is determined. In one embodiment, the second closing speed can be set to... .

[0068] Step S04: Determine the first downward rotation displacement and the second downward rotation displacement based on the distance between the user's eyeball center and the lower eyelid, the minimum displacement of the first eyeball relative displacement range, and the maximum displacement of the second eyeball relative displacement range.

[0069] In this embodiment, the first downward displacement and the second downward displacement can be used as displacement thresholds to distinguish between the downward-looking state and the fatigue state during the discrimination process.

[0070] Specifically, let D be the reference distance between the user's eyeball center and the lower eyelid, and let the minimum displacement of the first eyeball relative displacement range be... The maximum displacement of the second eyeball relative displacement range is First downward displacement Based on the reference distance Minimum displacement relative to the first eyeball The product relationship is determined. In one implementation, the first downward displacement... Can be set to Second downward displacement The maximum displacement can be determined based on the reference distance D and the relative displacement range of the second eyeball. The product relationship is determined. In one implementation, the second downward displacement... Can be set to Because the relative displacement range of the eyeballs is generally greater when looking downwards than when under fatigue, therefore... This enables the ability to separate and distinguish between the two types of states in the displacement dimension.

[0071] In practical implementation, a downward-looking state is determined only when the characteristics of a large downward eye movement, a fast eye closure speed, and strong eyelid-eye synchronization are simultaneously achieved. This eliminates false displacement interference caused by the driver's brief squinting, eye rubbing, or facial expression changes, reducing the false trigger rate of the downward-looking distraction alarm and minimizing unnecessary disturbance to normal driving. At this time, step S30 may include: If the downward displacement of the eyeball is greater than the first downward displacement, determine whether the eyelid closing speed is greater than the first closing speed, and whether the eyelid-eyeball motion correlation parameter is greater than the first correlation threshold. If the eyelid closing speed is greater than the first closing speed, and the eyelid-eye movement correlation parameter is greater than the first correlation threshold, then the user's state is determined to be looking down.

[0072] Specifically, in the process of determining the downward-looking state, the eyeballs are rotated downwards. With the determined first downward displacement Compare. If This indicates that the downward rotation of the eyeball within the frame interval has reached or exceeded the displacement judgment benchmark, thus meeting the criteria for suspected downward looking. Based on this, the eyelid closure speed is further determined. Is it greater than the first closing velocity? and eyelid-eye movement related parameters Is it greater than the first correlation threshold? .like and If all three conditions are met simultaneously, the user is determined to be in a downward-looking state, and an alarm is triggered to remind the user after the required duration of the downward-looking state has elapsed.

[0073] It should be understood that the value of the first correlation threshold S1 can be determined by statistical analysis based on the sample data in the downward view state. In this embodiment, the first correlation threshold S1 is preferably 0.8.

[0074] In practical implementation, fatigue can be identified when the downward eye movement is small, the closing speed is slow, and the eyelid-eye synchronization is weak. This allows for earlier detection of slow eye-closing behavior caused by drowsiness in drivers, distinguishing fatigue-induced half-closing from normal blinking or deliberate head-down movements in healthy individuals, thus reducing false alarm rates. At this point, step S30 may include: If the downward displacement of the eyeball is less than the second downward displacement, determine whether the eyelid closing speed is less than the second closing speed, and whether the eyelid-eyeball motion correlation parameter is less than the second correlation threshold. If the eyelid closure speed is less than the second closure speed, and the eyelid-eye movement correlation parameter is less than the second correlation threshold, then the user's state is determined to be fatigued.

[0075] Specifically, in the process of identifying fatigue, the downward displacement of the eyeballs can be considered. With the determined second downward displacement Compare. If This indicates that the downward rotation of the eyeball within the frame interval is relatively small, meeting the criteria for suspected fatigue-induced eye closure. Based on this, the eyelid closure speed is further assessed. Is it less than the second closing velocity? and eyelid-eye movement related parameters Is it less than the second correlation threshold? .like and If all three conditions are met simultaneously, the user is determined to be in a state of fatigue. Once the fatigue monitoring duration meets the requirements, the system will issue an alarm to remind the user.

[0076] It should be understood that the second correlation threshold The value of can be determined based on sample data under fatigue conditions through statistical analysis. In this embodiment, the first correlation threshold is... The preferred value is 0.2.

[0077] In this embodiment, based on the statistical regularity of a large sample, a statistically significant threshold can be derived by distinguishing the speed and displacement distribution intervals of the two states of looking down and fatigue. This avoids the defect that fixed thresholds are difficult to adapt to individual driving habits and avoids false alarms caused by inappropriate thresholds.

[0078] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the status determination method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0079] This application also provides a state discrimination device, please refer to... Figure 4 The state determination device includes: The opening and closing determination module 10 is used to determine the eyelid opening and closing degree based on the user's eye image; The parameter acquisition module 20 is used to acquire the user's eye movement parameters within an interval of frames when the eyelid opening degree is detected to be less than a preset opening degree threshold. The eye movement parameters include eyelid closing speed, eyeball downward displacement, and eyelid-eyeball movement correlation parameters. The eyelid-eyeball movement correlation parameters characterize the correlation between changes in eyelid opening degree and eyeball downward displacement. The parameter discrimination module 30 is used to discriminate the user's state based on the eye movement parameters.

[0080] The state discrimination device provided in this application, employing the state discrimination method described in the above embodiments, can solve the technical problem that existing static geometric feature discrimination methods cannot accurately distinguish between two behaviors: natural eyelid drooping due to looking downwards and slow eyelid closure due to fatigue. Compared with the prior art, the beneficial effects of the state discrimination device provided in this application are the same as those of the state discrimination method provided in the above embodiments, and other technical features in the state discrimination device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0081] This application provides a state discrimination device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the state discrimination method in the first embodiment described above.

[0082] The following is for reference. Figure 5The diagram illustrates a structural schematic suitable for implementing the state discrimination device in the embodiments of this application. The state discrimination device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The state discrimination device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0083] like Figure 5 As shown, the state discrimination device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the state discrimination device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the state discrimination device to communicate wirelessly or wiredly with other devices to exchange data. Although state discrimination devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0084] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0085] The state discrimination device provided in this application, employing the state discrimination method described in the above embodiments, can solve the technical problem that existing static geometric feature discrimination methods cannot accurately distinguish between two behaviors: natural eyelid drooping due to looking downwards and slow eyelid closure due to fatigue. Compared with the prior art, the beneficial effects of the state discrimination device provided in this application are the same as those of the state discrimination method provided in the above embodiments, and other technical features in this state discrimination device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0086] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0087] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0088] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the state discrimination method in the above embodiments.

[0089] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0090] The aforementioned computer-readable storage medium may be included in the state discrimination device; or it may exist independently and not assembled into the state discrimination device.

[0091] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the state determination device, cause the state determination device to perform the state determination method described above.

[0092] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0093] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0094] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0095] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described state discrimination method. This solves the technical problem that existing static geometric feature discrimination methods cannot accurately distinguish between two behaviors: natural eyelid drooping due to looking downwards and slow eyelid closure due to fatigue. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the state discrimination method provided in the above embodiments, and will not be repeated here.

[0096] The above description is only a part of the embodiments of this application and does not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

[0097] It should be noted that the data collection, tag management, rule setting, and push decision-making processes involved in this application are designed to work with other technical features to solve technical problems. They do not involve or support any illegal activities. Any data processing that may violate laws and regulations (such as unauthorized collection of privacy data, generation of discriminatory tags, setting unfair rules, or pushing illegal information) is not within the scope of protection of this application's technical solution. Of course, the user data in this application will be encrypted, anonymized, or de-identified before storage to ensure user data security.

Claims

1. A state discrimination method, characterized in that, The state determination method includes: Determine eyelid opening based on the user's eye image; When the eyelid opening degree is detected to be less than a preset opening degree threshold, the user's eye movement parameters within the interval frame number are obtained. The eye movement parameters include eyelid closing speed, eyeball downward displacement, and eyelid-eyeball movement correlation parameters. The eyelid-eyeball movement correlation parameters characterize the correlation between changes in eyelid opening degree and eyeball downward displacement. The user's state is determined based on the eye movement parameters.

2. The state discrimination method as described in claim 1, characterized in that, The step of obtaining the user's eye movement parameters within a frame interval includes: The eyelid closing speed is determined based on the first change in the eyelid opening degree within the specified frame interval; The eye tracking point in the eye image is tracked within the specified interval frame number, and the downward displacement of the eyeball is determined based on the second change of the eye tracking point within the specified interval frame number. Based on the eyelid opening and closing degree and the downward displacement of the eyeball, eyelid-eyeball motion-related parameters are determined.

3. The state discrimination method as described in claim 2, characterized in that, The step of determining eyelid-eye movement-related parameters based on the eyelid opening and closing degree and the downward displacement of the eyeball includes: Determine the covariance between the eyelid opening angle and the downward displacement of the eyeball; Based on the covariance, the first variance of the eyelid opening and closing degree, and the second variance of the downward displacement of the eyeball, the eyelid-eyeball motion correlation parameters are determined.

4. The state determination method as described in claim 1, characterized in that, The process of determining eyelid opening based on the user's eye image includes: Key point detection is performed on the eye image to obtain key points of the upper eyelid and the lower eyelid; The eyelid opening degree is determined based on the positional relationship between the key points of the upper eyelid and the key points of the lower eyelid in the same frame.

5. The state determination method as described in claim 1, characterized in that, Before determining the eyelid opening degree based on the user's eye image, the process also includes: Obtain the range of the first eyelid movement velocity and the range of the first eyeball relative displacement when the user is looking down; Obtain the range of second eyelid movement velocity and the range of relative displacement of the second eyeball when the user is in a fatigued state; The first closing speed and the second closing speed are determined based on the user's blinking speed, the minimum speed of the first eyelid movement speed range, and the maximum speed of the second eyelid movement speed range. The first downward rotation displacement and the second downward rotation displacement are determined based on the distance between the user's eyeball center and the lower eyelid, the minimum displacement of the first eyeball relative displacement range, and the maximum displacement of the second eyeball relative displacement range.

6. The state discrimination method according to any one of claims 1 to 5, characterized in that, The step of determining the user's state based on the eye movement parameters includes: If the downward displacement of the eyeball is greater than the first downward displacement, determine whether the eyelid closing speed is greater than the first closing speed, and whether the eyelid-eyeball motion correlation parameter is greater than the first correlation threshold. If the eyelid closing speed is greater than the first closing speed, and the eyelid-eye movement correlation parameter is greater than the first correlation threshold, then the user's state is determined to be looking down.

7. The state discrimination method according to any one of claims 1 to 5, characterized in that, The step of determining the user's state based on the eye movement parameters includes: If the downward displacement of the eyeball is less than the second downward displacement, determine whether the eyelid closing speed is less than the second closing speed, and whether the eyelid-eyeball motion correlation parameter is less than the second correlation threshold. If the eyelid closure speed is less than the second closure speed, and the eyelid-eye movement correlation parameter is less than the second correlation threshold, then the user's state is determined to be fatigued.

8. A state determination device, characterized in that, The device includes: The opening / closing determination module is used to determine the degree of eyelid opening / closing based on the user's eye image; The parameter acquisition module is used to acquire the user's eye movement parameters within an interval of frames when the eyelid opening degree is detected to be less than a preset opening degree threshold. The eye movement parameters include eyelid closing speed, eyeball downward displacement, and eyelid-eyeball movement correlation parameters. The eyelid-eyeball movement correlation parameters characterize the correlation between changes in eyelid opening degree and eyeball downward displacement. The parameter discrimination module is used to discriminate the user's state based on the eye movement parameters.

9. A state determination device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the state discrimination method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the state discrimination method as described in any one of claims 1 to 7.

Images