Eye state recognition method and device and electronic equipment

Abstract

The invention provides an eye state recognition method and device and electronic equipment, and relates to the field of driver fatigue driving detection, and the method comprises the steps: carrying out the key point detection of each frame of face image of a user, positioning the position region of two eyes of the user, calculating the closing degree of the two eyes in the position region of the two eyes, and carrying out the recognition of the fatigue driving of the driver. Determining a first prediction state corresponding to the two eyes based on the closed threshold value; the position areas of the two eyes are recognized through an eye state recognition model, and second prediction states corresponding to the two eyes are output; and outputting a final prediction state of the two eyes based on the first prediction state and the second prediction state so as to determine the closing condition of the two eyes. According to the method, the intelligent recognition capability of the model and the stability of the statistical method are fused for eye state recognition and correction, through multi-dimensional information integration, the limitation of a single judgment method is reduced, and the reliability of a prediction result is improved. The defect that in the prior art, squinting and the like of a driver caused by laughing or light stimulation are mistakenly recognized as eye closing can be overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of driver fatigue driving detection, and in particular to an eye state recognition method, device and electronic equipment. Background Art

[0002] Fatigue driving objectively leads to a decline in driving function, seriously affecting normal driving ability and making road traffic accidents more likely. Among the contactless methods, eye state performance, as a typical factor reflecting the driver's fatigue state, has attracted the attention of many researchers. In order to reduce the impact of various scene factors, infrared cameras fixed on the A-pillar or under the steering wheel are generally used to obtain images, and the algorithm is deployed on the vehicle-side domain controller device with limited resources.

[0003] Existing methods mainly use deep learning models to directly predict the eye closure state or facial key points, and use indicator parameters such as the continuous closure duration of the eyes to calculate the fatigue level. However, the driver's fatigue state is usually time-related and continuous. Existing methods tend to identify the driver's squinting eyes caused by smiling or light stimulation as closed eyes, and rarely consider the actual performance of the eyes in special states (such as squinting eyes caused by smiling or light stimulation, etc.), resulting in a large number of misidentifications, causing the system to frequently give false alarms, causing trouble to the driver. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an eye state recognition method, device and electronic device, which can overcome the shortcomings of the prior art that the driver's squinting due to smiling or light stimulation is easily misidentified as closed eyes, and can identify and correct the driver's eye state with high real-time performance and detection accuracy.

[0005] In a first aspect, an embodiment of the present invention provides an eye state recognition method, wherein the method comprises: obtaining a facial image stream of a user; the facial image stream comprises multiple frames of facial images; performing key point detection on each frame of facial image to locate the user's eye position area; calculating the degree of closure of both eyes included in the eye position area, and determining a first predicted state corresponding to the both eyes based on the degree of closure and a pre-configured closure threshold; wherein the first predicted state is used to predict whether the both eyes are in a closed state or in an open state; and, inputting the eye position area into a pre-trained eye state recognition model, and outputting a second predicted state corresponding to the both eyes through the eye state recognition model, wherein the second predicted state is also used to predict whether the both eyes are in a closed state or in an open state; outputting a final predicted state of the both eyes based on the first predicted state and the second predicted state; and the final predicted state is used to characterize the closure of the both eyes.

[0006] In combination with the first aspect, an embodiment of the present invention provides a first implementation of the first aspect, wherein the step of outputting the final prediction state of both eyes based on the first prediction state and the second prediction state includes: identifying a first weight parameter of the first prediction state and a second weight parameter of the second prediction state through a comprehensive recognition model; performing weighted calculation on the first prediction state and the second prediction state based on the first weight parameter and the second prediction state, and outputting a comprehensive prediction result; and determining the final prediction state of both eyes based on the comprehensive prediction result.

[0007] In combination with the first aspect, an embodiment of the present invention provides a second implementation of the first aspect, wherein the step of inputting the binocular position area into a pre-trained eye state recognition model includes: determining eye boundary points from the binocular position area; constructing a rectangular envelope box corresponding to the eye boundary points based on the three-dimensional coordinates of the eye boundary points; generating an eye image corresponding to the binocular position area based on the rectangular envelope box; and inputting the eye image into the pre-trained eye state recognition model.

[0008] In combination with the first aspect, an embodiment of the present invention provides a third implementation of the first aspect, wherein the step of constructing a rectangular envelope box corresponding to the eye boundary point based on the three-dimensional coordinates of the eye boundary point includes: calculating the width parameter and height parameter of the eye according to the three-dimensional coordinates of the eye boundary point; and generating a rectangular envelope box corresponding to the eye boundary point based on the width parameter and the height parameter.

[0009] In combination with the first aspect, an embodiment of the present invention provides a fourth implementation of the first aspect, wherein the step of generating an eye picture corresponding to the binocular position area based on a rectangular envelope frame includes: calculating the maximum width and height corresponding to the rectangular envelope frame according to a preset magnification; based on the maximum width and height and the center coordinates of the rectangular envelope frame, enlarging the rectangular envelope frame to construct an eye image area; and determining the picture corresponding to the eye image area as the eye picture corresponding to the binocular position area.

[0010] In combination with the first aspect, an embodiment of the present invention provides a fifth implementation of the first aspect, wherein the step of calculating the degree of closure of both eyes included in the binocular position area includes: determining a set of eye key point coordinates from the binocular position area, and determining eye closure key points from the eye key point coordinate set; calculating the three-dimensional Euclidean distance corresponding to the eye closure key points, and determining the degree of eye closure based on the three-dimensional Euclidean distance.

[0011] In combination with the first aspect, an embodiment of the present invention provides a sixth implementation of the first aspect, wherein the closure threshold includes a first threshold and a second threshold, wherein the first threshold is less than the second threshold; the step of determining a first predicted state corresponding to both eyes based on the closure degree and a pre-configured closure threshold comprises: judging whether the closure degree is less than the first threshold; if so, determining that both eyes are in a closed state; and judging whether the closure degree is greater than the second threshold; if so, determining that both eyes are in a non-closed state.

[0012] In combination with the first aspect, an embodiment of the present invention provides a seventh implementation of the first aspect, wherein the method also includes: if the degree of closure is at an intermediate value between the first threshold and the second threshold, determining the first predicted state corresponding to the previous frame of the degree of closure as the first predicted state corresponding to the current degree of closure.

[0013] In combination with the first aspect, an embodiment of the present invention provides an eighth implementation of the first aspect, wherein the method further includes: calculating the closure distribution of users within a preset time period based on closure statistics; and adaptively adjusting the closure threshold based on the closure distribution.

[0014] In combination with the first aspect, an embodiment of the present invention provides a ninth implementation of the first aspect, wherein the step of calculating the closure distribution of users within a preset time period based on closure statistics includes: based on a preset minimum statistical resolution, quantizing the closure within the preset time period to obtain closure distribution counts for multiple sub-time periods; and generating the user's closure distribution based on the closure distribution counts.

[0015] In combination with the first aspect, an embodiment of the present invention provides a tenth implementation of the first aspect, wherein the step of adaptively adjusting the closure threshold based on the closure distribution includes: obtaining an upper limit value and a lower limit value corresponding to the closure distribution; based on the upper limit value and the lower limit value, calculating a statistical result threshold corresponding to the closure; and adjusting the closure threshold according to the statistical result threshold.

[0016] In combination with the first aspect, an embodiment of the present invention provides an eleventh implementation manner of the first aspect, wherein the method also includes: determining whether the closure distribution count meets a preset statistical threshold; if not, removing the closure corresponding to the closure distribution count; and determining the upper limit value and the lower limit value based on the closure distribution corresponding to the current closure distribution count.

[0017] In combination with the first aspect, an embodiment of the present invention provides a twelfth implementation manner of the first aspect, wherein the step of determining the upper limit value corresponding to the closure distribution includes: performing weighted calculation on the maximum and second largest values ​​of the closure distribution count in the current closure distribution to determine the upper limit value corresponding to the closure distribution.

[0018] In a second aspect, an embodiment of the present invention provides an eye state recognition device, wherein the device includes: a data acquisition module, used to acquire a facial image stream of a user; the facial image stream includes multiple frames of facial images; a positioning module, used to perform key point detection on each frame of facial image to locate the user's eye position area; a data processing module, used to calculate the closure degree of both eyes included in the eye position area, and determine a first predicted state corresponding to the both eyes based on the closure degree and a pre-configured closure threshold; wherein the first predicted state is used to predict whether the both eyes are in a closed state or in an open state; and, inputting the eye position area into a pre-trained eye state recognition model, outputting a second predicted state corresponding to the both eyes through the eye state recognition model, wherein the second predicted state is also used to predict whether the both eyes are in a closed state or in an open state; an output module, used to output a final predicted state of the both eyes based on the first predicted state and the second predicted state; the final predicted state is used to characterize the closure of the both eyes.

[0019] In a third aspect, an embodiment of the present invention provides an electronic device, wherein the electronic device includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned eye state recognition method.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned eye state recognition method.

[0021] The embodiments of the present invention bring the following beneficial effects: the eye state recognition method, device and electronic device provided by the embodiments of the present invention obtain multiple frames of facial images by acquiring the facial image stream of the user, and perform key point detection on each frame of the facial image to locate the user's eye position area. Moreover, by calculating the closure degree of the eyes included in the area, the first predicted state of the eyes is determined in combination with the closure threshold, and the area is identified by the eye state recognition model to predict the second predicted state of the eyes. The final predicted state is further determined according to the prediction results of the two methods to determine the closure of the eyes. The embodiments of the present invention combine different prediction methods to determine the final state recognition result, which is hierarchical and complementary to accurately identify the driver's fatigue or distraction. The method based on the closure degree combined with the closure threshold can also improve the prediction accuracy, which can overcome the shortcomings of the prior art that the driver's squinting caused by smiling or light stimulation is easily misidentified as closed eyes, which helps to effectively identify the driver's eye closure state in actual driving scenarios, so as to perform real-time recognition and higher detection accuracy of the driver's eye state.

[0022] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0023] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A flowchart of an eye state recognition method provided by an embodiment of the present invention;

[0026] Figure 2 A flowchart of another eye state recognition method provided by an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of a key point of eye closure provided by an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of a closure distribution provided by an embodiment of the present invention;

[0029] Figure 5 A logic diagram of an eye state recognition method provided by an embodiment of the present invention;

[0030] Figure 6 A schematic diagram of the structure of an eye state recognition device provided by an embodiment of the present invention;

[0031] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] The present invention provides an eye state recognition method, device and electronic equipment, which can overcome the shortcomings of the prior art that the driver's squinting caused by smiling or light stimulation is easily misidentified as closed eyes, and can identify and correct the driver's eye state with high real-time performance and detection accuracy.

[0034] To facilitate understanding of this embodiment, firstly, a method for identifying eye states disclosed in an embodiment of the present invention is described in detail. Figure 1 A flow chart of an eye state recognition method is shown, and the method may include the following steps:

[0035] Step S102, obtaining a user's facial image stream.

[0036] Step S104: perform key point detection on each frame of facial image to locate the eye position area of ​​the user.

[0037] In a specific implementation, a driver's facial video stream is continuously captured by a camera facing the driver, and the facial image stream includes multiple frames of facial images. Furthermore, facial recognition technology is used to locate the facial area from the video frame. In the detected facial area, the positions of both eyes are further accurately located. In one embodiment, commonly used technologies include 68 facial landmark point detection models in the dlib library, which can accurately mark multiple facial feature points including the corners of the eyes.

[0038] Step S106, calculating the closure degree of the two eyes included in the two-eye position area, and determining a first predicted state corresponding to the two eyes based on the closure degree and a pre-configured closure threshold.

[0039] Step S108, inputting the eye position area into a pre-trained eye state recognition model, and outputting a second predicted state corresponding to the eyes through the eye state recognition model.

[0040] Step S110: outputting a final prediction state of both eyes based on the first prediction state and the second prediction state.

[0041] By analyzing the degree of eyelid opening and closing, the predicted state of closure of both eyes can be determined. When the eyelids cover most of the eyeball, it can be judged as closed. By observing the direction of the pupil and the frequency of blinking, long periods of looking away from the road or abnormally high blinking rates may indicate driver inattention or fatigue.

[0042] The prior art usually uses a deep learning model to directly predict the eye closure state, and calculates the fatigue level using indicator parameters such as the continuous closure time of the eyes. However, special circumstances (such as smiling or light stimulation) will also affect the actual performance of the eyes. To solve this problem, an embodiment of the present invention calculates the degree of closure corresponding to the position area of ​​both eyes, and determines a first predicted state based on the degree of closure and the closure threshold. Among them, the first predicted state is used to predict whether the eyes are in a closed state or in an open state. Furthermore, the second predicted state corresponding to the eyes is output through the eye state recognition model, and the second predicted state is also used to predict whether the eyes are in a closed state or in an open state.

[0043] Among them, by analyzing the features in the area where both eyes are located, such as the degree of eyelid coverage, an indicator that quantifies the degree of eye closure, that is, the degree of closure, can be calculated. The degree of closure can be determined by comparing the difference between the eyelid edge and the normal open eye state. If the degree of closure exceeds the preset closure threshold, the eyes are predicted to be in a closed state; otherwise, they are in a non-closed state. To further improve the accuracy of the judgment, the system will also use a more complex eye state recognition model to output a second predicted state. This model may be based on machine learning or deep learning. By training a large number of samples of open and closed eyes, it can provide a more refined state classification, and also distinguish between closed and non-closed states.

[0044] In order to reduce the misjudgment rate, the embodiment of the present invention determines the final prediction state based on the prediction results of the above two methods, and the final prediction state is used to characterize the closure of both eyes. In specific implementation, the first prediction state and the second prediction state can be fused, or comprehensively considered according to certain rules to finally determine the driver's eye state. The eye state recognition method of the embodiment of the present invention is hierarchical and complementary, which helps to effectively identify the driver's eye closure state in actual driving scenarios, so as to accurately identify the driver's fatigue or distraction.

[0045] Furthermore, based on the above embodiment, the embodiment of the present invention also provides another eye state recognition method. Figure 2 FIG. 4 is a flow chart showing another eye state recognition method provided by an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0046] Step S202, obtaining a user's facial image stream.

[0047] Step S204: perform key point detection on each frame of facial image to locate the eye position area of ​​the user.

[0048] The embodiment of the present invention uses an infrared camera to collect images of the user, and the generated raw data is pre-processed to obtain a face image. Furthermore, a key point detection model based on deep learning is used to accurately detect multiple key points of the face, such as eyes, etc. The position information of these key points is crucial for subsequent image analysis and processing.

[0049] The raw data generated by the infrared camera is pre-processed to obtain a standardized face image. First, the key point detection model after transfer learning is used to output 468 sets of face key point data and extract the eye key point coordinate set: k ={(x ik ,y ik ,z ik )}. Where k = 0 (left eye) or 1 (right eye), i represents the index of the element constituting the set, (x ik ,y ik ,z ik ) is the 3D coordinate of the key point. If the detection fails, abandon this frame data and wait for the next frame.

[0050] Step S206, determining an eye key point coordinate set from the eye position area, and determining an eye closure key point from the eye key point coordinate set.

[0051] Step S208, calculating the three-dimensional Euclidean distance corresponding to the eye closure key point, and determining the eye closure degree according to the three-dimensional Euclidean distance.

[0052] In the specific implementation, the eye key point coordinate set D is obtained through the above steps. k Finally, the eye closure key points are determined from the eye key point coordinate set, and the eye closure degree is calculated based on the three-dimensional Euclidean distance of these key points.

[0053] Specifically, define the eye closure key point P k = {P i,k | 1≤i≤8,i∈N}∈D k , k=0 (left eye) or 1 (right eye), there are 8 key points of eye closure in total, i refers to the set element index, which is a natural number from 1 to 8; N represents a natural number. Figure 3 A schematic diagram showing eye closure key points.

[0054] Let d i,j,k =||P i,k ,P j,k || 2 ,||*,*|| 2 To calculate the 3D Euclidean distance between two points, calculate the closure of the left and right eyes separately:

[0055]

[0056] Among them, A=[0.3,0.4,0.3], B k =[d 1,2,k ,d 3,4,k ,d 5,6,k ] T , S k ≥0. A is a 1×3 matrix of weight coefficients, and its value is determined by experiments; B k is a three-dimensional Euclidean distance 3×1 matrix. Let S k ≤0.5, MaxS k =max(S k ). Reference Figure 3 , d 1,2,k The 1 and 2 in S represent the three-dimensional Euclidean distance between the first and second eye closure key points, and so on. k This is the definition of eye closure, calculated by weighting the sum of the three distances and dividing by the fourth distance. 0.5 is an experimental value, and if it exceeds 0.5, it can be considered that the eyes are definitely open, and no larger value is required.

[0057] Step S210: determining a first predicted state corresponding to both eyes based on the closure degree and a pre-configured closure threshold.

[0058] In a specific implementation, the closed threshold includes a first threshold and a second threshold, wherein the first threshold is less than the second threshold. The first threshold may represent a threshold corresponding to a closed state, and the second threshold may be used to represent a threshold corresponding to a non-closed state. In one implementation, the embodiment of the present invention determines whether the degree of closedness is less than the first threshold; if so, determines that both eyes are in a closed state; and determines whether the degree of closedness is greater than the second threshold; if so, determines that both eyes are in a non-closed state.

[0059] In specific implementation, the first threshold and the second threshold of the embodiment of the present invention are obtained through adaptive learning, and the final threshold is related to the current calculation and the previous result, which can improve stability. Specifically refer to the following steps:

[0060] Adaptive learning of upper and lower thresholds for eye state judgment:

[0061] THRES i,j,k =α j ×THRES i-1,j,k +(1-α j 0×T i,j,k

[0062] Where i represents the number of times of adaptive learning, which is used to distinguish the current and previous parameter indexes and does not participate in the calculation itself; j = 0 (upper threshold, i.e., the second threshold) or 1 (lower threshold, i.e., the first threshold). k is used to indicate the left eye or the right eye. α j∈(0,1) is the learning rate, THRES i-1,j,k is the previous threshold, when i=0, THRES i,j,k =0, T i,j,k This is the statistical result threshold.

[0063] In specific implementation, the statistical result threshold is used to indicate the result determined by combining the statistical result of closure under a given input. Specifically, the steps of determining the statistical result threshold by combining the statistics of closure are as follows:

[0064] 1) Based on a preset minimum statistical resolution, the closure within a preset time period is quantified to obtain closure distribution counts of multiple sub-time periods; and the user's closure distribution is generated based on the closure distribution counts.

[0065] In order to improve the detection stability, the embodiment of the present invention further calculates the closure distribution of users within a preset time period based on closure statistics; and adaptively adjusts the closure threshold based on the closure distribution.

[0066] Specifically, the embodiment of the present invention can reduce erroneous judgments caused by momentary blinking or misidentification by statistically analyzing the distribution of eye closure within time T. Statistics within a long time window can smooth out instantaneous noise and provide more robust state judgment.

[0067] Here, a time T is defined, and the distribution of the closure degree of the left and right eyes in the time T is counted. From the above steps, it can be known that the closure degree S k is a continuous variable. The embodiment of the present invention defines a minimum statistical resolution u, where u mainly affects memory usage and calculation speed, 0.01≤u≤0.001. k Quantize to discrete values:

[0068]

[0069] Among them, floor(·) means rounding down, and the closure distribution is transformed into within T time The distribution counts in each segment. Each calculation gets an S k , DS is calculated by the above formula k , statistics for each DS k The number of occurrences is the closure distribution count. Figure 4 A schematic diagram of the closure distribution is shown.

[0070] Among them, the quantitative closure S kConsidering its distribution within time T can help the system distinguish between different behavior patterns such as rapid blinking, continuous eye closure (such as napping) or normal eye opening, which is particularly important for assessing the driver's fatigue state. The closure distribution shows the probability of the eyes being at a specified closure within time T, and after discretization, it can be considered as the count value of the number of times the specified closure occurs.

[0071] 2) Obtain the upper and lower limits corresponding to the closure distribution.

[0072] The upper limit and the lower limit are used to indicate the closure value determined according to the closure distribution, which respectively represent the closure corresponding to the maximum number of counts when the eyes are open and the closure when the eyes are completely closed. The upper limit and the lower limit are calculated to determine the statistical result threshold, so as to determine the closure threshold.

[0073] a-In the embodiment of the present invention, the upper limit value of the embodiment of the present invention is determined by the maximum value and the second largest value corresponding to the comprehensive closure distribution, and the maximum value and the second largest value are used to characterize the maximum value and the second largest value of the closure distribution count. In the specific implementation, the maximum value and the second largest value of the closure distribution count in the current closure distribution are weighted to further determine the upper limit value corresponding to the closure distribution.

[0074] The specific calculation is as follows:

[0075] ω=c1×Sp(S k )| max +(1-c1)×Sp(S k )| sub-max

[0076] Among them, c1∈(0,1) is the learning coefficient of different Sp, which can be simply understood as the weight of different Sp, and the specific value is determined through experiments; Sp(S k )| max 、Sp(S k )| sub-max are the S corresponding to the maximum and second largest probability values ​​respectively. k , probability is used to indicate the distribution probability of closure, that is, the number of times the above distribution is counted, and the statistic of this probability is the closure.

[0077] In summary, we get the upper limit value. The upper limit value ω is a quantity defined by the formula, which is the statistic S kThe upper limit value of (that is, the degree of closure) has the physical meaning of: the comprehensive value of the two degrees of closure that appear the most and the second most times, which can be understood as the degree of closure in the open eye state. The upper limit value determined by the embodiment of the present invention based on the maximum and second largest values ​​corresponding to the distribution probability of the degree of closure can define a reliable boundary and better adapt to individual differences and environmental changes. In order to more accurately distinguish between true eye closure events and possible misjudgments, such as misidentification caused by shadows, reflections from glasses, etc. The upper limit of closure, as a distinguishing criterion, helps to reduce false positives (false positives of eye closure).

[0078] b-Lower limit It is also a defined quantity, and its physical meaning is: the minimum closure degree of the closed eyes state, which can be simply understood as the closure degree of the closed eyes state. Specifically, it can be directly based on Figure 3 The closure distribution shown determines the lower limit.

[0079] Furthermore, in view of the possible recognition errors of the model, the embodiment of the present invention also performs noise filtering and outlier processing on the closure distribution within the above time T. In the specific implementation, it is determined whether the closure distribution count meets the preset statistical threshold; if not, the closure corresponding to the closure distribution count is removed; further, the upper limit and lower limit are determined based on the closure distribution corresponding to the current closure distribution count.

[0080] Specifically, a minimum statistical threshold ε is defined to remove data that are continuously lower than ε from the statistical value 0 in the statistical distribution. The minimum statistical threshold ε is the threshold corresponding to the distribution count, which is used to filter noise from the closure statistical distribution within time T and determine the statistic S k When the distribution count is less than the minimum statistical threshold ε, it can be understood that the count corresponding to the eye closure is small, which can indicate that the driver's eyes are not in an effective closed state, and then filtering is performed. After filtering the closure distribution based on the above steps, the optimized closure distribution can be obtained, and thus the optimized statistic S k The lower limit of and the statistic S k The upper limit value ω.

[0081] 3) Based on the upper and lower limits, calculate the statistical result threshold corresponding to the closure degree. The statistical result threshold T i,j,k The calculation is as follows:

[0082]

[0083] in, ω is the upper limit of the closure distribution, is the lower limit of the closure distribution. 0.6 and 0.4 are experimentally obtained values ​​used to calculate the upper and lower thresholds, i.e., the first threshold and the second threshold.

[0084] Further, based on the statistical result threshold adaptive learning, a first threshold and a second threshold of the closed threshold are obtained, and the eye state is judged based on the threshold. The upper and lower thresholds for eye state judgment in the embodiment of the present invention are updated through adaptive learning, which can be regarded as an iterative learning process: the latest threshold = the weighted sum of the last threshold and the current calculation result.

[0085] In summary, the first predicted state of both eyes can be determined, including the closed state and the non-closed state. Further, if the closure degree is in the middle of the first threshold and the second threshold, the first predicted state corresponding to the previous frame of the closure degree is determined as the first predicted state corresponding to the current closure degree.

[0086] In summary, the embodiments of the present invention can set a more detailed fatigue threshold based on the quantitative distribution of closure. For example, if a high degree of closure appears frequently in a short period of time, it may indicate that the driver is in an extremely fatigued state and needs to be warned or take measures. Based on the distribution of closure within the time window T, the system can dynamically adjust the closure threshold to adapt to different individuals or environmental changes. For example, when driving at night, due to the increase in the frequency of natural blinking, the system can automatically relax the closure threshold to reduce false alarms. In specific implementation, according to the Schmitt trigger principle, when the closure is lower than the lower limit (that is, the first threshold), 1 (closed state) is output, when it is higher than the upper limit (that is, the second threshold), 0 (non-closed state) is output, and when it is in the middle value, the previous result is output.

[0087] In summary, the embodiment of the present invention uses the closure degree of the left and right eyes calculated based on the eye key point data as a statistic, obtains the closure degree distribution of the left and right eyes within a certain period of time, and adaptively learns the upper and lower thresholds for eye state judgment after noise filtering. When the closure degree is lower than the lower limit, it is considered to be in a closed state, and when it is higher than the upper limit, it is considered to be in a non-closed state. By using an adaptive threshold to determine the state of the eyes, the judgment standard can be dynamically adjusted according to the specific habits and physiological characteristics of each user, effectively reducing misjudgments. Moreover, combined with the noise filtering mechanism, the instantaneous interference caused by blinking or eye movement can be removed, thereby improving the accuracy and efficiency of eye state monitoring.

[0088] Step S212: input the eye position area to a pre-trained eye state recognition model, and output a second predicted state corresponding to the eyes through the eye state recognition model.

[0089] In a specific implementation, the embodiment of the present invention determines the second predicted state of both eyes through the following steps:

[0090] 1) Determine the eye boundary points from the eye position region.

[0091] 2) Based on the three-dimensional coordinates of the eye boundary points, a rectangular envelope corresponding to the eye boundary points is constructed.

[0092] After key point detection in step S204, the eye key point coordinate set D can be obtained. k , further, based on the eye key point coordinate set, respectively calculate and obtain the rectangular envelope of the left and right eyes. In the specific implementation, according to the three-dimensional coordinates of the eye boundary point, the width parameter and height parameter of the eye are calculated; based on the width parameter and the height parameter, the rectangular envelope corresponding to the eye boundary point is generated. The rectangular envelope is represented by the following formula:

[0093]

[0094] Among them, (xmin k ,ymin k )、(xmax k ,ymax k ) are the coordinates of the upper left corner and lower right corner of the left and right eyes respectively. The width and height of the rectangular envelope are defined as:

[0095] w k =xmax k -xmin k

[0096] h k =ymax k -ymin k

[0097] The rectangular envelope can accurately select the eye area and maintain stable tracking even under complex backgrounds or lighting changes, thereby improving the accuracy of subsequent processing. Based on the eye area cropped out by the envelope, the interference of irrelevant information can be reduced, so that the model can be more focused on the analysis of eye features, thereby improving the recognition accuracy of eye states (such as open and closed, attention direction). In addition, only processing local eye areas can reduce the amount of calculation, speed up the processing flow, and facilitate real-time monitoring and instant feedback, which is particularly important for driving safety application scenarios that require rapid response. The embodiment of the present invention uses a model to identify the eye image corresponding to the rectangular envelope, which can not only improve the accuracy and real-time performance of the recognition system, but also enhance its robustness and wide application in a variety of environments.

[0098] 3) Based on the rectangular envelope, generate an eye image corresponding to the binocular position area.

[0099] In a specific implementation, the area corresponding to the rectangular envelope frame may be converted into an eye picture, so that the eye state recognition model can predict the eye state of the picture.

[0100] Furthermore, since the positions of eye key points obtained by different models or different input recognitions may be different, in order to ensure that the eyes are definitely within the rectangular area, the embodiment of the present invention also generates an eye image by magnifying the rectangular envelope formed by the original key points by a certain factor.

[0101] In a specific implementation, the embodiment of the present invention calculates the maximum width and height corresponding to the rectangular envelope according to a preset magnification; based on the maximum width and height and the center coordinates of the rectangular envelope, the rectangular envelope is enlarged to construct an eye image area; and the picture corresponding to the eye image area is determined as the eye picture corresponding to the binocular position area. Specifically, based on the above calculation formula, it can be known that the center coordinates of the rectangular envelope are:

[0102]

[0103] Furthermore, a rectangular envelope magnification r1 is defined, and the magnification r1 is determined by experiments. Take:

[0104] w′ k =w k ×r1

[0105] h′ k =h k ×r1

[0106] Let wh′ k =max(w′ k ,h′ k ), wh″ k For (cx k ,cy k ) is the center, the maximum width and height of the image under the image size limit is:

[0107] wh k = min(wh′ k ,wh″ k )

[0108] Furthermore, (cx k ,cy k ) is the center, wh k A square ROI image region is constructed for the width and height, and then the image corresponding to the image region is determined as the above-mentioned eye image.

[0109] Among them, in order to facilitate model input, the embodiment of the present invention also scales the eye image to unify the image size. Also, the eye image is normalized to convert the image 0-255 values ​​into the specific range required by the model, usually [0,1] or [-1,1]. In the specific implementation, the embodiment of the present invention scales the ROI image to 32×32 size and normalizes it to ensure the uniformity of the image size and the pixel value range meets the input requirements of the neural network or other machine learning models. In addition, the images processed as above can also be used to construct a data set to train the eye state recognition model.

[0110] 4) Inputting the eye image into a pre-trained eye state recognition model to determine the second predicted state corresponding to both eyes.

[0111] The eye state recognition model of the embodiment of the present invention is built based on a deep convolutional network, with an input of 32×32×3 and an output of two-dimensional, binary classification network. In addition, when the eye state recognition model performs image recognition, the loss function used is the cross entropy loss function, which is used to evaluate the difference between the probability distribution of the two types of outputs and the true distribution. The specific calculation formula is as follows:

[0112] loss i =-y i log(p i )+(1-y i )(1-log(p i ))

[0113] Among them, p i The probability that the network predicts that the i-th sample is closed, y i is the true label of the sample, 0 represents non-closed, 1 represents closed, and 0 and 1 are the second predicted states of the image recognition output of the eye state recognition model for the binocular position area.

[0114] In summary, the embodiment of the present invention locates the eye position area through the above steps to determine the eye key point data, calculates the rectangular envelope of the left and right eyes respectively, and then enlarges them by a certain magnification, and extracts the eye picture accordingly. The eye picture is generated based on the long side length of the rectangular envelope and corrects it to a square with a constant center. The picture is scaled to a specified size and normalized to construct a data set, which can be used to train the eye state recognition model, and the corresponding second predicted state can also be output by the eye state recognition model. The embodiment of the present invention can reduce the uncertainty of image changes, so that the features learned by the model are more consistent and generalized, thereby improving the accuracy of eye state (such as open and closed) recognition. In addition, the model can learn the ability to handle different lighting conditions, facial postures and expression changes during the training process, and improve the robustness in actual application scenarios. The embodiment of the present invention constructs a high-quality data set, which not only supports the current eye state recognition task, but also provides basic resources for future research in related fields, and facilitates the continuous optimization of the model and transfer learning in new scenarios.

[0115] Step S214: outputting the final predicted state of both eyes based on the first predicted state and the second predicted state.

[0116] Specifically, an embodiment of the present invention identifies a first weight parameter of a first prediction state and a second weight parameter of a second prediction state through a comprehensive recognition model; performs weighted calculation on the first prediction state and the second prediction state based on the first weight parameter and the second prediction state, and outputs a comprehensive prediction result; and determines the final prediction state of both eyes based on the comprehensive prediction result.

[0117] In specific implementation, the embodiment of the present invention uses the constructed comprehensive evaluation algorithm to output the final predicted state of the eye. i ∈{0,1},1≤i≤4,i∈N,rd i Used to characterize the first prediction state and the second prediction state.

[0118]

[0119] Among them, c i ∈(0,1) is the confidence parameter of the corresponding output. When Res≥τ, the final output of the system is 1 (closed state), otherwise the output is 0 (non-closed state). Among them, τ is the threshold obtained by experiment, c i Through experiments, a large number of labeled samples are sent to the model to obtain four state values, namely the two possible forms of the first predicted state and the two possible forms of the second predicted state. Since the true state value is known, the confidence parameter c i Both τ and τ are considered as unknowns and are calculated and determined according to certain rules (such as the least squares method).

[0120] In summary, Figure 5 The logic diagram of the eye state recognition method of the embodiment of the present invention is shown. The embodiment of the present invention determines the final prediction state based on the first prediction state corresponding to the closure degree and the second prediction state output by the model, which integrates the intelligent recognition ability of the model and the stability of the statistical method. Through multi-dimensional information integration, the limitations of a single judgment method are reduced and the reliability of the prediction results is improved. In addition, the above comprehensive method can better cope with complex and changeable actual scenes, whether it is the difference in physiological characteristics of different people, or the changes in external factors such as light and wearing glasses, it can maintain good recognition performance.

[0121] Furthermore, based on the above embodiment, the embodiment of the present invention also provides an eye state recognition device, Figure 6 FIG. 4 shows a schematic diagram of the structure of an eye state recognition device provided by an embodiment of the present invention. Figure 6 As shown, the device includes: a data acquisition module 100, which is used to acquire a facial image stream of a user; the facial image stream includes multiple frames of facial images; a positioning module 200, which is used to perform key point detection on each frame of facial image to locate the user's eye position area; a data processing module 300, which is used to calculate the closure degree of both eyes included in the eye position area, and determine the first predicted state corresponding to the both eyes based on the closure degree and a pre-configured closure threshold; wherein the first predicted state is used to predict whether the both eyes are in a closed state or in an open state; and, the eye position area is input into a pre-trained eye state recognition model, and the eye state recognition model outputs a second predicted state corresponding to the both eyes, wherein the second predicted state is also used to predict whether the both eyes are in a closed state or in an open state; an output module 400, which is used to output the final predicted state of the both eyes based on the first predicted state and the second predicted state; and the final predicted state is used to characterize the closure of the both eyes.

[0122] An eye state recognition device provided in an embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0123] Further, based on the above device embodiment, the embodiment of the present invention also provides Figure 2 Another eye state recognition device corresponding to the eye state recognition method shown, specifically, the above-mentioned output module 400 is also used to identify the first weight parameter of the first prediction state and the second weight parameter of the second prediction state through a comprehensive recognition model; perform weighted calculation on the first prediction state and the second prediction state based on the first weight parameter and the second prediction state, and output a comprehensive prediction result; based on the comprehensive prediction result, determine the final prediction state of both eyes.

[0124] The above-mentioned data processing module 300 is also used to determine the eye boundary points from the binocular position area; construct a rectangular envelope box corresponding to the eye boundary points based on the three-dimensional coordinates of the eye boundary points; generate an eye image corresponding to the binocular position area based on the rectangular envelope box; and input the eye image into a pre-trained eye state recognition model.

[0125] The data processing module 300 is further used to calculate the width parameter and height parameter of the eye according to the three-dimensional coordinates of the eye boundary point; and generate a rectangular envelope corresponding to the eye boundary point based on the width parameter and the height parameter.

[0126] The above-mentioned data processing module 300 is also used to calculate the maximum width and height corresponding to the rectangular envelope frame according to a preset magnification; based on the maximum width and height and the center coordinates of the rectangular envelope frame, the rectangular envelope frame is enlarged to construct an eye image area; and the picture corresponding to the eye image area is determined as the eye picture corresponding to the binocular position area.

[0127] The above-mentioned data processing module 300 is also used to determine the eye key point coordinate set from the eye position area, and determine the eye closure key point from the eye key point coordinate set; calculate the three-dimensional Euclidean distance corresponding to the eye closure key point, and determine the eye closure degree according to the three-dimensional Euclidean distance.

[0128] Furthermore, the above-mentioned closure threshold includes a first threshold and a second threshold, wherein the first threshold is smaller than the second threshold; the above-mentioned data processing module 300 is also used to determine whether the closure degree is smaller than the first threshold; if so, determine that both eyes are in a closed state; and, determine whether the closure degree is greater than the second threshold; if so, determine that both eyes are in a non-closed state.

[0129] The data processing module 300 is further configured to determine the first predicted state corresponding to the previous frame of the closure as the first predicted state corresponding to the current closure if the closure is at an intermediate value between the first threshold and the second threshold.

[0130] The data processing module 300 is further used to calculate the closure distribution of users within a preset time period based on closure statistics; and to adaptively adjust the closure threshold based on the closure distribution.

[0131] The above-mentioned data processing module 300 is also used to quantify the closure within a preset time period based on a preset minimum statistical resolution to obtain closure distribution counts of multiple sub-time periods; and generate the user's closure distribution based on the closure distribution counts.

[0132] The above-mentioned data processing module 300 is also used to obtain the upper limit value and the lower limit value corresponding to the closure distribution; based on the upper limit value and the lower limit value, calculate the statistical result threshold corresponding to the closure; and adjust the closure threshold according to the statistical result threshold.

[0133] The above-mentioned data processing module 300 is also used to determine whether the closure distribution count meets the preset statistical threshold; if not, remove the closure corresponding to the closure distribution count; and determine the upper limit and lower limit based on the closure distribution corresponding to the current closure distribution count.

[0134] The data processing module 300 is further used to perform weighted calculation on the maximum and second largest closure distribution counts in the current closure distribution to determine an upper limit value corresponding to the closure distribution.

[0135] An embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above Figures 1 to 5 The embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to perform the above Figures 1 to 5 Any of the steps of the method shown.

[0136] The embodiment of the present invention also provides a schematic diagram of the structure of an electronic device, such as Figure 7 FIG. 1 is a schematic diagram of the structure of the electronic device, wherein the electronic device includes a processor 71 and a memory 70, the memory 70 stores computer executable instructions that can be executed by the processor 71, and the processor 71 executes the computer executable instructions to implement the above Figures 1 to 5 Any of the methods shown.

[0137] exist Figure 7In the illustrated embodiment, the electronic device further includes a bus 72 and a communication interface 73, wherein the processor 71, the communication interface 73 and the memory 70 are connected via the bus 72. The memory 70 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 73 (which may be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. may be used. The bus 72 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. It may also be an AMBA (Advanced Microcontroller Bus Architecture) bus, wherein AMBA defines three types of buses, including an APB (Advanced Peripheral Bus) bus, an AHB (Advanced High-performance Bus) bus, and an AXI (Advanced Xtensible Interface) bus. The bus 72 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0138] The processor 71 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 71. The above processor 71 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor 71 reads the information in the memory and combines its hardware to complete the above Figures 1 to 5 Any of the methods shown.

[0139] The computer program product of an eye state recognition method, device and electronic device provided by the embodiment of the present invention includes a computer-readable storage medium storing a program code, and the instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can refer to the method embodiment, which will not be repeated here. A person skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the above method embodiment, which will not be repeated here. In addition, in the description of the embodiment of the present invention, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For those skilled in the art, the specific meaning of the above terms in the present invention can be understood in specific circumstances. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes. In the description of the present invention, it should be noted that the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0140] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for recognizing eye states, characterized in that: The method comprises: Acquire a facial image stream of a user; the facial image stream includes multiple frames of facial images; Performing key point detection on each frame of the facial image to locate the eye position area of ​​the user; Calculating the degree of closure of the two eyes included in the two-eye position area, and determining a first predicted state corresponding to the two eyes based on the degree of closure and a pre-configured closure threshold; wherein the first predicted state is used to predict whether the two eyes are in a closed state or in an open state; and, Inputting the eye position area into a pre-trained eye state recognition model, and outputting a second predicted state corresponding to the eyes through the eye state recognition model, wherein the second predicted state is also used to predict whether the eyes are in a closed state or an open state; The final predicted state of the two eyes is output based on the first predicted state and the second predicted state; the final predicted state is used to characterize the closure status of the two eyes.

2. The method according to claim 1, characterized in that The step of outputting the final predicted state of the binoculars based on the first predicted state and the second predicted state comprises: identifying a first weight parameter of the first prediction state and a second weight parameter of the second prediction state by a comprehensive recognition model; Performing weighted calculation on the first prediction state and the second prediction state based on the first weight parameter and the second prediction state, and outputting a comprehensive prediction result; Based on the comprehensive prediction result, the final prediction state of the two eyes is determined.

3. The method according to claim 1, characterized in that The step of inputting the eye position area into a pre-trained eye state recognition model comprises: determining eye boundary points from the two-eye position region; Based on the three-dimensional coordinates of the eye boundary points, construct a rectangular envelope corresponding to the eye boundary points; Based on the rectangular envelope, generating an eye picture corresponding to the binocular position area; The eye image is input into a pre-trained eye state recognition model.

4. The method according to claim 3, characterized in that The step of constructing a rectangular envelope corresponding to the eye boundary point based on the three-dimensional coordinates of the eye boundary point comprises: Calculating the width parameter and height parameter of the eye according to the three-dimensional coordinates of the eye boundary point; Based on the width parameter and the height parameter, a rectangular envelope corresponding to the eye boundary point is generated.

5. The method according to claim 3, characterized in that: The step of generating an eye picture corresponding to the binocular position area based on the rectangular envelope frame comprises: According to a preset magnification, the maximum width and height corresponding to the rectangular envelope are calculated; Based on the maximum width and height and the center coordinates of the rectangular envelope, the rectangular envelope is enlarged to construct an eye image area; The picture corresponding to the eye image area is determined as the eye picture corresponding to the binocular position area.

6. The method according to claim 1, characterized in that The step of calculating the closedness of both eyes included in the both-eyes position area comprises: Determine a set of eye key point coordinates from the binocular position region, and determine an eye closure key point from the set of eye key point coordinates; The three-dimensional Euclidean distance corresponding to the eye closure key point is calculated, and the eye closure degree is determined according to the three-dimensional Euclidean distance.

7. The method according to claim 1, characterized in that The closing threshold includes a first threshold and a second threshold, wherein the first threshold is smaller than the second threshold; The step of determining the first predicted state corresponding to the two eyes based on the closure degree and a preconfigured closure threshold comprises: Determining whether the degree of closure is less than the first threshold; If yes, determining that the eyes are closed; and, Determining whether the degree of closure is greater than the second threshold; If yes, it is determined that the eyes are in an open state.

8. The method according to claim 7, characterized in that The method further comprises: If the closure degree is at an intermediate value between the first threshold and the second threshold, the first prediction state corresponding to the previous frame of the closure degree is determined as the first prediction state corresponding to the current closure degree.

9. The method according to claim 7, characterized in that: The method further comprises: Statistics of the closure degree distribution of the user within a preset time period based on the closure degree; The closure threshold is adaptively adjusted based on the closure degree distribution.

10. The method according to claim 9, characterized in that The step of collecting statistics on the closure degree distribution of the users within a preset time period based on the closure degree includes: Based on a preset minimum statistical resolution, the closure degree within a preset time period is quantified to obtain the closure degree distribution counts of multiple sub-time periods; A closure distribution of the user is generated based on the closure distribution count.

11. The method according to claim 10, characterized in that The step of adaptively adjusting the closure threshold based on the closure distribution comprises: Obtaining an upper limit value and a lower limit value corresponding to the closure distribution; Based on the upper limit value and the lower limit value, calculating a statistical result threshold corresponding to the degree of closure; The closing threshold is adjusted according to the statistical result threshold.

12. The method according to claim 11, characterized in that The method further comprises: Determining whether the closure distribution count meets a preset statistical threshold; If not, remove the closure corresponding to the closure distribution count; The upper limit value and the lower limit value are determined based on the closure distribution corresponding to the current closure distribution count.

13. The method according to claim 11, characterized in that The step of determining an upper limit value corresponding to the closure distribution comprises: A weighted calculation is performed on the maximum and second largest closure distribution counts in the current closure distribution to determine an upper limit value corresponding to the closure distribution.

14. An eye state recognition device, characterized in that: The device comprises: A data acquisition module, used to acquire a facial image stream of a user; the facial image stream includes multiple frames of facial images; A positioning module, used for performing key point detection on each frame of the facial image to locate the eye position area of ​​the user; a data processing module, configured to calculate the degree of closure of the two eyes included in the two-eye position area, and determine a first predicted state corresponding to the two eyes based on the degree of closure and a pre-configured closure threshold; wherein the first predicted state is used to predict whether the two eyes are in a closed state or in an open state; and Inputting the eye position area into a pre-trained eye state recognition model, and outputting a second predicted state corresponding to the eyes through the eye state recognition model, wherein the second predicted state is also used to predict whether the eyes are in a closed state or an open state; An output module is used to output a final predicted state of the two eyes based on the first predicted state and the second predicted state; the final predicted state is used to characterize the closure status of the two eyes.

15. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the eye state recognition method described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the eye state recognition method described in any one of claims 1 to 13.

Images