Method and device for detecting eye state of driver and electronic equipment

Abstract

The invention provides a driver eye state detection method and device and electronic equipment, and the method comprises the steps: inputting a face region image of a driver wearing glasses into a face detection model, and outputting the eye contour coordinate information of a target eye; calculating eye state parameters of the target eye according to the eye contour coordinate information, and determining an eye region image; inputting the eye region image into an eye state detection model, and outputting a first eye state; the first eye state comprises an eye opening state, an eye closing state or a light reflecting state; and determining a final eye state of the driver according to the eye state parameter of the target eye and the first eye state. According to the mode, for a scene that a driver wears glasses, eye state parameters are calculated according to eye contour coordinate information of a target eye, a first eye state of the target eye is determined through an eye state detection model, and a final eye state of the driver is jointly determined in combination with the eye state parameters and the first eye state. And the recognition accuracy of the eye state of the driver is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device and electronic equipment for detecting the eye state of a driver. Background Art

[0002] The driver's eye condition is one of the important signs to judge whether the driver is tired, but nowadays, more and more drivers will wear myopia glasses or sunglasses to drive, especially when the sunlight is strong, the sunlight will be reflected on the lenses. At present, the driver's eye state recognition mainly relies on the opening and closing ratio of the eyes to judge the open and closed state of the eyes through the aspect ratio of the fixed point of the eyes. This method is aimed at the situation where the driver wears glasses, especially when there is sunlight reflection, and the misjudgment rate of the driver's eye state recognition is relatively high. Summary of the invention

[0003] The object of the present invention is to provide a method, device and electronic device for detecting the eye state of a driver, so as to improve the accuracy of identifying the eye state of the driver in a scenario where the driver wears glasses.

[0004] The present invention provides a method for detecting the eye state of a driver, the method comprising: obtaining a facial region image of the driver; wherein the driver wears glasses; inputting the facial region image into a pre-trained face detection model, and outputting eye contour coordinate information of the target eye of the driver through the face detection model; wherein the target eye includes: the left eye and the right eye of the driver; calculating the eye state parameters of the target eye according to the eye contour coordinate information; determining the eye region image corresponding to the target eye according to the eye contour coordinate information; inputting the eye region image into a pre-trained eye state detection model, and outputting the first eye state of the target eye through the eye state detection model; wherein the first eye state includes: an open eye state, a closed eye state or a reflective state; determining the final eye state of the driver according to the eye state parameters of the target eye and the first eye state.

[0005] Furthermore, the step of determining the final eye state of the driver according to the eye state parameters and the first eye state of the target eye includes: if the first eye state of at least one target eye is an open eye state, determining that the final eye state of the driver is an open eye state; if the first eye state of each target eye is a closed eye state, obtaining a preset threshold to determine the final eye state of the driver based on the preset threshold; if the first eye state of at least one target eye is a reflective state, for each target eye in a reflective state, detecting a reflective area in an eye area image corresponding to the target eye; processing the reflective area to obtain a processed eye area image; using the processed eye area image as a new eye area image, and repeatedly executing the step of inputting the eye area image into a pre-trained eye state detection model until the final eye state of the driver is determined.

[0006] Furthermore, if the first eye state of each target eye is a closed eye state, a preset threshold is obtained, and the step of determining the driver's final eye state based on the preset threshold includes: judging whether the eye state parameter of each target eye is greater than the preset threshold; if the eye state parameter of each target eye is not greater than the preset threshold, determining that the driver's final eye state is a closed eye state; if the eye state parameter of at least one target eye is greater than the preset threshold, determining that the driver's final eye state is an open eye state.

[0007] Furthermore, the step of processing the reflective area to obtain a processed eye area image includes: performing image restoration on the reflective area to obtain a restored eye area image; and performing histogram equalization on the restored eye area image to obtain a processed eye area image.

[0008] Furthermore, the eye contour coordinate information includes at least: left boundary point coordinates, right boundary point coordinates, upper boundary point coordinates of the upper eyelid, and lower boundary point coordinates of the lower eyelid of the target eye.

[0009] Furthermore, the step of obtaining the driver's facial area image includes: collecting the driver's image through a vehicle-mounted camera; extracting a first region of interest in the driver's image; wherein the first region of interest includes the driver's facial area; and reshaping the image size of the first region of interest to obtain a facial area image of the driver of a first specified size.

[0010] Furthermore, the step of determining an eye area image corresponding to the target eye based on the eye contour coordinate information includes: determining a minimum circumscribed rectangle of the target eye based on the eye contour coordinate information; extracting a second region of interest from the facial area image based on the minimum circumscribed rectangle; and processing the second region of interest to obtain an eye area image of a second specified size corresponding to the target eye.

[0011] The present invention provides a driver's eye state detection device, the device comprising: an acquisition module, used for acquiring a facial region image of the driver; wherein the driver wears glasses; a first output module, used for inputting the facial region image into a pre-trained face detection model, and outputting eye contour coordinate information of the driver's target eye through the face detection model; wherein the target eye includes: the driver's left eye and right eye; a calculation module, used for calculating eye state parameters of the target eye according to the eye contour coordinate information; a first determination module, used for determining the eye region image corresponding to the target eye according to the eye contour coordinate information; a second output module, used for inputting the eye region image into a pre-trained eye state detection model, and outputting the first eye state of the target eye through the eye state detection model; wherein the first eye state includes: an open eye state, a closed eye state or a reflective state; a second determination module, used for determining the driver's final eye state according to the eye state parameters of the target eye and the first eye state.

[0012] An electronic device provided by the present invention includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement any of the above-mentioned methods for detecting the eye state of a driver.

[0013] The present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement any of the above-mentioned methods for detecting the eye state of a driver.

[0014] The eye state detection method, device and electronic device of a driver provided by the present invention obtain the face area image of the driver; wherein the driver wears glasses; the face area image is input into a pre-trained face detection model, and the eye contour coordinate information of the target eye of the driver is output through the face detection model; wherein the target eye includes: the left eye and the right eye of the driver; according to the eye contour coordinate information, the eye state parameters of the target eye are calculated; according to the eye contour coordinate information, the eye area image corresponding to the target eye is determined; the eye area image is input into a pre-trained eye state detection model, and the first eye state of the target eye is output through the eye state detection model; wherein the first eye state includes: an open eye state, a closed eye state or a reflective state; according to the eye state parameters of the target eye and the first eye state, the final eye state of the driver is determined. This method aims at the scenario where the driver wears glasses. It calculates the eye state parameters according to the eye contour coordinate information of the target eye, determines the first eye state of the target eye through the eye state detection model, and jointly determines the driver's final eye state by combining the eye state parameters and the first eye state, thereby improving the accuracy of the driver's eye state recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A flowchart of a method for detecting the eye state of a driver provided by an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of a key location point provided by an embodiment of the present invention;

[0018] Figure 3 A flowchart of a method for detecting the eye state of a driver provided by an embodiment of the present invention;

[0019] Figure 4 A schematic diagram of the structure of a driver's eye state detection device provided by an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] At present, more and more drivers wear glasses to drive, especially when the sunlight is strong. The sunlight will reflect on the lenses of the glasses, making it difficult for the camera of the driver fatigue monitoring system to collect pictures to judge the driver's eye state. If the driver fatigue alarm is not timely, it will cause unpredictable safety hazards. At present, the driver's eye state recognition is mainly based on the opening and closing ratio of the eyes. This method has low robustness, especially for the driver's eye state detection when wearing glasses. When the sunlight is strong, the misjudgment rate is high, which has a significant impact on the driver's driving experience. Based on this, an embodiment of the present invention provides a driver's eye state detection method, device and electronic device. This technology can be applied to applications that need to identify the eye state of drivers wearing glasses.

[0023] To facilitate understanding of this embodiment, a method for detecting the eye state of a driver disclosed in an embodiment of the present invention is first introduced. Figure 1 As shown, the method comprises the following steps:

[0024] Step S102, obtaining a facial region image of the driver; wherein the driver wears glasses;

[0025] The above-mentioned facial area image can be a rectangular image, which usually includes the driver's facial information; the above-mentioned glasses can be myopia glasses, hyperopia glasses or sunglasses, etc.; in actual implementation, in order to detect the driver's eye state, it is usually necessary to obtain the driver's facial area image first.

[0026] Step S104, inputting the facial region image into a pre-trained face detection model, and outputting eye contour coordinate information of the driver's target eyes through the face detection model; wherein the target eyes include: the driver's left eye and right eye;

[0027] The above-mentioned face detection model can be implemented by a variety of convolutional neural networks, such as residual networks, VGG networks, etc.; the above-mentioned eye contour coordinate information usually includes at least: the left boundary point mark of the target eye, the right boundary point coordinates, the upper boundary point coordinates of the upper eyelid and the lower boundary point coordinates of the lower eyelid, wherein the left boundary point coordinates can be understood as the coordinates corresponding to the leftmost boundary point of the target eye; the right boundary point coordinates can be understood as the coordinates corresponding to the rightmost boundary point of the target eye; the upper boundary point coordinates can be understood as the coordinates corresponding to the uppermost boundary point of the upper eyelid; the lower boundary point coordinates can be understood as the coordinates corresponding to the lowermost boundary point of the lower eyelid; it should be noted that in order to improve the accuracy of detection, the coordinate information of other position points of the target eye can be added as needed on the basis of the above-mentioned four types of coordinate information. In actual implementation, the above-mentioned acquired facial area image can be input into a pre-trained face detection model, and the facial area image is detected by the face detection model to output the eye contour coordinate information of the driver's left eye and the eye contour coordinate information of the right eye.

[0028] Step S106, calculating the eye state parameters of the target eye according to the eye contour coordinate information;

[0029] The above eye state parameter can be used to indicate whether the target eye is in an open or closed state. The eye state parameter is usually a value calculated based on the above eye contour coordinate information. Generally, the larger the value, the greater the probability that the target eye is in an open state, and the smaller the value, the greater the probability that the target eye is in a closed state. In specific implementation, the eye parameter of the target eye can be calculated based on the above eye contour coordinate information in a preset calculation method.

[0030] Step S108, determining an eye region image corresponding to the target eye according to the eye contour coordinate information;

[0031] After obtaining the above-mentioned eye contour coordinate information, the eye area image corresponding to the target eye can be extracted from the facial area image based on the eye contour coordinate information. The eye area image can be a rectangular image, which usually includes the driver's eye information.

[0032] Step S110, inputting the eye region image into a pre-trained eye state detection model, and outputting a first eye state of the target eye through the eye state detection model; wherein the first eye state includes: an open eye state, a closed eye state, or a reflective state;

[0033] The above-mentioned open eye state indicates that the target eye is in an open state; the above-mentioned closed eye state indicates that the target eye is in a closed state; the above-mentioned reflective state can be understood as the reflective phenomenon caused by sunlight shining on the glasses worn by the driver. In the reflective state, there will usually be a reflective area with higher brightness on the eye area image; the above-mentioned eye state detection model can be implemented by a variety of convolutional neural networks, such as residual networks, VGG networks, etc.; in actual implementation, the above-mentioned acquired eye area image can be input into a pre-trained eye state detection model, and the eye area image is detected by the eye state detection model to output the first eye state of the driver's left eye and the first eye state of the driver's right eye, wherein the first eye state of the left eye and the first eye state of the right eye may be the same or different; for example, the left eye may be in an open eye state, a closed eye state or a reflective state, and the right eye may be in an open eye state, a closed eye state or a reflective state.

[0034] Step S112, determining the final eye state of the driver according to the eye state parameter of the target eye and the first eye state.

[0035] In actual implementation, the final eye state of the left eye can be determined according to the above-mentioned eye state parameters of the left eye and the first eye state of the left eye; the final eye state of the right eye can be determined according to the above-mentioned eye state parameters of the right eye and the first eye state of the right eye; the final eye state of the driver can be determined by combining the final eye state of the left eye and the final eye state of the right eye; for example, if the final eye state of the left eye and the final eye state of the right eye are both open eyes, then it can be determined that the final eye state of the driver is open eyes; if the final eye state of the left eye and the final eye state of the right eye are both closed eyes, then it can be determined that the final eye state of the driver is closed eyes, etc.

[0036] The above-mentioned driver's eye state detection method obtains the driver's face area image; wherein the driver wears glasses; the face area image is input into a pre-trained face detection model, and the eye contour coordinate information of the driver's target eye is output through the face detection model; wherein the target eye includes: the driver's left eye and right eye; according to the eye contour coordinate information, the eye state parameter of the target eye is calculated; according to the eye contour coordinate information, the eye area image corresponding to the target eye is determined; the eye area image is input into a pre-trained eye state detection model, and the first eye state of the target eye is output through the eye state detection model; wherein the first eye state includes: open eye state, closed eye state or reflective state; according to the eye state parameter of the target eye and the first eye state, the driver's final eye state is determined. This method is aimed at the scene where the driver wears glasses, the eye state parameter is calculated according to the eye contour coordinate information of the target eye, the first eye state of the target eye is determined through the eye state detection model, and the final eye state of the driver is determined by combining the eye state parameter and the first eye state, thereby improving the accuracy of the driver's eye state recognition.

[0037] The embodiment of the present invention further provides another method for detecting the eye state of a driver, which is implemented on the basis of the method in the above embodiment, and comprises the following steps:

[0038] Step 1: Collect the driver's image through the vehicle camera;

[0039] The above-mentioned vehicle-mounted camera is usually installed at the A-pillar position of the vehicle, which is located at the front of the vehicle body, on the pillars on both sides of the front windshield, and above the left and right rearview mirrors. The vehicle-mounted camera can adopt an automotive-grade infrared camera, etc.; the above-mentioned driver image usually includes at least the driver's facial area, for example, it can be an image of the driver's upper body, etc. In actual implementation, when the driver is driving, the driver's driver image can be collected in real time through the vehicle-mounted camera. The image size of the driver image can be 1280 pixels × 800 pixels × 3 channels, for example, it can be an RGB color image.

[0040] Step 2: extracting a first region of interest in the driver image; wherein the first region of interest includes the driver's face area;

[0041] In actual implementation, in order to improve the accuracy and reliability of image processing, it is usually necessary to preprocess the above-mentioned driver image. Specifically, the first region of interest in the driver image can be extracted first; the first region of interest can be understood as the area in the driver image that needs to be processed. In this embodiment, the first region of interest needs to include the driver's facial area. Specifically, based on the outline of the facial area in the driver image, the minimum circumscribed rectangle corresponding to the outline can be determined, and based on the minimum circumscribed rectangle, the first region of interest including the driver's facial area can be cropped out from the driver image.

[0042] Step 3, resizing the image of the first region of interest to obtain a facial region image of the driver of a first specified size; wherein the driver wears glasses;

[0043] The above-mentioned image resizing refers to re-adjusting the size of the above-mentioned first region of interest to be suitable for the input requirements of the subsequent face detection model; the above-mentioned first specified size can be set according to actual needs, for example, it can be 192 pixels × 192 pixels × 3 channels, etc.; in actual implementation, the image size of the above-mentioned first region of interest is usually not compatible with the input requirements of the subsequent face detection model, therefore, it is necessary to first adjust the image size of the first region of interest to obtain a facial area image that meets the requirements.

[0044] Step 4: input the facial region image into a pre-trained face detection model, and output eye contour coordinate information of the driver's target eyes through the face detection model; wherein the target eyes include: the driver's left eye and right eye;

[0045] Step 5, calculating the eye state parameters of the target eye according to the eye contour coordinate information;

[0046] In actual implementation, after the facial region image is input into the pre-trained face detection model, the eye contour coordinate information of the driver's left eye and the eye contour coordinate information of the right eye can be output, such as Figure 2 A schematic diagram of a key position point is shown, which includes eight key position points, namely, the left boundary point P1 of the target eye, the right boundary point P5, the upper boundary point P3 of the upper eyelid, the lower boundary point P7 of the lower eyelid, two boundary points P2 and P4 selected on the left and right sides of the upper boundary point of the upper eyelid, and two boundary points P6 and P8 selected on the left and right sides of the lower boundary point of the lower eyelid; for the convenience of explanation, taking the driver's left eye as an example, that is, collecting the coordinates of the eight key position points of the left eye, the eye contour coordinate information of the left eye can be expressed as: P = [P1, P2, P3, P4, P5, P6, P7, P8], and the eye state parameter of the target eye can be calculated according to the following formula:

[0047]

[0048] Among them, P i , i=1,2,...,8 are the position coordinates of the 8 key positions of the left eye, WEAR is the eye state parameter, ||P m -P n ||, m, n∈{1, 2, ..., 8} is P m and P n The distance between two key location points.

[0049] The right eye and the left eye are symmetrical, and the same calculation method can be used to obtain the WEAR of the right eye. Thus, the WEAR values ​​corresponding to the left and right eyes of the driver can be obtained.

[0050] Step 6: Determine the minimum circumscribed rectangle of the target eye according to the eye contour coordinate information;

[0051] In actual implementation, the outline indicated by the above eye outline coordinate information is usually an irregular shape. For the convenience of calculation, the minimum bounding rectangle of the target eye can be determined based on the eye outline coordinate information. The method for determining the minimum bounding rectangle can refer to related technologies and will not be repeated here.

[0052] Step 7: extracting a second region of interest from the facial region image according to the minimum circumscribed rectangle;

[0053] The second region of interest may be cropped from the facial region image according to the minimum circumscribed rectangle of the target eye, that is, the second region of interest may be understood as the minimum rectangular region including the target eye.

[0054] Step eight, processing the second region of interest to obtain an eye region image of a second specified size corresponding to the target eye;

[0055] The second specified size can be set according to actual needs, for example, it can be 32 pixels × 32 pixels, etc.; in actual implementation, the image size of the second region of interest is usually not compatible with the input requirements of the subsequent eye state detection model. At the same time, in order to further improve the accuracy and reliability of eye state detection, the second region of interest can be subjected to noise reduction processing, normalization processing, etc., and the image size of the second region of interest can be reshaped to the second specified size to obtain an eye region image that meets the requirements. The eye region image is usually a grayscale image. Among them, the methods of noise reduction processing, normalization processing, etc. can refer to the relevant technology and will not be repeated here.

[0056] Step nine, inputting the eye region image into a pre-trained eye state detection model, and outputting a first eye state of the target eye through the eye state detection model; wherein the first eye state includes: an open eye state, a closed eye state, or a reflective state;

[0057] Step 10: if the first eye state of at least one target eye is an open eye state, determining that the final eye state of the driver is an open eye state;

[0058] In actual implementation, if the eye state detection model is used to detect that the first eye state of the driver's left eye is an open state, or that the first eye state of the driver's right eye is an open state, or that both the first eye state of the driver's left eye and the first eye state of the right eye are open states, it can be determined that the driver's final eye state is an open state.

[0059] Step eleven: if the first eye state of each target eye is a closed eye state, obtaining a preset threshold value to determine the final eye state of the driver based on the preset threshold value;

[0060] This step eleven can be achieved by following steps A to B:

[0061] Step A, determining whether the eye state parameter of each target eye is greater than a preset threshold value, and if the eye state parameter of each target eye is less than the preset threshold value, determining that the driver's final eye state is a closed eye state;

[0062] Step B: if the eye state parameter of at least one target eye is not less than a preset threshold, determine that the driver's final eye state is an open eye state.

[0063] The above-mentioned preset threshold can be set according to actual needs; in actual implementation, when the eye state detection model is used to detect and determine that the first eye state of the driver's left eye is a closed eye state, and the first eye state of the right eye is also a closed eye state, taking into account the possible influence of factors such as the size of the driver's eyes, in order to further ensure the accuracy of the detection, the preset threshold can be obtained, and the eye state parameters of the left eye and the eye state parameters of the right eye are respectively compared with the preset threshold. If the eye state parameters of both eyes are less than the preset threshold, it can be determined that the driver's final eye state is a closed eye state, thereby strengthening the judgment conditions of the closed eye state and reducing the occurrence of false alarms.

[0064] If the eye state parameter of the left eye is not less than the preset threshold, or the eye state parameter of the right eye is not less than the preset threshold, or the target state parameters of both eyes are not less than the preset threshold, it can be considered that the driver is not actually in the eyes-closed state, that is, his final eye state is the eyes-open state.

[0065] Step twelve, if the first eye state of at least one target eye is a reflective state, for each target eye in the reflective state, detect the reflective area in the eye area image corresponding to the target eye; process the reflective area to obtain a processed eye area image; use the processed eye area image as a new eye area image, and repeat the step of inputting the eye area image into a pre-trained eye state detection model until the final eye state of the driver is determined.

[0066] The step of processing the reflective area to obtain a processed eye area image can be achieved by following steps C and D:

[0067] Step C, performing image restoration on the reflective area to obtain a restored eye area image;

[0068] Step D: performing histogram equalization processing on the repaired eye area image to obtain a processed eye area image.

[0069] In actual implementation, if the first eye state of at least one target eye is a reflective state, since the reflective area is generally brighter than the surrounding area, the brightness threshold α can be preset. brig To detect the reflective area in the eye area image, a binary mask image representing the reflective area can be generated in the following manner, wherein when the brightness value of the pixel point is greater than the preset brightness threshold α brig When the brightness value of the pixel is less than or equal to the preset brightness threshold α, the output is 1. brig When , the output is 0:

[0070]

[0071] Where x and y represent the coordinates of the pixel; gray (x, y) represents the brightness value of the pixel with coordinates (x, y); α brig is the preset brightness threshold;

[0072] The binary mask image is the same size as the corresponding eye area image including the reflective area, wherein in the binary mask image, the reflective area is white and the rest is black, and the reflective area can be determined based on the binary mask image.

[0073] In this embodiment, a method based on partial differential equations and a method based on texture synthesis can be used to implement image restoration and remove reflections. For each pixel, its restoration priority ρ is calculated. p , ρ p =C p *D p , C pis the confidence term, which indicates the confidence of the known pixels around pixel p. Pixel p is a pixel point on the boundary of the reflective area. p is a data item that represents the intensity of the boundary where pixel p is located. Select the boundary pixel p with the highest priority and search for the best matching texture block q around it. Copy the pixels of the best matching texture block q to the texture block in the area to be repaired, and then update the area in turn to remove the reflection. Confidence Ψ p is the inpainted block centered at pixel p, |Ψ p | is the size of the repair block. Data item is the normal vector of the image gradient, n p is the normal vector of the boundary pixel p. The specific image restoration method can refer to the relevant technology and will not be repeated here.

[0074] The above-mentioned histogram equalization is a method of adjusting contrast by using an image histogram; after obtaining the above-mentioned repaired eye area image, the repaired eye area image can be subjected to histogram equalization processing to enhance the target eye and obtain a processed eye area image, which is then input into a pre-trained eye state detection model for detection and identification to determine the driver's final eye state.

[0075] For easier understanding, see Figure 3 The flowchart of a method for detecting the eye state of a driver is shown. First, a driver image is collected through a vehicle-mounted camera, a facial area image of a first specified size of the driver is determined based on the driver image, the eye contour coordinate information of the target eye is detected through a face detection model, and a WEAR value (corresponding to the above-mentioned eye state parameter) is calculated, and the eye area image is determined according to the eye contour coordinate information, and is input into the eye state detection model. If the first eye state of at least one target eye is an open eye state, the final eye state of the driver is determined to be an open eye state; if the first eye state of each target eye is a closed eye state, it is determined whether the WEAR value of each target eye is greater than a preset threshold value T. If the eye state parameter of each target eye is not greater than the preset threshold value, it is determined that the final eye state of the driver is a closed eye state; if the eye state parameter of at least one target eye is greater than the preset threshold value, it is determined that the final eye state of the driver is a closed eye state.

[0076] If the first eye state of at least one target eye is a reflective state, the reflective area is detected, the reflective area is removed by image restoration, the target eye is enhanced by histogram equalization, and a processed eye area image is obtained, which is re-input into the eye state detection model for detection.

[0077] The above-mentioned driver's eye state detection method provides a method for identifying the driver's eye state when the driver wears glasses, especially when there is sunlight reflection. This method overcomes the reflection of the lenses when the driver wears glasses and is a more robust and better detection effect eye state detection method; considering that it is not stable enough to judge the state of the eyes based solely on eye state parameters, this method combines the eye state parameters and the first eye state to jointly determine the driver's final eye state, thereby improving the accuracy of identifying the driver's eye state.

[0078] The embodiment of the present invention provides a driver's eye state detection device, such as Figure 4 As shown, the device includes: an acquisition module 40, used to acquire a facial area image of a driver; wherein the driver wears glasses; a first output module 41, used to input the facial area image into a pre-trained face detection model, and output eye contour coordinate information of the driver's target eye through the face detection model; wherein the target eye includes: the driver's left eye and right eye; a calculation module 42, used to calculate the eye state parameters of the target eye according to the eye contour coordinate information; a first determination module 43, used to determine the eye area image corresponding to the target eye according to the eye contour coordinate information; a second output module 44, used to input the eye area image into a pre-trained eye state detection model, and output the first eye state of the target eye through the eye state detection model; wherein the first eye state includes: an open eye state, a closed eye state or a reflective state; a second determination module 45, used to determine the driver's final eye state according to the eye state parameters of the target eye and the first eye state.

[0079] The above-mentioned driver's eye state detection device, for the scenario where the driver wears glasses, calculates the eye state parameters based on the eye contour coordinate information of the target eye, determines the first eye state of the target eye through the eye state detection model, and combines the eye state parameters and the first eye state to jointly determine the driver's final eye state, thereby improving the accuracy of identifying the driver's eye state.

[0080] Furthermore, the second determination module is also used for: if the first eye state of at least one target eye is an open eye state, determining that the final eye state of the driver is an open eye state; if the first eye state of each target eye is a closed eye state, obtaining a preset threshold to determine the final eye state of the driver based on the preset threshold; if the first eye state of at least one target eye is a reflective state, for each target eye in a reflective state, detecting the reflective area in the eye area image corresponding to the target eye; processing the reflective area to obtain a processed eye area image; using the processed eye area image as a new eye area image, and repeatedly executing the step of inputting the eye area image into a pre-trained eye state detection model until the final eye state of the driver is determined.

[0081] Furthermore, the second determination module is also used to: determine whether the eye state parameters of each target eye are greater than a preset threshold; if the eye state parameters of each target eye are not greater than the preset threshold, determine that the driver's final eye state is a closed eye state; if the eye state parameters of at least one target eye are greater than the preset threshold, determine that the driver's final eye state is an open eye state.

[0082] Furthermore, the second determination module is also used to: perform image restoration on the reflective area to obtain a restored eye area image; and perform histogram equalization processing on the restored eye area image to obtain a processed eye area image.

[0083] Furthermore, the eye contour coordinate information includes at least: left boundary point coordinates, right boundary point coordinates, upper boundary point coordinates of the upper eyelid, and lower boundary point coordinates of the lower eyelid of the target eye.

[0084] Furthermore, the acquisition module is also used to: collect the driver's image through the vehicle-mounted camera; extract the first region of interest in the driver's image; wherein the first region of interest includes the driver's facial area; reshape the image size of the first region of interest to obtain the driver's facial area image of a first specified size.

[0085] Furthermore, the first determination module is also used to: determine the minimum circumscribed rectangle of the target eye based on the eye contour coordinate information; extract a second region of interest from the facial area image based on the minimum circumscribed rectangle; and process the second region of interest to obtain an eye area image of a second specified size corresponding to the target eye.

[0086] The driver's eye state detection device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned driver's eye state detection method embodiment. For the sake of brief description, for matters not mentioned in the embodiment of the driver's eye state detection device, reference may be made to the corresponding contents in the aforementioned driver's eye state detection method embodiment.

[0087] The embodiment of the present invention further provides an electronic device, see Figure 5 As shown, the electronic device includes a processor 130 and a memory 131, wherein the memory 131 stores machine executable instructions that can be executed by the processor 130, and the processor 130 executes the machine executable instructions to implement the above-mentioned driver's eye state detection method.

[0088] Further, Figure 5 The electronic device shown further includes a bus 132 and a communication interface 133 , and the processor 130 , the communication interface 133 and the memory 131 are connected via the bus 132 .

[0089] The memory 131 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0090] The processor 130 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 130. The above processor 130 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131 and completes the steps of the method of the above embodiment in combination with its hardware.

[0091] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned driver's eye state detection method. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0092] The driver's eye state detection method, device and computer program product of the electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0093] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting a driver's eye state, characterized in that: The method comprises: Acquire a facial region image of a driver, wherein the driver wears glasses; Input the facial region image into a pre-trained face detection model, and output eye contour coordinate information of the target eyes of the driver through the face detection model; wherein the target eyes include: the left eye and the right eye of the driver; Calculating eye state parameters of the target eye according to the eye contour coordinate information; Determining an eye region image corresponding to the target eye according to the eye contour coordinate information; Input the eye region image into a pre-trained eye state detection model, and output a first eye state of the target eye through the eye state detection model; wherein the first eye state includes: an open eye state, a closed eye state, or a reflective state; The final eye state of the driver is determined according to the eye state parameter of the target eye and the first eye state.

2. The method according to claim 1, characterized in that The step of determining the final eye state of the driver according to the eye state parameter of the target eye and the first eye state comprises: If the first eye state of at least one target eye is an open eye state, determining that the final eye state of the driver is an open eye state; If the first eye state of each target eye is a closed eye state, obtaining a preset threshold value to determine the final eye state of the driver based on the preset threshold value; If the first eye state of at least one target eye is a reflective state, for each target eye in the reflective state, a reflective area in an eye area image corresponding to the target eye is detected; the reflective area is processed to obtain a processed eye area image; the processed eye area image is used as a new eye area image, and the step of inputting the eye area image into a pre-trained eye state detection model is repeated until the final eye state of the driver is determined.

3. The method according to claim 2, characterized in that If the first eye state of each target eye is a closed eye state, the step of obtaining a preset threshold value to determine the final eye state of the driver based on the preset threshold value includes: Determining whether the eye state parameter of each target eye is greater than a preset threshold value, and if the eye state parameter of each target eye is not greater than the preset threshold value, determining that the final eye state of the driver is a closed eye state; If the eye state parameter of at least one target eye is greater than the preset threshold, it is determined that the final eye state of the driver is an open eye state.

4. The method according to claim 2, characterized in that: The step of processing the reflective area to obtain a processed eye area image comprises: Performing image restoration on the reflective area to obtain a restored eye area image; The repaired eye region image is subjected to a histogram equalization process to obtain a processed eye region image.

5. The method according to claim 1, characterized in that The eye contour coordinate information includes at least: the left boundary point coordinates, the right boundary point coordinates, the upper boundary point coordinates of the upper eyelid, and the lower boundary point coordinates of the lower eyelid of the target eye.

6. The method according to claim 1, characterized in that The steps of obtaining the driver's facial area image include: Collect driver images through the vehicle-mounted camera; Extracting a first region of interest in the driver image; wherein the first region of interest includes a facial region of the driver; The first region of interest is reshaped to obtain a facial region image of the driver of a first specified size.

7. The method according to claim 1, characterized in that The step of determining the eye area image corresponding to the target eye according to the eye contour coordinate information comprises: Determining the minimum circumscribed rectangle of the target eye according to the eye contour coordinate information; extracting a second region of interest from the facial region image according to the minimum circumscribed rectangle; The second region of interest is processed to obtain an eye region image of a second specified size corresponding to the target eye.

8. A driver's eye state detection device, characterized in that: The device comprises: An acquisition module, used for acquiring a facial region image of a driver, wherein the driver wears glasses; A first output module is used to input the facial region image into a pre-trained face detection model, and output eye contour coordinate information of the target eye of the driver through the face detection model; wherein the target eye includes: the left eye and the right eye of the driver; A calculation module, used for calculating the eye state parameters of the target eye according to the eye contour coordinate information; A first determining module, configured to determine an eye region image corresponding to the target eye according to the eye contour coordinate information; A second output module is used to input the eye area image into a pre-trained eye state detection model, and output a first eye state of the target eye through the eye state detection model; wherein the first eye state includes: an open eye state, a closed eye state or a reflective state; The second determination module is used to determine the final eye state of the driver according to the eye state parameter of the target eye and the first eye state.

9. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the driver's eye state detection method described in any one of claims 1-7.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the driver's eye state detection method described in any one of claims 1-7.

Images