Eye state detection method and apparatus, electronic device, and storage medium

Abstract

The application provides an eye state detection method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a to-be-processed image, and acquiring a target region image from the to-be-processed image, wherein the target region image comprises an eye region image and an eye surrounding region image, the eye surrounding region image is used for representing an image in a set region around the eye, then the eye region image and the eye surrounding region image are identified to obtain an identification result of the target region image, finally, the detection result of the eye state in the to-be-processed image is determined according to the identification result. The method can acquire the eye region image and the eye surrounding region image regardless of whether the driver wears glasses or not, and the identification results of the two images are fused to determine the state of the eye in the to-be-processed image, so that the eye state of the driver can be accurately detected.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to an eye state detection method, an eye state detection device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Eye condition detection has a wide range of applications. For example, automatic driver fatigue warning systems determine whether a driver is fatigued by detecting their eye condition. In daily life, some drivers do not wear glasses while driving, while others do. Furthermore, the type of glasses worn by different drivers may vary; some wear prescription glasses, while others wear sunglasses. Regardless of whether a driver wears glasses, accurate detection of their eye condition is essential.

[0003] Therefore, accurately detecting the driver's eye condition, regardless of whether the driver wears glasses, is a problem that urgently needs to be solved. Summary of the Invention

[0004] The present invention aims to solve, to a certain extent, the technical problems in the related technologies.

[0005] Therefore, the first objective of this invention is to propose an eye state detection method that acquires an image of the eye region and an image of the periorbital region regardless of whether the driver is wearing glasses, and fuses the recognition results of these two images to determine the state of the eyes in the image to be processed, thereby accurately detecting the driver's eye state.

[0006] The second objective of this invention is to provide an eye condition detection device.

[0007] The third objective of this invention is to provide an electronic device.

[0008] The fourth objective of this invention is to provide a computer-readable storage medium.

[0009] The fifth objective of this invention is to provide a computer program product.

[0010] To achieve the above objectives, a first aspect of the present invention provides an eye state detection method, comprising the following steps: acquiring an image to be processed; acquiring a target region image in the image to be processed, wherein the target region image includes an eye region image and a peri-eye region image, the peri-eye region image being used to characterize an image within a defined area surrounding the eye; recognizing the eye region image and the peri-eye region image to obtain a recognition result of the target region image; and determining a detection result of the eye state in the image to be processed based on the recognition result.

[0011] According to the eye state detection method of the present invention, an image to be processed is first acquired, and a target region image including an eye region image and a peri-eye region image is obtained from the image to be processed. The peri-eye region image is used to represent the image within a defined area surrounding the eye. Then, the eye region image and the peri-eye region image are identified to obtain the identification result of the target region image. Finally, based on the identification result, the detection result of the eye state in the image to be processed is determined. Thus, regardless of whether the driver is wearing glasses, this method acquires an eye region image and a peri-eye region image, and fuses the identification results of these two images to determine the eye state in the image to be processed, thereby accurately detecting the driver's eye state.

[0012] In addition, the eye state detection method proposed in the first aspect of the present invention may also have the following additional technical features:

[0013] According to one embodiment of the present invention, the recognition result includes at least one of: eyes open, eyes closed, and unable to be recognized.

[0014] According to an embodiment of the present invention, when the recognition result is that the eyes are open or closed, determining the detection result of the eye state in the image to be processed based on the recognition result includes: when the recognition result is that at least one of the eyes is open, determining that the detection result is that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open state; when the recognition result is that all the eyes are closed, determining that the detection result is that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in a closed state.

[0015] According to one embodiment of the present invention, if the identification result is unidentifiable, the unidentifiable includes: the target area image is black or the target area image is white.

[0016] According to an embodiment of the present invention, when it is impossible to identify that the target area image is black, determining the detection result of the eye state in the image to be processed based on the identification result includes: when the identification result is that the target area image is black, determining that the detection result is that the eyes in the image to be processed are wearing infrared blocking glasses.

[0017] According to an embodiment of the present invention, when it is not possible to identify that the target area image is white, determining the detection result of the eye state in the image to be processed based on the identification result includes: when the identification result is that the target area image is white, determining that the detection result is that the eyes in the image to be processed are wearing non-infrared blocking glasses, and the mirror surface of the non-infrared blocking glasses has reflection.

[0018] According to one embodiment of the present invention, before obtaining the recognition result of the target region image, the method further includes: recognizing the eye region image and the periorbital region image to obtain at least one key point of each eye in the eye region image; wherein, determining the detection result of the eye state in the image to be processed based on the recognition result includes: determining the opening degree of each eye based on at least one key point of each eye in the eye region image; and determining the detection result of the eye state in the image to be processed based on the recognition result of each eye and the opening degree of each eye.

[0019] According to one embodiment of the present invention, determining the detection result of the eye state in the image to be processed based on the recognition result of each eye and the opening degree of each eye includes: when the recognition result is that at least one eye is open, or the opening degree of at least one eye is greater than a set opening degree, determining that the detection result is that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open state; when the recognition result is that all eyes are closed, and the opening degree of each eye is less than or equal to a set opening degree, determining that the detection result is that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in a closed state.

[0020] According to an embodiment of the present invention, obtaining the target region image in the image to be processed includes: performing face detection on the image to be processed to obtain a face region image in the image to be processed; performing facial key point detection on the face region image to obtain at least one facial key point in the face region image; and determining the target region image in the image to be processed based on at least one of the facial key points.

[0021] According to an embodiment of the present invention, determining the target region image in the image to be processed based on at least one of the facial key points includes: determining the position information of the eye region and the position information of the peri-eye region in the image to be processed based on the at least one of the facial key points; and cropping the image to be processed according to the position information of the eye region and the position information of the peri-eye region to obtain the target region image.

[0022] To achieve the above objectives, a second aspect of the present invention provides an eye state detection device, comprising: a first acquisition module for acquiring an image to be processed; a second acquisition module for acquiring a target region image in the image to be processed, wherein the target region image includes an eye region image and a peri-eye region image, the peri-eye region image being used to characterize an image within a defined area surrounding the eye; an identification module for identifying the target region image to obtain an identification result of the target region image; and a determination module for determining a detection result of the eye state in the image to be processed based on the identification result.

[0023] According to an embodiment of the present invention, an eye state detection device acquires an image to be processed through a first acquisition module and acquires a target region image in the image to be processed through a second acquisition module. The target region image includes an eye region image and a peri-eye region image. The peri-eye region image represents the image within a defined area surrounding the eye. A recognition module identifies the target region image to obtain a recognition result. A determination module determines the detection result of the eye state in the image to be processed based on the recognition result. Therefore, regardless of whether the driver is wearing glasses, the device acquires both the eye region image and the peri-eye region image and fuses the recognition results of these two images to determine the eye state in the image to be processed, thereby accurately detecting the driver's eye state.

[0024] In addition, the eye condition detection device proposed in the second aspect embodiment of the present invention may also have the following additional technical features:

[0025] According to one embodiment of the present invention, the recognition result includes at least one of: eyes open, eyes closed, and unable to be recognized.

[0026] According to an embodiment of the present invention, when the recognition module obtains the recognition result of the target area image as either open or closed eyes, the determining module includes: a first determining unit, configured to determine, when the recognition result is that at least one of the eyes is open, that the detection result is that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open eye state; and a second determining unit, configured to determine, when the recognition result is that all the eyes are closed, that the detection result is that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in a closed eye state.

[0027] According to one embodiment of the present invention, when the recognition module obtains a recognition result of "unrecognizable" for the target region image, the "unrecognizable" condition includes: the target region image being black or the target region image being white.

[0028] According to an embodiment of the present invention, when the target area image cannot be identified as black, the determining module includes: a third determining unit, configured to determine, when the identification result is that the target area image is black, that the detection result is that the eyes in the image to be processed are wearing infrared blocking glasses.

[0029] According to an embodiment of the present invention, when the target area image cannot be identified as white, the determining module includes: a fourth determining unit, configured to determine, when the identification result is that the target area image is white, that the detection result is that the eyes in the image to be processed are wearing non-infrared blocking glasses, and the mirror surface of the non-infrared blocking glasses has reflection.

[0030] According to an embodiment of the present invention, when the recognition module obtains the recognition result of the target region image as open or closed eyes, the recognition module is further configured to: recognize the eye region image and the periorbital region image to obtain at least one key point of each eye in the eye region image; wherein, the determining module includes: a fifth determining unit, configured to determine the opening degree of each eye based on at least one key point of each eye in the eye region image; and a sixth determining unit, configured to determine the detection result of the eye state in the image to be processed based on the recognition result of each eye and the opening degree of each eye.

[0031] According to an embodiment of the present invention, the sixth determining unit includes: a first determining subunit, configured to determine, when the identification result is that at least one of the eyes is open, or the opening degree of at least one of the eyes is greater than a set opening degree, that the detection result is that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are in an open state; and a second determining subunit, configured to determine, when the identification result is that all the eyes are closed and the opening degree of each of the eyes is less than or equal to a set opening degree, that the detection result is that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are in a closed state.

[0032] According to an embodiment of the present invention, the second acquisition module includes: a first acquisition unit, configured to perform face detection on the image to be processed to acquire a face region image in the image to be processed; a second acquisition unit, configured to perform facial key point detection on the face region image to acquire at least one facial key point in the face region image; and a seventh determination unit, configured to determine a target region image in the image to be processed based on at least one of the facial key points.

[0033] According to an embodiment of the present invention, the seventh determining unit includes: a third determining subunit, configured to determine the position information of the eye region and the position information of the peri-eye region in the image to be processed based on the at least one of the facial key points; and a processing subunit, configured to perform cropping processing on the image to be processed according to the position information of the eye region and the position information of the peri-eye region to obtain the target region image.

[0034] To achieve the above objectives, a third aspect of the present invention provides an electronic device comprising: a processor and a memory; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, for implementing the eye state detection method of the first aspect embodiment.

[0035] The electronic device of this invention executes the above-described eye state detection method, acquiring images of the eye region and the periorbital region regardless of whether the driver is wearing glasses, and fusing the recognition results of these two images to determine the state of the eyes in the image to be processed, thereby accurately detecting the driver's eye state.

[0036] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the eye state detection method of the first aspect embodiment.

[0037] The computer-readable storage medium of this invention, by executing the above-described eye state detection method, acquires an image of the eye region and an image of the periorbital region regardless of whether the driver is wearing glasses, and fuses the recognition results of these two images to determine the state of the eyes in the image to be processed, thereby accurately detecting the driver's eye state.

[0038] To achieve the above objectives, a fifth aspect of the present invention provides a computer program product that, when executed by an instruction processor, performs the eye state detection method of the first aspect of the present invention.

[0039] The computer program product of this invention, by executing the above-described eye state detection method, acquires images of the eye region and the periorbital region regardless of whether the driver is wearing glasses, and fuses the recognition results of these two images to determine the state of the eyes in the image to be processed, thereby accurately detecting the driver's eye state.

[0040] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0041] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0042] Figure 1 This is a flowchart of an eye state detection method according to an embodiment of the present invention;

[0043] Figure 2 This is a flowchart of an eye state detection method according to an embodiment of the present invention;

[0044] Figure 3 This is a flowchart of an eye state detection method according to another embodiment of the present invention;

[0045] Figure 4 This is a flowchart of an eye state detection method according to yet another embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of an eye condition detection device according to an embodiment of the present invention;

[0047] Figure 6 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0048] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0049] The eye condition detection method, eye condition detection device, electronic device, and computer-readable storage medium of the present invention are described below with reference to the accompanying drawings.

[0050] Figure 1 This is a flowchart of an eye condition detection method according to an embodiment of the present invention.

[0051] It should be noted that the execution subject of this embodiment of the invention is an eye state detection device, which can be configured in an electronic device so that the electronic device can perform the function of eye state detection.

[0052] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0053] like Figure 1 As shown, the eye state detection method of this invention includes the following steps:

[0054] S101, Obtain the image to be processed.

[0055] In this embodiment, the image to be processed can be acquired by an image acquisition device (such as a camera). The image acquisition device can be set in multiple locations inside the vehicle, such as on the rearview mirror or in front of each seat. The specific location is not limited in this embodiment. Those skilled in the art can choose the location inside the vehicle where the driver's image can be captured to set the image acquisition device according to the actual situation.

[0056] It should be noted that the image acquisition device can operate continuously, meaning it can continuously acquire images of the driver inside the vehicle. Alternatively, it can operate only under certain conditions, such as when the driver manually activates the eye state detection function. This embodiment does not impose specific limitations on this, and those skilled in the art can make reasonable adjustments based on actual circumstances.

[0057] S102, acquire the target region image in the image to be processed, wherein the target region image includes the eye region image and the peri-eye region image, and the peri-eye region image is used to represent the image within a defined area around the eye.

[0058] The defined area can be set according to actual needs. For example, the target area image can be a rectangular area including the eyes and the area around the eyes, or it can be an area of ​​other shapes such as a circular area; there is no specific limitation here.

[0059] As a method for obtaining target region images, a multi-task convolutional neural network model is used to locate key points in the acquired face image to be processed, resulting in facial key points (e.g., 68 key points). These key points represent the positions of the eyes, facial contours, mouth, nose tip, nose root, eyebrows, etc. It should be noted that when the driver is wearing infrared blocking glasses, the key points for the eye position cannot be identified. In this case, the target region image in the image to be processed can be obtained by identifying key points such as the outermost positions of the left and right sides of the face, the nose tip or nose root, and the eyebrows. For example, during the training of the multi-task convolutional neural network model, the data types trained can include: face images without glasses, face images with regular glasses (including face images with regular sunglasses and regular myopia glasses, etc., which are not infrared blocking glasses), face images with infrared blocking glasses, and face images with reflective lenses, to improve the robustness of the multi-task convolutional neural network model for face detection in these situations.

[0060] S103, Recognize the image of the eye region and the image of the periorbital region to obtain the recognition result of the target region image.

[0061] Understandably, a multi-task convolutional neural network model includes a proposal network (P-Net), a refinement network (R-Net), and an output network (O-Net). Before using these network models, the target region image is preprocessed by scaling it to different sizes, forming an "image pyramid." Calculations are performed on each size image to detect the target region at different dimensions. Then, the target region images of different sizes are input into the proposal network, followed by the target region image output from the proposal network into the refinement network. The target region image output from the refinement network is then obtained. Finally, the target region image is input into the output network, which recognizes the target region image and outputs the recognition result. The recognition result can be at least one of the following: eyes open, eyes closed, or unrecognizable. When the driver is wearing non-infrared blocking glasses, their eyes are visible under infrared light, and it can be directly determined whether their eyes are open or closed by looking at the target area image. When wearing infrared blocking glasses, the target area image appears completely black, and in this case, it is classified as unrecognizable. When the lens of the glasses reflects light, it will make the eyes invisible, and the target area image should appear white. In this case, it is also classified as unrecognizable.

[0062] S104. Based on the recognition results, determine the detection results of the eye state in the image to be processed.

[0063] The following explanation of step S104 will be based on different recognition results.

[0064] The recognition result is: eyes open or eyes closed.

[0065] When the recognition result is that the left eye is open and the right eye is closed, it is determined that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open state; when the recognition result is that the right eye is open and the left eye is closed, it is determined that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open state; when the recognition result is that both the right and left eyes are open, it is determined that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open state; when the recognition result is that both the right and left eyes are closed, it is determined that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open state.

[0066] The recognition result is: Unable to recognize.

[0067] In cases where the target area image is either black or white, the recognition result can be classified as unrecognizable.

[0068] When the target area image is black, it indicates that the eyes in the image are wearing infrared blocking glasses. In this case, a first warning message needs to be issued to the driver. This message reminds the driver to change or remove the glasses, preventing the automatic fatigue driving warning system from failing to detect the driver's eyes, especially when the driver's eyes are closed, thus improving driving safety. When the target area image is white, it indicates that the eyes in the image are wearing non-infrared blocking glasses, and the lenses of these glasses are reflective. In this case, a second warning message can also be issued to the driver. This message reminds the driver to move their body or turn their head slightly to avoid light irritating the driver's eyes and causing them to be unable to see the road, thus improving driving safety.

[0069] Therefore, the eye state detection method according to embodiments of the present invention first acquires an image to be processed, and then acquires a target region image from the image to be processed, which includes an eye region image and a peri-eye region image. The peri-eye region image is used to represent the image within a defined area surrounding the eye. Then, the eye region image and the peri-eye region image are identified to obtain the identification result of the target region image. Finally, based on the identification result, the detection result of the eye state in the image to be processed is determined. Thus, regardless of whether the driver is wearing glasses, this method acquires an eye region image and a peri-eye region image, and fuses the identification results of these two images to determine the eye state in the image to be processed, thereby accurately detecting the driver's eye state.

[0070] Figure 2 This is a flowchart of an eye condition detection method according to an embodiment of the present invention.

[0071] like Figure 2 As shown, the eye state detection method of this invention includes:

[0072] S201, Obtain the image to be processed.

[0073] S202, Obtain the target region image in the image to be processed, wherein the target region image includes the eye region image and the peri-eye region image, and the peri-eye region image is used to represent the image within a defined area around the eye.

[0074] It should be noted that for details not disclosed in steps S201 and S202, please refer to the details disclosed in steps S102 and S202 of the above embodiments, which will not be repeated here.

[0075] S203, Recognize the image of the eye region and the image of the periorbital region to obtain the recognition result of the target region image.

[0076] The recognition result includes at least one of the following: eyes open, eyes closed, and unrecognizable. Unrecognizable includes situations where the target area image appears black or white.

[0077] Different recognition results correspond to different detection results, specifically including steps S204-S207.

[0078] S204, if the recognition result is that at least one eye is open, determine that the detection result is that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open state.

[0079] For example, if the recognition result is that the left eye is open and the right eye is closed, then the detection result is determined to be that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are in an open state; if the recognition result is that the right eye is open and the left eye is closed, then the detection result is determined to be that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are in an open state; if the recognition result is that the right eye is open and the left eye is open, then the detection result is determined to be that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are in an open state.

[0080] S205, if the recognition result is that all eyes are closed, determine that the detection result is that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are in a closed state.

[0081] In other words, when the recognition result is that both the right and left eyes are closed, the detection result is determined to be that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are in a closed state.

[0082] S206, if the recognition result is that the target area image is black, determine that the detection result is that the eyes in the image to be processed are wearing infrared blocking glasses.

[0083] S207, if the recognition result is that the target area image appears white, determine that the detection result is that the eyes in the image to be processed are wearing non-infrared blocking glasses, and the mirror surface of the non-infrared blocking glasses reflects light.

[0084] Therefore, the eye state detection method of this invention acquires images of the eye region and the periocular region regardless of whether the driver is wearing glasses. It then fuses the recognition results of these two images—open eyes, closed eyes, or unrecognizable (target region image appears black or white)—to determine the eye state in the image to be processed. This allows for accurate detection of the driver's eye state. Specifically, if at least one eye is open, the detection result indicates that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open state; if all eyes are closed, the detection result indicates that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in a closed state; if the target region image appears black, the detection result indicates that the eye in the image to be processed is wearing infrared blocking glasses; if the target region image appears white, the detection result indicates that the eye in the image to be processed is wearing non-infrared blocking glasses, and the lenses of the non-infrared blocking glasses reflect light. Furthermore, this method has high robustness in detecting eye state and low detection difficulty.

[0085] It is understandable that some drivers with smaller eyes may misinterpret the detection result of the eye state in the image to be processed as closed eyes. Therefore, in order to improve the accuracy of the detection result of the eye state in the image to be processed, before obtaining the recognition result of the target area image, the method further includes: recognizing the eye area image and the periorbital area image to obtain at least one key point of each eye in the eye area image. At this time, the specific process of the above step S104 can be as follows: Figure 3 As shown, it includes:

[0086] S301, determine the opening degree of each eye based on at least one key point of each eye in the eye region image.

[0087] Specifically, the eye opening and closing degree of the left eye can be calculated based on the coordinates of the highest position of the upper eyelid and the lowest position of the lower eyelid; the eye opening and closing degree of the right eye can be calculated based on the coordinates of the highest position of the upper eyelid and the lowest position of the lower eyelid.

[0088] S302, Based on the recognition results of each eye and the opening and closing degree of each eye, determine the detection results of the eye state in the image to be processed.

[0089] Specifically, if the recognition result is that at least one eye is open, or the opening degree of at least one eye is greater than the set opening degree, then the detection result is determined to be that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are in an open state; if the recognition result is that all eyes are closed, and the opening degree of each eye is less than or equal to the set opening degree, then the detection result is determined to be that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are in a closed state.

[0090] In other words, if the recognition result shows the left eye is open and the right eye is closed, the detection result is determined to be that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open state; if the recognition result shows the right eye is open and the left eye is closed, the detection result is determined to be that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open state; if the recognition result shows the right eye is open and the left eye is open, the detection result is determined to be that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open state; if the recognition result shows the closure degree of the left eye is greater than the set opening degree, the detection result is determined to be that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses. And the eyes are open; if the recognition result is that the closure degree of the right eye is greater than the set opening degree, it is determined that the detection result is that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are open; if the recognition result is that the closure degree of the left eye is greater than the set opening degree and the closure degree of the right eye is greater than the set opening degree, it is determined that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are open; if the recognition result is that the right eye is closed and the left eye is closed, and the closure degree of the left eye is greater than the set opening degree and the closure degree of the right eye is greater than the set opening degree, it is determined that the detection result is that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are open.

[0091] To improve the accuracy of acquiring the target region image in the image to be processed, the specific execution process of step S102 above can be as follows: Figure 4 As shown, it includes:

[0092] S401, Perform face detection on the image to be processed to obtain face region images in the image to be processed.

[0093] In this step, the image to be processed can be acquired through an image acquisition device (such as a camera). The image to be processed can be an image that includes a face or an image that does not include a face.

[0094] Then, the image to be processed is input into the face detection model to determine the position and size of the face, and the face region image output by the face detection model is obtained.

[0095] It should be noted that if face detection is performed on the image to be processed and no face region image is obtained, then subsequent processing of the image to be processed will be stopped.

[0096] S402, Perform facial landmark detection on the face region image to obtain at least one facial landmark in the face region image.

[0097] In this step, facial key points are detected based on the facial region image, such as the eyes, nose tip, corner of the mouth, eyebrows, and contour points of various facial components, to obtain a set of facial feature points.

[0098] S403, determine the target region image in the image to be processed based on at least one facial key point.

[0099] In this step: based on at least one facial key point, the position information of the left eye region, the right eye region, the periorbital region of the left eye, and the periorbital region of the right eye in the image to be processed are determined. Then, according to the obtained position information, the image to be processed is cropped to obtain the target region image.

[0100] In summary, the eye state detection method according to embodiments of the present invention first acquires an image to be processed, and then acquires a target region image from the image to be processed, which includes an eye region image and a peri-eye region image. The peri-eye region image is used to represent the image within a defined area surrounding the eye. Then, the eye region image and the peri-eye region image are identified to obtain the identification result of the target region image. Finally, based on the identification result, the detection result of the eye state in the image to be processed is determined. Therefore, regardless of whether the driver is wearing glasses, this method acquires an eye region image and a peri-eye region image, and fuses the identification results of these two images to determine the eye state in the image to be processed, thereby accurately detecting the driver's eye state.

[0101] Figure 5 This is a schematic diagram of an eye condition detection device according to an embodiment of the present invention.

[0102] like Figure 5 As shown, the eye state detection device 500 of this embodiment includes: a first acquisition module 501, a second acquisition module 502, an identification module 503, and a determination module 504.

[0103] The first acquisition module 501 is used to acquire the image to be processed. The second acquisition module 502 is used to acquire the target region image in the image to be processed, wherein the target region image includes an eye region image and a peri-eye region image, and the peri-eye region image is used to represent the image within a defined area surrounding the eye. The recognition module 503 is used to recognize the target region image to obtain the recognition result of the target region image. The determination module 504 is used to determine the detection result of the eye state in the image to be processed based on the recognition result.

[0104] According to one embodiment of the present invention, the recognition result includes at least one of: eyes open, eyes closed, and unable to be recognized.

[0105] According to an embodiment of the present invention, when the recognition module 503 obtains the recognition result of the target area image as open or closed eyes, the determination module 504 includes: a first determination unit, configured to determine that the detection result is that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open eye state when the recognition result is that at least one eye is open; and a second determination unit, configured to determine that the detection result is that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in a closed eye state when the recognition result is that all eyes are closed.

[0106] According to one embodiment of the present invention, if the recognition module 503 obtains a recognition result of "unrecognizable" for the target area image, "unrecognizable" includes: the target area image is black or the target area image is white.

[0107] According to an embodiment of the present invention, when it is impossible to identify that the target area image is black, the determination module 504 includes: a third determination unit, used to determine that the detection result is that the eyes in the image to be processed are wearing infrared blocking glasses when the identification result is that the target area image is black.

[0108] According to an embodiment of the present invention, when it is not possible to identify that the target area image is white, the determination module 504 includes: a fourth determination unit, used to determine, when the identification result is that the target area image is white, that the detection result is that the eyes in the image to be processed are wearing non-infrared blocking glasses and that the mirror surface of the non-infrared blocking glasses has reflection.

[0109] According to an embodiment of the present invention, when the recognition module 503 obtains the recognition result of the target area image as open or closed eyes, the recognition module 503 is further configured to: recognize the eye area image and the periorbital area image to obtain at least one key point of each eye in the eye area image; wherein, the determination module 504 includes: a fifth determination unit, configured to determine the opening degree of each eye based on at least one key point of each eye in the eye area image; and a sixth determination unit, configured to determine the detection result of the eye state in the image to be processed based on the recognition result of each eye and the opening degree of each eye.

[0110] According to an embodiment of the present invention, the sixth determining unit includes: a first determining subunit, configured to determine, when the identification result is that at least one eye is open, or the opening degree of at least one eye is greater than a set opening degree, that the detection result is that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in an open state; and a second determining subunit, configured to determine, when the identification result is that all eyes are closed, and the opening degree of each eye is less than or equal to a set opening degree, that the detection result is that the eye in the image to be processed is not wearing glasses or is wearing non-infrared blocking glasses and is in a closed state.

[0111] According to an embodiment of the present invention, the second acquisition module 502 includes: a first acquisition unit, configured to perform face detection on the image to be processed to acquire a face region image in the image to be processed; a second acquisition unit, configured to perform facial key point detection on the face region image to acquire at least one facial key point in the face region image; and a seventh determination unit, configured to determine a target region image in the image to be processed based on at least one facial key point.

[0112] According to an embodiment of the present invention, the seventh determining unit includes: a third determining subunit, configured to determine the position information of the eye region and the position information of the peri-eye region in the image to be processed based on at least one facial key point; and a processing subunit, configured to perform cropping processing on the image to be processed according to the position information of the eye region and the position information of the peri-eye region to obtain a target region image.

[0113] It should be noted that for details not disclosed in the eye state detection device of this embodiment, please refer to the details disclosed in the eye state detection method of this invention, which will not be repeated here.

[0114] According to an embodiment of the present invention, an eye state detection device acquires an image to be processed through a first acquisition module and acquires a target region image in the image to be processed through a second acquisition module. The target region image includes an eye region image and a peri-eye region image. The peri-eye region image represents the image within a defined area surrounding the eye. A recognition module identifies the target region image to obtain a recognition result. A determination module determines the detection result of the eye state in the image to be processed based on the recognition result. Therefore, regardless of whether the driver is wearing glasses, the device acquires both the eye region image and the peri-eye region image and fuses the recognition results of these two images to determine the eye state in the image to be processed, thereby accurately detecting the driver's eye state.

[0115] Based on the above embodiments, the present invention also proposes an electronic device.

[0116] The electronic device of this invention includes a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, so as to implement the above-described eye state detection method.

[0117] The electronic device of this invention, by executing the above-described eye state detection method, acquires an image of the eye region and an image of the periorbital region regardless of whether the driver is wearing glasses, and fuses the recognition results of these two images to determine the state of the eyes in the image to be processed, thereby accurately detecting the driver's eye state.

[0118] Based on the above embodiments, the present invention also proposes a computer-readable storage medium.

[0119] The computer-readable storage medium of this invention stores a computer program that, when executed by a processor, implements the above-described eye state detection method.

[0120] The computer-readable storage medium of this invention, by executing the above-described eye state detection method, acquires an image of the eye region and an image of the periorbital region regardless of whether the driver is wearing glasses, and fuses the recognition results of these two images to determine the state of the eyes in the image to be processed, thereby accurately detecting the driver's eye state.

[0121] Based on the above embodiments, the present invention also proposes a computer program product.

[0122] The computer program product of this invention executes the above-described eye state detection method when the instructions in the computer program product are executed by a processor.

[0123] The computer program product of this invention, by executing the above-described eye state detection method, acquires images of the eye region and the periorbital region regardless of whether the driver is wearing glasses, and fuses the recognition results of these two images to determine the state of the eyes in the image to be processed, thereby accurately detecting the driver's eye state.

[0124] Figure 6 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0125] like Figure 6 As shown, the electronic device 10 includes a processor 11, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 12 or a program loaded from a memory 16 into a random access memory (RAM) 13. The RAM 13 also stores various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0126] The following components are connected to I / O interface 15: memory 16 including hard disks, etc.; and communication section 17 including network interface cards such as LAN (Local Area Network) cards, modems, etc., which performs communication processing via a network such as the Internet; and driver 18 is also connected to I / O interface 15 as needed.

[0127] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 17. When the computer program is executed by the processor 11, the functions defined in the method of the present invention are performed.

[0128] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 16 including instructions, which can be executed by a processor 11 of an electronic device 10 to perform the above-described method. Optionally, the storage medium may be a computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0129] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0130] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0131] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0132] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0133] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0134] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0135] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a single module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0136] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for detecting eye condition, characterized in that, Includes the following steps: Obtain the image to be processed; Obtain a target region image from the image to be processed, wherein the target region image includes an eye region image and a peri-eye region image, and the peri-eye region image is used to characterize the image within a defined area around the eye; The eye region image and the peri-eye region image are identified using a multi-task convolutional neural network model to obtain the recognition result of the target region image. The data types used to train the multi-task convolutional neural network model include face images without glasses, face images with non-infrared blocking glasses, face images with infrared blocking glasses, and face images when the lenses of the glasses reflect light. The multi-task convolutional neural network model includes a selection network P-Net, an optimization network R-Net, and an output network O-Net. Based on the recognition results, the detection results of the eye state in the image to be processed are determined.

2. The method according to claim 1, characterized in that, The recognition result includes at least one of the following: eyes open, eyes closed, and unable to be recognized.

3. The method according to claim 2, characterized in that, When the recognition result is either open or closed eyes, determining the detection result of the eye state in the image to be processed based on the recognition result includes: If the recognition result is that at least one of the eyes is open, the detection result is determined to be that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are in an open state; If the recognition result is that all the eyes are closed, the detection result is determined to be that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are in a closed state.

4. The method according to claim 2, characterized in that, If the identification result is "unrecognizable", the "unrecognizable" includes: the target area image is black or the target area image is white.

5. The method according to claim 4, characterized in that, In the case where the target region image cannot be identified as black, determining the detection result of the eye state in the image to be processed based on the identification result includes: If the recognition result shows that the target area image is black, then the detection result indicates that the eyes in the image to be processed are wearing infrared blocking glasses.

6. The method according to claim 4, characterized in that, In the case where the target region image cannot be identified as white, determining the detection result of the eye state in the image to be processed based on the identification result includes: If the recognition result is that the target area image appears white, it is determined that the detection result is that the eyes in the image to be processed are wearing non-infrared blocking glasses, and the mirror surface of the non-infrared blocking glasses reflects light.

7. The method according to claim 3, characterized in that, Before obtaining the recognition result of the target region image, the method further includes: The eye region image and the periorbital region image are identified to obtain at least one key point for each eye in the eye region image; The step of determining the detection result of the eye state in the image to be processed based on the recognition result includes: Based on at least one key point of each eye in the eye region image, determine the opening degree of each eye; Based on the recognition results of each eye and the opening and closing degree of each eye, the detection result of the eye state in the image to be processed is determined.

8. The method according to claim 7, characterized in that, The step of determining the detection result of the eye state in the image to be processed based on the recognition result of each eye and the opening and closing degree of each eye includes: If the recognition result is that at least one of the eyes is open, or the opening degree of at least one of the eyes is greater than a set opening degree, then the detection result is determined to be that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are in an open state. If the recognition result is that all the eyes are closed and the opening degree of each eye is less than or equal to the set opening degree, then the detection result is determined to be that the eyes in the image to be processed are not wearing glasses or are wearing non-infrared blocking glasses and are in a closed state.

9. The method according to any one of claims 1-8, characterized in that, The step of obtaining the target region image in the image to be processed includes: Perform face detection on the image to be processed to obtain face region images in the image to be processed; Facial landmark detection is performed on the face region image to obtain at least one facial landmark in the face region image; The target region image in the image to be processed is determined based on at least one of the facial key points.

10. The method according to claim 9, characterized in that, The step of determining the target region image in the image to be processed based on at least one of the facial key points includes: Based on at least one of the facial key points, determine the positional information of the eye region and the positional information of the peri-eye region in the image to be processed; The image to be processed is cropped according to the location information of the eye region and the location information of the peri-eye region to obtain the target region image.

11. An eye condition detection device, characterized in that, include: The first acquisition module is used to acquire the image to be processed; The second acquisition module is used to acquire a target region image in the image to be processed, wherein the target region image includes an eye region image and a peri-eye region image, and the peri-eye region image is used to characterize the image within a defined area around the eye; The recognition module is used to recognize the eye region image and the peri-eye region image according to a multi-task convolutional neural network model to obtain the recognition result of the target region image; wherein, the data types for training the multi-task convolutional neural network model include face images without glasses, face images with non-infrared blocking glasses, face images with infrared blocking glasses, and face images when the lenses of the glasses reflect light; the multi-task convolutional neural network model includes a selection network P-Net, an optimization network R-Net, and an output network O-Net; The determination module is used to determine the detection result of the eye state in the image to be processed based on the recognition result.

12. An electronic device, characterized in that, include: Processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the eye state detection method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the eye state detection method as described in any one of claims 1-10.

Images