Eye opening and closing detection method, eye opening and closing detection device, and computer program product

Abstract

The eye opening and closing detection device of the present invention obtains a first image including the face of a person photographed by a camera device; generates a second image including the person's eye area from the first image; binarizes the second image and generates a third image in which pixels with grayscale values ​​less than a threshold are represented by a first brightness value and pixels with grayscale values ​​above the threshold are represented by a second brightness value; judges whether the person's eyes are in an open state or a closed state based on the height of the third image and the maximum height at which the longitudinal distance between the upper pixel and the lower pixel in the first brightness area with the first brightness value is the maximum; and outputs information related to the judgment result.

Description

Technical Field

[0001] The present invention relates to a technology for detecting the open or closed state of a person's eyes from an image. Background Art

[0002] Technology for detecting the openness or closedness of a person's eyes from images is gaining increasing attention as a foundational technology for inferring a person's emotions or state (for example, their level of wakefulness or eye fatigue). The following documents are known examples of technologies for detecting the openness or closedness of a person's eyes.

[0003] For example, the eye open / close judgment device of Patent Document 1 comprises: an edge detection unit that performs edge detection of an eye area based on an input image; a binarization processing unit that performs binarization processing on the eye area that has undergone edge detection; and an eye open / close judgment unit that performs an eye open / close judgment based on data of the eye area that has undergone binarization processing.

[0004] Moreover, for example, the image processing device of Patent Document 2 includes: an eye area determination unit that determines an eye area including eyes in a camera image including a face; an eye open / close determination unit that determines whether the eyes are open or closed; and an image processing unit that performs different processing on the eye area depending on whether the eyes are open or closed.

[0005] However, the above-mentioned prior art has low accuracy in detecting the open or closed state of a person's eyes, and further improvement is needed.

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Laid-Open No. 2008-84109

[0009] Patent Document 2: Japanese Patent Application Laid-Open No. 2016-9306 Summary of the Invention

[0010] The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a technology that can improve the accuracy of detecting the open or closed state of a person's eyes.

[0011] An embodiment of the present invention relates to an eye opening and closing detection method, which allows a computer to execute the following steps: obtaining a first image including the face of a person photographed by a camera device; generating a second image including the eye area of ​​the person from the first image; binarizing the second image and generating a third image in which pixels having grayscale values ​​less than a threshold are represented by a first brightness value and pixels having grayscale values ​​above the threshold are represented by a second brightness value; judging whether the eyes of the person are in an open state or a closed state based on the height of the third image and the maximum height at which the longitudinal distance between the upper pixel and the lower pixel in the first brightness area having the first brightness value is the maximum; and outputting information related to the judgment result.

[0012] According to the present invention, the accuracy of detecting the open or closed state of a person's eyes can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is an external view of the eye opening and closing detection system according to the first embodiment of the present invention.

[0014] Figure 2 This is a block diagram showing an example of the overall configuration of the eye opening and closing detection system according to the first embodiment of the present invention.

[0015] Figure 3 This is a flowchart showing an example of eye opening and closing detection processing by the eye opening and closing detection device according to the first embodiment of the present invention.

[0016] Figure 4 Yes Figure 3 This is a flowchart of an example of the eye state determination process of step S6.

[0017] Figure 5 Schematic diagram showing an example of a face area detected from a face image.

[0018] Figure 6 Schematic diagram showing an example of an eye detection area corresponding to the left eye.

[0019] Figure 7 Schematic diagram showing an example of an eye detection area corresponding to the right eye.

[0020] Figure 8 Schematic diagram showing an example of a binary image corresponding to the left eye.

[0021] Figure 9 Schematic diagram showing an example of a binary image corresponding to the right eye.

[0022] Figure 10 This is a schematic diagram showing an example of a display screen displayed on a monitor.

[0023] Figure 11 This is a flowchart showing an example of eye state determination processing according to the first modification of the first embodiment.

[0024] Figure 12 This is a flowchart showing an example of eye state determination processing according to the second modification of the first embodiment.

[0025] Figure 13 This is a block diagram showing an example of the overall configuration of an eye opening and closing detection system according to a second embodiment of the present invention.

[0026] Figure 14 This is a flowchart showing an example of eye state determination processing according to the second embodiment.

[0027] Figure 15 This is a schematic diagram showing an example of an eye detection area when the eyes are open.

[0028] Figure 16 FIG. 1 is a schematic diagram showing an example of an eye detection area when the eyes are closed.

[0029] Figure 17 This is a block diagram showing an example of the overall configuration of an eye opening and closing detection system according to a third embodiment of the present invention.

[0030] Figure 18 This is a flowchart showing an example of eye state determination processing according to the third embodiment.

[0031] Figure 19 This is a schematic diagram showing an example of an RGB image, an HSV image, and a binary image when the eyes are open.

[0032] Figure 20 This is a schematic diagram showing an example of an RGB image, an HSV image, and a binary image when the eyes are closed.

[0033] Figure 21 This is a block diagram showing an example of the overall configuration of an eye opening and closing detection system according to a fourth embodiment of the present invention.

[0034] Figure 22 This is a flowchart showing an example of processing by the eye opening and closing detection device according to the fourth embodiment of the present invention.

[0035] Figure 23 is a schematic diagram showing a binary image before performing the morphological gradient operation.

[0036] Figure 24 Schematic diagrams showing an expanded image and a contracted image obtained by performing expansion and contraction processing on a binary image.

[0037] Figure 25 is a schematic diagram showing a gradient image.

[0038] Figure 26 This is a block diagram showing an example of the overall configuration of an eye opening and closing detection system according to a fifth embodiment of the present invention.

[0039] Figure 27 This is a flowchart showing an example of processing of the eye opening and closing detection device according to the fifth embodiment of the present invention.

[0040] Figure 28 Schematic diagram showing a binary image in which the outer and inner canthi of the eyes are detected. DETAILED DESCRIPTION

[0041] Basic knowledge of the present invention

[0042] The eye opening and closing judgment in Patent Document 1 is to judge that the eye is open when the ratio of black pixels in the binarized pixels is greater than a threshold value, and to judge that the eye is closed when the ratio of black pixels is less than the threshold value.

[0043] Moreover, the judgment of whether the eyes are open or closed in the above-mentioned patent document 2 is that, in each of the right eye area and the left eye area, if the number of white pixels on one side after binarization is greater than a predetermined number of pixels, the eyes are judged to be open, and if the number of white pixels on one side after binarization is less than a predetermined number of pixels, the eyes are judged to be closed.

[0044] The size and shape of a person's eyes vary from person to person. Furthermore, the size and shape of a person's eyes may also change due to false eyelashes and makeup. Therefore, even if the conventional technology described above can detect the open or closed state of an eye based on the ratio of binarized pixels or the number of binarized pixels, it is difficult to detect the open or closed state of a person's eyes with high accuracy.

[0045] In order to solve the above-mentioned problem, an embodiment of the present invention relates to an eye opening and closing detection method, which allows a computer to execute the following steps: obtaining a first image including the face of a person photographed by a camera device; generating a second image including the eye area of ​​the person from the first image; binarizing the second image and generating a third image in which pixels with grayscale values ​​less than a threshold are represented by a first brightness value and pixels with grayscale values ​​above the threshold are represented by a second brightness value; judging whether the eyes of the person are in an open state or a closed state based on the maximum height at which the vertical distance between the upper pixel and the lower pixel in the first brightness area with the first brightness value is the largest, and outputting information related to the judgment result.

[0046] With this configuration, whether a person's eyes are open or closed is determined based on the height of a third image obtained by binarizing a second image including the eye region, and the maximum height at which the vertical distance between the upper and lower pixels within a first luminance region having a first luminance value is the largest. This reduces the influence of individual differences in eye size and shape, improving the accuracy of detecting the open or closed state of a person's eyes.

[0047] Moreover, the eye opening and closing detection method may also, in the judgment, judge whether the maximum height within the first brightness area is greater than the value obtained by multiplying the height of the third image by the first coefficient, and when it is judged that the maximum height within the first brightness area is greater than the value obtained by multiplying the height of the third image by the first coefficient, judge that the eyes are in an open state.

[0048] The ratio of the maximum height within the first brightness region to the height of the third image varies depending on whether the eyes are open or closed. When the eyes are open, the height of the third image is substantially the same as the maximum height within the first brightness region. Therefore, by comparing the maximum height within the first brightness region with the value obtained by multiplying the height of the third image by the first coefficient, it is possible to accurately determine whether the eyes are open.

[0049] Moreover, the eye opening and closing detection method may also, in the judgment, determine whether the maximum height within the first brightness area is less than the value obtained by multiplying the height of the third image by a second coefficient which is smaller than the first coefficient when the maximum height within the first brightness area is determined to be lower than the value obtained by multiplying the height of the third image by a first coefficient, and determine that the eyes are in a closed state when the maximum height within the first brightness area is determined to be less than the value obtained by multiplying the height of the third image by the second coefficient.

[0050] The ratio of the maximum height within the first brightness region to the height of the third image varies depending on whether the eyes are open or closed. The maximum height within the first brightness region is shorter when the eyes are closed than when the eyes are open. Therefore, by comparing the maximum height within the first brightness region with the value obtained by multiplying the height of the third image by a second coefficient that is smaller than the first coefficient, it is possible to accurately determine whether the eyes are closed.

[0051] Moreover, the eye opening and closing detection method may also generate a fourth image representing the facial area of ​​the person from the first image, and in the judgment, when it is judged that the maximum height within the first brightness area is greater than the value obtained by multiplying the height of the third image by the second coefficient, it is judged whether the maximum height within the first brightness area is less than the value obtained by multiplying the height of the fourth image by a third coefficient smaller than the second coefficient and the second coefficient. When the maximum height within the first brightness area is less than the value obtained by multiplying the height of the fourth image by the third coefficient and the second coefficient, it is judged that the eyes are in a closed state. When the maximum height within the first brightness area is greater than the value obtained by multiplying the height of the fourth image by the third coefficient and the second coefficient, it is judged that the eyes are in an open state.

[0052] According to this configuration, since not only the ratio of the maximum height within the first brightness area to the height of the third image including the human eye area is used, but also the ratio of the maximum height within the first brightness area to the height of the fourth image representing the human face area is used to determine whether the human eyes are in an open state or a closed state, the open or closed state of the eyes can be detected with higher accuracy.

[0053] Moreover, the eye opening and closing detection method may also generate a fourth image representing the facial area of ​​the person from the first image, and in the judgment, when it is judged that the maximum height within the first brightness area is less than the value obtained by multiplying the height of the third image by the first coefficient, it is judged whether the maximum height within the first brightness area is less than the value obtained by multiplying the height of the fourth image by a second coefficient smaller than the first coefficient and a third coefficient smaller than the second coefficient; when it is judged that the maximum height within the first brightness area is less than the value obtained by multiplying the height of the fourth image by the second coefficient and the third coefficient, it is judged that the eyes are in a closed state; when the maximum height within the first brightness area is greater than the value obtained by multiplying the height of the fourth image by the second coefficient and the third coefficient, it is judged that the eyes are in an open state.

[0054] According to this configuration, since not only the ratio of the maximum height within the first brightness area to the height of the third image including the human eye area is used but also the ratio of the maximum height within the first brightness area to the height of the fourth image representing the human face area is used to determine whether the human eyes are in a closed state, even if the third image cannot be correctly detected, the open or closed state of the eyes can be detected with high precision.

[0055] Moreover, the eye opening and closing detection method may also be that the second image is rectangular, and in the judgment, when it is judged that the maximum height within the first brightness area is greater than the value after multiplying the height of the third image by the second coefficient, based on the ratio of the height to the horizontal width of the second image, it is judged whether the person's eyes are in an open state or a closed state.

[0056] The height of a closed eye tends to be shorter than that of an open eye. Therefore, even if the height of the third image and the maximum height within the first luminance region cannot determine whether the eye is open or closed, the aspect ratio of the second image containing the eye region can be used to determine whether the eye is open or closed, thereby improving the accuracy of detecting whether the eye is open or closed.

[0057] Moreover, the eye opening and closing detection method may also be that the second image is represented by a red component, a green component and a blue component, and when it is judged that the maximum height within the first brightness area is greater than the value after multiplying the height of the third image by the second coefficient, a fifth image represented by a hue component, a saturation component and a lightness component is generated from the second image, and the hue component of the fifth image is binarized to generate a sixth image in which pixels having grayscale values ​​less than a threshold are represented by a third brightness value and pixels having grayscale values ​​greater than a threshold are represented by a fourth brightness value, and based on the third brightness area having the third brightness value, it is judged whether the person's eyes are in an open state or a closed state.

[0058] The difference between a visible light image of an open eye and a visible light image of a closed eye lies in whether light is reflected by the cornea. That is, light is reflected by the cornea of ​​an open eye, whereas a closed eye does not have a cornea. Light reflected by the cornea is prominently represented by the hue component in the color space represented by the hue component, saturation component, and lightness component. A feature quantity corresponding to the light reflected by the cornea is extracted by generating a sixth image obtained by binarizing the hue component of the fifth image represented by the hue component, saturation component, and lightness component. Therefore, even if the eye's open or closed state cannot be determined using the height of the third image and the maximum height within the first luminance region, the presence or absence of the feature quantity corresponding to the light reflected by the cornea in the sixth image obtained by binarizing the hue component of the fifth image represented by the hue component, saturation component, and lightness component can be used to determine the open or closed state of the eye, thereby improving the accuracy of detecting the open or closed state of the human eye.

[0059] Furthermore, the eye opening and closing detection method may further detect the position of the upper eyelid and the position of the lower eyelid based on the third image.

[0060] According to this configuration, since the positions of the upper eyelid and the lower eyelid are detected based on the third image, the person's emotions or state can be estimated based on the open or closed state of the eyes and the positions of the upper eyelid and the lower eyelid.

[0061] Furthermore, the eye opening and closing detection method may also be such that, when detecting the position of the upper eyelid and the position of the lower eyelid, the position of the upper eyelid and the position of the lower eyelid are detected by performing a morphological gradient operation on the third image.

[0062] According to this configuration, since the positions of the upper eyelid and the lower eyelid are detected by performing the morphological gradient operation on the binarized third image, the positions of the upper eyelid and the lower eyelid can be detected with high accuracy.

[0063] Furthermore, the eye opening and closing detection method may be such that the third image is a binary image of one of the left eye and the right eye of the person, and the positions of the outer corners and the inner corners of the eyes are detected based on the third image.

[0064] According to this configuration, since the positions of the outer and inner canthi are detected based on the third image, the person's emotions or state can be estimated based on the open or closed state of the eyes, the positions of the outer and inner canthi.

[0065] Moreover, the eye opening and closing detection method may be such that, when detecting the position of the outer canthus and the inner canthus, in the third image, the position of the pixel at the horizontal left end having the first brightness value is detected as the position of one of the outer canthus and the inner canthus, and the position of the pixel at the horizontal right end having the first brightness value is detected as the position of the other of the outer canthus and the inner canthus.

[0066] With this configuration, in the third image, which is a binary image for either the left or right eye, the position of the horizontal leftmost pixel having the first luminance value is detected as the position of one of the outer corner of the eye and the inner corner of the eye, and the position of the horizontal rightmost pixel having the first luminance value is detected as the position of the other of the outer corner of the eye and the inner corner of the eye. Therefore, the positions of the outer and inner corners of the eye can be easily detected.

[0067] Furthermore, the eye opening and closing detection method may further include superimposing information indicating whether the eyes are open or closed on the facial image of the person displayed on a display.

[0068] According to this configuration, since information indicating whether the eyes are open or closed is superimposed on the facial image of the person displayed on the display, the result of determining whether the eyes are open or closed can be displayed on the facial image in real time.

[0069] Furthermore, the present invention can be implemented not only as an eye opening and closing detection method that performs the characteristic processing described above, but also as an eye opening and closing detection device having a characteristic configuration corresponding to the characteristic method for performing the eye opening and closing detection method. Furthermore, it can be implemented as a computer program that causes a computer to execute the characteristic processing included in the eye opening and closing detection method. Therefore, the following other embodiments also have the same effects as the eye opening and closing detection method described above.

[0070] Another aspect of the present invention relates to an eye open and closed detection device including: an acquisition unit for acquiring a first image including a face of a person photographed by a camera device; an eye area detection unit for generating a second image including the eye area of ​​the person from the first image; a binarization processing unit for binarizing the second image and generating a third image in which pixels having grayscale values ​​less than a threshold are represented by a first brightness value and pixels having grayscale values ​​above the threshold are represented by a second brightness value; a judgment unit for judging whether the eyes of the person are in an open state or a closed state based on the height of the third image and the maximum height at which the longitudinal distance between the upper pixel and the lower pixel in the first brightness area having the first brightness value is the largest; and an output unit for outputting information related to the result of the judgment.

[0071] Another aspect of the present invention relates to an eye open and closed detection program, which causes a computer to perform the following processing: obtaining a first image including the face of a person photographed by a camera device; generating a second image including the eye area of ​​the person from the first image; binarizing the second image and generating a third image in which pixels having grayscale values ​​less than a threshold are represented by a first brightness value and pixels having grayscale values ​​above the threshold are represented by a second brightness value; judging whether the eyes of the person are open or closed based on the maximum height at which the vertical distance between the upper pixel and the lower pixel in the first brightness area having the first brightness value is the largest, and outputting information related to the judgment result.

[0072] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Note that the following embodiments are merely examples of the present invention and are not intended to limit the technical protection scope of the present invention.

[0073] First embodiment

[0074] Figure 1 This is an external view of the eye opening and closing detection system 100 according to the first embodiment of the present invention. Eye opening and closing detection system 100 is implemented using a portable terminal device such as a smartphone or tablet. However, this is merely an example; eye opening and closing detection system 100 can also be implemented by appropriately combining a desktop computer or cloud server with a camera and a display.

[0075] The eye opening and closing detection system 100 includes an eye opening and closing detection device 1, an imaging device 2, and a display 3. The eye opening and closing detection device 1 detects the opening and closing state of the eyes of a person U1 captured by the imaging device 2.

[0076] The imaging device 2 is composed of a camera mounted on a portable terminal device and is a camera capable of acquiring color visible light images at a predetermined frame rate.

[0077] Display 3 is composed of a display device such as a liquid crystal display device or an organic EL (ElectroLuminescence) display device incorporated into a portable terminal. Display 3 displays an image of the face of person U1 captured by imaging device 2. Furthermore, display 3 superimposes information on the eye opening and closing state detected by eye opening and closing detection device 1 on the image of person U1's face.

[0078] Figure 2 This is a block diagram showing an example of the overall configuration of an eye opening and closing detection system 100 according to the first embodiment of the present invention. The eye opening and closing detection device 1 includes a processor 10 and a memory 20. The processor 10 is, for example, a CPU (Central Processing Unit). The processor 10 includes an image acquisition unit 11, an eye region detection unit 12, a binarization unit 13, an eye state determination unit 14, and an output unit 15. The image acquisition unit 11, eye region detection unit 12, binarization unit 13, eye state determination unit 14, and output unit 15 are implemented, for example, by having the processor 10 execute an eye opening and closing detection program.

[0079] The image acquisition unit 11 acquires an image captured by the imaging device 2. Here, the acquired image includes the face of the person U1. Hereinafter, an image including a face is referred to as a facial image. The image acquisition unit 11 sequentially acquires the captured facial images at a predetermined frame rate. The facial image is an example of a first image including a person's face.

[0080] The eye region detection unit 12 generates an eye detection region that includes a person's eye region from a facial image. The eye region detection unit 12 detects a facial region representing a person's face from the facial image acquired by the image acquisition unit 11. The eye region detection unit 12 can detect the facial region by inputting the facial image into a classifier pre-created for facial region detection. This classifier, for example, is a Haar cascade classifier. The facial region is, for example, a rectangular region that encompasses the entire face. The facial region is an example of a region of the fourth image representing a person's face.

[0081] The eye region detection unit 12 inputs the face region into a classifier pre-created for detecting eye detection regions, thereby detecting eye detection regions. This classifier is, for example, a Haar cascade classifier. The eye detection region is a rectangular area whose top edge is adjacent to the upper eyelid, whose bottom edge is adjacent to the lower eyelid, whose left and right edges are adjacent to the inner corners of the eyes, and whose other edges are adjacent to the outer corners of the eyes. The eye region detection unit 12 detects an eye detection region including the left eye and an eye detection region including the right eye from the face region. The eye detection region is an example of a region of the second image that includes a person's eyes.

[0082] Hereinafter, the eye detection area including the left eye will be referred to as the "left eye detection area," and the eye detection area including the right eye will be referred to as the "right eye detection area." When the distinction between the two is not made, they are simply referred to as eye detection areas. Furthermore, the left eye refers to the eye on the left side when viewing person U1 from the front, and the right eye refers to the eye on the right side when viewing person U1 from the front. However, this is merely an example, and the relationship can also be reversed.

[0083] The binarization processing unit 13 binarizes the eye detection area to generate a binary image. In this binary image, pixels with grayscale values ​​less than a threshold are represented by a first brightness value, and pixels with grayscale values ​​above the threshold are represented by a second brightness value. A binary image is an example of a third image. When the eye detection area is composed of a color image, the binarization processing unit 13 can convert the eye detection area into a grayscale image having grayscale values ​​ranging from 0 to 255, for example, and perform binarization on the converted grayscale image. As the binarization process, for example, Otsu's binarization process can be used. The first brightness value is, for example, white, and the second brightness value is, for example, black. That is, in the first embodiment, a binary image is generated in which dark areas are represented by white and light areas are represented by black. The brightness value of white is represented by, for example, 255, and the brightness value of black is represented by, for example, 0.

[0084] Here, since the image captured by the camera 2 is quantized using 8 bits, the grayscale image has grayscale values ​​ranging from 0 to 255. However, this is merely an example. If the image captured by the camera 2 is quantized using another bit number, such as 16 bits, the grayscale image can have grayscale values ​​that can be expressed using that bit number.

[0085] The binarization processing unit 13 may generate binary images for each of the left eye detection region and the right eye detection region.

[0086] The eye state determination unit 14 determines whether the eyes of a person are open or closed based on the height of the binary image and the maximum height at which the vertical distance between the upper pixel and the lower pixel in the first brightness area having the first brightness value becomes the maximum.

[0087] Specifically, the eye state determination unit 14 determines whether the maximum height within the first luminance region is greater than the value obtained by multiplying the height of the binary image by a first coefficient. The first coefficient is, for example, 0.9. If the eye state determination unit 14 determines that the maximum height within the first luminance region is greater than the value obtained by multiplying the height of the binary image by the first coefficient, the eye state determination unit 14 determines that the eye is open.

[0088] Furthermore, if the maximum height within the first luminance region is determined to be less than the value obtained by multiplying the height of the binary image by the first coefficient, the eye state determination unit 14 determines whether the maximum height within the first luminance region is less than the value obtained by multiplying the height of the binary image by the second coefficient. The second coefficient is, for example, 0.6. If the maximum height within the first luminance region is determined to be less than the value obtained by multiplying the height of the binary image by the second coefficient, the eye state determination unit 14 determines that the eyes are closed.

[0089] The output unit 15 outputs information related to the determination result. The output unit 15 generates a display screen that superimposes information indicating the open or closed state of the eyes determined by the eye state determination unit 14 on the facial image acquired by the image acquisition unit 11 and displays the display screen on the display 3.

[0090] The memory 20 is a storage device such as RAM (Random Access Memory), SSD (Solid State Drive), or flash memory that can store various types of information. The memory 20 serves as a work area for the processor 10. Furthermore, the memory 20 may store information indicating the open or closed state of the eyes determined by the eye state determination unit 14 in association with the facial image acquired by the image acquisition unit 11.

[0091] The facial image associated with the information indicating the eye's open or closed state can be used as training data for machine learning to develop a recognition model for recognizing the eye's open or closed state from the facial image. Specifically, the output unit 15 can assign the information indicating the eye's open or closed state as a label to the facial image and store the labeled facial image in the memory 20.

[0092] Next, the eye opening and closing detection process of the eye opening and closing detection device 1 according to the first embodiment will be described.

[0093] Figure 3 This is a flowchart showing an example of the eye opening and closing detection process of the eye opening and closing detection device 1 according to the first embodiment of the present invention. Figure 3 The flowchart shown is executed at a predetermined sampling period. The predetermined sampling period is, for example, a frame period of the imaging device 2.

[0094] First, in step S1 , the image acquisition unit 11 acquires a facial image from the imaging device 2 .

[0095] Next, in step S2 , the eye region detecting unit 12 inputs the face image into a classifier for detecting a face region, and detects the face region.

[0096] Figure 5 FIG. 4 is a schematic diagram showing an example of a face region 41 detected from a face image 40. Figure 5 As shown in FIG. 4 , the eye region detecting unit 12 detects a rectangular region including the upper part of the forehead, the lower part of the chin, and the hairline of the ears as the face region 41. Here, the face region 41 does not include the entire hair, but may also include the entire hair. Figure 5 In FIG. 4 , the facial image 40 is an image of the person U1 photographed from the front, and therefore includes both the left eye and the right eye. Figure 5 As shown, the left eye of person U1 is open and the right eye of person U1 is closed.

[0097] Next, in step S3, the eye region detecting unit 12 inputs the face region 41 extracted in step S2 into the classifier for detecting the eye detection region to detect the eye detection region. Figure 5 As shown, the eye area detection unit 12 detects rectangular areas including the entire eyes as eye detection areas 421 and 422. The eye detection area 421 corresponds to the left eye, and the eye detection area 422 corresponds to the right eye.

[0098] Figure 6 is a schematic diagram showing an example of an eye detection area 421 corresponding to the left eye. Figure 74 is a schematic diagram showing an example of the eye detection area 422 corresponding to the right eye. Figure 6 As shown, the eye detection area 421 is a rectangular area including the entire open left eye. Figure 7 As shown, the eye detection area 422 is a rectangular area that includes the entire closed right eye. Figure 5 The facial region 41 shown is extracted with Figure 6 The eye detection area 421 corresponding to the left eye and Figure 7 The eye detection area 422 corresponding to the right eye is shown.

[0099] In the open-eye detection area 421, the upper edge is connected to the upper eyelid, the lower edge is connected to the lower eyelid, the right side is connected to the inner corner of the eye, and the left side is connected to the outer corner of the eye. In the closed-eye detection area 422, the upper edge is connected to the upper eyelid, the lower edge is connected to the tip of the eyelashes, the right side is connected to the outer corner of the eye, and the left side is connected to the inner corner of the eye.

[0100] Next, in step S4, the binarization processing unit 13 converts the eye detection areas 421 and 422 detected in step S3 into grayscale images. For example, the conversion process to grayscale images may include calculating the average grayscale values ​​of the red, green, and blue components of each pixel constituting the eye detection areas 421 and 422. However, this is merely an example, and other conversion processes may also be employed.

[0101] Next, in step S5 , the binarization processing unit 13 binarizes the eye detection areas 421 and 422 converted into grayscale images to generate binary images 431 and 432 .

[0102] Figure 8 is a schematic diagram showing an example of a binary image 431 corresponding to the left eye. Figure 9 1 is a schematic diagram showing an example of a binary image 432 corresponding to the right eye. The binarization processing unit 13 binarizes the eye detection areas 421 and 422 converted into grayscale images to generate binary images 431 and 432.

[0103] The left eye is open. Figure 8 For example, a binary image 431 is generated in the eye detection area 421 corresponding to the left eye, and pixels of the pupil, eyelashes, and a dark part of the white part of the eye are represented by white pixels in the binary image 431, and pixels of the other part of the white part of the eye and the bright part of the skin are represented by black pixels. Figure 8 In the example of , the binary image 431 includes a white area 51 composed of white pixels and a black area 52 composed of black pixels.

[0104] Close your right eye. Figure 9 For example, a binary image 432 is generated in the eye detection area 422 corresponding to the right eye, in which white pixels represent the darker pixels of the eyelashes and the edge of the upper eyelid, and black pixels represent the brighter pixels of the skin. Figure 9 For example, the binary image 432 includes a white area 61 composed of white pixels and a black area 62 composed of black pixels.

[0105] Next, in step S6, the eye state determination unit 14 performs an eye state determination process to determine whether the eyes of the person are open or closed based on the height of the binary image and the maximum height at which the longitudinal distance between the upper pixel and the lower pixel in the first brightness area having the first brightness value becomes the maximum. The eye state determination unit 14 determines whether the eyes of the person are open or closed for the left eye detection area and the right eye detection area, respectively. In addition, regarding the eye state determination process, the eye state determination unit 14 will use the image data to determine whether the eyes of the person are open or closed. Figure 4 This will be explained later.

[0106] Next, in step S7, the output unit 15 outputs the judgment result information indicating the judgment result of the eye open / closed state determined by the eye state determination unit 14. The output unit 15 generates a display screen and causes the display 3 to display the display screen. The display screen superimposes the judgment result information of the eye open / closed state determined in step S6 on the face image acquired in step S1. Figure 3 According to the process, a display screen in which judgment result information indicating the judgment result of the open or closed state of the eyes is superimposed on the facial image is displayed on the display 3 in real time.

[0107] Then, Figure 3 The eye state determination process of step S6 will be described.

[0108] Figure 4 Yes Figure 3 This is a flowchart of an example of the eye state determination process of step S6.

[0109] First, in step S11, the eye state determination unit 14 calculates the maximum height at which the vertical distance between the upper pixel and the lower pixel in the white area having the first luminance value becomes the maximum. Figure 8In the white area 51 of the binary image 431 shown, the height of each X-coordinate of the white area 51 is calculated by counting the number of white pixels on the Y-axis at each X-coordinate. The height is expressed as the number of white pixels in the vertical direction. The eye state determination unit 14 calculates the height at each X-coordinate while sequentially moving the X-coordinate from the left end to the right end of the white area 51 one pixel at a time. The eye state determination unit 14 calculates the X-coordinate with the maximum height within the white area 51 and the height at that X-coordinate.

[0110] In addition, when the eye detection area 421 corresponding to the left eye and the eye detection area 422 corresponding to the right eye are detected, the eye state determination process is performed on the binary images 431 and 432 corresponding to the left eye and the right eye, respectively. Figure 9 In the white area 61 of the binary image 432 shown, the height of each X coordinate of the white area 56 is also calculated by counting the number of white pixels on the Y axis at each X coordinate. The eye state determination unit 14 calculates the height at each X coordinate while sequentially moving the X coordinate from the left end to the right end of the white area 61 one pixel at a time. The eye state determination unit 14 calculates the X coordinate with the maximum height within the white area 61 and the height at that X coordinate.

[0111] Next, in step S12 , the eye state determination unit 14 determines whether the maximum height of the white area is greater than a value obtained by multiplying the height of the binary image of the eye detection area by 0.9.

[0112] Here, if it is determined that the maximum height of the white area is greater than the value obtained by multiplying the height of the binary image of the eye detection area by 0.9 ("Yes" in step S12), the eye state determination unit 14 determines in step S13 that the eyes are open. Figure 8 As shown, when the eyes are open, the ratio of the maximum height 54 of the white area 51 to the height 53 of the binary image 431 of the eye detection area 421 is close to 1. Therefore, when the maximum height 54 of the white area 51 is larger than the value obtained by multiplying the height of the binary image 431 of the eye detection area 421 by 0.9, the eye state determination unit 14 determines that the eyes in the eye detection area 421 are open.

[0113] On the other hand, when it is determined that the maximum height of the white area is less than the value obtained by multiplying the height of the binary image of the eye detection area by 0.9 ("No" in step S12), in step S14, the eye state determination unit 14 determines whether the maximum height of the white area is less than the value obtained by multiplying the height of the binary image of the eye detection area by 0.6.

[0114] Here, if it is determined that the maximum height of the white area is less than the value obtained by multiplying the height of the binary image of the eye detection area by 0.6 ("Yes" in step S14), the eye state determination unit 14 determines that the eyes are closed in step S15. Figure 9 As shown, when the eyes are closed, the maximum height 64 of the white area 61 is smaller than 60% of the height 63 of the binary image 432 of the eye detection area 422. Therefore, when the maximum height 64 of the white area 61 is smaller than the value obtained by multiplying the height of the binary image 432 of the eye detection area 422 by 0.6, the eye state determination unit 14 determines that the eyes in the eye detection area 422 are closed.

[0115] On the other hand, when it is determined that the maximum height of the white area is greater than or equal to the value obtained by multiplying the height of the binary image of the eye detection area by 0.6 (No in step S14), the eye state determination unit 14 determines in step S16 that the open or closed state of the eye cannot be determined.

[0116] When the eye state determination unit 14 determines that the eye opening or closing state cannot be determined, the output unit 15 may output determination result information indicating that the eye opening or closing state cannot be determined, or may not output determination result information.

[0117] Furthermore, in the first embodiment, when the answer to step S12 is “No”, the eye state determination unit 14 may determine that the eyes are closed without performing the process of step S14 .

[0118] Furthermore, in the first embodiment, after performing step S11, the eye state determination unit 14 may not perform step S12 but instead determine whether the maximum height of the white area is less than the value obtained by multiplying the height of the binary image of the eye detection area by 0.6. Furthermore, if the maximum height of the white area is determined to be less than the value obtained by multiplying the height of the binary image of the eye detection area by 0.6, the eye state determination unit 14 may determine that the eyes are closed. Furthermore, if the maximum height of the white area is determined to be greater than or equal to the value obtained by multiplying the height of the binary image of the eye detection area by 0.6, the eye state determination unit 14 may determine that the eyes are open.

[0119] Figure 10 It is a schematic diagram showing an example of a display screen displayed on the display 3 .

[0120] The output unit 15 displays the judgment result information indicating whether the eyes are open or closed by superimposing it on the face image of the person displayed on the display 3. Figure 10As shown, the display 3 displays the judgment result information 441 and 442 in a superimposed manner on the facial image 40. The judgment result information 441 indicates that the eyes are open, and is displayed near the frame indicating the eye detection area 421.

[0121] For example, the text "OPEN" indicating that the eyes are open is displayed on display 3 as judgment result information 441. On the other hand, judgment result information 442 indicating that the eyes are closed is displayed near the frame representing eye detection area 422. For example, the text "CLOSE" indicating that the eyes are closed is displayed on display 3 as judgment result information 442. The method of presenting judgment result information 441 and 442 is merely an example. Display 3 may also display eye detection area 421 indicating open eyes and eye detection area 422 indicating closed eyes in different ways. For example, such a different method may involve changing color.

[0122] Furthermore, the eye opening and closing detection system 100 may further include a speaker. For example, when the eye state determination unit 14 determines that the eyes are closed, the output unit 15 may output sound from the speaker.

[0123] As described above, whether a person's eyes are open or closed is determined based on the height of the binary image obtained by binarizing the eye detection region including the person's eyes and the maximum height at which the vertical distance between the upper and lower pixels within the first luminance region having the first luminance value is maximized. This reduces the influence of individual differences in eye size and shape, improving the accuracy of detecting the open or closed state of a person's eyes.

[0124] Next, an eye opening and closing detection system according to a first modified example of the first embodiment will be described.

[0125] In the first embodiment, if the eye detection area cannot be correctly detected, there is a possibility that even if the eyes are actually open, Figure 4 If the judgment result of step S12 is not "yes", it is not judged that the eyes are open in the processing after step S14. Here, in the first modification of the first embodiment, the height of the face area is also used to judge the open or closed state of the eyes.

[0126] The eye opening and closing detection system of the first modified example of the first embodiment has the same structure as the eye opening and closing detection system of the first embodiment. Figure 1 as well as Figure 2 The configuration of an eye opening and closing detection system according to a first modified example of the first embodiment will be described.

[0127] The eye state determination unit 14 of the first variant of the first embodiment, when determining that the maximum height within the first brightness area is greater than or equal to the value obtained by multiplying the height of the binary image by the second coefficient, determines whether the maximum height within the first brightness area is less than or equal to the value obtained by multiplying the height of the face area by a third coefficient smaller than the second coefficient and a second coefficient. Furthermore, the eye state determination unit 14 determines that the eyes are in a closed state if the maximum height within the first brightness area is less than or equal to the value obtained by multiplying the height of the face area by the third coefficient and the second coefficient. Furthermore, the eye state determination unit 14 determines that the eyes are in an open state if the maximum height within the first brightness area is greater than or equal to the value obtained by multiplying the height of the face area by the third coefficient and the second coefficient. The face area is an example of a fourth image representing an area of ​​a human face. The second coefficient is, for example, 0.6. The third coefficient is, for example, 0.15.

[0128] The value obtained by multiplying the height of the facial region by 0.15 can be said to correspond to the vertical length of the eyes, i.e., the eye height. Here, the eye state determination unit 14 determines whether the maximum height within the first luminance region is less than the value obtained by multiplying the height of the facial region by the third coefficient and the second coefficient. The third coefficient is not limited to 0.15; it can be any value that represents the ratio of the eye height to the height of the facial region.

[0129] Next, the eye state determination process of the first modified example of the first embodiment will be described. In the first modified example of the first embodiment, the eye open / close detection process other than the eye state determination process is Figure 3 The eye opening and closing detection process is the same as that of the first embodiment shown.

[0130] Figure 11 This is a flowchart showing an example of eye state determination processing according to the first modification of the first embodiment.

[0131] Because the processing of steps S21 to S25 is the same as Figure 4 The processing from step S11 to step S15 is the same, so the description thereof is omitted.

[0132] When it is determined that the maximum height of the white area is greater than the value obtained by multiplying the height of the binary image of the eye detection area by 0.6 ("No" in step S24), in step S26, the eye state judgment unit 14 determines whether the maximum height of the white area is less than the value obtained by multiplying the height of the face area by 0.15 and 0.6.

[0133] If the maximum height of the white area is determined to be smaller than the value obtained by multiplying the height of the face area by 0.15 and 0.6 (YES in step S26 ), the eye state determination unit 14 determines that the eyes are closed in step S27 .

[0134] On the other hand, if the maximum height of the white area is determined to be greater than or equal to the value obtained by multiplying the height of the face area by 0.15 and 0.6 (No in step S26 ), the eye state determination unit 14 determines in step S28 that the eyes are open.

[0135] As described above, in the first variant of the first embodiment, since the open or closed state of the eyes is determined not only by the ratio of the maximum height of the white area to the height of the binary image of the eye detection area but also by the ratio of the maximum height of the white area to the height of the face area, the open or closed state of the eyes can be detected with higher accuracy.

[0136] Next, an eye opening and closing detection system 100 according to a second modified example of the first embodiment will be described.

[0137] In the first embodiment, the height of the binary image of the eye detection area is used to determine whether the eyes are closed. In contrast, in the second variation of the first embodiment, the height of the face area is used to determine whether the eyes are closed.

[0138] The eye opening and closing detection system of the second modified example of the first embodiment has the same structure as the eye opening and closing detection system of the first embodiment. Figure 1 as well as Figure 2 The configuration of an eye opening and closing detection system according to a second modified example of the first embodiment will be described.

[0139] In the second variant of the first embodiment, the eye state determination unit 14, upon determining that the maximum height within the first brightness region is less than the value obtained by multiplying the height of the binary image by the first coefficient, determines whether the maximum height within the first brightness region is less than the value obtained by multiplying the height of the face region by a second coefficient that is smaller than the first coefficient and a third coefficient that is smaller than the second coefficient. Furthermore, if the eye state determination unit 14 determines that the eyes are closed if the maximum height within the first brightness region is less than the value obtained by multiplying the height of the face region by the second coefficient and the third coefficient. Furthermore, if the eye state determination unit 14 determines that the maximum height within the first brightness region is greater than the value obtained by multiplying the height of the face region by the second coefficient and the third coefficient, it determines that the eyes are open. The face region is an example of a region of the fourth image representing a human face. The second coefficient is, for example, 0.6. The third coefficient is, for example, 0.15.

[0140] Next, the eye state determination process of the second modified example of the first embodiment will be described. In the second modified example of the first embodiment, the eye open / close detection process other than the eye state determination process is Figure 3 The eye opening and closing detection process is the same as that of the first embodiment shown.

[0141] Figure 12 This is a flowchart showing an example of eye state determination processing according to the second modification of the first embodiment.

[0142] Because the processing of steps S31 to S33 is the same as Figure 4 The processing from step S11 to step S13 is the same, so the description thereof is omitted.

[0143] When it is determined that the maximum height of the white area is less than the value obtained by multiplying the height of the binary image of the eye detection area by 0.9 ("No" in step S32), in step S34, the eye state judgment unit 14 determines whether the maximum height of the white area is less than the value obtained by multiplying the height of the face area by 0.15 and 0.6.

[0144] Here, when it is determined that the maximum height of the white area is smaller than the value obtained by multiplying the height of the face area by 0.15 and 0.6 (YES in step S34 ), the eye state determination unit 14 determines that the eyes are closed in step S35 .

[0145] On the other hand, when it is determined that the maximum height of the white area is greater than or equal to the value obtained by multiplying the height of the face area by 0.15 and 0.6 (No in step S34 ), the eye state determination unit 14 determines in step S36 that the eyes are open.

[0146] As described above, in the second variant of the first embodiment, whether the eyes are closed is determined by using the ratio of the maximum height of the white area to the height of the face area instead of the ratio of the maximum height of the white area to the height of the binary image of the eye detection area. Therefore, even if the eye detection area cannot be correctly detected, the open or closed state of the eyes can be detected with high accuracy.

[0147] Second embodiment

[0148] In the first embodiment, if the maximum height of the white area is determined to be greater than or equal to the value obtained by multiplying the height of the binary image of the eye detection area by a second coefficient (e.g., 0.6), the eye state determination unit 14 determines that the eye open / close state cannot be determined. In contrast, in the second embodiment, if the maximum height of the white area is determined to be greater than or equal to the value obtained by multiplying the height of the binary image of the eye detection area by a second coefficient (e.g., 0.6), the eye open / close state is determined using the aspect ratio of the binary image.

[0149] Figure 13 1 is a block diagram showing an example of the overall configuration of an eye opening and closing detection system 100A according to a second embodiment of the present invention. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals and their descriptions are omitted.

[0150] The eye opening and closing detection device 1A includes a processor 10A and a memory 20. The processor 10A is similar to the processor 10 of the eye opening and closing detection device 1 according to the first embodiment, and further includes an aspect ratio determination unit 16.

[0151] When it is determined that the maximum height within the first brightness area is greater than the value obtained by multiplying the height of the binary image by the second coefficient, the aspect ratio determination unit 16 determines whether the human eyes are open or closed based on the aspect ratio (ratio of height to width) of the eye detection area.

[0152] Generally speaking, eyes in an open state and eyes in a closed state tend to have different aspect ratios. That is, the height of an eye in a closed state tends to be shorter than the height of an eye in an open state. The aspect ratio determination unit 16 determines whether an eye is open or closed by comparing the aspect ratio of the eye detection area with a threshold value.

[0153] More specifically, the aspect ratio determination unit 16 determines whether the aspect ratio of the eye detection area is greater than or equal to a threshold. The aspect ratio of the eye detection area is the value obtained by dividing the width (horizontal length) of the rectangular eye detection area by its height (vertical length). If the aspect ratio determination unit 16 determines that the eye detection area's aspect ratio is greater than or equal to the threshold, the eye is considered closed. Furthermore, if the aspect ratio determination unit 16 determines that the eye detection area's aspect ratio is less than the threshold, the eye is considered open.

[0154] In addition, in the second embodiment, the aspect ratio judgment unit 16 judges whether the human eyes are in an open state or a closed state based on the aspect ratio of the eye detection area, but the present invention is not limited to this. The aspect ratio judgment unit 16 may also judge whether the human eyes are in an open state or a closed state based on the aspect ratio of the binary image of the eye detection area.

[0155] Next, the eye state determination process of the second embodiment will be described. In the second embodiment, the eye open / close detection process other than the eye state determination process is Figure 3 The eye opening and closing detection process is the same as that of the first embodiment shown.

[0156] Figure 14 This is a flowchart showing an example of eye state determination processing according to the second embodiment.

[0157] Because the processing of steps S41 to S45 is the same as Figure 4 The processing from step S11 to step S15 is the same, so the description thereof is omitted.

[0158] When it is determined that the maximum height of the white area is greater than the value obtained by multiplying the height of the binary image of the eye detection area by 0.6 ("No" in step S44), in step S46, the aspect ratio determination unit 16 calculates the aspect ratio of the eye detection area by dividing the lateral width (horizontal length) of the eye detection area by the height (vertical length).

[0159] Figure 15 4 is a schematic diagram showing an example of an eye detection area 421 in a state where the eyes are open. Figure 16 Schematic diagram showing an example of the eye detection area 422 when the eyes are closed.

[0160] like Figure 15 as well as Figure 16 As shown, the eye detection areas 421 and 422 are rectangular. Figure 15 The eye detection area 421 shown includes eyes in an open state. Figure 16 The eye detection area 422 shown includes eyes in a closed state.

[0161] The horizontal width of eye detection area 421 is, for example, 263 pixels, and the height of eye detection area 421 is, for example, 80 pixels. The aspect ratio of eye detection area 421 is, for example, 3.29. Furthermore, the horizontal width of eye detection area 422 is, for example, 288 pixels, and the height of eye detection area 421 (should be 422) is, for example, 64 pixels. The aspect ratio of eye detection area 421 (should be 422) is, for example, 4.5.

[0162] As described above, the aspect ratio of the eye detection area 422 in the state where the eyes are closed is larger than the aspect ratio of the eye detection area 421 in the state where the eyes are open.

[0163] Return to Figure 14 Next, in step S47, the aspect ratio determination unit 16 determines whether the aspect ratio of the eye detection area is greater than or equal to a threshold. The threshold is, for example, 4.0. If the aspect ratio of the eye detection area is determined to be greater than or equal to the threshold ("YES" in step S47), the aspect ratio determination unit 16 determines in step S48 that the eyes are closed.

[0164] On the other hand, when it is determined that the aspect ratio of the eye detection area is smaller than the threshold value (No in step S47 ), the aspect ratio determination unit 16 determines in step S49 that the eyes are open.

[0165] In this way, even when the height of the binary image and the maximum height of the white area cannot be used to determine the open or closed state of the eyes, the aspect ratio of the eye detection area can be used to determine the open or closed state of the eyes, which can improve the detection accuracy of the open or closed state of a person's eyes.

[0166] Third embodiment

[0167] In the first embodiment, if the maximum height of the white area is determined to be greater than or equal to the value obtained by multiplying the height of the binary image of the eye detection area by a second coefficient (e.g., 0.6), the eye state determination unit 14 determines that the eye open / close state cannot be determined. In contrast, in the third embodiment, if the maximum height of the white area is determined to be greater than or equal to the value obtained by multiplying the height of the binary image of the eye detection area by a second coefficient (e.g., 0.6), the eye detection area represented by red, green, and blue components is converted into an HSV image represented by hue, saturation, and value components, and the eye open / close state is determined using the binary image of the hue component of the HSV image.

[0168] Figure 17 This is a block diagram showing an example of the overall configuration of an eye opening and closing detection system 100B according to a third embodiment of the present invention. In the second embodiment (should be the third embodiment), the same components as those in the first embodiment are denoted by the same reference numerals and their descriptions are omitted.

[0169] The eye opening and closing detection device 1B includes a processor 10B and a memory 20. The processor 10B further includes an HSV conversion unit 17, a binarization unit 18, and a white area determination unit 19, in addition to the processor 10 of the eye opening and closing detection device 1 according to the first embodiment.

[0170] When the eye state determination unit 14 determines that the maximum height within the first luminance region is greater than or equal to the value obtained by multiplying the height of the binary image by the second coefficient, the HSV conversion unit 17 generates an HSV image represented by hue, saturation, and value components from the eye detection region (RGB image) represented by red, green, and blue components. The HSV image is an example of a fifth image.

[0171] The HSV conversion unit 17 generates an HSV image from the eye detection area using a conversion formula for converting an RGB image into an HSV image. The conversion formula is conventional, so its description is omitted.

[0172] The binarization unit 18 binarizes the hue component of the HSV image, generating a binary image in which pixels with grayscale values ​​less than a threshold are represented by a third brightness value, and pixels with grayscale values ​​greater than the threshold are represented by a fourth brightness value. This binary image is an example of a sixth image. For example, the Otsu binarization process can be used as the binarization process. The binarization unit 18 performs binarization on the hue component of the HSV image. For example, the third brightness value represents white, and the fourth brightness value represents black. For example, the brightness value of white is represented by 255, and the brightness value of black is represented by 0.

[0173] The white area determination unit 19 determines whether the person's eyes are open or closed based on a third brightness area having a third brightness value. The third brightness value represents white. The white area determination unit 19 detects a white area having a plurality of continuous white pixels from the binary image. The white area determination unit 19 determines whether the person's eyes are open or closed based on the number or size of the white areas detected from the binary image. When the number of white areas detected from the binary image is greater than a threshold value, the white area determination unit 19 determines that the eyes are open. Furthermore, when the number of white areas detected from the binary image is less than a threshold value, the white area determination unit 19 determines that the eyes are closed.

[0174] Furthermore, the white area determination unit 19 may determine that the eyes are open when the number of pixels in the largest white area detected from the binary image is greater than a threshold. Furthermore, the white area determination unit 19 may determine that the eyes are closed when the number of pixels in the largest white area detected from the binary image is less than a threshold. Furthermore, the white area determination unit 19 may determine that the eyes are closed even when there is no white area in the binary image.

[0175] Next, the eye state determination process of the third embodiment will be described. In the third embodiment, the eye open / close detection process other than the eye state determination process is Figure 3 The eye opening and closing detection process is the same as that of the first embodiment shown.

[0176] Figure 18 This is a flowchart showing an example of eye state determination processing according to the third embodiment.

[0177] Because the processing of steps S61 to S65 is the same as Figure 4 The processing from step S11 to step S15 is the same, so the description thereof is omitted.

[0178] When it is determined that the maximum height of the white area is greater than or equal to the value obtained by multiplying the height of the binary image of the eye detection area by 0.6 (No in step S64), in step S66, the HSV conversion unit 17 generates an HSV image represented by the HSV color space from the eye detection area represented by the RGB color space.

[0179] Next, in step S67 , the binarization processing unit 18 binarizes the hue component of the HSV image to generate a binary image.

[0180] Next, in step S68, the white area determination unit 19 performs a labeling process on the binary image. In the labeling process, the same number is assigned to consecutive white pixels in the binary image. This labeling process allows the detection of multiple white areas consisting of consecutive white pixels in the binary image.

[0181] Figure 19 is a schematic diagram showing an example of an RGB image, an HSV image, and a binary image when the eyes are open. Figure 20 This is a schematic diagram showing an example of an RGB image, an HSV image, and a binary image in a state where the eyes are closed.

[0182] like Figure 19 As shown, in the eye detection area (RGB image) when the eyes are open, light is reflected into the cornea (black pupil). The difference between the RGB image of the open eyes and the RGB image of the closed eyes is whether there is light reflected into the cornea. That is, light is reflected into the cornea of ​​the open eyes, while there is no cornea in the closed eyes. The light reflected into the cornea is significantly expressed in the hue component of the HSV color space. The feature value equivalent to the light reflected into the cornea is extracted by generating an HSV image from the RGB image and generating a binary image of the hue component of the HSV image. As shown Figure 19 As shown in FIG, in the binary image of the hue component when the eyes are open, a white area corresponding to the light reflected on the cornea appears. Figure 20As shown, in the binary image of the hue component when the eye is closed, there is no white area corresponding to the light reflected by the cornea. The white area determination unit 19 can determine whether the eye is open or closed by counting the number of white areas in the binary image of the hue component.

[0183] Return to Figure 18 In step S69, the white area determination unit 19 determines whether the number of white areas is greater than a threshold value. If the number of white areas is determined to be greater than the threshold value ("Yes" in step S69), the white area determination unit 19 determines in step S70 that the eyes are open.

[0184] On the other hand, when it is determined that the number of white areas is smaller than the threshold value (No in step S69 ), the white area determination unit 19 determines in step S71 that the eyes are closed.

[0185] As described above, even when the height of the binary image of the eye detection area as an RGB image and the maximum height of the white area cannot be used to determine the open or closed state of the eyes, the open or closed state of the eyes can be determined by using a binary image generated based on the hue component of the HSV image of the eye detection area, thereby improving the detection accuracy of the open or closed state of a person's eyes.

[0186] In addition, the second embodiment and the third embodiment may be combined. Figure 14 In step S47, when it is determined that the aspect ratio of the eye detection area is smaller than the threshold value (“No” in step S47), the Figure 18 The processing of steps S66 to S71.

[0187] Fourth embodiment

[0188] In contrast to the first embodiment, which detects the open or closed state of the eyes, the fourth embodiment also detects the position of the upper eyelid and the position of the lower eyelid.

[0189] Figure 21 This is a block diagram showing an example of the overall configuration of an eye opening and closing detection system 100C according to a fourth embodiment of the present invention. The eye opening and closing detection system 100C in the fourth embodiment also detects the position of the upper eyelid and the position of the lower eyelid. In the fourth embodiment, components identical to those in the first embodiment are denoted by the same reference numerals, and their descriptions are omitted.

[0190] The processor 10C of the eye opening and closing detection device 1C further includes an iris detection processing unit 21 , an eyelid detection unit 22 , and a state estimation unit 24 , similar to the processor 10 of the eye opening and closing detection device 1 according to the first embodiment.

[0191] The iris detection processing unit 21 generates a binary image from the binary image generated by the binarization processing unit 13 by replacing pixels of a second luminance value that appear within a first luminance region having a first luminance value and that satisfy a predetermined condition with pixels of the first luminance value. Furthermore, the iris detection processing unit 21 calculates iris information including information related to at least one of the position and size of a person's iris using the binary image.

[0192] The eyelid detection unit 22 detects the positions of the upper and lower eyelids of the person U1 based on the binary image generated by the binarization unit 13. The eyelid detection unit 22 can detect the positions of the upper and lower eyelids by performing a morphological gradient operation on the binary image. The eyelid detection unit 22 detects the positions of the upper and lower eyelids for the binary image of the left eye and the binary image of the right eye, respectively.

[0193] The state estimation unit 24 estimates the state of the person based on the open / closed state of the person's eyes determined by the eye state determination unit 14 and the positions of the upper and lower eyelids detected by the eyelid detection unit 22. The state of the person is, for example, the person's level of wakefulness. For example, the state estimation unit 24 estimates the person's level of wakefulness based on the open / closed state of the eyes and the positions of the upper and lower eyelids.

[0194] The state of the person may be, for example, the degree of fatigue of the person. For example, the state estimation unit 24 may estimate the degree of fatigue of the person based on the open or closed state of the eyes and the position of the upper eyelid and the position of the lower eyelid.

[0195] Furthermore, the state estimation unit 24 may estimate the person's emotions based on the open or closed state of the person's eyes determined by the eye state determination unit 14 and the positions of the person's upper eyelids and lower eyelids detected by the eyelid detection unit 22 .

[0196] Next, the processing of the eye opening and closing detection device 1C according to the fourth embodiment will be described.

[0197] Figure 22 This is a flowchart showing an example of processing by the eye opening and closing detection device 1C according to the fourth embodiment of the present invention.

[0198] Because the processing of steps S81 to S86 is the same as Figure 3 The processing of steps S1 to S6 is the same, so their description is omitted.

[0199] Next, in step S87 , the iris detection processing unit 21 applies iris detection processing to the binary image generated by the binarization processing unit 13 to calculate iris information.

[0200] Here, the iris detection process performed by the iris detection processing unit 21 will be described.

[0201] First, the iris detection processing unit 21 divides the binary image into a plurality of local regions by separating the pixels in the X direction. For example, the iris detection processing unit 21 divides the binary image into ten equal parts in the horizontal direction. This results in the binary image being divided into ten strip-shaped local regions, each with its length in the Y direction. While the iris detection processing unit 21 divides the binary image into ten local regions, this is merely an example. The number of divisions may be an integer between 2 and 9, or an integer between 11 and 11. The Y direction refers to the longitudinal (vertical) direction of the image captured by the imaging device 2.

[0202] Next, the iris detection processing unit 21 calculates the average brightness value of each of the ten local areas.

[0203] Next, the iris detection processing unit 21 calculates the X coordinate of the estimated iris center position. The estimated iris center position is an estimate of the iris center position and is different from the final calculated iris center position. Due to the effects of double eyelids, thick eyelashes, false eyelashes, etc., these areas may appear exaggerated as white areas. In such cases, the white of the eye may be painted over. To avoid this, the estimated iris center position is calculated in this second embodiment (should be the fourth embodiment).

[0204] The iris detection processing unit 21 calculates the X-coordinate of the estimated iris center position based on the coordinates of the midpoint in the X direction of the local area with the highest average brightness value among the multiple local areas. Depending on the width of the local area in the X direction, the midpoint in the X direction of the local area may not be suitable as the X-coordinate of the estimated iris center position. In such cases, the left or right end of the local area in the X direction may be used as the X-coordinate of the estimated iris center position.

[0205] Next, the iris detection processing unit 21 calculates the Y coordinate of the estimated iris center position. The iris detection processing unit 21 detects the uppermost and lowermost white pixel endpoints in the local area where the estimated iris center position's X coordinate is located, and calculates the midpoint between the uppermost and lowermost endpoints as the Y coordinate of the estimated iris center position. Furthermore, due to the influence of eyelashes and makeup, the uppermost and lowermost endpoints may appear in the local area adjacent to the left or right. Alternatively, the iris detection processing unit 21 may calculate the uppermost and lowermost endpoints of the local area where the estimated iris center position's X coordinate is located and the two local areas adjacent to it on the left and right, average the three uppermost and three lowermost endpoints to obtain an average uppermost endpoint, and average the three lowermost endpoints to obtain an average lowermost endpoint, and calculate the midpoint between the average uppermost and average lowermost endpoints as the Y coordinate of the estimated iris center position.

[0206] Next, the iris detection processing unit 21 performs a color overlay process on the binary image. In visible light images, external light or background light may be reflected onto the cornea due to factors such as ambient brightness. When this reflection is significant, bright areas such as white appear within the dark or brown pupil. In this case, binarizing the eye image creates black islands within the pupil, making it difficult to accurately detect iris information. Therefore, the second embodiment (or fourth embodiment) performs a color overlay process to overlay these islands with black.

[0207] The details of the overlay process are as follows. First, the iris detection processing unit 21 sets a vertical line parallel to the Y direction for the X-coordinate of the estimated iris center position in the binary image. Next, the iris detection processing unit 21 detects the white pixel that first appears on the vertical line from the upper end of the binary image as the upper pixel. Next, the iris detection processing unit 21 detects the white pixel that first appears on the vertical line from the lower end of the binary image as the lower pixel. Next, the iris detection processing unit 21 determines whether the distance between the upper and lower pixels is greater than a first reference distance. Next, if the iris detection processing unit 21 determines that the distance between the upper and lower pixels is greater than the first reference distance, it determines that the black pixels located between the upper and lower pixels of the vertical line are black pixels that meet a predetermined condition and replaces these black pixels with white pixels. On the other hand, if the iris detection processing unit 21 determines that the distance between the upper and lower pixels is less than the first reference distance, it does not replace the vertical line. The first reference distance is, for example, an appropriate distance based on the assumed iris diameter.

[0208] The iris detection processing unit 21 performs this coating process on each vertical line within the range of the left reference distance from the estimated iris center position toward the left in the X direction, and also performs this coating process on each vertical line within the range of the right reference distance from the estimated iris center position toward the right in the X direction. The sum of the left reference distance range and the right reference distance range is an example of a second reference distance. The left reference distance range and the right reference distance range can be, for example, equal ranges. For example, a distance slightly larger than the assumed iris diameter is used as the second reference distance. This allows the coating process to be applied preferentially to vertical lines located in the pupil area.

[0209] Next, the iris detection processing unit 21 detects the left and right end pixels of the pupil area. In the white area of ​​the binary image, the iris detection processing unit 21 examines changes in brightness pixel by pixel in the X direction, starting from the estimated center position of the iris. Furthermore, the iris detection processing unit 21 detects the first black pixel that appears on the left side of the X direction as the left end pixel, and the first black pixel that appears on the right side of the X direction as the right end pixel.

[0210] Next, the iris detection processing unit 21 calculates the middle position between the left end pixel and the right end pixel as the X coordinate of the iris center position.

[0211] Next, the iris detection processing unit 21 detects the upper and lower pixels of the pupil area. In the white area of ​​the binary image, the iris detection processing unit 21 examines changes in brightness pixel by pixel in the Y direction, starting from the X coordinate of the iris center position. Furthermore, the iris detection processing unit 21 detects the black pixel that appears first in the upper Y direction as the upper pixel and the black pixel that appears first in the lower Y direction as the lower pixel.

[0212] Next, the iris detection processing unit 21 calculates the middle position between the upper pixel and the lower pixel as the Y coordinate of the iris center position. The iris center position is calculated through the above steps.

[0213] The above is a description of the iris detection process. The iris detection processing unit 21 calculates iris information including the iris center position.

[0214] Next, in step S88 , the eyelid detection unit 22 detects the position of the upper eyelid and the position of the lower eyelid by performing morphological gradient calculation.

[0215] Figure 23 Schematic diagram showing a binary image 70 before the morphological gradient operation is performed.

[0216] exist Figure 23In the example of , a binary image 70 is generated in the eye detection area, in which dark areas such as the pupil and eyelashes are represented in white, and bright areas such as the white of the eye and skin are represented in black. Figure 23 The white area D1 shown is composed of white pixels.

[0217] First, the eyelid detection unit 22 applies a dilation process to the binary image 70. Dilation replaces the pixel of interest with a white pixel if there is at least one white pixel near the pixel of interest. Next, the eyelid detection unit 22 applies an erosion process to the binary image 70. Erosion replaces the pixel of interest with a black pixel if there is at least one black pixel near the pixel of interest.

[0218] Figure 24 Schematic diagrams showing an expanded image 81 and a contracted image 82 obtained by performing expansion and contraction processing on a binary image 70 .

[0219] The white area D1 included in the expanded image 81 is expanded compared to the white area D1 included in the binary image 70 by the expansion process. The white area D1 included in the contracted image 82 is contracted compared to the white area D1 included in the binary image 70 by the contraction process.

[0220] Next, the eyelid detection unit 22 calculates a gradient image by subtracting the contracted image 82 from the expanded image 81 .

[0221] Figure 25 Schematic diagram showing the gradient image 83 . The gradient image 83 includes the edge E1 of the white area D1 of the binary image 70 .

[0222] Next, the eyelid detection unit 22 detects the uppermost position of edge E1 as the upper eyelid position P10. Alternatively, the eyelid detection unit 22 may detect the intersection of a vertical line passing through the iris center position P0 detected by the iris detection processing unit 21 and the upper edge E1 as the upper eyelid position P10. Furthermore, the eyelid detection unit 22 detects the intersection of a vertical line passing through the iris center position P0 detected by the iris detection processing unit 21 and the lower edge E1 as the lower eyelid position P11. This allows detection of the lower eyelid position, which exhibits a subtle brightness change and is difficult to clearly represent in the binary image 70. Alternatively, the eyelid detection unit 22 may detect the portion of the upper edge E1 connecting the left and right ends of the gradient image 83 as the upper eyelid line.

[0223] Alternatively, the eyelid detection unit 22 may detect the uppermost position of the edge E1 as the upper eyelid position P10, and detect the lower eyelid position P11 at the intersection of a straight line extending vertically downward from the detected upper eyelid position P10 and the lowermost edge E1. In this case, since detection of the iris center position P0 is unnecessary, the processing of step S87 can be omitted.

[0224] Next, in step S89, the state estimation unit 24 estimates the state of the person based on the open or closed state of the person's eyes determined by the eye state determination unit 14 and the positions of the person's upper and lower eyelids detected by the eyelid detection unit 22. At this time, the state estimation unit 24 calculates the distance between the upper and lower eyelid positions when the eyes are open. The state estimation unit 24 calculates the ratio of the distance between the upper and lower eyelid positions relative to the height of the eye detection area. Furthermore, if the calculated ratio is below a threshold, the state estimation unit 24 estimates that the person's level of wakefulness is relatively low. Furthermore, if the calculated ratio is above the threshold, the state estimation unit 24 estimates that the person's level of wakefulness is relatively high. Furthermore, if both eyes are closed, the state estimation unit 24 estimates that the person is asleep and not awake.

[0225] Furthermore, the state estimation unit 24 may estimate that the person's fatigue level is relatively high when the calculated ratio is below the threshold, and may estimate that the person's fatigue level is relatively low when the calculated ratio is above the threshold.

[0226] Next, in step S90, the output unit 15 generates a display screen by superimposing the estimation result of the state estimation unit 24 on the facial image calculated in step S1, and displays the display screen on the display 3. For example, the display screen may display the facial image and the state of the person (e.g., the degree of awakening). Alternatively, the output unit 15 may superimpose the result of determining whether the person's eyes are open or closed on the facial image and display the result on the display 3.

[0227] Alternatively, the state estimation unit 24 may estimate whether the upper eyelid is in a spasmodic state based on the time series data of the upper eyelid position P10. Specifically, the state estimation unit 24 estimates that the upper eyelid is in a spasmodic state if the upper eyelid position P10 moves up and down at predetermined intervals within a predetermined time.

[0228] In this way, by using the results of determining the open or closed state of the eyes and the results of detecting the positions of the upper eyelids and the lower eyelids, it is possible to estimate the state or emotion of a person.

[0229] Fifth embodiment

[0230] In contrast to the first embodiment, which detects the open or closed state of the eyes, the fifth embodiment further detects the positions of the outer canthus and the inner canthus of the eyes.

[0231] Figure 26 This is a block diagram showing an example of the overall configuration of an eye opening and closing detection system 100D according to a fifth embodiment of the present invention. The eye opening and closing detection device 1D in the fifth embodiment further detects the positions of the outer and inner canthi of the eyes. In the fifth embodiment, components identical to those in the first and fourth embodiments are denoted by the same reference numerals, and their descriptions are omitted.

[0232] The processor 10D of the eye opening and closing detection device 1D is similar to the processor 10 of the eye opening and closing detection device 1 according to the first embodiment, and further includes an iris detection processing unit 21 , an outer canthus and inner canthus detection unit 23 , and a state estimation unit 24D.

[0233] The outer and inner canthus detection unit 23 detects the positions of the outer and inner canthus of the eye, respectively, from the binary image generated by the binarization processing unit 13. The outer and inner canthus detection unit 23 detects the positions of the outer and inner canthus of the eye, respectively, based on the binary image of one of the left and right eyes. Here, the outer and inner canthus detection unit 23 detects the positions of the outer and inner canthus of the eye, respectively, for the binary image of the left eye and the binary image of the right eye.

[0234] The outer corner and inner corner detection unit 23 detects the position of the horizontal left end pixel having the first brightness value as the position of one of the outer corner and the inner corner of the eye in the binary image, and detects the position of the horizontal right end pixel having the first brightness value as the position of the other of the outer corner and the inner corner of the eye.

[0235] The state estimation unit 24D estimates the state of the person based on the open / closed state of the person's eyes determined by the eye state determination unit 14, the positions of the outer and inner canthi of the person's eyes detected by the outer and inner canthi of the eyes detection unit 23, and the center position of the iris detected by the iris detection processing unit 16 (should be 21). Examples of the state of the person include confusion, hesitation, or nervousness. For example, the state estimation unit 24D estimates whether the person is confused, hesitant, or nervous based on the open / closed state of the eyes, the positions of the outer and inner canthi of the eyes, and the center position of the iris.

[0236] Specifically, when a person is confused, shaken, or nervous, the number of times the eyes open and close (blink) increases compared to normal, and the pupils tend to move slightly left and right. State estimation unit 24D determines whether a person has blinked a predetermined number of times or more within a predetermined time period based on time-series data of the eye's opening and closing states.

[0237] Furthermore, the state estimation unit 24D calculates the distance between the iris center and the outer canthus or the distance between the iris center and the inner canthus. Based on the time-series data of the distance between the iris center and the outer canthus or the distance between the iris center and the inner canthus, the state estimation unit 24D determines whether the iris center has slightly moved left or right. Specifically, the state estimation unit 24D determines that the iris center has slightly moved left or right if the distance between the iris center and the outer canthus changes a predetermined number of times within a predetermined time period. Alternatively, the state estimation unit 24D may determine that the iris center has slightly moved left or right if the distance between the iris center and the inner canthus changes a predetermined number of times within a predetermined time period.

[0238] Furthermore, when the state estimation unit 24D determines that the person has blinked a predetermined number of times or more within a predetermined time and that the center position of the iris is moving slightly left or right, it estimates that the person is confused, shaken, or nervous.

[0239] In addition, the state estimation unit 24D can also estimate the person's emotions based on the open or closed state of the person's eyes determined by the eye state determination unit 14, the position of the person's outer canthus and inner canthus detected by the outer canthus and inner canthus detection unit 23, and the center position of the iris detected by the iris detection processing unit 16 (should be 21).

[0240] Next, the processing of the eye opening and closing detection device 1D according to the fifth embodiment will be described.

[0241] Figure 27 This is a flowchart showing an example of processing of the eye opening and closing detection device 1D according to the fifth embodiment of the present invention.

[0242] Because the processing of steps S101 to S106 is the same as Figure 3 The processing of steps S1 to S6 is the same, so their description is omitted.

[0243] Next, in step S107 , the outer canthus and inner canthus detecting unit 23 detects the positions of the outer canthus and the inner canthus.

[0244] Figure 28Schematic diagram showing a binary image 70 in which the positions of the outer canthus and the inner canthus of the eye are detected.

[0245] Since the frame of the binary image 70 is the circumscribed rectangle of the white area D1, in the binary image 70, the X coordinate of the left end of the white area D1 is the X coordinate of the left end of the binary image 70 (X11), and the X coordinate of the right end of the white area D1 is the X coordinate of the right end of the binary image 70 (X12). These two X coordinates have already been calculated when the binary image 70 is generated. Here, the outer corner and inner corner detection unit 23 calculates the position of the outer corner I2 and the position of the inner corner P13 using the two calculated X coordinates. In addition, since Figure 28 This is a binary image 70 for the left eye. The left end of the white area D1 is at the outer corner of the eye position P12, and the right end of the white area D1 is at the inner corner of the eye position P13.

[0246] Specifically, the outer and inner canthus detection unit 23 searches for white pixels, pixel by pixel, starting from the bottom end toward the top of the binary image 70 at the X coordinate (X11). Furthermore, the outer and inner canthus detection unit 23 determines the Y coordinate of the first white pixel detected as the Y coordinate of the outer canthus position P12. Similarly, the outer and inner canthus detection unit 23 searches for white pixels, pixel by pixel, starting from the bottom end toward the top of the binary image 70 at the X coordinate (X12). Furthermore, the outer and inner canthus detection unit 23 determines the Y coordinate of the first white pixel detected as the Y coordinate of the inner canthus position P13. In this manner, the outer canthus position P12 and the inner canthus position P13 are detected.

[0247] Because the processing of step S108 is the same as Figure 22 The processing of step S87 is the same as that of step S87, so its description is omitted.

[0248] Next, in step S109, the state estimation unit 24D estimates the state of the person based on the open or closed state of the person's eyes determined by the eye state determination unit 14, the position of the outer canthus and the position of the inner canthus of the person detected by the outer canthus and inner canthus detection unit 23, and the position of the center of the iris detected by the iris detection processing unit 16 (should be 21).

[0249] Next, in step S110, the output unit 15 generates a display screen by superimposing the estimation result of the state estimation unit 24D on the facial image calculated in step S1, and displays the display screen on the display 3. For example, the display screen may display the facial image and the estimation result of the person's state. Alternatively, the output unit 15 may superimpose the result of determining whether the person's eyes are open or closed on the facial image and display the result on the display 3.

[0250] As described above, the state or emotion of a person can be estimated using the judgment result of the open or closed state of the eyes, the detection result of the positions of the outer and inner canthi, and the detection result of the iris center position.

[0251] Furthermore, the eye opening and closing detection system 100D may further include an input device that performs input operations based on eye movements. For example, the input device may move a pointer displayed on the display 3 based on the distance between the center of the iris and the outer or inner canthus of the eye.

[0252] In each of the above-described embodiments, each component may be implemented using dedicated hardware or by executing a software program suitable for that component. Each component may also be implemented by having a program execution unit, such as a CPU or processor, read and execute a software program stored on a storage medium, such as a hard disk or semiconductor memory. Furthermore, the program may be stored and transferred on a storage medium or transferred via a network, allowing execution of the program on a separate independent computer system.

[0253] Part or all of the functions of the devices described in the embodiments of the present invention can typically be implemented as an integrated circuit (LSI). Part or all of these functions can be implemented as separate chips, or as a single chip containing part or all of them. Furthermore, integrated circuits are not limited to LSIs and can also be implemented using dedicated circuits or general-purpose processors. Alternatively, a programmable FPGA (Field Programmable Gate Array) or a reconfigurable processor capable of reconfiguring the connections and settings of circuit cells within the LSI can be utilized after LSI manufacturing.

[0254] Furthermore, part or all of the functions of the apparatus according to the embodiment of the present invention may be realized by causing a processor such as a CPU to execute a program.

[0255] Furthermore, the numbers used above are examples given to specifically illustrate the present invention, and the present invention is not limited to these exemplified numbers.

[0256] Furthermore, the order in which the steps shown in the flowcharts are executed is merely an example for the purpose of illustrating the present invention in detail. Other orders are possible while achieving the same results. Furthermore, some of the steps may be executed simultaneously (in parallel) with other steps.

[0257] Industrial applicability

[0258] The technology related to the present invention can improve the accuracy of detecting the open or closed state of a person's eyes and thus has practical value as a technology for detecting the open or closed state of a person's eyes from an image.

Claims

1. A method for detecting whether eyes are open or closed, characterized in that: Have the computer perform the following steps: acquiring a first image including a face of a person captured by a camera; generating a second image including an eye region of the person from the first image; Binarizing the second image to generate a third image, wherein pixels having grayscale values ​​less than a threshold are represented by first brightness values, and pixels having grayscale values ​​greater than the threshold are represented by second brightness values; determining whether the eyes of the person are open or closed based on a height of the third image and a maximum height at which a longitudinal distance between an upper pixel and a lower pixel in a first brightness region having the first brightness value is the largest; Output information related to the judgment result, In the judgment, it is determined whether the maximum height within the first brightness area is greater than the value obtained by multiplying the height of the third image by the first coefficient. When it is determined that the maximum height within the first brightness area is greater than the value obtained by multiplying the height of the third image by the first coefficient, it is determined that the eyes are in an open state.

2. The eye opening and closing detection method according to claim 1, characterized in that: In the judgment, when it is judged that the maximum height within the first brightness area is lower than the value obtained by multiplying the height of the third image by a first coefficient, it is judged whether the maximum height within the first brightness area is less than the value obtained by multiplying the height of the third image by a second coefficient which is smaller than the first coefficient; when it is judged that the maximum height within the first brightness area is less than the value obtained by multiplying the height of the third image by the second coefficient, it is judged that the eyes are in a closed state.

3. The eye opening and closing detection method according to claim 2, characterized in that: further generating a fourth image representing a facial region of the person from the first image, In the judgment, when it is judged that the maximum height within the first brightness area is greater than the value after multiplying the height of the third image by the second coefficient, it is judged whether the maximum height within the first brightness area is less than the value after multiplying the height of the fourth image by a third coefficient smaller than the second coefficient and the second coefficient; when the maximum height within the first brightness area is less than the value after multiplying the height of the fourth image by the third coefficient and the second coefficient, it is judged that the eyes are in a closed state; when the maximum height within the first brightness area is greater than the value after multiplying the height of the fourth image by the third coefficient and the second coefficient, it is judged that the eyes are in an open state.

4. The eye opening and closing detection method according to claim 1, wherein: further generating a fourth image representing a facial region of the person from the first image, In the judgment, when it is judged that the maximum height within the first brightness area is lower than the value after multiplying the height of the third image by the first coefficient, it is judged whether the maximum height within the first brightness area is smaller than the value after multiplying the height of the fourth image by a second coefficient smaller than the first coefficient and a third coefficient smaller than the second coefficient. When it is judged that the maximum height within the first brightness area is smaller than the value after multiplying the height of the fourth image by the second coefficient and the third coefficient, it is judged that the eyes are in a closed state. When the maximum height within the first brightness area is greater than the value after multiplying the height of the fourth image by the second coefficient and the third coefficient, it is judged that the eyes are in an open state.

5. The eye opening and closing detection method according to claim 2, characterized in that: The second image is rectangular, In the judgment, when it is judged that the maximum height within the first brightness area is greater than the value after multiplying the height of the third image by the second coefficient, based on the ratio of the height to the horizontal width of the second image, it is judged whether the person's eyes are in an open state or a closed state.

6. The eye opening and closing detection method according to claim 2, characterized in that: The second image is represented by a red component, a green component, and a blue component, When it is determined that the maximum height within the first luminance region is greater than or equal to the value obtained by multiplying the height of the third image by the second coefficient, a fifth image represented by a hue component, a saturation component, and a lightness component is further generated from the second image. The hue component of the fifth image is further binarized to generate a sixth image, in which pixels having grayscale values ​​less than a threshold are represented by a third brightness value, and pixels having grayscale values ​​greater than the threshold are represented by a fourth brightness value. Furthermore, based on the third brightness region having the third brightness value, it is determined whether the eyes of the person are in an open state or a closed state.

7. The eye opening and closing detection method according to any one of claims 1 to 6, characterized in that: The position of the upper eyelid and the position of the lower eyelid are also detected based on the third image.

8. The eye opening and closing detection method according to claim 7, characterized in that: When detecting the position of the upper eyelid and the position of the lower eyelid, the position of the upper eyelid and the position of the lower eyelid are detected by performing a morphological gradient operation on the third image.

9. The eye opening and closing detection method according to any one of claims 1 to 6, characterized in that: The third image is a binary image of one of the left eye and the right eye of the person, The positions of the outer canthus and the inner canthus are also detected based on the third image.

10. The eye opening and closing detection method according to claim 9, characterized in that: When detecting the position of the outer corner of the eye and the position of the inner corner of the eye, in the third image, the position of the pixel at the horizontal left end having the first brightness value is detected as the position of one of the outer corner of the eye and the inner corner of the eye, and the position of the pixel at the horizontal right end having the first brightness value is detected as the position of the other of the outer corner of the eye and the inner corner of the eye.

11. The eye opening and closing detection method according to any one of claims 1 to 6, characterized in that: Information indicating whether the eyes are open or closed is also superimposed and displayed on the face image of the person displayed on the display.

12. An eye opening and closing detection device, characterized in that include: an acquisition unit configured to acquire a first image including a face of a person captured by a camera; an eye region detecting unit configured to generate a second image including the eye region of the person from the first image; a binarization processing unit, binarizing the second image to generate a third image, wherein pixels having grayscale values ​​less than a threshold are represented by first brightness values, and pixels having grayscale values ​​greater than the threshold are represented by second brightness values; a determination unit that determines whether the eyes of the person are open or closed based on a height of the third image and a maximum height at which a longitudinal distance between an upper pixel and a lower pixel in a first luminance region having the first luminance value is maximum; and The output unit outputs information related to the judgment result. In the judgment, it is determined whether the maximum height within the first brightness area is greater than the value obtained by multiplying the height of the third image by the first coefficient. When it is determined that the maximum height within the first brightness area is greater than the value obtained by multiplying the height of the third image by the first coefficient, it is determined that the eyes are in an open state.

13. A computer program product comprising an eye opening and closing detection program, characterized in that: The eye opening and closing detection program causes the computer to perform the following processing: acquiring a first image including a face of a person captured by a camera; generating a second image including an eye region of the person from the first image; Binarizing the second image to generate a third image, wherein pixels having grayscale values ​​less than a threshold are represented by first brightness values, and pixels having grayscale values ​​greater than the threshold are represented by second brightness values; determining whether the eyes of the person are open or closed based on a height of the third image and a maximum height at which a longitudinal distance between an upper pixel and a lower pixel in a first brightness region having the first brightness value is the largest; Output information related to the judgment result, In the judgment, it is determined whether the maximum height within the first brightness area is greater than the value obtained by multiplying the height of the third image by the first coefficient. When it is determined that the maximum height within the first brightness area is greater than the value obtained by multiplying the height of the third image by the first coefficient, it is determined that the eyes are in an open state.

Images