Eye detection method, eye detection apparatus, and program product

Abstract

An eye detection device acquires a color image containing a face of a person by an imaging device; generates a gray scale image by multiplying a red component value, a green component value, and a blue component value by a prescribed ratio corresponding to a characteristic of a lens of a pair of glasses worn by the person for each pixel of the color image; detects an eye of the person from the gray scale image; and outputs eye information related to the detected eye.

Description

Technical Field

[0001] This invention relates to a technique for detecting human eyes from images. Background Technology

[0002] Eye detection technology, which identifies human eyes from images, is a fundamental technology required for gaze detection and personal verification.

[0003] In eye detection technology, when photographing a person wearing glasses, the reflection of specific wavelengths of light from the glasses lenses, or external light or background light entering the lenses, can introduce noise into the eye portion of the captured image, potentially reducing the accuracy of eye detection. Therefore, it is necessary to suppress external light or background light entering the glasses lenses.

[0004] For example, the imaging method in Patent Document 1 involves illuminating light into a person's eyeball at a certain angle of incidence, photographing an image of the eyeball in order to obtain an iris pattern through the reflected light, and judging the quality of the photographed image of the eyeball based on a predefined threshold. If the image is judged to be bad, light is illuminating the eyeball at another angle of incidence different from the certain angle of incidence, and the image of the eyeball is photographed again.

[0005] Moreover, for example, the photographic method in Patent Document 2 uses near-infrared light vibrating in the direction of a specified polarization axis to detect the opening of the eye without being affected by reflected light from the eyeglasses.

[0006] However, the existing technologies described above are insufficient to improve the accuracy of detecting human eyes from color images, and further improvements are needed.

[0007] Existing technical documents

[0008] Patent documents

[0009] Patent Document 1: Japanese Patent Publication No. 3337913

[0010] Patent Document 2: Japanese Patent Publication No. 2015-194884 Summary of the Invention

[0011] This invention was made to solve the above-mentioned problems, and its purpose is to provide a technique that can improve the accuracy of detecting human eyes from color images.

[0012] One aspect of the present invention relates to an eye detection method in which a computer: acquires a color image of a person's face captured by a camera device; generates a grayscale image for each pixel of the color image by multiplying the red component value, green component value, and blue component value by a predetermined ratio corresponding to the characteristics of the lens of the glasses worn by the person; detects the person's eyes from the grayscale image; and outputs eye information related to the detected eyes.

[0013] According to the present invention, the accuracy of detecting human eyes from color images can be improved. Attached Figure Description

[0014] Figure 1 This is an external view of the eye detection system according to the first embodiment of the present invention.

[0015] Figure 2 This is a block diagram illustrating an example of the overall configuration of the eye detection system according to the first embodiment of the present invention.

[0016] Figure 3 This is a flowchart illustrating an example of eye detection processing of the eye detection device according to the first embodiment of the present invention.

[0017] Figure 4 This is a schematic diagram illustrating an example of a color image captured by a camera device in the first embodiment.

[0018] Figure 5 It means from Figure 4 This is a schematic diagram of an example of a grayscale image generated from a color image.

[0019] Figure 6 This is a block diagram illustrating an example of the overall configuration of the eye detection system according to the second embodiment of the present invention.

[0020] Figure 7 This is a flowchart illustrating an example of the eye detection process of the eye detection device according to the second embodiment of the present invention.

[0021] Figure 8 This is a schematic diagram illustrating an example of a facial region detected by the eye detection unit.

[0022] Figure 9 This is a schematic diagram illustrating an example of the left eye region detected by the eye detection unit.

[0023] Figure 10 This is a schematic diagram illustrating an example of the right eye region detected by the eye detection unit.

[0024] Figure 11This is a block diagram illustrating an example of the overall configuration of the eye detection system according to the third embodiment of the present invention.

[0025] Figure 12 This is a flowchart illustrating an example of the eye detection process of the eye detection device according to the third embodiment of the present invention.

[0026] Figure 13 This is a schematic diagram illustrating an example of a binary image of the facial region in the third embodiment.

[0027] Figure 14 This is a schematic diagram illustrating an example of a binary image of the facial region in a variation of the third embodiment.

[0028] Figure 15 This is a block diagram illustrating an example of the overall configuration of the eye detection system according to the fourth embodiment of the present invention.

[0029] Figure 16 This is a flowchart illustrating an example of the eye detection process of the eye detection device according to the fourth embodiment of the present invention.

[0030] Figure 17 This is a schematic diagram illustrating an example of a facial region where the average luminance value is higher than a threshold in the fourth embodiment.

[0031] Figure 18 This is a schematic diagram illustrating an example of a facial region where the average brightness value is below a threshold in the fourth embodiment. Detailed Implementation

[0032] Basic knowledge of this invention

[0033] Patent Document 1 requires an illumination device for changing the incident angle of light illuminating the eyeball. Furthermore, Patent Document 2 requires an illumination device for illuminating near-infrared light vibrating in the direction of a predetermined polarization axis.

[0034] Any of the aforementioned existing technologies requires a special lighting device to suppress external light or background reflections into the lenses of the glasses. Therefore, it is difficult to improve the accuracy of detecting human eyes from images by using devices such as smartphones or tablets that do not have lighting devices or only have simple lighting devices.

[0035] To address the aforementioned problems, one aspect of the present invention relates to an eye detection method in which a computer: acquires a color image of a person's face captured by a camera device; generates a grayscale image for each pixel of the color image by multiplying the red, green, and blue component values ​​by a predetermined ratio corresponding to the characteristics of the lenses of the glasses worn by the person; detects the person's eyes from the grayscale image; and outputs eye information related to the detected eyes.

[0036] According to this configuration, for each pixel of a color image, a grayscale image is generated by multiplying the red, green, and blue component values ​​by a predetermined ratio corresponding to the characteristics of the lenses of eyeglasses worn by a person. The person's eyes are then detected from the grayscale image. Therefore, even with a simple device such as a smartphone or tablet, external light or background reflected onto the lens portion of the eyeglasses in the color image can be removed, improving the accuracy of detecting the person's eyes from the color image.

[0037] Furthermore, in the eye detection method described above, the ratio by which the red component value is multiplied can also be higher than the ratio by which the green component value and the blue component value are multiplied.

[0038] According to this configuration, since a grayscale image with more red components than green and blue components is generated, when the lenses of glasses worn by a person have the property of reflecting green and blue light, the green and blue light reflected into the lens portion of the glasses can be reduced.

[0039] Furthermore, in the eye detection method described above, the ratio by which the blue component value is multiplied can also be 0.

[0040] According to this configuration, since a grayscale image with the blue component removed is generated, when the lenses of glasses worn by a person have the property of reflecting light with the blue component, the blue component of light reflected into the lens portion of the glasses can be removed.

[0041] Furthermore, the eye detection method also calculates the average brightness value of the eye region containing the detected eye; determines whether the calculated average brightness value is higher than a threshold; and outputs projection information indicating that external light or background is reflected into the lens of the glasses if the calculated average brightness value is higher than the threshold.

[0042] When external light or background is reflected onto the lens of the glasses, the eye area is brighter than when external light or background is not reflected onto the lens. Therefore, by comparing the average brightness value of the eye area with a threshold, it is possible to detect residual external light or background that was not removed when generating the grayscale image and is still reflected onto the lens.

[0043] Furthermore, in the eye detection method, a binary image is generated from the color image; a speculative region in the binary image is extracted, which is presumed to contain the glasses; the length of a continuous white region of multiple white pixels within the extracted speculative region in the horizontal direction is used to determine whether the person is wearing the glasses; if it is determined that the person is not wearing the glasses, the person's eyes are detected from the color image; if it is determined that the person is wearing the glasses, a grayscale image is generated in the step of generating a grayscale image.

[0044] According to this configuration, if it is determined that the person is not wearing glasses, a grayscale image is not generated; instead, the person's eyes are detected from a color image. On the other hand, if it is determined that the person is wearing glasses, a grayscale image is generated to remove external light or background from the lens portion of the glasses projected into the color image. Therefore, the processing for removing external light or background from the lens portion of the glasses projected into the color image is only performed when the person is wearing glasses.

[0045] Furthermore, the eye detection method also detects a facial region containing the person's face from the color image; calculates an average brightness value of the facial region; determines that eye detection can be performed if the calculated average brightness value is higher than a threshold; and determines that eye detection cannot be performed if the calculated average brightness value is lower than the threshold.

[0046] When a person is in a relatively dark environment, external light easily reflects onto the lenses of their glasses. That is, in eye detection processing using color images, the brightness of the shooting environment affects the occurrence of reflection onto the glasses lenses. Here, if the average brightness value of the face area is above a threshold, it is determined that eye detection can be performed; if the average brightness value of the face area is below the threshold, it is determined that eye detection cannot be performed. Therefore, processing to remove external light or background reflected onto the lenses of the glasses included in the color image is only performed when eye detection is possible.

[0047] Furthermore, the eye detection method can also notify the person that the eye detection cannot be performed if it is determined that the eye detection cannot be performed.

[0048] According to this structure, since the person notified cannot perform an eye test when it is determined that the eye test cannot be performed, the person who receives the notification can perform the eye test by moving to a brighter place.

[0049] Furthermore, in the eye detection method, the grayscale image is binarized to generate a first image, in which pixels with grayscale values ​​less than a threshold are represented by a first brightness value, and pixels with grayscale values ​​above the threshold are represented by a second brightness value; a second image is also generated by replacing pixels with the second brightness value with pixels with the first brightness value, wherein the pixels with the second brightness value are pixels that appear in the first image within a first brightness region having the first brightness value and satisfy a predetermined condition; iris information containing at least one of the information related to the position and size of the person's iris is calculated using the second image; and the iris information is output.

[0050] According to this configuration, a second image is generated by replacing pixels with a second brightness value with pixels with a first brightness value, where the pixels with the second brightness value are pixels that appear in the first image within the first brightness region and meet predetermined conditions. Consequently, island regions with second brightness values ​​appearing within the pupil region of the first brightness region are painted over with the first brightness value. Furthermore, iris information is calculated using the painted binary image, i.e., the second image. As a result, the influence of external light or background reflected onto the cornea can be suppressed, further improving the detection accuracy of iris information.

[0051] Furthermore, the present invention can be implemented not only as an eye detection method performing the characteristic processes described above, but also as an eye detection device having a configuration corresponding to the characteristic processes of the eye detection method. It can also be implemented as a computer program that causes a computer to perform the characteristic processes included in the eye detection method. Therefore, the other embodiments described below also have the same effects as the eye detection method described above.

[0052] Another aspect of the present invention relates to an eye detection apparatus, comprising: an acquisition unit for acquiring a color image of a person's face captured by a camera device; a generation unit for generating a grayscale image for each pixel of the color image by multiplying the red component value, green component value, and blue component value by a predetermined ratio corresponding to the characteristics of the lens of the glasses worn by the person; a detection unit for detecting the person's eyes from the grayscale image; and an output unit for outputting eye information related to the detected eyes.

[0053] Another aspect of the present invention relates to an eye detection program that allows a computer to perform the following processes: acquiring a color image of a person's face captured by a camera device; generating a grayscale image for each pixel of the color image by multiplying the red component value, green component value, and blue component value by a predetermined ratio corresponding to the characteristics of the lens of the glasses worn by the person; detecting the person's eyes from the grayscale image; and outputting eye information related to the detected eyes.

[0054] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Furthermore, the following embodiments are merely examples illustrating the present invention and are not intended to limit the scope of protection of the present invention.

[0055] First Implementation Method

[0056] Figure 1 This is an external view of an eye detection system 100 according to a first embodiment of the present invention. The eye detection system 100 is composed of a portable terminal device such as a smartphone or tablet computer. However, this is only one example, and the eye detection system 100 can also be configured by appropriately combining a desktop computer or cloud server with a camera and a display.

[0057] The eye detection system 100 includes an eye detection device 1, a camera device 2, and a display 3. The eye detection device 1 calculates eye information related to the eyes of a person U1 captured by the camera device 2.

[0058] The imaging device 2 consists of a camera mounted on a portable terminal device. The imaging device 2 is a camera capable of acquiring color visible light images at a specified frame rate.

[0059] The display 3 is composed of a liquid crystal display device or an organic EL (ElectroLuminescence) display device, which is mounted on a portable terminal device. The display 3 displays an image of the face of the person U1 captured by the camera device 2. Furthermore, the display 3 overlays the eye information calculated by the eye detection device 1 (described later) onto the image of the person U1's face.

[0060] Figure 2 This is a block diagram illustrating an example of the overall configuration of an eye detection system 100 according to a first embodiment of the present invention. The eye detection device 1 includes a processor 10 and a memory 20. The processor 10 is, for example, a CPU (Central Processing Unit). The processor 10 includes an image acquisition unit 11, an image generation unit 12, an eye detection unit 13, and an output unit 14. The image acquisition unit 11, the image generation unit 12, the eye detection unit 13, and the output unit 14 are implemented, for example, by having the processor 10 execute an eye detection program.

[0061] The image acquisition unit 11 acquires the color image captured by the camera device 2. Here, the acquired color image includes the face of the person U1. Hereinafter, the image including the face will be referred to as a face image. The image acquisition unit 11 acquires the captured face images sequentially at a predetermined frame rate.

[0062] The image generation unit 12 converts a color image into a grayscale image. For each pixel of the color image acquired by the image acquisition unit 11, the image generation unit 12 generates a grayscale image by multiplying the red, green, and blue component values ​​by a predetermined ratio corresponding to the characteristics of the lenses of the glasses worn by person U1. The image generation unit 12 also generates a grayscale image by multiplying the red, green, and blue component values ​​of each pixel of the color image by a predetermined ratio corresponding to the characteristics of the lenses of the glasses worn by person U1, and summing the red, green, and blue component values ​​multiplied by the predetermined ratio.

[0063] At this time, the ratio of multiplying the red component value is higher than the ratios of multiplying the green and blue component values. For example, the ratio of multiplying the red component value is 0.9, the ratio of multiplying the green component value is 0.1, and the ratio of multiplying the blue component value is 0. In this case, the image generation unit 12 calculates the pixel value V of each pixel of the grayscale image using the following formula (1).

[0064] V=0.9×R+0.1×G+0×B…… (1)

[0065] In the above formula (1), R represents the red component value of each pixel in the color image, G represents the green component value of each pixel in the color image, and B represents the blue component value of each pixel in the color image.

[0066] Furthermore, it could be, for example, that the ratio of multiplying the red component value is 1.0, the ratio of multiplying the green component value is 0, and the ratio of multiplying the blue component value is 0.

[0067] The multiplication rate for the red component values ​​should preferably be higher than the usual multiplication rate for red component values ​​when converting from a color image to a grayscale image (e.g., 0.299). The multiplication rate for the green component values ​​should preferably be lower than the usual multiplication rate for green component values ​​when converting from a color image to a grayscale image (e.g., 0.587). The multiplication rate for the blue component values ​​should preferably be lower than the usual multiplication rate for blue component values ​​when converting from a color image to a grayscale image (e.g., 0.114).

[0068] Furthermore, the ratio by which the red component value is multiplied is preferably in the range of 0.3 to 1.0, more preferably in the range of 0.9 to 1.0. Moreover, the ratio by which the green component value is multiplied is preferably in the range of 0 to 0.58, more preferably in the range of 0 to 0.1. The ratio by which the blue component value is multiplied is preferably 0.

[0069] LED (Light Emitting Diode) displays in personal computers or smartphones emit blue light with wavelengths similar to ultraviolet light, ranging from 380 to 500 nanometers. Since this blue light can have adverse effects on the eyes, glasses with blue light reduction functions exist. These glasses suppress blue light from reaching the eyes by reflecting it through the lenses. When photographing a person U1 wearing these blue light reduction glasses, the reflected blue component is superimposed on the lens portion of the photographed image, reducing the accuracy of eye detection for person U1. Here, the image generation unit 12, when converting the color image acquired by the image acquisition unit 11 into a grayscale image, multiplies the blue component value of each pixel in the color image by a ratio of 0. This generates a grayscale image that suppresses the superimposition of blue components on the lens portion of the photographed image, thereby reducing noise in the eye area.

[0070] Furthermore, the ratios by which the red, green, and blue component values ​​are multiplied are just an example and can be changed according to the characteristics of the eyeglass lenses.

[0071] Furthermore, since the image captured by camera device 2 is quantized in 8 bits, the grayscale image has grayscale values ​​from 0 to 255; however, this is only one example. If the image captured by camera device 2 is quantized in 16 bits or other bit quantities, the grayscale image can have grayscale values ​​that can be represented by that number of bits.

[0072] The eye detection unit 13 detects the eyes of the person U1 from the grayscale image generated by the image generation unit 12. The eye detection unit 13 also detects the facial region representing the person's face from the grayscale image generated by the image generation unit 12. The eye detection unit 13 can detect the facial region by inputting the grayscale image into a classifier pre-created for detecting the facial region. This classifier is, for example, a Haar cascade classifier. The facial region is, for example, a rectangular region having the overall size of the face.

[0073] The eye detection unit 13 detects the eye region by inputting the face region into a classifier pre-created for detecting the eye region. This classifier is, for example, a Haar cascade classifier. The eye region is the area containing the eye. For example, the eye region is a rectangular area whose upper edge connects to the upper eyelid, its lower edge connects to the lower eyelid, one of its left and right edges connects to the inner corner of the eye, and the other of its left and right edges connects to the outer corner of the eye. Alternatively, the eye region can also be a rectangular area with a size that includes a predetermined margin for the size of either the left or right eye. The eye detection unit 13 detects the eye region containing the left eye and the eye region containing the right eye from the face region.

[0074] Hereinafter, the area containing the left eye will be referred to as the "left eye area," and the area containing the right eye will be referred to as the "right eye area." Without distinguishing between the two, they will be simply referred to as the eye area. Furthermore, the left eye refers to the eye located on the left side when viewing person U1 from the front, and the right eye refers to the eye located on the right side when viewing person U1 from the front. However, this is just an example; this relationship can also be reversed.

[0075] The output unit 14 outputs eye information related to the eyes detected by the eye detection unit 13. The eye information includes, for example, the position of the eye, i.e., the position of the eye region. The output unit 14 generates a display screen and displays the display screen on the monitor 3. The display screen is a picture in which the eye region detected by the eye detection unit 13 is superimposed on the color image acquired by the image acquisition unit 11.

[0076] The memory 20 is a storage device capable of storing various types of information, such as RAM (Random Access Memory), SSD (Solid State Drive), or flash memory. The memory 20 is used as the operating area of ​​the processor 10. Furthermore, the memory 20 can also store eye information output through the output unit 14.

[0077] Next, the eye detection process of the eye detection device 1 according to the first embodiment will be described.

[0078] Figure 3 This is a flowchart illustrating an example of the eye detection process of the eye detection device according to the first embodiment of the present invention. Additionally, Figure 3 The flowchart shown is executed at a specified sampling period. The specified sampling period is, for example, the frame period of the camera device 2.

[0079] First, in step S1, the image acquisition unit 11 acquires a color image from the camera device 2.

[0080] Figure 4 This is a schematic diagram showing an example of a color image captured by the camera device 2 in the first embodiment.

[0081] exist Figure 4 In the color image 31 shown, a person is wearing glasses with blue light reduction function. Therefore, noise from the blue light component reflected by the left lens of the glasses is superimposed on the person's left eye.

[0082] Return to Figure 3Next, in step S2, the image generation unit 12 multiplies the red, green, and blue component values ​​of each pixel in the color image acquired by the image acquisition unit 11 by a predetermined ratio corresponding to the characteristics of the lenses of the glasses worn by person U1, and generates a grayscale image by summing the red, green, and blue component values ​​multiplied by the predetermined ratio. For example, the image generation unit 12 generates a grayscale image by multiplying the red component value of each pixel in the color image by 0.9, the green component value by 0.1, and the blue component value by 0.

[0083] Figure 5 It means from Figure 4 A schematic diagram illustrating an example of a grayscale image generated from a color image.

[0084] exist Figure 5 The grayscale image 32 shown can remove noise from the lens portion superimposed on the left side of the glasses by setting the multiplication rate of the red component value to be higher than usual and the multiplication rate of the green and blue component values ​​to be lower than usual.

[0085] Return to Figure 3 Next, in step S3, the eye detection unit 13 detects the eyes of the person U1 from the grayscale image generated by the image generation unit 12. Here, the eye detection unit 13 inputs the grayscale image into a classifier for detecting face regions to detect face regions. The eye detection unit 13 detects a rectangular region including the upper part of the forehead, the lower part of the chin, and the hairline of the ears as the face region. Alternatively, the face region can also be a region including the entire hair. Then, the eye detection unit 13 inputs the detected face region into a classifier for detecting eye regions to detect eye regions. The eye region is a rectangular region including the entire eye.

[0086] Next, in step S4, the output unit 14 outputs eye information related to the human eye. The output unit 14 generates a display screen and displays the generated display screen on the monitor 3. This display screen overlays the eye information related to the human eye onto the color image. The eye information is, for example, a rectangular eye region. The output unit 14 generates a display screen that overlays the outer edge of the eye region of at least one of the left and right eyes onto the color image and displays the display screen on the monitor 3. This is achieved by repeating the process at a predetermined sampling period. Figure 3 The eye detection processing shown can display a screen on display 3 in real time, overlaying eye information onto the face image.

[0087] Thus, for each pixel of the color image 31, a grayscale image 32 is generated by multiplying the red, green, and blue component values ​​by a predetermined ratio corresponding to the characteristics of the lenses of the glasses worn by the person, and the person's eyes are detected from the grayscale image 32. Therefore, even with a simple device such as a smartphone or tablet, external light or background reflected onto the lens portion of the glasses included in the color image 31 can be removed, improving the accuracy of detecting the person's eyes from the color image 31.

[0088] Furthermore, the eye information detected by the eye detection unit 13 can also be used for processing such as gaze detection, iris detection, or estimating gaze distance. Moreover, the eye information can also be used to determine whether the object being detected is a living person. Additionally, the eye information can be used to determine whether the captured color image is a still image or a moving image.

[0089] Second Implementation Method

[0090] In the first embodiment, human eyes are detected from the generated grayscale image, and eye information related to the detected eyes is output. In the second embodiment, the average brightness value of the eye region containing the detected eyes is calculated, and if the calculated average brightness value is higher than a threshold, imaging information indicating that external light or background has been reflected onto the lens of the glasses is output.

[0091] Figure 6 This is a block diagram illustrating an example of the overall configuration of the eye detection system 100A according to the second embodiment of the present invention. Furthermore, in the second embodiment, the same reference numerals are used for the same constituent elements as in the first embodiment, and their descriptions are omitted.

[0092] The eye detection device 1A includes a processor 10A and a memory 20. Compared with the processor 10 of the eye detection device 1 of the first embodiment, the processor 10A also includes a mapping determination unit 15 and an iris detection processing unit 16.

[0093] The imaging determination unit 15 calculates the average brightness value of the eye region detected by the eye detection unit 13. The imaging determination unit 15 determines whether the calculated average brightness value is higher than a threshold. If the imaging determination unit 15 determines that the calculated average brightness value is higher than the threshold, it outputs imaging information indicating that external light or background has been reflected onto the lens of the glasses.

[0094] For example, the image generation unit 12 generates a grayscale image for each pixel of the color image by multiplying the red, green, and blue component values ​​by predetermined ratios corresponding to the characteristics of blue light reflected from the lenses of the eyeglasses. In this case, although the reflected noise of the blue component contained in the eye area can be removed, reflected noise other than the blue component may remain. Here, in the second embodiment, it is determined whether external light or background is reflected into the lenses of the eyeglasses from the generated grayscale image.

[0095] Typically, when photographing a person without vision, or when no external light or background is reflected in the lenses of the glasses being photographed, the average brightness of the eye area tends to be lower because the pupil (iris) is black or dark brown. Conversely, when external light or background is reflected in the lenses of the glasses being photographed, the average brightness of the eye area tends to increase because the pupil (iris) becomes brighter. Therefore, by comparing the average brightness of the eye area with a threshold, it is possible to determine whether external light or background is reflected in the lenses of the glasses.

[0096] The iris detection processing unit 16 generates a binary image (first image) that binarizes the eye region. In this binary image, pixels with grayscale values ​​less than a threshold are represented by a first brightness value, and pixels with grayscale values ​​greater than or equal to the threshold are represented by a second brightness value. When the mapping determination unit 15 outputs mapping information, the iris detection processing unit 16 changes the threshold used to compare grayscale values ​​during eye region binarization. More specifically, when the mapping determination unit 15 outputs mapping information, the iris detection processing unit 16 increases the threshold used to compare grayscale values ​​during eye region binarization. The first brightness value is, for example, white, and the second brightness value is, for example, black. That is, in the second embodiment, a binary image is generated where dark areas are represented by white and bright areas by black. The brightness value of white is, for example, represented by 255, and the brightness value of black is, for example, represented by 0.

[0097] Furthermore, the iris detection processing unit 16 generates a binary image (second image) in which pixels with second brightness values ​​are replaced with pixels with first brightness values. The pixels with the second brightness values ​​are pixels that appear in the generated binary image (first image) within a first brightness region having the first brightness value and satisfy predetermined conditions. Then, the iris detection processing unit 16 uses the binary image (second image) to calculate iris information containing information related to at least one of the position and size of a person's iris.

[0098] In addition, the first brightness value can also be black, and the second brightness value can also be white.

[0099] The output unit 14 outputs eye information related to the eyes. The eye information may also include iris information. The output unit 14 generates a display screen that overlays the iris information calculated by the iris detection processing unit 16 onto the face image acquired by the image acquisition unit 11, and displays the display screen on the display 3.

[0100] Next, the eye detection processing of the eye detection device 1A according to the second embodiment will be described.

[0101] Figure 7 This is a flowchart illustrating an example of the eye detection process of the eye detection device 1A according to the second embodiment of the present invention.

[0102] Because the processing of steps S11 to S13 is related to... Figure 3 The processes in steps S1 to S3 are the same, so their descriptions are omitted.

[0103] Next, in step S14, the mapping determination unit 15 calculates the average brightness value of the eye region including the eye detected by the eye detection unit 13. Additionally, the mapping determination unit 15 calculates the average brightness value of the left eye region including the left eye and the average brightness value of the right eye region including the right eye.

[0104] Figure 8 This is a schematic diagram illustrating an example of a facial region detected by the eye detection unit 13. Figure 9 This is a schematic diagram illustrating an example of the left eye region detected by the eye detection unit 13. Figure 10 This is a schematic diagram showing an example of the right eye region detected by the eye detection unit 13.

[0105] The eye detection unit 13 inputs a grayscale image into a classifier for detecting face regions, and detects face regions 33. Furthermore, the eye detection unit 13 inputs the detected face regions 33 into a classifier for detecting eye regions, detecting the left eye region 331 and the right eye region 332. Here, external light or background light is reflected onto the right lens of the glasses. Therefore, the right eye region 332 is brighter than the left eye region 331. More specifically, the average brightness value of the left eye region 331 is 109.08, while the average brightness value of the right eye region 332 is 139.73.

[0106] Return to Figure 7Next, in step S15, the mapping determination unit 15 determines whether the calculated average brightness value is higher than a threshold. Furthermore, if the grayscale image has grayscale values ​​from, for example, 0 to 255, the threshold is half of that value, 128. Here, if it is determined that the calculated average brightness value is lower than the threshold, the process proceeds to step S18. Additionally, the mapping determination unit 15 determines whether the average brightness value of the left eye region is higher than the threshold, and also determines whether the average brightness value of the right eye region is higher than the threshold.

[0107] On the other hand, if it is determined that the calculated average brightness value is higher than the threshold, in step S16, the imaging determination unit 15 outputs imaging information indicating that external light or background has been reflected onto the lens of the glasses to the iris detection processing unit 16. In addition, the imaging determination unit 15 includes information indicating whether the imaging occurred in the left eye region or the right eye region in the imaging information.

[0108] Next, in step S17, the iris detection processing unit 16 changes the threshold used for binarization processing. The iris detection processing unit 16 performs binarization processing on the eye region of the grayscale image, but sets the threshold used for this binarization processing to a predetermined value added to that threshold. Additionally, the iris detection processing unit 16 changes the threshold used for binarization processing of the regions where projection occurs between the left and right eye regions.

[0109] Next, in step S18, the iris detection processing unit 16 performs binarization processing on the eye region of the grayscale image and generates a binary image.

[0110] Next, in step S19, the iris detection processing unit 16 calculates iris information by applying iris detection processing to the generated binary image.

[0111] Here, the iris detection processing performed by the iris detection processing unit 16 will be explained.

[0112] First, the iris detection processing unit 16 divides the binary image into multiple local regions sequentially in the X direction according to a predetermined number of pixels. For example, the iris detection processing unit 16 divides the binary image into 10 equal parts horizontally. Thus, the binary image is divided into 10 strip-shaped local regions with the Y direction as its length direction. Here, the iris detection processing unit 16 divides the binary image into 10 local regions, but this is only one example. The number of divisions can also be an integer between 2 and 9, or an integer greater than 11. The Y direction refers to the longitudinal (vertical) direction of the image captured by the imaging device 2.

[0113] Next, the iris detection and processing unit 16 calculates the average brightness value of each of the 10 local areas.

[0114] Next, the iris detection and processing unit 16 calculates the X-coordinate of the estimated iris center position. The estimated iris center position is a predicted position of the iris center position and differs from the final calculated iris center position. Due to the influence of double eyelids, thick eyelashes, and false eyelashes, these areas may sometimes appear large as white regions. In such cases, the sclera (white of the eye) may be obscured. To avoid this, the estimated iris center position is calculated in the second embodiment.

[0115] The iris detection processing unit 16 calculates the X-coordinate of the estimated iris center position using the coordinates of the midpoint of the local region with the largest average brightness value in the X direction among multiple local regions. However, depending on the width of the local region in the X direction, using the midpoint of the local region as the X-coordinate of the estimated iris center position may sometimes be inappropriate. In such cases, the left or right end of the local region in the X direction can also be used as the X-coordinate of the estimated iris center position for calculation.

[0116] Next, the iris detection processing unit 16 calculates the Y-coordinate of the estimated iris center position. In a local region where the X-coordinate of the estimated iris center position exists, the iris detection processing unit 16 detects the uppermost and lowermost points of white pixels, and calculates the Y-coordinate of the estimated iris center position using the midpoint between the uppermost and lowermost points. Furthermore, due to the influence of eyelashes and makeup, the uppermost and lowermost points may sometimes appear in a local region adjacent to the left or right of the local region. Here, the iris detection processing unit 16 can also calculate the uppermost and lowermost points in the local region where the X-coordinate of the estimated iris center position exists and in two local regions adjacent to the left and right of the local region. The average uppermost point is calculated by averaging the three calculated uppermost points, and the average lowermost point is calculated by averaging the three calculated lowermost points. The midpoint between the average uppermost and average lowermost points is then used to calculate the Y-coordinate of the estimated iris center position.

[0117] Next, the iris detection processing unit 16 performs coating processing on the binary image. In visible light images, due to ambient brightness, external light or background may sometimes be reflected onto the cornea. When this reflection is significant, bright areas of white or other colors appear within the black or brown pupil. In such cases, if the eye image is binarized, black island areas will appear within the pupil area, making it impossible to detect iris information with high accuracy. Therefore, in the second embodiment, coating processing is performed to cover the black island areas.

[0118] The details of the coating process are as follows. First, the iris detection processing unit 16 sets a vertical line parallel to the Y direction for the X-coordinate of the binary image at the predicted iris center position. Second, the iris detection processing unit 16 detects the first white pixel appearing on the vertical line from the upper side of the binary image as the upper pixel. Next, the iris detection processing unit 16 detects the first white pixel appearing on the vertical line from the lower side of the binary image as the lower pixel. Next, the iris detection processing unit 16 determines whether the distance between the upper and lower pixels is greater than a first reference distance. Next, if the iris detection processing unit 16 determines that the distance between the upper and lower pixels is greater than the first reference distance, it determines that a black pixel located between the upper and lower pixels on the vertical line is a black pixel that meets a predetermined condition and replaces the black pixel with a white pixel. On the other hand, if the iris detection processing unit 16 determines that the distance between the upper and lower pixels is less than the first reference distance, it does not perform the replacement of the vertical line. The first reference distance can be, for example, an appropriate distance based on the assumed iris diameter.

[0119] The iris detection processing unit 16 performs this coating process on each longitudinal line within a range of a left reference distance from the estimated center position of the iris towards the X direction, and performs the same coating process on each longitudinal line within a range of a right reference distance from the estimated center position of the iris towards the X direction. The sum of the left and right reference distance ranges is an example of a second reference distance. The left and right reference distance ranges are, for example, the same range. As the second reference distance, for example, a distance slightly larger than the assumed iris diameter is used. Thus, the coating process can be applied selectively to longitudinal lines located in the pupil region.

[0120] Next, the iris detection processing unit 16 detects the left and right pixels of the pupil region. Within the white region of the binary image, the iris detection processing unit 16 sequentially investigates the changes in brightness values ​​of each pixel in the X direction, starting from the predicted center position of the iris. Furthermore, the iris detection processing unit 16 detects the first black pixel appearing on the left side in the X direction as the left-end pixel and the first black pixel appearing on the right side in the X direction as the right-end pixel.

[0121] Secondly, the iris detection processing unit 16 calculates the X-coordinate of the iris center position by using the midpoint between the left and right pixels as the midpoint.

[0122] Next, the iris detection processing unit 16 detects the upper and lower pixels of the pupil region. Within the white region of the binary image, the iris detection processing unit 16 sequentially investigates the changes in brightness values ​​of each pixel in the Y direction, starting from the X-coordinate of the iris center position. Furthermore, the iris detection processing unit 16 detects the first black pixel appearing in the upper part of the Y direction as the upper pixel and the first black pixel appearing in the lower part of the Y direction as the lower pixel.

[0123] Next, the iris detection processing unit 16 calculates the Y-coordinate of the iris center position using the midpoint between the upper and lower pixels as the coordinate. The iris center position is then calculated using this method.

[0124] Next, the iris detection processing unit 16 calculates the iris radius. Here, the iris detection processing unit 16 can calculate the iris radius using the distance between the iris center position and the left-hand pixel, the distance between the iris center position and the right-hand pixel, or the average of the two distances. Alternatively, the iris detection processing unit 16 can calculate the iris radius using the distance between the iris center position and the upper-hand pixel, the distance between the iris center position and the lower-hand pixel, or the average of the two distances. Alternatively, the iris detection processing unit 16 can also calculate the iris radius using the average of these four distances. Furthermore, the iris detection processing unit 16 can also calculate the outer edge of the iris using a circle centered at the iris center position and with the iris radius as its radius.

[0125] The above is an explanation of iris detection and processing.

[0126] Next, in step S20, the output unit 14 outputs eye information related to the human eye. The eye information includes iris information calculated by the iris detection processing unit 16. The output unit 14 generates a display screen that overlays the iris information calculated by the iris detection processing unit 16 onto a color image and displays it on the display 3. The iris information includes, for example, at least one of the iris center position, iris radius, and outer edge of the iris for the left and right eyes.

[0127] As mentioned above, the eye area is brighter when external light or background is reflected onto the lens of the glasses than when external light or background is not reflected onto the lens of the glasses. Therefore, by comparing the average brightness value of the eye area with a threshold, it is possible to detect residual external light or background that was not removed when generating the grayscale image and is reflected onto the lens of the glasses.

[0128] In addition, in the second embodiment, the projection determination unit 15 outputs projection information to the iris detection processing unit 16; however, the present invention is not limited to this. Regarding the projection of external light or background onto the lenses of glasses, it can sometimes be eliminated by changing the environment. Here, the processor 10A may also include a notification unit that notifies the person that projection has occurred on the lenses of the glasses. The projection determination unit 15 may also output projection information to the notification unit. The notification unit, if it receives projection information, notifies the person that projection has occurred on the lenses of the glasses. In this case, the notification unit may also display an image on the display 3 to notify the person that projection has occurred on the lenses of the glasses. Furthermore, the notification unit may also output a sound from a speaker to notify the person that projection has occurred on the lenses of the glasses.

[0129] Furthermore, the aforementioned iris detection processing can also be performed in the first embodiment. Specifically, after performing... Figure 3 After the processing of steps S1 to S3, the following can also be performed. Figure 7 The processing of steps S18 to S20.

[0130] Furthermore, the eye detection device 1A may also include a visual distance estimation unit, which estimates the visual distance representing the distance between an object and a person's eyes using iris information calculated by the iris detection processing unit 16. The visual distance estimation unit may also calculate a first value representing the number of pixels indicating the size of the iris detected by the iris detection processing unit 16. Moreover, the visual distance estimation unit may acquire the resolution of a color image and, based on the first value and a second value representing the actual size of a predetermined iris, calculate a third value representing the actual size corresponding to one pixel. Furthermore, the visual distance estimation unit may also estimate the visual distance corresponding to the acquired resolution and the calculated third value based on relational information representing the relationship between resolution, the third value, and visual distance.

[0131] Third Implementation Method

[0132] In the first embodiment, for each pixel of the color image, a grayscale image is generated by multiplying the red, green, and blue component values ​​by a predetermined ratio corresponding to the characteristics of the lenses of the glasses worn by the person. In contrast, in the third embodiment, firstly, it is determined whether the person is wearing glasses; if it is determined that the person is wearing glasses, a grayscale image is generated using the method of the first embodiment.

[0133] Figure 11 This is a block diagram illustrating an example of the overall configuration of the eye detection system 100B according to a third embodiment of the present invention. Furthermore, in the third embodiment, the same reference numerals are used for the same constituent elements as in the first embodiment, and their descriptions are omitted.

[0134] The eye detection device 1B includes a processor 10B and a memory 20. The processor 10B, relative to the processor 10 of the first detection device 1 in the first embodiment, also includes an eyeglass wearing determination unit 17.

[0135] The glasses-wearing determination unit 17 generates a binary image from the color image acquired by the image acquisition unit 11. The glasses-wearing determination unit 17 extracts a glasses-predicting region in the binary image, presuming the presence of glasses. Within the extracted glasses-predicting region, the glasses-wearing determination unit 17 determines whether a person is wearing glasses based on the horizontal length of a continuous white region of multiple white pixels. If the glasses-wearing determination unit 17 determines that the person is not wearing glasses, the eye detection unit 13 detects the person's eyes from the color image. Conversely, if the glasses-wearing determination unit 17 determines that the person is wearing glasses, the image generation unit 12 generates a grayscale image from the color image.

[0136] Next, the eye detection process of the eye detection device 1B according to the third embodiment will be described.

[0137] Figure 12 This is a flowchart illustrating an example of the eye detection process of the eye detection device 1B according to the third embodiment of the present invention.

[0138] Because the processing in step S31 is related to Figure 3 The process of step S1 shown is the same, so its description is omitted.

[0139] Next, in step S32, the glasses wearing determination unit 17 detects the face region from the color image acquired by the image acquisition unit 11. The glasses wearing determination unit 17 inputs the color image into a classifier for detecting the face region and detects the face region.

[0140] Next, in step S33, the glasses-wearing determination unit 17 binarizes the detected face region and generates a binary image. In this binary image, pixels with grayscale values ​​less than a threshold are represented by a first brightness value, and pixels with grayscale values ​​above the threshold are represented by a second brightness value. When the face region is composed of a color image, the glasses-wearing determination unit 17 can transform the face region into a grayscale image, for example, with grayscale values ​​from 0 to 255, and perform binarization processing on the transformed grayscale image. For example, Otsu's binarization processing can be used as the binarization processing. The first brightness value is, for example, white, and the second brightness value is, for example, black. That is, in this embodiment, a binary image is generated where white represents dark areas and black represents bright areas. The brightness value of white is, for example, 255, and the brightness value of black is, for example, 0.

[0141] Furthermore, the glasses wearing determination unit 17 performs general transformation processing when converting a color image to a grayscale image. For example, this general transformation processing to a grayscale image can involve calculating the average grayscale values ​​of the red, green, and blue components of each pixel constituting the face region. Alternatively, the glasses wearing determination unit 17 can also calculate the pixel value V of each pixel in the grayscale image using the following formula (2).

[0142] V=0.299×R+0.587×G+0.114×B……(2)

[0143] In the above formula (2), R represents the red component value of each pixel in the color image, G represents the green component value of each pixel in the color image, and B represents the blue component value of each pixel in the color image.

[0144] Figure 13 This is a schematic diagram illustrating an example of a binary image of the facial region in the third embodiment. Figure 13 The example generates a binary image 34, in which darker areas such as the face region are represented by white, including the eyes, eyebrows, nostrils, lips, hair, and the frames of glasses, and brighter areas such as the skin are represented by black.

[0145] Return to Figure 12 Next, in step S34, the glasses wearing determination unit 17 extracts the glasses prediction region from the binary image, which is presumed to contain glasses. Based on the proportions of a typical human face, the region extending from the top edge of the face area to the line representing the 3 / 10 position and up to the line representing the 6 / 10 position is extracted as the glasses prediction region. Figure 13 As shown, the glasses inference region 341 is the region of the binary image 34 of the face region from the top to the line representing the position of 3 / 10 to the line representing the position of 6 / 10.

[0146] Return to Figure 12 Next, in step S35, the glasses wearing determination unit 17 performs a labeling process on the extracted glasses-inferred region 341. In the labeling process, multiple consecutive white pixels in the binary image are assigned the same number. Through this labeling process, multiple white regions composed of consecutive white pixels can be detected from the glasses-inferred region 341. Figure 13 As shown, when a person is wearing glasses, a white area 342 representing the frame of the glasses is detected. The white area 342 is composed of a plurality of white pixels that are continuous in the horizontal direction.

[0147] As described above, by pre-extracting the binary image of the glasses inference region 341 from the binary image 34 of the face region, the range for labeling is narrowed, which can shorten the time required for labeling.

[0148] Return to Figure 12 Next, in step S36, the glasses wearing determination unit 17 calculates the width (horizontal length) of each of the multiple white regions of multiple consecutive white pixels within the extracted glasses prediction region 341.

[0149] Next, in step S37, the glasses wearing determination unit 17 determines whether the width of the longest white area is more than 2 / 3 of the width of the glasses' estimated area. Here, if the width of the longest white area is more than 2 / 3 of the width of the glasses' estimated area (yes in step S37), in step S38, the image generation unit 12 multiplies the red component value, green component value, and blue component value of each pixel of the color image acquired by the image acquisition unit 11 by a predetermined ratio corresponding to the characteristics of the lenses of the glasses worn by person U1, and generates a grayscale image by summing the red component value, green component value, and blue component value multiplied by the predetermined ratio.

[0150] The white area 342 representing the frame of the glasses is composed of multiple consecutive white pixels in the horizontal direction. A width where the width of the longest white area is more than 2 / 3 of the width of the glasses' estimated area indicates that the person being photographed is wearing glasses. Therefore, when the length of the white area 342 is more than 2 / 3 of the width of the glasses' estimated area 341, the glasses-wearing determination unit 17 can determine that the person is wearing glasses.

[0151] On the other hand, if the width of the longest white area is not more than 2 / 3 of the width of the glasses-predicted area (No in step S37), the process proceeds to step S39. The fact that the width of the longest white area is not more than 2 / 3 of the width of the glasses-predicted area means that the person being photographed is not wearing glasses.

[0152] Furthermore, because the processing of steps S38 to S40 is related to... Figure 3 The processes from steps S2 to S4 are the same, so their descriptions are omitted.

[0153] Thus, if it is determined that the person is not wearing glasses, a grayscale image is not generated; instead, the person's eyes are detected from the color image. On the other hand, if it is determined that the person is wearing glasses, a grayscale image is generated to remove external light or background from the lens portion of the glasses projected into the color image. Therefore, the process of removing external light or background from the lens portion of the glasses projected into the color image is only performed when the person is wearing glasses.

[0154] Here, the determination of eyeglass wearing in a modified example of the third embodiment will be explained.

[0155] Figure 14 This is a schematic diagram illustrating an example of a binary image of the facial region in a variation of the third embodiment. Figure 14 For example, a binary image 35 is generated, in which darker areas such as the face, eyes, eyebrows, nostrils, lips, hair, and glasses frames are represented by white, and brighter areas such as skin are represented by black.

[0156] In the above Figure 13 This indicates that the white area of ​​the eyeglass frame is detected as an island. However, in cases where external light enters the eyeglass frame or where color is applied to the frame, the white area may not be detected as an island. Figure 14 The frame of the glasses cannot be detected as an island, but is composed of multiple white areas 352.

[0157] Here, the glasses-wearing determination unit 17 can also determine whether the number of white areas with a width greater than a predetermined length is greater than a predetermined number. If the number of white areas with a width greater than a predetermined length is greater than a predetermined number, it can be determined that the person being filmed is wearing glasses. Furthermore, if the number of white areas with a width greater than a predetermined length is less than a predetermined number, it can be determined that the person being filmed is not wearing glasses.

[0158] A variation of the third embodiment, in Figure 12 In step S37, the glasses wearing determination unit 17 determines whether the number of white areas with a width greater than a predetermined length is greater than a predetermined number. If the determination is that the number of white areas with a width greater than a predetermined length is greater than a predetermined number ("Yes" in step S37), the process proceeds to step S38. Otherwise, if the determination is that the number of white areas with a width greater than a predetermined length is less than a predetermined number ("No" in step S37), the process proceeds to step S39.

[0159] Alternatively, the eye detection processing of the third embodiment and the eye detection processing of the second embodiment can be combined. In this case, the eye detection processing can be performed... Figure 12 After the processing of steps S31 to S39, proceed with... Figure 7 The processing after step S14.

[0160] Fourth Implementation Method

[0161] In eye detection processing using color images, the brightness of the shooting location affects the reflection of light into the lenses of the glasses. Therefore, in the fourth embodiment, the average brightness value of the face region in the color image is calculated. If the average brightness value is higher than a threshold, the following eye detection processing is performed; if the average brightness value is lower than the threshold, the following eye detection processing is not performed.

[0162] Figure 15 This is a block diagram illustrating an example of the overall configuration of the eye detection system 100C according to the fourth embodiment of the present invention. Furthermore, in the fourth embodiment, the same reference numerals are used for the same constituent elements as in the first embodiment, and their descriptions are omitted.

[0163] The eye detection device 1C includes a processor 10C and a memory 20. The processor 10C, compared with the processor 10 of the first detection device 1 in the first embodiment, also includes an eye detection judgment unit 18 and a notification unit 19.

[0164] The eye detection determination unit 18 detects a facial region containing a person's face from the color image acquired by the image acquisition unit 11. The eye detection determination unit 18 calculates the average brightness value of the facial region. If the calculated average brightness value is higher than a threshold, the eye detection determination unit 18 determines that eye detection can be performed. Conversely, if the calculated average brightness value is lower than the threshold, the eye detection determination unit 18 determines that eye detection cannot be performed.

[0165] If the eye detection determination unit 18 determines that eye detection can be performed, the image generation unit 12 generates a grayscale image from the color image.

[0166] The notification unit 19, when the eye detection judgment unit 18 determines that eye detection cannot be performed, notifies the person that eye detection cannot be performed. The notification unit 19, when the eye detection judgment unit 18 determines that eye detection cannot be performed, displays a screen on the monitor 3 to notify the person that eye detection cannot be performed. Additionally, the notification unit 19, when the eye detection judgment unit 18 determines that eye detection cannot be performed, can also output an audio signal from a speaker to notify the person that eye detection cannot be performed.

[0167] Next, the eye detection process of the eye detection device 1C according to the fourth embodiment will be described.

[0168] Figure 16 This is a flowchart illustrating an example of eye detection processing in the eye detection device 1C according to the fourth embodiment of the present invention.

[0169] Because the processing in step S51 is related to Figure 3 The process of step S1 shown is the same, so its description is omitted.

[0170] Next, in step S52, the eye detection determination unit 18 detects the face region from the color image acquired by the image acquisition unit 11. The eye detection determination unit 18 inputs the color image to a classifier for detecting the face region and detects the face region.

[0171] Next, in step S53, the eye detection determination unit 18 calculates the average brightness value of the detected face region. Alternatively, if the face region is composed of a color image, the eye detection determination unit 18 can also transform the face region into a grayscale image with, for example, a grayscale value of 0 to 255, and calculate the average brightness value of the face region in the transformed grayscale image. Furthermore, the transformation process here is a general transformation process from a color image to a grayscale image.

[0172] Next, in step S54, the eye detection and judgment unit 18 determines whether the average brightness value is higher than the threshold.

[0173] Figure 17 This is a schematic diagram illustrating an example of a facial region in the fourth embodiment where the average brightness value is higher than a threshold. Figure 18 This is a schematic diagram illustrating an example of a facial region where the average brightness value is below a threshold in the fourth embodiment.

[0174] exist Figure 17 Detecting face region 361 from color image 36, in Figure 18 Detect facial region 371 from color image 37.

[0175] Color images with a visible light wavelength range differ from those with an infrared wavelength range, and their advantage is that they can detect the eye even when the person is wearing glasses. However, color images acquired in relatively dark environments may have reduced eye detection accuracy because external light or background can easily be reflected into the lenses of the glasses. In particular, in relatively dark environments, when a color image is acquired using a smartphone's camera device 2 and displayed on the smartphone's display 3, the image on the display 3 can easily be reflected into the lenses of the glasses.

[0176] For example, Figure 17 The color image 36 shown was obtained in a location with a brightness of approximately 850 lux. Figure 18 The color image 37 shown was obtained in a location with a brightness of approximately 10 lux.

[0177] The average brightness value of the face region 361 detected from color image 36 is 130, and the average brightness value of the face region 371 detected from color image 37 is 96. For example... Figure 18As shown, in color images captured in locations where the average brightness value of the face region 371 is below 100, external light or background light is reflected onto the lenses of the glasses. Therefore, it is difficult to accurately detect the eyes from the face region 371 where external light or background light is reflected onto the lenses of the glasses.

[0178] Here, the eye detection judgment unit 18 determines whether the average brightness value of the face region is higher than a threshold. Furthermore, if the average brightness value of the face region is determined to be higher than the threshold, the following eye detection processing is performed; if the average brightness value of the face region is determined to be lower than the threshold, the following eye detection processing is not performed. The threshold value is, for example, 100.

[0179] Return to Figure 16 If it is determined that the average brightness value is higher than the threshold (Yes in step S54), in step S55, the image generation unit 12 generates a grayscale image by multiplying the red component value, green component value and blue component value of each pixel of the color image acquired by the image acquisition unit 11 by a predetermined ratio corresponding to the characteristics of the lens of the glasses worn by person U1, and summing the red component value, green component value and blue component value after multiplying by the predetermined ratio.

[0180] Furthermore, because the processing of steps S55 to S57 is related to... Figure 3 The processes from steps S2 to S4 are the same, so their descriptions are omitted.

[0181] On the other hand, if it is determined that the average brightness value is below the threshold (No in step S54), in step S58, the notification unit 19 notifies the person that eye detection cannot be performed. At this time, the notification unit 19 causes the display 3 to display a screen to notify the person that eye detection cannot be performed. For example, the screen may display a notification urging the person to move to a bright location, and explain that eye detection may not be possible because the current shooting location is relatively dark.

[0182] As mentioned above, when a person is in a relatively dark environment, external light easily enters the lens of their glasses. That is, in eye detection processing using color images, the brightness of the shooting location affects the occurrence of light entering the lens of the glasses. Here, if the average brightness value of the face area is higher than a threshold, it is determined that eye detection can be performed; if the average brightness value of the face area is lower than the threshold, it is determined that eye detection cannot be performed. Therefore, processing to remove external light or background light entering the lens of the glasses included in the color image is only performed when eye detection is possible.

[0183] Furthermore, since the person notified cannot undergo an eye test if it is determined that the test cannot be performed, the eye test can be conducted by having the person who received the notification move to a well-lit area.

[0184] Alternatively, the eye detection processing of the fourth embodiment and the eye detection processing of the second embodiment can be combined. In this case, it is also possible to perform... Figure 16 After the processing in steps S51 to S56, proceed with... Figure 7 The processing after step S14.

[0185] Furthermore, the eye detection processing of the fourth embodiment and the eye detection processing of the third embodiment can also be combined. In this case, it is also possible to perform... Figure 16 The processing of steps S51 to S54, and if step S54 is "yes", proceed as follows: Figure 12 The processing after step S33.

[0186] Furthermore, the eye detection processing of the fourth embodiment, the eye detection processing of the third embodiment, and the eye detection processing of the second embodiment can also be combined. In this case, it is also possible to perform... Figure 16 The processing of steps S51 to S54, and if step S54 is "yes", proceed as follows: Figure 12 The processing of steps S33 to S39 is then performed. Figure 7 The processing after step S14.

[0187] Furthermore, in the above embodiments, each component can be constructed using dedicated hardware or implemented by executing software programs suitable for each component. Each component can also be implemented by having a program execution unit such as a CPU or processor read and execute software programs recorded on recording media such as hard disks or semiconductor memory. Moreover, the program can be transferred while recorded on a recording medium, transferred via a network, or executed by a separate computer system.

[0188] Some or all of the functions of the apparatus involved in the embodiments of the present invention can typically be implemented as an integrated circuit LSI (Large Scale Integration). These functions can be individually chip-based or formed as a subset of chips. Furthermore, the integrated circuit is not limited to LSIs; it can also be implemented using dedicated circuits or general-purpose processors. Alternatively, a field-programmable gate array (FPGA) or a reconfigurable processor that can reconstruct the connections or settings of the circuit cells within the LSI can be utilized after LSI fabrication.

[0189] Furthermore, some or all of the functions of the device involved in the embodiments of the present invention can also be implemented by having a processor such as a CPU execute a program.

[0190] Furthermore, the figures used above are examples given for the purpose of specifically illustrating the present invention, and the present invention is not limited to these exemplified figures.

[0191] Furthermore, the order in which the steps shown in the flowchart are executed is merely an example to illustrate the present invention, and other orders may be used within the scope of achieving the same effect. Also, some of the steps may be executed simultaneously (in parallel) with other steps.

[0192] Industrial availability

[0193] The technology involved in this invention has practical value as a technology for detecting human eyes from images because it can improve the accuracy of detecting human eyes from color images.

Claims

1. An eye detection method, characterized in that, Have the computer perform the following steps: Acquire a color image containing a human face captured by a camera device; For each pixel of the color image, a grayscale image is generated by multiplying the red, green, and blue component values ​​by a predetermined ratio corresponding to the characteristics of the lenses of the glasses worn by the person. Detect the person's eyes from the grayscale image; Output eye information related to the detected eye.

2. The eye detection method according to claim 1, characterized in that, The ratio by which the red component value is multiplied is higher than the ratio by which the green component value and the blue component value are multiplied.

3. The eye detection method according to claim 2, characterized in that, The ratio by which the blue component value is multiplied is 0.

4. The eye detection method according to any one of claims 1 to 3, characterized in that, It also calculates the average brightness value of the eye region containing the detected eye; It also determines whether the calculated average brightness value is higher than a threshold. If the calculated average brightness value is determined to be higher than the threshold, the system also outputs projection information indicating that external light or background light is reflected onto the lens of the glasses.

5. The eye detection method according to any one of claims 1 to 3, characterized in that, A binary image is also generated from the color image; The glasses-inferred region, which is inferred to be the location of the glasses, is also extracted from the binary image. The method also determines whether the person is wearing the glasses based on the horizontal length of a continuous white region of multiple white pixels within the extracted speculative area of ​​the glasses. If it is determined that the person is not wearing the glasses, the person's eyes are also detected from the color image; If it is determined that the person is wearing the glasses, the grayscale image is generated in the step of generating the grayscale image.

6. The eye detection method according to any one of claims 1 to 3, characterized in that, The facial region containing the person's face is also detected from the color image; The average brightness value of the facial region is also calculated; If the calculated average brightness value is higher than the threshold, it is further determined that the eye detection can be performed; If the calculated average brightness value is below the threshold, it is determined that the eye cannot be detected.

7. The eye detection method according to claim 6, characterized in that, If it is determined that the eye cannot be tested, the person will also be notified that the eye test cannot be performed.

8. The eye detection method according to any one of claims 1 to 3, characterized in that, The grayscale image is also binarized to generate a first image. In the first image, pixels with grayscale values ​​less than a threshold are represented by a first brightness value, and pixels with grayscale values ​​above the threshold are represented by a second brightness value. A second image is also generated by replacing pixels with the second brightness value with pixels with the first brightness value, wherein the pixels with the second brightness value are pixels that appear in the first image within a first brightness region having the first brightness value and meet a specified condition; The second image is also used to calculate iris information containing information related to at least one of the location and size of the person's iris; It also outputs the iris information.

9. An eye detection device, characterized in that... include: The acquisition unit is used to acquire a color image containing a human face captured by a camera device; The generation unit generates a grayscale image by multiplying the red, green, and blue component values ​​of each pixel of the color image by a predetermined ratio corresponding to the characteristics of the lenses of the glasses worn by the person. The detection unit detects the person's eyes from the grayscale image; and, The output unit outputs information related to the detected eye.

10. A software product comprising an eye detection program, characterized in that, The eye detection program causes the computer to perform the following processes: Acquire a color image containing a human face captured by a camera device; For each pixel of the color image, a grayscale image is generated by multiplying the red, green, and blue component values ​​by a predetermined ratio corresponding to the characteristics of the lenses of the glasses worn by the person. Detect the person's eyes from the grayscale image; Output information related to the detected eye.

Images