Eye opening / closing determination device and eye opening / closing determination method

Abstract

To provide an eye open / close determination apparatus and an eye open / close determination method capable of improving a determination accuracy of an eye open / close state based on a photographed face image.SOLUTION: A video acquisition unit 10 in an eye open / close determination apparatus 100 acquires video data from a camera 60. A face orientation detection unit 20 detects an orientation of a face based on a face image in the video data acquired by the video acquisition unit 10. An eye image extraction unit 30 extracts an eye image including a region of the eyes based on the face image in the video data acquired by the video acquisition unit 10. An eye open / close determination unit 40 determines an open / close state of the eyes based on an opened amount of the eyes acquired from the eye image, an opened eye reference value, and a closed eye reference value. The eye open / close determination unit 40 includes a plurality of opened eye reference values and a plurality of closed eye reference values according to the orientation of the face, and switches the opened eye reference value and the closed eye reference value based on the orientation of the face detected by the face orientation detection unit 20.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an eye opening / closing determination device and an eye opening / closing determination method. [Background technology]

[0002] For example, a driver monitor mounted on a vehicle captures the inside of the vehicle cabin and stores the image including the driver in a storage device. The captured image of the inside of the vehicle cabin is used to check the driving situation, such as the direction the driver's face is facing, and the behavior of passengers in the vehicle cabin.

[0003] For example, Patent Document 1 describes a conventional detection device that detects the open / closed state of the eyes. This detection device includes a face direction detection unit, an eye detection unit, and an open / closed degree calculation unit. The face direction detection unit detects the direction of a person's face based on a face image that is an image of the person's face captured by a camera. The eye detection unit detects the size of the person's eyes based on the face image. The open / closed degree calculation unit calculates the open / closed degree, which indicates the degree of opening of the person's eyes based on the size of the person's eyes in the face image, by correcting it to a value when the person's face is viewed from the front based on the detected face direction. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-134490 A Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 describes that the degree of eye opening and closing is corrected with a calculated value according to the direction of the face. The inventor noticed that when the direction of the face is changed to the left or right, the actual measurement value of the eye opening amount differs from the calculated value even if the eye opening amount is the same due to the influence of the undulation of the eyelids, and thought that there was room for improvement in improving the accuracy of determining the eye opening and closing state. The inventor also noticed that when a camera is installed diagonally above the front of the face, the actual measurement value of the eye opening amount changes significantly when the face is changed to the left or right, and the accuracy of determining the eye opening and closing state may decrease.

[0006] The present invention has been made in consideration of the above circumstances, and its object is to provide an eye opening / closing determination device and an eye opening / closing determination method that can improve the accuracy of determining the eye opening / closing state based on a photographed facial image. [Means for solving the problem]

[0007] One aspect of the present invention is an eye opening / closing determination device, which includes a video acquisition unit that acquires video data from a camera, a face direction detection unit that detects a face direction based on a face image in the video data acquired by the video acquisition unit, an eye image extraction unit that extracts an eye image including an eye region based on the face image in the video data acquired by the video acquisition unit, and an eye opening / closing determination unit that determines an eye opening / closing state based on an eye opening amount, an eye opening reference value, and an eye closing reference value acquired from the eye image, the eye opening / closing determination unit having a plurality of the eye opening reference values ​​and the eye closing reference values ​​according to the face direction, and switching between the eye opening reference value and the eye closing reference value based on the face direction detected by the face direction detection unit.

[0008] Another aspect of the present invention is an eye opening / closing determination method for an eye opening / closing determination device, comprising: a video acquisition step of acquiring video data from a camera, a face direction detection step of detecting a face direction based on a face image in the video data acquired by the video acquisition step, an eye image extraction step of extracting an eye image including an eye region based on the face image in the video data acquired by the video acquisition step, and an eye opening / closing determination step of determining an eye opening / closing state based on an eye opening amount, an eye opening reference value, and an eye closing reference value acquired from the eye image, wherein the eye opening / closing determination step has a plurality of the eye opening reference values ​​and the eye closing reference values ​​according to the face direction, and switches the eye opening reference value and the eye closing reference value based on the face direction detected by the face direction detection step. Effect of the Invention

[0009] According to the present invention, it is possible to improve the accuracy of determining whether the eyes are open or closed based on a captured face image. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of a video processing device including an eye opening / closing determination device according to an embodiment. [Diagram 2] FIG. 13 is a schematic diagram for explaining the direction of a face. [Diagram 3] FIG. 3(a) is a schematic diagram showing an eye image in an open eye state, and FIG. 3(b) is a schematic diagram showing an eye image in a closed eye state. [Figure 4] 11 is a schematic diagram for explaining an eye-open reference value and an eye-closed reference value. FIG. [Diagram 5] 10A and 10B are schematic diagrams for explaining changes in the amount of eye opening depending on the face direction. [Figure 6] 1 is a schematic diagram for explaining determination of an open / closed state of the eyes using both eyes or one eye. FIG. [Figure 7] 10 is a flowchart showing a procedure for determining whether the eyes are open or closed in the eye open / closed state determination device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present invention will be described below based on preferred embodiments with reference to Figures 1 to 7. The same or equivalent components and members shown in each drawing are given the same reference numerals, and duplicated descriptions are omitted as appropriate. Also, some of the members that are not important for explaining the embodiments are omitted in each drawing.

[0012] (Embodiment) 1 is a block diagram showing a configuration of a video processing device 110 including an eye opening / closing determination device 100 according to an embodiment. The video processing device 110 is, for example, a driver monitor mounted in a vehicle and captures images of passengers in the vehicle cabin continuously over time. The video processing device 110 may record the captured images, or may discard the images without recording them.

[0013] The image processing device 110 is disposed, for example, on a dashboard below the windshield of the vehicle, on a rear-view mirror, etc. The image processing device 110 captures an image including, for example, a driver in the vehicle as an object to be captured, and performs image recognition of the entire face or facial parts such as the eyes of the captured object.

[0014] The video processing device 110 includes a camera 60, a recording unit 70, an external output unit 80, and an eye open / closed determination device 100. The eye open / closed determination device 100 detects the direction of the face based on the video data of the face image. The eye open / closed determination device 100 obtains the amount of eye opening from an eye image obtained by cutting out the area of ​​each eye, and determines the open / closed state of the eyes included in the video data based on the open eye reference value and the closed eye reference value. The eye open / closed determination device 100 determines the open / closed state of the eyes by determining the difference value obtained by subtracting the closed eye reference value from the open eye reference value as 100%, and obtains the degree of eye opening, which is the ratio of the amount of eye opening to the difference value, to determine the open / closed state of the eyes.

[0015] The camera 60 is an imaging device having a detector such as a CMOS sensor or a CCD sensor, and captures an image including, for example, a driver in a vehicle as an object to be captured. The camera 60 continuously acquires image data and transmits it to the image acquisition unit 10 described later.

[0016] The recording unit 70 is, for example, a removable medium such as an SD card or a USB memory, or a hard disk, and is capable of recording and deleting video data acquired by the video acquisition unit 10 described below. The recording unit 70 may be configured to be removable from the video processing device 110, in which case the recording unit 70 can be removed from the video processing device 110 and the video data can be played back on another PC or the like.

[0017] The external output unit 80 outputs to an external device the video data recorded in the recording unit 70, data on the eye open / closed state determined by the eyes open / closed state determination device 100, etc. The external output unit 80 may be configured to output data related to the direction of the driver's face detected by the eyes open / closed state determination device 100, etc. to an external device.

[0018] The eyes open / closed determination device 100 has a video acquisition unit 10, a face direction detection unit 20, an eye image extraction unit 30, and an eyes open / closed determination unit 40. The eyes open / closed determination device 100 is configured with, for example, a CPU and executes the processes of the above-mentioned units by operating according to a computer program. The storage unit 101 is configured with a data storage device such as a RAM (Random Access Memory), a flash memory, a hard disk storage device, etc., and stores the computer program executed by the eyes open / closed determination device 100, temporary processing data, etc. The storage unit 101 also stores a recognition dictionary for recognizing the face, eyes, etc. of the subject from the captured video.

[0019] Video acquisition unit 10 acquires video data from camera 60, performs processing such as data compression, and outputs the data to face direction detection unit 20, eye image extraction unit 30, and recording unit 70. Note that video acquisition unit 10 may be configured to include camera 60.

[0020] The face direction detection unit 20 recognizes the facial parts in the video data input from the video acquisition unit 10 based on the recognition dictionary and learning-type computation model stored in the storage unit 101. The recognition dictionary stored in the storage unit 101 contains shape data and the like relating to each part of the face, and the face direction detection unit 20 recognizes the facial parts by extracting a shape pattern appearing in the video data and comparing it with the shape data contained in the recognition dictionary. The face direction detection unit 20 can recognize the facial parts from the video using a known image recognition processing method that has been developed in various fields, and may use a method of recognizing the facial parts using, for example, a learning-type computation model using a neural network.

[0021] The face direction detection unit 20 detects the face direction of the subject based on data such as the recognized facial parts. Fig. 2 is a schematic diagram for explaining the face direction. Fig. 2 shows, for example, the driver's face as seen from the front of the vehicle, with the front direction of the vehicle being the X-axis direction, the left direction of the vehicle being the Y-axis direction, and the upward direction of the vehicle being the Z-axis direction. Rotation around the X-axis is roll (rolling), rotation around the Y-axis is pitch (pitching), and rotation around the Z-axis is yaw (yawing).

[0022] The direction of a face in video data changes due to pitching and yawing. That is, by detecting pitching of the face direction, it is possible to know that the subject has changed the direction of the face in the up-down direction. Also, by detecting yawing of the face direction, it is possible to know that the subject has changed the direction of the face in the left-right direction.

[0023] The face direction detection unit 20 detects the pitch angle based on the up and down movement of a line L1 connecting the contours of the upper eyelids of the left and right eyes as shown in Fig. 2, for example. The face direction detection unit 20 also detects the yaw angle based on the movement of the center line L2 of the nose and the positions P1 and P2 of the corners of the left and right eyes. In addition to these parts, the face direction detection unit 20 can detect the pitch angle and yaw angle of the face based on the movement of each part of the face, such as the eyebrows, nose, and mouth. The face direction detection unit 20 can use known techniques that have been developed in various technical fields for detecting the direction of the face.

[0024] The eye image extraction unit 30 recognizes each eye part in the video data input from the video acquisition unit 10 based on the recognition dictionary and learning-type computation model stored in the storage unit 101, and extracts an eye image obtained by cutting out the area of ​​each eye. The eye image extraction unit 30 recognizes the eye part by extracting a shape pattern appearing in the video data and comparing it with shape data included in the recognition dictionary. The eye image extraction unit 30 can recognize the eye part from the video using a known image recognition processing method that has been developed in various fields, and may use a method of recognizing the face, eyes, etc. using a learning-type computation model based on a neural network, for example.

[0025] The eye open / closed determination unit 40 calculates the eye opening amount T based on the eye image of each eye extracted by the eye image extraction unit 30, and determines the eye open / closed state based on the eye open reference value and the eye closed reference value. The eye open / closed determination unit 40 determines the eye open / closed state by determining the eye open / closed degree, which is the ratio of the eye open amount T to the difference value, assuming the difference value obtained by subtracting the eye closed reference value from the eye open reference value to be 100%.

[0026] Fig. 3(a) is a schematic diagram showing an eye image in an open state, and Fig. 3(b) is a schematic diagram showing an eye image in a closed state. Fig. 3(a) shows the eye image in a fully open state, and the eye-open reference value is set to a slightly smaller value with a margin for the eye-opening amount T in this state. Fig. 3(b) shows the eye image in a fully closed state, and the eye-closed reference value is set to a slightly larger value with a margin for the eye-opening amount T in this state.

[0027] The eye opening / closing determination unit 40 calculates the eye opening amount T based on the dimension in the eye height direction. The eye opening amount T is the largest value among the distances between the contour line of the upper eyelid and the contour line of the lower eyelid. For example, the eye opening / closing determination unit 40 may divide the distance from the inner corner to the outer corner of the eye into two equal parts to determine the horizontal center position, and may determine the distance between the contour line of the upper eyelid and the contour line of the lower eyelid at the center position as the eye opening amount T. Note that the eye opening amount T may use the actual dimension or a normalized value.

[0028] 4 is a schematic diagram for explaining the eye-opening reference value and the eye-closing reference value. The eye-opening / closing determination unit 40 first uses the default setting of the eye-opening reference value and the eye-closing reference value to divide the eye-opening amount T calculated from the eye images acquired from time to time into data of the eye-opening state and data of the eye-closing state.

[0029] The eye open / closed determination unit 40 calculates the average value of the data in the eye open state and the average value of the data in the eye closed state, and sets a margin for each average value as the eye open reference value ST1 and the eye closed reference value ST2. For example, by periodically updating the eye open reference value ST1 and the eye closed reference value ST2, it is possible to set appropriate reference values ​​according to the driver and the surrounding environment (such as brightness).

[0030] The degree of eye opening is set to 100% when the amount of eye opening T is the eye opening reference value, and 0% when the amount of eye opening T is the eye closing reference value. The eye opening / closing determination unit 40 sets the difference value obtained by subtracting the eye closing reference value from the eye opening reference value to 100%, and calculates the ratio of the amount of eye opening T to the difference value to determine the degree of eye opening / closing. When the amount of eye opening T exceeds the eye opening reference value or is smaller than the eye closing reference value, the amount of eye opening T is replaced and calculated so that the eye opening reference value and the eye closing reference value are the upper and lower limits, respectively, thereby suppressing errors when recognizing the eye image and enabling stable calculation of the degree of eye opening / closing. The eye opening / closing determination unit 40 may also divide the range from the eye opening reference value to the eye closing reference value into, for example, 10 equal parts, and evaluate the degree of eye opening / closing in multiple stages.

[0031] The eye opening / closing determination unit 40 switches between the eye opening reference value ST1 and the eye closing reference value ST2 depending on the facial direction of the subject input from the facial direction detection unit 20. Fig. 5 is a schematic diagram for explaining the eye opening amount that changes depending on the facial direction. Even if the left eye is opened to the exact same degree in the outline of the left eye as seen from the front and the outline of the left eye as seen from diagonally forward to the left, the eye opening amount T changes depending on the undulating shape of the upper and lower eyelids.

[0032] When the subject's face rotates around the Z axis and the yaw angle of the face changes, the eye opening amount T is expected to change slightly less than when viewed from the front. Also, for example, when the subject's face is photographed from diagonally above and in the front, it is likely to be affected by the upper eyelids, and the change in the eye opening amount T due to the change in the yaw angle of the face direction is expected to be more noticeable.

[0033] The eye open / closed determination unit 40 changes the settings of the eye open reference value ST1 and the eye closed reference value ST2 according to the face direction. For example, the eye open / closed determination unit 40 sets the eye open reference value ST1 to a smaller value as the yaw angle change of the face direction increases. Also, the eye open / closed determination unit 40 sets the eye closed reference value ST2 to a larger value as the yaw angle change of the face direction increases. The eye open / closed determination unit 40 may set the eye open reference value ST1 and the eye closed reference value ST2 to, for example, several stages according to the yaw angle change of the face direction. Note that the eye open / closed determination unit 40 may change only the eye open reference value ST1, assuming that the effect of the yaw angle change of the face direction is small, with the setting of the eye closed reference value ST2 kept constant.

[0034] Next, the operation of the eye open / closed determination unit 40 for determining the open / closed state of both eyes or one eye will be described. FIG. 6 is a schematic diagram for explaining the determination of the open / closed state of both eyes or one eye. When the face direction has changed significantly and it has been determined that it is difficult to determine the open / closed state of one eye, the eye open / closed determination unit 40 determines the open / closed state of only the other eye. When the face direction has changed significantly and one eye is hidden and cannot be seen or the outline of one eye is indistinguishable, the eye open / closed determination unit 40 determines that it is difficult to determine the open / closed state of one eye because the degree of opening / closing of the eyes becomes indefinite for that eye.

[0035] When the eye open / closed state determination unit 40 determines that it is possible to determine the open / closed state of both eyes while determining the open / closed state of only one eye, it returns to determining the open / closed state of both eyes. The eye open / closed state determination unit 40 may return to determining the open / closed state of both eyes a predetermined time (e.g., one second) after it becomes possible to calculate the open / closed degree of one eye whose open / closed degree was indefinite.

[0036] Next, the operation of the eye open / closed determination device 100 will be described based on the process of determining whether the eyes are open or closed. Fig. 7 is a flow chart showing the procedure of the process of determining whether the eyes are open or closed in the eye open / closed determination device 100. The video acquisition unit 10 of the eye open / closed determination device 100 acquires video data from the camera 60 (S1). The face direction detection unit 20 recognizes the face part in the video data input from the video acquisition unit 10, and detects the face direction of the subject to be photographed (S2). In addition, the eye image extraction unit 30 recognizes the part of each eye in the video data input from the video acquisition unit 10, and extracts an eye image obtained by cutting out the area of ​​each eye (S3).

[0037] The eye opening / closing determination unit 40 switches between the eye opening reference value ST1 and the eye closing reference value ST2 (S4) based on the face direction detected by the face direction detection unit 20. The eye opening / closing determination unit 40 calculates the ratio of the eye opening amount T to the difference value between the switched eye opening reference value ST1 and the switched eye closing reference value ST2, determines whether the eyes are open or closed (S5), and ends the process.

[0038] The eye opening / closing determination device 100 has multiple eye opening reference values ​​ST1 and eye closing reference values ​​ST2, for example, in several stages, and by switching between the eye opening reference value ST1 and the eye closing reference value ST2 based on the direction of the face, it is possible to improve the accuracy of determining the eye opening / closing state based on the captured face image.

[0039] When determining the eye open / closed state using video data obtained by photographing the face of an object from an obliquely upward forward position, since there are individual differences in eye shape and eyelid shape, etc., when the yaw angle of the face direction changes, the reference value of the eye open / closed degree changes, and it may not be possible to accurately determine the eye open / closed state. The eye open / closed state determination device 100 can increase the accuracy of determining the eye open / closed state based on the face image by switching the eye open reference value ST1 and the eye closed reference value ST2 according to the change in the yaw angle of the face direction.

[0040] When the eye opening / closing determination device 100 determines that the face direction has changed significantly and it is difficult to determine the opening / closing state of one of the eye images, it can continue to determine the opening / closing state of the eyes while suppressing deterioration in accuracy by determining the opening / closing state of the eyes based on the other eye image.

[0041] When the eye open / closed state determination device 100 determines that it is possible to determine the open / closed state of both eyes while determining the open / closed state of only the other eye, it returns to determining the open / closed state of the eyes based on both eyes.

[0042] The eye open / closed state determination device 100 may be configured to switch back to determining the open / closed state using both eyes a predetermined time (e.g., one second) after it becomes possible to calculate the open / closed state of one eye for which the open / closed state was uncertain. If the operation of switching back to determining the open / closed state using both eyes is performed immediately from a video frame in which it becomes possible to calculate the open / closed state of one eye for which the open / closed state was uncertain, the open / closed state of one eye may become uncertain again in the subsequent frame, causing the determination result to become unstable, for example, if the orientation of the subject's face is not stable.

[0043] For this reason, the eye opening / closing judgment device 100 switches to judging the open / closed state using the other eye, whose open / closed degree is correctly calculated, the moment the open / closed degree of one eye becomes uncertain, and when returning to judgment using both eyes, the switch is made a predetermined time after the open / closed degree of one eye becomes calculable.

[0044] Furthermore, when returning to the determination using both eyes, the eye opening / closing determination device 100 may switch from determining the eye opening / closing state using one eye to determining the eye opening / closing state using both eyes when the eye is closed and then opened by blinking. The eye for detecting blinking may not be limited to one eye or both eyes at the same time. By detecting blinking, the eye opening / closing determination device 100 can evaluate that it is able to calculate the opening / closing state of both eyes at a more stable timing, and therefore can reliably determine the opening / closing state of the eyes after switching.

[0045] The degree of eye opening and closing may not be calculated correctly due to reflections of the eyeglass lenses or light reflections. Even if the eye opening and closing determination device 100 cannot calculate the degree of opening and closing for one eye due to reflections of the eyeglass lenses or light reflections, if the other eye is not affected by the eyeglass lenses, it can calculate the degree of opening and closing for only the unaffected eye and determine the eye opening and closing state. The eye opening and closing determination device 100 estimates the position of the eye from the previous frame in the video data, and can determine that there is reflection of the eyeglass lenses or light reflection when there is a pixel with high brightness near the position of the eye.

[0046] (Modification) The eye opening / closing determination device 100 may update the eye opening reference value ST1 and the eye closing reference value ST2 periodically, for example, every hour or every day. The eye opening / closing determination unit 40 of the eye opening / closing determination device 100 uses, for example, the current eye opening reference value and the eye closing reference value to divide the eye opening amount T calculated from the eye images acquired from time to time into data of the eye opening state and data of the eye closing state. Each reference value may be stored in the storage unit 101 or the like in association with each time period.

[0047] The eye open / closed determination unit 40 calculates the average value of the data in the eye open state and the average value of the data in the eye closed state, and updates each average value with a margin to a new eye open reference value ST1 and closed eye reference value ST2. By periodically updating the eye open reference value ST1 and closed eye reference value ST2, the eye open / closed determination device 100 can improve the accuracy of determining the eye open / closed state for a new driver when the driver is replaced.

[0048] The eye opening / closing determination device 100 periodically updates the eye opening reference value ST1 and the eye closing reference value ST2, thereby enabling the eye opening reference value ST1 and the eye closing reference value ST2 to be correctly set in response to physical changes in the amount of eye opening T caused by changes in the amount of light at night, on cloudy or sunny days, in long tunnels, etc.

[0049] Although the example in which the eye opening / closing determination device 100 sets a plurality of eye opening reference values ​​ST1 and eye closing reference values ​​ST2 corresponding to the yaw angle change of the face direction has been shown, a plurality of eye opening reference values ​​ST1 and eye closing reference values ​​ST2 may also be set for the pitch angle and roll angle changes. Also, the eye opening / closing determination device 100 may set a plurality of eye opening reference values ​​ST1 and eye closing reference values ​​ST2 for a combination of two or more of the yaw angle change, the pitch angle change, and the roll angle change, and for each of the combined angle changes.

[0050] Next, features of the eye opening / closing determination device 100 and the eye opening / closing determination method according to the above-described embodiment and modified examples will be described. The eye opening / closing determination device 100 includes a video acquisition unit 10, a face direction detection unit 20, an eye image extraction unit 30, and an eye opening / closing determination unit 40. The video acquisition unit 10 acquires video data from a camera 60. The face direction detection unit 20 detects the face direction based on the face image in the video data acquired by the video acquisition unit 10. The eye image extraction unit 30 extracts an eye image including an eye area based on the face image in the video data acquired by the video acquisition unit 10. The eye opening / closing determination unit 40 determines the eye opening / closing state based on the eye opening amount, the eye opening reference value, and the eye closing reference value acquired from the eye image. The eye opening / closing determination unit 40 has a plurality of eye opening reference values ​​and eye closing reference values ​​according to the face direction, and switches between the eye opening reference value and the eye closing reference value based on the face direction detected by the face direction detection unit 20. This allows the eye opening / closing determination device 100 to improve the accuracy of determining the eye opening / closing state based on the captured face image.

[0051] Furthermore, when it is difficult to determine the eye open / closed state for one eye image, the eye open / closed state determination unit 40 determines the eye open / closed state based on the other eye image. This allows the eye open / closed state determination device 100 to suppress deterioration in accuracy in determining the eye open / closed state.

[0052] Furthermore, when the state has changed to one of the eye images where the eye open / closed state can be determined, the eye open / closed state determination unit 40 returns to a state where the eye open / closed state is determined based on both eye images from a state where the eye open / closed state is determined based on the other eye image, thereby enabling the eye open / closed state determination device 100 to improve the accuracy of determining the eye open / closed state.

[0053] After detecting a blink, the eye open / close determination unit 40 returns to a state in which the eye open / close state is determined based on both eye images. This allows the eye open / close determination device 100 to evaluate that it has accurately calculated the open / close state of both eyes, and to reliably determine the open / close state of the eyes after switching.

[0054] The eye open / closed determination method in the eye open / closed determination device 100 includes a video acquisition step, a face direction detection step, an eye image extraction step, and an eye open / closed determination step. The video acquisition step acquires video data from a camera. The face direction detection step detects the face direction based on the face image in the video data acquired by the video acquisition step. The eye image extraction step extracts an eye image including an eye area based on the face image in the video data acquired by the video acquisition step. The eye open / closed determination step determines the eye open / closed state based on the eye opening amount, the eye open reference value, and the eye closed reference value acquired from the eye image. The eye open / closed determination step has a plurality of eye open reference values ​​and eye closed reference values ​​according to the face direction, and switches between the eye open reference value and the eye closed reference value based on the face direction detected by the face direction detection step. According to this eye open / closed determination method, it is possible to improve the accuracy of determining the eye open / closed state based on the captured face image.

[0055] The above description is based on the embodiments of the present invention. These embodiments are merely examples, and it will be understood by those skilled in the art that various modifications and changes are possible within the scope of the claims of the present invention, and that such modifications and changes are also within the scope of the claims of the present invention. Therefore, the descriptions and drawings in this specification should be treated as illustrative rather than restrictive. [Explanation of symbols]

[0056] 10 image acquisition unit, 20 face direction detection unit, 30 eye image extraction unit, 40 eye opening / closing determination unit, 60 camera, 100 eye opening / closing determination device.

Claims

1. an image acquisition unit that acquires image data from a camera; a face direction detection unit that detects a face direction based on a face image in the video data acquired by the video acquisition unit; an eye image extraction unit that extracts an eye image including an eye region based on a face image in the video data acquired by the video acquisition unit; an eye open / close determination unit that determines an eye open / closed state based on an eye open amount, an eye open reference value, and an eye closed reference value acquired from the eye image; Equipped with The eye opening / closing determination unit has a plurality of eye open reference values ​​and eye closed reference values ​​corresponding to the face direction, and switches between the eye open reference value and the eye closed reference value based on the face direction detected by the face direction detection unit.

2. 2. The eye open / closed state determination device according to claim 1, wherein when it is difficult to determine the eye open / closed state for one eye image, the eye open / closed state determination unit determines the eye open / closed state based on the other eye image.

3. The eye opening / closing determination device according to claim 2, characterized in that, when the state changes to one of the eye images and it becomes possible to determine the eye opening / closing state, the eye opening / closing determination unit returns to a state in which the eye opening / closing state is determined based on the other eye image, but rather to a state in which the eye opening / closing state is determined based on both eye images.

4. 4. The eye opening / closing determination device according to claim 3, wherein the eye opening / closing determination unit returns to a state in which the eye opening / closing state is determined based on the images of both eyes after detecting a blink of an eye.

5. an image acquisition step of acquiring image data from a camera; a face direction detection step of detecting a face direction based on a face image in the video data acquired by the video acquisition step; an eye image extraction step of extracting an eye image including an eye region based on a face image in the video data acquired by the video acquisition step; an eye open / closed state determination step of determining an eye open / closed state based on an eye openness amount, an eye open reference value, and an eye closed reference value acquired from the eye image; Equipped with The eye opening / closing determination step has a plurality of the eye opening reference values ​​and the eye closing reference values ​​corresponding to the face direction, and switches the eye opening reference value and the eye closing reference value based on the face direction detected by the face direction detection step.

Images