Pupil detection device and pupil detection method

Abstract

To enhance calculation efficiency and detection accuracy with a simple device configuration.SOLUTION: A sight line detection device 1 comprises: a camera 10 which acquires face images at continuous timing by imaging the face of an object person A; a light source 13 which emits light toward the face of the object person A; and an image processing device 20 which processes the face image acquired by the camera 10 at the irradiation timing of light. The image processing device 20 comprises: a sight line detection unit 23 which detects a position of the pupil of the object person A on the face image; and a pupil position prediction unit 25 which predicts the position of the pupil during eye closure of the object person A by searching for a position of a feature part of the eye in an eye closure state on the face image. The sight line detection unit 23 detects the position of the pupil by tracking the pupil by utilizing the position of the pupil predicted by the pupil position prediction unit 25.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a pupil detection device and a pupil detection method for detecting the position of a pupil from an image of a subject. [Background technology]

[0002] Conventionally, devices have been used that acquire an image of a subject's face, including the eyes, and detect the position of the subject's pupil based on that image (see, for example, Patent Document 1 below). This device detects the position of the pupil based on a difference image obtained by subtracting a bright pupil image, in which the pupil is brightly captured, from a dark pupil image, in which the pupil is darkly captured. In order to improve computational efficiency and accuracy, a window is set in the next frame of the image from the pupil position detected in the previous frame of the image to a portion where the pupil is predicted to be located, and the pupil is searched for within that window. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-268026 Summary of the Invention [Problem to be solved by the invention]

[0004] In the device described in Patent Document 1, when the subject closes their eyes by blinking or the like (hereinafter referred to as "closed eyes"), the position of the window set on the image tends to deviate from the actual position of the pupil, making it impossible to detect the pupil. Setting a larger window can avoid this situation, but it reduces calculation efficiency.

[0005] The present invention has been made in view of the above-mentioned problems, and has as its object to provide a pupil detection device and pupil detection method that can improve calculation efficiency and detection accuracy. [Means for solving the problem]

[0006] In order to solve the above problem, a pupil detection device according to one embodiment of the present invention includes a camera that captures facial images of a subject at successive timings by capturing images of the subject's face, a light source that irradiates light toward the subject's face, and a computing device that processes the facial images captured by the camera at the timings when the light is irradiated. The computing device has a pupil position detection unit that detects the position of the subject's pupil on the facial image, and a pupil position prediction unit that predicts the position of the subject's pupil when the eyes are closed by searching for the positions of facial features when the eyes are closed on the facial image. The pupil position detection unit detects the position of the pupil by tracking the pupil using the pupil position predicted by the pupil position prediction unit.

[0007] In addition, a pupil detection method according to another aspect of the present invention is a pupil detection method that uses a camera that captures facial images of a subject at successive timings by capturing images of the subject's face, a light source that irradiates light toward the subject's face, and a computing device that processes the facial images captured by the camera at the timings when the light is irradiated, and includes a pupil position detection step in which the computing device detects the position of the subject's pupil on the facial image, and a pupil position prediction step in which the computing device predicts the position of the subject's pupil when the eyes are closed by searching for the positions of facial features when the eyes are closed on the facial image, and in the pupil position detection step, the position of the pupil is detected by tracking the pupil using the pupil position predicted in the pupil position prediction step.

[0008] According to the pupil detection device of the above embodiment or the pupil detection method of the other embodiment, the positions of the pupils are detected on face images captured at successive timings by a camera, and the positions of facial features when the eyes are closed are searched for from the face images, thereby predicting the positions of the pupils when the eyes are closed on the face images. When detecting the positions of the pupils, the predicted positions of the pupils on the face images are used to track the pupils. This allows the position of the pupils to be detected by tracking them even when the subject has their eyes closed, thereby achieving highly accurate pupil detection processing with high computational efficiency.

[0009] Here, the facial feature may be the eye feature of the subject. In this case, by searching for the position of the eye feature in the facial image and predicting the position of the pupil from that position, the accuracy of predicting the pupil position can be improved, thereby realizing a more accurate pupil detection process.

[0010] The pupil position prediction unit may also predict the pupil position using a machine learning model that uses a neural network. In this case, the accuracy of predicting the pupil position in the face image can be reliably improved using a simple learning method, thereby realizing stable pupil detection processing.

[0011] The pupil position prediction unit may also predict the position of the pupil by inputting a partial image cut out from a face image to a machine learning model as input data and using the machine learning model to search for the position of the partial image containing facial features. In this case, by using the partial image cut out from the face image as input data, the position of the pupil on the face image can be predicted by simple processing. As a result, the computational efficiency of the pupil detection process can be further improved.

[0012] The pupil position prediction unit may also crop a partial image to a size that includes all of the facial features. In this case, the entire facial feature can be included in the cropped partial image of the face image, improving the accuracy of predicting the pupil position. As a result, more stable pupil detection processing can be achieved.

[0013] The computing device may further include a model learning unit that trains the machine learning model, and the model learning unit may cut out a facial image acquired when the subject's eyes are closed based on the position of the pupil detected by the pupil position detection unit just before the subject closes their eyes, and use the cut-out facial image as training data to train the machine learning model. In this way, appropriate training data can be created, and by training using the training data, the prediction accuracy of the pupil position can be improved. As a result, more stable pupil detection processing can be achieved.

[0014] The computing device may further include a pupil distance detection unit that detects the distance from the camera to the pupil, and the pupil position prediction unit may variably set the size of the partial image cutout as input data according to the pupil distance. In this case, the size of the partial image used as input data can be appropriately set, thereby improving the accuracy of predicting the pupil position. As a result, a more stable pupil detection process can be achieved.

[0015] The computing device may further include a pupil distance detection unit that detects the distance from the camera to the pupil, and the model learning unit may variably set the crop size of the facial image used as training data according to the pupil distance. This allows the crop size of the facial image used as training data to be appropriately set, thereby improving the accuracy of predicting the pupil position. As a result, a more stable pupil detection process is realized.

[0016] The model learning unit may also identify an area in the cropped face image where facial features exist, and set the crop size of the face image as training data based on the identified area. This allows the crop size of the face image used as training data to be appropriately set according to the area of ​​the facial features, thereby improving the accuracy of predicting the pupil position. As a result, more stable pupil detection processing is achieved.

[0017] The model learning unit may also identify an area in the cut-out face image where facial features exist, and set the cut-out position of the face image as training data so that the identified area is located at the center of the image. This allows the cut-out position of the face image used as training data to be appropriately set according to the area of ​​the facial features, thereby improving the accuracy of predicting the position of the pupil. As a result, a more stable pupil detection process is realized.

[0018] Furthermore, the machine learning model is a model that predicts the amount of displacement of a partial image including facial features relative to a face image cut out based on the position of the pupil detected by the pupil position detection unit immediately before the subject closes their eyes, and the model learning unit may generate shifted images while shifting the cut-out face image, and train the learning model using the shifted images and the amount of displacement of the shifted images as training data. In this case, the prediction accuracy of the pupil position on the face image can be improved regardless of the magnitude of the displacement on the face image between the multiple partial images used as input data. As a result, more stable pupil detection processing can be achieved.

[0019] Furthermore, when the subject closes his / her eyes and then opens them again, the pupil position detection unit may detect the pupil position by setting a window on the face image based on the pupil position predicted by the pupil position prediction unit at the timing when the subject closes his / her eyes. Conventionally, when the subject opens his / her eyes after closing them, it has tended to be impossible to detect the pupil for several frames of images. With this configuration, pupil detection can be resumed stably and without delay from the frame immediately after the subject opens his / her eyes.

[0020] Furthermore, the pupil position detection unit may detect the position of the pupil by tracking the position of the pupil using the position of the pupil detected on the face image of successive frames, and if detection of the position of the pupil on the face image of the immediately preceding frame fails, may track the position of the pupil using the position of the pupil predicted by the pupil position prediction unit. With this configuration, when successively detecting the position of the pupil, even if the subject closes their eyes by blinking or the like and the image of the pupil disappears on the image, the position of the pupil can be stably tracked, and when the subject opens their eyes and the image of the pupil reappears on the image, the position of the pupil can be stably detected. [Effects of the Invention]

[0021] According to the present invention, it is possible to improve the calculation efficiency and detection accuracy of pupil detection processing with a simple device configuration. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a perspective view showing a gaze detection device according to an embodiment; [Figure 2] FIG. 2 is a plan view showing the lens portion of the camera. [Figure 3] FIG. 1 is a diagram illustrating a hardware configuration of an image processing apparatus according to an embodiment. [Figure 4] 1 is a block diagram showing a functional configuration of a gaze detection device according to an embodiment. [Figure 5] 5 is a diagram showing an image of a partial image created by the partial image creating unit 24 of FIG. 4. FIG. [Figure 6] FIG. 2 is a diagram showing the layer structure of a machine learning model 1 used by a pupil position prediction unit 25. [Figure 7] FIG. 2 is a diagram showing the layer structure of a machine learning model 2 used by a pupil position prediction unit 25. [Figure 8] 10 is a diagram showing an example of closed-eye images GPL1 and GPR1 of the left and right pupils created by the learning image creation unit 26 for the face image GF2. FIG. [Figure 9] FIG. 10 is a diagram showing an example of learning data "learning image 3" created by the learning image creation unit 26 for the facial image GF2. [Figure 10] 4 is a flowchart showing the operation procedure of the gaze detection device 1. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, preferred embodiments of a pupil detection device and a pupil detection method according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts will be given the same reference numerals, and duplicated explanations will be omitted.

[0024] [Configuration of gaze detection device] First, the configuration of gaze detection device 1, which is a pupil detection device according to an embodiment, will be described with reference to FIGS. 1 to 4. Gaze detection device 1 is a computer system that detects the pupil and corneal reflex of a subject by capturing an image of the subject's face and uses the detection results to detect the subject's gaze direction. This device implements a pupil detection method according to this embodiment. The subject is a person whose gaze direction is to be detected, and can also be called a test subject. The purposes for which gaze detection device 1 and pupil detection method are used are not limited in any way. For example, gaze detection device 1 can be used to detect distracted driving, check whether the driver is checking the safety of the side mirrors or rearview mirror, detect driver drowsiness, investigate the level of interest in products, input data into computers used in amusement machines, etc., diagnostic devices for diagnosing autism in infants, communication systems used between remote locations, and observation devices for remotely observing events.

[0025] As shown in FIG. 1, the gaze detection device 1 includes a pair of cameras 10 that function as stereo cameras and an image processing device (arithmetic device) 20. In the following, as needed, the pair of cameras 10 will be referred to as a left camera 10 on the left side of the subject A. L and the right camera 10 on the right side of the subject A. R In this embodiment, the gaze detection device 1 further includes a display device 30 that is the object that subject A looks at. However, since the purpose of use of the gaze detection device 1 is not limited to the above, the object in the line of sight of subject A is not limited to the display device 30 and could be, for example, the windshield of a car. Therefore, the display device 30 is not an essential element of the gaze detection device 1. Each camera 10 is connected to an image processing device 20 wirelessly or by wire, and various data or commands are transmitted and received between the camera 10 and the image processing device 20. Camera calibration is performed in advance for each camera 10.

[0026] The camera 10 photographs the face of subject A, including both eyes, to obtain a facial image. The pair of cameras 10 are arranged at a predetermined interval along the horizontal direction, and are positioned lower than the face of subject A in order to prevent reflected light from appearing in the facial image when subject A is wearing glasses. The elevation angle of the camera 10 relative to the horizontal direction is set to, for example, a range of 20 to 35 degrees, taking into consideration both reliable detection of the pupils and avoiding obstruction of subject A's field of view. Each camera 10 is calibrated in advance.

[0027] In this embodiment, the camera 10 is a camera capable of capturing a plurality of frames of facial images at successive regular intervals. The camera 10 captures an image of the subject A in response to a command from the image processing device 20, and outputs the facial images to the image processing device 20.

[0028] FIG. 2 shows a schematic diagram of the lens portion of camera 10. As shown in this figure, camera 10 includes objective lens 11 housed in circular opening 12, and light source 13 attached to the outside of opening 12. Light source 13 is a device for irradiating illumination light toward the face of subject A and consists of multiple light-emitting elements 13a and multiple light-emitting elements 13b. Light-emitting elements 13a are semiconductor light-emitting elements (LEDs) that emit light with a central wavelength of 850 nm, and are arranged in a ring shape at equal intervals along the edge of opening 12. Light-emitting elements 13b are semiconductor light-emitting elements that emit light with a central wavelength of 940 nm, and are arranged in a ring shape at equal intervals outside light-emitting elements 13a. Therefore, the distance from the optical axis of camera 10 to light-emitting elements 13b is greater than the distance from the optical axis to light-emitting elements 13a. Each of light-emitting elements 13a and 13b is arranged to emit illumination light along the optical axis of camera 10. The arrangement of light source 13 is not limited to the configuration shown in FIG. 2; other arrangements are possible as long as the camera can be considered as a pinhole model. The light source 13 emits illumination light at a timing according to a command from the image processing device 20 .

[0029] The image processing device 20 is a computer (arithmetic device) that controls the camera 10 and the light source 13 and detects the gaze direction using a facial image of the subject A. The image processing device 20 may be constructed using a stationary or portable personal computer (PC), a workstation, or other types of computers. Alternatively, the image processing device 20 may be constructed using a combination of multiple computers of any types. When multiple computers are used, these computers are connected via a communication network such as the Internet or an intranet.

[0030] 3 shows a general hardware configuration of an image processing device 20. The image processing device 20 includes a CPU (processor) 101 that executes an operating system or application programs, a main memory unit 102 consisting of ROM and RAM, an auxiliary memory unit 103 consisting of a hard disk or flash memory, a communication control unit 104 consisting of a network card or wireless communication module, an input device 105 such as a keyboard or mouse, and an output device 106 such as a display or printer.

[0031] Each functional element of the image processing device 20, which will be described later, is realized by loading predetermined software onto the CPU 101 or main memory unit 102, operating the communication control unit 104, input device 105, output device 106, etc. under the control of the CPU 101, and reading and writing data from and to the main memory unit 102 or auxiliary memory unit 103. Data or databases required for processing are stored in the main memory unit 102 or auxiliary memory unit 103.

[0032] As shown in FIG. 4, the image processing device 20 includes functional components such as a lighting control unit 21, an image acquisition unit 22, a gaze detection unit (pupil position detection unit, pupil distance detection unit) 23, a partial image creation unit 24, a pupil position prediction unit 25, a learning image creation unit 26, and a model learning unit 27. The lighting control unit 21 controls the lighting timing of the light source 13. The image acquisition unit 22 is a functional element that synchronizes the lighting timing of the light source 13 with the shooting timing of the camera 10, thereby acquiring facial image data from the camera 10 at the lighting timing. The gaze detection unit 23 is a functional element that detects the direction of the visual axis (also called the gaze) based on a gaze vector obtained from the facial image. The visual axis (gaze) is a line connecting the center of the subject's pupil and the subject's gaze point (the point at which the subject is looking). The term "visual axis" includes the meanings (concepts) of starting point, ending point, and direction. The "gaze vector" is a vector representation of the direction of the subject's visual axis and is one form of representing the "direction of the visual axis." There are no limitations on the destination of the output of the visual axis direction detection result from the image processing device 20. For example, the image processing device 20 may display the determination result as an image, graphic, or text on a monitor, store it in a storage device such as a memory or a database, or transmit it to another computer system via a communication network.

[0033] Here, the basic operation of the image processing device 20 for detecting the line of sight will be described.

[0034] First, lighting control unit 21 controls the lighting timing of light-emitting element 13a and light-emitting element 13b included in light source 13 so that they alternately light up in synchronization with the shooting timing of camera 10, and image acquisition unit 22 acquires, in accordance with the alternating lighting timing, a bright pupil image (facial image) in which the pupil appears relatively bright and a dark pupil image (facial image) in which the pupil appears relatively dark from each camera 10. Next, gaze detection unit 23 detects the position of the pupil center and the position of the corneal reflex in the facial image from each camera 10, using a difference image (or division image) of the bright pupil image and the dark pupil image from each camera 10 as the target. Then, gaze detection unit 23 acquires the three-dimensional coordinates of the centers of the left and right pupils of subject A and the distances from each camera 10 to the left and right pupils by performing calculations using the positions of the pupil centers detected from the facial images from the two cameras 10 using a stereo method. Furthermore, gaze detection unit 23 calculates the visual axes (gaze vectors) of the left and right eyes based on the calculated positions of the pupil center and corneal reflex in the face image captured by either camera 10, and the three-dimensional coordinates of the left and right pupils. Furthermore, gaze detection unit 23 may calculate a gaze point on a predetermined visual symmetry plane by referring to the calculated visual axes. The above process is repeatedly performed on pairs of alternating bright and dark pupil images.

[0035] Here, the gaze detection unit 23 detects the position of the pupil center and the position of the corneal reflex for the bright pupil image and the dark pupil image within a window of a predetermined size set in the bright pupil image and the dark pupil image. The position of this window is set to include the position of the pupil based on the position of the pupil in the current frame predicted using at least the three-dimensional position of the pupil detected in the previous frame of the face image to be detected (for example, the method described in Japanese Patent Application Laid-Open No. 2007-268026 is adopted). In other words, the window position is set to track the position of the pupil between consecutive frames. However, if the gaze detection unit 23 fails to detect the position of the pupil center for the face image in the previous frame, the position of the window on the face image in the current frame is set using the three-dimensional position of the pupil predicted by the pupil position prediction unit 25, which will be described later. Here, the gaze detection unit 23 determines whether its detection of the pupil center position has failed (i.e., whether the eye is closed due to blinking or the like) when the area of ​​the pupil (e.g., the number of pixels in the pupil region of a binarized image using a threshold value) detected on the difference image (or division image) between the bright pupil image and the dark pupil image approaches zero, indicating that the pupil is not present. Generally, eye closure due to blinking or the like occurs instantaneously, so the pupil suddenly becomes undetectable in a certain frame. Therefore, if the head does not move between immediately before and immediately after eye closure, the eye or pupil will be in the same position immediately before and immediately after eye closure. Therefore, the closed eye image can be associated with the pupil position. The gaze detection unit 23 detects the pupil position and the pupil position prediction unit 25 predict the pupil position in parallel for consecutively acquired face images. In this case, if the gaze detection unit 23 determines that detection in the previous frame has failed, the prediction result by the pupil position prediction unit 25 is used to set the window position in the current frame. On the other hand, if gaze detection unit 23 fails to detect the pupil position in a certain frame, it is also possible to cause pupil position prediction unit 25 to execute prediction processing for that frame. In this case, gaze detection unit 23 uses the prediction result by pupil position prediction unit 25 to set the window position in the next frame.

[0036] Next, we will explain the functions of the other components of the image processing device 20. Note that the processes performed by the partial image creation unit 24, pupil position prediction unit 25, learning image creation unit 26, and model learning unit 27, which will be described below, are performed separately for the bright pupil image and the dark pupil image continuously acquired by each of the two cameras 10.

[0037] The partial image creation unit 24 cuts out a rectangular partial image of a predetermined size from the facial image acquired from each camera 10 by the image acquisition unit 22. For example, if the facial image is 150 pixels wide by 80 pixels high, the partial image is 30 pixels wide by 20 pixels high. The size of the partial image is set to a size that includes the entire eyelashes, which are a characteristic feature of the eyes on the face of subject A. At this time, the partial image creation unit 24 may variably set the cutout size of the partial image so that the entire eye feature is included, based on the distance from the camera 10 to the pupil detected or predicted by the gaze detection unit 23 or the pupil position prediction unit 25 for the frame immediately before the facial image to be processed, and convert the cutout partial image into image data of a predetermined size (e.g., 8 pixels by 5 pixels). The partial image creation unit 24 then repeatedly shifts the cutout region of the partial image in two-dimensional directions (e.g., by shifting 10 pixels horizontally and 5 pixels vertically) within the facial image to create multiple partial images that overlap each other horizontally and vertically.

[0038] 5 shows an image of a partial image created by the partial image creating unit 24. For example, if a facial image G F1 is 150 pixels wide x 80 pixels high, the cutout size of the partial image is 30 pixels wide x 20 pixels high, the shift amount in the horizontal direction is 10 pixels, and the shift amount in the vertical direction is 5 pixels, the partial image creation unit 24 creates a facial image G F1 A total of 169 partial images were taken from the GP F1 Create a.

[0039] Furthermore, if the pupil position on the face image has been predicted by pupil position prediction unit 25 in the frame immediately before the frame to be processed, partial image creation unit 24 sets a rectangular window centered on that position and cuts out a partial image from within that window. For example, partial image creation unit 24 sets a window of 60 pixels horizontally and 40 pixels vertically centered on the predicted position, and cuts out a partial image from within that window by shifting it by 5 pixels vertically and horizontally at a size of 40 pixels horizontally and 20 pixels vertically.

[0040] Furthermore, if the pupil position prediction unit 25 has predicted the three-dimensional position of the pupil in the frames immediately preceding the target frame and two frames prior to the target frame, the partial image creation unit 24 predicts the pupil position in the target frame using a constant velocity model under the assumption that the pupil moves at a constant velocity in three-dimensional space. In this case, the pupil position is predicted by projecting the position in three-dimensional space onto a two-dimensional facial image. The partial image creation unit 24 then sets a rectangular window centered on the predicted position and functions to extract a partial image from within the window. For example, the partial image creation unit 24 sets a window measuring 60 pixels horizontally and 40 pixels vertically centered on the predicted position, and extracts a partial image from within the window by shifting it by 5 pixels vertically and horizontally at a size of 40 pixels horizontally and 20 pixels vertically.

[0041] Pupil position prediction unit 25 predicts the position of the pupil on the face image of subject A with the eyes closed by searching for the position of the characteristic part of the eye (the eyelashes in this embodiment) on the face image with the eyes closed, based on the plurality of partial images created by partial image creation unit 24 for the face images of each camera 10. This prediction of the pupil position is performed for each of the left and right eyes.

[0042] That is, the pupil position prediction unit 25 predicts the pupil position using a machine learning model that uses a CNN (convolutional neural network). First, the pupil position prediction unit 25 uses the machine learning model 1 that has been trained in advance by the model training unit 27 to predict, from among multiple partial images, a partial image that has a high likelihood of being an image containing a characteristic part of the eye (eyelash part) in a closed eye state (hereinafter simply referred to as a "closed eye image"). FIG. 6 shows an example of the layer configuration of the machine learning model 1 used by the pupil position prediction unit 25. However, the layer configuration shown in FIG. 6 is only an example, and other layer configurations may be adopted. The machine learning model 1 is configured with a preprocessing unit having a convolutional layer and an activation function such as a ReLU (Rectified Linear Unit) function that converts the output of the convolutional layer, and a postprocessing unit having, in this order, a smoothing layer, a fully connected layer, an activation function such as a ReLU function that converts the output of the fully connected layer, and a fully connected layer. The machine learning model 1 processes an input image and calculates the likelihood of an eye-closed image. Here, the input partial image is converted into image data of a predetermined size based on the distance from camera 10 to the pupil so that the partial image includes the entire feature of the eye, and therefore the size of the image to be processed is unlikely to change in calculations by machine learning model 1. This means that machine learning model 1 can maintain large and small features in the feature extraction process without reducing the image, making it possible to omit the pooling layer in the pre-processing unit. Furthermore, one of the roles of the pooling layer, which is the function of reducing the effect of misalignment of feature parts in the input image, can be realized by the function of partial image creation unit 24 to create multiple partial images by overlapping them so that the entire feature is included.

[0043] Furthermore, the pupil position prediction unit 25 inputs the partial image predicted as described above into the machine learning model 2 previously trained by the model learning unit 27, and predicts the amount of two-dimensional positional deviation of the partial image from the closed-eye image based on the output value of the machine learning model 2. For example, the pupil position prediction unit 25 predicts the amount of two-dimensional positional deviation, Δx in the x-axis direction and Δy in the y-axis direction, in subpixel units, with the horizontal direction of the image as the x-axis and the vertical direction of the image as the y-axis. FIG. 7 shows an example of the layer configuration of the machine learning model 2 used by the pupil position prediction unit 25. However, the layer configuration shown in FIG. 7 is merely an example, and other layer configurations may be adopted. The machine learning model 2 is configured by a preprocessing unit having a convolutional layer and a pooling layer, and a postprocessing unit having a smoothing layer, three fully connected layers, an activation function such as a ReLU function that converts the output of the fully connected layer, and a linear function that converts the output of the fully connected layer. This machine learning model 2 processes the input image and calculates the likelihood for each two-dimensional displacement (Δx, Δy) relative to the closed-eye image.

[0044] Additionally, the pupil position prediction unit 25 calculates the pupil position in the partial image based on the deviation (Δx, Δy) from the closed-eye image predicted as described above, by referencing data relating to the relationship between the closed-eye image and the pupil position stored in advance in the image processing device 20, so as to cancel out the deviation. Here, the relational data relating to pupil position stored in advance in the image processing device 20 has different values ​​for each of the left and right pupils. This allows the pupil position prediction unit 25 to predict the pupil position in the face image when the eyes are closed. The pupil position prediction unit 25 then calculates the three-dimensional positions of the left and right pupils using a stereo method based on the predicted positions of the left and right pupils in the face image for two face images simultaneously acquired by the two cameras 10. The pupil position prediction unit 25 repeatedly performs this prediction of the left and right pupil positions for each of the face images in consecutive frames.

[0045] The learning image creation unit 26 creates learning data (training data) for pre-learning of the above-described machine learning model 1 and machine learning model 2. It is preferable that the creation of learning data by the learning image creation unit 26 is executed immediately after the completion of the calibration process for detecting the gaze direction disclosed in Japanese Patent Application Laid-Open No. 2005-230049 and the like.

[0046] That is, during the calibration process, the learning image creation unit 26 acquires multiple consecutive frames of facial images from each camera 10 before and after subject A closes his or her eyes in response to an instruction output from the image processing device 20 to subject A, and determines the position of the pupil last detected by the gaze detection unit 23 based on these facial images. The learning image creation unit 26 then acquires a facial image of a frame immediately after the frame in which the pupil position was determined, when subject A closes his or her eyes, and cuts out an image of a rectangular area of ​​a predetermined size based on the pupil position as a closed-eye image. For example, if the facial image is 150 pixels wide by 80 pixels high, the closed-eye image will be 30 pixels wide by 20 pixels high. The size of this closed-eye image is set to a size that includes the entire eyelashes, which are a characteristic feature of subject A's eyes. The relationship between the pupil position and the cutout position of the closed-eye image is preset to be different for the left and right pupils, and this relationship is stored in the image processing device 20 and referenced when the pupil position prediction unit 25 predicts the pupil position. Furthermore, the learning image creation unit 26 repeats the same process to create multiple frames of closed-eye images of the left and right pupils, and stores these as positive (correct) learning data "learning image 1" in the image processing device 20. At this time, the learning image creation unit 26 can also output instruction information from the image processing device 20 (for example, output to the display device 30) to have the subject A turn his / her face in multiple directions and close his / her eyes (for example, directly in front of the display device 30, 30 degrees to the right, and 30 degrees to the left), and create images of the closed-eye images of the left and right pupils in each direction as positive (correct) learning data "learning image 1."

[0047] Figure 8 shows the facial image G F2The left and right pupil closed eye images GP created by the learning image creation unit 26 are L1 ,GP R1 In this way, the facial image G F2 From the above, the pupil position P detected just before closing the eye is L1 ,P R1 Based on this, a closed-eye image that includes the entire eyelashes is automatically created.

[0048] In addition, the training image creation unit 26 simultaneously creates the multiple frames of positive training data "Training Image 1" described above and negative (incorrect) training data "Training Image 2." That is, the training image creation unit 26 creates multiple frames of negative training data by shifting the closed-eye image, which is the positive training data, by a predetermined amount in two dimensions from the reference position of the closed-eye image in the facial image. For example, eight patterns of images are created as negative training data by shifting one closed-eye image two-dimensionally by 5 pixels each within a range of ±5 pixels vertically and ±5 pixels horizontally. In addition, four patterns of images are created as negative training data by shifting 20 pixels each within a range of ±10 pixels vertically and ±10 pixels horizontally. However, the training image creation unit 26 may create the negative training data in an image size different from that of the positive training data. The training image creation unit 26 stores the created negative training data "Training Image 2" in the image processing device 20.

[0049] In addition, similar to the partial image creation unit 24, the learning image creation unit 26 may variably set the cut-out size of the learning data so that the entire characteristic part of the eye (the eyelashes in this embodiment) is included, based on the distance from the camera 10 to the pupil detected by the gaze detection unit 23 for the frame immediately before the face image to be processed, and convert the cut-out learning data into image data of a predetermined size (for example, 8 pixels x 5 pixels).

[0050] In addition, while creating the multiple frames of positive learning data "Learning Image 1," the learning image creation unit 26 also creates learning data "Learning Image 3" for the machine learning model 2 for determining the pupil position. Specifically, the learning image creation unit 26 creates multiple frames of images (shifted images) shifted by a predetermined amount in two dimensions from the reference position of the closed-eye image, which is the positive learning data in the face image, as "Learning Image 3." However, the shift amount of "Learning Image 3" is set to a smaller value than the shift amount of "Learning Image 2." For example, for one closed-eye image, multiple images shifted by one pixel in two dimensions within a range of ±2 pixels vertically and ±2 pixels horizontally are created as the learning data "Learning Image 3." However, the learning image creation unit 26 may create the learning data "Learning Image 3" with an image size larger than the learning data "Learning Image 1." The learning image creation unit 26 stores the created multiple learning data “learning image 3” in the image processing device 20 together with the shift amount Δx in the x-axis direction and the shift amount Δy in the y-axis direction of each learning image relative to the closed-eye image.

[0051] Figure 9 shows a facial image G of subject A taken with his eyes closed. F2 1. An example of the learning data "learning image 3" created by the learning image creation unit 26 for the face image G F2 The closed-eye image, which includes the entire eyelashes, is shifted by ±2 pixels vertically and ±2 pixels horizontally to create the training image GP. R2 ~GP R4 will be created automatically.

[0052] The model learning unit 27 uses the learning data created by the learning image creation unit 26 to train the machine learning model 1 and the machine learning model 2 used by the pupil position prediction unit 25. That is, the model learning unit 27 trains the machine learning model 1 using a plurality of positive learning data "learning images 1" and a plurality of negative learning data "learning images 2" stored in the image processing device 20. The model learning unit 27 also trains the machine learning model 2 using a combination of a plurality of learning data "learning images 3" stored in the image processing device 20 and the deviation amounts Δx and Δy of each learning image as training data for positive images.

[0053] Next, the operation procedure of the gaze detection device 1 will be described, along with the steps of the pupil detection method according to this embodiment. FIG.

[0054] First, when the gaze direction detection process for subject A is started, the lighting control unit 21 and image acquisition unit 22 of the image processing device 20 start controlling the lighting timing of the light source 13 and controlling the acquisition of images from the camera 10 (step S101). Thereafter, the learning image creation unit 26 of the image processing device 20 acquires multiple images each of "learning image 1," "learning image 2," and "learning image 3," which are learning data, during the calibration process for gaze direction detection (step S102). Next, the model learning unit 27 of the image processing device 20 uses the learning data to train machine learning model 1 and machine learning model 2 (step S103).

[0055] Then, the gaze detection unit 23 of the image processing device 20 performs a gaze direction detection process (including pupil position detection) targeting the face images of frames continuously acquired from the camera 10 (step S104; pupil position detection step). In parallel with this, the partial image creation unit 24 of the image processing device 20 acquires multiple partial images from the face image (step S105). Thereafter, the pupil position prediction unit 25 of the image processing device 20 performs a pupil position prediction process based on the multiple partial images (step S106; pupil position prediction step). Furthermore, the image processing device 20 determines whether or not a face image of the next frame exists (step S107). If a face image of the next frame exists, the processes of steps S104 to S106 are repeated. Here, the window setting in the face image in step S104 is set so as to track the three-dimensional position of the pupil predicted by the process of step S106 targeting the frame previous to the face image to be processed.

[0056] The effects of the gaze detection device 1 according to the embodiment of the present disclosure and the pupil detection method using the same will be described.

[0057] According to the gaze detection device 1, the position of the pupil is detected on the face image captured at successive timings by the camera 10, and the position of the eye feature when the eyes are closed is searched for in the face image, thereby predicting the position of the pupil when the eyes are closed on the face image. When detecting the position of the pupil, a window is set using the predicted pupil position on the face image, and the pupil is tracked. This allows the pupil position to be detected by tracking the pupil even when the subject A has his / her eyes closed, so the window size can be made relatively small, enabling highly accurate pupil detection processing with high computational efficiency. In addition, because a complex optical system is not required, pupil detection processing can be achieved with a simple device configuration.

[0058] A conventional gaze detection device described in JP 2017-102731 A includes a subject detection device that detects the three-dimensional position of the subject's head, a narrow-field camera, and a pan-tilt mechanism that adjusts the attitude of the narrow-field camera. In this device, the attitude and zoom value of the narrow-field camera are controlled based on the three-dimensional position of the subject's head detected by the subject detection device, and the position of the subject's pupil is detected using bright pupil images and dark pupil images acquired by the narrow-field camera with the controlled attitude and zoom value. However, this conventional device requires two types of detection optical systems, which tends to complicate the device configuration. In contrast, the gaze detection device 1 of this embodiment does not require a complex optical system, allowing pupil detection processing to be achieved with a simple device configuration.

[0059] In this embodiment, a machine learning model using a neural network is used to predict the pupil position. In this case, the accuracy of predicting the pupil position in a face image can be reliably improved by a simple learning method, and stable pupil detection processing can be achieved.

[0060] Furthermore, in this embodiment, a partial image cut out from a face image is input as input data to a machine learning model, and the machine learning model is used to search for the position of the partial image containing eye features, thereby predicting the position of the pupil. In this case, by using the partial image cut out from the face image as input data, the position of the pupil on the face image can be predicted by simple processing. As a result, the computational efficiency of the pupil detection process can be further improved.

[0061] In this case, the partial image used as input data for the machine learning model is cropped to a size that includes the entire eye feature. In this case, the entire eye feature can be included in the cropped partial image of the face image, improving the accuracy of predicting the pupil position. As a result, more stable pupil detection processing is achieved.

[0062] The image processing device 20 further includes a model learning unit 27 that trains a machine learning model. The model learning unit 27 cuts out a facial image acquired when subject A has his / her eyes closed based on the position of the pupil detected immediately before subject A closes his / her eyes, and uses the cut-out facial image as training data to train the machine learning model. This makes it possible to create training data in which the relationship between the cut-out facial image and the pupil position is appropriately set, and by training using this training data, the prediction accuracy of the pupil position can be improved. As a result, a more stable pupil detection process is realized. Furthermore, this embodiment enables automatic annotation, in which training data is automatically acquired during the calibration process. This eliminates the need to manually acquire training data, reducing the user's workload during training.

[0063] The image processing device 20 also has a function for detecting the distance from the camera 10 to the pupil, and variably sets the size of the cutout of the partial image used as input data according to the distance to the pupil. This function allows the size of the partial image used as input data to be appropriately set, thereby improving the accuracy of predicting the pupil position. As a result, a more stable pupil detection process is realized. Furthermore, in the machine learning model used to predict the pupil position, it is possible to omit the pooling layer that is provided to enable prediction even when the image size changes. This allows for faster calculations during learning and prediction.

[0064] The image processing device 20 also has a function for detecting the distance from the camera 10 to the pupil, and variably sets the crop size of the face image used as training data according to the distance to the pupil. This function allows the crop size of the face image used as training data to be appropriately set, thereby improving the accuracy of predicting the pupil position. As a result, a more stable pupil detection process is realized.

[0065] The machine learning model 2 used in the image processing device 20 is a model that predicts the two-dimensional shift amount of the position of a partial image containing eye features, and the model learning unit 27 generates shifted images by shifting the extracted face image two-dimensionally, and trains the learning model using the shifted images and the two-dimensional shift amounts of the shifted images as training data. In this case, the prediction accuracy of the pupil position on the face image can be improved regardless of the magnitude of the shift amount on the face image between multiple partial images used as input data during prediction. As a result, more stable pupil detection processing is realized.

[0066] Furthermore, the gaze detection unit 23 of the image processing device 20 may detect the position of the pupil by tracking the position of the pupil using the position of the pupil detected on the face image of successive frames, and if detection of the position of the pupil on the face image of the immediately preceding frame fails, may track the position of the pupil using the position of the pupil predicted by the pupil position prediction unit 25. With this configuration, when successively detecting the position of the pupil, the position of the pupil can be stably tracked even if the subject A closes his / her eyes by blinking or the like and the image of the pupil disappears on the image, and when the subject A opens his / her eyes and the image of the pupil appears on the image again, the pupil position that was tracked when the eyes were closed can be used for window setting, thereby enabling the position of the pupil to be detected stably and without delay immediately after the eyes are opened.

[0067] The present invention is not limited to the above-described embodiment, and the configuration of the above-described embodiment can be modified in various ways.

[0068] For example, the image processing device 20 according to the above embodiment is equipped with a learning image creation unit 26 and a model learning unit 27, but the functions of either or both of the learning image creation unit 26 and the model learning unit 27 may be implemented in a computer other than the image processing device 20, and either or both of the creation of training data and the learning of the machine learning model may be performed by the other computer.

[0069] Furthermore, in this embodiment, the eye feature that is the search target may be the eyelid, eyebrow, etc. instead of the eyelashes. Furthermore, the search target may be a facial feature such as the nostrils instead of the eye feature. When a facial feature such as the nostrils is the search target, it is preferable that the image processing device 20 stores in advance information on the relative three-dimensional positions between the pupil and the facial feature, and further predicts the position of the pupil based on information on the rotational state and movement state of the head obtained from the pupil and the facial feature.

[0070] Furthermore, while the gaze detection device 1 detects the position of the pupil by tracking the pupil using the predicted pupil position and setting the position of a window on the face image, a configuration in which the attitude and zoom value of the narrow-field camera can be controlled may be adopted, similar to the configuration described in JP 2017-102731 A. In this case, the gaze detection device 1 is configured to track the pupil by controlling the attitude and zoom value of the narrow-field camera using the predicted three-dimensional position of the pupil. Even with this modification, the pupil can be tracked even when subject A has his / her eyes closed, achieving highly accurate pupil detection processing with high computational efficiency.

[0071] Furthermore, although the image processing device 20 of the gaze detection device 1 predicts the pupil position using a machine learning model that uses a neural network, it may also operate to predict the pupil position using other image processing such as template matching. Even with such an operation, it is possible to search for the positions of eye features in a facial image and predict the pupil position based on these positions.

[0072] Furthermore, instead of calculating the three-dimensional position of the pupil using a stereo method, the image processing device 20 may function to determine the three-dimensional position of the pupil using the position of the pupil, the position of the nostrils, the position of other facial features on the image, or the position of a marker attached to the face, using the technique described in Japanese Patent Application Laid-Open No. 2007-268164. Furthermore, the image processing device 20 may function to determine the three-dimensional position of the pupil relative to the overall structure (shape) of the face by applying a dot pattern to the face and detecting the pupil while simultaneously grasping the overall shape of the face using this dot pattern. Furthermore, the image processing device 20 may function to determine the three-dimensional position of the pupil relative to the overall structure (shape) of the face by using a TOF (Time Of Flight) camera to determine the distance to the face for each pixel.

[0073] In creating training data using the image processing device 20 according to the embodiment described above, closed-eye images are generated that include the entire eyelashes, which are a characteristic feature of the eye. If the size of the closed-eye image is too small, the eyelashes will protrude from the image, reducing the accuracy of the machine learning model's prediction of the pupil position. Conversely, if the size of the closed-eye image is too large, features other than the eyelashes (e.g., eyebrows, hair, etc.) will be included in the image, reducing the accuracy of the machine learning model's prediction of the pupil position due to the influence of these features. The image processing device 20 may have a function for automatically optimizing the size of the closed-eye image to prevent a reduction in the accuracy of the prediction of the pupil position due to the size of the closed-eye image. For example, the training image creation unit 26 of the image processing device 20 first acquires a relatively large closed-eye image, then performs image analysis to identify the area where the eyelashes are present in the acquired closed-eye image, and sets the size of the closed-eye image so that a gap larger than the shift amount of the partial image is secured around the eyelashes. This function allows the crop size of the face image used as training data to be set appropriately according to the eyelash area, improving the accuracy of predicting the pupil position, resulting in more stable pupil detection processing.

[0074] Furthermore, if the pupil position and eyelash position immediately before closing the eye in the closed-eye image are misaligned, the eyelash position may be located at the edge of the closed-eye image, potentially reducing the accuracy of the pupil position prediction by the machine learning model. In other words, positioning the eyelashes near the center of the closed-eye image is considered preferable for improving the accuracy of pupil position prediction. In this case, the closed-eye image can be set to a relatively small size, thereby shortening the calculation time for predicting the pupil position. To prevent a decrease in the accuracy of pupil position prediction due to misalignment of the eyelashes in the closed-eye image, the learning image creation unit 26 of the image processing device 20 identifies the area where the eyelashes are present in the closed-eye image once acquired through image analysis and sets the cropping area of ​​the closed-eye image so that the eyelashes are located in the center of the closed-eye image. In this case, the learning image creation unit 26 also stores data indicating the relationship between the pupil position and the cropping position of the closed-eye image in the image processing device 20. This function allows the position of the facial image cutout used as training data to be appropriately set according to the eyelash area, improving the accuracy of predicting the pupil position, resulting in more stable pupil detection processing.

[0075] Furthermore, the gaze detection unit 23 of the image processing device 20 of this embodiment may set a window for pupil position detection as follows. Specifically, when subject A closes his / her eyes and then opens them again, the gaze detection unit 23 uses the predicted pupil position tracked by the pupil position prediction unit 25 to set a window of a predetermined size in the gaze detection unit 23, and can resume pupil detection based on the difference image between the bright pupil image and the dark pupil image. Conventional pupil detection devices tend to be unable to detect the pupil for several frames after the subject A's eyes are opened after being closed. The above-described function of the gaze detection unit 23 allows stable pupil detection to be resumed without delay from the frame immediately after subject A opens his / her eyes again after closing them.

[0076] Furthermore, the pupil position prediction unit 25 of the image processing device 20 according to this embodiment predicts the pupil position using two networks, machine learning model 1 and machine learning model 2. On the other hand, as a modified example, the image processing device 20 may predict the pupil position using one network that integrates the prediction functions of the two networks, or may predict the pupil position using three or more networks. For example, the three networks may be "machine learning model 1" that predicts an eye-closed image using multiple partial images cut out from within a window of a face image, "machine learning model 2" that predicts the amount of deviation of the partial images from the eye-closed image, and "machine learning model 3" that predicts an eye-closed image using multiple partial images cut out from the entire face image. [Explanation of symbols]

[0077] 1...Gaze detection device (pupil detection device), 10...Camera, 13...Light source, 20...Image processing device (arithmetic device), 23...Gaze detection unit (pupil position detection unit, pupil distance detection unit), 24...Partial image creation unit, 25...Pupil position prediction unit, 27...Model learning unit, A...Subject, G F1 ,G F2 ...Facial image, GP F1 ...Partial image, GP L1 ,GP R1 …Closed eyes image, P L1 ,P R1 ...pupil position.

Claims

1. a camera that captures facial images of a subject at successive timings by capturing the subject's face; a light source that projects light toward the subject's face; a computing device that processes a face image acquired by the camera at the timing of irradiating the light, The computing device a pupil position detection unit that detects the position of the pupil of the subject on the face image; a pupil position prediction unit that predicts the position of the pupil of the subject when the eyes are closed by searching for the position of the facial feature part when the eyes are closed on the face image, the pupil position detection unit detects the position of the pupil by tracking the pupil using the position of the pupil predicted by the pupil position prediction unit; the pupil position prediction unit predicts the position of the pupil by a machine learning model using a neural network, inputs a partial image cut out from the face image as input data to the machine learning model, and predicts the position of the pupil by searching for a position of the partial image including a characteristic part of the face using the machine learning model; The computing device further includes a model learning unit that cuts out the face image acquired when the subject has their eyes closed, based on the position of the pupil detected by the pupil position detection unit immediately before the subject closes their eyes, and uses the cut-out face image as training data to train the machine learning model. Pupil detection device.

2. the facial feature is an eye feature of the subject; 2. The pupil detection device according to claim 1.

3. the pupil position prediction unit cuts out a partial image with a size that includes the entire facial feature.

3. The pupil detection device according to claim 1 or 2.

4. the computing device further includes a pupil distance detection unit that detects a distance from the camera to the pupil; the pupil position prediction unit variably sets a cutout size of the partial image as the input data in accordance with the distance to the pupil. The pupil detection device according to any one of claims 1 to 3.

5. the computing device further includes a pupil distance detection unit that detects a distance from the camera to the pupil; the model learning unit variably sets a cutout size of the face image as training data in accordance with the pupil distance; The pupil detection device according to any one of claims 1 to 3.

6. the model learning unit identifies an area in which the facial features exist from the cut-out face image, and sets a cut-out size of the face image as training data based on the identified area; The pupil detection device according to any one of claims 1 to 3.

7. the model learning unit identifies an area in the cut-out face image where the facial features exist, and sets a cut-out position of the face image as training data so that the identified area is located at the center of the image. The pupil detection device according to claim 1 or 6.

8. the machine learning model is a model that predicts a positional deviation amount of a partial image including a characteristic part of the face with respect to the face image cut out based on the position of the pupil detected by the pupil position detection unit immediately before the subject closes his / her eyes, the model learning unit generates a shifted image while shifting the extracted face image, and uses the shifted image and the shift amount of the shifted image as training data to train the machine learning model; The pupil detection device according to claim 1 or 5.

9. A camera that captures facial images of a subject at successive timings by capturing the face of the subject; a light source that projects light toward the subject's face; a computing device that processes a face image acquired by the camera at the timing of irradiating the light, The computing device a pupil position detection unit that detects the position of the pupil of the subject on the face image; a pupil position prediction unit that predicts the position of the pupil of the subject when the eyes are closed by searching for the position of the facial feature part when the eyes are closed on the face image, the pupil position detection unit detects the position of the pupil by tracking the pupil using the position of the pupil predicted by the pupil position prediction unit; a pupil position is detected by tracking the pupil position using the pupil position detected on the face image of successive frames, and if detection of the pupil position on the face image of the immediately preceding frame fails, the pupil position is tracked using the pupil position predicted by the pupil position prediction unit; when the subject closes his / her eyes and then opens them again, the position of the pupil is detected by setting a window on the face image based on the position of the pupil predicted by the pupil position prediction unit at the timing when the subject closed his / her eyes. Pupil detection device.

10. A pupil detection method using a camera that captures facial images of a subject at successive timings by capturing an image of the subject's face, a light source that irradiates light onto the subject's face, and a computing device that processes the facial images acquired by the camera at the timings of the light irradiation, a pupil position detection step in which the computing device detects the position of the pupil of the subject on the face image; a pupil position prediction step in which the arithmetic device predicts the position of the pupil of the subject when the eyes are closed by searching for the positions of the facial feature parts in the closed eye state on the face image, the pupil position detecting step detects the position of the pupil by tracking the pupil using the position of the pupil predicted in the pupil position predicting step; the pupil position prediction step predicts the position of the pupil by a machine learning model using a neural network, inputs a partial image cut out from the face image as input data to the machine learning model, and predicts the position of the pupil by searching for a position of the partial image including a characteristic part of the face using the machine learning model; The computing device further includes a model learning step of cutting out the face image acquired when the subject has their eyes closed, based on the position of the pupil detected by the pupil position detection step immediately before the subject closes their eyes, and using the cut-out face image as training data to train the machine learning model. Pupil detection method.

11. A pupil detection method using a camera that captures facial images of a subject at successive timings by capturing an image of the subject's face, a light source that irradiates light toward the subject's face, and a computing device that processes the facial images acquired by the camera at the timings when the light is irradiated, a pupil position detection step in which the computing device detects the position of the pupil of the subject on the face image; a pupil position prediction step in which the arithmetic device predicts the position of the pupil of the subject when the eyes are closed by searching for the positions of the facial feature parts in the closed eye state on the face image, the pupil position detecting step detects the position of the pupil by tracking the pupil using the position of the pupil predicted in the pupil position predicting step; detecting a position of the pupil by tracking the position of the pupil using the position of the pupil detected on the face image of successive frames, and if detection of the position of the pupil on the face image of the immediately preceding frame fails, tracking the position of the pupil using the position of the pupil predicted by the pupil position prediction step; When the subject closes his / her eyes and then opens them again, the position of the pupil is detected by setting a window on the face image based on the position of the pupil predicted by the pupil position prediction step at the timing when the subject closed his / her eyes. Pupil detection method.

Images