Gaze Estimation System

Abstract

To provide a gaze estimation system capable of achieving both of reduction in processing load and accurate gaze estimation.SOLUTION: A gaze estimation system 1 determines whether a gaze 24 of a driver 20 obtained from a captured image is oriented to a first object range 41 requiring high accuracy or a second object range 42 which is outside the first object range 41 and does not require higher accuracy than the first object range 41. As a result, when the gaze 24 is determined to be oriented to the first object range 41, first gaze estimation processing is performed that performs gaze estimation processing using an appearance based method to generate first estimated gaze information. Meanwhile, when the gaze 24 is determined to be oriented to the second object range 42, second gaze estimation processing is performed that performs gaze estimation processing using a model based method to generate second estimated gaze information.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a gaze estimation system. [Background technology]

[0002] There are two types of gaze estimation technology using image processing: model-based methods and appearance-based methods. Gaze estimation using the model-based method involves, for example, creating a three-dimensional model including an eyeball model, and estimating the gaze by comparing the model with an image including the subject's eyes (see, for example, Patent Document 1). Gaze estimation using the appearance-based method involves estimating the gaze by comparing an image including the subject's eyes with multiple learning images that have been machine-learned by a learning device (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-88236 A [Patent Document 2] JP 2019-28843 A Summary of the Invention [Problem to be solved by the invention]

[0004] By the way, the appearance-based method can estimate the gaze of the subject with high accuracy by using deep learning, but the processing load of the computing device increases and a large amount of learning is required for machine learning. On the other hand, the model-based method reduces the processing load of the device compared to the appearance-based method, but it is not easy to estimate the gaze of the subject with high accuracy. Therefore, when estimating the gaze of the subject, it is desirable to achieve both a reduction in the processing load and high-accuracy gaze estimation.

[0005] The present invention has been made in consideration of the above problems, and has an object to provide a gaze estimation system that can achieve both a reduction in processing load and highly accurate gaze estimation. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the gaze estimation system of the present invention comprises an image acquisition unit that acquires photographed images including the face of a person to be measured in succession in time series, and a processing unit that executes an estimation process of the gaze of the person to be measured based on each of the acquired photographed images, wherein the processing unit executes a first gaze estimation process that performs a gaze estimation process using an appearance-based method based on a learning image including the face of the person to be measured to generate first estimated gaze information, and a second gaze estimation process that performs a gaze estimation process using a model-based method based on the learning image to generate second estimated gaze information, and further comprises a first determination unit that determines whether the gaze of the person to be measured is directed toward a first target range requiring relatively high accuracy, or toward a second target range that is outside the first target range and does not require higher accuracy than the first target range, depending on a movement accompanied by a change in the gaze of the person to be measured obtained from each of the photographed images, and executes the first gaze estimation process when the first determination unit determines that the gaze is directed toward the first target range, and executes the second gaze estimation process when the processing unit determines that the gaze is directed toward the second target range. Effect of the Invention

[0007] The gaze estimation system according to the present invention has the advantage of being able to achieve both a reduction in the processing load in estimating the gaze of a subject based on each acquired captured image and high-precision gaze estimation. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing an application example of a gaze estimation system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a schematic configuration of the gaze estimation system of FIG. [Diagram 3] FIG. 3(A) is a state transition diagram showing an overview of the operation of the gaze estimation system of FIG. 1, and FIG. 3(B) is a schematic diagram showing an example of a target range to which the measurement subject directs his / her gaze. [Figure 4] FIG. 4 is a schematic diagram showing an overview of the first gaze estimation process executed in the gaze estimation system of FIG. [Diagram 5] FIG. 5 is a flowchart showing an outline of the second gaze estimation process executed in the gaze estimation system of FIG. [Figure 6] FIG. 6(A) is a schematic diagram showing the relationship between a three-dimensional (3D) model of the eyeball and a two-dimensional (2D) image in step S16 of FIG. 5, and FIG. 6(B) is a schematic diagram showing an overview of the iris search performed in steps S17 and S18 of FIG. 5. [Figure 7] FIG. 7 is a flowchart showing an example of an algorithm of the gaze estimation process executed by the gaze estimation system of FIG. [Figure 8] FIG. 8 is a schematic diagram showing an example of a target range to which a person to be measured directs his / her gaze in a gaze estimation system according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Below, a gaze estimation system according to an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments shown below. The components in the following embodiments include those that a person skilled in the art can easily imagine, or those that are substantially the same. Also, the components in the following embodiments can be omitted, replaced, or modified in various ways without departing from the gist of the invention.

[0010] [Embodiment] 1 and 2, the gaze estimation system 1 of this embodiment is a system that captures an image of a subject and estimates the gaze of the subject based on a captured image including the face of the subject. In this embodiment, the gaze estimation system 1 is applied to a vehicle. The gaze estimation system 1 includes a photographing unit 12 and a processing unit 13.

[0011] The photographing unit 12 is an example of an image acquisition section (or image acquisition device) that captures an image of the face 21 of the driver 20 of the vehicle 2 and acquires images including the face 21 of the driver 20 continuously in time series. In this embodiment, the driver 20 is the person to be measured. The photographing unit 12 includes a light source and a camera. The light source and the camera are arranged adjacent to each other on a single substrate, for example.

[0012] The light source is, for example, an LED (Light Emitting Diode) that emits near-infrared light toward the outside of the photographing unit 12. Near-infrared light is an electromagnetic wave with a wavelength of approximately 0.7 to 2.5 μm, and has a wavelength close to red visible light, and is used, for example, as "invisible light" in infrared cameras, infrared communications, etc. The light source is turned on (emits infrared light) in response to a turn-on signal input from the processing unit 13, and turned off in response to a turn-off signal. The light source is placed on or near the object to be viewed.

[0013] The camera is disposed near the light source, and captures an image of a capture range (e.g., an angle of view) set outside the capture unit 12. When the driver 20 is present in the capture range set outside the capture unit 12, the camera captures an image of the driver 20 illuminated by infrared light emitted from the light source. The capture unit 12 is installed, for example, in a meter unit in front of the driver's seat 101 or on an upper part of a column cover in order to capture an image of the face 21 of the driver 20 seated in the driver's seat 101 in the passenger compartment 10 of the vehicle 2. The capture unit 12 is connected to a processing unit 13, and outputs the captured image to the processing unit 13.

[0014] The camera is placed at a position where it receives reflected light reflected in the opposite direction to the direction of emission of infrared light emitted from the light source. When the camera is placed at a position where it receives reflected light reflected in the opposite direction to the direction of emission of infrared light, the optical axis of the camera and the optical axis of the light source overlap. The optical axis of the camera and the optical axis of the light source overlap means that both optical axes are coaxial, but it is sufficient that both optical axes are parallel and the reflected light can be received at a position. It is preferable that the camera can capture at least the entire face 21 with sufficient resolution. The capture range can be determined based on the range to be detected that is required by the application. Examples of applications include measuring the interest of a vending machine, automatically counting the number of viewers of advertisements such as digital signage, and turning on (or increasing the brightness of) a display system only when the display system is viewed.

[0015] The processing unit 13 is an example of a processing unit (or processing device) and executes an estimation process of the line of sight of the driver 20 based on the captured image acquired by the photographing unit 12. The processing unit 13 has, for example, a processing circuit (not shown) that realizes various processing functions in the line of sight estimation system 1. The processing circuit is realized, for example, by a processor. The processor means, for example, a circuit such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). The processing unit 13 realizes each processing function, for example, by executing a program read from a storage circuit (storage unit) not shown.

[0016] As shown in FIG. 2, the processing unit 13 includes a first gaze estimation processing unit 14A, a second gaze estimation processing unit 14B, a first determination unit 15A, a second determination unit 15B, and a third determination unit 15C.

[0017] Among the processing unit 13, the first gaze estimation processing unit 14A executes a first gaze estimation process that performs gaze estimation processing by an appearance-based method based on a learning image including the face 21 of the driver 20 to generate first estimated gaze information. The first estimated gaze information is, for example, the gaze angle of the gaze 24 of the driver 20. The first gaze estimation process is executed by the first gaze estimation processing unit 14A. The gaze estimation technology by the appearance-based method is disclosed in, for example, the following document. "High-precision gaze estimation using appearance-based methods for specific environment applications, Qu Manting (Keio University), Takahashi Yuki (Yazaki Corporation), et al., 26th Symposium on Image Sensing (SSII2020), IS2-18, June 2020."

[0018] Gaze estimation using appearance-based methods uses the subject's eye image itself as input information, learns the combination of gaze and eye image using machine learning, and estimates the gaze position for a new eye image. Machine learning methods include, for example, Convolutional Neural Networks (CNN), a deep learning method. CNN is a type of DNN (Deep Neural Network), which is a multi-layered pattern recognition method called a neural network, that is compatible with 2D data and has been reported to have high pattern recognition capabilities for images. To perform learning with CNN, a correct label corresponding to the input image is required. In this case, a face image is given as input information and the gaze angle is given as the correct label.

[0019] In estimating gaze using an appearance-based method, it is necessary to create a learning dataset and generate a model by machine learning. At this time, the accuracy of the learning dataset affects the accuracy of the learning. Therefore, when creating a learning dataset, it is necessary to ensure that the subject (or test subject) is looking accurately at the visual target. For example, as shown in FIG. 4, the subject wears a wearable gaze measurement device and gaze measurement results are collected. Image 31 was obtained by photographing the subject while wearing the gaze measurement device. On the other hand, when actually estimating the gaze of the subject, a face image without the gaze measurement device is input, so in order to make it closer to the image during actual use, as shown in Image 32, a process is performed to paint the arm part of the gaze measurement device that is over the face image with the same color as the face.

[0020] The second gaze estimation processing unit 14B of the processing unit 13 executes a second gaze estimation process that performs a gaze estimation process by a model-based method based on a learning image including the face 21 of the driver 20 to generate second estimated gaze information. The second estimated gaze information is, for example, the gaze angle of the gaze 24 of the driver 20. The second gaze estimation process is executed by the second gaze estimation processing unit 14B. A gaze estimation technique by a model-based method is disclosed in, for example, JP 2018-88236 A.

[0021] The process shown in FIG. 5 is an example of the second gaze estimation process, and each step is performed sequentially by the processing unit 13 executing a program read from the storage unit.

[0022] In step S12, the photographing unit 12 photographs an image including the face of the driver 20 and outputs a signal of the image. The processing unit 13 takes in one frame of the image signal from the photographing unit 12.

[0023] In step S13, the processing unit 13 converts the data format of the image (pre-processing), including grayscaling. For example, it generates image data of a two-dimensional (2D) array in which 8-bit data expressing luminance in a gradation range of "0 to 255" is arranged vertically and horizontally according to the scanning direction in the frame at the time of shooting, for each pixel position in one frame.

[0024] In step S14, the processing unit 13 performs face detection using, for example, the "Viola-Jones method" based on the image converted in step S13, and extracts a face image including a face from one frame of two-dimensional image data. That is, the face image is extracted using a detector that is characterized by the shading difference of the face and that is created by learning using "Boosting". The technology of the "Viola-Jones method" is described in, for example, the following document. “Viola, Paul and Michael J. Jones, “Rapid Object Detection using aBoosted Cascade of Simple Features”, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. Volume: 1, pp. 511-518.”

[0025] In step S15, the processing unit 13 detects eye areas from the face image extracted in step S14, for example, by using the above-mentioned Viola-Jones method.

[0026] In step S16, when the gaze is detected by a model-based method using a 3D eyeball model described later, the processing unit 13 estimates the eyeball center. Here, the central coordinates of the rectangular area of ​​the eye detected in step S15 are assumed to be the eyeball center. For example, it is also possible to use a method of determining the eyeball center based on the positions of the outer corner and inner corner of the eye, or a method of estimating the skeleton from facial feature points and simultaneously calculating the eyeball center position.

[0027] In step S17, the processing unit 13 performs a rough search for the iris (pupil or iris) by applying a template matching technique to the data of the rectangular region of the eye detected in step S15. Specifically, a black circle image is matched as a template against a binarized image of the eye image obtained by cutting out the periphery of the eye, and the coordinates of the image center (center of the black circle) of the black circle image with the highest likelihood are set as the center position of the iris in the eye image, and the radius of the black circle image with the highest likelihood is set as the radius of the iris in the eye image. The processing in step S17 is performed to roughly estimate the center position and radius of the iris in the eye image.

[0028] In step S18, the processing unit 13 uses the center position and radius of the iris found in step S17 and applies a particle filter technique to detect the center position and radius of the iris with higher accuracy.

[0029] In step S19, the processing unit 13 obtains numerical data of the eyeball center coordinates and the coordinates of the center position of the iris for one frame of image by the processes of steps S13 to S18 described above, and outputs the numerical data. The eyeball center coordinates and the coordinates of the center position of the iris allow the direction of gaze to be specified. Furthermore, by repeating the loop processes of steps S11 to S20 until the video output by the photographing unit 12 is interrupted, gaze detection is realized in real time. When calculating the eyeball rotation angle from the coordinates on the image plane using these data, a conversion is performed using a 3D eyeball model as shown in FIG. 6(A).

[0030] In the 3D eyeball model shown in FIG. 6A, the eyeball E is a sphere represented by an eyeball center O and an eyeball radius R. In addition, on the surface of the eyeball E, there is a circular iris F that constitutes the black eye, and in the center of the iris F, there is a circular pupil G. The direction of the line of sight can be specified as the direction from the eyeball center O toward the center of the iris F or the pupil G, and can be represented by a rotation angle yaw with respect to a reference direction in the horizontal plane, and a rotation angle pitch with respect to a reference direction in the up-down direction. In addition, when the eyeball center O is used as a reference, the central coordinates of the iris F or the pupil G can be represented by the eyeball radius R, yaw, and pitch. On the other hand, since the image captured by the photographing unit 12 represents a two-dimensional plane, when a two-dimensional image captured by the photographing unit 12 is applied to the 3D eyeball model, it is necessary to perform two-dimensional / three-dimensional mutual conversion. For example, the conversion is performed using the following equation.

[0031] X=-R×cos(pitch)×sin(yaw)···(1) Y = R × sin(pitch) (2) X: Distance in the x direction from the eyeball center O on the two-dimensional image plane Y: Distance in the y direction from the eyeball center O on the two-dimensional image plane

[0032] Next, the process of searching for the iris from a face image will be described with reference to Fig. 6(B). In step S41, the processing unit 13 uses the result of step S15 in Fig. 5 to cut out the eyes and their surrounding rectangular areas from the two-dimensional image data of one entire frame, and obtains data D41.

[0033] In step S42, the processing unit 13 generates data D42 by binarizing the data D41 acquired in step S41 so that the gradation for each pixel is only two values, black / white, and then performs template matching on this data D42 in step S17 in Fig. 5. That is, an image of a black circle shape similar to the shape of the iris is used as a template, and while scanning the image of data D42 with this template, the approximate position of the iris with the most similar features is searched for, and the position and the radius or diameter of the iris are identified.

[0034] In step S43, the processing unit 13 applies Sobel filter processing to the eye data D41 acquired in step S41. Specifically, the eye data D41 is scanned horizontally from left to right, and edges are detected by outputting black (tone value: 0) in areas with no luminance change and approaching white (tone value: 255) as the gradient of the luminance change increases. This results in eye image data D43 from which edges have been extracted.

[0035] In step S44, the processing unit 13 executes particle filter processing on the eye image data D43 obtained in step S43, starting from the approximate position of the iris obtained in step S42. Iris detection using the particle filter is performed in the following order [1] to [4].

[0036] [1] The processing unit 13 searches for the rough position of the iris by template matching. [2] The processing unit 13 calculates the eyeball rotation angle from the iris position in [1] above based on the 3D eyeball model. [3] The processing unit 13 scatters particles for sampling iris ellipse candidates in the vicinity of the rough eyeball rotation angle obtained in [2] above. [4] The processing unit 13 calculates the likelihood at the iris ellipse candidate positions scattered in [3]. Here, the likelihood is calculated by summing the scores of all points on the circumference of the iris ellipse candidate 44a, calculating +1 if the point is extracted as an edge by referring to the edge image D44 of the Sobel filter, and +0 if it is not an edge. [5] The likelihood is calculated for each scattered particle, and the trend of change in the likelihood with respect to the scattering position is captured and the detection result is output. In other words, from the scattering results of multiple particles, the point at which the likelihood is maximized with respect to the scattering position change is output as the detection result.

[0037] For the eyeball rotation angle vicinity in [3] above, multiple patterns are created by adding blur caused by random numbers to the yaw and pitch values ​​of the 3D eyeball model shown in Figure 6 (A). At this time, the amount of change in blur detected in the next frame is generated from the detection result of the previous frame. In other words, the greater the difference in the detection result from the median of the random numbers, the greater the amount of change in blur caused by the random numbers.

[0038] As shown in FIG. 3A, the processing unit 13 executes model-based gaze estimation (second gaze estimation process), and executes appearance-based gaze estimation (first gaze estimation process) if the result of the gaze estimation process satisfies condition A. On the other hand, when the processing unit 13 executes the first gaze estimation process, if the result of the face direction calculation process executed before executing the first gaze estimation process satisfies condition B, the processing unit 13 executes the second gaze estimation process. Condition A is, for example, a gaze angle estimated in the second gaze estimation process within ±θ°. θ°(yaw, pitch) is a threshold value for transitioning to appearance-based gaze estimation based on the gaze angle obtained in the model-based gaze estimation, and indicates a corresponding range in the appearance-based gaze estimation. Condition B is a face direction angle calculated in the face direction calculation process equal to or greater than ±φ°. φ°(yaw, pitch) is a threshold value for the face direction angle for transitioning from appearance-based gaze estimation to model-based gaze estimation. φ° can be set based on the corresponding range θ° in the appearance-based gaze estimation. For example, it can be set based on a sharing ratio that is considered regarding the relationship between the face direction and the gaze angle when viewing an object. Details of the sharing ratio are shown in, for example, the following document. The processing unit 13 performs initial setting immediately after the system is started, and executes the second gaze estimation process after the initial setting. "Analysis of the Coordination of Head and Eye Movements When Gazing at a Two-Dimensional Indicator, Mitsuho Yamada, IEICE Transactions on Electronics, Information and Communication Engineers, Vol. J75-D2 No. 5 pp971-981, May 1992"

[0039] The first determination unit 15A determines whether the line of sight 24 of the driver 20 is directed toward the first target range 41 or the second target range 42, depending on the action accompanied by a change in the line of sight 24 of the driver 20 obtained from each captured image. The action accompanied by a change in the line of sight 24 of the driver 20 obtained from each captured image is, for example, an action of turning the face or an action of turning the body. In this embodiment, the action is an action of turning the face. The first determination unit 15A calculates the face direction angle of the driver 20 from each captured image, and determines whether the line of sight 24 is directed toward the first target range 41 or the second target range 42, based on the face direction angle.

[0040] The first target range 41 is set to a range within the visible range of the driver 20 that requires relatively high-precision estimation of the line of sight. The first target range 41 is, for example, a driving visual field range, as shown in FIG. 3B. In the driving visual field range, it is required to estimate the direction of the line of sight, such as whether the driver 20 saw the brake lights of a vehicle traveling ahead, whether the driver saw a red light, or whether the driver saw a pedestrian at a crosswalk in the distance. The second target range 42 is a range within the visible range of the driver 20 that is outside the first target range 41 and does not require a line of sight estimation with higher precision than the first target range 41. The second target range 42 is, for example, a range into which the driver 20 looks when checking for safety or looking aside.

[0041] When second estimated gaze information is obtained, second determination section 15B determines whether gaze 24 is directed toward first target range 41 or second target range 42 based on the second estimated gaze information.

[0042] The third determination unit 15C determines whether the gaze estimation process executed immediately before is the first gaze estimation process or the second gaze estimation process.

[0043] Next, an algorithm of the gaze estimation process executed by the gaze estimation system 1 will be described with reference to Fig. 3 and Fig. 7. In the process shown in Fig. 7, each step is sequentially performed by the processing unit 13 executing a program read from the storage unit in response to power-on (e.g., turning on the IGN of the vehicle 2).

[0044] In step S31, the photographing unit 12 photographs an image including the face of the driver 20 and outputs a signal of the image. The processing unit 13 captures one frame of the image signal from the photographing unit 12. At this time, the processing unit 13 converts the data format of the image, including grayscaling, and changes the size, as necessary.

[0045] In step S32, the processing unit 13 performs face detection using, for example, the above-mentioned "Viola-Jones method" based on the captured image captured in step S31, and extracts a facial image including a face from one frame of two-dimensional image data.

[0046] In step S33, the processing unit 13 detects the eyes in the face image extracted in step S32. The processing unit 13 can detect the positions of the eyes, for example, by the above-mentioned "Viola-Jones method." However, since the face direction is calculated in a later step, a method of detecting facial feature points and identifying a rectangular portion that includes the eye feature points is preferable. For example, a general facial feature point detection algorithm shown on the following site may be used. Facial point annotations: https: / / ibug.doc.ic.ac.uk / resources / facial-point-annotations /

[0047] In step S34, the processing unit 13 calculates the face direction in the face image extracted in step S32. The algorithm executed in step S33 includes an algorithm for calculating the face direction. This is a method of estimating the rotation angle (yaw, pitch, roll) of the face direction by creating a standard face model in advance and comparing the position of each point corresponding to the detected two-dimensional feature point arrangement. Note that the face direction angle is used in a later step, and is temporarily stored in, for example, a storage unit in the processing unit 13. Note that the algorithm executed in steps S32 to S34 may be realized using software other than the above-mentioned method.

[0048] In step S35, the processing unit 13 determines whether or not there is a gaze estimation process executed immediately before. The processing unit 13 determines, for example, whether or not a software use log is recorded in the storage unit. Here, the software use log includes a log corresponding to the first gaze estimation process and a log corresponding to the second gaze estimation process. When the software use log is not recorded in the storage unit, the processing unit 13 determines that there is no gaze estimation process executed immediately before, and proceeds to step S41. When the software use log is not recorded in the storage unit, for example, a case where the gaze estimation process is executed for an initial frame or the system is initialized is included. Or, a case where the face 21 or the eyes 22 of the driver 20 cannot be detected in the face detection or eye detection in the previous stage, that is, the face 21 of the driver 20 is facing to the side or backward, and the face 21 or the eyes 22 does not exist within the image range is included. When the software use log is recorded in the storage unit as a result of the determination in step S35, the processing proceeds to step S36.

[0049] In step S36, processing unit 13 determines whether the gaze estimation process executed immediately before was the first gaze estimation process or the second gaze estimation process. Step S36 is executed by third determination unit 15C. Processing unit 13 refers to the software usage log recorded in the storage unit and determines whether it corresponds to the first gaze estimation process or the second gaze estimation process. If the software usage log corresponds to the first gaze estimation process, processing unit 13 proceeds to step S37. On the other hand, if the software usage log corresponds to the second gaze estimation process, processing unit 13 proceeds to step S41.

[0050] In step S37, processing unit 13 determines whether the face direction angle is ±φ° or more. Step S37 is executed by first determination unit 15A. The face direction angle is the one calculated in step S34. When processing unit 13 determines that the face direction angle is ±φ° or more, it determines that line of sight 24 of driver 20 is directed toward second target range 42, and proceeds to step S41. On the other hand, when processing unit 13 determines that the face direction angle is not ±φ° or more, it determines that line of sight 24 of driver 20 is directed toward first target range 41, and proceeds to step S38.

[0051] In step S38, the processing unit 13 executes the first gaze estimation process, and then the process proceeds to step S39.

[0052] In step S39, the processing unit 13 outputs a result of the executed gaze estimation process, and the process proceeds to step S40.

[0053] In step S40, the processing unit 13 records a software usage log indicating the executed gaze estimation process in the storage unit, and then returns to step S31.

[0054] In step S41, the processing unit 13 executes the second gaze estimation process, and the process proceeds to step S42. The processing unit 13 temporarily records the result of the executed gaze estimation process in the storage unit.

[0055] In step S42, processing unit 13 determines whether the gaze angle is within ±θ°. Step S42 is executed by second determination unit 15B. The gaze angle is the one estimated in step S41. When processing unit 13 determines that the gaze angle is within ±θ°, it determines that the gaze 24 of driver 20 is directed toward first target range 41, erases the result of the gaze estimation process temporarily recorded in step S41, and proceeds to step S38. On the other hand, when processing unit 13 determines that the gaze angle is not within ±θ°, it determines that the gaze 24 of driver 20 is directed toward second target range 42, and proceeds to step S39.

[0056] The gaze estimation system 1 described above determines whether the gaze 24 of the driver 20 obtained from a captured image is directed toward a first target range 41 that requires a relatively high level of accuracy, or toward a second target range 42 that is outside the first target range 41 and does not require a higher level of accuracy than the first target range 41. As a result, if it is determined that the gaze 24 is directed toward the first target range 41, a first gaze estimation process is performed to generate first estimated gaze information by performing a gaze estimation process using an appearance-based method. On the other hand, if it is determined that the gaze 24 is directed toward the second target range 42, a second gaze estimation process is performed to generate second estimated gaze information by performing a gaze estimation process using a model-based method.

[0057] With the above configuration, when the driver 20 is directing his / her gaze to the first target range 41, which requires high accuracy, the gaze can be estimated with high accuracy by the appearance-based method, and when the driver is directing his / her gaze to the second target range 42, which does not require high accuracy, the gaze can be estimated by the model-based method, which has a lower processing load than the appearance-based method. As a result, the processing load can be reduced compared to when estimating the gaze using only the appearance-based method, and the gaze can be estimated with high accuracy compared to when estimating the gaze using only the model-based method, so that both the reduction in processing load and the high-accuracy gaze estimation can be achieved. In addition, when the gaze estimation system 1 is applied to a vehicle, the gaze in the driving field of view in front of the vehicle and the gaze outside the driving field of view, such as the driver 20 checking for safety or looking aside, can be estimated.

[0058] Furthermore, when the action is a facial direction action of the driver 20, the gaze estimation system 1 calculates the facial direction angle of the driver 20 from each captured image, and determines whether the gaze 24 is directed toward the first target range 41 or the second target range 42 based on the facial direction angle. As a result, the processing load can be reduced by determining which target range the gaze 24 is directed toward based on the facial direction angle of the driver 20, compared to, for example, a case in which the second gaze estimation process is executed for each image and whether the gaze 24 of the driver 20 is directed toward the first target range 41 or the second target range 42.

[0059] Furthermore, the gaze estimation system 1 determines whether the gaze 24 is directed toward the first target range 41 or the second target range 42 based on the second gaze estimation gaze information, and executes the first gaze estimation process when it is determined that the gaze 24 is directed toward the first target range 41. As a result, even if it erroneously determines that the gaze 24 of the driver 20 is directed toward the second target range 42 instead of the first target range 41 and performs gaze estimation in the second gaze estimation process, it is possible to execute the first gaze estimation process and estimate the gaze directed toward the first target range 41 with high accuracy.

[0060] Furthermore, the gaze estimation system 1 determines whether the gaze estimation process executed immediately before is the first gaze estimation process or the second gaze estimation process, and if the gaze estimation process executed immediately before is the second gaze estimation process, executes the second gaze estimation process, and if it is the first gaze estimation process, performs a determination by the first determination unit 15A. As a result, for example, if the gaze estimation process executed immediately before is the second gaze estimation process, the second gaze estimation process is performed without a determination by the first determination unit 15A, and therefore it is possible to reduce the amount of processing for determination by the first determination unit 15A.

[0061] [Variations] In the above embodiment, the gaze estimation system 1 is applied to a vehicle 2 such as an automobile, but is not limited thereto, and may be applied to, for example, a ship or an aircraft other than the vehicle 2. In addition, the gaze estimation system 1 is separated into the photographing unit 12 and the processing unit 13, but is not limited thereto, and may be configured as an integrated unit.

[0062] The gaze estimation system 1 may also be applied to digital signage. For example, by installing the photographing unit 12 on the digital signage, it is possible to analyze where passersby looked on the digital signage. In the example shown in FIG. 8, the photographing unit 12 is installed on the top of the central digital signage 51. The photographing unit 12 may be installed at the bottom, but it is necessary to create a gaze database in which gaze angle information and face images are associated to learn a trained model used for gaze estimation by the appearance-based method according to the installation position and the range in which the measurement subject appears. In such a gaze estimation system, within the digital signage 51 in which the photographing unit 12 is installed, gaze estimation by the appearance-based method can be used to accurately estimate which part of the digital signage 51 is focused on. Furthermore, within the digital signage outside 52, gaze estimation by the model-based method can be used to estimate gaze information such as which of the digital signage outside 52 adjacent to the left and right of the digital signage 51 is being looked at. In addition, by recording the gaze position and the duration of gaze, it is possible to measure the degree of attention to the displayed content.

[0063] In the above embodiment, the processing circuit has been described as having each processing function realized by a single processor, but this is not limited to the above. The processing circuit may be configured to realize each processing function by combining multiple independent processors and having each processor execute a program. The processing functions of the processing circuit may be realized by being appropriately distributed or integrated into a single or multiple processing circuits. The processing functions of the processing circuit may be realized in whole or in part by a program, or may be realized as hardware using wired logic or the like.

[0064] The program executed by the processor described above is provided by being preinstalled in a storage circuit or the like. The program may be provided by being recorded on a computer-readable storage medium in a format that can be installed on these devices or in a format that can be executed. The program may also be provided or distributed by being stored on a computer connected to a network such as the Internet and downloaded via the network. [Explanation of symbols]

[0065] 1. Gaze Estimation System 2 Vehicles 12 Photography Unit 13 Processing section 14A First line of sight estimation processing unit 14B Second gaze estimation processing unit 15A 1st judgment section 15B 2nd judgment section 15C 3rd Judgment Section 20 Driver 21 Face 22 eyes

Claims

1. an image acquisition unit that acquires photographed images including a face of a subject in a time series; a processing unit that executes an estimation process of the line of sight of the measurement subject based on each of the acquired photographic images, The processing unit includes: a first gaze estimation process for performing a gaze estimation process by an appearance-based method based on a learning image including a face of the measurement subject to generate first estimated gaze information, and a second gaze estimation process for performing a gaze estimation process by a model-based method based on the learning image to generate second estimated gaze information, a first determination unit that determines, according to a movement accompanied by a change in the line of sight of the measurement subject obtained from each of the captured images, whether the line of sight is directed toward a first target range requiring a relatively high degree of accuracy or toward a second target range outside the first target range and not requiring a higher degree of accuracy than the first target range; When the first determination unit determines that the line of sight is directed toward the first target range, the first line of sight estimation process is executed; When it is determined that the line of sight is directed toward the second target range, the second line of sight estimation process is executed. The gaze estimation system is characterized by:

2. The movement is a facial movement of the measurement subject, The first determination unit is calculating a face direction angle of the measurement subject from each of the captured images, and determining whether the line of sight is directed toward the first target range or the second target range based on the face direction angle; The gaze estimation system according to claim 1 .

3. The processing unit includes: a second determination unit that, when the second estimated gaze information is obtained, determines whether the gaze is directed toward the first target range or the second target range based on the second estimated gaze information; When the second determination unit determines that the line of sight is directed toward the first target range, the first line of sight estimation process is executed. The gaze estimation system according to claim 1 .

4. The processing unit includes: a third determination unit that determines whether a previously executed gaze estimation process is the first gaze estimation process or the second gaze estimation process, When the third determination unit determines that the gaze estimation process executed immediately before is the second gaze estimation process, the second gaze estimation process is executed, and when the third determination unit determines that the gaze estimation process executed immediately before is the first gaze estimation process, a determination is made by the first determination unit. The gaze estimation system according to claim 3 .

Images