Line-of-sight estimation device, line-of-sight estimation system and line-of-sight estimation method

Abstract

To provide a line-of-sight estimation device, a line-of-sight system, and a line-of-sight estimation method capable of accurately estimating a person's line of sight over a wide distance range.SOLUTION: A line-of-sight estimation device 10 comprises: an image acquisition part 11a that acquires a picked-up image including at least a part of a person from a camera 20; a distance acquisition part 11b that acquires a distance to the person; an extraction part 11c that extracts multiple line-of-sight evaluation elements of different types from the picked-up image; and a line-of-sight estimation part 11d that changes estimation processing using the multiple line-of-sight evaluation elements based on the distance acquired by the distance acquisition part, thereby estimating the line-of-sight of the person.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a gaze estimation device, a gaze estimation system, and a gaze estimation method for estimating a person's gaze.

Background Art

[0002] In recent years, there has been an increasing demand for confirming the commercial effects of advertisements such as digital signage. In this case, it is determined whether a person near the advertisement has seen the advertisement. Therefore, it is necessary to estimate the gaze of a person near the advertisement.

[0003] Patent Document 1 below describes a gaze detection device that detects a person's gaze by detecting the pupil center of gravity from a captured image of a camera. In this gaze detection device, light from a light source is irradiated onto a person's eyes. From the eye region included in the captured image of the camera, the corneal reflection image of the light source and the pupil center of gravity are detected. The person's gaze direction is detected according to the positional relationship between the pupil center of gravity and the corneal reflection image. Based on the gaze direction and the distance from the display unit to the person's face, the fixation position of the person on the display unit is obtained.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the gaze detection device described in Patent Document 1 above, if a person is at a position farther than the distance range in which the person's eyes can be detected from the captured image of the camera, the gaze of the person cannot be estimated. On the other hand, as described above, when detecting a person's gaze to determine whether the person has looked at digital signage, the distance to the person can extend to a range where the eyes cannot be detected.

[0006] In view of such problems, an object of the present invention is to provide a gaze estimation device, a gaze system, and a gaze estimation method capable of appropriately estimating a person's line of sight over a wide distance range.

Means for Solving the Problems

[0007] The gaze estimation device according to the first aspect of the present invention includes an image acquisition unit that acquires an imaging image including at least a part of a person from a camera, a distance acquisition unit that acquires the distance to the person, an extraction unit that extracts a plurality of gaze evaluation elements of different types from the imaging image, and a gaze estimation unit that changes the estimation process using the plurality of gaze evaluation elements according to the distance acquired by the distance acquisition unit and estimates the person's line of sight.

[0008] According to the gaze estimation device according to this aspect, since the estimation process using a plurality of gaze evaluation elements is changed according to the distance to the person, the person's line of sight can be appropriately estimated over a wide distance range.

[0009] In the gaze estimation device according to this aspect, the plurality of gaze evaluation elements may include the orientation of the person's eyeballs and the orientation of the face.

[0010] According to this configuration, since the orientation of the person's eyeballs and the orientation of the face, which have a high probability of gaze estimation, are included in the gaze evaluation elements, the person's line of sight can be accurately estimated.

[0011] In the gaze estimation device according to this aspect, the plurality of gaze evaluation elements may further include the orientation of the person's body.

[0012] According to this configuration, since the gaze evaluation elements further include the orientation of the person's body, for example, even in a relatively far distance range where the orientation of the person's eyeballs or face cannot be extracted from the imaging image, the person's line of sight can be estimated from the orientation of the person's body.

[0013] In the gaze estimation device according to this aspect, the gaze estimation unit may be configured to change the gaze evaluation elements used for gaze estimation for each distance range.

[0014] According to this configuration, by using, for each distance range, the line-of-sight evaluation elements that can be appropriately obtained and the optimal line-of-sight evaluation elements in each distance range for line-of-sight estimation, the line of sight of a person can be appropriately estimated.

[0015] In this case, the line-of-sight estimation unit may be configured to use, for each of the distance ranges, one of the line-of-sight evaluation elements that is assumed to have the highest reliability for line-of-sight estimation.

[0016] According to this configuration, since the optimal line-of-sight evaluation elements in each distance range are used for line-of-sight estimation in each distance range, the line of sight of a person can be appropriately estimated by simple processing.

[0017] Alternatively, the line-of-sight estimation unit may be configured to use, for each of the distance ranges, the line-of-sight evaluation elements preset as extractable from the captured image in each of the distance ranges among the plurality of line-of-sight evaluation elements for line-of-sight estimation in each of the distance ranges.

[0018] According to this configuration, since the line-of-sight estimation elements that can be extracted in each distance range are used for line-of-sight estimation, for a predetermined distance range, line-of-sight estimation is performed using a plurality of line-of-sight evaluation elements. Therefore, in this distance range, the line of sight of a person can be comprehensively estimated using a plurality of line-of-sight evaluation elements, and even if any of these line-of-sight evaluation elements cannot be extracted from the captured image, the line-of-sight estimation result of a person can be obtained from other line-of-sight evaluation elements.

[0019] In this configuration, when the line-of-sight estimation unit performs line-of-sight estimation in the distance range where a plurality of the line-of-sight evaluation elements are set, the line-of-sight estimation unit may be configured to integrate the line-of-sight estimation results based on each of the line-of-sight evaluation elements to obtain the line-of-sight estimation result of the person.

[0020] According to this configuration, for the distance range in which a plurality of line-of-sight evaluation elements are set, the human line of sight can be comprehensively estimated from the estimated results of the line of sight based on these line-of-sight evaluation elements. Also, even if any one of the line-of-sight evaluation elements cannot be extracted from the captured image, the estimated result of the human line of sight can be obtained from the other line-of-sight evaluation elements.

[0021] In this case, the line-of-sight estimation unit may be configured to apply weighting according to the reliability when each of the line-of-sight evaluation elements is extracted from the captured image to each of the line-of-sight evaluation elements, and integrate the estimated results of the line of sight based on each of the line-of-sight evaluation elements.

[0022] According to this configuration, since the reliability when each line-of-sight evaluation element is extracted is taken into account and the estimated result of the human line of sight is obtained, the human line of sight can be accurately estimated.

[0023] Alternatively, the line-of-sight estimation unit may be configured to apply weighting according to the probability of each of the line-of-sight evaluation elements in the line-of-sight estimation to each of the line-of-sight evaluation elements, and integrate the estimated results of the line of sight based on each of the line-of-sight evaluation elements.

[0024] For example, when the line-of-sight evaluation elements are the direction of the eyeballs, the direction of the face, and the direction of the body, the probability of line-of-sight estimation is usually eyeball direction > face direction > body direction. According to the above configuration, weighting according to such probability is applied to each line-of-sight evaluation element to obtain the line-of-sight estimation result, so the human line of sight can be accurately estimated.

[0025] In the line-of-sight estimation device according to this aspect, the line-of-sight estimation unit may be configured to apply weighting according to the distance obtained by the distance acquisition unit to each of the line-of-sight evaluation elements, integrate the estimated results of the line of sight based on each of the line-of-sight evaluation elements, and obtain the integrated estimated result as the human line of sight.

[0026] According to this configuration, as will be described later with reference to Modification Example 3, weightings according to the distance are applied to the respective line-of-sight evaluation elements, and the line-of-sight estimation results based on the respective line-of-sight evaluation elements are integrated, so that a person's line of sight can be estimated accurately.

[0027] In this case, for example, as shown in FIG. 8, ranges of distances at which the weightings of the respective line-of-sight evaluation elements increase are arranged on the distance axis, and the weightings can be set such that there exists on the distance axis a range in which one of the weightings of two adjacent line-of-sight evaluation elements decreases while the other increases.

[0028] According to this configuration, appropriate weightings can be applied to the respective line-of-sight evaluation elements. Therefore, a person's line of sight can be estimated accurately.

[0029] In the line-of-sight estimation device according to the present aspect, the distance acquisition unit may be configured to acquire the distance to the person from a distance sensor.

[0030] According to this configuration, the distance to the person can be detected accurately. Therefore, the line of sight of the person can be estimated accurately according to the distance to the person.

[0031] Alternatively, the distance acquisition unit may be configured to acquire the distance to the person from the captured image.

[0032] According to this configuration, since the distance to the person is acquired from the captured image from the camera, there is no need to use a distance sensor separately. Therefore, the configuration necessary for line-of-sight estimation can be simplified.

[0033] A line-of-sight estimation device according to a second aspect of the present invention includes an image acquisition unit that acquires a captured image including at least a part of a person from a camera, an extraction unit that extracts a plurality of line-of-sight evaluation elements of different types from the captured image, and a line-of-sight estimation unit that applies weightings according to the reliability when each of the line-of-sight evaluation elements is extracted from the captured image to the respective line-of-sight evaluation elements, integrates the line-of-sight estimation results based on the respective line-of-sight evaluation elements, and acquires the integrated estimation result as the line of sight of the person.

[0034] According to the line-of-sight estimation device according to this aspect, since a plurality of line-of-sight evaluation elements of different types are used for estimating a person's line of sight, the line of sight of a person can be estimated in a wide distance range. Further, since weighting according to the reliability when each line-of-sight evaluation element is extracted is applied to each line-of-sight evaluation element and the line-of-sight estimation results based on each line-of-sight evaluation element are integrated, the line of sight of a person can be accurately estimated.

[0035] In the line-of-sight estimation device according to the second aspect, the line-of-sight estimation unit may be configured to further apply weighting according to the certainty of each line-of-sight evaluation element in line-of-sight estimation to each line-of-sight evaluation element and integrate the line-of-sight estimation results based on each line-of-sight evaluation element.

[0036] For example, when the line-of-sight evaluation elements are the direction of the eyeball, the direction of the face, and the direction of the body, the certainty of line-of-sight estimation is usually eyeball direction > face direction > body direction. According to the above configuration, since weighting according to such certainty is applied to each line-of-sight evaluation element and a line-of-sight estimation result is obtained, the line of sight of a person can be accurately estimated.

[0037] The line-of-sight estimation system according to the third aspect of the present invention includes the line-of-sight estimation device according to the first or second aspect and a camera.

[0038] According to the line-of-sight estimation system according to this aspect, the same effects as those of the first aspect and the second aspect can be achieved.

[0039] The line-of-sight estimation system according to the fourth aspect of the present invention includes the line-of-sight estimation device according to the first aspect, a camera, and a distance sensor.

[0040] According to the line-of-sight estimation system according to this aspect, the same effects as those of the first aspect can be achieved.

[0041] A fifth aspect of the present invention is a method for estimating a line of sight in an information processing apparatus, which includes acquiring a captured image including at least a part of a person, acquiring a distance to the person, extracting a plurality of line-of-sight evaluation elements different in type from each other from the captured image, and changing a line-of-sight estimation process using the plurality of line-of-sight evaluation elements according to the distance acquired by the distance acquisition unit to estimate the line of sight of the person.

[0042] According to the line-of-sight estimation system according to this aspect, the same effect as that of the first aspect can be achieved.

[0043] A sixth aspect of the present invention is a method for estimating a line of sight in an information processing apparatus, which includes acquiring a captured image including at least a part of a person, extracting a plurality of line-of-sight evaluation elements different in type from each other from the captured image, applying a weighting according to the reliability when each of the line-of-sight evaluation elements is extracted from the captured image to each of the line-of-sight evaluation elements, integrating the estimation results of the line of sight based on each of the line-of-sight evaluation elements, and acquiring the integrated estimation result as the line of sight of the person.

[0044] According to the line-of-sight estimation system according to this aspect, the same effect as that of the second aspect can be achieved.

Advantages of the Invention

[0045] As described above, according to the present invention, it is possible to provide a line-of-sight estimation apparatus, a line-of-sight system, and a line-of-sight estimation method capable of appropriately estimating a person's line of sight in a wide distance range.

[0046] The advantages or significance of the present invention will become clearer from the following description of the embodiments. However, the following embodiments are merely examples when implementing the present invention, and the present invention is not limited to those described in the following embodiments at all.

Brief Description of the Drawings

[0047]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

MODE FOR CARRYING OUT THE INVENTION

[0048] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0049] <Embodiment 1> FIG. 1 is a diagram showing the configuration of the line-of-sight estimation system 1.

[0050] The line-of-sight estimation system 1 includes a line-of-sight estimation device 10, a camera 20, and a distance sensor 30. The camera 20 and the distance sensor 30 are connected to the line-of-sight estimation device 10 by a communication line. The camera 20, the distance sensor 30, and the line-of-sight estimation device 10 may communicate wirelessly.

[0051] The line-of-sight estimation device 10 is constituted by, for example, a personal computer. In this case, the line-of-sight estimation device 10 is constituted by installing an application program for line-of-sight estimation on the personal computer. However, it is not limited to this, and the line-of-sight estimation device 10 may be a dedicated machine.

[0052] The line-of-sight estimation device 10 estimates the line of sight of a person near the advertising device 40 based on the captured image of the camera 20 and the detection result of the distance sensor 30. The line-of-sight estimation device 10 determines whether a person near the advertising device 40 has looked at the advertising device 40 based on the estimated line of sight, and stores data based on the determination result at any time. This determination can be made, for example, based on whether the extension line of the line of sight is included in the display area 41 based on the direction of the person's line of sight, the distance to the person, and the display area 41 of the advertising device 40 on a predetermined three-dimensional coordinate region. However, the determination method is not limited to this. For example, this determination may be made using a machine learning model that uses information on the direction of the line of sight and whether the display area 41 has been viewed as teacher data.

[0053] The advertising device 40 is, for example, a digital signage. The advertising device 40 displays a predetermined advertising screen in the display area 41 under the control of a control device (not shown). The camera 20 is installed on the upper surface of the advertising device 40. The camera 20 faces the same direction as the advertising device 40. Therefore, the camera 20 captures an image of a person near the front of the advertising device 40 from the side of the advertising device 40.

[0054] The captured image captured by the camera 20 is transmitted to the gaze estimation device 10 at any time. The gaze estimation device 10 extracts a person's eyes, face, and body from the captured image for one frame. The gaze estimation device 10 is equipped with a recognition engine for recognizing these parts of a person from the captured image.

[0055] During the recognition process of these parts, the recognition engine outputs the reliability of the recognition results for these parts, that is, the matching rate of the data of each part obtained from the captured image with respect to the reference data of these parts. The gaze estimation device 10 extracts the area on the captured image where this reliability is equal to or higher than a predetermined threshold as the area of each part. In this way, the gaze estimation device 10 extracts a person's eyes, face, and body from the captured image.

[0056] Furthermore, the gaze estimation device 10 obtains the direction of the eyeballs, the direction of the face, and the direction of the body from the extracted person's eyes, face, and body. The gaze estimation device 10 estimates the gaze of the person from the obtained direction of the eyeballs, the direction of the face, and the direction of the body, respectively. That is, the gaze estimation device 10 estimates the direction of the eyeballs, the direction of the face, and the direction of the body on a predetermined three-dimensional coordinate axis set in the imaging field of the camera 20 as the direction of the gaze of the person, respectively, and estimates the straight lines extending from near the person's eyes in each direction as the gaze.

[0057] The direction of the eyeballs is obtained, for example, as the direction in which the black eyes are directed. The direction of the face is obtained, for example, as the direction in which the nose is directed. The direction of the body is obtained, for example, as the direction in which the skeletal center (the center between the left and right shoulders) is directed. The gaze estimation device 10 obtains, for example, keypoint data from the captured image and obtains the direction of the face and the direction of the body from the obtained keypoint data.

[0058] Here, the keypoint data is coordinate data of the following positions on the above three-dimensional coordinate axis.

[0059] <Keypoint> Skeletal center (center between the left and right shoulders), left ankle, right ankle, left ear, right ear, left elbow, right elbow, left eye, right eye, left thigh, right thigh, left knee, right knee, neck, nose, left shoulder, right shoulder, left wrist, right wrist, left corner of the mouth, right corner of the mouth On the upper surface of the advertising device 40, a distance sensor 30 is further installed. The distance sensor 30 is arranged adjacent to the camera 20 and faces the same direction as the camera 20. The distance sensor 30 acquires the distance to a person in the vicinity in front of the advertising device 40.

[0060] The distance sensor 30 is, for example, a TOF (Time Of Flight) camera. The distance sensor 30 acquires the distance to an object existing within the field of view that covers the imaging field of view of the camera 20. The distance sensor 30 may be a distance sensor other than a TOF camera.

[0061] The distance sensor 30 includes, for example, a light emitting source, an image sensor, and an optical system that condenses the reflected light from the object of the light emitted from the light emitting source onto the image sensor. The distance is calculated for each pixel based on the time difference between the light emission timing of the light emitting source and the light reception timing of each pixel on the image sensor. The position of each pixel corresponds to the position of each cell that divides the field of view of the distance sensor 30 in a matrix form. Therefore, the distance calculated for each pixel corresponds to the distance to the object in the direction of the corresponding cell.

[0062] The distance sensor 30 generates a distance image for one frame in which distance information is embedded at the position of each pixel. The distance sensor 30 transmits the generated distance image to the gaze estimation device 10 at any time.

[0063] The gaze estimation device 10 acquires the distance to the person in the captured image based on the distance image acquired from the distance sensor 30 and the captured image acquired from the camera 20. Specifically, the gaze estimation device 10 acquires the distance of the pixel position on the distance image corresponding to the person's body extracted from the captured image as the distance to the person.

[0064] In this case, the line-of-sight estimation device 10 acquires, as the distance to the person, the distance of the pixel position on the distance image corresponding to a position near the center of the region of the person's body on the captured image. When the body cannot be extracted from the captured image, the line-of-sight estimation device 10 acquires, as the distance to the person, the distance of the pixel position on the distance image corresponding to a position near the center of the region of the person's face on the captured image. Further, when neither the body nor the entire face can be extracted from the captured image, the line-of-sight estimation device 10 acquires, as the distance to the person, the distance of the pixel position on the distance image corresponding to a position near the center of a part of the face (for example, the region of the person's eyes) on the captured image.

[0065] The line-of-sight estimation device 10 estimates the line of sight of the person based on the distance to the person, the orientation of the person's eyeballs, the orientation of the face, and the orientation of the body. The method of estimating the line of sight will be described later with reference to FIGS. 3 and 4.

[0066] FIG. 2 is a block diagram showing the configuration of the line-of-sight estimation device 10 that constitutes the line-of-sight estimation system 1.

[0067] The line-of-sight estimation device 10 includes a control unit 11, a storage unit 12, a display unit 13, an input unit 14, and a communication unit 15. The control unit 11 includes an arithmetic processing circuit such as a CPU and controls each unit according to a program stored in the storage unit 12. The storage unit 12 includes a memory such as a ROM, a RAM, and a hard disk and stores the above program. The display unit 13 includes a display such as a liquid crystal panel and displays a predetermined screen under the control of the control unit 11. The input unit 14 has input means such as a keyboard and a mouse. The communication unit 15 communicates with the camera 20 and the distance sensor 30 under the control of the control unit 11.

[0068] In the present embodiment, the control unit 11 executes the functions of the image acquisition unit 11a, the distance acquisition unit 11b, the extraction unit 11c, and the line-of-sight estimation unit 11d according to a program stored in the storage unit 12.

[0069] The image acquisition unit 11a acquires a captured image from the camera 20. The distance acquisition unit 11b acquires a distance image from the distance sensor 30 and obtains the distance to the person in the captured image. The method for obtaining the distance to the person is as described above. The extraction unit 11c extracts a plurality of gaze evaluation elements of different types from the captured image. In the present embodiment, as the gaze evaluation elements, the above-described eye direction, face direction, and body direction are extracted. The extraction methods for the eye direction, face direction, and body direction are as described above. The gaze estimation unit 11d changes the estimation process using the plurality of gaze evaluation elements (eye direction, face direction, and body direction) according to the distance obtained by the distance acquisition unit 11b, and estimates the gaze of the person in the captured image.

[0070] FIG. 3 is a diagram showing a processing mode used for gaze estimation.

[0071] In the first embodiment, three processing modes from the first mode to the third mode are prepared according to the distance. The first mode is a mode for estimating the gaze only by the eye direction. The second mode is a mode for estimating the gaze only by the face direction. The third mode is a mode for estimating the gaze only by the body direction.

[0072] When the distance to the person obtained by the distance acquisition unit 11b is in the range greater than 0 and less than or equal to D1, the gaze is estimated in the first mode. When the distance to the person is in the range greater than D1 and less than or equal to D2, the gaze is estimated in the second mode. When the distance to the person is in the range greater than D2 and less than or equal to D3, the gaze is estimated in the third mode.

[0073] The distance range from 0 to D1 is a range within which the orientation of the eyeball can be obtained more appropriately than other gaze evaluation elements. That is, the distance range from 0 to D1 is a range assumed to be able to extract a person's eyes from the captured image more appropriately than the face and body. The distance range from D1 to D2 is a range within which the orientation of the face can be obtained more appropriately than other gaze evaluation elements. That is, the distance range from D1 to D2 is a range assumed to be unable to extract a person's eyes appropriately from the captured image, but able to extract the person's face more appropriately than the body. The distance range from D2 to D3 is a range within which the orientation of the body can be obtained more appropriately than other gaze evaluation elements. That is, the distance range from D2 to D3 is a range assumed to be unable to extract a person's eyes and face appropriately from the captured image, but able to extract the person's body appropriately.

[0074] Here, the area of the eyes on the captured image gradually becomes smaller as the distance to the person increases. For this reason, the area of the eyes becomes more difficult to extract as the distance increases, and the orientation of the eyeball also becomes more difficult to extract as the distance increases. From this perspective, the upper limit distance D1 of the first mode is set.

[0075] Also, if the distance to the person is too small, the entire face area on the captured image is difficult to be included in the captured image. For this reason, it is difficult to extract the orientation of the face from the captured image in a small distance range. On the other hand, after the entire face is included in the captured image, the area of the face on the captured image gradually becomes smaller as the distance to the person increases. For this reason, the area of the face becomes more difficult to extract as the distance increases, and the orientation of the face also becomes more difficult to extract as the distance increases. From this perspective, the upper limit distance D2 of the second mode is set.

[0076] Similarly, if the distance to the person is too small, the entire body area on the captured image is difficult to be included in the captured image. After the entire body is included in the captured image, the area of the body on the captured image gradually becomes smaller as the distance to the person increases. For this reason, the area of the body also becomes more difficult to extract as the distance increases after the entire body is included in the captured image, and the orientation of the body also becomes more difficult to extract as the distance increases. From this perspective, the upper limit distance D3 of the third mode is set.

[0077] In this way, the upper limit distances D1, D2, and D3 are set as the upper limits of the distances at which the direction of the eyeball, the direction of the face, and the direction of the body can be appropriately extracted from the captured image. However, the distances D1 to D3 do not necessarily have to be set from this perspective. For example, the distance D1 may be set to a distance at which the reliability of detecting the direction of the eyeball and the direction of the face is about the same, and the distance D2 may be set to a distance at which the reliability of detecting the direction of the face and the direction of the body is about the same.

[0078] FIG. 4 is a flowchart showing the gaze estimation process.

[0079] In the flowchart of FIG. 4, steps S101, S102, and S103 are executed by the control unit 11 according to the functions of the image acquisition unit 11a, the extraction unit 11c, and the distance acquisition unit 11b in FIG. 2, respectively, and steps S104 to S106 are executed by the control unit 11 according to the function of the gaze estimation unit 11d in FIG. 2. Hereinafter, an explanation will be given assuming that the control unit 11 executes the process of FIG. 4 according to these functions.

[0080] The control unit 11 acquires a captured image for one frame from the camera 20 (S101), and extracts gaze evaluation elements (the direction of the eyeball, the direction of the face, the direction of the body) for estimating the gaze of the person on the captured image from the acquired captured image (S102). The control unit 11 acquires a distance image from the distance sensor 30 and acquires the distance to the person on the captured image (S103). The control unit 11 sets the processing mode corresponding to the distance to the person among the three processing modes (the first mode to the third mode) shown in FIG. 3 as the processing mode to be used for gaze estimation (S104).

[0081] The control unit 11 determines whether or not a line-of-sight evaluation element corresponding to the processing mode set in step S104 has been extracted in step S102 (S105). For example, when the processing mode set in step S104 is the first mode, the control unit 11 determines whether or not a line-of-sight evaluation element of the direction of the eyeball has been extracted in step S102. For example, when a person's face is not facing the direction of the camera 20 and the captured image does not include the person's eyes, the line-of-sight evaluation element of the direction of the eyeball cannot be extracted in step S102. In such a case, the determination in step S105 is NO.

[0082] If the determination in step S105 is NO, the control unit 11 ends the current process without obtaining the estimation result of the line of sight of the person. On the other hand, if the determination in step S105 is YES, the control unit 11 uses the line-of-sight evaluation element corresponding to the processing mode set in step S104 to obtain the estimation result of the line of sight of the person by the above method, and stores the obtained estimation result in the storage unit 12 (S106). Thereby, the control unit 11 ends the process of FIG. 4.

[0083] When the captured image includes a plurality of people, the control unit 11 executes the processes of steps S102 to S106 for each person. This is the same in the modification example and the second embodiment shown below. After the process of FIG. 4 is completed, the control unit 11 returns the process to step S101 and executes the next estimation process. In this way, the control unit 11 repeatedly executes the process of FIG. 4.

[0084] <Effect of Embodiment 1> According to this embodiment, the following effects can be achieved.

[0085] As shown in FIG. 2, the line-of-sight estimation device 10 includes an image acquisition unit 11a that acquires a captured image including at least a part of a person from a camera 20, a distance acquisition unit 11b that acquires the distance to the person, an extraction unit 11c that extracts a plurality of line-of-sight evaluation elements (orientation of the eyeballs, orientation of the face, orientation of the body) different in type from each other from the captured image, and a line-of-sight estimation unit 11d that estimates the line of sight of the person by changing the estimation process (first mode, second mode, third mode) using the plurality of line-of-sight evaluation elements according to the distance acquired by the distance acquisition unit 11b.

[0086] According to this configuration, since the estimation process using the plurality of line-of-sight evaluation elements (orientation of the eyeballs, orientation of the face, orientation of the body) is changed according to the distance to the person, the line of sight of the person can be appropriately estimated in a wide distance range.

[0087] As shown in FIG. 3, the plurality of line-of-sight evaluation elements include the orientation of the person's eyeballs and the orientation of the face.

[0088] According to this configuration, since the orientation of the person's eyeballs and the orientation of the face, which have a high probability of correct line-of-sight estimation, are included in the line-of-sight evaluation elements, the line of sight of the person can be accurately estimated.

[0089] As shown in FIG. 3, the plurality of line-of-sight evaluation elements further include the orientation of the person's body.

[0090] According to this configuration, since the line-of-sight evaluation elements further include the orientation of the person's body, for example, even in a relatively far distance range where the orientation of the person's eyeballs or the orientation of the face cannot be extracted from the captured image, the line of sight of the person can be estimated from the orientation of the person's body.

[0091] As shown in FIG. 3, the line-of-sight estimation unit 11d changes the line-of-sight evaluation elements (orientation of the eyeballs, orientation of the face, orientation of the body) used for line-of-sight estimation for each distance range.

[0092] According to this configuration, by using the line-of-sight evaluation elements that can be appropriately acquired in each distance range and the optimal line-of-sight evaluation elements in each distance range for line-of-sight estimation for each distance range, the line of sight of the person can be appropriately estimated.

[0093] As shown in FIG. 3, for each of the distance ranges, the line-of-sight estimation unit 11d uses one line-of-sight evaluation element that is assumed to have the highest reliability for line-of-sight estimation.

[0094] According to this configuration, since the optimal line-of-sight evaluation element in each distance range is used for line-of-sight estimation in each distance range, the line of sight of a person can be appropriately estimated by simple processing.

[0095] As shown in FIG. 2, the distance acquisition unit 11b acquires the distance to a person from the distance sensor 30.

[0096] According to this configuration, the distance to a person can be accurately detected. Therefore, the line of sight of the person can be accurately estimated according to the distance to the person.

[0097] As shown in FIGS. 1 and 2, the line-of-sight estimation system 1 includes a line-of-sight estimation device 10, a camera 20, and a distance sensor 30.

[0098] According to this configuration, as described above, in the line-of-sight estimation device 10, the estimation process (first mode, second mode, third mode) using a plurality of line-of-sight evaluation elements (direction of the eyeball, direction of the face, direction of the body) is changed according to the distance to a person. Therefore, the line of sight of a person can be appropriately estimated in a wide distance range.

[0099] <Modification Example 1> FIG. 5 is a flowchart showing the line-of-sight estimation process according to Modification Example 1.

[0100] Compared with the flowchart of FIG. 4, the flowchart of FIG. 5 has steps S111 and S112 added. The processes of steps S111 and S112 are performed by the control unit 11 by the function of the line-of-sight estimation unit 11d.

[0101] If the control unit 11 cannot extract the line-of-sight evaluation element (the direction of the eyeball, the direction of the face, or the direction of the body) corresponding to the processing mode (the first mode, the second mode, or the third mode) set in step S104 in step S102 (S105: NO), it determines whether the processing mode used for line-of-sight estimation can be changed to another processing mode (S111). Then, if the determination in step S111 is YES, the control unit 11 changes the processing mode used for line-of-sight estimation to another processing mode (S112).

[0102] For example, when the processing mode set in step S104 is the first mode, if the control unit 11 cannot extract the direction of the eyeball, which is the line-of-sight evaluation element corresponding to the first mode, in step S102 (S105: NO), it determines whether it can extract the direction of the face or the direction of the body, which are other line-of-sight evaluation elements, in step S102 (S111). Then, if the control unit 11 can extract the direction of the face or the direction of the body (S111: YES), it changes the processing mode used for line-of-sight estimation to the second mode or the third mode corresponding to the direction of the face or the direction of the body (S112).

[0103] Here, when there are multiple other changeable processing modes, the control unit 11 changes the processing mode used for line-of-sight estimation to the processing mode associated with a closer distance range. For example, when the other changeable processing modes are the second mode and the third mode corresponding to the direction of the face and the direction of the body respectively, the control unit 11 changes the processing mode used for line-of-sight estimation to the second processing mode (direction of the face) associated with a closer distance range. This is because usually, using the line-of-sight evaluation element of the processing mode associated with a closer distance range results in higher accuracy (certainty) of line-of-sight estimation. That is, the priority (certainty) of each line-of-sight evaluation element in line-of-sight estimation is: direction of the eyeball > direction of the face > direction of the body.

[0104] The control unit 11 obtains the estimated result of the person's line of sight using the processing mode thus changed (S106). On the other hand, when the determination in step S111 is NO, the control unit 11 ends the processing of FIG. 5 without obtaining the estimated result of the person's line of sight.

[0105] According to the processing of Modification Example 1, even if the determination in step S105 is NO, if the determination in step S111 is YES, the estimated result of the person's line of sight can be obtained. Therefore, more line-of-sight estimation results can be obtained compared to the processing of FIG. 4.

[0106] In the processing of FIG. 5, since the processing mode set in step S112 is not a processing mode according to the distance to the person, the accuracy of the line-of-sight estimation result may be inferior compared to the case where the processing mode set in step S104 is used for line-of-sight estimation. For this reason, in the processing of FIG. 5, a flag indicating which of the processing modes set in step S104 and step S112 was used for line-of-sight estimation may be associated with the line-of-sight estimation result and stored in the storage unit 12. Furthermore, the type of line-of-sight evaluation element set in step S112 may be associated with the line-of-sight estimation result. Thereby, in subsequent statistical processing of the line-of-sight estimation result, by using this information, a more accurate statistical processing result can be obtained.

[0107] <Modification Example 2> FIG. 6 is a diagram showing a processing mode used for line-of-sight estimation according to Modification Example 2.

[0108] In the above-described Embodiment 1, as shown in FIG. 3, three processing modes from the first mode to the third mode are prepared according to the distance, and one line-of-sight evaluation element (eye direction / face direction / body direction) is assigned to each processing mode. In contrast, in Modification Example 2, as shown in FIG. 6, a plurality of line-of-sight evaluation elements are assigned to the first mode and the second mode.

[0109] That is, in the relatively short distance range from 0 to D1, it is optimal to use the orientation of the eyeball for line-of-sight estimation. However, when a part or the whole of the face or a part of the body is reflected in the captured image, it may be possible to extract the orientation of the face or the body from these images. Therefore, in the first mode corresponding to the distance from 0 to D1, as line-of-sight evaluation elements for line-of-sight estimation, in addition to the orientation of the eyeball, the orientation of the face and the orientation of the body are assigned.

[0110] Also, in the medium distance range from D1 to D2, it is optimal to use the orientation of the face for line-of-sight estimation. However, when a part or the whole of the body is reflected in the captured image, it may be possible to extract the orientation of the body from this image. On the other hand, when the distance exceeds D1, as described above, the orientation of the eyeball cannot be properly extracted from the captured image. Therefore, in the second mode corresponding to the distance from D1 to D2, as line-of-sight evaluation elements for line-of-sight estimation, in addition to the orientation of the face, the orientation of the body is assigned.

[0111] Also, in the relatively long distance range from D2 to D3, it is optimal to use the orientation of the body for line-of-sight estimation. Also, in this distance range, as described above, neither the orientation of the eyeball nor the orientation of the face can be properly extracted from the captured image. Therefore, in the third mode corresponding to the distance from D2 to D3, as the line-of-sight evaluation element for line-of-sight estimation, only the orientation of the body is assigned.

[0112] Also in Modification 2, the line-of-sight estimation process is the same as in FIG. 4. However, in Modification 2, the process of step S106 in FIG. 4 is changed as follows from Embodiment 1.

[0113] (Estimation Process 1) In Estimation Process 1, when each line-of-sight evaluation element (orientation of the eyeball, orientation of the face, orientation of the body) is extracted from the captured image, weighting according to the reliability is applied to each line-of-sight evaluation element, and the line-of-sight estimation results based on each line-of-sight evaluation element are integrated. Specifically, the line-of-sight estimation result is obtained by the following formula.

[0114]

Equation

[0115] Here, the above-described three-dimensional coordinate axes for defining the line-of-sight evaluation elements (the direction of the eyeballs, the direction of the face, and the direction of the body) are represented by x, y, and z. Also, the variable i is a variable indicating the type of line-of-sight evaluation element used for line-of-sight estimation. In the example of FIG. 6, since there are three types of line-of-sight evaluation elements, i.e., the direction of the eyeballs, the direction of the face, and the direction of the body, when the first mode is used for line-of-sight estimation, the variable i is an integer from 1 to 3. In this case, for example, the variable i = 1 indicates the direction of the eyeballs, the variable i = 2 indicates the direction of the face, and the variable i = 3 indicates the direction of the body. n is the total number of types of line-of-sight evaluation elements used for line-of-sight estimation.

[0116] In Equation (1), X(x, y, z) i represents the direction of the line-of-sight evaluation element (i) in terms of the components of the x-axis, y-axis, and z-axis of the above-described three-dimensional coordinate axes. wc i is a weighting according to the reliability when each line-of-sight evaluation element (the direction of the eyeballs, the direction of the face, the direction of the body) is extracted from the captured image.

[0117] As described above, the line-of-sight estimation device 10 is equipped with a recognition engine for extracting a person's eyes, face, and body from the captured image for one frame. This recognition engine outputs the reliability of the recognition result for these parts during the recognition process of these parts. The weighting wc in the above Equation (1) i is a value according to the respective reliabilities when extracting a person's eyes, face, and body from the captured image, and is applied to the line-of-sight evaluation elements of the direction of the eyeballs, the direction of the face, and the direction of the body, respectively. Each weighting wc i is set in the range of 0.0 to 1.0.

[0118] Therefore, angle(x, y, z) in Equation (1) is obtained by multiplying the X-axis component, Y-axis component, and Z-axis component of the direction of each line-of-sight evaluation element (i) by the weighting wc i corresponding to the reliability of each line-of-sight evaluation element (i) and integrating (adding) them vectorially.

[0119] When executing the first mode of FIG. 6, the control unit 11 determines the orientation X (x, y, z) corresponding to each of the three types of line-of-sight evaluation elements (i = 1, 2, 3) of the orientation of the eyeball, the orientation of the face, and the orientation of the body i and the weighting wc i are applied to Equation (1) to obtain the estimated line-of-sight result angle (x, y, z) for the person.

[0120] When executing the second mode, the control unit 11 determines the orientation X (x, y, z) corresponding to each of the two types of line-of-sight evaluation elements (i = 1, 2) of the orientation of the face and the orientation of the body i and the weighting wc i are applied to Equation (1) to obtain the estimated line-of-sight result angle (x, y, z) for the person.

[0121] When executing the third mode, the control unit 11 determines the orientation X (x, y, z) corresponding to only the orientation of the body as the line-of-sight evaluation element (i = 1) i and the weighting wc i are applied to Equation (1) to obtain the estimated line-of-sight result angle (x, y, z) for the person.

[0122] According to this estimation process, the reliability when each line-of-sight evaluation element is extracted is taken into account, and the estimated result of the person's line of sight is obtained, so that the person's line of sight can be accurately estimated.

[0123] Note that, without applying the weighting wc i to each line-of-sight evaluation element, the orientation X (x, y, z) i of the line-of-sight evaluation element (i) used for line-of-sight estimation may be simply integrated vectorially. In this case, 1 is set for the weighting wc i in the above Equation (1) for any line-of-sight evaluation element. However, in this case, since the estimated line-of-sight result is obtained without taking into account the reliability of the line-of-sight evaluation element as described above, the line-of-sight estimation accuracy is lower than the above. In order to obtain a more accurate line-of-sight estimation result, it is preferable to apply the weighting wc i corresponding to the reliability of each line-of-sight evaluation element to the above Equation (1) to obtain the estimation result.

[0124] (Estimation Process 2) In Estimation Process 2, weights corresponding to the probabilities of each line-of-sight evaluation element in line-of-sight estimation are applied to each line-of-sight evaluation element, and the line-of-sight estimation results based on each line-of-sight evaluation element are integrated. Specifically, the line-of-sight estimation result is obtained by the following formula.

[0125] [Number]

[0126] In Equation (2), the weight wc in Equation (1) i is changed to the weight wk i . When not considering the distance to a person, the probability of line-of-sight estimation is usually Eye direction > Face direction > Body direction. That is, since the eye direction is directly related to the line of sight, the line-of-sight estimation accuracy based on the eye direction is the highest. Also, since a person is more likely to face in the direction of the line of sight, the line-of-sight estimation accuracy based on the face direction is the second highest after the eye direction. Also, since a person is less likely to turn the body than the face in the direction of the line of sight, the line-of-sight estimation accuracy based on the body direction is the lowest. Therefore, when not considering the distance to a person, the probability of line-of-sight estimation is usually Eye direction > Face direction > Body direction.

[0127] The weight wk in Equation (2) i is set considering the probabilities of such line-of-sight evaluation elements. For example, for each line-of-sight evaluation element of eye direction, face direction, and body direction, the weight wk i is set to 0.5, 0.3, and 0.2 respectively, as shown in the following formula.

[0128] wk(Eye direction, Face direction, Body direction) = (0.5, 0.3, 0.2) Therefore, angle(x, y, z) in Equation (1) is obtained by multiplying the X-axis component, Y-axis component, and Z-axis component of the direction of each line-of-sight evaluation element (i) by the weight wk i corresponding to the probability of each line-of-sight evaluation element (i) and integrating (adding) them vectorially.

[0129] When executing the first mode of FIG. 6, the control unit 11 has orientations X(x, y, z) corresponding to three types of line-of-sight evaluation elements (i = 1, 2, 3) of the orientation of the eyeball, the orientation of the face, and the orientation of the body respectively i and weights wk i are applied to Equation (2) to obtain the estimated line-of-sight result angle(x, y, z) for the person

[0130] When executing the second mode, the control unit 11 has orientations X(x, y, z) corresponding to two types of line-of-sight evaluation elements (i = 1, 2) of the orientation of the face and the orientation of the body respectively i and weights wk i are applied to Equation (2) to obtain the estimated line-of-sight result angle(x, y, z) for the person

[0131] When executing the third mode, the control unit 11 has the orientation X(x, y, z) corresponding only to the orientation of the body as the line-of-sight evaluation element (i = 1) i and weights wk i are applied to Equation (2) to obtain the estimated line-of-sight result angle(x, y, z) for the person

[0132] According to this estimation process, weights corresponding to the reliability of each line-of-sight evaluation element are applied to each line-of-sight evaluation element, and the line-of-sight estimation result is obtained. Therefore, the line of sight of a person can be accurately estimated

[0133] Note that weights wc corresponding to the reliability of each line-of-sight evaluation element in the above estimation process 1 i and weights wk corresponding to the likelihood of each line-of-sight evaluation element in the estimation process 2 i Both can be applied to obtain the estimated line-of-sight result for the person from the following equation

[0134]

Equation

[0135] According to this estimation process, when extracting each line-of-sight evaluation element, the reliability when extracting each line-of-sight evaluation element and the probability of the line-of-sight estimation by each line-of-sight evaluation element are taken into account, and the estimation result of the person's line of sight is obtained. Therefore, the person's line of sight can be estimated accurately.

[0136] <Effect of Modification Example 2> As shown in FIG. 6, the line-of-sight estimation unit 11d uses, for line-of-sight estimation in each distance range, the line-of-sight evaluation elements preset as extractable from the captured image among a plurality of line-of-sight evaluation elements (orientation of the eyeball, orientation of the face, orientation of the body).

[0137] According to this configuration, since the line-of-sight estimation elements that can be extracted in each distance range are used for line-of-sight estimation, for a predetermined distance range, line-of-sight estimation is performed using a plurality of line-of-sight evaluation elements. Therefore, in this distance range, the person's line of sight can be comprehensively estimated using a plurality of line-of-sight evaluation elements, and even if any of these line-of-sight evaluation elements cannot be extracted from the captured image, the estimation result of the person's line of sight can be obtained from the other line-of-sight evaluation elements.

[0138] As shown in the above formulas (1) to (3), when the line-of-sight estimation unit 11d performs line-of-sight estimation in a set distance range of a plurality of line-of-sight evaluation elements (orientation of the eyeball, orientation of the face, orientation of the body / orientation of the face, orientation of the body), the line-of-sight estimation results based on each line-of-sight evaluation element, that is, the orientation X (x, y, z) of the line-of-sight evaluation element (i) i are integrated to obtain the estimation result of the person's line of sight.

[0139] According to this configuration, for the distance range in which a plurality of line-of-sight evaluation elements are set, the person's line of sight can be comprehensively estimated from the line-of-sight estimation results based on these line-of-sight evaluation elements, and even if any of the line-of-sight evaluation elements cannot be extracted from the captured image, the estimation result of the person's line of sight can be obtained from the other line-of-sight evaluation elements.

[0140] In the estimation process 1 using the above formula (1), the line-of-sight estimation unit 11d weights wc according to the reliability when each line-of-sight evaluation element is extracted from the captured image iApply to each line-of-sight evaluation element to obtain the orientation X(x, y, z), which is the line-of-sight estimation result based on each line-of-sight evaluation element i Integrate them.

[0141] According to this configuration, since the reliability when each line-of-sight evaluation element is extracted is taken into account and the line-of-sight estimation result of a person is obtained, the line-of-sight of a person can be estimated accurately.

[0142] In the estimation process 2 using the above formulas (2) and (3), the line-of-sight estimation unit 11d assigns weights wk i according to the certainty of each line-of-sight evaluation element in line-of-sight estimation, and applies them to each line-of-sight evaluation element to obtain the orientation X(x, y, z), which is the line-of-sight estimation result based on each line-of-sight evaluation element i Integrate them.

[0143] As described above, the certainty of line-of-sight estimation is generally in the order of eye orientation > face orientation > body orientation. Therefore, according to the estimation process 2, weights wk i corresponding to such certainty are applied to each line-of-sight evaluation element to obtain the line-of-sight estimation result, so the line-of-sight of a person can be estimated accurately.

[0144] <Modification Example 3> FIG. 7 is a flowchart showing the line-of-sight estimation process according to Modification Example 3.

[0145] In the above-described Embodiment 1, depending on the distance to a person, any one of the first process, the second process, and the third process was used for distance estimation. In contrast, in the modification example, weights corresponding to the distance to a person are applied to each line-of-sight evaluation element, the line-of-sight estimation results based on each line-of-sight evaluation element are integrated, and the estimation result obtained by this integration is acquired as the line-of-sight of a person.

[0146] In the flowchart of FIG. 7, the processes of steps S121, S122, and S124 are executed by the control unit 11 by the functions of the image acquisition unit 11a, the extraction unit 11c, and the distance acquisition unit 11b shown in FIG. 2, respectively, and the processes of the other steps are executed by the control unit 11 by the function of the gaze estimation unit 11d in FIG. 2. Hereinafter, it will be described on the assumption that the control unit 11 executes the process of FIG. 7 by these functions.

[0147] The control unit 11 acquires a captured image for one frame from the camera 20 (S121), and extracts a plurality of types of gaze evaluation elements (direction of the eyeballs, direction of the face, direction of the body) for estimating the gaze of the person in the captured image from the acquired captured image (S122). The control unit 11 determines whether or not at least one of these gaze evaluation elements has been extracted (S123). For example, if none of the eyes, face, or body can be extracted from the captured image, the determination in step S123 is NO. In this case, the control unit 11 ends the current process without obtaining the gaze estimation result.

[0148] When any of a plurality of types of gaze evaluation elements (direction of the eyeballs, direction of the face, direction of the body) has been extracted (S123: YES), the control unit 11 acquires a distance image from the distance sensor 30 and acquires the distance to the person in the captured image (S124). Then, the control unit 11 applies a weighting according to the acquired distance to each gaze evaluation element (S125), and obtains an estimation result of the gaze with respect to the person (S126).

[0149] FIG. 8 is a graph schematically showing an example of a method for setting the weighting applied to each gaze evaluation element according to the distance.

[0150] The horizontal axis in FIG. 8 is the distance, and the vertical axis is the weighting value. Here, the maximum value of the distance is set to 20 m.

[0151] In the distance range ΔD1 where the image of the eyes including the black eyes can be appropriately extracted from the captured image and the direction of the eyeballs can be appropriately obtained, the weighting applied to the gaze evaluation element of the direction of the eyeballs is set to 1, and the weighting applied to the other evaluation elements is set to 0.

[0152] When the distance to the person exceeds the distance range ΔD1, it becomes increasingly difficult to extract the image of the eye including the black eye from the captured image. Therefore, in the distance range ΔD2, the weighting applied to the line-of-sight evaluation element of the direction of the eyeball is adjusted to decrease from 1 to 0 as the distance increases. On the other hand, in this distance range ΔD2, since the face image can be extracted so that the direction of the face can be extracted from the captured image, the weighting of the line-of-sight evaluation element of the direction of the face is set to increase from 0 to 1 as the weighting of the line-of-sight evaluation element of the direction of the eyeball decreases. At each distance in the distance range ΔD2, the weightings of both are set so that the sum of the weighting value of the direction of the eyeball and the weighting value of the direction of the face becomes 1.

[0153] In the distance range ΔD3 following the distance range ΔD2, since the entire face is included relatively large in the captured image, the direction of the face can be appropriately extracted. Therefore, in the distance range ΔD3, the weighting applied to the line-of-sight evaluation element of the direction of the face is set to 1, and the weightings applied to other evaluation elements are set to 0.

[0154] When the distance to the person exceeds the distance range ΔD3, it becomes increasingly difficult to extract the face image from the captured image. Therefore, in the distance range ΔD4, the weighting applied to the line-of-sight evaluation element of the direction of the face is adjusted to decrease from 1 to 0 as the distance increases. On the other hand, in this distance range ΔD4, since the body image can be extracted so that the direction of the body can be extracted from the captured image, the weighting of the line-of-sight evaluation element of the direction of the body is set to increase from 0 to 1 as the weighting of the line-of-sight evaluation element of the direction of the face decreases. At each distance in the distance range ΔD4, the weightings of both are set so that the sum of the weighting value of the direction of the face and the weighting value of the direction of the body becomes 1.

[0155] In the distance range ΔD5 following the distance range ΔD4, since the entire body is included relatively large in the captured image, the direction of the body can be appropriately extracted. Therefore, in the distance range ΔD5, the weighting applied to the line-of-sight evaluation element of the direction of the body is set to 1, and the weightings applied to other evaluation elements are set to 0.

[0156] In the storage unit 12 shown in FIG. 2, information defining the relationship between the distance as shown in FIG. 8 and the weighting of each line-of-sight evaluation element is stored. This information may be a table associating the weighting of each line-of-sight evaluation element with each distance, or may be a calculation formula defining the relationship of FIG. 8 for each line-of-sight evaluation element.

[0157] The control unit 11 acquires, for each line-of-sight evaluation element, the weighting corresponding to the distance acquired in step S124 of FIG. 7 from the above information stored in the storage unit 12, and applies these acquired weightings to the respective line-of-sight evaluation elements in step S125. Then, in step S126, the control unit 11 obtains an estimated result of the line of sight from the following formula.

[0158]

Equation

[0159] The weighting wd in Equation (4) i is the weighting of the line-of-sight evaluation element (i) and is obtained from the information defining the relationship of FIG. 8. n is the number of line-of-sight evaluation elements extracted in step S122 of FIG. 7.

[0160] For example, when the line-of-sight evaluation elements extracted in step S122 are the direction of the eyeball and the direction of the face, n is 2. In this case, the control unit 11 substitutes X(x, y, z)1 indicating the direction of the eyeball, X(x, y, z)2 indicating the direction of the face, and the respective weightings wd1 and wd2 of the direction of the eyeball and the direction of the face corresponding to the distance acquired in step S124 into the above Equation (4) to obtain a detection result of the line of sight (steps S125 and S126 of FIG. 7).

[0161] Also, when the gaze evaluation elements extracted in step S122 are the direction of the eyeballs, the direction of the face, and the direction of the body, n becomes 3. In this case, the control unit 11 substitutes X(x, y, z)1 indicating the direction of the eyeballs, X(x, y, z)2 indicating the direction of the face, X(x, y, z)3 indicating the direction of the body, and the respective weightings wd1, wd2, wd3 of the direction of the eyeballs, the direction of the face, and the direction of the body corresponding to the distance obtained in step S124 into the above formula (4) respectively, and obtains the detection result of the gaze (steps S125 and S126 in FIG. 7).

[0162] <Effect of Modification Example 3> As shown in FIG. 8 and the above formula (4), the gaze estimation unit 11d applies the weighting wd i corresponding to the distance obtained by the distance acquisition unit 11b to each gaze evaluation element (direction of the eyeballs, direction of the face, direction of the body), and integrates the direction X(x, y, z) i which is the estimation result of the gaze based on each gaze evaluation element, and obtains the integrated estimation result as the gaze of a person.

[0163] According to this configuration, as described above, the weighting according to the distance is applied to each gaze evaluation element, and the estimation results of the gaze based on each gaze evaluation element are integrated, so that the gaze of a person can be accurately estimated.

[0164] As shown in FIG. 8, the distance ranges in which the weightings of each gaze evaluation element increase are arranged on the distance axis, and the weighting wd i is adjusted so that there is a range on the distance axis where one of the weightings of two adjacent gaze evaluation elements decreases while the other increases.

[0165] According to this configuration, an appropriate weighting can be applied to each gaze evaluation element. Therefore, the gaze of a person can be accurately estimated.

[0166] <Modification Example 4> In the above Embodiment 1, the distance sensor 30 was used to acquire the distance to a person. In contrast, in Modification Example 4, the distance to a person is acquired from the captured image obtained from the camera 20. In Modification Example 4, the distance sensor 30 is omitted from the configuration of the above Embodiment 1.

[0167] FIG. 9 is a diagram schematically showing a method for acquiring the distance to a person according to Modification Example 4.

[0168] For example, the control unit 11 (distance acquisition unit 11b) extracts a person's face from the captured image P0 and detects the size W1 of the face. Then, the control unit 11 acquires the distance to the person based on the detected size W1 of the face. Here, the maximum width of the face is used as the size W1 of the face. The area of the person's face in the captured image P0 becomes smaller as the person moves away from the camera 20. On the other hand, the physical size of a person's face does not vary significantly from person to person. Therefore, the distance to the person can be detected from the size W1 of the person's face in the captured image P0.

[0169] In this case, the storage unit 12 stores in advance information defining the relationship between the face size W1 and the distance to the person. As this information, for example, a table defining the relationship between the face size W1 and the distance to the person is used. The control unit 11 acquires the distance to the person based on the face size W1 obtained from the captured image P0 and this information.

[0170] Here, the size W1 of the face on the captured image P0 was used to detect the distance to the person, but other parameters such as the shoulder width on the captured image P0 may be used to detect the distance to the person. The parameter used to detect the distance to the person is preferably a parameter that does not vary significantly from person to person and can be acquired smoothly from the captured image P0.

[0171] In addition, the distance to the person may be detected based on the height H1 from the lower end of the captured image P0 to the person's feet. As the person moves away from the camera 20, the area of the person on the captured image P0 becomes smaller and moves upward. Therefore, the distance to the person can be detected from the height H1 from the lower end of the captured image P0 to the position of the person's feet.

[0172] In this case, the storage unit 12 stores in advance information defining the relationship between the height H1 from the lower end of the captured image P0 to the position of the person's feet and the distance to the person. As this information, for example, a table defining the relationship between the height H1 and the distance to the person is used. The control unit 11 acquires the distance to the person based on the height H1 acquired from the captured image P0 and this information.

[0173] Note that instead of the height H1 to the position of the feet, the height from the lower end of the captured image P0 to the position of another part of the person may be used to detect the distance to the person. For example, the height H2 from the lower end of the captured image P0 to the position of the person's waist may be used to detect the distance to the person.

[0174] However, since the height from the feet to the waist is likely to vary depending on the person's height, if the height H2 is used to detect the distance to the person, the detection accuracy of the distance is likely to decrease slightly. Therefore, in order to detect the distance to the person more accurately, it is preferable to use the height H from the lower end of the captured image P0 to the position of the person's feet as described above.

[0175] In addition, a monocular camera distance measurement technique using a machine learning model may be used to detect the distance to the person. In this case, the storage unit 12 is equipped with an engine that executes this measurement technique. The control unit 11 uses this engine to detect the distance to the person on the captured image.

[0176] According to the configuration of Modification Example 4, since the distance to the person is acquired from the captured image from the camera 20, it is not necessary to separately use the distance sensor 30 as in Embodiment 1. Therefore, the configuration necessary for line-of-sight estimation can be simplified.

[0177] <Embodiment 2> <In the above Embodiment 1, the estimation process by a plurality of line-of-sight evaluation elements is changed according to the distance to a person. In contrast, in Embodiment 2, weighting according to the reliability when each line-of-sight evaluation element is extracted from a captured image is applied to each line-of-sight evaluation element, the estimation results of the line of sight based on each line-of-sight evaluation element are integrated, and the integrated estimation result is obtained as the line of sight of the person.>

[0178] <FIG. 10 is a diagram showing the configuration of the line-of-sight estimation system 1 according to Embodiment 2.>

[0179] <Compared with the configuration of FIG. 1, in Embodiment 2, the distance sensor 30 is omitted from the line-of-sight estimation system 1. Other configurations of the line-of-sight estimation system 1 according to Embodiment 2 are the same as those in Embodiment 1. However, in Embodiment 2, as described above, the line-of-sight estimation process in the line-of-sight estimation device 10 is different from that in Embodiment 1.>

[0180] <FIG. 11 is a block diagram showing the configuration of the line-of-sight estimation device 10 that constitutes the line-of-sight estimation system 1 according to Embodiment 2.>

[0181] <Compared with the configuration of FIG. 2, in Embodiment 2, the distance acquisition unit 11b is omitted from the functions executed by the control unit 11. Also, the distance sensor 30 is excluded from the targets with which the communication unit 15 communicates. Other configurations are the same as those in Embodiment 1. However, in Embodiment 2, as described above, the function of the line-of-sight estimation process in the line-of-sight estimation unit 11d is different from that in Embodiment 1.>

[0182] <FIG. 12(a) is a flowchart showing the line-of-sight estimation process according to Embodiment 2.>

[0183] In the flowchart of FIG. 12(a), the processes of steps S201 and S202 are executed by the control unit 11 by the functions of the image acquisition unit 11a and the extraction unit 11c shown in FIG. 11, respectively, and the processes of the other steps are executed by the control unit 11 by the function of the gaze estimation unit 11d in FIG. 11. Hereinafter, the description will be made on the assumption that the control unit 11 executes the process of FIG. 12 by these functions.

[0184] The control unit 11 acquires a captured image for one frame from the camera 20 (S201), and extracts a plurality of types of gaze evaluation elements (eye direction, face direction, body direction) for estimating the gaze of a person in the captured image from the acquired captured image (S202). The control unit 11 determines whether or not at least one of these gaze evaluation elements has been extracted (S203). For example, if none of the eyes, face, or body can be extracted from the captured image, the determination in step S203 is NO. In this case, the control unit 11 ends the current process without obtaining the gaze estimation result.

[0185] When any one of a plurality of types of gaze evaluation elements (eye direction, face direction, body direction) has been extracted (S203: YES), the control unit 11 acquires the reliability of each extracted gaze evaluation element (S204). Here, as described above, the reliability is the reliability of the recognition results of these parts output by the recognition engine during the recognition process of the parts of the person's eyes, face, and body from the captured image for one frame acquired in step S201. The reliabilities of the recognition results of the person's eyes, face, and body are the reliabilities of the eye direction, face direction, and body direction, respectively.

[0186] The control unit 11 obtains an estimation result of the gaze for the person from the reliabilities acquired for each gaze evaluation element (S205). This process is performed using the above formula (1).

[0187] That is, the control unit 11 applies a weighting wc i corresponding to the reliability acquired in step S204 to each gaze evaluation element. Similar to the above, the weighting wc iis set in the range of 0.0 to 1.0. For a line-of-sight evaluation element with a reliability of 0, the weighting wc i is set to 0.0. Alternatively, for a line-of-sight evaluation element that could not be extracted in step S203, the weighting wc i may be set to 0.0.

[0188] The control unit 11 applies the weighting wc i thus applied to each line-of-sight evaluation element and the direction X(x, y, z) of each line-of-sight evaluation element i to the above formula (1) to obtain an estimated result of the line of sight for the person. At this time, a line-of-sight evaluation element that could not be extracted in step S202 is excluded from the calculation process. The control unit 11 stores the obtained estimated result of the line of sight in the storage unit 12. Thereby, the process of FIG. 12(a) ends. Thereafter, the control unit 11 returns the process to step S201 and executes the next estimation process.

[0189] Note that step S203 may be omitted from the flowchart of FIG. 12(a). In this case, for example, if all line-of-sight evaluation elements cannot be extracted, the direction (x, y, z) of the line-of-sight evaluation element applied to the formula (1) in step S205 i does not exist. Therefore, similar to the case where the determination in step S203 in the flowchart of FIG. 12(a) is NO, an estimated result of the line of sight for the person cannot be obtained. Therefore, in terms of whether an estimated result of the line of sight can be obtained, it is the same as the case where the flowchart includes step S203. However, in the process of FIG. 12(a), when the determination in step S203 is NO, the steps of step S204 and S205 are skipped, and thus the processing load on the control unit 11 can be reduced accordingly.

[0190] <Effects of Embodiment 2> As shown in FIG. 11, the line-of-sight estimation device 10 includes an image acquisition unit 11a that acquires a captured image including at least a part of a person from a camera, an extraction unit 11c that extracts a plurality of line-of-sight evaluation elements different in type from each other from the captured image, and a line-of-sight estimation unit 11d that applies weighting according to the reliability when each line-of-sight evaluation element is extracted from the captured image to each line-of-sight evaluation element, integrates the line-of-sight estimation results based on each line-of-sight evaluation element, and acquires the integrated estimation result as the line of sight of the person.

[0191] According to this configuration, since a plurality of line-of-sight evaluation elements different in type from each other are used for estimating the line of sight of a person, the line of sight of a person can be estimated in a wide distance range. Further, since weighting according to the reliability when each line-of-sight evaluation element is extracted is applied to each line-of-sight evaluation element and the line-of-sight estimation results based on each line-of-sight evaluation element are integrated, the line of sight of a person can be accurately estimated.

[0192] <Modified Example> In the process of FIG. 12(a), only the reliability when each line-of-sight evaluation element is extracted from the captured image is taken into account. However, further, the probability of each of the line-of-sight evaluation elements in line-of-sight estimation may be taken into account and line-of-sight estimation may be performed.

[0193] FIG. 12(b) is a flowchart showing a line-of-sight estimation process according to a modified example of Embodiment 2.

[0194] In the flowchart of FIG. 12(b), step S205 of the flowchart of FIG. 12(a) is changed to step S211. The processing of the other steps in the flowchart of FIG. 12(b) is the same as the processing of the corresponding steps in the flowchart of FIG. 12(a).

[0195] In step S211, the control unit 11 acquires the line-of-sight estimation result of the person based on the reliability when each line-of-sight evaluation element is extracted from the captured image and the probability of each of the line-of-sight evaluation elements in line-of-sight estimation. This process is performed using the above formula (3).

[0196] The control unit 11 applies weights wc i corresponding to the reliability of each line-of-sight evaluation element, similar to step S205 in Fig. 12(a), to each line-of-sight evaluation element. Further, the control unit 11 applies weights wk i corresponding to the likelihood of each of the line-of-sight evaluation elements in line-of-sight estimation, similar to the estimation process 2 using the above formula (2), to each line-of-sight evaluation element. Then, the control unit 11 applies the weights wc i and wk i applied to each line-of-sight evaluation element and the direction X(x, y, z) i of each line-of-sight evaluation element to the above formula (3) to obtain the line-of-sight estimation result for the person. At this time, the line-of-sight evaluation elements that could not be extracted in step S202 are excluded from the calculation process.

[0197] The control unit 11 stores the obtained line-of-sight estimation result in the storage unit 12. Thereby, the process in Fig. 12(b) ends. After that, the control unit 11 returns the process to step S201 and executes the next round of estimation processing. Similar to the case of Fig. 12(a), step 203 may be omitted from the flowchart of Fig. 12(b).

[0198] According to this modification example, since weights wc i corresponding to the likelihood of each of the line-of-sight evaluation elements in line-of-sight estimation are further applied to each line-of-sight evaluation element to obtain the line-of-sight estimation result, the line of sight of a person can be accurately estimated.

[0199] Note that the weights wk i may be adjusted according to the scene, environment, etc. where line-of-sight estimation is performed. For example, when estimating the line of sight of a user facing the display screen of the terminal device for product purchase, etc., mainly when the user's face is included in the captured image, the weights wk of each line-of-sight evaluation element may be adjusted so that the weights of the direction of the eyeballs and the direction of the face increase as follows.

[0200] wk(direction of eyeballs, direction of face, direction of body) = (0.6, 0.4, 0.1) Also, in a situation where it is necessary to estimate the line of sight of a person who is relatively far from the camera 20, since it is difficult to extract the direction of the eyeball from the captured image, the weight wk of the direction of the eyeball may be set low as shown in the following equation.

[0201] wk(direction of eyeball, direction of face, direction of body) = (0.1, 0.4, 0.6) By adjusting the weight wk for each scene in this way, it is possible to improve the accuracy of line-of-sight estimation according to each scene.

[0202] <Other modification examples> In the above-described Embodiments 1 and 2, the direction of the eyeball, the direction of the face, and the direction of the body are exemplified as line-of-sight evaluation elements. However, the line-of-sight evaluation elements used for line-of-sight estimation are not limited to these. For example, since the probability of line-of-sight estimation is eyeball direction > face direction > body direction, even when these three line-of-sight estimation elements can be obtained from the captured image, the line-of-sight evaluation elements used for line-of-sight estimation may be limited to the direction of the eyeball and the direction of the face.

[0203] As a result, compared to the case where the above three types of line-of-sight evaluation elements are used for line-of-sight estimation, although the upper limit distance at which line-of-sight estimation is possible becomes shorter, the accuracy of line-of-sight estimation can be improved. Therefore, in a wide distance range where the direction of the eyeball and the direction of the face can be extracted, the line of sight of a person can be accurately estimated.

[0204] Also, line-of-sight evaluation elements other than the above three may be further used for line-of-sight estimation.

[0205] Also, in the above-described Embodiments 1 and 2, the camera 20 is installed on the upper surface of the advertising device 40. However, the camera 20 may be installed separately from the advertising device 40. Also, in the above-described embodiment, the display area 41 of the advertising device 40 is exemplified as the object of fixation. However, an advertising medium such as a poster may be the object of fixation.

[0206] In the above-described Embodiment 1, the gaze estimation device 10, the camera 20, and the distance sensor 30 are connected by a communication line. In Embodiment 2, the gaze estimation device 10 and the camera 20 are connected by a communication line. However, when the gaze estimation device 10 is installed at a location far from the camera 20 and the distance sensor 30, they may be connected via an external network such as a LAN or a public communication network.

[0207] Also, the gaze estimation device 10 does not necessarily have to be a single unit. The gaze estimation device 10 may be configured by a plurality of information processing devices that share the functions of the image acquisition unit 11a, the distance acquisition unit 11b, the extraction unit 11c, and the gaze estimation unit 11d.

[0208] In the above-described Embodiments 1 and 2, one camera 20 is used, but a plurality of cameras 20 may be arranged. When these plurality of cameras 20 can form a stereo camera, the distance to a person on each captured image may be measured based on the parallax of these cameras.

[0209] In addition, the embodiments of the present invention can be appropriately modified within the scope described in the claims.

Explanation of Reference Numerals

[0210] 1 Gaze Estimation System 10 Gaze Estimation Device 11 Control Unit 11a Image Acquisition Unit 11b Distance Acquisition Unit 11c Extraction Unit 11d Gaze Estimation Unit 20 Camera 30 Distance Sensor

Claims

1. An image acquisition unit that acquires a captured image including at least a part of a person from a camera, A distance acquisition unit that acquires the distance to the person, An extraction unit that extracts a plurality of line-of-sight evaluation elements of different types from the captured image, A line-of-sight estimation unit that changes the estimation process using the plurality of line-of-sight evaluation elements according to the distance acquired by the distance acquisition unit and estimates the line of sight of the person, comprising: A line-of-sight estimation device characterized by the above.

2. In the line-of-sight estimation device according to Claim 1, The plurality of line-of-sight evaluation elements include the orientation of the person's eyeballs and the orientation of the face, A line-of-sight estimation device characterized by the above.

3. In the line-of-sight estimation device according to Claim 2, The plurality of line-of-sight evaluation elements further include the orientation of the person's body, A line-of-sight estimation device characterized by the above.

4. In the line-of-sight estimation device according to Claim 1, The line-of-sight estimation unit changes the line-of-sight evaluation elements used for line-of-sight estimation for each distance range, A line-of-sight estimation device characterized by the above.

5. In the line-of-sight estimation device according to Claim 4, The line-of-sight estimation unit uses, for line-of-sight estimation for each of the distance ranges, one of the line-of-sight evaluation elements assumed to have the highest reliability, A line-of-sight estimation device characterized by the above.

6. In the line-of-sight estimation device according to Claim 4, The line-of-sight estimation unit uses, for line-of-sight estimation for each of the distance ranges, the line-of-sight evaluation elements preset as extractable from the captured image in each of the distance ranges among the plurality of line-of-sight evaluation elements, A line-of-sight estimation device characterized by the above.

7. In the line-of-sight estimation device according to Claim 6, When performing line-of-sight estimation in the distance range where a plurality of the line-of-sight evaluation elements are set, the line-of-sight estimation unit integrates the line-of-sight estimation results based on each of the line-of-sight evaluation elements to obtain the line-of-sight estimation result of the person, A line-of-sight estimation device characterized by the above.

8. In the line-of-sight estimation device according to Claim 7, The line-of-sight estimation unit applies weighting according to the reliability when each of the line-of-sight evaluation elements is extracted from the captured image to each of the line-of-sight evaluation elements and integrates the line-of-sight estimation results based on each of the line-of-sight evaluation elements, A line-of-sight estimation device characterized by the above.

9. In the line-of-sight estimation device according to Claim 7, The line-of-sight estimation unit applies a weighting according to the probability of each of the line-of-sight evaluation elements in line-of-sight estimation to each of the line-of-sight evaluation elements, and integrates the line-of-sight estimation results based on each of the line-of-sight evaluation elements. A line-of-sight estimation device characterized by the above.

10. In the line-of-sight estimation device according to Claim 1, the line-of-sight estimation unit applies a weighting according to the distance acquired by the distance acquisition unit to each of the line-of-sight evaluation elements, integrates the line-of-sight estimation results based on each of the line-of-sight evaluation elements, and acquires the integrated estimation result as the line of sight of the person. A line-of-sight estimation device characterized by the above.

11. In the line-of-sight estimation device according to Claim 10, the distance ranges in which the weightings of each of the line-of-sight evaluation elements increase are arranged on the distance axis, and a range in which one of the weightings of two adjacent line-of-sight evaluation elements decreases while the other increases exists on the distance axis. A line-of-sight estimation device characterized by the above.

12. In the line-of-sight estimation device according to Claim 1, the distance acquisition unit acquires the distance to the person from a distance sensor. A line-of-sight estimation device characterized by the above.

13. In the line-of-sight estimation device according to Claim 1, the distance acquisition unit acquires the distance to the person from the captured image. A line-of-sight estimation device characterized by the above.

14. An image acquisition unit that acquires a captured image including at least a part of a person from a camera, an extraction unit that extracts a plurality of line-of-sight evaluation elements of different types from the captured image, and a line-of-sight estimation unit that applies a weighting according to the reliability when each of the line-of-sight evaluation elements is extracted from the captured image to each of the line-of-sight evaluation elements, integrates the line-of-sight estimation results based on each of the line-of-sight evaluation elements, and acquires the integrated estimation result as the line of sight of the person. A line-of-sight estimation device characterized by the above.

15. In the line-of-sight estimation device according to Claim 14, the line-of-sight estimation unit further applies a weighting according to the probability of each of the line-of-sight evaluation elements in line-of-sight estimation to each of the line-of-sight evaluation elements, and integrates the line-of-sight estimation results based on each of the line-of-sight evaluation elements. A line-of-sight estimation device characterized by the above.

16. A line-of-sight estimation device according to any one of Claims 1 to 11, 13, and 14, and the camera. A line-of-sight estimation system characterized by the above.

17. The line-of-sight estimation device according to Claim 12, the camera, and the distance sensor. A line-of-sight estimation system characterized by the above.

18. A method for estimating a line of sight in an information processing apparatus, comprising: acquiring a captured image including at least a part of a person; acquiring a distance to the person; extracting a plurality of line-of-sight evaluation elements of different types from the captured image; changing a line-of-sight estimation process using the plurality of line-of-sight evaluation elements according to the distance acquired by the distance acquisition unit, and estimating the line of sight of the person; A line-of-sight estimation method characterized by the above.

19. A method for estimating a line of sight in an information processing apparatus, comprising: acquiring a captured image including at least a part of a person; extracting a plurality of line-of-sight evaluation elements of different types from the captured image; applying a weighting according to the reliability when each of the line-of-sight evaluation elements is extracted from the captured image to each of the line-of-sight evaluation elements, integrating the line-of-sight estimation results based on each of the line-of-sight evaluation elements, and acquiring the integrated estimation result as the line of sight of the person; A line-of-sight estimation method characterized by the above.

Images