Line-of-sight estimation device, line-of-sight estimation system, and line-of-sight estimation method

Abstract

A line-of-sight estimation device (10) comprises: an image acquisition unit (11a) that acquires, from a camera (20), a captured image including at least a portion of a person; a distance acquisition unit (11b) that acquires the distance to the person; an extraction unit (11c) that extracts a plurality of different types of line-of-sight evaluation elements from the captured image; and a line-of-sight estimation unit (11d) that changes estimation processes based on the plurality of line-of-sight evaluation elements, depending on the distance acquired by the distance acquisition unit, and estimates the line of sight of the person.

Description

Gaze estimation device, gaze estimation system, and gaze estimation method

[0001] The present invention relates to a gaze estimation device, a gaze estimation system, and a gaze estimation method for estimating a person's gaze.

[0002] In recent years, there has been an increasing demand for verifying the commercial effectiveness of advertisements on digital signage and other devices. In this case, it is necessary to determine whether people near the advertisement have seen the advertisement. To do this, it is necessary to estimate the gaze direction of people near the advertisement.

[0003] The following Patent Document 1 describes an eye gaze detection device that detects a person's gaze by detecting the pupil center from an image captured by a camera. In this eye gaze detection device, light from a light source is irradiated onto the person's eye. The corneal reflection image of the light source and the pupil center are detected from the eye area included in the image captured by the camera. The person's gaze direction is detected based on the positional relationship between the pupil center and the corneal reflection image. The person's gaze position on the display unit is determined based on the gaze direction and the distance from the display unit to the person's face.

[0004] Patent No. 5655644

[0005] The gaze detection device described in Patent Document 1 cannot estimate the gaze of a person if the person is located farther away than the distance range in which the person's eyes can be detected from the image captured by the camera. On the other hand, when detecting a person's gaze to determine whether or not the person is gazing at digital signage as described above, the distance to the person may extend beyond the range in which the eyes cannot be detected.

[0006] In view of the above problem, an object of the present invention is to provide a gaze estimation device, a gaze system, and a gaze estimation method that are capable of properly estimating a person's gaze over a wide distance range.

[0007] A gaze estimation device according to a first aspect of the present invention comprises an image acquisition unit that acquires an image including at least a portion of a person from a camera, a distance acquisition unit that acquires the distance to the person, an extraction unit that extracts a plurality of gaze evaluation elements of different types from the captured image, and a gaze estimation unit that estimates the gaze of the person by changing the estimation process using the plurality of gaze evaluation elements according to the distance acquired by the distance acquisition unit.

[0008] According to the gaze estimation device of this aspect, the estimation process using multiple gaze evaluation elements is changed depending on the distance to the person, so that the gaze of a person can be appropriately estimated over a wide range of distances.

[0009] In the gaze estimation device according to this aspect, the plurality of gaze evaluation elements may include the direction of the person's eyeballs and the direction of their face.

[0010] According to this configuration, the gaze direction of a person whose gaze estimation is highly reliable is included in the gaze evaluation elements, so that the gaze of the person can be estimated with high accuracy.

[0011] In the gaze estimation device according to this aspect, the plurality of gaze evaluation elements may further include a body orientation of the person.

[0012] According to this configuration, the gaze evaluation element further includes the direction of a person's body, so that, for example, even in a relatively long distance range where the direction of a person's eyeballs or face cannot be extracted from a captured image, the gaze of the person can be estimated from the direction of the person's body.

[0013] In the gaze estimation device according to this aspect, the gaze estimation unit may be configured to change the gaze evaluation element used for gaze estimation for each distance range.

[0014] According to this configuration, a person's gaze can be appropriately estimated by using gaze evaluation elements that can be appropriately acquired in each distance range or gaze evaluation elements that are optimal for each distance range for gaze estimation for each distance range.

[0015] In this case, the gaze estimation unit may be configured to use, for each of the distance ranges, one of the gaze evaluation elements that is assumed to have the highest reliability for gaze estimation.

[0016] According to this configuration, the gaze evaluation element that is optimal for each distance range is used for gaze estimation in each distance range, so that a person's gaze can be estimated appropriately through simple processing.

[0017] Alternatively, the gaze estimation unit may be configured to use, among the plurality of gaze evaluation elements, a gaze evaluation element that has been pre-set as being extractable from the captured image in each of the distance ranges for gaze estimation in each of the distance ranges.

[0018] According to this configuration, gaze estimation elements that can be extracted in each distance range are used to estimate the gaze, and therefore, for a given distance range, gaze estimation is performed using multiple gaze evaluation elements. Therefore, in this distance range, the gaze of a person can be comprehensively estimated using multiple gaze evaluation elements, and even if one of these gaze evaluation elements cannot be extracted from the captured image, the estimation result of the gaze of a person can be obtained from the other gaze evaluation elements.

[0019] In this configuration, when performing gaze estimation within the distance range in which multiple gaze evaluation elements are set, the gaze estimation unit can be configured to integrate the gaze estimation results based on each of the gaze evaluation elements to obtain an estimation result of the person's gaze.

[0020] According to this configuration, for a distance range in which multiple gaze evaluation elements are set, a person's gaze can be comprehensively estimated from the gaze estimation results based on these gaze evaluation elements, and even if any of the gaze evaluation elements cannot be extracted from the captured image, the estimation results of the person's gaze can be obtained from the other gaze evaluation elements.

[0021] In this case, the gaze estimation unit may be configured to apply a weighting to each of the gaze evaluation elements according to the reliability of each element when extracted from the captured image, and to integrate the gaze estimation results based on each element.

[0022] According to this configuration, the reliability of each gaze evaluation element when extracted is taken into consideration when acquiring the estimation result of the gaze of a person, so that the gaze of a person can be estimated with high accuracy.

[0023] Alternatively, the gaze estimation unit may be configured to apply a weighting to each of the gaze evaluation elements according to the likelihood of each of the gaze evaluation elements in the gaze estimation, and to integrate the gaze estimation results based on each of the gaze evaluation elements.

[0024] For example, when the gaze evaluation elements are eye direction, face direction, and body direction, the likelihood of gaze estimation is usually eye direction > face direction > body direction. According to the above configuration, a weighting according to such likelihood is applied to each gaze evaluation element to obtain a gaze estimation result, so that a person's gaze can be estimated with high accuracy.

[0025] In the gaze estimation device of this aspect, the gaze estimation unit can be configured to apply weighting according to the distance acquired by the distance acquisition unit to each of the gaze evaluation elements, integrate the gaze estimation results based on each of the gaze evaluation elements, and acquire the integrated estimation result as the gaze of the person.

[0026] According to this configuration, as will be shown in Modification Example 3 later, weighting according to distance is applied to each gaze evaluation element, and the gaze estimation results based on each gaze evaluation element are integrated, thereby enabling a person's gaze to be estimated with high accuracy.

[0027] In this case, for example, as shown in Figure 8, the distance ranges where the weighting of each of the gaze evaluation elements is high are lined up on the distance axis, and the weighting can be set so that there is a range on the distance axis where one of the weightings of two adjacent gaze evaluation elements decreases while the other increases.

[0028] According to this configuration, it is possible to appropriately weight each of the gaze evaluation elements, thereby enabling accurate estimation of a person's gaze.

[0029] In the gaze estimation device according to this aspect, the distance acquisition unit may be configured to acquire the distance to the person from a distance sensor.

[0030] According to this configuration, the distance to a person can be detected with high accuracy, and the line of sight of the person can be estimated with high accuracy based on the distance to the person.

[0031] Alternatively, the distance acquisition unit may be configured to acquire the distance to the person from the captured image.

[0032] According to this configuration, the distance to the person is acquired from the captured image by the camera, so there is no need to use a separate distance sensor, and the configuration required for gaze estimation can be simplified.

[0033] A gaze estimation device according to a second aspect of the present invention comprises an image acquisition unit that acquires an image from a camera that includes at least a portion of a person, an extraction unit that extracts a plurality of gaze evaluation elements of different types from the captured image, and a gaze estimation unit that applies weighting to each of the gaze evaluation elements according to the reliability of each element when extracted from the captured image, integrates the gaze estimation results based on each of the gaze evaluation elements, and acquires the integrated estimation result as the gaze of the person.

[0034] According to the gaze estimation device of this aspect, a plurality of different gaze evaluation elements are used to estimate a person's gaze, so that the person's gaze can be estimated over a wide distance range. Furthermore, weighting is applied to each gaze evaluation element according to the reliability at the time of extraction of each gaze evaluation element, and gaze estimation results based on each gaze evaluation element are integrated, so that the person's gaze can be estimated with high accuracy.

[0035] In the gaze estimation device according to the second aspect, the gaze estimation unit may be configured to further apply weighting to each of the gaze evaluation elements according to the likelihood of each of the gaze evaluation elements in the gaze estimation, and to integrate the gaze estimation results based on each of the gaze evaluation elements.

[0036] For example, when the gaze evaluation elements are eye direction, face direction, and body direction, the likelihood of gaze estimation is usually eye direction > face direction > body direction. According to the above configuration, a weighting according to such likelihood is applied to each gaze evaluation element to obtain a gaze estimation result, so that a person's gaze can be estimated with high accuracy.

[0037] A gaze estimation system according to a third aspect of the present invention includes the gaze estimation device according to the first or second aspect and a camera.

[0038] According to the gaze estimation system of this aspect, the same effects as those of the first and second aspects can be achieved.

[0039] A gaze estimation system according to a fourth aspect of the present invention includes the gaze estimation device according to the first aspect, a camera, and a distance sensor.

[0040] The gaze estimation system according to this aspect can achieve the same effects as the first aspect.

[0041] A fifth aspect of the present invention is a gaze estimation method in an information processing device, which acquires an image including at least a portion of a person, acquires the distance to the person, extracts a plurality of gaze evaluation elements of different types from the image, and modifies a gaze estimation process using the plurality of gaze evaluation elements according to the distance acquired by the distance acquisition unit to estimate the gaze of the person.

[0042] The gaze estimation system according to this aspect can achieve the same effects as the first aspect.

[0043] A sixth aspect of the present invention is a gaze estimation method in an information processing device, which acquires an image including at least a portion of a person, extracts a plurality of gaze evaluation elements of different types from the image, applies weighting to each of the gaze evaluation elements according to the reliability of each element when extracted from the image, integrates the gaze estimation results based on each of the gaze evaluation elements, and acquires the integrated estimation result as the gaze of the person.

[0044] The gaze estimation system according to this aspect can achieve the same effects as those of the second aspect.

[0045] As described above, according to the present invention, it is possible to provide a gaze estimation device, a gaze system, and a gaze estimation method that are capable of properly estimating a person's gaze over a wide distance range.

[0046] The effects and significance of the present invention will become more apparent from the following description of the embodiments, however, the embodiments shown below are merely examples of how the present invention can be implemented, and the present invention is not limited to the embodiments described below.

[0047] FIG. 1 is a diagram illustrating a configuration of a gaze estimation system according to a first embodiment. FIG. 2 is a block diagram illustrating a configuration of a gaze estimation device constituting the gaze estimation system according to the first embodiment. FIG. 3 is a diagram illustrating processing modes used for gaze estimation according to the first embodiment. FIG. 4 is a flowchart illustrating gaze estimation processing according to the first embodiment. FIG. 5 is a flowchart illustrating gaze estimation processing according to a first modified example. FIG. 6 is a diagram illustrating processing modes used for gaze estimation according to a second modified example. FIG. 7 is a flowchart illustrating gaze estimation processing according to a third modified example. FIG. 8 is a graph schematically illustrating an example of a method for setting weights applied to each gaze evaluation element depending on distance according to a third modified example. FIG. 9 is a diagram schematically illustrating a method for acquiring the distance to a person according to a fourth modified example. FIG. 10 is a diagram illustrating a configuration of a gaze estimation system according to a second embodiment. FIG. 11 is a block diagram illustrating a configuration of a gaze estimation device constituting the gaze estimation system according to the second embodiment. FIG. 12( a) is a flowchart illustrating gaze estimation processing according to the second embodiment. FIG. 12( b) is a flowchart illustrating gaze estimation processing according to a second modified example.

[0048] However, the drawings are for illustrative purposes only and do not limit the scope of the present invention.

[0049] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0050] First Embodiment FIG. 1 is a diagram showing the configuration of a gaze estimation system 1. As shown in FIG.

[0051] The gaze estimation system 1 includes a gaze estimation device 10, a camera 20, and a distance sensor 30. The camera 20 and the distance sensor 30 are connected to the gaze estimation device 10 via a communication line. The camera 20 and the distance sensor 30 may communicate with the gaze estimation device 10 wirelessly.

[0052] The gaze estimation device 10 is configured, for example, by a personal computer. In this case, the gaze estimation device 10 is configured by installing an application program for gaze estimation on the personal computer. However, the present invention is not limited to this, and the gaze estimation device 10 may be a dedicated device.

[0053] The gaze estimation device 10 estimates the gaze of a person near the advertising device 40 based on the captured image of the camera 20 and the detection result of the distance sensor 30. The gaze estimation device 10 determines whether a person near the advertising device 40 has looked at the advertising device 40 based on the estimated gaze, and stores data based on the determination result as needed. This determination can be made, for example, by determining whether an extension of the gaze is included in the display area 41 of the advertising device 40 based on the gaze direction of the person, the distance to the person, and the display area 41 of the advertising device 40 in a predetermined three-dimensional coordinate area. However, the determination method is not limited to this, and for example, this determination may be made using a machine learning model that uses information on the gaze direction and whether the display area 41 has been looked at as training data.

[0054] The advertising device 40 is, for example, a digital signage. The advertising device 40 displays a predetermined advertising screen in a display area 41 under the control of a control device (not shown). A camera 20 is installed on the top surface of the advertising device 40. The camera 20 faces the same direction as the advertising device 40. Therefore, the camera 20 captures an image of a person in the vicinity of the front of the advertising device 40 from the advertising device 40 side.

[0055] The captured image captured by the camera 20 is transmitted as needed to the gaze estimation device 10. The gaze estimation device 10 extracts the eyes, face, and body of a person from one frame of the captured image received. The gaze estimation device 10 is equipped with a recognition engine for recognizing these features of a person from the captured image.

[0056] During the recognition process for these features, the recognition engine outputs the reliability of the recognition results for these features, i.e., the consistency rate of the data for each feature acquired from the captured image with the reference data for these features. The gaze estimation device 10 extracts, as the region of each feature, the region on the captured image where this reliability is equal to or greater than a predetermined threshold. In this way, the gaze estimation device 10 extracts the person's eyes, face, and body from the captured image.

[0057] Furthermore, the gaze estimation device 10 acquires the eyeball direction, facial direction, and body direction from the extracted person's eyes, face, and body. The gaze estimation device 10 estimates the gaze of the person from the acquired eyeball direction, facial direction, and body direction. That is, the gaze estimation device 10 estimates the eyeball direction, facial direction, and body direction on predetermined three-dimensional coordinate axes set in the imaging field of view of the camera 20 as the gaze direction of the person, and estimates straight lines pointing in each direction from near the person's eyes as the gaze.

[0058] The eye direction is acquired, for example, as the direction in which the pupil is facing. The face direction is acquired, for example, as the direction in which the nose is facing. The body direction is acquired, for example, as the direction in which the skeletal center (the center between the left and right shoulders) is facing. The gaze estimation device 10, for example, acquires key point data from the captured image and acquires the face direction and body direction from the acquired key point data.

[0059] Here, the key point data is coordinate data of the following positions on the above three-dimensional coordinate axes.

[0060] <Key points> Skeleton center (center between left and right shoulders), left ankle, right ankle, left ear, right ear, left elbow, right elbow, left eye, right eye, left thigh, right thigh, left knee, right knee, neck, nose, left shoulder, right shoulder, left wrist, right wrist, left corner of mouth, right corner of mouth A distance sensor 30 is also installed on the top surface of the advertising device 40. The distance sensor 30 is arranged adjacent to the camera 20 and faces the same direction as the camera 20. The distance sensor 30 acquires the distance to a person standing near in front of the advertising device 40.

[0061] The distance sensor 30 is, for example, a time-of-flight (TOF) camera. The distance sensor 30 acquires the distance to an object present in a field of view that covers the imaging field of view of the camera 20. The distance sensor 30 may be a distance sensor other than a TOF camera.

[0062] The distance sensor 30 includes, for example, a light source, an image sensor, and an optical system that focuses light emitted from the light source and reflected by an object onto the image sensor. The distance is calculated for each pixel based on the time difference between the light emission timing of the light source and the light reception timing of each pixel on the image sensor. The position of each pixel corresponds to the position of each square obtained by dividing the field of view of the distance sensor 30 into a matrix. Therefore, the distance calculated for each pixel corresponds to the distance to the object in the direction of the corresponding square.

[0063] The distance sensor 30 generates a distance image for one frame in which distance information is embedded at the position of each pixel, and transmits the generated distance image to the line-of-sight estimation device 10 as needed.

[0064] The gaze estimation device 10 acquires the distance to a person on the captured image based on the distance image acquired from the distance sensor 30 and the captured image acquired from the camera 20. Specifically, the gaze estimation device 10 acquires, as the distance to the person, the distance to a pixel position on the distance image that corresponds to the body of the person extracted from the captured image.

[0065] In this case, the gaze estimation device 10 acquires, for example, the distance to the pixel position on the distance image corresponding to a position near the center of the person's body area on the captured image as the distance to the person. If the body cannot be extracted from the captured image, the gaze estimation device 10 acquires, as the distance to the person, the distance to the pixel position on the distance image corresponding to a position near the center of the person's face area on the captured image. Furthermore, if neither the body nor the entire face can be extracted from the captured image, the gaze estimation device 10 acquires, as the distance to the person, the distance to the pixel position on the distance image corresponding to a position near the center of a part of the face (for example, the person's eye area) on the captured image.

[0066] The gaze estimation device 10 estimates the gaze of a person based on the distance to the person and the direction of the person's eyes, face, and body. The gaze estimation method will be described later with reference to FIGS. 3 and 4.

[0067] FIG. 2 is a block diagram showing the configuration of the gaze estimation device 10 that constitutes the gaze estimation system 1.

[0068] The gaze estimation device 10 includes a control unit 11, a storage unit 12, a display unit 13, an input unit 14, and a communication unit 15. The control unit 11 includes a processing circuit such as a CPU, and controls each unit according to a program stored in the storage unit 12. The storage unit 12 includes memory such as a ROM, RAM, and hard disk, and stores the program. The display unit 13 includes a display device such as a liquid crystal panel, and displays a predetermined screen under the control of the control unit 11. The input unit 14 includes input means such as a keyboard and a mouse. The communication unit 15 communicates with the camera 20 and the distance sensor 30 under the control of the control unit 11.

[0069] In this embodiment, the control unit 11 executes the functions of the image acquisition unit 11 a, the distance acquisition unit 11 b, the extraction unit 11 c, and the line of sight estimation unit 11 d by using a program stored in the storage unit 12.

[0070] The image acquisition unit 11a acquires a captured image from the camera 20. The distance acquisition unit 11b acquires a distance image from the distance sensor 30 and acquires the distance to the person in the captured image. The method for acquiring the distance to the person is as described above. The extraction unit 11c extracts a plurality of different types of gaze evaluation elements from the captured image. In this embodiment, the above-mentioned eye direction, face direction, and body direction are extracted as the gaze evaluation elements. The method for extracting the eye direction, face direction, and body direction is as described above. The gaze estimation unit 11d estimates the gaze of the person in the captured image by changing the estimation process using the plurality of gaze evaluation elements (eye direction, face direction, and body direction) depending on the distance acquired by the distance acquisition unit 11b.

[0071] FIG. 3 is a diagram showing processing modes used for gaze estimation.

[0072] In the first embodiment, three processing modes, a first mode, a second mode, and a third mode, are prepared according to the distance. The first mode is a mode in which the gaze is estimated based only on the direction of the eyeballs. The second mode is a mode in which the gaze is estimated based only on the direction of the face. The third mode is a mode in which the gaze is estimated based only on the direction of the body.

[0073] When the distance to the person acquired by distance acquisition unit 11b is in the range of greater than 0 and equal to or less than D1, gaze estimation is performed in the first mode. When the distance to the person is in the range of greater than D1 and equal to or less than D2, gaze estimation is performed in the second mode. When the distance to the person is in the range of greater than D2 and equal to or less than D3, gaze estimation is performed in the third mode.

[0074] The distance 0 to D1 is a distance range in which the direction of the eyeball can be obtained more appropriately than the other gaze evaluation elements. In other words, the distance 0 to D1 is a distance range in which it is assumed that a person's eyes can be extracted more appropriately from a captured image than the face and body. The distance D1 to D2 is a distance range in which it is assumed that a person's face can be extracted more appropriately than the body, although the distance D1 to D2 cannot extract a person's eyes appropriately from a captured image. The distance D2 to D3 is a distance range in which it is assumed that a person's body direction can be obtained more appropriately than the other gaze evaluation elements. In other words, the distance D2 to D3 is a distance range in which it is assumed that a person's eyes and face cannot be extracted appropriately from a captured image, but the person's body can be extracted appropriately.

[0075] The eye area in the captured image gradually becomes smaller as the distance to the person increases. Therefore, the eye area becomes more difficult to extract as the distance increases, and the eye direction also becomes more difficult to extract as the distance increases. From this perspective, the upper limit distance D1 for the first mode is set.

[0076] Furthermore, if the distance to the person is too small, the entire face area in the captured image is difficult to include in the captured image. Therefore, it is difficult to extract the face direction from the captured image when the distance is small. On the other hand, after the entire face is included in the captured image, the face area in the captured image gradually becomes smaller as the distance to the person increases. Therefore, it becomes more difficult to extract the face area as the distance increases, and it also becomes more difficult to extract the face direction as the distance increases. From this perspective, the upper limit distance D2 for the second mode is set.

[0077] Similarly, if the distance to the person is too small, the entire body area in the captured image is difficult to include in the captured image. After the entire body is included in the captured image, the body area in the captured image gradually becomes smaller as the distance to the person increases. Therefore, after the entire body is included in the captured image, the body area becomes more difficult to extract as the distance increases, and the body orientation also becomes more difficult to extract as the distance increases. From this perspective, the upper limit distance D3 for the third mode is set.

[0078] In this way, the upper limit distances D1, D2, and D3 are set as the upper limit of the distances at which the eyeball direction, face direction, and body direction can be properly extracted from the captured image. However, the distances D1 to D3 do not necessarily have to be set from this perspective. For example, the distance D1 may be set to a distance at which the eyeball direction and face direction can be detected with similar accuracy, and the distance D2 may be set to a distance at which the face direction and body direction can be detected with similar accuracy.

[0079] FIG. 4 is a flowchart showing the gaze estimation process.

[0080] In the flowchart of Fig. 4, steps S101, S102, and S103 are executed by the control unit 11 using the functions of the image acquisition unit 11a, extraction unit 11c, and distance acquisition unit 11b in Fig. 2, respectively, and steps S104 to S106 are executed by the control unit 11 using the function of the line of sight estimation unit 11d in Fig. 2. In the following, the description will be made assuming that the control unit 11 executes the processing in Fig. 4 using these functions.

[0081] The control unit 11 acquires one frame of a captured image from the camera 20 (S101), and extracts gaze evaluation elements (eye direction, face direction, and body direction) from the acquired captured image to estimate the gaze of the person in the captured image (S102). The control unit 11 acquires a distance image from the distance sensor 30 and acquires the distance to the person in the captured image (S103). The control unit 11 sets the processing mode corresponding to the distance to the person among the three processing modes (first to third modes) shown in Fig. 3 as the processing mode to be used for gaze estimation (S104).

[0082] The control unit 11 determines whether or not the gaze evaluation element corresponding to the processing mode set in step S104 has been extracted in step S102 (S105). For example, if the processing mode set in step S104 is the first mode, the control unit 11 determines whether or not the gaze evaluation element of eye direction has been extracted in step S102. For example, if a person's face is not facing the direction of the camera 20 and the person's eyes are not included in the captured image, the gaze evaluation element of eye direction cannot be extracted in step S102. In such a case, the determination in step S105 is NO.

[0083] If the determination in step S105 is NO, the control unit 11 ends the current process without obtaining an estimation result of the person's gaze. On the other hand, if the determination in step S105 is YES, the control unit 11 uses the gaze evaluation element corresponding to the processing mode set in step S104 to obtain an estimation result of the person's gaze by the above-mentioned method, and stores the obtained estimation result in the storage unit 12 (S106). This causes the control unit 11 to end the process of FIG. 4.

[0084] If the captured image includes multiple people, the control unit 11 executes the process of steps S102 to S106 for each person. This also applies to the modified example and embodiment 2 described below. After the process of FIG. 4 is completed, the control unit 11 returns the process to step S101 and executes the next round of estimation processing. In this way, the control unit 11 repeatedly executes the process of FIG. 4.

[0085] <Effects of First Embodiment> According to this embodiment, the following effects can be achieved.

[0086] As shown in Figure 2, the gaze estimation device 10 includes an image acquisition unit 11a that acquires an image including at least a portion of a person from a camera 20, a distance acquisition unit 11b that acquires the distance to the person, an extraction unit 11c that extracts multiple gaze evaluation elements of different types (eye direction, face direction, body direction) from the captured image, and a gaze estimation unit 11d that estimates the person's gaze by changing the estimation process (first mode, second mode, third mode) using the multiple gaze evaluation elements depending on the distance acquired by the distance acquisition unit 11b.

[0087] With this configuration, the estimation process using multiple gaze evaluation elements (eye direction, face direction, body direction) is changed depending on the distance to the person, making it possible to properly estimate a person's gaze over a wide range of distances.

[0088] As shown in FIG. 3, the gaze evaluation elements include the direction of the person's eyes and the direction of their face.

[0089] According to this configuration, the gaze direction of a person whose gaze estimation is highly reliable is included in the gaze evaluation elements, so that the gaze of the person can be estimated with high accuracy.

[0090] As shown in FIG. 3, the multiple gaze evaluation elements further include the orientation of the person's body.

[0091] According to this configuration, the gaze evaluation element further includes the direction of a person's body, so that, for example, even in a relatively long distance range where the direction of a person's eyeballs or face cannot be extracted from a captured image, the gaze of the person can be estimated from the direction of the person's body.

[0092] As shown in FIG. 3, the gaze estimation unit 11d changes the gaze evaluation elements (eye direction, face direction, and body direction) used for gaze estimation for each distance range.

[0093] According to this configuration, a person's gaze can be appropriately estimated by using gaze evaluation elements that can be appropriately acquired in each distance range or gaze evaluation elements that are optimal for each distance range for gaze estimation for each distance range.

[0094] As shown in FIG. 3, the gaze estimation unit 11d uses one gaze evaluation element that is assumed to have the highest reliability for each distance range for gaze estimation.

[0095] According to this configuration, the gaze evaluation element that is optimal for each distance range is used for gaze estimation in each distance range, so that a person's gaze can be estimated appropriately through simple processing.

[0096] As shown in FIG. 2, the distance acquisition unit 11b acquires the distance to the person from the distance sensor 30.

[0097] According to this configuration, the distance to a person can be detected with high accuracy, and the line of sight of the person can be estimated with high accuracy based on the distance to the person.

[0098] As shown in FIGS. 1 and 2, the gaze estimation system 1 includes a gaze estimation device 10, a camera 20, and a distance sensor 30.

[0099] According to this configuration, as described above, in the gaze estimation device 10, the estimation process (first mode, second mode, third mode) using multiple gaze evaluation elements (eye direction, face direction, body direction) is changed depending on the distance to the person, so that the gaze of a person can be appropriately estimated over a wide range of distances.

[0100] <Modification 1> FIG. 5 is a flowchart showing the line-of-sight estimation process according to Modification 1. In FIG.

[0101] Compared to the flowchart of Fig. 4, the flowchart of Fig. 5 has additional steps S111 and S112. The processing of steps S111 and S112 is performed by the control unit 11 using the function of the line-of-sight estimation unit 11d.

[0102] If the control unit 11 cannot extract in step S102 a gaze evaluation element (eye direction, face direction, or body direction) corresponding to the processing mode (first mode, second mode, or third mode) set in step S104 (S105: NO), the control unit 11 determines whether the processing mode used for gaze estimation can be changed to another processing mode (S111). If the determination in step S111 is YES, the control unit 11 changes the processing mode used for gaze estimation to another processing mode (S112).

[0103] For example, when the processing mode set in step S104 is the first mode, if the control unit 11 cannot extract the eye direction, which is the gaze evaluation element corresponding to the first mode, in step S102 (S105: NO), the control unit 11 determines whether the face direction or body direction, which is another gaze evaluation element, was extracted in step S102 (S111). If the face direction or body direction was extracted (S111: YES), the control unit 11 changes the processing mode used for gaze estimation to the second mode or the third mode corresponding to the face direction or body direction (S112).

[0104] Here, if there are multiple other changeable processing modes, the control unit 11 changes the processing mode used for gaze estimation to the processing mode associated with the closer distance range. For example, if the other changeable processing modes are the second mode and the third mode, which correspond to the face direction and the body direction, respectively, the control unit 11 changes the processing mode used for gaze estimation to the second processing mode (face direction), which corresponds to the closer distance range. This is because the accuracy (likelihood) of gaze estimation is usually higher when the gaze evaluation element of the processing mode associated with the closer distance range is used. In other words, the priority (likelihood) of each gaze evaluation element in gaze estimation is eye direction > face direction > body direction.

[0105] The control unit 11 acquires the gaze estimation result for the person using the processing mode changed in this way (S106). On the other hand, if the determination in step S111 is NO, the control unit 11 ends the processing in FIG. 5 without acquiring the gaze estimation result for the person.

[0106] According to the process of the first modification, even if the determination in step S105 is NO, if the determination in step S111 is YES, it is possible to obtain an estimation result of the gaze of the person. Therefore, more gaze estimation results can be obtained than in the process of FIG.

[0107] 5 , the processing mode set in step S112 is not a processing mode corresponding to the distance to the person, and therefore the accuracy of the gaze estimation result may be lower than when the processing mode set in step S104 is used for gaze estimation. Therefore, in the processing of FIG. 5 , a flag indicating whether the processing mode set in step S104 or step S112 was used for gaze estimation may be linked to the gaze estimation result and stored in the storage unit 12. Furthermore, the type of gaze evaluation element set in step S112 may be linked to the gaze estimation result. This information may be used in subsequent statistical processing of the gaze estimation result, thereby enabling more accurate statistical processing results to be obtained.

[0108] <Modification 2> FIG. 6 is a diagram showing processing modes used for gaze estimation according to Modification 2. In FIG.

[0109] In the first embodiment, three processing modes, a first mode, a second mode, and a third mode, are prepared according to the distance, and one gaze evaluation element (eye direction / face direction / body direction) is assigned to each processing mode, as shown in Fig. 3. In contrast, in the second modification, a plurality of gaze evaluation elements are assigned to the first mode and the second mode, as shown in Fig. 6.

[0110] That is, in the relatively short distance range of 0 to D1, it is optimal to use the direction of the eyeballs for gaze estimation, but if part or all of the face or part of the body is reflected in the captured image, it may be possible to extract the direction of the face or the direction of the body from these images. For this reason, in the first mode corresponding to the distances 0 to D1, the direction of the eyeballs, as well as the direction of the face and the direction of the body, are assigned as gaze evaluation elements for gaze estimation.

[0111] In addition, in the medium distance range of D1 to D2, it is optimal to use the face direction for gaze estimation, but if part or all of the body is reflected in the captured image, it may be possible to extract the body direction from this image. On the other hand, when the distance exceeds D1, as described above, the eye direction cannot be properly extracted from the captured image. For this reason, in the second mode corresponding to the distances D1 to D2, the face direction and the body direction are assigned as gaze evaluation elements for gaze estimation.

[0112] In addition, in the relatively long distance range of distances D2 to D3, it is optimal to use body orientation for gaze estimation. In this distance range, as described above, it is not possible to properly extract both the eye orientation and the face orientation from the captured image. Therefore, in the third mode corresponding to distances D2 to D3, only body orientation is assigned as a gaze evaluation element for gaze estimation.

[0113] In the second modification, the gaze estimation process is the same as that in Fig. 4. However, in the second modification, the process in step S106 in Fig. 4 is changed from that in the first embodiment as follows.

[0114] (Estimation Process 1) In estimation process 1, a weight is applied to each gaze evaluation element (eye direction, face direction, body direction) according to the reliability when the element was extracted from the captured image, and the gaze estimation results based on each gaze evaluation element are integrated. Specifically, the gaze estimation result is obtained by the following formula:

[0115]

[0116] Here, the above-mentioned three-dimensional coordinate axes for defining the gaze evaluation elements (eye direction, face direction, and body direction) are indicated by x, y, and z. Furthermore, variable i is a variable indicating the type of gaze evaluation element used for gaze estimation. In the example of FIG. 6 , when the first mode is used for gaze estimation, there are three types of gaze evaluation elements: eye direction, face direction, and body direction, so variable i is an integer between 1 and 3. In this case, for example, variable i=1 indicates eye direction, variable i=2 indicates face direction, and variable i=3 indicates body direction. n is the total number of types of gaze evaluation elements used for gaze estimation.

[0117] In formula (1), X(x, y, z) ｉ is the direction of the line of sight evaluation element (i) expressed by the x-axis, y-axis, and z-axis components of the three-dimensional coordinate axes mentioned above. ｉ is a weighting according to the reliability when each gaze evaluation element (eye direction, face direction, body direction) is extracted from the captured image.

[0118] As described above, the gaze estimation device 10 is equipped with a recognition engine for extracting a person's eyes, face, and body from one frame of captured image. When performing recognition processing for these features, this recognition engine outputs the reliability of the recognition results for these features. The weighting wc in the above formula (1) is ｉ are values ​​corresponding to the reliability of extracting a person's eyes, face, and body from a captured image, and are applied to the gaze evaluation elements of eye direction, face direction, and body direction, respectively. ｉ is set in the range of 0.0 to 1.0.

[0119] Therefore, the angle (x, y, z) in the formula (1) is a weighting wc according to the reliability of each gaze evaluation element (i) to the X-axis component, Y-axis component, and Z-axis component of the direction of each gaze evaluation element (i). ｉ The result is a vectorial integration (addition) of the two.

[0120] When the first mode of FIG. 6 is executed, the control unit 11 calculates the orientations X(x, y, z) corresponding to the three types of gaze evaluation elements (i=1, 2, 3) of the eyeball orientation, the face orientation, and the body orientation. ｉ and weighting wc ｉ is applied to equation (1) to obtain the estimated result angle(x, y, z) of the line of sight of the person.

[0121] When the second mode is executed, the control unit 11 calculates the orientations X(x, y, z) corresponding to the two types of gaze evaluation elements (i=1, 2) of the face orientation and the body orientation. ｉ and weighting wc ｉ is applied to equation (1) to obtain the estimated result angle(x, y, z) of the line of sight of the person.

[0122] When the third mode is executed, the control unit 11 determines only the body orientation as the orientation X (x, y, z) corresponding to the line-of-sight evaluation element (i=1). ｉ and weighting wc ｉ is applied to equation (1) to obtain the estimated result angle(x, y, z) of the line of sight of the person.

[0123] According to this estimation process, the reliability of each gaze evaluation element when extracted is taken into consideration when acquiring an estimation result of a person's gaze, so that a person's gaze can be estimated with high accuracy.

[0124] In addition, weighting wc for each gaze evaluation element ｉ The direction X(x, y, z) of the gaze evaluation element (i) used for gaze estimation is ｉ In this case, the weighting wc in the above formula (1) may be simply integrated vectorially. ｉ is set to 1 for all gaze evaluation elements. However, in this case, the gaze estimation result is obtained without taking into account the reliability of the gaze evaluation elements as described above, and therefore the gaze estimation accuracy is lower than that described above. In order to obtain a gaze estimation result with higher accuracy, weighting wc according to the reliability of each gaze evaluation element is added to the above formula (1). ｉ It is preferable to apply the following to obtain the estimation result.

[0125] (Estimation Process 2) In estimation process 2, a weighting according to the reliability of each gaze evaluation element in gaze estimation is applied to each gaze evaluation element, and gaze estimation results based on each gaze evaluation element are integrated. Specifically, the gaze estimation result is obtained by the following formula:

[0126]

[0127] In equation (2), the weighting wc in equation (1) ｉ But the weighting wk ｉhas been changed to. When the distance to a person is not taken into consideration, the accuracy of gaze estimation is usually eye direction > face direction > body direction. In other words, because eye direction is directly related to gaze, the accuracy of gaze estimation based on eye direction is the highest. Furthermore, because people tend to turn their faces in the direction of gaze, the accuracy of gaze estimation based on face direction is the second highest after eye direction. Furthermore, because people find it more difficult to turn their bodies in the direction of gaze than their faces, the accuracy of gaze estimation based on body direction is the lowest. Therefore, when the distance to a person is not taken into consideration, the accuracy of gaze estimation is usually eye direction > face direction > body direction.

[0128] Weighting wk in equation (2) ｉ is set taking into consideration the likelihood of each of these gaze evaluation elements. For example, for each of the gaze evaluation elements, i.e., the eye direction, the face direction, and the body direction, a weight wk is set as follows: ｉ are set to 0.5, 0.3, and 0.2, respectively.

[0129] wk (eye direction, face direction, body direction) = (0.5, 0.3, 0.2) Therefore, angle (x, y, z) in equation (1) is a weighting wk that corresponds to the likelihood of each gaze evaluation element (i) on the X-axis component, Y-axis component, and Z-axis component of the direction of each gaze evaluation element (i). ｉ The result is a vectorial integration (addition) of the two.

[0130] When the first mode of FIG. 6 is executed, the control unit 11 calculates the orientations X(x, y, z) corresponding to the three types of gaze evaluation elements (i=1, 2, 3) of the eyeball orientation, the face orientation, and the body orientation. ｉ and weighting wk ｉ is applied to equation (2) to obtain the estimated result angle(x, y, z) of the line of sight of the person.

[0131] When the second mode is executed, the control unit 11 calculates the orientations X(x, y, z) corresponding to the two types of gaze evaluation elements (i=1, 2) of the face orientation and the body orientation. ｉ and weighting wk ｉ is applied to equation (2) to obtain the estimated result angle(x, y, z) of the line of sight of the person.

[0132] When the third mode is executed, the control unit 11 determines only the body orientation as the orientation X (x, y, z) corresponding to the line-of-sight evaluation element (i=1). ｉ and weighting wk ｉ is applied to equation (2) to obtain the estimated result angle(x, y, z) of the line of sight of the person.

[0133] According to this estimation process, a weighting according to the accuracy of gaze estimation by each gaze evaluation element is applied to each gaze evaluation element to obtain a gaze estimation result, so that a person's gaze can be estimated with high accuracy.

[0134] In addition, weighting wc according to the reliability of each gaze evaluation element in the above estimation process 1 ｉ and weighting wk according to the likelihood of each gaze evaluation element in estimation process 2. ｉ By applying both of the above, an estimation result of the gaze of the person may be obtained from the following formula:

[0135]

[0136] According to this estimation process, the reliability of each gaze evaluation element when extracted and the accuracy of gaze estimation by each gaze evaluation element are taken into consideration to obtain an estimation result of a person's gaze, so that a person's gaze can be estimated with high accuracy.

[0137] <Effect of Modification Example 2> As shown in FIG. 6 , the gaze estimation unit 11d uses, from among a plurality of gaze evaluation elements (eye direction, face direction, body direction), a gaze evaluation element that has been preset as one that can be extracted from a captured image in each distance range, for gaze estimation in each distance range.

[0138] According to this configuration, gaze estimation elements that can be extracted in each distance range are used to estimate the gaze, and therefore, for a given distance range, gaze estimation is performed using multiple gaze evaluation elements. Therefore, in this distance range, the gaze of a person can be comprehensively estimated using multiple gaze evaluation elements, and even if one of these gaze evaluation elements cannot be extracted from the captured image, the estimation result of the gaze of a person can be obtained from the other gaze evaluation elements.

[0139] As shown in the above formulas (1) to (3), when performing gaze estimation within a distance range in which a plurality of gaze evaluation elements (eye direction, face direction, body direction / face direction, body direction) are set, the gaze estimation unit 11d calculates the gaze estimation result based on each gaze evaluation element, that is, the direction X(x, y, z) of the gaze evaluation element (i). ｉ are integrated to obtain the estimation result of the human gaze.

[0140] According to this configuration, for a distance range in which multiple gaze evaluation elements are set, a person's gaze can be comprehensively estimated from the gaze estimation results based on these gaze evaluation elements, and even if any of the gaze evaluation elements cannot be extracted from the captured image, the estimation results of the person's gaze can be obtained from the other gaze evaluation elements.

[0141] In the estimation process 1 using the above formula (1), the gaze estimation unit 11d assigns a weighting wc according to the reliability when each gaze evaluation element is extracted from the captured image. ｉ is applied to each gaze evaluation element to obtain the gaze direction X(x, y, z) ｉ Integrate.

[0142] According to this configuration, the reliability of each gaze evaluation element when extracted is taken into consideration when acquiring the estimation result of the gaze of a person, so that the gaze of a person can be estimated with high accuracy.

[0143] In the estimation process 2 using the above formulas (2) and (3), the gaze estimation unit 11d assigns weights wk according to the likelihood of each gaze evaluation element in gaze estimation. ｉ is applied to each gaze evaluation element to obtain the gaze direction X(x, y, z) ｉ Integrate.

[0144] As described above, the accuracy of gaze estimation is usually in the order of eye direction > face direction > body direction. Therefore, according to estimation process 2, weighting wk according to such accuracy is performed. ｉ is applied to each gaze evaluation element to obtain a gaze estimation result, so that a person's gaze can be estimated with high accuracy.

[0145] Modification Example 3 FIG. 7 is a flowchart showing the line-of-sight estimation process according to Modification Example 3. In FIG.

[0146] In the first embodiment, one of the first process, the second process, and the third process is used for distance estimation depending on the distance to the person. In contrast, in the modified example, weighting according to the distance to the person is applied to each gaze evaluation element, gaze estimation results based on each gaze evaluation element are integrated, and the estimation result obtained by this integration is acquired as the gaze of the person.

[0147] In the flowchart of Fig. 7, the processes of steps S121, S122, and S124 are executed by the control unit 11 using the functions of the image acquisition unit 11a, extraction unit 11c, and distance acquisition unit 11b shown in Fig. 2, respectively, and the processes of the other steps are executed by the control unit 11 using the function of the line of sight estimation unit 11d shown in Fig. 2. In the following, the description will be made assuming that the control unit 11 executes the process of Fig. 7 using these functions.

[0148] The control unit 11 acquires one frame of a captured image from the camera 20 (S121) and extracts multiple types of gaze evaluation elements (eye direction, face direction, and body direction) from the acquired captured image to estimate the gaze of a person in the captured image (S122). The control unit 11 determines whether or not at least one of these gaze evaluation elements has been extracted (S123). For example, if none of the eyes, face, or body has been extracted from the captured image, the determination in step S123 is NO. In this case, the control unit 11 ends the current process without acquiring a gaze estimation result.

[0149] If any of the multiple types of gaze evaluation elements (eye direction, face direction, body direction) can be extracted (S123: YES), the control unit 11 acquires a distance image from the distance sensor 30 and acquires the distance to the person in the captured image (S124).The control unit 11 then applies weighting according to the acquired distance to each gaze evaluation element (S125) and acquires an estimation result of the gaze toward the person (S126).

[0150] FIG. 8 is a graph showing an example of a method for setting weights to be applied to the respective gaze evaluation elements according to distance.

[0151] 8, the horizontal axis represents distance and the vertical axis represents weighting value. Here, the maximum distance is set to 20 m.

[0152] In a distance range ΔD1 where an image of the eye including the pupil can be properly extracted from the captured image and the direction of the eyeball can be properly acquired, the weighting applied to the gaze evaluation element of the direction of the eyeball is set to 1, and the weighting applied to the other evaluation elements is set to 0.

[0153] When the distance to the person exceeds distance range ΔD1, it becomes increasingly difficult to extract an image of the eyes, including the pupils, from the captured image, so in distance range ΔD2, the weighting applied to the gaze evaluation element of eyeball direction is adjusted to decrease from 1 to 0 as the distance increases. On the other hand, in this distance range ΔD2, a facial image can be extracted so that the facial direction can be extracted from the captured image, so the weighting of the gaze evaluation element of facial direction is set to increase from 0 to 1 as the gaze evaluation element of eyeball direction decreases. At each distance in distance range ΔD2, the weighting values ​​of the eyeball direction and the facial direction are set so that the sum of their weighting values ​​becomes 1.

[0154] In the distance range ΔD3 following the distance range ΔD2, the entire face is included in the captured image to a relatively large extent, so the facial orientation can be appropriately extracted. For this reason, in the distance range ΔD3, the weight applied to the gaze evaluation element of the facial orientation is set to 1, and the weights applied to the other evaluation elements are set to 0.

[0155] When the distance to the person exceeds distance range ΔD3, it becomes increasingly difficult to extract a face image from the captured image, so in distance range ΔD4, the weighting applied to the face direction gaze evaluation element is adjusted to decrease from 1 to 0 as the distance increases. On the other hand, in this distance range ΔD4, it is possible to extract a body image so that the body direction can be extracted from the captured image, so the weighting of the body direction gaze evaluation element is set to increase from 0 to 1 as the face direction gaze evaluation element decreases. At each distance in distance range ΔD4, the weighting values ​​of the face direction and the body direction are set so that the sum of their weighting values ​​becomes 1.

[0156] In the distance range ΔD5 following the distance range ΔD4, the entire body is included in the captured image to a relatively large extent, so the body orientation can be appropriately extracted. For this reason, in the distance range ΔD5, the weight applied to the gaze evaluation element of the body orientation is set to 1, and the weight applied to the other evaluation elements is set to 0.

[0157] The storage unit 12 in Fig. 2 stores information that defines the relationship between the distance and the weighting of each gaze evaluation element as shown in Fig. 8. This information may be a table that associates each distance with the weighting of each gaze evaluation element, or may be a calculation formula that defines the relationship in Fig. 8 for each gaze evaluation element.

[0158] The control unit 11 obtains weights corresponding to the distances obtained in step S124 of Fig. 7 for each gaze evaluation element from the information stored in the storage unit 12, and applies these obtained weights to each gaze evaluation element in step S125. Then, in step S126, the control unit 11 obtains a gaze estimation result from the following formula:

[0159]

[0160] Weighting wd in equation (4) ｉ is the weighting of the gaze evaluation element (i), and is acquired from the information defining the relationship in Fig. 8. n is the number of gaze evaluation elements extracted in step S122 in Fig. 7.

[0161] For example, if the gaze evaluation elements extracted in step S122 are the eye direction and the face direction, n is 2. In this case, the control unit 11 calculates X(x, y, z) indicating the eye direction. １ and X (x, y, z) indicating the face direction. ２ and weights wd of the eyeball direction and face direction corresponding to the distances acquired in step S124. １ , wd ２ and are respectively substituted into the above equation (4) to obtain the gaze detection result (steps S125 and S126 in FIG. 7).

[0162] If the gaze evaluation elements extracted in step S122 are the eye direction, the face direction, and the body direction, n is 3. In this case, the control unit 11 calculates X(x, y, z) indicating the eye direction. １ and X (x, y, z) indicating the face direction. ２ and X (x, y, z) which indicates the body orientation ３ and weights wd for the eyeball direction, face direction, and body direction corresponding to the distances acquired in step S124. １ , wd ２ , wd ３ and are respectively substituted into the above equation (4) to obtain the gaze detection result (steps S125 and S126 in FIG. 7).

[0163] <Effects of Modification 3> As shown in FIG. 8 and the above formula (4), the line-of-sight estimation unit 11d calculates the weighting wd according to the distance acquired by the distance acquisition unit 11b. ｉ is applied to each gaze evaluation element (eye direction, face direction, body direction), and the gaze direction X(x, y, z) is the gaze estimation result based on each gaze evaluation element. ｉ The integrated result is then estimated as the human gaze.

[0164] According to this configuration, as described above, weighting according to distance is applied to each gaze evaluation element, and the gaze estimation results based on each gaze evaluation element are integrated, so that a person's gaze can be estimated with high accuracy.

[0165] As shown in FIG. 8, the distance ranges where the weighting of each gaze evaluation element is high are aligned on the distance axis, and the weighting wd is set so that there is a range on the distance axis where one of the weightings of two adjacent gaze evaluation elements decreases while the other increases. ｉ has been adjusted.

[0166] According to this configuration, it is possible to appropriately weight each of the gaze evaluation elements, thereby enabling accurate estimation of a person's gaze.

[0167] <Modification 4> In the above-described first embodiment, the distance sensor 30 is used to acquire the distance to a person. In contrast, in modification 4, the distance to a person is acquired from a captured image acquired by the camera 20. In modification 4, the distance sensor 30 is omitted from the configuration of the above-described first embodiment.

[0168] FIG. 9 is a diagram schematically showing a method for acquiring the distance to a person according to the fourth modification.

[0169] For example, the control unit 11 (distance acquisition unit 11b) extracts a person's face from the captured image P0 and detects the size W1 of the face. Then, the control unit 11 acquires the distance to the person based on the detected size W1 of the face. Here, the maximum width of the face is used as the size W1 of the face. The area of ​​the person's face in the captured image P0 becomes smaller as the person moves farther away from the camera 20. On the other hand, the physical size of a person's face does not vary significantly from person to person. Therefore, the distance to the person can be detected from the size W1 of the person's face in the captured image P0.

[0170] In this case, information defining the relationship between the face size W1 and the distance to the person is stored in advance in the storage unit 12. For example, a table defining the relationship between the face size W1 and the distance to the person is used as this information. The control unit 11 obtains the distance to the person based on the face size W1 obtained from the captured image P0 and this information.

[0171] Although the size W1 of the face on the captured image P0 is used to detect the distance to the person here, other parameters such as shoulder width on the captured image P0 may also be used to detect the distance to the person. It is preferable that the parameters used to detect the distance to the person do not vary significantly from person to person and can be smoothly acquired from the captured image P0.

[0172] Alternatively, the distance to the person may be detected from the height H1 from the bottom of the captured image P0 to the person's feet. The farther the person is from the camera 20, the smaller the area of ​​the person in the captured image P0 becomes and the higher it moves. Therefore, the distance to the person can be detected from the height H1 from the bottom of the captured image P0 to the position of the person's feet.

[0173] In this case, information defining the relationship between the height H1 from the bottom of the captured image P0 to the position of the person's feet and the distance to the person is pre-stored in the storage unit 12. For example, a table defining the relationship between the height H1 and the distance to the person is used as this information. The control unit 11 obtains the distance to the person based on the height H1 obtained from the captured image P0 and this information.

[0174] Instead of the height H1 to the feet, the height from the bottom of the captured image P0 to the position of another part of the person may be used to detect the distance to the person. For example, the height H2 from the bottom of the captured image P0 to the position of the person's waist may be used to detect the distance to the person.

[0175] However, since the height from the feet to the waist tends to vary depending on the height of a person, using height H2 to detect the distance to a person tends to result in a slight decrease in the accuracy of distance detection. Therefore, in order to detect the distance to a person more accurately, it is preferable to use height H from the bottom of captured image P0 to the position of the person's feet, as described above.

[0176] Alternatively, a monocular camera distance measurement technique using a machine learning model may be used to detect the distance to a person. In this case, an engine that executes this measurement technique is installed in the storage unit 12. The control unit 11 uses this engine to detect the distance to the person in the captured image.

[0177] According to the configuration of the fourth modification, the distance to the person is acquired from the captured image from the camera 20, so there is no need to use a separate distance sensor 30 as in the first embodiment. Therefore, the configuration required for gaze estimation can be simplified.

[0178] In the first embodiment, the estimation process using a plurality of gaze evaluation elements is changed depending on the distance to the person. In contrast, in the second embodiment, weighting is applied to each gaze evaluation element according to the reliability at the time of extracting each gaze evaluation element from the captured image, and gaze estimation results based on each gaze evaluation element are integrated, and the integrated estimation result is acquired as the gaze of the person.

[0179] FIG. 10 is a diagram showing the configuration of a gaze estimation system 1 according to the second embodiment.

[0180] 1 , in the second embodiment, the distance sensor 30 is omitted from the gaze estimation system 1. The other configurations of the gaze estimation system 1 according to the second embodiment are the same as those of the first embodiment. However, as described above, in the second embodiment, the gaze estimation process in the gaze estimation device 10 is different from that of the first embodiment.

[0181] FIG. 11 is a block diagram showing the configuration of a gaze estimation device 10 constituting a gaze estimation system 1 according to the second embodiment.

[0182] 2 , in the second embodiment, the distance acquisition unit 11b is omitted from the functions executed by the control unit 11. Also, the distance sensor 30 is excluded from the targets with which the communication unit 15 communicates. The other configurations are the same as those of the first embodiment. However, in the second embodiment, as described above, the function of the gaze estimation process in the gaze estimation unit 11d is different from that of the first embodiment.

[0183] FIG. 12A is a flowchart showing the gaze estimation process according to the second embodiment.

[0184] In the flowchart of Fig. 12(a), the processes of steps S201 and S202 are executed by the control unit 11 using the functions of the image acquisition unit 11a and the extraction unit 11c shown in Fig. 11, respectively, and the processes of the other steps are executed by the control unit 11 using the functions of the gaze estimation unit 11d shown in Fig. 11. In the following, the description will be made assuming that the control unit 11 executes the process of Fig. 12 using these functions.

[0185] The control unit 11 acquires one frame of a captured image from the camera 20 (S201) and extracts multiple types of gaze evaluation elements (eye direction, face direction, and body direction) from the acquired captured image to estimate the gaze of a person in the captured image (S202). The control unit 11 determines whether or not at least one of these gaze evaluation elements has been extracted (S203). For example, if none of the eyes, face, or body has been extracted from the captured image, the determination in step S203 is NO. In this case, the control unit 11 ends the current process without acquiring a gaze estimation result.

[0186] If any of the multiple types of gaze evaluation elements (eye direction, face direction, body direction) can be extracted (S203: YES), the control unit 11 acquires the reliability of each of the extracted gaze evaluation elements (S204). Here, the reliability is the reliability of the recognition results of the human eyes, face, and body parts output by the recognition engine when recognizing these parts from one frame of captured image acquired in step S201, as described above. The reliability of the recognition results of the human eyes, face, and body is the reliability of the eye direction, face direction, and body direction, respectively.

[0187] The control unit 11 acquires an estimation result of the gaze of the person from the reliability acquired for each gaze evaluation element (S205). This process is performed using the above formula (1).

[0188] That is, the control unit 11 assigns a weighting wc according to the reliability acquired in step S204. ｉ is applied to each gaze evaluation element. As above, weighting wc ｉ is set in the range of 0.0 to 1.0. For gaze evaluation elements with a reliability of 0, a weighting wc ｉ is set to 0.0. Alternatively, for the gaze evaluation element that could not be extracted in step S203, a weighting wc ｉ may be set to 0.0.

[0189] The control unit 11 calculates the weighting wc applied to each of the gaze evaluation elements in this way. ｉ and the direction X (x, y, z) of each gaze evaluation element ｉ and are applied to the above formula (1) to obtain an estimation result of the gaze of the person. At this time, the gaze evaluation elements that could not be extracted in step S202 are excluded from the calculation process. The control unit 11 stores the obtained gaze estimation result in the storage unit 12. This ends the process of FIG. 12(a). Thereafter, the control unit 11 returns the process to step S201 to execute the next estimation process.

[0190] 12A. In this case, for example, if it is not possible to extract all the gaze evaluation elements, the orientation (x, y, z) of the gaze evaluation element to be applied to the formula (1) in step S205 is omitted. ｉ 12(a) のメソッド Because there is no such a condition, an estimation result of the gaze of the person cannot be obtained, just as when the determination in step S203 in the flowchart of Fig. 12(a) is NO. Therefore, in terms of whether or not an estimation result of the gaze can be obtained, it is the same as when step S203 is included in the flowchart. However, in the process of Fig. 12(a) , if the determination in step S203 is NO, steps S204 and S205 are skipped, and the processing load on the control unit 11 can be reduced accordingly.

[0191] <Effects of Embodiment 2> As shown in FIG. 11 , the gaze estimation device 10 includes an image acquisition unit 11a that acquires a captured image including at least a part of a person from a camera, an extraction unit 11c that extracts a plurality of gaze evaluation elements of different types from the captured image, and a gaze estimation unit 11d that applies weighting to each gaze evaluation element according to the reliability of each element when extracted from the captured image, integrates the gaze estimation results based on each gaze evaluation element, and acquires the integrated estimation result as the gaze of the person.

[0192] According to this configuration, a plurality of different gaze evaluation elements are used to estimate a person's gaze, so that the person's gaze can be estimated over a wide distance range. In addition, weighting is applied to each gaze evaluation element according to the reliability at the time of extraction of each gaze evaluation element, and the gaze estimation results based on each gaze evaluation element are integrated, so that the person's gaze can be estimated with high accuracy.

[0193] <Modification> In the process of Figure 12 (a), only the reliability when extracting each gaze evaluation element from the captured image is taken into account, but gaze estimation may also be performed by taking into account the likelihood of each of the gaze evaluation elements in gaze estimation.

[0194] FIG. 12B is a flowchart showing a gaze estimation process according to a modification of the second embodiment.

[0195] In the flowchart of Fig. 12(b), step S205 in the flowchart of Fig. 12(a) is changed to step S211. The processing of the other steps in the flowchart of Fig. 12(b) is the same as the processing of the corresponding steps in the flowchart of Fig. 12(a).

[0196] In step S211, the control unit 11 acquires a gaze estimation result of the person based on the reliability of extracting each gaze evaluation element from the captured image and the likelihood of each of the gaze evaluation elements in gaze estimation. This process is performed using the above formula (3).

[0197] The control unit 11 performs weighting wc according to the reliability of each line-of-sight evaluation element, similarly to step S205 in FIG. ｉ Further, the control unit 11 applies a weighting wk according to the likelihood of each of the gaze evaluation elements in gaze estimation, similar to the estimation process 2 using the above formula (2). ｉ Then, the control unit 11 applies the weighting wc applied to each of the gaze evaluation elements. ｉ and wk ｉ and the direction X (x, y, z) of each gaze evaluation element ｉ and are applied to the above formula (3) to obtain an estimation result of the gaze of the person. At this time, the gaze evaluation elements that could not be extracted in step S202 are excluded from the calculation process.

[0198] The control unit 11 stores the acquired gaze estimation result in the storage unit 12. This ends the processing of Fig. 12(b). Thereafter, the control unit 11 returns the processing to step S201 and executes the next round of estimation processing. As in the case of Fig. 12(a), step 203 may be omitted from the flowchart of Fig. 12(b).

[0199] According to this modification, weighting wc according to the reliability of each of the gaze evaluation elements in gaze estimation is ｉ is further applied to each gaze evaluation element to obtain a gaze estimation result, so that a person's gaze can be estimated with high accuracy.

[0200] In addition, the weighting wk ｉmay be adjusted depending on the scene, environment, etc. in which the gaze estimation is performed. For example, when estimating the gaze of a user facing the display screen of a terminal device for purchasing a product, and the user's face is mainly included in the captured image, the weighting wk of each gaze evaluation element may be adjusted so that the weighting wk of the eye direction and the face direction is increased, as shown in the following equation.

[0201] wk (eye direction, face direction, body direction) = (0.6, 0.4, 0.1) Furthermore, in a situation where the gaze of a person located at a relatively long distance from camera 20 needs to be estimated, it is difficult to extract the eye direction from the captured image, so the weighting wk of the eye direction may be set low, as in the following equation.

[0202] wk (eye direction, face direction, body direction)=(0.1, 0.4, 0.6) By adjusting the weighting wk for each scene in this way, it is possible to improve the accuracy of gaze estimation according to each scene.

[0203] <Other Modifications> In the above-described first and second embodiments, the eye direction, face direction, and body direction are exemplified as gaze evaluation elements, but the gaze evaluation elements used for gaze estimation are not limited to these. For example, since the accuracy of gaze estimation is eye direction > face direction > body direction, even if these three gaze estimation elements can be acquired from a captured image, the gaze evaluation elements used for gaze estimation may be limited to the eye direction and face direction.

[0204] This allows for a higher accuracy of gaze estimation, although the upper limit distance for gaze estimation is shorter than when the above three types of gaze evaluation elements are used for gaze estimation. Therefore, a person's gaze can be estimated with high accuracy over a wide distance range in which the eye direction and face direction can be extracted.

[0205] Furthermore, gaze evaluation elements other than the above three gaze evaluation elements may also be used for gaze estimation.

[0206] In addition, in the above-described first and second embodiments, the camera 20 is installed on the upper surface of the advertising device 40, but the camera 20 may be installed separately from the advertising device 40. In addition, in the above-described embodiments, the display area 41 of the advertising device 40 is exemplified as the gaze target, but the gaze target may be an advertising medium such as a poster.

[0207] Furthermore, in the above-mentioned embodiment 1, the gaze estimation device 10, the camera 20, and the distance sensor 30 are connected by a communication line, and in the above-mentioned embodiment 2, the gaze estimation device 10 and the camera 20 are connected by a communication line, but if the gaze estimation device 10 is installed in a location far away from the camera 20 and the distance sensor 30, they may be connected via an external network such as a LAN or a public communication network.

[0208] Furthermore, the gaze estimation device 10 does not necessarily have to be a single unit, and the gaze estimation device 10 may be configured by multiple information processing devices that share the functions of the image acquisition unit 11a, the distance acquisition unit 11b, the extraction unit 11c, and the gaze estimation unit 11d.

[0209] Furthermore, although one camera 20 is used in the above-described first and second embodiments, it is also possible to provide a plurality of cameras 20. If these plurality of cameras 20 can constitute a stereo camera, the distance to a person in each captured image may be measured based on the parallax of these cameras.

[0210] In addition, the embodiments of the present invention can be modified as appropriate within the scope of the claims.

[0211] REFERENCE SIGNS LIST 1 Gaze estimation system 10 Gaze estimation device 11 Control unit 11a Image acquisition unit 11b Distance acquisition unit 11c Extraction unit 11d Gaze estimation unit 20 Camera 30 Distance sensor

Claims

1. An image acquisition unit that acquires a captured image including at least a part of a person from a camera, a distance acquisition unit that acquires the distance to the person, an extraction unit that extracts a plurality of line-of-sight evaluation elements different in type from each other from the captured image, and a line-of-sight estimation unit that changes an estimation process using the plurality of line-of-sight evaluation elements according to the distance acquired by the distance acquisition unit and estimates the line of sight of the person. A line-of-sight estimation device characterized by comprising:

2. The line-of-sight estimation device according to claim 1, wherein the plurality of line-of-sight evaluation elements include the direction of the person's eyeballs and the direction of the face. A line-of-sight estimation device characterized by comprising:

3. The line-of-sight estimation device according to claim 2, wherein the plurality of line-of-sight evaluation elements further include the direction of the person's body. A line-of-sight estimation device characterized by comprising:

4. The line-of-sight estimation device according to claim 1, wherein the line-of-sight estimation unit changes the line-of-sight evaluation elements used for line-of-sight estimation for each distance range. A line-of-sight estimation device characterized by comprising:

5. The line-of-sight estimation device according to claim 4, wherein the line-of-sight estimation unit uses, for line-of-sight estimation, one of the line-of-sight evaluation elements assumed to have the highest reliability for each of the distance ranges. A line-of-sight estimation device characterized by comprising:

6. The line-of-sight estimation device according to claim 4, wherein the line-of-sight estimation unit uses, for line-of-sight estimation in each of the distance ranges, the line-of-sight evaluation elements preset as extractable from the captured image in each of the distance ranges among the plurality of line-of-sight evaluation elements. A line-of-sight estimation device characterized by comprising:

7. The line-of-sight estimation device according to claim 6, wherein when the line-of-sight estimation unit performs line-of-sight estimation in the distance range where a plurality of the line-of-sight evaluation elements are set, the line-of-sight estimation device integrates the line-of-sight estimation results based on each of the line-of-sight evaluation elements to obtain the line-of-sight estimation result of the person. A line-of-sight estimation device characterized by comprising:

8. The line-of-sight estimation device according to claim 7, wherein the line-of-sight estimation unit applies a weighting according to the reliability when each of the line-of-sight evaluation elements is extracted from the captured image to each of the line-of-sight evaluation elements and integrates the line-of-sight estimation results based on each of the line-of-sight evaluation elements. A line-of-sight estimation device characterized by comprising:

9. In the line-of-sight estimation device according to claim 7, the line-of-sight estimation unit applies a weighting according to the probability of each of the line-of-sight evaluation elements in the line-of-sight estimation to each of the line-of-sight evaluation elements, and integrates the line-of-sight estimation results based on each of the line-of-sight evaluation elements. A line-of-sight estimation device characterized by the above.

10. In the line-of-sight estimation device according to claim 1, the line-of-sight estimation unit applies a weighting according to the distance acquired by the distance acquisition unit to each of the line-of-sight evaluation elements, integrates the line-of-sight estimation results based on each of the line-of-sight evaluation elements, and obtains the integrated estimation result as the line of sight of the person. A line-of-sight estimation device characterized by the above.

11. In the line-of-sight estimation device according to claim 10, the distance ranges in which the weightings of each of the line-of-sight evaluation elements increase are arranged on the distance axis, and a range in which one of the weightings of two adjacent line-of-sight evaluation elements decreases while the other increases exists on the distance axis. A line-of-sight estimation device characterized by the above.

12. In the line-of-sight estimation device according to claim 1, the distance acquisition unit acquires the distance to the person from a distance sensor. A line-of-sight estimation device characterized by the above.

13. In the line-of-sight estimation device according to claim 1, the distance acquisition unit acquires the distance to the person from the captured image. A line-of-sight estimation device characterized by the above.

14. An image acquisition unit that acquires a captured image including at least a part of a person from a camera, an extraction unit that extracts a plurality of line-of-sight evaluation elements of different types from the captured image, and a weight according to the reliability when each of the line-of-sight evaluation elements is extracted from the captured image. A line-of-sight estimation device comprising: a line-of-sight estimation unit that applies a weighting to each of the line-of-sight evaluation elements, integrates the line-of-sight estimation results based on each of the line-of-sight evaluation elements, and obtains the integrated estimation result as the line of sight of the person.

15. In the line-of-sight estimation device according to claim 14, the line-of-sight estimation unit further applies a weighting according to the probability of each of the line-of-sight evaluation elements in the line-of-sight estimation to each of the line-of-sight evaluation elements, and integrates the line-of-sight estimation results based on each of the line-of-sight evaluation elements. A line-of-sight estimation device characterized by the above.

16. A line-of-sight estimation system comprising the line-of-sight estimation device according to any one of claims 1 to 11, 13, and 14, and the camera.

17. A gaze estimation system comprising the gaze estimation device according to claim 12, the camera, and the distance sensor.

18. A gaze estimation method in an information processing device, the method comprising: acquiring a captured image including at least a part of a person; acquiring a distance to the person; extracting a plurality of gaze evaluation elements of different types from the captured image; and estimating the gaze of the person by changing a gaze estimation process using the plurality of gaze evaluation elements according to the distance acquired by the distance acquisition unit.

19. A gaze estimation method in an information processing device, the method comprising: acquiring a captured image including at least a part of a person; extracting a plurality of gaze evaluation elements of different types from the captured image; applying a weighting according to the reliability when each of the gaze evaluation elements is extracted from the captured image to each of the gaze evaluation elements, integrating gaze estimation results based on each of the gaze evaluation elements, and acquiring the integrated estimation result as the gaze of the person.

Images