Estimation device for visual line direction and estimation method therefor

Abstract

To provide an estimation device and an estimation method for a visual line direction that estimate the visual line direction of a person appearing in a two-dimensional image.SOLUTION: Estimation processing is performed to estimate a visual line direction of a person shown in a two-dimensional image. In the estimation processing, three-dimensional attitude information of a target person is acquired from the two-dimensional image that shows a target person whose visual line direction is to be estimated. In the estimation processing, three-dimensional position information of an object shown in the two-dimensional image is also acquired from the two-dimensional image. In the estimation processing, input information is further input to a neural network model that outputs the visual line direction of the person, and output information of the neural network model is acquired as the visual line direction of the target person. The input information for the neural network includes the three-dimensional attitude information and the three-dimensional position information acquired through the estimation processing.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an apparatus and method for estimating a person's gaze direction. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2024-029913 discloses a device for estimating the gaze direction of a person captured in a two-dimensional camera image of a space. This conventional device estimates the three-dimensional pose and three-dimensional coordinates of a person captured in the two-dimensional camera image, including the person's face. The conventional device also references the estimated three-dimensional pose and three-dimensional coordinates to estimate the three-dimensional pose and three-dimensional coordinates of the person's head captured in the two-dimensional camera image, and estimates the forward direction of the person's face area as the gaze direction.

[0003] In addition to JP 2024-029913 A, JP 2007-006427 A and JP 2020-027390 A can be exemplified as documents showing the technical state of the technical field related to the present disclosure. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2024-029913 [Patent Document 2] Japanese Patent Application Laid-Open No. 2007-006427 [Patent Document 3] Japanese Patent Publication No. 2020-027390 Summary of the Invention [Problem to be solved by the invention]

[0005] However, if the accuracy of estimating the 3D pose and 3D coordinates of a person's head in a 2D camera image is low, it is expected that the accuracy of the forward direction of the person's facial region will also be low. As a result, the gaze direction of the person in the 2D camera image cannot be accurately estimated, making it difficult to estimate what the person is focusing on. Therefore, it is desirable to study methods for estimating the gaze direction of a person in a 2D camera image from various angles and to make improvements.

[0006] An object of the present disclosure is to provide a technology capable of estimating the gaze direction of a person appearing in a two-dimensional image. [Means for solving the problem]

[0007] A first aspect of the present disclosure is a device for estimating the gaze direction of a person appearing in a two-dimensional image, and has the following features. The device includes a storage device and a processing circuit. The storage device stores a two-dimensional image of a person whose gaze direction is to be estimated, and a neural network model that outputs the gaze direction of the person. The processing circuit is configured to perform an estimation process to estimate the gaze direction of the person. The estimation process includes obtaining three-dimensional posture information of the target person from the two-dimensional image, obtaining three-dimensional position information of an object appearing in the two-dimensional image from the two-dimensional image, and inputting input information into the neural network model to obtain output information of the neural network model as the gaze direction of the target person. The input information of the neural network model includes the three-dimensional posture information and the three-dimensional position information.

[0008] A second aspect of the present disclosure is a method for causing a computer to perform an estimation process for estimating the gaze direction of a person appearing in a two-dimensional image, and has the following features. The estimation process includes acquiring three-dimensional posture information of a target person whose gaze direction is to be estimated from a two-dimensional image in which the target person appears, acquiring three-dimensional position information of an object appearing in the two-dimensional image from the two-dimensional image, and inputting input information to a neural network model that outputs the gaze direction of the person, and acquiring output information of the neural network model as the gaze direction of the target person. The input information of the neural network model includes the three-dimensional posture information and the three-dimensional position information. [Effects of the Invention]

[0009] According to the present disclosure, by inputting the 3D posture information of a target person and the 3D position information of an object acquired from a 2D image as input information for a neural network model that outputs the gaze direction of the person, it is possible to acquire the output information of this neural network model as the gaze direction of the target person. In other words, according to the present disclosure, it is possible to estimate the gaze direction of a person appearing in a 2D image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating an example configuration of an estimation device according to an embodiment of the present disclosure. [Figure 2] 2 is a block diagram illustrating a first example of a functional configuration of the estimation device shown in FIG. 1. FIG. [Figure 3] 1. FIG. 4 is a block diagram illustrating a second example of the functional configuration of the estimation device shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals, and the description thereof will be simplified or omitted.

[0012] 1. Example of the configuration of the estimation device Fig. 1 is a block diagram showing an example configuration of an estimation device according to an embodiment of the present disclosure. Fig. 1 illustrates a data processing device 10 and a display device 20 as a configuration of the estimation device according to the embodiment. The display device communicates with the data processing device 10 via a communication network (not shown). The communication network is not particularly limited, and a wired or wireless network may be used.

[0013] The data processing device 10 includes at least one processing circuit 11 and at least one storage device 12. Examples of the processing circuit 11 include a general-purpose processor, a specific application processor, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), and a field-programmable gate array (FPGA). Examples of the storage device 12 include a hard disk drive (HDD), a solid state drive (SSD), a volatile memory, and a non-volatile memory.

[0014] The processing circuit 11 develops various programs stored in the storage device 12 and processes various data stored in the storage device 12. The data processing by the processing circuit 11 includes processing of a two-dimensional image IMG. The processing of the two-dimensional image IMG includes processing to estimate the gaze direction GD (Gaze Direction) of a person appearing in the two-dimensional image IMG. The person whose gaze direction GD is to be estimated (hereinafter also referred to as "target person TG") is any person appearing in the two-dimensional image IMG, and is set by the data processing device 10. When the data processing device 10 receives input information specifying the target person TG, the target person TG may be set in accordance with this input information.

[0015] Here, the two-dimensional image IMG is, for example, one acquired from an RGB camera. The two-dimensional image IMG may be a composite of multiple images acquired from the RGB camera at different times, or may be a composite of multiple images acquired from multiple RGB cameras (for example, a person image and a background image).

[0016] In this embodiment, a one-shot two-dimensional image is considered as the two-dimensional image IMG. This is because it is assumed that the only information available for estimating the gaze direction GD is a one-shot two-dimensional image. When an RGB camera captures an image, the one-shot two-dimensional image corresponds to one of the time-series images that make up the image. When the two-dimensional image IMG is a composite of multiple images, this composite image corresponds to the one-shot two-dimensional image.

[0017] The gaze direction GD estimation process uses a neural network (NN) model NNM stored in the storage device 12. The neural network model NNM is constructed to output the gaze direction GD. An example of the neural network model NNM that outputs the gaze direction GD is a convolutional neural network (CNN).

[0018] The neural network model NNM is trained by, for example, supervised learning using training data including correct answer data. The training of the neural network model NNM is performed, for example, by i and output y i This is done using equation (1) (i = 1, , N, N ≥ 2). y → i =f(Θ → ,x → i )···(1) In formula (1), the superscript arrow indicates a vector set. → i contains a 2D image IMG. The input x → i The 2D image IMG is a set of images with a width of W and a height of T, and is expressed as a W×T vector. → i is the estimated probability value. Output y → i The function f(Θ → ) is the parameter set Θ →It is a function of the neural network model NNM that holds the above and outputs a two-dimensional vector.

[0019] The display device 20 displays various data. Examples of the display device 20 include a liquid crystal display, an organic electroluminescence display, and a head-up display. The various data displayed on the display device 20 is provided to a user of the estimation device according to the embodiment. The various data displayed on the display device 20 includes the gaze direction GD. When displaying the gaze direction GD, the data processing device 10 may generate a composite image in which an arrow indicating the gaze direction GD is superimposed on the two-dimensional image IMG that was the basis for estimating the gaze direction GD. The data processing device 10 may estimate an object located ahead of the gaze direction GD, i.e., an object that the target person TG focuses on, and display information about this focused object on the display device 20.

[0020] 2. Functional configuration example 2-1. First configuration example Fig. 2 is a block diagram showing a first example of the functional configuration of data processing device 10 shown in Fig. 1. In the example shown in Fig. 2, functional blocks of data processing device 10 include a three-dimensional posture estimation unit 13, a three-dimensional position estimation unit 14, and a gaze direction calculation unit 15. These functional blocks are realized, for example, by cooperation between processing circuitry 11 and storage device 12.

[0021] The three-dimensional posture estimation unit (3D posture estimation unit) 13 performs a process (3DPS estimation process) to estimate the three-dimensional posture 3DPS of a person depicted in the two-dimensional image IMG. In the 3DPS estimation process, for example, a bounding box is assigned to a person (target person TG) depicted in the two-dimensional image IMG (RGB image). Then, key points of the person (target person TG) are extracted from this bounding box to estimate the three-dimensional posture of the person. The three-dimensional posture is represented by lines connecting parts such as joints, head, hands, and feet. The position of each part is represented in a three-dimensional coordinate system (X, Y, Z). Note that this estimation process is a well-known technique, and the method is not particularly limited. For example, MeTRAbs, TransPose, etc. are used in the 3DPS estimation process. The three-dimensional posture PS_TG of the target person TG is transmitted to the gaze direction calculation unit 15.

[0022] The three-dimensional position estimation unit (3D position estimation unit) 14 performs a process (3DCD estimation process) of estimating the three-dimensional position (3D Coordinate) of an object reflected in the two-dimensional image IMG. In the 3DCD estimation process, an object OB reflected in the two-dimensional image IMG is detected using, for example, a YOLO (You Only Look Once) network or an SSD (Single Shot multibox Detector) network. The object OB to be detected is, for example, a static object such as a building, structure, or natural object, or a dynamic object such as a person (a person other than the target person TG), a robot, a bicycle, or a car. Information about the detected object OB includes information about the two-dimensional position of the object OB in the two-dimensional image IMG. The two-dimensional position of the object is expressed in a two-dimensional coordinate system (X, Y).

[0023] In the 3DCD estimation process, a depth image is also generated from the two-dimensional image IMG (RGB image). The depth image can be generated using a known machine learning model. In the 3DCD estimation process, depth information of the object OB reflected in the two-dimensional image IMG (i.e., distance information from the camera to the object) is further obtained from the depth image and added to the two-dimensional position information. This generates data on the three-dimensional position CD_OB of the object OB. The three-dimensional position of the object OB is expressed in a three-dimensional coordinate system (X, Y, Z). Note that if the camera that acquires the two-dimensional image IMG is a camera that can acquire a depth image (e.g., an RGB-D camera), the three-dimensional position CD_OB may be generated using a depth image acquired simultaneously with the two-dimensional image IMG. The data on the three-dimensional position CD_OB is transmitted to the gaze direction calculation unit 15.

[0024] The gaze direction calculation unit 15 performs a process (GD calculation process) to calculate the gaze direction GD of the target person TG (hereinafter also referred to as "gaze direction GD_TG"). In the GD calculation process, a neural network model NNM1 is used. In addition to the two-dimensional image IMG, the three-dimensional posture PS_TG of the target person TG received from the three-dimensional posture estimation unit 13 and the three-dimensional position CD_OB of the object OB received from the three-dimensional position estimation unit 14 are used as input to the neural network model NNM1. That is, the input variables of the neural network model NNM1 are the two-dimensional image IMG, the three-dimensional posture PS_TG, and the three-dimensional position CD_OB. The gaze direction GD_TG is obtained as output information of the neural network model NNM1.

[0025] 2-2. Second configuration example Fig. 3 is a block diagram showing a second example of the functional configuration of the data processing device 10 shown in Fig. 1. In the example shown in Fig. 3, functional blocks of the data processing device 10 include a face direction estimation unit 16, a three-dimensional posture estimation unit 17, a three-dimensional position estimation unit 18, and a gaze direction calculation unit 19. These functional blocks are realized, for example, by cooperation between the processing circuit 11 and the storage device 12.

[0026] The face direction estimation unit 16 performs a process (FD estimation process) to estimate the face direction of the target person TG. In the FD estimation process, for example, a face image IMG_TGF of the target person TG is extracted from a two-dimensional image IMG (RGB image). Then, a depth image generated from this face image IMG_TGF is used to estimate the position of the front of the face of the target person TG. The face direction FD_TG is estimated based on this position of the front. The face direction FD_TG is expressed as a three-dimensional vector. Data of the face direction FD_TG is transmitted to the gaze direction calculation unit 15.

[0027] In a second example of the FD estimation process, a depth image is generated from the two-dimensional image IMG (RGB image) without extracting the face image IMG_TGF. Then, the face direction FD_TG is estimated based on the position of the front of the face of the target person TG estimated using this depth image. In a third example of the FD estimation process, the three-dimensional posture of the target person TG shown in the two-dimensional image IMG is estimated. An example of a method for estimating the three-dimensional posture of the target person TG is the method using the three-dimensional posture estimation unit 13 described in FIG. 2. Then, the face direction FD_TG is estimated based on this three-dimensional posture of the target person TG. In the second or third example, data on the face direction FD_TG is also sent to the gaze direction calculation unit 15.

[0028] The function of three-dimensional posture estimation unit (3D posture estimation unit) 17 is the same as that of three-dimensional posture estimation unit 13 described in Fig. 2. Furthermore, the function of three-dimensional position estimation unit (3D position estimation unit) 18 is the same as that of three-dimensional position estimation unit 14 described in Fig. 2.

[0029] The gaze direction calculation unit 19 performs a process (GD calculation process) to calculate the gaze direction GD_TG. In the GD calculation process, a neural network model NNM2 is used. In addition to the two-dimensional image IMG, the neural network model NNM2 receives as input the face direction FD_TG received from the face direction estimation unit 16, the three-dimensional posture PS_TG of the target person TG received from the three-dimensional posture estimation unit 17, and the three-dimensional position CD_OB of the object OB received from the three-dimensional position estimation unit 18. That is, the input variables of the neural network model NNM2 are the two-dimensional image IMG, the face direction FD_TG, the three-dimensional posture PS_TG, and the three-dimensional position CD_OB. The gaze direction GD_TG is obtained as output information of the neural network model NNM2.

[0030] 3.Effects According to the embodiment, a neural network model NNM, which uses a two-dimensional image showing a target person, the three-dimensional posture of the target person, and the three-dimensional position of an object shown in the two-dimensional image as input variables and outputs the gaze direction of the target person, inputs the three-dimensional posture PS_TG of the target person TG and the three-dimensional position CD_OB of the object OB along with the two-dimensional image IMG, thereby obtaining the gaze direction GD_TG as output information. Alternatively, by adding the face direction FD_TG of the target person TG to the input variables, the gaze direction GD_TG can be obtained as output information. In other words, according to the embodiment, the gaze direction GD_TG of the target person TG shown in the two-dimensional image IMG can be estimated. Even if the two-dimensional image IMG is a one-shot image, it is possible to estimate the gaze direction GD_TG of the target person TG shown in the two-dimensional image IMG. [Explanation of symbols]

[0031] 10...data processing device, 11...processing circuit, 12...storage device, 13, 17...3D posture estimation unit, 14, 18...3D position estimation unit, 15, 19...gaze direction calculation unit, 16...face direction estimation unit, TG...target person, IMG...2D image, NNM, NNM1, NNM2...neural network model, CD_OB...3D position, FD_TG...face direction, GD_TG...gaze direction, PS_TG...3D posture, IMG_TGF...face image

Claims

1. An apparatus for estimating the gaze direction of a person appearing in a two-dimensional image, comprising: a storage device that stores a two-dimensional image of a person whose gaze direction is to be estimated and a neural network model that outputs the gaze direction of the person; a processing circuit that performs an estimation process to estimate the gaze direction of the target person, The estimation process acquiring three-dimensional posture information of the target person from the two-dimensional image; acquiring three-dimensional position information of an object shown in the two-dimensional image from the two-dimensional image; inputting input information into the neural network model and acquiring output information of the neural network model as the gaze direction of the target person; The input information of the neural network model includes the three-dimensional posture information and the three-dimensional position information. A gaze direction estimation device comprising:

2. 10. The apparatus of claim 1, the estimation process further includes acquiring face direction information of the target person appearing in the two-dimensional image; The input information of the neural network model further includes the face direction information. A gaze direction estimation device characterized by:

3. 3. The device according to claim 1 or 2, The two-dimensional image includes a one-shot two-dimensional image. A gaze direction estimation device characterized by:

4. A method for causing a computer to perform an estimation process for estimating the gaze direction of a person appearing in a two-dimensional image, comprising: The estimation process acquiring three-dimensional posture information of a target person whose gaze direction is to be estimated from a two-dimensional image in which the target person appears; acquiring three-dimensional position information of an object shown in the two-dimensional image from the two-dimensional image; inputting input information into a neural network model that outputs a gaze direction of a person, and acquiring output information of the neural network model as the gaze direction of the target person; The input information of the neural network model includes the three-dimensional posture information and the three-dimensional position information. A gaze direction estimation method comprising:

5. 5. The method of claim 4, the estimation process further includes acquiring face direction information of the target person appearing in the two-dimensional image; The input information of the neural network model further includes the face direction information. A gaze direction estimation method comprising:

6. 3. The method of claim 1 or 2, The two-dimensional image includes a one-shot two-dimensional image. A method for estimating gaze direction, comprising:

Images