Estimation device and estimation method for line-of-sight direction

Abstract

The invention relates to a line-of-sight direction estimation device and estimation method for estimating the line-of-sight direction of a person shown in a two-dimensional image. The line-of-sight direction estimation device performs estimation processing for estimating the line-of-sight direction of a person shown in a two-dimensional image. In the estimation process, three-dimensional posture information of a subject person is acquired from a two-dimensional image of the subject person in which estimation in the line-of-sight direction is reflected. In the estimation process, three-dimensional position information of the object reflected in the two-dimensional image is acquired from the two-dimensional image. In the estimation process, the input information is input to a neural network model that outputs the direction of the line of sight of the person, and output information of the neural network model is acquired as the direction of the line of sight of the subject person. The input information of the neural network includes the three-dimensional posture information and the three-dimensional position information acquired in the estimation process.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an apparatus and a method of inferring a gaze direction of a person. BACKGROUND

[0002] Japanese Patent Application Publication No. 2024-029913 discloses an apparatus that infers a gaze direction of a person appearing in a two-dimensional camera image in which a space is captured. The existing apparatus infers a three-dimensional pose and three-dimensional coordinates of the person including a face of the person appearing in the two-dimensional camera image. In addition, the existing apparatus refers to the inferred three-dimensional pose and three-dimensional coordinates to infer a three-dimensional pose and three-dimensional coordinates of a head of the person appearing in the two-dimensional camera image, and infers a front direction of a face region of the person as the gaze direction.

[0003] As documents indicating the technical level of the technical field associated with the present disclosure, in addition to Japanese Patent Application Publication No. 2024-029913, Japanese Patent Application Publication No. 2007-006427 and Japanese Patent Application Publication No. 2020-027390 can be exemplified.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2024-029913

[0005] Patent Document 2: Japanese Patent Application Publication No. 2007-006427

[0006] Patent Document 3: Japanese Patent Application Publication No. 2020-027390

[0007] However, in a case where the accuracy of the inference of the three-dimensional pose and three-dimensional coordinates of the head of the person appearing in the two-dimensional camera image is low, it is expected that the accuracy of the inference of the front direction of the face region of the person also becomes low. In this way, the gaze direction of the person appearing in the two-dimensional camera image cannot be accurately inferred, and it is difficult to infer what the person is looking at. Therefore, with respect to the technique of inferring the gaze direction of the person appearing in the two-dimensional camera image, it is desirable to make various studies and improvements. SUMMARY

[0008] An object of the present disclosure is to provide a technique capable of inferring a gaze direction of a person appearing in a two-dimensional image.

[0009] A first aspect of the present disclosure is an apparatus that infers a gaze direction of a person appearing in a two-dimensional image, having the following features.

[0010] The above-described apparatus includes a storage apparatus and a processing circuit. The above-described storage apparatus stores a two-dimensional image of a target person in which the inference of the above-described gaze direction is reflected, and a neural network model that outputs a gaze direction of a person. The above-described processing circuit is configured to perform inference processing that infers the gaze direction of the above-described target person.

[0011] The inference processing includes: acquiring three-dimensional posture information of the object person from the two-dimensional image; acquiring three-dimensional position information of an object appearing in the two-dimensional image from the two-dimensional image; and inputting input information to the neural network model to acquire output information of the neural network model as the line-of-sight direction of the object person.

[0012] The input information of the neural network model includes the three-dimensional posture information and the three-dimensional position information.

[0013] A second aspect of the present disclosure is a method of causing a computer to perform inference processing of inferring a line-of-sight direction of a person appearing in a two-dimensional image, and has the following features.

[0014] The inference processing includes: acquiring three-dimensional posture information of the object person from the two-dimensional image; acquiring three-dimensional position information of an object appearing in the two-dimensional image from the two-dimensional image; and inputting input information to the neural network model to acquire output information of the neural network model as the line-of-sight direction of the object person.

[0015] The input information of the neural network model includes the three-dimensional posture information and the three-dimensional position information.

[0016] According to the present disclosure, by inputting three-dimensional posture information of an object person and three-dimensional position information of an object acquired from a two-dimensional image as input information of a neural network model that outputs a line-of-sight direction of a person, it is possible to acquire output information of the neural network model as the line-of-sight direction of the object person. That is, according to the present disclosure, it is possible to infer a line-of-sight direction of a person appearing in a two-dimensional image. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a block diagram showing a configuration example of an inference device according to an embodiment of the present disclosure.

[0018] Figure 2 is a block diagram showing a first functional configuration example of the inference device shown in Figure 1

[0019] Figure 3 is a block diagram showing a second functional configuration example of the inference device shown in Figure 1

[0020] REFERENCE NUMERALS

[0021] ​​10 … data processing device; 11 … processing circuit; 12 … storage device; 13, 17 … three-dimensional posture inference unit; 14, 18 … three-dimensional position inference unit; 15, 19 … line-of-sight direction calculation unit; 16 … face direction inference unit; TG … target person; IMG … two-dimensional image; NNM, NNM1, NNM2 … neural network model; CD_OB … three-dimensional position; FD_TG … face direction; GD_TG … line-of-sight direction; PS_TG … three-dimensional posture; IMG_TGF … face image. DETAILED DESCRIPTION

[0022] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In each drawing, the same reference signs are attached to the same or equivalent portions, and the description thereof is simplified or omitted.

[0023] 1. Configuration example of inference device

[0024] Figure 1 is a block diagram showing a configuration example of an inference device to which the embodiment of the present disclosure relates. In Figure 1 , a data processing device 10 and a display device 20 are depicted as a configuration of an inference device to which the embodiment relates. The display device communicates with the data processing device 10 via a communication line network (not shown). Furthermore, the communication line network is not particularly limited, and a wired and wireless network can be used.

[0025] The data processing device 10 has at least one processing circuit 11 and at least one storage device 12. As the processing circuit 11, a general-purpose processor, a special-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or the like can be exemplified. As the storage device 12, a hard disk drive (HDD), a solid state drive (SSD), a volatile memory, a non-volatile memory, or the like can be exemplified.

[0026] The processing circuit 11 expands various programs stored in the storage device 12 and performs processing of various data stored in the storage device 12. The data processing involved by the processing circuit 11 includes processing of the two-dimensional image IMG. The processing of the two-dimensional image IMG includes an estimation process of a gaze direction GD of a person appearing in the two-dimensional image IMG. The person who is the object of estimation of the gaze direction GD (hereinafter, also referred to as "target person TG") is an arbitrary person appearing in the two-dimensional image IMG and is set by the data processing apparatus 10. In a case where the data processing apparatus 10 receives input information that specifies the target person TG, the target person TG can be set in accordance with the input information.

[0027] Here, as the two-dimensional image IMG, a two-dimensional image obtained from an RGB camera can be exemplified. The two-dimensional image IMG can be a two-dimensional image in which a plurality of images obtained from an RGB camera at different times are synthesized, or a two-dimensional image in which a plurality of images (for example, a person image and a background image) obtained from a plurality of RGB cameras are synthesized.

[0028] In the embodiment, a two-dimensional image of one shot is considered as the two-dimensional image IMG. This is because a case where information for estimating the gaze direction GD is only a two-dimensional image of one shot can be assumed. In a case where an RGB camera obtains a video, an arbitrary one of time-series images that constitute the video corresponds to a two-dimensional image of one shot. In a case where the two-dimensional image IMG is an image in which a plurality of images are synthesized, the synthesized image corresponds to a two-dimensional image of one shot.

[0029] In the estimation process of the gaze direction GD, a neural network (NN) model NNM stored in the storage device 12 is used. The neural network model NNM is constructed so as to output the gaze direction GD. As the neural network model NNM that outputs the gaze direction GD, a convolutional neural network (CNN) can be exemplified.

[0030] For example, the neural network model NNM is trained by supervised learning using training data including correct data. For example, training of the neural network model NNM is performed using an equation (1) that represents a relationship between an input x i and an output y i (i = 1,..., N. N ≥ 2).

[0031] y → i = f(Θ → , x → i ) ··· (1)

[0032] In the equation (1), the superscript arrow indicates that it is a vector set. The input x→ i The two-dimensional image IMG is a set of images of a lateral width W, a longitudinal width T, expressed by a vector of W x T. The output y → i The two-dimensional image IMG is a set of images of a lateral width W, a longitudinal width T, expressed by a vector of W x T. The output y → i is a probability value. The output y → i The gaze direction Θ → is a function of a neural network model NNM that holds a set of parameters Θ → , and outputs a two-dimensional vector.

[0033] The display device 20 displays various data. As the display device 20, a liquid crystal display, an organic EL display, a head-up display, or the like can be exemplified. The various data displayed on the display device 20 is provided to a user of the inference device according to the embodiment. The various data displayed on the display device 20 includes the gaze direction GD. At the time of display of the gaze direction GD, the data processing device 10 can generate a composite image in which an arrow indicating the gaze direction GD is overlaid on the two-dimensional image IMG on which the inference of the gaze direction GD is based. The data processing device 10 can also infer an object located in front of the gaze direction GD, that is, an object that is the object of attention of the target person TG, and display information of the object of attention on the display device 20.

[0034] 2. Functional configuration example

[0035] 2-1. First configuration example

[0036] Figure 2 is a block diagram illustrating a first example of a functional configuration of the data processing device 10 shown in Figure 1 . In the example shown in Figure 2 , as functional modules of the data processing device 10, a three-dimensional pose inference unit 13, a three-dimensional position inference unit 14, and a gaze direction calculation unit 15 are depicted. These functional modules are realized by, for example, cooperation of the processing circuit 11 and the storage device 12.

[0037] A three-dimensional posture inference unit (3D posture inference unit) 13 performs a process of inferring a three-dimensional posture 3DPS of a person appearing in the two-dimensional image IMG (3DPS inference process). In the 3DPS inference process, for example, a bounding box is given to a person (target person TG) appearing in the two-dimensional image IMG (RGB image). Also, a key point of the person (target person TG) is extracted from the bounding box to infer the three-dimensional posture of the person. The three-dimensional posture is expressed with parts such as joints, head, hands, feet, and lines connecting the parts. The position of each part is expressed by a three-dimensional coordinate system (X, Y, Z). Furthermore, such an inference process is a known technique, and the method is not particularly limited. For example, MeTRAbs, TransPose, and the like are used for the 3DPS inference process. The three-dimensional posture PS_TG of the target person TG is sent to the line-of-sight direction calculation unit 15.

[0038] A three-dimensional position inference unit (3D position inference unit) 14 performs a process of inferring a three-dimensional position (3D coordinate) of an object appearing in the two-dimensional image IMG (3DCD inference process). In the 3DCD inference process, for example, a one-shot (YOLO, You Only Look Once) network, a single-stage multibox detector (SSD, Single Shot multibox Detector) network, or the like is used to detect an object OB appearing in the two-dimensional image IMG. The object OB to be detected is, for example, a static object such as a building, a structure, a natural object, a person (a person other than the target person TG), a robot, a bicycle, a car, or the like. Information of the detected object OB includes information of a two-dimensional position of the object OB in the two-dimensional image IMG. The two-dimensional position of the object is expressed by a two-dimensional coordinate system (X, Y).

[0039] In the 3DCD inference process, in addition, a depth image is generated from the two-dimensional image IMG (RGB image). The generation of the depth image can be performed using a known machine learning model. In the 3DCD inference process, also, depth information (that is, distance information from the camera to the object) of the object OB appearing in the two-dimensional image IMG is obtained from the depth image and is added to the information of the two-dimensional position. Thereby, data of a three-dimensional position CD_OB of the object OB is generated. The three-dimensional position of the object OB is expressed by a three-dimensional coordinate system (X, Y, Z). In this case, in a case where the camera that obtains the two-dimensional image IMG is a camera capable of obtaining a depth image (for example, an RGB-D camera), the three-dimensional position CD_OB can be generated using a depth image obtained at the same time as the two-dimensional image IMG. The data of the three-dimensional position CD_OB is sent to the line-of-sight direction calculation unit 15.

[0040] The gaze direction calculation unit 15 performs processing (GD calculation processing) to calculate the gaze direction GD (hereinafter also referred to as "gaze direction GD_TG") of the object character TG. In the GD calculation processing, a neural network model NNM1 is used. The inputs to the neural network model NNM1 include, in addition to the two-dimensional image IMG, the three-dimensional pose PS_TG of the object character TG received from the three-dimensional pose inference unit 13, and the three-dimensional position CD_OB of the object OB received from the three-dimensional position inference unit 14. That is, the input variables of the neural network model NNM1 are the two-dimensional image IMG, the three-dimensional pose PS_TG, and the three-dimensional position CD_OB. The gaze direction GD_TG can be obtained as the output information of the neural network model NNM1.

[0041] 2-2. Example of the second composition

[0042] Figure 3 It means Figure 1 The block diagram shows a second example of the functional configuration of the data processing device 10. Figure 3 In the example shown, the face direction inference unit 16, the three-dimensional pose inference unit 17, the three-dimensional position inference unit 18, and the gaze direction calculation unit 19 are depicted as functional modules of the data processing device 10. For example, these functional modules are implemented through the cooperation of the processing circuit 11 and the storage device 12.

[0043] The face direction inference unit 16 performs face direction inference processing (FD inference processing) on ​​the subject TG. In the FD inference processing, for example, a face image IMG_TGF of the subject TG is extracted from a two-dimensional image IMG (RGB image). Furthermore, a depth image generated based on this face image IMG_TGF is used to infer the frontal position of the subject TG's face. The face direction FD_TG is inferred based on this frontal position. The face direction FD_TG is represented by a three-dimensional vector. The face direction FD_TG data is sent to the gaze direction calculation unit 15.

[0044] In the second example of FD inference processing, a depth image is generated from the two-dimensional image IMG (RGB image) without extracting the face image IMG_TGF. Furthermore, the face orientation FD_TG is inferred based on the frontal position of the subject's face TG inferred from this depth image. In the third example of FD inference processing, the three-dimensional pose of the subject TG, which is mapped from the two-dimensional image IMG, is inferred. An example of the method for inferring the three-dimensional pose of the subject TG can be found in... Figure 2 The method described in the three-dimensional pose inference unit 13 involves inferring the face direction FD_TG based on the three-dimensional pose of the subject's TG. In the second or third example, the face direction FD_TG data is also sent to the gaze direction calculation unit 15.

[0045] The function of the three-dimensional pose inference unit (3D pose inference unit) 17 is the same as that of the three-dimensional pose inference unit 13 described in Figure 2 Figure 2 The function of the three-dimensional position inference unit (3D position inference unit) 18 is the same as that of the three-dimensional position inference unit 14 described in

[0046] The line-of-sight direction calculation unit 19 performs processing for calculating the line-of-sight direction GD_TG (GD calculation processing). In the GD calculation processing, the neural network model NNM2 is used. In the input of the neural network model NNM2, in addition to the two-dimensional image IMG, the face direction FD_TG received from the face direction inference unit 16, the three-dimensional pose PS_TG of the target person TG received from the three-dimensional pose inference unit 17, and the three-dimensional position CD_OB of the object OB received from the three-dimensional position inference unit 18 are used. That is, the input variables of the neural network model NNM2 are the two-dimensional image IMG, the face direction FD_TG, the three-dimensional pose PS_TG, and the three-dimensional position CD_OB. As the output information of the neural network model NNM2, the line-of-sight direction GD_TG can be obtained.

[0047] 3. Effects

[0048] According to the embodiment, by inputting the three-dimensional pose PS_TG of the target person TG and the three-dimensional position CD_OB of the object OB to the neural network model NNM that maps a two-dimensional image of an existing target person, a three-dimensional pose of the target person, and a three-dimensional position of an object appearing in the two-dimensional image as input variables, and a line-of-sight direction of the target person as output, together with the two-dimensional image IMG, as output information, the line-of-sight direction GD_TG can be obtained. Alternatively, by adding the face direction FD_TG of the target person TG to the input variables, as output information, the line-of-sight direction GD_TG can be obtained. That is, according to the embodiment, the line-of-sight direction GD_TG of the target person TG appearing in the two-dimensional image IMG can be inferred. Even if the two-dimensional image IMG is an image of single auto focus, the line-of-sight direction GD_TG of the target person TG appearing in the two-dimensional image IMG can be inferred.​

Claims

1.An apparatus for estimating a gaze direction of a person appearing in a two-dimensional image, the apparatus comprising: a storage device that stores a two-dimensional image of a target person for which the gaze direction is to be estimated, and a neural network model that outputs a gaze direction of a person; and a processing circuitry that performs an estimation process for estimating the gaze direction of the target person, the estimation process including: acquiring three-dimensional posture information of the target person from the two-dimensional image; acquiring three-dimensional position information of an object appearing in the two-dimensional image from the two-dimensional image; and inputting input information to the neural network model to acquire output information of the neural network model as the gaze direction of the target person, the input information of the neural network model including the three-dimensional posture information and the three-dimensional position information. 2.The apparatus for estimating a gaze direction according to claim 1, wherein the estimation process further includes a process of acquiring face direction information of the target person appearing in the two-dimensional image, and the input information of the neural network model further includes the face direction information. 3.The apparatus for estimating a gaze direction according to claim 1 or 2, wherein the two-dimensional image includes a single auto-focusing two-dimensional image. 4.A method for causing a computer to perform an estimation process for estimating a gaze direction of a person appearing in a two-dimensional image, the estimation process including: acquiring three-dimensional posture information of a target person for which the gaze direction is to be estimated from a two-dimensional image of the target person; acquiring three-dimensional position information of an object appearing in the two-dimensional image from the two-dimensional image; and inputting input information to a neural network model that outputs a gaze direction of a person to acquire output information of the neural network model as the gaze direction of the target person, the input information of the neural network model including the three-dimensional posture information and the three-dimensional position information. 5.The method for estimating a gaze direction according to claim 4, wherein the estimation process further includes a process of acquiring face direction information of the target person appearing in the two-dimensional image, and the input information of the neural network model further includes the face direction information. 6.The method for estimating a gaze direction according to claim 4 or 5, wherein the two-dimensional image includes a single auto-focusing two-dimensional image. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

Images