Learning device, estimation device, their methods, and programs

Abstract

Provided are an estimation device, etc., with which it is possible to estimate an angle of sight with high accuracy and assesses an effective visual field capturing the features of both the angle of sight and effective visual field assessment. This training device converts a face image for training to an intermediate feature by a neural network function, using a model parameter ^θg or ^θp, converts the intermediate feature to an estimate sight angle vector by a neural network function, using the model parameter ^θg, updates the model parameter ^θg, using the estimate sight angle vector and a sight angle correct-answer label for the face image for training, converts an intermediate feature v to an estimate effective visual field probability vector by a neural network function, using the model parameter ^θp, and updates the model parameter ^θp, using the estimate effective visual field probability vector and an effective visual field correct-answer label for the face image for training.

Description

[Technical field]

[0001] The present invention relates to a technique for estimating a gaze angle and a useful visual field from a face image, and a technique for learning parameters used in the estimation. [Background technology]

[0002] The gaze angle is a rotation angle that indicates the direction of the pupils of the left and right eyes. If the gaze angle can be estimated from an image, it is possible to understand the state of the eyeball movement, such as what a person is gazing at or looking around, and to analyze the person's state and inner understanding. Gaze angle estimation is generally performed using a neural network. In conventional technology, only the gaze angle is learned by regression from the eye area image using a neural network model. The correct answer label of the gaze angle is vector data, which is continuous data of the rotation angle values ​​of the horizontal and vertical components of the eyeball. In learning the parameters used for gaze angle estimation using a neural network, for example, the features of the image are extracted using a convolution layer and a pooling layer that are widely used in image recognition, and the gaze angle can be estimated by regressing the gaze angle to vector data using a fully connected layer.

[0003] Non-Patent Document 1 describes a method for estimating the gaze angle using a neural network.

[0004] On the other hand, there is a method that uses the effective visual field as a measure to determine whether a person recognizes an object. The effective visual field refers to the range of horizontal angles of the eyeball from -15 degrees to 15 degrees and vertical angles from -12 degrees to 8 degrees (see Figure 1), and humans can clearly recognize objects within this range. Conversely, the ability to recognize objects outside the effective visual field drops drastically. For example, in a use case where the object is a robot, the effective visual field scale can be used to determine whether a robot equipped with a camera is recognized by a human captured on the camera. Specifically, even if a human is facing the robot, if the gaze angle is outside the effective visual field, it can be determined that the robot is not recognized, and if it is within the effective visual field, it can be determined that the robot is recognized. Figure 1 is a diagram showing the visual field and visual field characteristics of a human. Figure 2 is a diagram showing the inside of the effective visual field (the range where the presence of an object can be recognized) and the outside of the effective visual field (the range where the presence of an object cannot be recognized). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling, "Appearance-Based Gaze Estimation in the Wild", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015 Summary of the Invention [Problem to be solved by the invention]

[0006] In the learning of model parameters used for conventional gaze angle estimation, a method is used in which only the gaze angle is regressed using a neural network, and whether the gaze angle is within or outside the effective visual field is determined by a rule judgment from the estimated gaze angle. In addition, the conventional technology only minimizes the error of the gaze angle, and performs learning without considering whether it is within or outside the effective visual field. The tendency of gaze angle estimation error is that the larger the absolute value of the correct angle, the larger the error becomes, and there is a tendency for angle errors to become large around horizontal angles of -15 degrees and 15 degrees (areas surrounded by dashed lines in Figure 3), which are the boundaries of the effective visual field, and around vertical angles of -12 degrees and 8 degrees. For this reason, in the method of judging the effective visual field using rules from the estimated gaze angle, there is a problem in that the accuracy of judging the effective visual field decreases near the boundaries of the effective visual field.

[0007] An object of the present invention is to provide an estimation device, a learning device, and methods and programs thereof that are capable of performing highly accurate gaze angle estimation and effective visual field judgment that captures the characteristics of both the gaze angle and effective visual field judgment. [Means for solving the problem]

[0008] In order to solve the above problem, according to one aspect of the present invention, a learning device is provided that calculates a model parameter ^θ g or ^θ p Using the neural network function, the training face image S b The shared network part converts the intermediate features v into the model parameters ^θ g Using the neural network function, estimate the intermediate feature v and the gaze angle vector Z' g and the gaze angle network part that converts the gaze angle vector Z' g and training face image S b Using the gaze angle correct label for g A gaze angle model parameter optimization part that updates the gaze angle model parameter ^θ p Using the neural network function, estimate the intermediate feature v and the effective visual field probability vector Z' p and the estimated effective visual field probability vector Z'p and training face image S b Using the correct label of the effective field of view for p and an effective visual field model parameter optimization unit that updates the model parameter ^θ g The trained model parameters θ g and the model parameters ^θ p The trained model parameters θ p Get the. Effect of the Invention

[0009] According to the present invention, it is possible to perform highly accurate gaze angle estimation and effective visual field determination that captures the characteristics of both the gaze angle and the effective visual field determination. [Brief description of the drawings]

[0010] [Figure 1] A diagram showing human visual field and visual field characteristics. [Diagram 2] FIG. [Diagram 3] FIG. 4 is a diagram for explaining an estimated angle error. [Figure 4] FIG. 1 is a diagram showing an example of the configuration of an estimation system according to a first embodiment. [Diagram 5] FIG. 2 is a functional block diagram of the learning device according to the first embodiment. [Figure 6] FIG. 4 is a diagram showing an example of a processing flow of the learning device according to the first embodiment. [Figure 7] FIG. 1 is a functional block diagram of an estimation device according to a first embodiment. [Figure 8] FIG. 4 is a diagram showing an example of a processing flow of the estimation device according to the first embodiment. [Figure 9] FIG. 13 is a diagram showing an example of the configuration of a computer to which the present technique is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described. In the drawings used in the following description, components having the same functions and steps performing the same processing are denoted with the same reference numerals, and duplicated description will be omitted. In the following description, symbols such as "^" used in the text should be written directly above the character immediately preceding it, but due to limitations in text notation, they are written immediately after the character in question. In the formulas, these symbols are written in their original positions. Furthermore, unless otherwise specified, processing performed on each element of a vector or matrix is ​​assumed to be applied to all elements of that vector or matrix.

[0012] <Key Points of the First Embodiment> In this embodiment, model parameters used to estimate the gaze angle and model parameters used to determine the effective visual field are simultaneously learned by a learning device. The learning data includes face images for learning and the correct label Y g and the correct label for the effective visual field Y p The neural network is composed of a shared network section, a gaze angle network section and an effective visual field network section that branch out from the shared network section. The shared network section takes a face image for learning as input, and is expected to learn to acquire features of both gaze angle and effective visual field. The gaze angle network section in the latter section regresses the gaze angle, and the effective visual field network section calculates the probability within the effective visual field, and minimizes the error of the estimated value using each correct answer label, and updates the model parameters. During estimation, a gaze angle estimation section and an effective visual field determination section are used, respectively. The gaze angle estimation section estimates the gaze angle using the network architecture of the shared network section and the subsequent gaze angle network section, and the model parameters learned by the learning device. Similarly, the effective visual field determination section determines the effective visual field using the network architecture of the shared network section and the subsequent effective visual field network section, and the model parameters learned by the learning device.

[0013] First Embodiment FIG. 4 shows an example of the configuration of an estimation system according to the first embodiment.

[0014] The estimation system includes a learning device 100 and an estimation device 200.

[0015] The learning device 100 and the estimation device 200 are special devices configured by loading a special program into a publicly known or dedicated computer having, for example, a central processing unit (CPU), a main storage device (RAM), and the like. The learning device 100 and the estimation device 200 execute each process under the control of, for example, the central processing unit. Data input to the learning device 100 and the estimation device 200 and data obtained in each process are stored in, for example, the main storage device, and the data stored in the main storage device is read out to the central processing device as necessary and used for other processes. At least a part of each processing unit of the learning device 100 and the estimation device 200 may be configured by hardware such as an integrated circuit. Each storage unit included in the learning device 100 and the estimation device 200 can be configured by, for example, a main storage device such as a random access memory (RAM), or middleware such as a relational database or a key-value store. However, each storage unit does not necessarily need to be provided inside the learning device 100 and the estimation device 200, and may be configured as an auxiliary storage device formed of a hard disk, an optical disk, or a semiconductor memory element such as a flash memory, and may be configured to be provided outside the learning device 100 and the estimation device 200.

[0016] First, the learning device 100 will be described.

[0017] <Learning device 100> The learning device 100 uses learning data D=(S1,T g 1,T p 1),…,(S |M| ,T g |M| ,T p |M| ) is input, and model parameters θ g ,θ pand the trained model parameters θ g ,θ p Here, M is the data size of the training data D. m (m=1,...,M) are training face images. The training face images may be, for example, either images with only the face cut out or images with only the eye region cut out. For example, the training face images have a resolution of 224x224 pixels and three RGB channels. g m (m=1,...,M) is the correct label of the gaze angle (hereinafter also referred to as the "gaze angle correct label"). For example, it is vector data that stores the horizontal and vertical rotation angles of the gaze angle, and has a data format such as [-35,56]. p m (m=1,…,M) is the correct label of the effective visual field (hereinafter also referred to as the “effective visual field correct label”). For example, if it is within the effective visual field, it is [1], and if it is outside the effective visual field, it is [0].

[0018] FIG. 5 shows a functional block diagram of the learning device 100, and FIG. 6 shows the processing flow thereof.

[0019] The learning device 100 includes a shared network unit 120, a gaze angle network unit 130, a gaze angle model parameter optimization unit 140, an effective visual field network unit 150, and an effective visual field model parameter optimization unit 160.

[0020] An overview of each part will be explained below.

[0021] The shared network unit 120 receives a learning face image as an input, learns the features of both the gaze angle and the effective visual field, and acquires intermediate features using an arbitrary neural network that outputs intermediate features.

[0022] The gaze angle network unit 130 obtains an estimate of the gaze angle using any neural network that receives the intermediate features as input and outputs an estimate of the gaze angle.

[0023] The gaze angle model parameter optimization unit 140 receives the gaze angle estimate and the gaze angle correct label as input, calculates the error of the gaze angle estimate, and optimizes the model parameter θ g Update.

[0024] The visual field network unit 150 acquires the estimated visual field probability using any neural network that receives the intermediate features as input and outputs the probability that the intermediate features are within the visual field (estimated visual field probability).

[0025] The visual field model parameter optimization unit 160 receives the estimated visual field probability and the correct label of the visual field, calculates the error of the estimated visual field probability, and optimizes the model parameter θ p Update.

[0026] The above process shows the learning procedure for one batch (a block of data selected from the learning data), and this can be repeated any number of times to learn all the data.

[0027] The details of each part will be explained below.

[0028] <Shared network section 120> Input: Training face images S1, …, S |M| , the updated model parameters ^θ g or ^θ p Among these, parameters corresponding to the neural network constituting the shared network unit 120 are Output: 1 batch of intermediate features v The shared network unit 120 is configured with an arbitrary neural network, for example, with four convolution layers.

[0029] Prior to the conversion process, the shared network unit 120 updates the model parameters ^θ g (output value of the line of sight angle model parameter optimization unit 140) or ^θ pAmong the output values ​​of the visual field model parameter optimization unit 160, parameters corresponding to any neural network constituting the shared network unit 120 are received.

[0030] The shared network unit 120 updates the model parameters ^θ g or ^θ p Using parameters corresponding to any neural network constituting the shared network unit 120, a face image for learning S is generated by a function of any neural network. b is converted into intermediate features v (S120). b are the training face images S1,…,S |M| A batch of training face images S b For example, a batch of training face images S b If v consists of Q images, the intermediate feature corresponding to the q-th image is v q Let v = [v1, v2, …, v Q ], where q = 1, 2, …, Q.

[0031] <Line of Sight Angle Network Part 130> Input: 1 batch of intermediate features v, updated model parameters ^θ g Among these, the parameters corresponding to the neural network constituting the line of sight angle network unit 130 are Output: Estimated line of sight angle vector Z' g The gaze angle network unit 130 is configured with any neural network, for example, with two fully connected layers.

[0032] Prior to the conversion process, the line of sight angle network unit 130 updates the model parameters ^θ g Among the (output values ​​of the gaze angle model parameter optimization unit 140), parameters corresponding to any neural network constituting the gaze angle network unit 130 are received.

[0033] The gaze angle network unit 130 uses the received parameters to estimate the intermediate feature v by a function of an arbitrary neural network, and generates a gaze angle vector Z' g (S130). The estimated gaze angle vector is vector data storing the estimated values ​​of the horizontal rotation angle and vertical rotation angle of the gaze angle. In other words, the estimated gaze angle vector is an estimate of the gaze angle vector. The gaze angle vector is vector data storing the horizontal rotation angle and vertical rotation angle of the gaze angle. The rotation angle ranges from -180 degrees to 180 degrees, and for example, the gaze angle vector is a vector such as [-35, 56]. For example, the qth intermediate feature v of a certain batch is q Let Z' be the estimated line of sight angle vector corresponding to g,q Let Z' g =[Z' g,1 ,Z' g,2 ,…,Z' g,Q ]. Estimated line of sight angle vector Z' g,q will result in a vector such as [-35,56].

[0034] <Gaze angle model parameter optimization unit 140> Input: Estimated line of sight angle vector Z' g , gaze angle correct label T g 1,…,T g |M| Output: Updated model parameters ^θ g or the trained model parameters θ g Model parameters ^θ g and θ g is a combination of parameters corresponding to the neural network constituting shared network unit 120 and parameters corresponding to the neural network constituting line of sight angle network unit 130.

[0035] The gaze angle model parameter optimization unit 140 calculates the estimated gaze angle vector Z' g and gaze angle correct label T g 1,…,T g |M|Using the model parameters ^θ g For example, the gaze angle model parameter optimization unit 140 updates the estimated gaze angle vector Z' g and gaze angle correct label T g 1,…,T g |M| Calculate the error between and and set the model parameters ^θ to minimize the error. g The error can be, for example, the MSE error or the MAE error, and the parameter can be updated using the gradient descent method or the like.

[0036] <Effective Field of View Network Section 150> Input: 1 batch of intermediate features v, updated model parameters ^θ p Among these, the parameters corresponding to the neural network constituting the effective visual field network section 150 are Output: Estimated effective field probability vector Z' p The visual field network unit 150 is configured with any neural network, for example, with two fully connected layers.

[0037] Prior to the conversion process, the effective visual field network unit 150 updates the model parameters ^θ p Among the output values ​​of the effective visual field model parameter optimization unit 160, parameters corresponding to any neural network constituting the effective visual field network unit 150 are received.

[0038] The effective visual field network unit 150 uses the received parameters to estimate a batch of intermediate features v by a function of an arbitrary neural network, and generates an effective visual field probability vector Z' p(S150). The estimated effective visual field probability vector is a vector consisting of estimated values ​​of the effective visual field probability (estimated effective visual field probability), and the effective visual field probability is the probability of whether the gaze direction of the training face image is within the effective visual field. In other words, the effective visual field probability is the probability of whether the camera capturing the training face image is within the effective visual field of the subject of the training face image. For example, the q-th intermediate feature v q Let Z' be the estimated effective field probability corresponding to p,q Let Z' p =[Z' p,1 ,Z' p,2 ,…,Z' g,Q ].

[0039] <Effective visual field model parameter optimization unit 160> Input: Estimated effective field probability vector Z' p , the correct label of the effective visual field T p 1,…,T p |M| Output: Updated model parameters ^θ p or the trained model parameters θ p Model parameters ^θ p and θ p is a combination of parameters corresponding to the neural network constituting the shared network section 120 and parameters corresponding to the neural network constituting the visual field network section 150.

[0040] The effective visual field model parameter optimization unit 160 calculates the estimated effective visual field probability vector Z' p and the correct label T p 1,…,T p |M| Using the model parameters ^θ p For example, the effective visual field model parameter optimization unit 160 updates the estimated effective visual field probability vector Z' p and the correct label T p 1,…,T p |M| Calculate the error between and and set the model parameters ^θ to minimize the error.p The error can be, for example, a cross-entropy error, and the parameter can be updated using a gradient descent method or the like.

[0041] The above-mentioned processes S120 to S160 are repeated until a predetermined condition is satisfied (S170). The predetermined condition is a condition for determining whether the parameter update has converged or not, and may be, for example, (i) the number of updates has exceeded a predetermined number, or (ii) the difference between the parameters before and after the update is smaller than a predetermined value.

[0042] Furthermore, the above-mentioned processes S120 to S170 are performed on all the batch data (learning data). For example, it is determined whether or not there is unprocessed batch data (S180), and if there is unprocessed batch data, the above-mentioned processes S120 to S170 are performed (NO in S180), and if there is no unprocessed batch data (YES in S180), the process ends.

[0043] After performing the above process on all the batch data, the final updated model parameters ^θ g , ^θ p The trained model parameters θ g , θ p The output is as follows:

[0044] Next, the estimation device 200 will be described.

[0045] <Estimation device 200> Prior to the estimation process, the estimation device 200 calculates the learned model parameters θ g ,θ p The estimation device 200 receives a face image S of an estimation target as input and receives a trained model parameter θ g The gaze angle is estimated using the trained model parameters θ p The effective visual field probability is estimated using the estimated line of sight angle vector Z g , Effective visual field judgment result Z p Output.

[0046] FIG. 7 shows a functional block diagram of the estimation device 200, and FIG. 8 shows the process flow thereof.

[0047] The estimation device 200 includes a gaze angle estimation unit 210 and an effective visual field determination unit 220.

[0048] The details of each part will be explained below.

[0049] <Gaze angle estimation unit 210> Input: Face image S, model parameters θ g Output: Estimated gaze angle vector Z for face image S g Prior to the estimation process, the gaze angle estimation unit 210 calculates the model parameter θ g Receive.

[0050] The gaze angle estimation unit 210 estimates the network architecture of the shared network unit 120 and the subsequent gaze angle network unit 130, and the model parameters θ g Using the above, the gaze angle is estimated from the face image S (S210), and the estimated value (estimated gaze angle vector Z g For example, when the shared network unit 120 is configured with an arbitrary neural network having four convolution layers, and the gaze angle network unit 130 is configured with an arbitrary neural network having two fully connected layers, the gaze angle estimation unit 210 is configured with a neural network having four convolution layers and two fully connected layers corresponding to the shared network unit 120 and the gaze angle network unit 130, and the model parameter θ g is used.

[0051] <Effective visual field determination unit 220> Input: Face image S, model parameters θ p Output: Judgment result Z of the useful field of view for face image S p Prior to the estimation process, the effective visual field determination unit 220 determines the model parameter θ p Receive.

[0052] The effective visual field determination unit 220 determines the network architecture of the shared network unit 120 and the subsequent effective visual field network unit 150, and the model parameters θ p Using the above, the effective visual field probability is estimated from the face image S, and it is determined whether or not the face is within the effective visual field based on the estimated value of the effective visual field probability (S220). For example, the effective visual field determination unit 220 determines whether or not the face is within the effective visual field based on the magnitude relationship between the estimated value of the effective visual field probability and a predetermined threshold. For example, the effective visual field determination unit 220 outputs a determination result indicating that the face is within the effective visual field when the estimated value of the effective visual field probability is equal to or greater than a predetermined threshold (e.g., 0.5), and outputs a determination result indicating that the face is outside the effective visual field when the estimated value of the effective visual field probability is less than the predetermined threshold. However, the effective visual field determination unit 220 may output the estimated value of the effective visual field probability itself as the determination result. For example, when the shared network unit 120 is configured with an arbitrary neural network consisting of four convolution layers, and the effective visual field network unit 150 is configured with an arbitrary neural network consisting of two fully connected layers, the effective visual field determination unit 220 is configured with a neural network consisting of four convolution layers and two fully connected layers corresponding to the shared network unit 120 and the effective visual field network unit 150, and the model parameter θ p is used.

[0053] <Effects> The above configuration enables highly accurate gaze angle estimation and effective visual field determination that captures the characteristics of both the gaze angle and the effective visual field determination.

[0054] <Modification> In this embodiment, the estimation device 200 estimates the gaze angle and the effective visual field probability, but it may be configured to estimate only one of them. Even in this case, during learning, the model parameters used to estimate the gaze angle and the effective visual field are learned in a single neural network system, so that it is possible to perform a highly accurate gaze angle estimation or effective visual field judgment that captures the characteristics of both the gaze angle and the effective visual field judgment.

[0055] <Effects> With the above configuration, it is possible to perform highly accurate gaze angle estimation and effective visual field determination that captures the characteristics of both the gaze angle and the effective visual field determination.

[0056] <Other Modifications> The present invention is not limited to the above-mentioned embodiment and modified examples. For example, the above-mentioned various processes may be executed not only in chronological order as described, but also in parallel or individually depending on the processing capacity of the device executing the processes or as necessary. In addition, appropriate modifications are possible within the scope of the present invention.

[0057] <Programs and recording media> The various processes described above can be implemented by loading a program for executing each step of the above method into the memory unit 2020 of the computer shown in FIG. 9 and operating the control unit 2010, input unit 2030, output unit 2040, etc.

[0058] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other type of recording medium.

[0059] The program may be distributed, for example, by selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of a server computer and transferring the program from the server computer to other computers via a network.

[0060] A computer that executes such a program, for example, first stores the program recorded on a portable recording medium or the program transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from the portable recording medium and execute a process according to the program, or may execute a process according to the received program each time a program is transferred from the server computer to the computer. The above-mentioned process may be executed by a so-called ASP (Application Service Provider) type service that realizes a processing function only by issuing an execution instruction and obtaining a result, without transferring a program from the server computer to the computer. Note that the program in this embodiment includes information used for processing by an electronic computer and equivalent to a program (data that is not a direct command to the computer but has a nature that specifies the processing of the computer, etc.).

[0061] In addition, in this embodiment, the present device is configured by executing a specific program on a computer, but at least a part of the processing contents may be realized by hardware.

Claims

1. Model parameters ^θ g or ^θ p Using the neural network function, the training face image S b a shared network part that converts the vectors into intermediate features v; The model parameter ^θ g Using the neural network function, the intermediate feature v is estimated as the gaze angle vector Z' g A line of sight network part converting The estimated line of sight angle vector Z' g and the learning face image S b Using the gaze angle correct label for g a gaze angle model parameter optimization unit that updates the gaze angle model parameter The model parameter ^θ p Using the neural network function, estimate the intermediate feature v and the effective visual field probability vector Z' p A network section for converting the effective visual field into The estimated effective visual field probability vector Z' p and the learning face image S b Using the effective visual field correct label for p and an effective visual field model parameter optimization unit that updates the effective visual field model parameter The model parameter ^θ g The trained model parameters θ g and the model parameters ^θ p The trained model parameters θ p Get Learning device.

2. The model parameter θ learned by the learning device of claim 1 g An estimation device using The network architecture of the shared network section and the subsequent line of sight angle network section, and the model parameters θ g and a gaze angle estimation unit that estimates a gaze angle from a face image S of an estimation target using the above. Estimation device.

3. The model parameter θ learned by the learning device of claim 1 p An estimation device using The network architecture of the shared network section and the subsequent effective visual field network section, and the model parameters θ p and an effective visual field determination unit that estimates an effective visual field probability from the face image S using the above and determines whether or not the face image S is within the effective visual field based on the estimated value of the effective visual field probability. Estimation device.

4. The model parameter θ learned by the learning device of claim 1 g and the model parameters θ p An estimation device using The network architecture of the shared network section and the subsequent line of sight angle network section, and the model parameters θ g a gaze angle estimation unit that estimates a gaze angle from a face image S of an estimation target using the above; The network architecture of the shared network section and the subsequent effective visual field network section, and the model parameters θ p and an effective visual field determination unit that estimates an effective visual field probability from the face image S using the above, and determines whether or not the face image S is within the effective visual field based on the estimated value of the effective visual field probability. Estimation device.

5. Model parameters ^θ g or ^θ p Using the neural network function, the training face image S b a shared network step to transform the vectors into intermediate features v; The model parameter ^θ g Using the neural network function, the intermediate feature v is estimated as the gaze angle vector Z' g A line of sight network step to convert The estimated line of sight angle vector Z' g and the learning face image S b Using the gaze angle correct label for g a gaze angle model parameter optimization step for updating The model parameter ^θ p Using the neural network function, estimate the intermediate feature v and the effective visual field probability vector Z' p and a network step of converting the effective field of view into The estimated effective visual field probability vector Z' p and the learning face image S b Using the effective visual field correct label for p and optimizing the effective visual field model parameters by updating The model parameter ^θ g The trained model parameters θ g and the model parameters ^θ p The trained model parameters θ p Get How to learn.

6. The model parameter θ learned by the learning method of claim 5 g An estimation method using The network architecture of the shared network step and the subsequent gaze angle network step, and the model parameters θ g and estimating a gaze angle from a face image S of an estimation target using the above. Estimation method.

7. The model parameter θ learned by the learning method of claim 5 p An estimation method using The network architecture of the shared network step and the subsequent effective field of view network step, and the model parameters θ p and estimating an effective visual field probability from the face image S using the above, and judging whether or not the face image S is within the effective visual field based on the estimated value of the effective visual field probability. Estimation method.

8. A program for causing a computer to function as the learning device of claim 1 or the estimation device of any one of claims 2 to 4.

Images