Learning device, estimation device, their methods, and programs

Abstract

Provided is a parameter learning device for use in estimating a face orientation and face key points that represent more universal face features. This learning device converts a learning face image into intermediate features via a neural network function using a model parameter ^θk or ^θp. When the learning face image consists of a face image for learning face key points, the learning device converts the intermediate features into estimated face key point vectors via a neural network function using the model parameter ^θk, and updates the model parameter ^θk using the estimated face key point vectors and face key point correct labels for the learning face image. When the learning face image consists of a face image for learning a face orientation, the learning device converts the intermediate features into an estimated face orientation vector via a neural network function using a model parameter, and updates the model parameter ^θp using the estimated face orientation vector and a face orientation correct label.

Description

[Technical field]

[0001] The present invention relates to an estimation technique for estimating face key points and face orientation from a face image, and a learning technique for parameters used in the estimation. [Background technology]

[0002] Facial keypoints are plots of eyebrows, eyes, nose, mouth, and contours of a human face as several to a dozen feature points, and are features that quantitatively capture facial expressions and facial movements. Estimating facial keypoints from facial images is important for understanding human states and inner thoughts. On the other hand, facial orientation is the rotation angle of the three axes (roll, pitch, yaw) of the face, and estimating the facial orientation from a facial image makes it possible to analyze facial movements and attention directions. Facial keypoints and facial orientations are generally estimated using neural networks. In conventional techniques, facial keypoints and facial orientations are estimated independently using independent neural network models trained only with the respective learning data (Non-Patent Documents 1 and 2). Correct labels for both facial keypoints and facial orientations are vector data. Facial keypoints are expressed as continuous data of the two-dimensional or three-dimensional coordinate values ​​of each keypoint in the image. For example, any number of keypoints, such as 21 points or 68 points, can be used. Facial orientation is continuous data of the rotation angle values ​​of the roll, pitch, and yaw components of the rotation angle. Facial keypoints and facial direction can be estimated using a neural network by extracting image features using a convolutional layer and pooling layer, which are widely used in image recognition, and then regressing the facial keypoints and facial direction into vector data using a subsequent fully connected layer.

[0003] In Non-Patent Documents 1 and 2, face keypoints and face orientation are estimated using a neural network. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Zongping Deng, Ke Li, Qijun Zhao, Yi Zhang, Hu Chen, "Effective face landmark localization via single deep network", CoRR, 2017. [Non-Patent Document 2] Nataniel Ruiz, Eunji Chong, James M. Rehg, "Fine-Grained Head Pose Estimation Without Keypoints", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2018 Summary of the Invention [Problem to be solved by the invention]

[0005] In the conventional technology, the model parameters used to estimate face key points and the model parameters used to estimate face direction are trained by independent neural networks using independent training data. On the other hand, both face key points and face direction are features that capture face movement, and if model parameters can be trained taking into account the features of both, more universal face features can be acquired, and highly accurate estimation of face key points and face direction can be expected. However, in the conventional technology, since the model parameters used to estimate face key points and the model parameters used to estimate face direction are trained independently, there is a problem that only local face features can be captured.

[0006] The present invention aims to provide an estimation device that can estimate facial keypoints and facial orientation that capture more universal facial features by learning model parameters used to estimate facial keypoints and facial orientation using a single neural network system, a learning device for parameters used in estimation, and methods and programs for carrying out the above estimation. [Means for solving the problem]

[0007] In order to solve the above problem, according to one aspect of the present invention, a learning device is provided that calculates a model parameter ^θ k or ^θ p Using the neural network function, the training face image S b The intermediate feature v b A shared network part converts the face image S b consists of face images for face keypoint training, the model parameters ^θ k Using the neural network function, the intermediate feature v b Estimate the face keypoint vector Z' k,b and the face keypoint network part converting the estimated face keypoint vector Z' k,b and training face image S b The correct face keypoint label for T k 1,…,T k |M| Using the model parameters ^θ k A face keypoint model parameter optimization unit that updates the face image S b When the face orientation training face images are composed of the model parameters ^θ p Using the neural network function, the intermediate feature v b Estimate face direction vector Z' p,b and the face orientation network part converting the estimated face orientation vector Z' p,b and face orientation correct label T p 1,…,T p |N| Using the model parameters ^θ p and a face direction model parameter optimization unit that updates the model parameter ^θ k The trained model parameters θ k and the model parameters ^θ p The trained model parameters θ p Get the. Effect of the Invention

[0008] According to the present invention, it is possible to estimate face key points and face orientation that capture more general facial features. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an estimation system according to a first embodiment. [Diagram 2] FIG. 2 is a functional block diagram of the learning device according to the first embodiment. [Diagram 3] FIG. 4 is a diagram showing an example of a processing flow of the learning device according to the first embodiment. [Figure 4] FIG. 1 is a functional block diagram of an estimation device according to a first embodiment. [Diagram 5] FIG. 4 is a diagram showing an example of a processing flow of the estimation device according to the first embodiment. [Figure 6] FIG. 13 is a diagram showing an example of the configuration of a computer to which the present technique is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described. In the drawings used in the following description, components having the same functions and steps performing the same processing are denoted with the same reference numerals, and duplicated description will be omitted. In the following description, symbols such as "^" used in the text should be written directly above the character immediately preceding it, but due to limitations in text notation, they are written immediately after the character in question. In the formulas, these symbols are written in their original positions. Furthermore, unless otherwise specified, processing performed on each element of a vector or matrix is ​​assumed to be applied to all elements of that vector or matrix.

[0011] <Key Points of the First Embodiment> In this embodiment, both the learning data of the model parameters used to estimate the face keypoints and the learning data of the model parameters used to estimate the face direction are learned by a learning device that is a single neural network system. The face images for learning the model parameters used to estimate the face keypoints and the face images for learning the model parameters used to estimate the face direction are alternately input to a shared network section in the learning device, where more universal face features acquired from both the learning face images are learned. A face keypoint network section and a face direction network section are branched out at the rear of the shared network section, and the error of the estimated value is minimized using the respective correct answer labels, and the model parameters are updated. During estimation, a face keypoint estimation section and a face direction estimation section are used, respectively. The face keypoint estimation section estimates the face keypoints using the network architecture of the shared network section and the subsequent face keypoint network section, and the model parameters learned by the learning device. Similarly, the face direction estimation section estimates the face direction using the network architecture of the shared network section and the subsequent face direction network section, and the model parameters learned by the learning device.

[0012] First Embodiment FIG. 1 shows an example of the configuration of an estimation system according to the first embodiment.

[0013] The estimation system includes a learning device 100 and an estimation device 200.

[0014] The learning device 100 and the estimation device 200 are special devices configured by loading a special program into a publicly known or dedicated computer having, for example, a central processing unit (CPU), a main storage device (RAM), and the like. The learning device 100 and the estimation device 200 execute each process under the control of, for example, the central processing unit. Data input to the learning device 100 and the estimation device 200 and data obtained in each process are stored in, for example, the main storage device, and the data stored in the main storage device is read out to the central processing device as necessary and used for other processes. At least a part of each processing unit of the learning device 100 and the estimation device 200 may be configured by hardware such as an integrated circuit. Each storage unit included in the learning device 100 and the estimation device 200 can be configured by, for example, a main storage device such as a random access memory (RAM), or middleware such as a relational database or a key-value store. However, each storage unit does not necessarily need to be provided inside the learning device 100 and the estimation device 200, and may be configured as an auxiliary storage device formed of a hard disk, an optical disk, or a semiconductor memory element such as a flash memory, and may be configured to be provided outside the learning device 100 and the estimation device 200.

[0015] First, the learning device 100 will be described.

[0016] <Learning device 100> The learning device 100 estimates the face keypoints by using model parameters θ k (Hereinafter, the face keypoint parameters θ k The training data K=(S k 1,T k 1),…,(S k |M| ,T k |M| ) and the model parameters θ p (Hereinafter, the face direction parameter θ pThe training data P = (S p 1,T p 1),…,(S p |N| ,T p |N| ) and the model parameters θ k ,θ p and the trained model parameters θ k ,θ p Here, M is the data size of the learning data K, and N is the data size of the learning data P. S k m (m=1,…,M) are the face keypoint parameters θ k The face image is a training image in the training data for learning the face recognition system. The face image is, for example, an image in which only the face is cut out, and has a resolution of 224x224 pixels and three RGB channels. k m (m=1,…,M) are the face keypoint parameters θ k It is a correct label in the learning data for learning (hereinafter, also called "correct face keypoint label"). For example, it is vector data that stores the coordinates of each point of the face keypoint, and has a data format such as [[X1, Y1], ..., [X68, Y68]]. p n (n=1,…,N) is the face direction parameter θ p The learning face image S in the learning data K is a learning face image in the learning data for learning k m Similarly, for example, an image of only a face may have a resolution of, say, 224x224 pixels and 3 RGB channels. p n (n=1,…,N) is the face direction parameter θ p It is a correct label in the training data for learning the face orientation (hereinafter also referred to as the "face orientation correct label"), and is a vector that stores the three rotation angles of the face orientation: roll, pitch, and yaw, and has a data format such as [roll, pitch, yaw].

[0017] FIG. 2 shows a functional block diagram of the learning device 100, and FIG. 3 shows the processing flow thereof.

[0018] The learning device 100 includes a data selection unit 110 , a shared network unit 120 , a facial keypoint network unit 130 , a facial keypoint model parameter optimization unit 140 , a facial pose network unit 150 , and a facial pose model parameter optimization unit 160 .

[0019] An overview of each part will be explained below.

[0020] The data selection unit 110 selects face keypoint parameters θ k A training face image (hereinafter referred to as "face keypoint training face image") S k 1,…,S k |M| and the face direction parameter θ p Learning face images (hereinafter also referred to as "face orientation learning face images") S p 1,…,S p |N| Take and as input, and select the training data to be used in one batch.

[0021] The shared network unit 120 receives one batch of face images for learning as input, and acquires intermediate features using any neural network that outputs intermediate features.

[0022] The facial keypoint network unit 130 obtains estimates of the facial keypoints using any neural network that takes the intermediate features as input and outputs estimates of the facial keypoints.

[0023] The face keypoint model parameter optimization unit 140 receives the estimated values ​​of the face keypoints and the correct labels of the face keypoints, calculates the error of the estimated coordinates of the face keypoints, and optimizes the model parameters θ k Update.

[0024] The face pose network unit 150 obtains an estimate of the face pose using any neural network that takes the intermediate features as input and outputs an estimate of the face pose.

[0025] The face direction model parameter optimization unit 160 receives the estimated face direction value and the correct label of the face direction, calculates the error of the estimated rotation angle of the face direction, and optimizes the model parameter θ p Update.

[0026] The above process shows the learning procedure for one batch (a block of data selected from the learning data), and this can be repeated any number of times to learn all the data.

[0027] The details of each part will be explained below.

[0028] <Data Selection Unit 110> Input: Face image S for learning face keypoints k 1,…,S k |M| , face image for face orientation learning S p 1,…,S p |N| Output: 1 batch of training face images S b , data identifier i b The data selection unit 110 selects face images S for face keypoint learning. k 1,…,S k |M| and face orientation learning face image S p 1,…,S p |N| The input is the data used for one learning session = one batch of learning face images S b (S110). Note that b is an index indicating a batch number. For example, the face image S for face keypoint learning is selected. k 1,…,S k |M| , face image for face orientation learning S p 1,…,S p |N|Each of these is divided into B batches, b = 1, 2, ..., 2B. In this case, the training face image S b For example, a batch of training face images S b is created for each face image for face keypoint learning and face image for face orientation learning, and all the learning face images in one batch are composed of face images for face keypoint learning, or all are composed of face images for face orientation learning. As a data selection procedure, for example, the first batch of learning is a batch composed entirely of face images for face keypoint learning, the second batch of learning is a batch composed entirely of face images for face orientation learning, and so on. In addition, the data identifier i b is a batch of training face images S b is an identifier for identifying whether the face image is composed of face keypoint learning face image data or face pose learning face image data.

[0029] <Shared network section 120> Input: 1 batch of training face images S b , data identifier i b , the updated model parameters ^θ k or ^θ p Among these, parameters corresponding to the neural network constituting the shared network unit 120 are Output: 1 batch of intermediate features v b , data identifier i b The shared network unit 120 is configured with an arbitrary neural network, for example, with four convolution layers.

[0030] Prior to the conversion process, the shared network unit 120 updates the model parameters ^θ k (output value of the face keypoint model parameter optimization unit 140) or ^θ p Among the output values ​​of the face direction model parameter optimization unit 160, parameters corresponding to any neural network constituting the shared network unit 120 are received.

[0031] The shared network unit 120 updates the model parameters ^θ k or ^θ p Using parameters corresponding to any neural network constituting the shared network unit 120, one batch of learning face images S is generated by a function of the arbitrary neural network. b The intermediate feature v b For example, the training face image S b If v consists of Q images, the intermediate feature corresponding to the q-th image is v b q Let v b =[v b 1,v b 2,…,v b Q ], where q = 1, 2, …, Q.

[0032] In addition, the shared network unit 120 stores the data identifier i b is output as is.

[0033] <Face Keypoint Network Division 130> Input: 1 batch of intermediate features v b , data identifier i b , the updated model parameters ^θ k Among these, parameters corresponding to the neural network constituting the face keypoint network unit 130 Output: Estimated face keypoint vector Z' k,b The face keypoint network unit 130 is configured with any neural network, for example, with two fully connected layers.

[0034] Prior to the conversion process, the face keypoint network unit 130 updates the model parameters ^θ k Among the output values ​​of the face keypoint model parameter optimization unit 140, parameters corresponding to any neural network constituting the face keypoint network unit 130 are received.

[0035] The face keypoint network unit 130 receives face images S b The data identifier i b If so (YES in S129), a batch of intermediate features v is generated by an arbitrary neural network function using the received parameters. b Estimate the face keypoint vector Z' k,b (S130). The estimated face keypoint vector is vector data that stores the coordinates of the estimated values ​​of each point of the face keypoint in an array. In other words, the estimated face keypoint vector is an estimated value of the face keypoint vector. The face keypoint vector is vector data that stores the coordinates of each point of the face keypoint. For example, in the case of a model that estimates 68 two-dimensional keypoints, if X1 is the X coordinate value of the first keypoint and Y1 is the Y coordinate value of the first keypoint, the face keypoint vector will be a vector such as [[X1,Y1],...,[X68,Y68]]. In addition, in the case of a model that estimates three-dimensional keypoints, if Z1 is the Z coordinate value of the first keypoint, the face keypoint vector will be a vector such as [[X1,Y1,Z1],...,[X68,Y68,Z68]]. For example, the qth intermediate feature v of the bth batch b q Let Z' be the estimated face keypoint vector corresponding to k,b,q Let Z' k,b =[Z' k,b,1 ,Z' k,b,2 ,…,Z' k,b,Q ]. The estimated face keypoint vector Z' k,b,q results in a vector such as [[X1,Y1],…,[X68,Y68]] or [[X1,Y1,Z1],…,[X68,Y68,Z68]].

[0036] In addition, the learning face image S b The data identifier i b , the facial keypoint network unit 130 and the facial keypoint model parameter optimizer unit 140 do not perform any processing.

[0037] <Facial keypoint model parameter optimization unit 140> Input: Estimated face keypoint vector Z' k,b , face keypoint correct label T k 1,…,T k |M| Output: Updated model parameters ^θ k or the trained model parameters θ k Model parameters ^θ k and θ k is a concatenation of parameters corresponding to the neural network constituting the shared network unit 120 and parameters corresponding to the neural network constituting the face keypoint network unit 130.

[0038] The facial keypoint model parameter optimization unit 140 optimizes the estimated facial keypoint vector Z' k,b and face keypoint correct label T k 1,…,T k |M| Using the model parameters ^θ k For example, the face keypoint model parameter optimization unit 140 updates the estimated face keypoint vector Z' k,b and face keypoint correct label T k 1,…,T k |M| Calculate the error between and and set the model parameters ^θ to minimize the error. k The error can be, for example, the MSE error or the MAE error, and the parameter can be updated using the gradient descent method.

[0039] The above-mentioned processes S120 to S140 are repeated until a predetermined condition is satisfied (S170). The predetermined condition is a condition for determining whether the parameter update has converged or not, and may be, for example, (i) the number of updates has exceeded a predetermined number, or (ii) the difference between the parameters before and after the update is smaller than a predetermined value.

[0040] <Face Orientation Network Section 150> Input: 1 batch of intermediate features v b , data identifier i b , the updated model parameters ^θ p Among these, parameters corresponding to the neural network constituting the face direction network unit 150 Output: Estimated face direction vector Z' p,b The face direction network unit 150 is configured with any neural network, for example, with two fully connected layers.

[0041] Prior to the conversion process, the face direction network unit 150 updates the model parameters ^θ p Among the output values ​​of the face direction model parameter optimization unit 160, parameters corresponding to any neural network constituting the face direction network unit 150 are received.

[0042] The face orientation network unit 150 receives the learning face image S b The data identifier i b If so (NO in S129), a batch of intermediate features v is generated by an arbitrary neural network function using the received parameters. b Estimate face direction vector Z' p,b (S150). The estimated face direction vector is vector data storing estimated values ​​of the rotation angles in three directions, roll, pitch, and yaw, of the face direction. In other words, the estimated face direction vector is an estimated value of a face direction vector. The face direction vector is a vector storing the rotation angles in three directions, roll, pitch, and yaw, of the face direction. For example, the rotation angle value has a range of -180 degrees to 180 degrees, and the face direction vector is a vector such as [20, 43, -151]. For example, the q-th intermediate feature v of the b-th batch is b q Let Z' be the estimated face direction vector corresponding to p,b,q Let Z' p,b =[Z' p,b,1 ,Z' p,b,2 ,…,Z' p,b,Q]. The estimated face direction vector Z' p,b,q will result in a vector such as [20, 43, -151].

[0043] In addition, the learning face image S b The data identifier i b indicates, the face pose network unit 150 and the face pose model parameter optimizer unit 160 do not perform any processing.

[0044] <Facial orientation model parameter optimization unit 160> Input: Estimated face direction vector Z' p,b , face orientation correct label T p 1,…,T p |N| Output: Updated model parameters ^θ p or the trained model parameters θ p Model parameters ^θ p and θ p is a concatenation of parameters corresponding to the neural network constituting shared network section 120 and parameters corresponding to the neural network constituting face direction network section 150.

[0045] The face direction model parameter optimization unit 160 calculates the estimated face direction vector Z' p,b and face orientation correct label T p 1,…,T p |N| Using the model parameters ^θ p For example, the face direction model parameter optimization unit 160 updates the estimated face direction vector Z' p,b and face orientation correct label T p 1,…,T p |N| Calculate the error between and and set the model parameters ^θ to minimize the error. p The error can be, for example, the MSE error or the MAE error, and the parameter can be updated using the gradient descent method.

[0046] The above-mentioned processes S120 to S160 are repeated until a predetermined condition is satisfied (S170).

[0047] Furthermore, the above-mentioned processes S110 to S170 are performed on all the batch data (learning data). For example, it is determined whether or not there is unprocessed batch data (S180), and if there is unprocessed batch data, the above-mentioned processes S110 to S170 are performed (NO in S180), and if there is no unprocessed batch data (YES in S180), the process ends.

[0048] After performing the above process on all the batch data, the final updated model parameters ^θ k , ^θ p The trained model parameters θ k , θ p The output is as follows:

[0049] Next, the estimation device 200 will be described.

[0050] <Estimation device 200> Prior to the estimation process, the estimation device 200 calculates the learned model parameters θ k ,θ p The estimation device 200 receives a face image S of an estimation target as input and receives a trained model parameter θ k We estimate face keypoints using the trained model parameters θ p The face orientation is estimated using the estimated face keypoint vector Z k , the estimated face direction vector Z p Output.

[0051] FIG. 4 shows a functional block diagram of the estimation device 200, and FIG. 5 shows the process flow thereof.

[0052] The estimation device 200 includes a face keypoint estimation unit 210 and a face direction estimation unit 220 .

[0053] The details of each part will be explained below.

[0054] <Facial key point estimation unit 210> Input: Face image S, model parameters θ k Output: Estimated face keypoint vector Z for face image S k Prior to the estimation process, the face keypoint estimation unit 210 calculates the model parameters θ k Receive.

[0055] The facial keypoint estimation unit 210 is configured to estimate the facial keypoints by using the shared network 120, followed by the facial keypoint network 130, and the model parameters θ k Using the above, face key points are estimated from the face image S (S210), and the estimated value (estimated face key point vector Z k For example, if the shared network unit 120 is configured with an arbitrary neural network consisting of four convolution layers, and the face keypoint network unit 130 is configured with an arbitrary neural network consisting of two fully connected layers, the face keypoint estimation unit 210 is configured with a neural network consisting of four convolution layers and two fully connected layers corresponding to the shared network unit 120 and the face keypoint network unit 130, and the model parameter θ k is used.

[0056] <Facial orientation estimation unit 220> Input: Face image S, model parameters θ p Output: Estimated face direction vector Z for face image S p Prior to the estimation process, the face direction estimation unit 220 calculates the model parameter θ p Receive.

[0057] The face direction estimation unit 220 estimates the network architecture of the shared network unit 120 and the subsequent face direction network unit 150, and the model parameters θ p The face direction is estimated from the face image S (S220) using the estimated face direction vector Z pFor example, if the shared network unit 120 is configured with an arbitrary neural network having four convolution layers, and the face direction network unit 150 is configured with an arbitrary neural network having two fully connected layers, the face direction estimation unit 220 is configured with a neural network having four convolution layers and two fully connected layers corresponding to the shared network unit 120 and the face direction network unit 150, and the model parameter θ p is used.

[0058] <Effects> With the above configuration, it is possible to estimate face keypoints and face poses that capture more general facial features.

[0059] <Modification> In this embodiment, the estimation device 200 estimates face key points and face direction, but may estimate only one of them. Even in this case, during learning, model parameters used for estimating face key points and face direction are learned in a single neural network system, so that it is possible to estimate face key points or face direction capturing more general facial features.

[0060] In this embodiment, the learning is performed by alternately using the batches composed entirely of face images for face keypoint learning and the batches composed entirely of face images for face orientation learning as learning data, but it is not necessary to alternately use one batch at a time, and for example, ten batches may be alternately used. However, it is preferable to alternately use one batch at a time to prevent bias toward any of the learning data.

[0061] <Other Modifications> The present invention is not limited to the above-mentioned embodiment and modified examples. For example, the above-mentioned various processes may be executed not only in chronological order as described, but also in parallel or individually depending on the processing capacity of the device executing the processes or as necessary. In addition, appropriate modifications are possible within the scope of the present invention.

[0062] <Programs and recording media> The various processes described above can be implemented by loading a program for executing each step of the above method into the memory unit 2020 of the computer shown in FIG. 6 and operating the control unit 2010, input unit 2030, output unit 2040, etc.

[0063] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other type of recording medium.

[0064] The program may be distributed, for example, by selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of a server computer and transferring the program from the server computer to other computers via a network.

[0065] A computer that executes such a program, for example, first stores the program recorded on a portable recording medium or the program transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from the portable recording medium and execute a process according to the program, or may execute a process according to the received program each time a program is transferred from the server computer to the computer. The above-mentioned process may be executed by a so-called ASP (Application Service Provider) type service that realizes a processing function only by issuing an execution instruction and obtaining a result, without transferring a program from the server computer to the computer. Note that the program in this embodiment includes information used for processing by an electronic computer and equivalent to a program (data that is not a direct command to the computer but has a nature that specifies the processing of the computer, etc.).

[0066] In addition, in this embodiment, the present device is configured by executing a specific program on a computer, but at least a part of the processing contents may be realized by hardware.

Claims

1. Model parameters ^θ k or ^θ p Using the neural network function, the face image S for training is generated. b The intermediate feature v b A shared network part that converts The learning face image S b is composed of face images for face keypoint training, the model parameters ^θ k The intermediate feature v is calculated by the neural network function using b Estimate the face keypoint vector Z' k,b A face keypoint network part that converts the estimated face keypoint vector Z' k,b and the learning face image S b The correct face keypoint label for T k 1 ,…,T k |M| Using the model parameters ^θ k a face keypoint model parameter optimization unit for updating The learning face image S b is composed of face images for face orientation learning, the model parameters ^θ p Using the neural network function, the intermediate feature v b Estimate face direction vector Z' p,b A face orientation network part that converts the face orientation into a face orientation network part. The estimated face direction vector Z′ p,b and face orientation correct label T p 1 ,…,T p |N| Using the model parameters ^θ p a face pose model parameter optimization unit that updates The model parameter ^θ k The trained model parameters θ k and the model parameters ^θ p The trained model parameters θ p Get Learning device.

2. The model parameter θ learned by the learning device of claim 1 k An estimation device using The network architecture of the shared network and the subsequent face keypoint network, and the model parameters θ k and a face key point estimation unit that estimates face key points from a face image S of an estimation target using the face key point estimation unit. Estimation device.

3. The model parameter θ learned by the learning device of claim 1 p An estimation device using The network architecture of the shared network unit and the subsequent face orientation network unit, and the model parameters θ p and a face direction estimation unit that estimates a face direction from the face image S using the face direction estimation unit. Estimation device.

4. The model parameter θ learned by the learning device of claim 1 k and the model parameters θ p An estimation device using The network architecture of the shared network and the subsequent face keypoint network, and the model parameters θ k a face key point estimation unit that estimates face key points from a face image S of an estimation target using the above; The network architecture of the shared network unit and the subsequent face orientation network unit, and the model parameters θ p and a face direction estimation unit that estimates a face direction from the face image S using the face direction estimation unit. Estimation device.

5. Model parameters ^θ k or ^θ p Using the neural network function, the training face image S b The intermediate feature v b A shared network step to convert The learning face image S b is composed of face images for face keypoint training, the model parameters ^θ k The intermediate feature v is calculated by the neural network function using b Estimate the face keypoint vector Z' k,b A face keypoint network step to convert the estimated face keypoint vector Z' k,b and the learning face image S b The correct face keypoint label for T k 1 ,…,T k |M| Using the model parameters ^θ k a face keypoint model parameter optimization step for updating The learning face image S b is composed of face images for face orientation learning, the model parameters ^θ p Using the neural network function, the intermediate feature v b Estimate face direction vector Z' p,b A face orientation network step to convert The estimated face direction vector Z′ p,b and face orientation correct label T p 1 ,…,T p |N| Using the model parameters ^θ p and optimizing the face pose model parameters by updating The model parameter ^θ k The trained model parameters θ k and the model parameters ^θ p The trained model parameters θ p Get How to learn.

6. The model parameter θ learned by the learning method of claim 5 k An estimation method using The network architecture of the shared network step and the subsequent face keypoint network step, and the model parameters θ k and a face keypoint estimation step of estimating face keypoints from a face image S of an estimation target using the face keypoint estimation step. Estimation method.

7. The model parameter θ learned by the learning method of claim 5 p An estimation method using The network architecture of the shared network step and the subsequent face orientation network step, and the model parameters θ p and estimating a face direction from the face image S using the face direction estimation step. Estimation method.

8. A program for causing a computer to function as the learning device of claim 1 or the estimation device of any one of claims 2 to 4.

Images