Learning device, learning method, and program

Abstract

This learning device comprises: a training data acquisition means for acquiring training data that includes an original image; an image inversion means for generating an inverted image in which the original image is axisymmetrically inverted; a feature point coordinate calculation means for calculating the coordinates of feature points of the original image and the coordinates of feature points of the inverted image; a symmetry evaluation value calculation unit for calculating a symmetry evaluation value that indicates the degree of difference between the coordinates of feature points of the original image and the coordinates of feature points of the inverted image when coordinate conversion is carried out so as to cancel out the difference in coordinates between the coordinates of feature points of the original image and the coordinates of feature points of the inverted image due to the effect of image inversion; and a learning control means for controlling the learning of the feature point coordinate calculation means so that the degree of difference indicated by the symmetry evaluation value is reduced.

Description

[Technical Field]

[0001] The present invention relates to a learning device. , studies Regarding learning methods and programs. [Background technology]

[0002] In processing face images, in addition to the original face image, a face image obtained by flipping the original face image horizontally may be used. For example, Patent Document 1 describes that when performing pattern recognition on a face image, in order to prepare as few sample face images as possible, in addition to the original face image, a face image obtained by flipping the original face image horizontally is used. For example, an image processing device described in Patent Document 1 stores multiple normalized facial image data, calculates and stores average image data, variance image data, and covariance matrix data for all of the multiple facial image data and facial image data obtained by flipping the multiple facial image data horizontally. When a new input facial image is obtained, the image processing device refers to the average image data, variance image data, and covariance matrix data to calculate the Mahalanobis distance between the input facial image and feature information of the stored facial image. If the calculated Mahalanobis distance is smaller than a predetermined value, the image processing device outputs a control signal indicating that the input facial image is close to the stored facial image.

[0003] Furthermore, Patent Document 2 describes that, in order to evaluate the reliability of a face image detected by face detection, learning is performed using training data and learning is performed using a mirror image of the training data. In the image processing device described in Patent Document 2, the face detection means detects a face image facing approximately frontally using a detector that has been trained using training data to detect a face image facing approximately frontally. The face evaluation means evaluates the reliability of the face image detected by the face detection means. The mirror image detection means detects a face image facing approximately frontally by utilizing a detector that has been trained using the mirror image of the training data to detect a face image facing approximately frontally. The face image re-evaluation means increases the reliability of the face image detected by the face detection means when the mirror image detection means detects a face image in an area that overlaps with the face image detected by the face detection means by a predetermined percentage or more. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2000-187733 [Patent Document 2] Japanese Patent Application Publication No. 2010-160640 Summary of the Invention [Problem to be solved by the invention]

[0005] When learning image processing of oriented objects, such as facial images, if the symmetry between the original image and its line-symmetrically inverted form can be reflected in the learning results, it is expected that processing that is robust to the orientation of the object can be learned.

[0006] An example of the object of this disclosure is a learning device that can solve the above-mentioned problems. , studies The purpose of this project is to provide learning methods and programs. [Means for solving the problem]

[0007] According to a first aspect of the present invention, a learning device includes training data acquisition means for acquiring training data including an original image; image inversion means for generating an inverted image by inverting the original image in an axisymmetric manner; feature point coordinate calculation means for calculating the coordinates of feature points in the original image and the coordinates of feature points in the inverted image; symmetry evaluation value calculation means for calculating a symmetry evaluation value indicating the degree of difference between the coordinates of feature points in the original image and the coordinates of feature points in the inverted image when coordinate transformation is performed so as to cancel out the difference in coordinates between the feature points in the original image and the feature points in the inverted image due to the effect of image inversion; and learning control means for controlling the learning of the feature point coordinate calculation means so as to reduce the degree of difference indicated by the symmetry evaluation value.

[0009] According to a third aspect of the present invention, a learning method includes: a computer acquiring training data including an original image and ground truth values ​​of coordinates of feature points of the original image; generating an inverted image by inverting the original image in line symmetry; calculating coordinates of feature points of the original image and coordinates of feature points of the inverted image; calculating a symmetry evaluation value indicating a degree of difference between the coordinates of feature points of the original image and the coordinates of feature points of the inverted image when coordinate transformation is performed so as to cancel out differences in coordinates between the coordinates of feature points of the original image and the coordinates of feature points of the inverted image due to the influence of image inversion; and controlling learning of a model that calculates the coordinates of the feature points so as to reduce the degree of difference indicated by the symmetry evaluation value.

[0010] According to a fourth aspect of the present invention, programa program for causing a computer to execute the steps of: acquiring training data including an original image and correct values ​​of coordinates of feature points of the original image; generating an inverted image by inverting the original image in line symmetry; calculating coordinates of feature points of the original image and coordinates of feature points of the inverted image; calculating a symmetry evaluation value indicating the degree of difference between the coordinates of feature points of the original image and the coordinates of feature points of the inverted image when coordinate transformation is performed so as to cancel out the difference in coordinates between the coordinates of feature points of the original image and the coordinates of feature points of the inverted image due to the influence of image inversion; and controlling learning of a model that calculates the coordinates of the feature points so as to reduce the degree of difference indicated by the symmetry evaluation value. In be. [Effects of the Invention]

[0011] According to the present invention, when learning image processing of an object with a direction, the symmetry between image data and image data that is an image of that image data that is line-symmetrically inverted can be reflected in the learning results. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a learning device according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of input and output of data in the learning device according to the embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of the configuration of a face authentication device that uses a feature point coordinate calculation unit that has been trained by a learning device according to an embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of a processing procedure in which the learning device according to the embodiment learns the feature point coordinate calculation unit. [Figure 5] FIG. 10 is a diagram illustrating an example of a processing procedure for performing face authentication by the face authentication device according to the embodiment. [Figure 6] FIG. 10 is a diagram illustrating another example of the configuration of the learning device according to the embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of a processing procedure in a learning method according to an embodiment. [Figure 8]FIG. 1 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] The following describes embodiments of the present invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. Fig. 1 is a diagram illustrating an example of the configuration of a learning device according to an embodiment. In the configuration illustrated in Fig. 1, the learning device 100 includes a training data acquisition unit 110, an image inversion unit 120, a feature point coordinate calculation unit 130, a coordinate inversion unit 140, and a learning control unit 150. The feature point coordinate calculation unit 130 includes a convolutional neural network 131 and a fully connected layer neural network 132. The learning control unit 150 includes a loss function calculation unit 151 and an update amount calculation unit 154. The loss function calculation unit 151 includes a learning loss calculation unit 152 and a symmetry loss calculation unit 153.

[0014] The learning device 100 learns a model that calculates the coordinates of feature points in an image. In particular, the learning device 100 learns the model so that it is robust to the orientation of an object depicted in the image. To this end, the learning device 100 learns the model using an original image and an inverted image obtained by inverting the original image in a line symmetry. In this case, inverting an image in a line symmetry means inverting the image around a certain straight line. Calculating the coordinates of the feature points is also referred to as feature point detection, and the accuracy of the coordinates of the feature points is also referred to as feature point detection accuracy.

[0015] The following describes an example in which the learning device 100 learns a model that detects feature points in a facial image for face recognition. In this case, the model that the learning device 100 learns calculates the coordinates of feature points that indicate areas of the facial image where facial features, such as the eyes, nose, and mouth, are likely to appear. High accuracy of these coordinates allows the size and tilt of the facial image to be properly normalized based on these coordinates in face recognition, which is expected to enable highly accurate face recognition.

[0016] In the case of face recognition, a model being robust to the orientation of an object in an image means that the accuracy of feature point detection is roughly the same whether the face of the person being recognized is facing right or left. Here, the right or left orientation is not limited to a completely right or left orientation, but may also be a diagonal right or diagonal left orientation.

[0017] If there is a difference in the accuracy of detecting feature points in a facial image depending on whether the facial image is facing left or right, it is possible that the authentication accuracy will decrease depending on the facial orientation, such as facial authentication being more likely to fail when the image is facing right. In contrast, since the model obtained by learning using the learning device 100 is robust to the orientation of the face, it is expected that face authentication can be performed with relatively high accuracy regardless of whether the face in the face image to be authenticated is oriented left or right.

[0018] However, the use of the model that is the target of learning by learning device 100 is not limited to a specific use. For example, learning device 100 may learn a model for estimating the pose of a human body, such as a model that detects the positions of joints in a whole-body image of a person as feature points.

[0019] Alternatively, the learning device 100 may be configured to train a model for detecting minutiae (feature points in a fingerprint image, such as the end points or bifurcation points of fingerprint ridges) in fingerprint authentication. In this way, the model trained by the learning device 100 can be used for various biometric authentications performed using images.

[0020] Furthermore, the line-symmetric flipping of the image by the learning device 100 is not limited to horizontal flipping. For example, in the case of human body pose estimation, the learning device 100 may flip an image of a person lying sideways upside down.

[0021] The training data acquisition unit 110 acquires training data for model learning. Specifically, the training data acquisition unit 110 acquires supervised learning data including sample data of a face image and training data indicating correct values ​​of the coordinates of feature points in the face image. The training data acquisition unit 110 is an example of a training data acquisition means.

[0022] The image inverter 120 generates a facial image by horizontally inverting a facial image included in the training data. The original horizontally inverted image is also called an original image, and the generated image is also called an inverted image or a horizontally inverted image. The image inverting unit 120 is an example of an image inverting means.

[0023] The feature point coordinate calculation unit 130 calculates the coordinates of the feature points of the original image and the coordinates of the feature points of the inverted image. The feature point coordinate calculation unit 130 is an example of a feature point coordinate calculation means. 1, the learning device 100 includes two feature point coordinate calculation units 130: a feature point coordinate calculation unit 130 that calculates the coordinates of feature points in an original image, and a feature point coordinate calculation unit 130 that calculates the coordinates of feature points in an inverted image. When distinguishing between these two feature point coordinate calculation units 130, the feature point coordinate calculation unit 130 that calculates the coordinates of feature points in the original image will be referred to as feature point coordinate calculation unit 130-1. Furthermore, the feature point coordinate calculation unit 130 that calculates the coordinates of feature points in an inverted image will be referred to as feature point coordinate calculation unit 130-2.

[0024] The convolutional neural network 131 receives a facial image as input and outputs the feature quantities of the facial image in the form of a feature vector. The number of dimensions of the feature vector output by the convolutional neural network 131 is not limited to a specific number of dimensions. For example, the convolutional neural network 131 may output a 1024-dimensional feature vector or a 2048-dimensional feature vector. Alternatively, the convolutional neural network 131 may output a feature vector with a number of dimensions other than these.

[0025] The convolutional neural network 131 may output feature vectors in a matrix format, such as in the format of image data. When distinguishing between the convolutional neural network 131 included in the feature point coordinate calculation unit 130-1 and the convolutional neural network 131 included in the feature point coordinate calculation unit 130-2, the convolutional neural network 131 included in the feature point coordinate calculation unit 130-1 will be referred to as the convolutional neural network 131-1, and the convolutional neural network 131 included in the feature point coordinate calculation unit 130-2 will be referred to as the convolutional neural network 131-2.

[0026] The fully connected layer neural network 132 converts the feature vector output by the convolutional neural network 131 into coordinates of feature points of the face image. For example, if the number of feature points to be detected by the feature point coordinate calculation unit 130 is fixed at five, the fully connected layer neural network 132 may output a 10-dimensional vector (2 dimensions × 5 dimensions = 10 dimensions).

[0027] When distinguishing between the fully connected layer neural network 132 provided in the feature point coordinate calculation unit 130-1 and the fully connected layer neural network 132 provided in the feature point coordinate calculation unit 130-2, the fully connected layer neural network 132 provided in the feature point coordinate calculation unit 130-1 will be referred to as the fully connected layer neural network 132-1, and the fully connected layer neural network 132 provided in the feature point coordinate calculation unit 130-2 will be referred to as the fully connected layer neural network 132-2.

[0028] The combination of the convolutional neural network 131 and the fully connected layer neural network 132 corresponds to an example of a model that is a target model for learning by the learning device 100, and that calculates the coordinates of feature points of an image. Convolutional neural network 131-1 and convolutional neural network 131-2 are configured as convolutional neural networks of the same structure. Fully connected layer neural network 132-1 and fully connected layer neural network 132-2 are configured as neural networks with fully connected layers of the same structure. The combination of convolutional neural network 131-1 and fully connected layer neural network 132-1 corresponds to an example of a first model. The combination of convolutional neural network 131-2 and fully connected layer neural network 132-2 corresponds to an example of a second model.

[0029] During learning, learning device 100 sets the parameter values ​​of convolutional neural network 131-1 and convolutional neural network 131-2 to the same values. Learning device 100 also sets the parameter values ​​of fully connected layer neural network 132-1 and fully connected layer neural network 132-2 to the same values. As a result, a model formed by combining convolutional neural network 131-1 and fully connected layer neural network 132-1 and a model formed by combining convolutional neural network 131-2 and fully connected layer neural network 132-2 are configured as the same model.

[0030] The learning of the convolutional neural network 131-1 and the fully connected layer neural network 132-1 is also referred to as the learning of the feature point coordinate calculation unit 130-1. The learning of the convolutional neural network 131-2 and the fully connected layer neural network 132-2 is also referred to as the learning of the feature point coordinate calculation unit 130-2. The learning of the feature point coordinate calculation unit 130-1 and the learning of the feature point coordinate calculation unit 130-2 are also collectively referred to as the learning of the feature point coordinate calculation unit 130.

[0031] However, the model to be learned by learning device 100 is not limited to a specific type of model. Any model that receives an input image, outputs the coordinates of feature points of the image, and is trainable can be used as the model to be learned by learning device 100.

[0032] The coordinate inversion unit 140 performs coordinate transformation on the coordinates of the feature points calculated by the feature point coordinate calculation unit 130-2 so as to cancel out the influence of image inversion. Specifically, the coordinate inversion unit 140 transforms the coordinates of the feature points so as to horizontally invert the feature points on the same axes as the axes on which the image was transformed by the image inversion unit 120. The coordinate transformation performed by the coordinate inversion unit 140 corresponds to an example of coordinate transformation performed to cancel out the difference in coordinates between the feature points of the original image and the feature points of the inverted image due to the influence of image inversion.

[0033] However, the coordinate transformation performed to cancel out the difference in coordinates between the coordinates of feature points in the original image and the coordinates of feature points in the inverted image due to the influence of image inversion is not limited to the above-described image transformation by the coordinate inversion unit 140. For example, instead of transforming the coordinates of feature points in the inverted image, the coordinate inversion unit 140 may perform coordinate transformation equivalent to the coordinate transformation due to image inversion by the image inversion unit 120 on the coordinates of feature points in the original image calculated by the feature point coordinate calculation unit 130-1 and the correct coordinates of feature points in the original image.

[0034] The learning control unit 150 controls the learning of the feature point coordinate calculation unit 130. Specifically, the learning control unit 150 calculates update amounts for the parameter values ​​of the convolutional neural network 131-1, the fully connected layer neural network 132-1, the convolutional neural network 131-2, and the fully connected layer neural network 132-2. Then, the learning control unit 150 controls the feature point coordinate calculation unit 130 to change the parameter values ​​by the calculated update amounts. Updating the parameter values ​​of a model based on training data is called learning, and controlling the learning is called controlling the learning.

[0035] In particular, the learning control unit 150 controls the learning of the feature point coordinate calculation unit 130 so as to reduce the degree of difference between the coordinates of the feature points of the original image calculated by the feature point coordinate calculation unit 130-1 and the coordinates obtained by converting the coordinates of the feature points of the inverted image calculated by the feature point coordinate calculation unit 130-2 into coordinates obtained by the coordinate inversion unit 140. Specifically, the learning control unit 150 controls the learning of the feature point coordinate calculation unit 130 so as to reduce the total loss value, for example, by minimizing or locally minimizing the total loss value calculated by the loss function calculation unit 151. The total loss value is also referred to as a total loss value. The total loss value is an example of a total evaluation value.

[0036] However, the evaluation function used by the learning control unit 150 for learning the feature point coordinate calculation unit 130 is not limited to a loss function. The learning control unit 150 may use an evaluation function whose evaluation function value increases as the accuracy of the feature point coordinates calculated by the feature point coordinate calculation unit 130 increases.

[0037] Furthermore, learning control unit 150 controls learning of feature point coordinate calculation unit 130 so as to set the same parameter values ​​to convolutional neural networks 131-1 and 131-2 and to set the same parameter values ​​to fully connected layer neural networks 132-1 and 132-2. As a result, a model formed by combining convolutional neural networks 131-1 and 132-1 and a model formed by combining convolutional neural networks 131-2 and 132-2 are configured as the same model.

[0038] The loss function calculation unit 151 calculates the value of the loss function used for learning by the feature point coordinate calculation unit 130. The learning loss calculation unit 152 calculates the learning loss. The learning loss is also called learning loss. The learning loss indicates the degree of difference between the coordinates of the feature points calculated by the feature point coordinate calculation unit 130 and the correct answers indicated in the training data.

[0039] The loss function calculation unit 151 includes a learning loss calculation unit 152 for each of the original image and the inverted image. When distinguishing between these two learning loss calculation units 152, the learning loss calculation unit 152 on the original image side will be referred to as learning loss calculation unit 152-1, and the learning loss calculation unit 152 on the inverted image side will be referred to as learning loss calculation unit 152-2.

[0040] The learning loss calculation unit 152-1 calculates a learning loss indicating the degree of difference between the coordinates of the feature points of the original image calculated by the feature point coordinate calculation unit 130-1 and the correct answer indicated in the training data. The learning loss calculated by the learning loss calculation unit 152-1 is also referred to as a first learning loss or a learning loss of the original image. The first learning loss corresponds to an example of a first learning evaluation value. The learning loss calculation unit 152-1 corresponds to an example of a first learning evaluation value calculation means.

[0041] The learning loss calculation unit 152-2 calculates a learning loss indicating the degree of difference between the coordinates obtained by converting the coordinates of the feature points of the original image calculated by the feature point coordinate calculation unit 130-2 by the coordinate inversion unit 140 and the correct answer indicated in the training data. The learning loss calculated by the learning loss calculation unit 152-2 is also referred to as a second learning loss or a learning loss of an inverted image. The second learning loss corresponds to an example of a second learning evaluation value. The learning loss calculation unit 152-2 corresponds to an example of a second learning evaluation value calculation means.

[0042] As the learning loss, various losses based on the distance between coordinates can be used. For example, the learning loss calculation unit 152 may use any of the L2 loss, L1 loss, and Wing loss as the learning loss, but is not limited to these.

[0043] The symmetry loss calculation unit 153 calculates a symmetry loss that indicates the degree of difference between the coordinates of the feature points of the original image calculated by the feature point coordinate calculation unit 130-1 and the coordinates obtained by converting the coordinates of the feature points of the inverted image calculated by the feature point coordinate calculation unit 130-2 by the coordinate inversion unit 140. The symmetry loss is also referred to as a symmetry loss. The symmetry loss corresponds to an example of a symmetry evaluation value. The symmetry loss calculation unit 153 corresponds to an example of a symmetry evaluation value calculation means.

[0044] As the symmetric loss, various losses based on the distance between coordinates can be used. For example, the symmetric loss calculation unit 153 may use any of the L2 loss, the L1 loss, and the Wing loss as the symmetric loss, but is not limited to these.

[0045] Symmetry loss calculation unit 153 may be configured to output the symmetry loss to the outside of learning device 100. In this case, symmetry loss calculation unit 153 corresponds to an example of output means. Furthermore, learning device 100 corresponds to an example of a symmetry evaluation device.

[0046] For example, the symmetry loss calculation unit 153 may have a display screen and display the symmetry loss numerically, or may have a communication function and transmit data indicating the symmetry loss to another device. By referring to the symmetry loss, the user of the learning device 100 can check how robust the feature point coordinate calculation unit 130 is to the orientation of the face.

[0047] The loss function calculation unit 151 calculates a total loss value based on the first learning loss, the second learning loss, and the symmetric loss. The loss function calculation unit 151 calculates the total loss value so that the total loss value increases as the value of the first learning loss increases, the total loss value increases as the value of the second learning loss increases, and the total loss value increases as the value of the symmetric loss increases. For example, the loss function calculation unit 151 may calculate the total loss value as the sum of the first learning loss, the second learning loss, and the symmetric loss, but is not limited to this.

[0048] As described above, the learning control unit 150 controls the learning of the feature point coordinate calculation unit 130 so as to reduce the total loss value. At this point, the learning control unit 150 controls the learning of the feature point coordinate calculation unit 130 so as to reduce the degree of difference indicated by the symmetry evaluation value.

[0049] The update amount calculation unit 154 calculates the update amounts of the parameter values ​​of the convolutional neural network 131 and the fully connected layer neural network 132 based on the total loss value. The method by which the update amount calculation unit 154 calculates the update amounts of the parameter values ​​may be, but is not limited to, backpropagation. The update amount calculation unit 154 may use various methods within the framework of regression learning to calculate the update amounts of the parameter values.

[0050] Update amount calculation unit 154 calculates the same parameter value update amount so as to set the same parameter values ​​for convolutional neural network 131-1 and convolutional neural network 131-2. For example, update amount calculation unit 154 may calculate an update amount for the parameters of convolutional neural network 131-1 and an update amount for the parameters of convolutional neural network 131-2. Then, update amount calculation unit 154 may calculate the sum of these update amounts as the parameter value update amount common to convolutional neural network 131-1 and convolutional neural network 131-2. The update amount calculation unit 154 calculates the amount of update of the parameter value for each of the parameters to be learned that the convolutional neural network 131 has.

[0051] Furthermore, the update amount calculation unit 154 calculates the same parameter value update amounts so as to set the same parameter values ​​in the fully connected layer neural network 132-1 and the fully connected layer neural network 132-2. For example, the update amount calculation unit 154 may calculate an update amount for the parameters of the fully connected layer neural network 132-1 and an update amount for the parameters of the fully connected layer neural network 132-2. Then, the update amount calculation unit 154 may calculate the sum of these update amounts as a parameter value update amount common to the fully connected layer neural network 132-1 and the fully connected layer neural network 132-2. The update amount calculation unit 154 calculates the parameter value update amount for each of the parameters to be learned that the fully connected layer neural network 132 has.

[0052] FIG. 2 is a diagram showing an example of data input and output in the learning device 100. 2, the training data acquisition unit 110 acquires training data and outputs image data included in the training data to the image inversion unit 120 and the convolutional neural network 131-1. The image data output by the training data acquisition unit 110 corresponds to the data of the original image. In addition, the training data acquisition unit 110 outputs correct answer data of the coordinates of the feature amounts associated with the output image data to the learning loss calculation unit 152-1 and the learning loss calculation unit 152-2.

[0053] The image inverting unit 120 generates image data of an inverted image by inverting the original image indicated by the image data output by the training data acquiring unit 110, and outputs the image data to the convolutional neural network 131-2. In the feature point coordinate calculation unit 130-1, the convolutional neural network 131-1 receives image data of the original image from the training data acquisition unit 110 and outputs feature vector data of the original image to the fully connected layer neural network 132-1. The fully connected layer neural network 132-1 receives the feature vector data of the original image output by the convolutional neural network 131-1 and outputs coordinate data of the feature points of the original image to the learning loss calculation unit 152-1 and the symmetry loss calculation unit 153.

[0054] In the feature point coordinate calculation unit 130-2, the convolutional neural network 131-2 receives image data of the inverted image from the image inverting unit 120 and outputs vector data of the inverted image to the fully connected layer neural network 132-2. The fully connected layer neural network 132-2 receives feature vector data of the inverted image output by the convolutional neural network 131-2 and outputs coordinate data of feature points of the inverted image to the coordinate inverting unit 140.

[0055] The coordinate inversion unit 140 receives the coordinate data of the feature points of the inverted image from the fully connected layer neural network 132-2 and performs coordinate transformation on the coordinates of the feature points of the inverted image as described above. The coordinate inversion unit 140 outputs coordinate data indicating the transformed coordinates to the symmetry loss calculation unit 153 and the learning loss calculation unit 152-2.

[0056] In the loss function calculation unit 151, the learning loss calculation unit 152-1 calculates a first learning loss based on the coordinate data of the feature points of the original image from the fully connected layer neural network 132-1 and the correct answer data from the training data acquisition unit 110. The learning loss calculation unit 152-2 calculates a second learning loss based on the coordinate data of the feature points of the inverted image after the coordinate transformation by the coordinate inversion unit 140 and the correct answer data from the training data acquisition unit 110.

[0057] The symmetry loss calculation unit 153 calculates the symmetry loss based on the coordinate data of the feature points of the original image from the fully connected layer neural network 132-1 and the coordinate data of the feature points of the inverted image after the coordinate transformation by the coordinate inversion unit 140. The loss function calculation unit 151 calculates a total loss value based on the first learning loss, the second learning loss, and the symmetric loss, and outputs the total loss value to the update amount calculation unit 154.

[0058] The update amount calculation unit 154 calculates the update amounts for the parameter values ​​of the convolutional neural network 131 and the fully connected layer neural network 132 based on the total loss value from the loss function calculation unit 151. The update amount calculation unit 154 outputs the calculated update amounts to the feature point coordinate calculation units 130-1 and 130-2, causing them to update the parameter values ​​of the convolutional neural network 131 and the fully connected layer neural network 132.

[0059] 3 is a diagram showing an example of the configuration of a face authentication device that uses a feature point coordinate calculation unit 130 that has been trained by the learning device 100. In the configuration shown in Fig. 3, the face authentication device 200 includes an image acquisition unit 210, a face detection unit 220, a feature point coordinate calculation unit 130, a normalization unit 240, a feature extraction unit 250, a matching unit 260, and an output unit 270. The feature point coordinate calculation unit 130 includes a convolutional neural network 131 and a fully connected layer neural network 132.

[0060] The feature point coordinate calculation unit 130 included in the face authentication device 200 has been trained by the training device 100. In Fig. 3, the feature point coordinate calculation unit 130 and its respective units are given the same reference numerals (130, 131, 132) as in Fig. 1. The face recognition device 200 receives an input of a face image and performs face recognition on the input face image. However, as described above, the use of the feature point coordinate calculation unit 130 trained by the learning device 100 is not limited to face recognition.

[0061] The image acquisition unit 210 acquires a face image of the person to be authenticated. The face detection unit 220 detects the face portion in the face image acquired by the image acquisition unit 210 .

[0062] The feature point coordinate calculation unit 130 calculates the coordinates of feature points from the face portion of the face image detected by the face detection unit 220. Here, the feature point coordinate calculation units 130-1 and 130-2 in FIG. 1 are the same. In particular, the convolutional neural networks 131-1 and 131-2 are the same model. Furthermore, the fully connected layer neural networks 132-1 and 132-2 are the same model. Therefore, either the feature point coordinate calculation unit 130-1 or 130-2 may be used as the feature point coordinate calculation unit 130 of the face authentication device 200.

[0063] The convolutional neural network 131 calculates a feature vector from the face portion of the face image. The fully connected layer neural network 132 converts the feature vector calculated by the convolutional neural network 131 into coordinates of feature points. However, the configuration of the model included in the feature point coordinate calculation unit 130 is not limited to a specific configuration.

[0064] The normalization unit 240 normalizes the face image in which the face portion is detected by the face detection unit 220 to make the size and inclination uniform, based on the coordinates of the feature points calculated by the feature point coordinate calculation unit 130. The feature extraction unit 250 calculates a feature vector from the face image normalized by the normalization unit 240. The feature vector calculated by the feature extraction unit 250 corresponds to the feature amount of the face image for face authentication.

[0065] The matching unit 260 performs face authentication by comparing the feature vector calculated by the feature extraction unit 250 with the feature vector of a face image registered in advance. The output unit 270 outputs the result of face authentication by the matching unit 260.

[0066] FIG. 4 is a diagram showing an example of a processing procedure in which the learning device 100 learns the feature point coordinate calculation unit 130.

[0067] (Step S101) The training data acquisition unit 110 reads out a combination of a face image and the correct coordinates of the feature points of the face image from the training data. The face image read out from the training images by the training data acquisition unit 110 corresponds to an example of an original image. After step S101, the process branches to step S111 and step S121. Step S111 and steps S121 to S123 can be executed in parallel.

[0068] (Step S111) The feature point coordinate calculation unit 130-1 calculates the coordinates of the feature points of the original image.

[0069] (Step S121) The image inverter 120 generates an inverted image by flipping the original image horizontally. After step S121, the process proceeds to step S122.

[0070] (Step S122) The feature point coordinate calculation unit 130-2 calculates the coordinates of the feature points of the inverted image. After step S122, the process proceeds to step S123.

[0071] (Step S123) The coordinate inversion unit 140 performs coordinate conversion on the coordinates of the feature points of the inverted image calculated by the feature point coordinate calculation unit 130-2 so as to cancel out the influence of the left-right inversion of the image by the image inversion unit 120.

[0072] After steps S111 and S123, the process branches to steps S131, S141, and S151. Steps S131, S141, and S151 can be executed in parallel.

[0073] (Step S131) The learning loss calculation unit 152-1 calculates the learning loss (first learning loss) of the original image based on the coordinates of the feature points of the original image calculated by the feature point coordinate calculation unit 130-1 and the correct answer indicated in the training data.

[0074] (Step S141) The symmetry loss calculation unit 153 calculates the symmetry loss based on the coordinates of the feature points of the original image calculated by the feature point coordinate calculation unit 130-1 and the coordinates obtained by converting the coordinates of the feature points of the inverted image calculated by the feature point coordinate calculation unit 130-2 by the coordinate inversion unit 140.

[0075] (Step S151) The learning loss calculation unit 152-2 calculates the learning loss (second learning loss) of the inverted image based on the coordinates into which the coordinates of the feature points of the original image calculated by the feature point coordinate calculation unit 130-2 are converted by the coordinate inversion unit 140 and the correct answer indicated in the training data.

[0076] After steps S131, S141, and S151, the process proceeds to step S161.

[0077] (Step S161) The loss function calculation unit 151 calculates a total loss value based on the first learning loss, the second learning loss, and the symmetric loss. After step S161, the process proceeds to step S162.

[0078] (Step S162) The update amount calculation unit 154 calculates the update amounts of the parameter values ​​of the convolutional neural network 131 and the fully connected layer neural network 132 based on the total loss value. After step S162, the process proceeds to step S163.

[0079] (Step S163) The feature point coordinate calculation unit 130 updates the parameter values ​​of the convolutional neural network 131 and the fully connected layer neural network 132 based on the update amounts of the parameter values ​​calculated by the update amount calculation unit 154. After step S163, the process proceeds to step S164.

[0080] (Step S164) The learning control unit 150 determines whether a learning termination condition is met. The learning termination condition here is not limited to a specific condition. For example, the learning termination condition may be whether the accuracy of the feature point coordinates calculated by the feature point coordinate calculation unit 130 is equal to or greater than a predetermined threshold. Alternatively, the learning termination condition may be whether learning has been performed for a predetermined number of epochs.

[0081] If the learning control unit 150 determines that the learning termination condition is not met (step S164: NO), the process returns to step S101. On the other hand, if the learning control unit 150 determines that the learning termination condition is met (step S164: YES), the learning device 100 terminates the processing of FIG.

[0082] FIG. 5 is a diagram showing an example of a processing procedure in which the face authentication device 200 performs face authentication.

[0083] (Step S201) The image acquisition unit 210 acquires a face image of the person to be authenticated. After step S201, the process proceeds to step S202.

[0084] (Step S202) The face detection unit 220 detects the face portion in the face image of the authentication target. After step S202, the process proceeds to step S203.

[0085] (Step S203) The feature point coordinate calculation unit 130 calculates the coordinates of feature points from the face portion of the face image detected by the face detection unit 220. After step S203, the process proceeds to step S204.

[0086] (Step S204) Based on the coordinates of the feature points calculated by the feature point coordinate calculation unit 130, the normalization unit 240 normalizes the face image in which the face portion is detected by the face detection unit 220 to make the size and inclination uniform. After step S204, the process proceeds to step S205.

[0087] (Step S205) The feature extraction unit 250 calculates a feature vector from the face image normalized by the normalization unit 240 . After step S205, the process proceeds to step S206.

[0088] (Step S206) The matching unit 260 performs face authentication by comparing the feature vector calculated by the feature extraction unit 250 with the feature vector of a face image registered in advance. After step S206, the process proceeds to step S207.

[0089] (Step S207) The output unit 270 outputs the result of the face authentication performed by the matching unit 260. After step S207, the face authentication device 200 ends the processing of FIG.

[0090] As described above, the training data acquisition unit 110 acquires training data including an original image. The image inversion unit 120 generates an inverted image by inverting the original image in line symmetry. The feature point coordinate calculation unit 130 calculates the coordinates of feature points in the original image and the coordinates of feature points in the inverted image. The symmetry loss calculation unit 153 calculates a symmetry loss that indicates the degree of difference between the coordinates of feature points in the original image and the coordinates of feature points in the inverted image when coordinate transformation is performed to cancel out the difference in coordinates caused by the effect of image inversion. The learning control unit 150 controls the learning of the feature point coordinate calculation unit 130 so as to reduce the degree of difference indicated by the symmetry loss.

[0091] According to the learning device 100, by training the feature point coordinate calculation unit 130 so as to reduce the degree of difference indicated by symmetry loss, the symmetry between an original image and an inverted image obtained by horizontally inverting the original image can be reflected in the training of the feature point coordinate calculation unit 130. When face image data is input to the trained feature point coordinate calculation unit 130, the learning device 100 can reduce asymmetry in the obtained coordinates depending on the left-right orientation of the face, such as when the face is facing left, the accuracy of the obtained coordinates is low, or when the face is facing right, the coordinates of the feature points tend to be more inward than the actual coordinates, and when the face is facing left, the coordinates of the feature points tend to be more outward than the actual coordinates. In this respect, the learning device 100 can obtain the coordinates of the feature points of an image with high accuracy.

[0092] Furthermore, the feature point coordinate calculation unit 130 calculates the coordinates of the feature points of the inverted image using a second model having the same structure as the first model used to calculate the coordinates of the feature points of the original image. The learning control unit 150 controls the learning of the feature point coordinate calculation unit 130 so that the same parameter values ​​are set for the first model and the second model. According to the learning device 100, processing on the original image and processing on the inverted image can be performed in parallel, and the asymmetry of the coordinates obtained depending on the left and right direction of the face can be reduced.

[0093] Furthermore, the learning loss calculation unit 152-1 calculates a first learning loss that indicates the degree of difference between the coordinates of feature points in the original image and the correct values ​​of the coordinates of feature points in the original image shown in the training data. The learning loss calculation unit 152-2 calculates a second learning loss that indicates the degree of difference between the coordinates of feature points in the inverted image and the correct values ​​when coordinate transformation is performed to cancel out the difference in coordinates due to the influence of image inversion. The learning control unit 150 controls the learning of the feature point coordinate calculation unit 130 so as to reduce the total loss value based on the symmetry loss, the first learning loss, and the second outer periphery loss.

[0094] With the learning device 100, regardless of whether the facial image is oriented left or right, the coordinates of the feature points calculated by the feature point coordinate calculation unit 130 can be made to approach the correct coordinates or the coordinates obtained by flipping the correct coordinates left and right, as with the image. In this respect, the learning device 100 is expected to reduce the asymmetry of the coordinates obtained depending on the left and right orientation of the face, and to obtain the coordinates of the feature points with high accuracy.

[0095] The face authentication device 200 also includes a feature point coordinate calculation unit 130 that has been trained by the training device 100 described above. It is described that the face authentication device 200 can obtain the coordinates of the feature points of the face image with a relatively high degree of accuracy regardless of whether the face is facing left or right. As a result, it is expected that the face authentication device 200 can perform face authentication with a relatively high degree of accuracy regardless of whether the face is facing left or right.

[0096] In addition, the symmetry loss calculation unit 153 outputs a symmetry loss indicating the degree of difference between the coordinates of the feature points of the original image and the coordinates of the feature points of the inverted image when coordinate transformation is performed so as to cancel out the difference in coordinates due to the influence of image inversion. By referring to the symmetry loss, the user of the learning device 100 can check how robust the feature point coordinate calculation unit 130 is to the orientation of the face.

[0097] 6 is a diagram showing another example of the configuration of a learning device according to an embodiment. In the configuration shown in Fig. 6, a learning device 610 includes a training data acquisition unit 611, an image inversion unit 612, a feature point coordinate calculation unit 613, a symmetry evaluation value calculation unit 614, and a learning control unit 615.

[0098] With this configuration, the training data acquisition unit 611 acquires training data including an original image. The image inversion unit 612 generates an inverted image by inverting the original image in line symmetry. The feature point coordinate calculation unit 613 calculates the coordinates of feature points in the original image and the coordinates of feature points in the inverted image. The symmetry evaluation value calculation unit 614 calculates a symmetry evaluation value indicating the degree of difference between the coordinates of feature points in the original image and the coordinates of feature points in the inverted image when coordinate transformation is performed to cancel out the difference in coordinates caused by the effect of image inversion. The learning control unit 615 controls the learning of the feature point coordinate calculation unit 613 so as to reduce the degree of difference indicated by the symmetry evaluation value.

[0099] The training data acquisition unit 611 corresponds to an example of training data acquisition means. The image inversion unit 612 corresponds to an example of image inversion means. The feature point coordinate calculation unit 613 corresponds to an example of feature point coordinate calculation means. The symmetry evaluation value calculation unit 614 corresponds to an example of symmetry evaluation value calculation means. The learning control unit 615 corresponds to an example of learning control means.

[0100] According to the learning device 610, by training the feature point coordinate calculation unit 613 so as to reduce the degree of difference indicated by the symmetry evaluation value, the symmetry between the original image and the inverted image obtained by inverting the original image in line symmetry can be reflected in the training of the feature point coordinate calculation unit 613. When image data is input to the trained feature point coordinate calculation unit 613, the learning device 610 can reduce asymmetry in the obtained coordinates depending on the orientation of the object depicted in the image, such as low accuracy of the obtained coordinates for a left-facing image, or the coordinates of feature points tending to be more inward than the actual coordinates for a right-facing image, and the coordinates of feature points tending to be more outward than the actual coordinates for a left-facing image. In this respect, the learning device 610 can obtain the coordinates of the feature points of an image with high accuracy.

[0101] 7 is a diagram showing an example of a processing procedure in a learning method according to an embodiment. The method shown in FIG. 7 includes acquiring training data (step S611), generating a reversed image (step S612), calculating feature point coordinates (step S613), calculating a symmetry evaluation value (step S614), and controlling learning (step S615).

[0102] In acquiring training data (step S611), the computer acquires training data including an original image and correct values ​​of the coordinates of feature points of the original image. In generating a reversed image (step S612), the computer generates a reversed image by line-symmetrically reversing the original image. In calculating the feature point coordinates (step S613), the computer calculates the coordinates of the feature points in the original image and the coordinates of the feature points in the inverted image.

[0103] In calculating a symmetry evaluation value (step S614), the computer calculates a symmetry evaluation value that indicates the degree of difference between the coordinates of the feature points of the original image and the coordinates of the feature points of the inverted image when coordinate transformation is performed to cancel out the difference in coordinates due to the influence of image inversion. In controlling the learning (step S615), the computer controls the learning of a model that calculates the coordinates of feature points so that the degree of difference indicated by the symmetry evaluation value becomes smaller.

[0104] According to the learning method shown in Fig. 7, by training a model so as to reduce the degree of difference indicated by the symmetry evaluation value, it is possible to reflect the symmetry between an original image and an inverted image obtained by inverting the original image in a line symmetry manner in the model training. The learning method shown in Fig. 7 can reduce asymmetry in the obtained coordinates depending on the orientation of the object in the image when image data is input to a trained model, such as low accuracy of the obtained coordinates for a left-facing image, or the coordinates of feature points tending to be more inward than the actual coordinates for a right-facing image, and the coordinates of feature points tending to be more outward than the actual coordinates for a left-facing image. In this respect, the learning method shown in Fig. 7 makes it possible to obtain the coordinates of feature points of an image with high accuracy.

[0105] FIG. 8 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. In the configuration shown in FIG. 8, a computer 700 includes a CPU (Central Processing Unit) 710, a main memory device 720, an auxiliary memory device 730, an interface 740, and a non-volatile recording medium 750.

[0106] One or more of the learning device 100, face recognition device 200, and learning device 610, or a part thereof, may be implemented in a computer 700. In this case, the operation of each of the above-described processing units is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program. The CPU 710 also allocates storage areas in the main storage device 720 corresponding to each of the above-described storage units in accordance with the program. Communication between each device and other devices is executed by an interface 740 having a communication function and performing communication under the control of the CPU 710.

[0107] When learning device 100 is implemented in computer 700, the operation of each unit is stored in the form of a program in auxiliary storage device 730. CPU 710 reads the program from auxiliary storage device 730, loads it into main storage device 720, and executes the above-described processing in accordance with the program.

[0108] CPU 710 also allocates a storage area in main memory 720 for learning device 100 to perform processing in accordance with the program. Communication between learning device 100 and other devices is achieved by interface 740, which has a communication function and performs communication under the control of CPU 710. Interaction between learning device 100 and a user is achieved by interface 740, which has a display device and an input device, displaying various images under the control of CPU 710 and accepting user operations.

[0109] When the face authentication device 200 is implemented in a computer 700, the operation of each unit thereof is stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0110] Furthermore, the CPU 710 allocates a storage area in the main memory device 720 for the processing of the face authentication device 200 in accordance with the program. Communication between the face authentication device 200 and other devices is performed by an interface 740 having a communication function and performing communication under the control of the CPU 710. Interaction between the face authentication device 200 and a user is performed by the interface 740 having a display device and an input device, displaying various images under the control of the CPU 710, and accepting user operations.

[0111] When the learning device 610 is implemented in the computer 700, the operation of each unit is stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0112] Furthermore, CPU 710 allocates a storage area in main memory 720 for the learning device 610 to perform processing in accordance with the program. Communication between learning device 610 and other devices is performed by interface 740, which has a communication function and performs communication under the control of CPU 710. Interaction between learning device 610 and a user is performed by interface 740, which has a display device and an input device, displaying various images under the control of CPU 710 and accepting user operations.

[0113] One or more of the above-described programs may be recorded on nonvolatile recording medium 750. In this case, interface 740 may read the programs from nonvolatile recording medium 750. CPU 710 may then directly execute the programs read by interface 740, or may temporarily store the programs in main storage device 720 or auxiliary storage device 730 and then execute them.

[0114] Note that a program for executing all or part of the processing performed by learning device 100, face recognition device 200, and learning device 610 may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed to perform the processing of each part. Note that the term "computer system" here includes hardware such as an OS (Operating System) and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CD-ROMs (Compact Disc Read Only Memory), as well as storage devices such as hard disks built into computer systems. The program may be one that realizes part of the aforementioned functions, or may be one that can realize the aforementioned functions in combination with a program already stored in the computer system.

[0115] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Industrial Applicability]

[0116] The present invention may be applied to a learning device, a symmetry evaluation device, a learning method, and a recording medium. [Explanation of symbols]

[0117] 100, 610 Learning Device 110, 611 Training data acquisition section 120, 612 Image inversion section 130, 613 Feature point coordinate calculation unit 131 Convolutional Neural Networks 132 Fully connected neural network 140 Coordinate inversion section 150, 615 Learning control unit 151 Loss function calculation unit 152 Learning loss calculation unit 153 Symmetrical loss calculation unit 154 Update amount calculation section 200 Facial Recognition Device 210 Image acquisition unit 220 Face detection unit 240 Normalization section 250 Feature Extraction Unit 260 Collation Unit 270 Output section 614 Symmetry evaluation value calculation unit

Claims

1. training data acquisition means for acquiring training data including an original image; an image inverting means for generating an inverted image by inverting the original image in line symmetry; a feature point coordinate calculation means for calculating the coordinates of feature points of the original image and the coordinates of feature points of the inverted image; a symmetry evaluation value calculation means for calculating a symmetry evaluation value indicating the degree of difference between the coordinates of the feature points of the original image and the coordinates of the feature points of the reversed image when coordinate transformation is performed so as to cancel out the difference in coordinates caused by the influence of image inversion; a learning control means for controlling learning of the feature point coordinate calculation means so that the degree of difference indicated by the symmetry evaluation value becomes smaller; A learning device comprising:

2. the feature point coordinate calculation means calculates the coordinates of the feature points of the inverted image using a second model having the same structure as a first model used to calculate the coordinates of the feature points of the original image; the learning control means controls learning of the feature point coordinate calculation means so as to set the same parameter values ​​to the first model and the second model. The learning device according to claim 1 .

3. a first learning evaluation value calculation means for calculating a first learning evaluation value indicating a degree of difference between coordinates of feature points of the original image and correct values ​​of coordinates of feature points of the original image indicated in the training data; a second learning evaluation value calculation means for calculating a second learning evaluation value indicating a degree of difference between the coordinates of the feature points of the inverted image and the correct value when coordinate transformation is performed so as to cancel out the difference in coordinates between the feature points of the inverted image and the correct value due to the influence of image inversion; Furthermore, the learning control means controls learning of the feature point coordinate calculation means so that a comprehensive evaluation value based on the symmetry evaluation value, the first learning evaluation value, and the second learning evaluation value becomes smaller. The learning device according to claim 1 or 2.

4. The computer Acquire training data including an original image and correct coordinate values ​​of feature points of the original image; generating an inverted image by inverting the original image in a line symmetric manner; Calculating the coordinates of the feature points of the original image and the coordinates of the feature points of the inverted image; calculating a symmetry evaluation value indicating the degree of difference between the coordinates of the feature points of the original image and the coordinates of the feature points of the inverted image when coordinate transformation is performed so as to cancel out the difference in coordinates caused by the effect of image inversion between the coordinates of the feature points of the original image and the coordinates of the feature points of the inverted image; controlling learning of a model that calculates the coordinates of the feature points so that the degree of difference indicated by the symmetry evaluation value becomes smaller; A learning method that includes:

5. On the computer, Obtaining training data including an original image and correct values ​​of coordinates of feature points of the original image; generating an inverted image by inverting the original image in an axisymmetric manner; Calculating coordinates of feature points of the original image and coordinates of feature points of the inverted image; calculating a symmetry evaluation value indicating the degree of difference between the coordinates of the feature points of the original image and the coordinates of the feature points of the inverted image when coordinate transformation is performed so as to cancel out the difference in coordinates caused by the influence of image inversion between the coordinates of the feature points of the original image and the coordinates of the feature points of the inverted image; Controlling learning of a model that calculates coordinates of the feature points so that the degree of difference indicated by the symmetry evaluation value becomes smaller; A program to execute.

Images