Adaptive image cropping for face recognition

Abstract

By adding a side network to the face recognition network, the output of early convolutional blocks can be used to determine relative bounding box values. The relative bounding box values ​​can be used to refine existing bounding box values ​​with a view to improving the embedding vector generated by the face recognition network.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. patent application serial number 16 / 355,665, filed on March 15, 2019, entitled “ADAPTIVE IMAGE CROPPING FOR FACE RECOGNITION,” the contents of which are incorporated herein by reference as if reproduced in full. Technical Field

[0003] The present application relates generally to face recognition and, more particularly, to adaptive cropping of images in the context of extracting facial features in the images, wherein the features are used for face recognition. Background Art

[0004] A face recognition system is a technology that is able to identify or verify a person from a digital image or video frame from a video source.

[0005] Face recognition has been widely studied in computer vision. Recently, face recognition systems that perform face recognition have been greatly improved by using convolutional neural networks combined with deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Reference is now made, by way of example, to the accompanying drawings which illustrate example implementations, in which:

[0007] Figure 1 The face recognition device provided by the present application is shown;

[0008] Figure 2 It shows that the present application can provide Figure 1 A joint face comparison and recognition system implemented in a face recognition device, the joint face comparison and recognition system comprising an image processing unit, a face recognition network and a side network with multiple branches;

[0009] Figure 3 This application shows that Figure 2 The first branch of the side network;

[0010] Figure 4 This application shows that Figure 2 The second branch of the side network;

[0011] Figure 5 This application shows that Figure 2 The third branch of the side network;

[0012] Figure 6 This application shows that Figure 2 An image processing unit;

[0013] Figure 7The spatial relationship between a pair of bounding boxes is shown. DETAILED DESCRIPTION

[0014] Existing face recognition systems include a face detection network and a face recognition network. Typically, the face detection network passes a face image to the face recognition network. The face recognition network extracts facial features from the face image received from the face detection network. The face recognition network outputs an "embedding," a real-valued multidimensional vector that defines the features of the face in the image. Another element (not shown) can compare the output embedding to a dataset of embeddings associated with known faces.

[0015] The face image passed from the face detection network to the face recognition network can be understood as being associated with a bounding box. The way the bounding box is defined determines how well the face is aligned in the image and how big the margins around the face are.

[0016] In many cases, errors on the part of the face detection network (e.g., errors in matching faces in images) can propagate to the face recognition network.

[0017] Unfortunately, existing face recognition systems cannot optimize and align images in one network. In fact, most existing face recognition systems have a network for face recognition and a network for detecting and extracting facial key points. These networks then use formulas to align input images. Therefore, existing face recognition systems cannot provide the best edge for the face recognition network.

[0018] Aspects of the present application can be shown to improve the edges and alignment provided by any face detection algorithm for face recognition tasks on face images.

[0019] It can be shown that when the alignment method used in the face detection network is the same as the alignment method used in the face recognition network, the performance of the system combining the face detection network and the face recognition network tends to be optimized. However, in practice, the two networks often use different alignment methods.

[0020] Various aspects of the present application relate to an integrated and general method for extracting facial features and improving alignment simultaneously. Other aspects of the present application relate to improving the bounding box of a facial image before passing it to a face recognition network.

[0021] It can be shown that, in accordance with aspects of the present application, the ability to improve the edges and alignment of a source image (expressed as bounding box improvements) can improve the accuracy of any face recognition network.

[0022] The face recognition method described in this application is based on feature embedding of facial images, which can be found in existing patents and published patent applications. In U.S. Patent No. 7,515,740, it is proposed to use principal component analysis (PCA) to determine feature maps. U.S. Patent No. 7,054,468 proposes the use of FISHERFACES kernel for face recognition. In Chinese Patent Application Publication CN104866810A, a face recognition method based on a deep convolutional neural network is proposed. A face recognition method and device are described in U.S. Patent Application Publication No. 2016 / 0379041A1 and U.S. Patent Application Publication No. 2017 / 0046563A1. In U.S. Patent Application Publication No. 2017 / 0262695A1, a system for face detection, representation and recognition is proposed.

[0023] Various aspects of the present application are used to perform bounding box adjustments on an image to be re-input into a face recognition network. The output of the face detection network includes an initial bounding box for the face recognition network. Various aspects of the present application are used to improve the bounding box generated by the face detection network until the bounding box has been optimized for the face recognition task.

[0024] In summary, aspects of the present application relate to a joint face alignment and recognition system 114, including a face recognition network, a side network, and provide a new loss function for training the joint face alignment and recognition system 114. In aspects of the present application, the side network can adjust the edges of the face image using the bounding box output from the face detection network.

[0025] According to one aspect of the present invention, there is provided a method for operating a face recognition system, the face recognition system comprising a side network and a face recognition network for receiving a cropped image of a face and generating an embedding vector representing features of the face, the face recognition network being implemented as a convolutional neural network, the convolutional neural network comprising a series connection of a plurality of face recognition convolutional blocks terminating in a face recognition fully connected network, the cropped image of the face having a cropped bounding box relative to an original bounding box of the original image of the face. The method comprises: receiving in the side neural network an output of a face recognition convolutional block selected from the plurality of face recognition convolutional blocks; processing in the side neural network the output to generate a relative bounding box value; and defining a new bounding box of the image using the relative bounding box value. In other aspects of the present application, a face recognition device comprising a processor is used to perform the method, and a computer-readable medium is provided for adapting a processor in the face recognition device to perform the method.

[0026] Other aspects and features of the present invention will become apparent to those of ordinary skill in the art upon reviewing the following description of specific implementations of the invention in conjunction with the accompanying drawings.

[0027] Figure 1 A face recognition device 104 provided in accordance with at least one example aspect of the present application is shown. An image acquisition device 102 (e.g., a camera) may capture an image of a face and transmit the image including the face to the face recognition device 104. The face recognition device 104 may compare features found in the image with features found in a registered image pre-stored in a registered image database 106, thereby performing face recognition. In one example, a user of the face recognition device may register an image of the user's face. The registered image database 106 may store the image of the user's face as a registered image. The face recognition device 104 is shown to include a processor 108 and a memory 110. The memory 110 stores instructions for implementing a face detection network 112 and a joint face comparison and recognition system 114.

[0028] The face recognition device 104 can be implemented as follows: Figure 2 A joint face alignment and recognition system 114 is shown.

[0029] Figure 2 The illustrated joint face matching and recognition system 114 includes a face recognition network 200 . Figure 2 The illustrated joint face alignment and recognition system 114 also includes various elements designed to optimize the bounding box of the face image passed to the face recognition network 200. Such elements include a side network 210, an image processing unit 202, and a loss function 222. The face recognition network 200 is connected to receive input from the image processing unit 202. In some implementations, the image processing unit 202 can be part of the side network 210.

[0030] Traditionally, the face recognition network 200 includes a series of convolution blocks, including: a first convolution block 204-1, a second convolution block 204-2, ... and an Nth convolution block 204-N. The convolution blocks can be referenced collectively or individually by reference numeral 204. This series of convolution blocks 204 terminates at a face recognition network fully connected network 206. As is well known, a general fully connected network has nodes. A node is a computing unit that has one or more input connections, each of which is associated with a corresponding weight or bias (an activation function that combines the input and output connections in some way). These nodes can also be called neurons, which is where the term "neural network" comes from. The nodes are arranged in layers in a general fully connected network: the first layer (input layer); one or more hidden layers; and the last layer (output layer). According to various aspects of the present application, the nodes of the first layer and the nodes of the hidden layer of the face recognition network fully connected network 206 implement the ReLU activation function, while the nodes of the last layer implement the softmax activation function.

[0031] In the face recognition network 200, the first convolution block 204-1 is connected to the image processing unit 202 on the input side and to the second convolution block 204-2 on the output side. Then, the second convolution block 204-2 is connected to the first convolution block 204-1 on the input side and to the subsequent convolution block 204 on the output side. Next, the Nth convolution block 204-N is connected to the previous convolution block 204 on the input side and to the face recognition network fully connected network 206 on the output side.

[0032] The side network 210 includes a first branch 212-1, a second branch 212-2, ..., and an R-th branch 212-R. These branches may be collectively or individually referenced by reference numeral 212. Figure 2 In the illustrated implementation, the first branch 212-1 is connected to the output side of the first convolution block 204-1 on the input side and to the side network fully connected network 214 on the output side. For example, the second branch 212-2 is connected to the output side of the second convolution block 204-2 on the input side and to the side network fully connected network 214 on the output side. The R-th branch 212-N is connected to the output side of the R-th convolution block 204-R on the input side and to the side network fully connected network 214 on the output side.

[0033] In one exemplary implementation, the face recognition network 200 is implemented according to the known “Inception ResNet V1”. In another exemplary implementation, the face recognition network 200 is implemented according to the known “ResNet 64”. The side network 210 can be customized for each implementation of the face recognition network 200. The implementation based on “Inception ResNet V1” and the corresponding customized side network 210 can be referred to as “JFA-Inception ResNet V1”. The implementation based on “ResNet 64” and the corresponding customized side network 210 can be referred to as “JFA-ResNet64”. In each implementation, “JFA” is an acronym standing for the phrase Joint Face Alignment.

[0034] For the JFA-Inception V1 network, the side network 210 has three branches. Figure 3 As shown, the first branch 212-1 includes a 2×2 max pooling block 304 connected to the output of the first convolutional block 204-1 of the Inception ResNet V1 implementation of the face recognition network 200. The max pooling block 304 has a stride of 2. The max pooling block 304 is followed by a convolutional block 306. The convolutional block 306 has 256 filters of size 3×3.

[0035] like Figure 4 As shown, the second branch 212-2 includes a convolution block 406 having 256 4×4 filters. The convolution block 406 is connected to a third convolution block (not shown) of the Inception ResNet V1 implementation of the face recognition network 200.

[0036] like Figure 5 As shown, the third branch 212-3 of the side network 210 includes a convolution block 506 having 256 filters of size 5×5, connected to the fifth convolution block (not shown) of the Inception ResNet V1 implementation of the face recognition network 200.

[0037] It should be noted that the general branch 212 may include a combination of a resizing block, a maximum pooling block, and a convolution block.

[0038] The outputs of the three branches 212-1, 212-2, and 212-3 are cascaded into a combined output vector. A 1×1 convolution block (not shown) is applied to the combined output vector, the output of the 1×1 convolution block is flattened, and the side network fully connected network 214 maps the output of the 1×1 convolution block to four outputs: z1, z2, z3, and z4. As described below, the four outputs can be used to estimate the coordinates of the corners of the refined bounding box.

[0039] For JFA-ResNet 64, the side network 210 is implemented by three branches 212. The first branch 212-1 is connected to the output of the first convolutional block 204-1 of the ResNet 64 implementation of the face recognition network 200. The second branch 212-2 is connected to the output of the third convolutional block (not shown) of the ResNet 64 implementation of the face recognition network 200. The third branch (not shown) is connected to the output of the fifth convolutional block (not shown) of the ResNet 64 implementation of the face recognition network 200.

[0040] The outputs of the three branches 212 are concatenated into a combined output vector. A 1×1 convolution block (not shown) is applied to the combined output vector, the output of the 1×1 convolution block is flattened, and the side network fully connected network 214 maps the output of the 1×1 convolution block to four outputs: z1, z2, z3, and z4.

[0041] Figure 6 An example arrangement of the image processing unit 202 is shown. Figure 6 In the arrangement of , the image processing unit 202 includes a batch of images 602 connected to an image picker 604, which in turn is connected to an image cropper 606. The image cropper 606 is also connected to the side network fully connected network 214 and the first convolutional block 204-1.

[0042] In operation, in the face recognition network 200, the first convolution block 204-1 receives an image from the image processing unit 202 and passes the output to the second convolution block 204-2. The second convolution block 204-2 then passes the output to the subsequent convolution block 204. The previous convolution block 204 then passes the output to the Nth convolution block 204-N. The Nth convolution block 204-N passes the output to the face recognition network fully connected network 206. As is conventional, the image from the image processing unit 202 has attributes such as width (in pixels) and height (in pixels) and possibly several color channels. The output passed by the Nth convolution block 204-N to the face recognition network fully connected network 206 is expected to be a vector of a predetermined number of elements, which matches the number of input nodes in the face recognition network fully connected network 206.

[0043] The face recognition network fully connected network 206 receives the vector from the Nth convolutional block 204-N at the first layer. The face recognition network fully connected network 206 processes the received vector to generate a feature embedding. In the face recognition network fully connected network 206, the node layer before the softmax layer can be called a feature layer. The feature layer generates a feature embedding.

[0044] As described above, the nodes in the last layer of the face recognition network fully connected network 206 implement the softmax activation function. Therefore, it can be understood that the face recognition network fully connected network 206 generates a cross entropy loss. The cross entropy loss of the last layer (softmax layer) is referred to as "softmax loss" in this article. It should be noted that the softmax loss that relies on deep convolutional neural networks is generally unable to make useful improvements to the accuracy of embedding.

[0045] To address this problem, several loss functions are considered, including the loss functions called center loss, large margin softmax loss, angular softmax loss, and triplet loss. As we all know, these loss functions aim to maximize the inter-class distance and minimize the intra-class distance.

[0046] While triplet loss, center loss, large margin softmax loss, and angular softmax loss have useful effects on the training of networks of the face recognition network 200 type, these losses are usually applied to the feature layer and the last layer. Nevertheless, all of these losses are constrained based on the embedding vector, and the early layers build the feature layer. Therefore, loss functions that can affect the early layers can speed up training and also improve accuracy.

[0047] In various aspects of the present application, the structure induces the discriminative power of a characteristic layer from an earlier layer.

[0048] In particular, the discriminative power of feature embedding can be improved using face matching and detection methods. The loss functions cited above consider the features at the output of the Nth convolutional block 204-N. Unlike these loss functions, the methods found in various aspects of the present application are based on the consideration of multiple convolutional blocks 204.

[0049] In this way, it can be shown that estimating some parameters of the probability distribution function of the data can improve the accuracy of the softmax layer in the fully connected network 206 of the face detection network.

[0050] The relationship between the estimation and classification of the early convolutional blocks 204 of the face recognition network 200 can be improved to build a more powerful feature vector. The side network 210 is used to align the input image for subsequent transmission to the face recognition network 200 for embedding extraction. The side network 210 also generates a feature vector as part of the estimated bounding box. The feedback loop represented by the output transmission from the side network 210 to the image processing unit 202 is used to refine the bounding box of the input face image to improve the bounding box of the face recognition network 200.

[0051] It is contemplated that the face recognition network 200 is implemented as a deep convolutional neural network for face recognition, with softmax as its last layer and the center loss can be obtained from the feature layer.

[0052] It can be shown that aspects of the present application improve the relationship between bounding box estimation and classification of the convolutional block 204 of the face recognition network 200 and generate more powerful feature vectors to improve recognition overall. This is achieved by using the side network 210 to detect faces and return refined bounding boxes.

[0053] During training, the weights and biases used in the joint face alignment and recognition system 114 are adjusted based on the optimized loss function determined when processing the training data, and images with refined bounding boxes are returned to the face recognition network 200 to improve the learning of facial features. Subsequently, at test time, images with refined bounding boxes can be used to improve reasoning. This process is iterative, so that improved feature vectors can improve bounding box estimates, which in turn produce increasingly refined feature vectors.

[0054] As is well known, training a neural network can be based on a sequence of stages called feedforward, loss, and backpropagation. The loss stage involves determining the value of a loss function. The backpropagation stage involves adjusting various aspects of the neural network being trained to try to minimize the value of the loss function.

[0055] It can be shown that the loss function plays an important role in training a face recognition network, such as face recognition network 200. To train the convolutional block 204 and the branches of the side network 210 of the face recognition network 200, consider a multi-class classification problem with M classes. In this problem, the cross entropy of the softmax layer is given by

[0056]

[0057] Among them, y i is the true label of the i-th data, Represents the activation function of the mth fully connected layer of the i-th data in the last layer of the face recognition network 200.

[0058] It is well known that in order to improve the accuracy of the softmax layer, margin can be added to the activation of the true label to improve the training of the softmax after the feature vector and weight normalization of the last layer. The result of adding margin is called large margin cosine loss (LMCL), denoted as L ms And determined as follows:

[0059]

[0060] Where s is the surface scale parameter and m is the margin added to the softmax loss. In this equation, is the LMCL of the i-th input.

[0061] Although the margin is added to improve the discriminative power of the feature, the center loss can be directly added to the feature layer to improve the feature layer. Assume that the feature layer is the last layer before the softmax layer in the fully connected network 206 of the face detection network. In addition, assume that the feature layer consists of the i-th data x i Under these assumptions, the center loss can be given by the following equation:

[0062]

[0063] in, is the center of class m, is the feature of the i-th data belonging to the m-th class. m is the number of data belonging to the mth class, and M is the total number of classes. The center loss in equation (3) represents the average value of the variance of the feature vector.

[0064] Face recognition network 200 face recognition "fr" loss function L fr It can be determined by combining equation (2) and equation (3) as follows:

[0065] L fr =L ms +λL cl (4)

[0066] The side network 210 ideally returns the “true” bounding box of the input image to the image processing unit 202 .

[0067] Figure 7 A ground-truth bounding box 702 is shown. During training, a new random cropped bounding box 704 may be selected for the i-th face data.

[0068] It can be seen that if the image is poorly cropped or cropped in an inappropriate manner for the face recognition task, the side network 210 returns a value that can determine a bounding box that is close to the "true" bounding box that needs to be applied to the image. Therefore, a refined version of the image is constructed and applied to the face recognition network 200. Consider an input face image that has been cropped relative to the original face image. The position of the bounding box 704 of the cropped face image is defined in terms of four pairs of coordinates as {(x1, y1), (x1, y2), (x2, y1), (x2, y2)}. It should be noted that the true bounding box 702, that is, the bounding box used to train the face recognition network 200, can be defined in terms of four pairs of coordinates. The goal of the side network 210 is to estimate the geometry of the input image and the In the following, variables w and h are used to represent the width and height of a given image, respectively. For a given image, the side network 210 determines x1, x2, y1, y2, and the following relative bounding box values ​​as follows:

[0069]

[0070] in, and The output of the side network 210 is considered to be a vector of length four relative bounding box values ​​{z1, z2, z3, z4}. Then, the mean square error (MSE) can be applied to refine the bounding box. MSE can be expressed as L bx , and can be used when estimating the value in equation (5) as follows:

[0071]

[0072] Among them, z i，k Based on the i-th input z k Sure, is the kth output of the side network 210 for the i-th input data. Finally, the term Used to represent the MSE loss of the i-th input.

[0073] During inference, the image processing unit 202 may determine and The estimate is as follows:

[0074]

[0075] It should be noted that the ideal output of the side network 210 is the value {z1, z2, z3, z4}. However, these values is the actual output of the side network 210 as it attempts to steadily improve its estimate of the values ​​{z1, z2, z3, z4}. The image processing unit 202 may use the relative bounding box values A refined bounding box is established for the image provided to the face recognition network 200. Thus, during inference, for each image, the side network 210 improves the input image and its bounding box. The new bounding box then crops the face image and is fed again to the face recognition network 200. This process can be considered iterative, such that each successive iteration improves the bounding box. Simulations show that even for very poorly cropped face images, two iterations are generally suitable.

[0076] The total loss function for the i-th input can be expressed as And can be determined as three losses in the loss function 222 and L cl A linear combination of . and The convex hull of can be expressed as follows:

[0077]

[0078] Among them, η i It is called a "hyperparameter" and its actual value is between 0 and 1. During the training phase, the hyperparameter η can be set for each batch of images based on the ratio of poorly cropped images to well-cropped images. i Therefore, the expression in equation (4) can be expanded based on equation (8) to give the following equation:

[0079]

[0080] During training, the poorly cropped images are processed by the data augmentation unit (as part of the image cropper 606, see Figure 6 Given the ratio of the total cropped image to the cropped original face, when the total loss for each image is determined using Equation (9), the given ratio can be used for η i . Training starts with η close to 1 i start.

[0081] For any face image, the bounding box does not change the identity of the person to whom the face belongs. Based on this fact, consider the i The face data point of a given identity can be denoted as x. In addition, the transformation group considering all possible bounding box cropping transformations is denoted as G. Then, x belongs to c i The probability that g(x) belongs to c can be expressed as i probability. Hence, we can say that bounding box transformations do not change the identity. Hence, the probability density function of the face is invariant with respect to G. To achieve a classifier function that is invariant with respect to G, we can show that the classifier function should be a function of the maximal invariant statistic. A statistic denoted m(x) can be considered to define a maximal invariant statistic if it satisfies the following two conditions: (1) a condition that the statistic has the invariance property such that for any g∈G, m(g(x))=m(x); and (2) a condition that the statistic has maximality, where for the statistic to have maximality, for any x1 and x2, a choice is made such that m(x1)=m(x2), then for x2=g(x2), there exists g∈G.

[0082] Consider that the bounding box applied to a face image can be represented as a classifier function g. Therefore, the operation of refining the bounding box can be viewed as an operation that approximates the maximum invariant statistic. This is true because the two conditions for defining the maximum invariant statistic described above are satisfied by the operation of refining the bounding box. To satisfy the first condition, that the statistic has the invariance property, we can show that the facial data points inside the refined bounding box are also inside the bounding box before refinement. To satisfy the second condition, that the statistic has maximality, we can show that for the first crop and the second crop of the same face image, the facial data points in the refined bounding box are the same, and then there exists a crop that can transform one crop into the other crop.

[0083] By studying the relationship between softmax loss and center loss, it can be shown that these two losses can improve each other. bx The relationship between and the accuracy of softmax.

[0084] The operation of refining the bounding box involves estimating the face alignment parameters of the input face image.

[0085] It can be shown that the estimation of face alignment parameters is useful for performing operations to improve the bounding box of an image provided to the face recognition network 200. It can also be shown that during training, the estimation of face alignment parameters may help the convolution block 204 to improve the accuracy of the softmax layer.

[0086] Consider the maximum likelihood loss. The training goal is to maximize the conditional probability of multiple labels y given a specific x (provided by label unit 224, see Figure 2 ),Right now:

[0087]

[0088] where θ represents the parameters of the probability distribution function (PDF) of label y given a particular x, where Y is the set of all possible y. θ (y|x) is maximized, so during training, p θ The gradient of (y|x) can be set to zero. The Taylor series can be used as the gradient of the maximum likelihood expansion with respect to θ A further approximation can be obtained by using only the gradient It can be shown that for the maximum likelihood of θ, The second term of the Taylor series can be determined as, with respect to θ, Therefore, it can be approximated as follows

[0089]

[0090] in,

[0091]

[0092] It can be shown that when the input data point set x is an independent and identically distributed data set, the function I(θ) tends to the known Fisher information matrix of θ.

[0093] It can be considered as follows L 2 Norm (also called Euclidean norm, Euclidean distance, and L 2 distance):

[0094]

[0095] where u is the upper limit of I(θ).

[0096] From equation (12), it can be inferred that the minimum mean squared error (MMSE) θ improves the training of the maximum likelihood classifier. In other words, g(x, y) tends to the MMSE of θ to achieve the maximum likelihood classifier. Based on this result, if there is some information about the PDF parameter set, the mean squared error (MSE) loss function improves the overall accuracy of the classifier.

[0097] The accuracy of the unknown parameter estimation affects the accuracy of the classifier in the maximum likelihood criterion. It can be seen that in order to improve the accuracy of the face recognition network 200, a good strategy involves training the feature vector of the convolution block 204 so that the parameters of the convolution block 204 tend to the MMSE of the unknown parameters. In this regard, it can be considered that the center loss L cl It can be the naive implementation MSE of the Nth convolutional block 204-N.

[0098] On the other hand, reliable estimation of the unknown parameters of the PDF can improve the performance of the softmax layer. It can be seen that the side network 210 that attempts to estimate some alignment parameters of the face image can improve the training of the intermediate layer feature vectors of the convolution block 204. Therefore, the side network 210 can not only be used for bounding box refinement, but also based on equation (12), it can improve the accuracy of the softmax layer.

[0099] In equation (8), the parameter η i ∈[0, 1] controls the relative contribution of face recognition and bounding box estimation to the loss function Equation (9) during training. In particular, if η i = 1, the bounding box refinement does not participate in the training.i , the contribution of the i-th sample in the bounding box refinement increases. It can be seen that for the case with a complete face, the parameter η i Set to 1, for incomplete faces, the parameter η i Set to the ratio of false matches between the input and the ground-truth face.

[0100] Figure 7 A ground-truth bounding box 702 is shown. During training, a new random cropped bounding box 704 may be selected for the i-th face data. i The value of can be determined as:

[0101]

[0102] Among them, A c，i is the intersection area between the cropped bounding box 704 and the true bounding box 702, where A t，i is the area of ​​the ground-truth bounding box 702 of the i-th face image in training.

[0103] For example, training can be done from η i = 1. Then, by increasing the number of cycles, the cropping of the input image is increased.

[0104] Various aspects of the present application are directed to an integrated approach for bounding box alignment and classification for face recognition based on training of a detection network that provides feature maps for alignment and classification.

[0105] We analyze the probability distribution function (PDF) of faces to show that such PDFs are invariant to cropping transformations. Understanding this invariance enables the development of an iterative approach to bounding box refinement for face recognition.

[0106] Aspects of the present application relate to the relationship between the estimation loss function and the classification loss function, and the combination of these two loss functions from the data PDF, and show that this term improves classification training.

[0107] The side network fully connected network 214 layer returns four relative bounding box values Forms a new bounding box to replace the existing one.

[0108] During training, for each image label of an image showing a face identity with a correct bounding box, a loss function 222 is passed.

[0109] In various aspects of the present application, the side network 210 can find the best edge of the face recognition network so that the network selects the edge.

[0110] This product can be used in a smartphone or PC for high-tech image synthesis capabilities Video surveillance: face search in large data sets for identification and verification purposes. This technology can be used for object detection or recognition, where bounding boxes are used to select inputs and can be improved. Conveniently, it can be shown that aspects of this application can improve any bounding box using training data.

[0111] The above implementations of the present application are only examples. Those skilled in the art may make changes, modifications and variations to the specific implementations without departing from the scope of the present application as defined by the appended claims.

Claims

1. A method for operating a face recognition system, characterized in that: The face recognition system comprises a side neural network and a face recognition network for receiving a cropped image of a face and generating an embedding vector representing features of the face, the face recognition network being implemented as a convolutional neural network, the convolutional neural network comprising a series connection of a plurality of face recognition convolutional blocks terminating in a face recognition fully connected network, the cropped image of the face having a cropped box relative to an original bounding box of an original image of the face, the method comprising: receiving, in the side neural network, an output of a face recognition convolution block selected from the plurality of face recognition convolution blocks; processing the output in the side neural network to generate a relative bounding box value; A new bounding box for the image is defined using the relative bounding box values.

2. The method according to claim 1, characterized in that The side neural network includes multiple branches that implement one or more of a maximum pooling block, a convolution block, and a resizing block.

3. The method according to claim 2, characterized in that Also includes: In the side neural network, the output of each of the plurality of branches is cascaded to generate a combined output vector; The combined output vector is processed to generate the relative bounding box value.

4. The method according to claim 3, characterized in that The processing the combined output vector includes applying a 1×1 convolution block to the combined output vector, thereby generating a 1×1 convolution block output.

5. The method according to claim 4, characterized in that It also includes flattening the 1×1 convolution block output.

6. The method according to any one of claims 1 to 5, characterized in that The side neural network includes branches that implement a maximum pooling block and a side neural network convolution block.

7. The method according to claim 6, characterized in that The maximum pooling block includes a 2×2 maximum pooling block connected to an output of a first face recognition convolution block among the multiple face recognition convolution blocks.

8. The method according to claim 7, characterized in that The max pooling block has a stride of 2.

9. The method according to claim 4, characterized in that The side neural network convolutional block has 256 filters of size 3×3.

10. The method according to claim 4, characterized in that The side neural network convolution block is connected to the output of the maximum pooling block.

11. The method according to claim 6, characterized in that The side neural network includes a branch that implements a side neural network convolutional block with 256 4×4 filters.

12. The method according to claim 11, characterized in that The side neural network convolution block is connected to the third face recognition convolution block among the multiple face recognition convolution blocks.

13. The method according to claim 6, characterized in that The side neural network includes a branch that implements a side neural network convolutional block with 256 filters of size 5×5.

14. The method according to claim 13, characterized in that The side neural network convolution block is connected to the fifth face recognition convolution block among the multiple face recognition convolution blocks.

15. A face recognition device, characterized in that: include: a memory for storing instructions; a processor that executes the instructions to: Implementing a face recognition network, the face recognition network is used to: receiving a cropped image of a face; generating an embedding vector representing features of the face; wherein the face recognition network is implemented as a convolutional neural network, the convolutional neural network comprising a series connection of a plurality of face recognition convolutional blocks terminating in a face recognition fully connected network, the cropped image of the face having a cropping box relative to an original bounding box of an original image of the face; Implement a side neural network to: Receiving an output of a face recognition convolution block selected from the plurality of face recognition convolution blocks; processing the output to generate a relative bounding box value; implementing an image processing unit to: receiving the relative bounding box value; A new bounding box for the image is defined using the relative bounding box values.

16. The face recognition device according to claim 15, characterized in that: The side neural network includes multiple branches that implement one or more of a maximum pooling block, a convolution block, and a resizing block.

17. The face recognition device according to claim 16, characterized in that: The processor is further configured to implement the side neural network to: cascading the output of each of the plurality of branches to generate a combined output vector; The combined output vector is processed to generate the relative bounding box value.

18. A computer readable medium storing instructions, characterized in that: The instructions may be executed by a processor of a face recognition device, and the instructions enable the processor to execute the method according to any one of claims 1 to 14.

19. A computer program storing instructions, characterized in that: The instructions may be executed by a processor of a face recognition device, and the instructions enable the processor to execute the method according to any one of claims 1 to 14.

20. A face recognition device, characterized in that: Comprising means for performing the method according to any one of claims 1 to 14.

Images