Face positioning in images

Abstract

The present specification relates to a method of face localization in images. According to a first aspect of the present specification, a computer-implemented method of training a neural network for face localization is described, the method comprising: inputting a training image to the neural network, the training image comprising one or more faces; processing the training image using the neural network; outputting, from the neural network, one or more sets of output data for each of a plurality of training anchors in the training image, each set of output data comprising a predicted face class, a predicted location of a corresponding face box, and one or more corresponding feature vectors; updating parameters of the neural network according to an objective function. For each positive anchor in the training image, the objective function comprises: a classification loss for comparing a predicted class of the positive anchor with a known class of the positive anchor; a box regression loss for comparing a predicted location of a face box of the positive anchor with a known location of the face box; a feature loss for comparing pixel-based attributes of the one or more feature vectors of the positive anchor with known pixel-based attributes of a face associated with the positive anchor.

Description

Technical Field

[0001] This specification relates to methods for face localization in images. Specifically, this specification relates to the use and training of neural networks for face localization in images. Background Technology

[0002] Face localization in an image is the process of determining whether a face exists in an image and its location. Face localization in an image can also include identifying bounding boxes to determine the location of each face present in the image. In many applications, such as facial attribute (e.g., expression and / or age) and / or facial identification, automatic face localization is a prerequisite step in face image analysis. Typically, training a neural network (face classification / detection neural network) to perform face localization in an image involves using only classification and bounding box regression losses.

[0003] Face detection differs from general object detection in several ways. Face detection has a smaller ratio variation (e.g., from 1:1 to 1:1.5), but a much larger scaling variation (e.g., from a few pixels to several kilopixels). Using a general object detector for face detection and localization presents significant challenges. Summary of the Invention

[0004] According to a first aspect of this specification, a computer-implemented method for training a neural network for face localization is described, the method comprising: inputting a training image into the neural network, the training image including one or more faces; processing the training image using the neural network; outputting one or more sets of output data from the neural network for each of a plurality of training anchor points in the training image, each set of output data including a predicted face classification, a predicted location of a corresponding face bounding box, and one or more corresponding feature vectors; and updating the parameters of the neural network according to an objective function. For each positive anchor point in the training image, the objective function includes: a classification loss for comparing the predicted classification of the positive anchor point with a known classification of the positive anchor point; a bounding box regression loss for comparing the predicted location of the face bounding box of the positive anchor point with a known location of the face bounding box; and a feature loss for comparing pixel-based attributes of the one or more feature vectors of the positive anchor point with known pixel-based attributes of the face associated with the positive anchor point.

[0005] The one or more feature vectors may include the predicted locations of multiple facial feature points of a face, wherein the feature loss includes a facial feature point regression loss, used to compare the predicted locations of the multiple facial feature points with the known locations of the multiple facial feature points.

[0006] The one or more feature vectors may include an encoded representation of a face within a predicted face bounding box, and the method may further include: generating a three-dimensional representation of the face using a mesh decoder neural network, the mesh decoder neural network including one or more geometric convolutional layers; generating a two-dimensional image of the face from the three-dimensional representation of the face using a differentiable renderer, wherein the feature loss includes a dense regression loss that compares the generated two-dimensional image of the face with a measured image of the face within the predicted face bounding box.

[0007] The feature vector may also include camera parameters and / or lighting parameters, wherein the differentiable renderer uses the camera parameters and / or lighting parameters when generating the two-dimensional image of the face from the three-dimensional representation of the face.

[0008] The dense regression loss can be the pixel difference between the generated two-dimensional image of the face and the measured image of the face within the predicted face bounding box.

[0009] The grid decoder may include multiple geometric convolutional layers and multiple magnification layers, wherein the multiple magnification layers are interleaved with the multiple geometric convolutional layers.

[0010] For each negative anchor point in the training image, the objective function includes a classification loss for comparing the predicted classification of the negative anchor point with its known classification.

[0011] The method may further include iterating the method using one or more other training images until a threshold condition is met.

[0012] The neural network may include: a first plurality of convolutional layers, including an input layer, a plurality of convolutional filters, and one or more skip connections; and a second plurality of convolutional layers, laterally connected to the first plurality of convolutional layers, for processing the outputs of the first plurality of convolutional layers in a top-down manner. The second plurality of convolutional layers may include one or more deformable convolutional layers.

[0013] Each lateral connection can be used to merge the output of one of the first plurality of convolutional layers with the output of the previous layer of the second plurality of convolutional layers.

[0014] The neural network may also include one or more context modules, each context module being used to process the output of one of the second plurality of convolutional layers using one or more deformable convolutional networks.

[0015] According to another aspect of this specification, a face localization method is described, the method comprising: using a neural network to identify one or more faces in an input image; and using the neural network to identify one or more corresponding face bounding boxes in the input image, wherein the neural network has been trained using one of the training methods described herein.

[0016] According to another aspect of this specification, a computer-implemented face localization method is described, the method comprising: inputting an image, the image including one or more faces, into a neural network; processing the image using the neural network; and outputting one or more sets of output data from the neural network, each set of output data including a predicted face classification, a predicted location of a corresponding face bounding box, and one or more corresponding feature vectors. The neural network includes: a first plurality of convolutional layers, including an input layer, a plurality of convolutional filters, and one or more skip connections; and a second plurality of convolutional layers, laterally connected to the first plurality of convolutional layers, for processing the outputs of the first plurality of convolutional layers in a top-down manner.

[0017] Each lateral connection can be used to merge the output of one of the first plurality of convolutional layers with the output of the previous layer of the second plurality of convolutional layers. The second plurality of convolutional layers may include one or more deformable convolutional layers.

[0018] Neural networks may also include one or more context modules, each of which is used to process the output of a layer in a second or more convolutional layers using one or more deformable convolutional networks.

[0019] Neural networks can be trained using any of the training methods described in this article.

[0020] According to another aspect of this specification, a system is described, the system comprising: one or more processors; and a memory including computer-readable instructions that, when executed by the one or more processors, cause the system to perform one or more of the methods described herein.

[0021] According to another aspect of this specification, a computer program product is described, the computer program product including computer-readable instructions that, when executed by a computing device, cause the computing device to perform one or more of the methods described herein.

[0022] As used herein, the terms “face localization” and / or “face location” are preferably used to refer to one or more of the following: face detection; face alignment; pixel-based face resolution; and / or 3D dense correspondence regression. Attached Figure Description

[0023] Embodiments will now be described by way of non-limiting examples with reference to the accompanying drawings, in which:

[0024] Figure 1 A schematic overview of methods for facial localization in an image is shown;

[0025] Figure 2 A schematic overview of a method for training a neural network using multi-task loss to perform face localization in an image is shown;

[0026] Figure 3 A flowchart illustrating a method for training a neural network to perform face localization in an image is shown;

[0027] Figure 4 An example of a neural network structure for face localization in an image is shown;

[0028] Figure 5 An example of lateral connections in a neural network used for face localization in an image is shown;

[0029] Figure 6 Another example of a neural network architecture for face localization in an image is shown;

[0030] Figure 7 An example of a computer system for performing facial localization is shown. Detailed Implementation

[0031] In the embodiments described herein, the facial recognition neural network can be used to output face scores (i.e., indicators of the presence or absence of a face at a location) and face bounding boxes, as well as other feature vectors. For example, the facial recognition neural network can output the locations of one or more facial feature points. Alternatively, the facial recognition neural network can output the embedding of regions within the face bounding box. These outputs can be used in a multi-task loss function to train the facial recognition neural network.

[0032] Compared to other face localization methods, using this multi-task loss function can improve the performance of face classification / detection neural networks. For example, face classification / detection neural networks can have a lower error rate.

[0033] In some embodiments, a lightweight backbone network is employed. This allows the face classification / detection neural network to run in real time on a single CPU core.

[0034] Figure 1A schematic overview of a method 100 for face localization in an image is shown. An image 102, including one or more faces 104, is input into a neural network 106. The neural network 106 processes the image through multiple neural network layers and outputs one or more sets of output face data 108. Each set of output face data 108 includes a predicted face classification of the face 106 in the image 102, the predicted location of the corresponding face bounding box, and one or more corresponding feature vectors encoding one or more attributes of the corresponding face in the image 102.

[0035] Input image 102 (i.e., I) is an image including one or more human faces. Input image 102 may include a set of pixel values ​​from an array (e.g., a two-dimensional or three-dimensional array). For example, in a color image In this context, H is the height of the image (in pixels), W is the width of the image (in pixels), and the image has three color channels (e.g., RGB or CIELAB). In some embodiments, image 102 may be black and white / grayscale.

[0036] The neural network 106 comprises multiple layers of nodes, each associated with one or more parameters. The parameters of each node in the neural network may include one or more weights and / or biases. A node takes one or more outputs from a node in the previous layer as input. The outputs of the node in the previous layer are used by the node to generate activation values ​​using an activation function and the parameters of the neural network. One or more layers of the trained generator neural network 106 may be convolutional layers. One or more layers of the trained generator neural network 106 may be deformable convolutional layers.

[0037] In some embodiments, the neural network 106 may include a residual network and a feature pyramid network. The neural network 106 may include lateral connections between layers of the residual network and layers of the feature pyramid network. The feature pyramid can process the output of the residual network in a top-down manner. The following describes... Figures 4 to 6 Describe an example of a neural network architecture that uses residual networks and feature pyramids.

[0038] The parameters of neural network 106 can be used with a multi-task objective function. (This is also referred to as the loss function in this paper) is used for training. The multi-task objective function can include two or more of the following: face classification loss, face bounding box regression loss, facial landmark regression loss, and / or dense face regression loss. The multi-task loss function can include supervised loss and / or self-supervised loss. The following combines... Figure 2 and Figure 3 Examples describing methods for training neural networks.

[0039] For each potential face identified in image 102 by the neural network, a set of output facial data 108 includes a predicted face classification 108a for the face 106 in image 102. The predicted classification 108a includes the probability that an anchor point in the image is a face.

[0040] For each potential face identified by the neural network in image 102, a set of output face data 108 also includes a predicted bounding box 108b of the face 106 in image 102. For example, the predicted bounding box 108b may be the coordinates of the corner points of the bounding box 110 of the potential face.

[0041] For each potential face identified in image 102 by the neural network, a set of output facial data 108 also includes one or more feature vectors 108c encoding one or more attributes of the face. For example, feature vector 108c may include the location of one or more facial feature points, such as the eyes, nose, chin, and / or mouth on a face. Alternatively, feature vector 108c may include nesting of faces contained within bounding box 110.

[0042] Figure 2 A schematic overview of a method for training a neural network using a multi-task loss function to perform face localization in an image is shown. The method includes training a neural network 106 on a set of training images 102 using a multi-task loss function 204. For each of the multiple images 202 in the training dataset, the training image 202 is input into the neural network 106 and processed through multiple neural network layers. The neural network 106 outputs a set of predicted face data 206 based on the input training images 202. The predicted face data 206 is compared with known face data of the corresponding training image 202 using the multi-task loss function 204, and this comparison is used to update the parameters of the neural network 106.

[0043] Training image 202 is obtained from the training set. The training set includes multiple images containing human faces, and a set of known facial data for each image. The known facial data includes the location of each face in the corresponding image, and the location of the bounding box of each face in the corresponding image. In some embodiments, the known facial data also includes the location of one or more facial feature points for each face in the corresponding image.

[0044] Each training image can be associated with multiple training anchors. Each anchor represents a "prior" bounding box in the image where a face may be present. Each anchor can be defined as a reference box including its center location and scale. However, it should be understood that anchors can be defined in other ways. Anchors are used for face detection to distinguish between positive and negative training samples. A positive anchor is defined as an anchor where a face is present. A negative anchor is defined as an anchor where a face is not present. An example of using anchors is provided in the following literature: "Faster R-CNN: Towards Real-Time Object Detection with Regional Proposal Networks" (Shaoqing Ren et al., arXiv:1506.01497, the contents of which are incorporated herein by reference).

[0045] The predicted facial data 206 includes facial classification 210. The predicted facial classification 210 includes the probability p that the associated anchor point i is a face. i In some embodiments, the predicted face classification 210 may have two parts corresponding to "faces present" and "faces not present".

[0046] Anchor point prediction for face classification 210 and classification loss Related, among which, The anchor point i is the known face classification (i.e., 1 represents a positive anchor point, and 0 represents a negative anchor point). The classification loss compares the predicted classification of the anchor point with its known classification. For example, the classification loss could be a softmax loss in the binary case, such as cross-entropy loss or binary hinge loss. However, those skilled in the art will understand that other loss functions can be used. In some embodiments, the classification loss may be used only for positive anchor points. In other embodiments, the classification loss may be used for both positive and negative anchor points.

[0047] Predicted facial data 206 includes predicted face bounding box locations 212 (i.e., t i The predicted face bounding box position 212 includes data on the predicted position of the bounding box of the corresponding face for identifying anchor point i. The predicted face bounding box position 212 can be, for example, the coordinates of the corner points of the predicted bounding box for the face, such as the coordinates of all four corner points of the box or the coordinates of the two opposite corner points of the box's diagonals. Alternatively, the predicted face bounding box position 212 can be the coordinates of the center of the bounding box, as well as the height and width of the box. Other examples are also possible.

[0048] Predicted face bounding box position of positive anchor point 212 and bounding box regression loss Related, among which, It is the known face bounding box position of the positive anchor point i. The bounding box regression loss will predict the face bounding box position 212 (i.e., t).i The known position of the corresponding face bounding box Comparisons can be made. For example, the box regression loss can be a robust loss function, such as a smoothed L1 loss function. However, those skilled in the art will understand that other loss functions can be used. In some embodiments, the box regression objective can be normalized to the center position of the box as well as its width and height.

[0049] The predicted facial data also includes one or more predicted feature vectors, each encoding one or more pixel-based attributes of the region with the face bounding box. These feature vectors are then compared with the feature loss. In connection, the feature loss is used to compare the pixel-based properties of one or more feature vectors of the positive anchor point with the known pixel-based properties of the face associated with the positive anchor point.

[0050] An example of such a feature vector is facial feature point data 214. Facial feature point data 214 includes the predicted coordinates / positions of one or more facial feature points within the predicted face bounding box. For example, five facial feature points corresponding to the eyes, corners of the mouth, and nose could be used.

[0051] Facial feature point data 214 and facial feature point regression loss Related, among which, This is the known facial feature point location of the positive anchor point i. The facial feature point regression loss will predict the facial feature point location 214 (i.e., l). i (and the known locations of the corresponding facial feature points) Comparisons can be made. For example, the facial feature point regression loss can be a robust loss function, such as a smoothed L1 loss function. However, those skilled in the art will understand that other loss functions can be used. In some embodiments, the facial feature point location target can be normalized to the center location.

[0052] Alternatively, the feature vector may include nested 2^16 potential faces within the bounding box (i.e., P). ST Nesting 216 may include shape and texture parameters of the face within the corresponding bounding box. In some embodiments, camera parameters 218 and / or lighting parameters 220 may also be output by neural network 106. Camera parameters 218 may include one or more of the following: camera position; camera pose; and / or camera focal length. Lighting (or illumination) parameters may include one or more of the following: light source position; light source color value; and / or ambient light color. Camera parameters 218 and / or lighting parameters 220 may be referred to as rendering parameters.

[0053] Nested 216 can be used with dense regression loss function L pixelRelatedly, the dense regression loss does not directly compare the nesting 216 with the known nesting. Instead, the nesting is post-processed to generate a predicted face image 222, which is then compared with the potential face image (i.e., the image within the bounding box) using a regression loss function. Therefore, this branch of the process can be considered self-supervised.

[0054] To generate predicted facial images 222, a mesh decoder neural network 224 can be used to process nesting to generate a 3D representation of the potential face, such as a 3D mesh. The mesh decoder neural network 224 comprises multiple layers of nodes, each associated with one or more parameters. The parameters of each node in the neural network may include one or more weights and / or biases. A node takes one or more outputs from nodes in the previous layer as input. The outputs of nodes in the previous layer are used by the node to generate activation values ​​using an activation function and the parameters of the neural network. The mesh decoder neural network 224 includes one or more geometric convolutional layers (also referred to herein as mesh convolutional layers). Geometric convolutional layers are a type of convolutional filter layer in neural networks used in geometric deep learning that can be directly applied to the mesh domain.

[0055] The mesh decoder may also include one or more magnification layers to magnify the mesh (i.e., increase the number of vertices in the mesh). Magnification layers can be interleaved with geometric convolutional layers. For example, geometric convolutional layers can alternate with magnification layers.

[0056] The mesh decoder 224 takes nested parameters 216 as input and processes the input using a series of neural network layers to output a shape map and texture map of a two-dimensional potential face from the training images. The output of the mesh decoder neural network can be represented by the symbol... The shape and texture map includes three-dimensional shape and texture information associated with the input visual data. In some embodiments, the output is a set of parameters describing the (x, y, z) coordinates of points in the grid and the corresponding (r, g, b) values ​​for each point in the grid. In some embodiments, the output may be a colored grid within a unit sphere.

[0057] The trellis decoder can be pre-trained, and its parameters can be fixed during the training of the neural network 106. Examples of trellis decoder neural networks and how to train them can be described in conjunction with co-pending GB patent application No. 902524.6 (the contents of which are incorporated herein by reference).

[0058] A mesh can be based on an undirected connected graph. Define, where, It is a set of n vertices containing joint shape (e.g., (x, y, z)) and texture (e.g., (r, g, b)) information, ε∈{0,1} n×nIt is an adjacency matrix that defines the connection state between vertices. However, it should be understood that an alternative representation of a mesh can be used.

[0059] Examples of convolution operators on graphs / grids can be achieved using recursive polynomials (e.g., Chebyshev polynomials) via kernel g. θ Define the mesh filter. When defining this convolution operator on a mesh, it is useful to define some intermediate variables as follows. The non-normalized graph Laplacian can be defined as... in, It is a diagonal matrix, where D ii =∑ j ε ij Then, the normalized graph Laplace can be defined as L = I n -D -1 / 2 εD -1 / 2 , among which, I n It is an identity matrix.

[0060] Laplace can be diagonalized using Fourier bases. Make L=U∧U T ,in, Then, the graph Fourier representation of a given grid can be defined as The reciprocal of.

[0061] Given these definitions, the convolution operator g on a graph / mesh can be defined. θ Example of filter g. θ It can be parameterized as a K-order truncated Chebyshev polynomial expansion, such as

[0062]

[0063] in, It is a vector of Chebyshev coefficients. It is in the scaled Laplace The k-th order Chebyshev polynomial is obtained at time T. k (x) can use T k (x)=2xT k-1 (x)-T k-2 (x) is calculated recursively, where T0(x) = 1 and T0(x) = 1.

[0064] Then, spectral convolution can be defined as

[0065]

[0066] in, It is input, and This filtering operation is highly efficient, requiring only... Calculation.

[0067] Alternatively, other types of geometric / mesh convolutions (or graph convolutions) can be used. For example, other sets of orthogonal polynomials can be used instead of Chebyshev polynomials.

[0068] The joint shape and texture map are further processed using a differentiable renderer 226 to generate a 2D rendered representation 222 of the potential face within the bounding box. The representation may include a two-dimensional projection of the combined shape and texture map onto the image plane. Rendering parameter sets 218 and 220 can be used to render representations of two-dimensional visual data, such as parameters for defining the camera model used in rendering; that is, the 2D rendering representation 222 is obtained through... For example, a differentiable renderer 226 could be used to render a colored mesh. A 3D mesh renderer that projects onto a 2D image plane. An example of such a differentiable renderer226 can be found in Genova et al. (CVPR, 2018.2,3), but other renderers may also be used.

[0069] Dense regression loss L pixel The rendered 2D face 222 (i.e., The predicted face image is compared to the original face image in training image 202 (i.e., the measured image within the predicted face bounding box). Pixel differences can be used to perform the comparison. An example of dense regression loss is given as follows:

[0070]

[0071] Among them, I * This refers to anchor point clipping of the training image based on measured values, where H and W represent the width and height of the clipped anchor points, respectively. ∥x∥ n It is the Ln norm, such as the L-1 norm. Alternatively, other pixel loss functions can be used.

[0072] The multi-task objective function 204 comprises a combination of classification loss, bounding box regression loss, and feature loss. The goal of training is to minimize this multi-task objective function for each training anchor point i. An example of the multi-task loss function for anchor point i is:

[0073]

[0074] Where, {λ i} are constants that control the relative importance of different loss functions in the multi-task objective function. For example, these parameters can be in the range (0,1]. For example, λ1 = 0.25, λ2 = 0.1, and λ3 = 0.01.

[0075] The multi-task objective function 204 is used to update the parameters of the neural network 106 during training. The parameters of the neural network 106 can be updated using an optimization process designed to optimize the multi-task loss function 208. For example, the optimization process could be a gradient descent algorithm. The optimization process can use backpropagation, for example, by backpropagating the multi-task loss function 204 to the parameters of the neural network 106. The optimization process can be associated with a learning rate that varies during training. For example, the learning rate can start with an initial value (e.g., 10^(–3)) and then increase after a first threshold number of epochs, for example, increasing to 10^(–2) after 5 epochs. The learning rate can then be decreased after further threshold number of epochs, for example, by a predefined factor at a predefined number of epochs. For example, the learning rate might decrease by a factor of 10 at 55 and 68 epochs.

[0076] Figure 3 A flowchart illustrating a method for training a neural network to perform face localization in an image is shown. This method can correspond to the combination described above. Figure 2 The method described.

[0077] In operation 3.1, training images are input into the neural network. These training images include one or more faces. The training images can be selected from a training dataset consisting of multiple images, each including one or more faces located at known locations and with known bounding boxes. One or more features of the faces, such as the locations of facial feature points, may also be known.

[0078] In operation 3.2, a neural network is used to process the input training images.

[0079] In Operation 3.3, the neural network outputs one or more sets of output data for each of the multiple training anchors in the training image. Each set of output data includes the predicted face classification, the predicted location of the corresponding face bounding box, and one or more corresponding feature vectors.

[0080] Operations 3.1 through 3.3 can be iterated over a subset of training examples in the training data to form a set of training examples. This set can be used to determine the objective function in operation 3.4, for example, by averaging the multi-task objective function over the set. Alternatively, operation 3.4 can be performed individually for each training example.

[0081] In operation 3.4, the parameters of the neural network are updated according to an objective function. For each positive anchor point in the training image, the objective function includes: a classification loss for comparing the predicted classification of the positive anchor point with the known classification of the positive anchor point; a bounding box regression loss for comparing the predicted location of the face bounding box of the positive anchor point with the known location of the face bounding box; and a feature loss for comparing the pixel-based attributes of the one or more feature vectors of the positive anchor point with the known pixel-based attributes of the face associated with the positive anchor point. The update may be performed after each iteration of operations 3.1 to 3.3. Alternatively, the update may be performed after multiple iterations of operations 3.1 to 3.3. In these embodiments, the multi-task loss function may include one or more expected values ​​obtained on the training example set.

[0082] Operations 3.1 to 3.4 can iterate on the training dataset until a threshold condition is met. The threshold condition can be the number of training epochs. The number of training epochs can be in the range of 50-150, for example, in the range of 70-90, such as 80.

[0083] Figure 4 An example of a neural network structure for face localization in an image is shown. In some embodiments, neural network 106 may use this structure.

[0084] Neural network 400 includes a backbone network 402. Backbone network 402 may include multiple convolutional layers. Backbone network 402 may also include multiple downsampling layers that reduce the dimensionality of the input to that layer. The backbone network may be initialized as a pre-trained classification network, such as a ResNet network (e.g., ResNet-152). For example, for an input image 404 of size 640×640 color pixels, the structure of backbone network 402 can be given as follows:

[0085]

[0086]

[0087] The convolutional building blocks are shown in square brackets, and downsampling is performed by conv3_1, conv4_1, and conv5_1 with a stride of 2. The backbone 402 processes the input image 404 in a "bottom-up" manner (i.e., from low-index layers to high-index layers) through a series of convolutional layers (labeled C2-C6 in this example) and generates an output for each convolutional layer. Although six layers are shown in this example, fewer or more layers can be used. During training, one or more layers of the pre-trained backbone, such as the first two layers, can be fixed to improve accuracy.

[0088] During training, one or more convolutional layers can be fixed to their initial pre-trained values. For example, the first two layers can be fixed. This can lead to higher neural network accuracy after training.

[0089] Neural network 400 also includes a feature pyramid network 406. Feature pyramid network 406 comprises multiple feature pyramid levels (labeled P2-P6 in this example, where P6 directly acts on / shares with the last layer of backbone network 402). Feature pyramid levels may include convolutional layers. Convolutional layers may be deformable convolutional layers. Convolutional layers may be interleaved with one or more magnification layers. Feature pyramid 406 is used to process the outputs of corresponding layers in backbone network 402 in a "top-down" manner (i.e., from high-index layers to low-index layers) to generate multiple feature maps.

[0090] An example of a feature pyramid structure is shown below:

[0091]

[0092] The output dimension of each layer is indicated in the first column, the "Anchor" column indicates the scale associated with each layer of the pyramid, and the "Stride" column indicates the stride of the convolution performed by each layer. For example, in this example, layer P2 is associated with anchor scales of 16×16, 20.16×20.16, and 25.4×25.4, and performs convolutions with a stride of 4. However, it should be understood that different scales and convolutions can be chosen. Lower levels of the feature pyramid can be associated with smaller scales and / or have smaller strides compared to higher levels.

[0093] The neural network 400 includes multiple lateral connections 408 between the outputs of layers in the backbone network 402 and layers in the feature pyramid 406. In some embodiments, the lateral connections 408 may process the outputs of the associated backbone network layers 402 and then input them, along with the outputs of higher feature pyramid layers, into the corresponding feature pyramid layer. In other words, each feature pyramid layer after the highest layer receives a combination of the output of the corresponding backbone network layer and the output of the previous higher layer of the feature pyramid as input; for example, feature pyramid layer P3 receives a combination of the output of backbone network layer C3 and the output of feature pyramid layer P4 as input. Higher layers of the feature pyramid (in this example, layers P6 and P5) receive their inputs from the backbone network without combining them with the output of the previous layer.

[0094] Each level of the feature pyramid outputs a set of features. These features can be further processed to generate output data 108 and / or a multi-task loss 208. The number of features in each level can be fixed to be the same. For example, each level can be fixed to have 256 features.

[0095] Lateral connections can be used to combine bottom-up and top-down paths. Generally, top-down paths may be semantically stronger but spatially coarser, while bottom-up paths may be spatially finer but semantically weaker. By combining paths using lateral connections, the performance of neural networks in face localization can be improved.

[0096] In some embodiments, the backbone network 402 can be a MobileNet network, such as MobileNet-0.25. This provides a more lightweight model that can run virtually in real-time on a single CPU. These lightweight models can reduce data size by using 7×7 convolutions with a stride of 4 on the input image. Dense anchors can be tiled on P4-P5 to improve performance. In some embodiments, deformable convolutional layers are replaced with convolutional layers to accelerate processing. During training, one or more layers of the pre-trained backbone network, such as the first two layers, can be fixed to improve accuracy.

[0097] Figure 5 An example of lateral connections in a neural network used for face localization in an image is shown. In this example, lateral connections are applied to the P3 level of the feature pyramid, although they can be applied to any lateral connection between layers with the necessary modifications.

[0098] Lateral connection 500 provides the output of layer 502 of backbone network 402 (in this example, layer C3 of the backbone network) to the corresponding layer of feature pyramid network 406 (in this example, layer P3 of the feature pyramid). Lateral connection 500 can be used to process the output of layer 502 of backbone network 402 to generate a processed output 504. For example, this processing can change the dimension of the output of layer 502 of backbone network 402 so that it can be used as input to the corresponding feature pyramid layer. In the example shown, a 1×1 convolution is used to reduce the dimension of the output of layer 502 of backbone network from 80×80×512 to 80×80×256. Each lateral connection can apply a different convolution to reduce or increase the dimension of the associated output layer to match the corresponding feature pyramid layer.

[0099] The output from the next higher layer 506 of the feature pyramid (P4 in this example) is upsampled to increase its dimension. In the example shown, the output of layer P4 of the pyramid network is magnified by a factor of 2, from 40×40×256 to 80×80×256. For example, magnification can be performed by nearest neighbor upsampling. The magnified output is then added to the processed output 504 from the lateral connection 500 to generate a merged graph 508. Element-wise addition can be used to merge the magnified output with the processed output 504.

[0100] The merged graph 508 is used as input to the layer of the feature pyramid network 406 corresponding to the lateral connections 500 (in this example, layer P3). Layer P3 applies a 3×3 deformable convolution to the merged graph 508 to generate the output feature map 510. Using deformable convolutions reduces aliasing effects from upsampling and performs non-rigid context modeling.

[0101] Figure 6 Another example of a neural network architecture for face localization in an image is shown. The neural network 600 in this example is essentially the same as... Figure 4 The neural network is the same, but a context module 602 is added to each feature pyramid layer. The context module 602 can increase the receptive field and enhance the rigid context modeling capability of the neural network 600.

[0102] Each context module 602 obtains a feature map 604 as input from the corresponding feature pyramid layer. A series of deformable convolutions are performed using the feature map 604. Each deformable convolution after the first is applied to the output 606a-606c of the previous deformable convolution. In the example shown, three 3×3 deformable convolutions are applied. However, it should be understood that fewer or more deformable convolutions can be applied, and deformable convolutions of different sizes can be used.

[0103] The outputs of each deformable convolution 606a-606c are combined to form an enhanced output 608. The enhanced output 608 can be further processed to generate output data 108 and / or a multi-task loss 208.

[0104] Figure 7 A schematic example of a system / apparatus for performing any of the methods described herein is shown. The system / apparatus shown is an example of a computing device. Those skilled in the art will understand that other types of computing devices / systems may alternatively be used to implement the methods described herein, such as distributed computing systems.

[0105] Device (or system) 700 includes one or more processors 702. The one or more processors control the operation of other components of the system / device 700. The one or more processors 702 may include, for example, a general-purpose processor. The one or more processors 702 may be a single-core device or a multi-core device. The one or more processors 702 may include a central processing unit (CPU) or a graphics processing unit (GPU). Alternatively, the one or more processors 702 may include dedicated processing hardware, such as a RISC processor or programmable hardware with embedded firmware. Multiple processors may be included.

[0106] The system / device includes working or volatile memory 704. One or more processors can access volatile memory 704 to process data and can control the storage of data in memory. Volatile memory 704 can include any type of RAM, such as static RAM (SRAM) or dynamic flash memory, such as an SD card.

[0107] The system / device includes non-volatile memory 706. Non-volatile memory 706 stores a set of operation instructions 708 for controlling the operation of processor 702 in the form of computer-readable instructions. Non-volatile memory 706 can be any type of memory, such as read-only memory (ROM), flash memory, or magnetically driven memory.

[0108] One or more processors 702 are used to execute operation instructions 408 to cause the system / device to perform any of the methods described herein. Operation instructions 708 may include code relating to hardware components of the system / device 700 (i.e., drivers) and code relating to basic operations of the system / device 700. Generally, one or more processors 702 use volatile memory 704 to temporarily store data generated during the execution of the operation instructions 708, thereby executing one or more instructions of the operation instructions 708 that are permanently or semi-permanently stored in non-volatile memory 706.

[0109] The methods described herein can be implemented in digital electronic circuits, integrated circuits, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These may include computer program products (e.g., software stored on, for example, a disk, optical disk, memory, or programmable logic device), including computer-readable instructions, when executed by a computer (e.g., in conjunction with...). Figure 7 When the computer described herein is executed, the computer performs one or more methods as described herein.

[0110] Any system feature described herein can also be provided as a method feature, and vice versa. As used herein, device plus functional features can be represented according to their corresponding structures. Specifically, method aspects can be applied to system aspects, and vice versa.

[0111] Furthermore, any, some, and / or all features in one aspect may be applied in any suitable combination to any, some, and / or all features in any other aspect. It should also be understood that specific combinations of the various features described and defined in any aspect of the invention may be implemented and / or provided and / or used independently.

[0112] Although several embodiments have been shown and described, those skilled in the art will understand that changes may be made to these embodiments without departing from the principles of the invention, the scope of which is defined in the claims.

Claims

1. A computer-implemented method of training a neural network for face localization, the method comprising: The method comprises: inputting a training image to the neural network, the training image comprising one or more faces; processing the training image using the neural network; outputting, from the neural network, for each of a plurality of training anchors in the training image, one or more sets of output data, each set of output data comprising a predicted face class, a predicted location of a corresponding face box, and one or more corresponding feature vectors; updating parameters of the neural network according to an objective function, wherein, for each positive anchor in the training image, the objective function comprises: a classification loss for comparing a predicted class of the positive anchor with a known class of the positive anchor; a box regression loss for comparing a predicted location of a face box of the positive anchor with a known location of the face box; a feature loss for comparing pixel-based attributes of the one or more feature vectors of the positive anchor with known pixel-based attributes of a face associated with the positive anchor; wherein the one or more feature vectors comprise an encoded representation of a face within a predicted face box, the method further comprising: generating a three-dimensional representation of the face using a mesh decoder neural network, the mesh decoder neural network comprising a plurality of geometric convolution layers and a plurality of upsampling layers, the plurality of upsampling layers interleaved with the plurality of geometric convolution layers, convolution operators of the geometric convolution layers being recursive polynomials, the recursive polynomials comprising Chebyshev polynomials and / or orthogonal polynomials; generating a two-dimensional image of the face from the three-dimensional representation of the face using a differentiable renderer, the feature loss comprising a dense regression loss for comparing the generated two-dimensional image of the face with a ground truth image of the face within the predicted face box; the dense regression loss being a pixel-wise difference between the generated two-dimensional image of the face and the ground truth image of the face within the predicted face box.

2. The method of claim 1, wherein, the one or more feature vectors comprising predicted locations of a plurality of facial landmark points of the face, wherein the feature loss comprises a facial landmark point regression loss for comparing the predicted locations of the plurality of facial landmark points with known locations of the plurality of facial landmark points.

3. The method of claim 1, wherein, the feature vector further comprising camera parameters and / or lighting parameters, wherein the differentiable renderer uses the camera parameters and / or lighting parameters when generating the two-dimensional image of the face from the three-dimensional representation of the face.

4. The method according to any one of claims 1 to 3, characterized in that, for each negative anchor in the training image, the objective function comprises a classification loss for comparing a predicted class of the negative anchor with a known class of the negative anchor.

5. The method according to any one of claims 1 to 3, characterized in that, further comprising iterating the method using one or more further training images until a threshold condition is satisfied.

6. The method according to any one of claims 1 to 3, characterized in that, the neural network comprising: a first plurality of convolution layers comprising an input layer, a plurality of convolution filters, and one or more skip connections; a second plurality of convolution layers laterally connected to the first plurality of convolution layers and configured to process an output of the first plurality of convolution layers in a top-down manner.

7. The method of claim 6, wherein, each lateral connection is configured to merge an output of one of the first plurality of convolution layers with an output of a preceding layer of the second plurality of convolution layers.

8. The method of claim 6, wherein, The second plurality of convolutional layers comprises one or more deformable convolutions.

9. The method of claim 6, wherein, The neural network further comprises one or more context modules, each context module configured to process the output of one of the second plurality of convolutional layers using one or more deformable convolutional networks.

10. A computer-implemented face positioning method, comprising: The method comprises: identifying one or more faces in an input image using a neural network; identifying one or more corresponding face boxes in the input image using the neural network, wherein the neural network has been trained using the method of any one of claims 1 to 9.

11. A computer-implemented face positioning method, comprising: The method comprises: inputting an image to a neural network, the image comprising one or more faces; wherein the neural network has been trained using the method of any one of claims 1 to 5; processing the image using the neural network; outputting one or more sets of output data from the neural network, each set of output data comprising a predicted face class, a predicted position of a corresponding face box, and one or more corresponding feature vectors, wherein the neural network comprises: a first plurality of convolutional layers comprising an input layer, a plurality of convolutional filters, and one or more skip connections; a second plurality of convolutional layers laterally connected to the first plurality of convolutional layers and configured to process the output of the first plurality of convolutional layers in a top-down manner.

12. The method of claim 11, wherein, Each lateral connection is configured to merge the output of one of the first plurality of convolutional layers with the output of a preceding layer of the second plurality of convolutional layers.

13. The method according to claim 11 or 12, characterized in that, The second plurality of convolutional layers comprises one or more deformable convolutions.

14. The method of claim 11 or 12, wherein, The neural network further comprises one or more context modules, each context module configured to process the output of one of the second plurality of convolutional layers using one or more deformable convolutional networks.

15. A system, comprising: comprises: one or more processors; a memory comprising computer-readable instructions that, when executed by the one or more processors, cause the system to perform the method of any one of claims 1 to 14.

16. A computer program product, characterised in that, comprises computer-readable instructions that, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 14.

Images