Multi-task face and landmark detector

Abstract

Methods and systems for face detection techniques for image processing neural networks are provided. In one example, a method can include collecting a set of multi-channel outputs of context modules, providing the multi-channel outputs to both a face detection head and a landmark localization head of a neural network. The face detection head can then generate a bounding box, which is also provided to the landmark localization head. Based on the outputs of the context modules and the bounding box, the landmark localization head can provide an output including a set of landmark indicators.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Russian Patent Application No. 2020101638, filed on January 17, 2020, entitled "MULTI-TASK FACE AND LANDMARK DETECTOR". The entire application is incorporated herein by reference for all purposes. Technical Field

[0003] This disclosure generally relates to systems and methods for performing face detection. Background Technology

[0004] Facial image analysis technology has many practical applications in automotive, security, retail, and social networks. Facial analysis typically begins with basic tasks such as bounding box detection and landmark localization. One technique in use is to sequentially apply single-task models to independently solve the face detection problem and the landmark (or "keypoint") detection problem. These single-task models may be incorporated into applications that are based on convolutional neural networks (CNNs) or may be used in other ways.

[0005] However, developing a software system consisting of many sequentially applied CNNs can be extremely challenging because it is best to train each CNN individually and address the errors produced by previous models. While different heuristics and specialized training procedures can be applied to achieve robustness across the entire system, single-task CNNs cannot benefit from the shared deep representations and additional supervision provided by multiple tasks.

[0006] Meanwhile, recent research suggests that multi-task CNNs, which generate multiple predicted outputs, may offer improved accuracy and / or speed compared to single-task counterparts, but can be difficult to train correctly. However, despite recent successes of multi-task models in face analysis, their accuracy remains unsatisfactory compared to competing single-task models.

[0007] The most popular multi-task model, MTCNN, uses cascaded shallow CNNs but does not share feature representations. Modern end-to-end multi-task methods are mainly represented by single-shot methods. For feature point localization, these models use regression heads or keypoint heatmaps. Heatmap-based methods have lower face detection accuracy, while regression-based methods have poor feature point localization. This is because regression-based methods cannot withstand strong feature point prediction heads. Furthermore, there may be misalignment between the spatially discrete features of the activation map and the continuous positions of facial feature points. Shallow convolutional layers cannot properly handle this misalignment. Summary of the Invention

[0008] Considering the shortcomings of other techniques, this paper discloses an accurate multi-task face detection and feature point detection model called "MaskFace". The MaskFace model partially extends existing face detection methods, such as the RetinaFace model (Guo, Zhou, Yu, Kotsia, and Zafeiriou, "Retinaface: Single-stage dense face localization in the wild", 2019) and the SSH model (Najibi, Samanguei, Chellappa, and Davis, "SSH: Single-stage headless face detector", 2017), by adopting the concept of the MaskR-CNN model (He, Gkioxari, Dollar, and Girshick, "Maskr-CNN", 2017). In the first stage, the MaskFace model predicts bounding boxes, and in the second stage, the predicted bounding boxes are used to extract face features from a shared representation.

[0009] MaskFace employs two prediction heads: a face detection head and a feature point localization head (e.g., a face feature point localization head). The face detection head outputs the bounding box of the face. Face features are then extracted from a high-resolution layer using the predicted bounding box, enabling precise feature point localization. To achieve good pixel-to-pixel alignment during feature extraction, we incorporate a RoIAlign layer on top of the Mask R-CNN model (“Maskr-cnn”, 2017). The extracted face features are used to predict the localization mask for the feature points.

[0010] Unlike Mask R-CNN and other multi-stage methods, MaskFace predicts bounding boxes in a single forward pass, which improves performance. For feature extraction, MaskFace uses a Region of Interest (RoI) Align (RoIAlign) layer (“Maskr-CNN”, 2017), which advantageously provides good pixel-to-pixel alignment between the predicted bounding boxes and discrete feature maps. MaskFace uses a feature pyramid (Lin, Dollar, Girshick, He, Hariharan, Belongie, “Feature pyramid networks for object detection”, 2017) and a context module (“Retinaface: Single-stage dense face localization in the wild”, 2019), which advantageously improves the detection of small faces. The feature pyramid transfers deep features to shallow layers, while the context module increases the receptive field and makes the prediction layers stronger. MaskFace’s feature point prediction is as fast as the original Mask R-CNN head, and the increased computational cost of feature point prediction is negligible for images with few faces.

[0011] It should be understood that the above overview is provided to present the concept choices further described in the detailed embodiments in a simplified form. This is not intended to identify key or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims appended to the detailed embodiments. Furthermore, the claimed subject matter is not limited to implementations that address any shortcomings pointed out above or in any part of this disclosure. Attached Figure Description

[0012] This disclosure can be better understood by referring to the following description of non-limiting embodiments, in which:

[0013] Figure 1 A neural network architecture for face detection image processing according to one or more embodiments of the present disclosure is shown;

[0014] Figure 2 The illustration shows a context module design according to one or more embodiments of this disclosure; and

[0015] Figures 3 to 5 A flowchart is shown of a method for processing an input image for face detection according to one or more embodiments of the present disclosure. Detailed Implementation

[0016] This paper discloses systems and methods for image processing. Figure 1A neural network architecture for the MaskFace model, which can be used by various image processing systems, is described, incorporating a feature pyramid network, a context module, a face detection head, and a feature point localization head. Figure 2 Depicting and Figure 1 The design incorporates a context module compatible with neural network architectures. Figures 3 to 5 Describing neural network architectures (such as Figure 1 The architecture) can be used to process images to improve face detection.

[0017] Figure 1 A neural network architecture 100 for face detection image processing according to one or more embodiments of the present disclosure is illustrated. The neural network architecture 100 can process images (e.g., photographs or pictures) such as input image 105, which may potentially include one or more faces. The neural network architecture 100 includes a Feature Pyramid Network (FPN) (e.g., the FPN described in "Feature pyramid networks for object detection" 2017). The output of the FPN is provided to a context module stage (e.g., the context module stage may be similar to the inception module described by Szegedy, Liu, Jia, Sermanet, Reed, Anguelov, Erhan, Vanhoucke, and Rabinovich in "Going deeper with convolutions" 2015). The output of the context module stage is then provided to a first head 140 of the neural network architecture 100, and also to a second head 150 of the neural network architecture 100. In the neural network architecture 100, the first head 140 (e.g., a face detection head) advantageously provides bounding boxes to the second head 150 (e.g., a feature point localization head). Then, the neural network architecture 100 can output a set of feature point indicators 155, which can be used in a variety of ways.

[0018] The first portion 110 of the FPN may include feature maps of the final layer 111, which may be numbered C2 to CN. For example, the neural network architecture 100 is depicted as including feature maps numbered C2 to C6 of the final layer 111.

[0019] The feature map of the final layer 111 can be calculated based on the input image 105. For example, in an embodiment such as the depicted embodiment, C2 can be calculated based on the input image 105; C3 can be calculated based on C2; C4 can be calculated based on C3; C5 can be calculated based on C4; and C6 can be calculated based on C5.

[0020] In some implementations, the feature maps from C2 to C6 of the final layer 111 can be calculated using strides of 4, 8, 16, 32, and 64, respectively. For example, the stride is a power of 2 raised to the index number. (In other implementations, the feature maps of the final layer 111 can have strides spanning from 2 to any number, and / or the stride of the feature maps may not be a power of 2 raised to the index number; for example, it may be a power of 3 raised to the index number.)

[0021] The second part 120 of the FPN may include individual feature maps 121, which are numbered P2 to PN. For example, the neural network architecture 100 is depicted as including feature maps 121 numbered P2 to P6.

[0022] Feature map 121 can be calculated based on the feature map of the last layer 111. For example, in an embodiment such as the depicted embodiment, P5 can be calculated based on C5; P4 can be calculated based on C4 and P5; P3 can be calculated based on C3 and P4; and P2 can be calculated based on C2 and P3. In some embodiments, P6 can be calculated by applying a max-pooling layer with a stride of 2 to C5.

[0023] Feature maps 121 from P2 to P5 can be calculated using feature maps of the last layer 111 with strides of 4, 8, 16, 32, and 64, respectively. For example, the stride is a power of 2 with the index number as the exponent. (In other embodiments, feature maps 121 can have strides spanning from 2 to any number, and / or the stride of the feature map may not be a power of 2 with the index number as the exponent; for example, it may be a power of 3 with the index number as the exponent.) The various feature maps 121 can have the same spatial size as the corresponding feature maps of the last layer 111.

[0024] The first portion 110 and the second portion 120 of the FPN can interact accordingly to generate feature maps 121 from P2 to P6, which can then be a set of outputs of the FPN. In some embodiments, each feature map 121 can have 256 channels.

[0025] The use of FPN (Feature Pyramid Network) allows for the combination of low-resolution, semantically strong features with high-resolution, semantically weak features through top-down paths and lateral connections. The result is a feature pyramid with rich semantics at all levels, which can facilitate the detection of small faces.

[0026] Then, the set of multi-channel outputs of the FPN (e.g., feature maps 121 numbered P2 to P6) is provided as input to the context module stage 130, which has a corresponding set of context modules 131, which may be numbered M2 to MN. (The implementation of the context modules is in...) Figure 2 (As depicted and further discussed herein.) The set of context modules 131 can then provide a corresponding set of multi-channel outputs, which can be collected and provided to both the first head 140 and the second head 150. For example, in an embodiment such as the depicted embodiment, the output of context module M2 can be calculated based on feature map P2; the output of context module M3 can be calculated based on feature map P3; the output of context module M4 can be calculated based on feature map P4; the output of context module M5 can be calculated based on feature map P5; and the output of context module M6 can be calculated based on feature map P6.

[0027] A first head 140 (which may be a face detection head) can predict a set of boundary regions 145 based on the set of multi-channel outputs of the set of context modules 131. The boundary regions 145 may correspond to the input image 105 and may, for example, indicate portions, areas, and / or regions of the input image 105 that correspond to detected faces. In some embodiments, the boundary regions 145 may be rectangular bounding boxes. In some embodiments, the boundary regions 145 may have other shapes (e.g., circles, hexagons, or any other regular or irregular shape). The first head 140 may use a 1×1 filter. The prediction of the set of bounding boxes can be completed in a single forward pass.

[0028] In some implementations, 1×1 convolutional layers with shared weights can be applied to the set of multichannel outputs of the set of context modules 131 for anchor box regression and classification. The neural network architecture 100 can use translation-invariant anchor boxes (e.g., anchor boxes that may resemble those described by Ren, He, Girshick, and Sun in "Faster r-cnn: Towards real-time object detection with region proposal networks" in 2015). The base anchor boxes may have {16} at levels corresponding to context modules M2 through M6, respectively. 2 32 2 ,64 2 128 2 256 2 The area of ​​}. For each level from M2 to M6, the neural network architecture 100 can use a basic anchor box size of {2}. 0 ,21 / 3 ,2 2 / 3 The anchor frames can advantageously provide dense scale coverage. The aspect ratio of some or all anchor frames can be 1.0. In some implementations, each level from M2 to M6 can have three anchor frames, which can cover a scale range of approximately 16 to 406 pixels across levels. For a 640×640 pixel input image, there could be approximately 112,000 anchor frames in total.

[0029] If the intersection-over-union (IoU) overlap between the anchor box and the ground truth box is greater than 0.5, the anchor box can be considered a positive sample (and / or assigned a positive label). If the overlap is less than 0.3, the anchor box can be considered a negative sample and / or assigned a negative label. During training, some or all anchor boxes with overlaps between 0.3 and 0.5 may be ignored. Furthermore, a low-quality matching strategy can be used for anchor box assignment. For each ground truth box, a set of anchor boxes with the highest overlap with the ground truth box may be found. For each anchor box in this set, if the anchor box does not match, a ground truth box with the highest IoU may be matched. Experiments show that using a low-quality matching strategy can advantageously improve accuracy.

[0030] The boundary region 145 predicted by the first head 140 can then be provided to the second head 150, which can be a feature point localization head (e.g., a face feature point localization head). The second head 150 can treat the prediction from the first head 140 (e.g., the boundary region 145) as a region of interest (RoI) for feature extraction to perform feature point localization (e.g., face feature point localization). The second head 150 can use a RoI Align layer for feature extraction (the layer can be similar to the layer described in "Maskr-CNN" in 2017) and can accordingly extract a set of feature point indicators for the input image 105 based on the multi-channel output of the context module 131 and the boundary region 145.

[0031] Prediction proposals can be filtered. For example, predictions with a confidence level less than approximately 0.02 may be ignored. In some implementations, a non-maximum suppression threshold of approximately 0.7 can be applied to the remaining predictions. The proposal can then be matched with the ground truth bounding boxes. If the IoU overlap between the proposal and the ground truth bounding box is greater than approximately 0.5, the proposal can be used to extract feature point features from the corresponding layers of the FPN, corresponding to M2 through M6.

[0032] Using Equation 1 below, the width w can be... roi And the height is h roi The post-FPNRoI is assigned to the layer corresponding to MN in the FPN:

[0033]

[0034] Where k0 = 4. In this equation, if the area of ​​the predicted bounding box is less than 112... 2 Then the bounding box can be assigned to the FPN layer corresponding to M2; if the area is between 112 2 up to 224 2 Between these points, the bounding boxes can be assigned to the FPN layer corresponding to M3; and so on. The relatively high-resolution layer of FPH corresponding to M2 can be used for feature extraction with a stride of 4. Experiments show that high-resolution feature maps can advantageously improve the accuracy of feature point localization for relatively small faces within the input image.

[0035] As discussed in this paper, the neural network architecture 100 can use a RoIAlign layer to extract features from the assigned feature map. The RoIAlign layer facilitates the correct alignment of the extracted features with the input RoI. The RoIAlign layer can output 14×14 resolution features, which can then be fed into multiple subsequent convolutional layers (e.g., 3×3 convolutions with 256 filters and a stride of 1), a single transposed convolutional layer (e.g., 4×4 transposed convolutions with K filters and a stride of 2), and a bilinear interpolation layer that upsamples the mask to 56×56 resolution. The size of the output mask tensor can be K×56×56. K can be the number of facial feature points.

[0036] Compared to the computational cost for overall feature extraction, the computational cost associated with the feature point localization head of the neural network architecture 100 disclosed in this paper may be slightly increased, thus allowing it to be used advantageously at a very low relative cost while providing improved accuracy in feature point localization.

[0037] The location of the feature points can be modeled as a one-hot mask, and the neural network architecture disclosed in this paper can be used to predict K masks, each corresponding to one of the K feature points (e.g., facial feature points such as left eye, right eye, etc.).

[0038] For neural network architecture 100, the multi-task loss of an image can be defined as Equation 2 below:

[0039] <![CDATA[L=L cls +L box +λ kp L kp ]]> (2)

[0040] in:

[0041] L cls It could be an anchor-box binary classification loss (face and background);

[0042] L box It could be the regression loss for the anchor frame position; and

[0043] Lkp It can be done by using the parameter λ kp Weighted key point location loss.

[0044] For neural network architecture 100, anchor box classification can use the focus loss derived from the following Equation 3:

[0045]

[0046] Additionally, where:

[0047] N pos It can be the number of positive anchor boxes that should be classified as faces (p i It should equal 1);

[0048] N neg This could be the number of negative anchor boxes that should be categorized as background (p i It should equal 0);

[0049] Pos can be a set of indices for a positive anchor box;

[0050] Neg can be a set of indices for negative anchor boxes;

[0051] p i Anchor box i could be the predicted probability of a face;

[0052] α can be a balancing parameter between the classification losses of positive and negative anchor boxes; and

[0053] γ can be a focusing parameter to reduce the loss of well-classified samples.

[0054] For neural network architecture 100, bounding box regression can employ a smooth L1 loss according to Equation 4 below. L1 ):

[0055]

[0056] Additionally, where:

[0057] t i It can be a vector representing the four parameterized coordinates of the predicted bounding box (e.g., the vector associated with the positive anchor box i);

[0058] It can be a vector representing the four parameterized coordinates of the real box associated with the negative anchor box i.

[0059] For neural network architecture 100, the prediction of feature point locations can be achieved by applying the cross-entropy loss of each feature point mask according to Equations 5 and 6 below:

[0060]

[0061] Additionally, where:

[0062] L i，k，j，l It can be the predicted logit of feature point k of positive sample i;

[0063] M i，k，j，l It can be the mask of feature point k of positive sample i;

[0064] It can be the index of the mask pixel where the true feature point k in the positive sample i is located; and

[0065] It can be the index of the mask pixel where the real feature point k in the positive sample i is located.

[0066] For each of the K keypoints of a face, the training objective may be a one-hot m×m binary mask, in which only one pixel is labeled as foreground. In some implementations, parameters α and γ can be set to 0.25 and 2, respectively. Based on experimental results, the neural network architecture 100 can select the keypoint loss weights λ. kp The optimal value is approximately 0.25, which provides a good balance between the accuracy of face detection and the accuracy of feature point localization.

[0067] Then, the second head 150 can output a feature point indicator 155. In some embodiments, the feature point indicator 155 may include one or more coordinates corresponding to the bounding box of the face on the input image 105. In some embodiments, the feature point indicator 155 may include one or more coordinates corresponding to feature points of the face on the input image 105 (e.g., facial feature points such as locations associated with facial structure and / or facial features). In various embodiments, the feature point indicator 155 can be used to modify a copy of the input image 105 (e.g., by labeling the copy). In various embodiments, the feature point indicator 155 may be presented as a separate data file corresponding to the input image 105, or as metadata embedded in an annotated copy of the input image 105.

[0068] In some implementations, a customer may use cloud-based computing services to provide the input image to a neural network architecture such as neural network architecture 100, and then receive the feature point indicator 155 in some way (e.g., as a direct annotation on a copy of the input image, as a separate data file, and / or as metadata). In some implementations, the face may be cropped from a copy of the input image or blurred within the copy of the input image. In some implementations, face recognition technology can be performed according to the improved face detection techniques discussed herein. In various implementations, the extraction of the feature point indicator 155 enables various subsequent applications that can utilize the location of the detected face within the image.

[0069] Go to Figure 2 The context module 200 includes an input 205, an output 260, and various branches from the input 205 to the output 260, wherein various sequences of convolutional filters (or layers) are processed along these branches. The design of the context module 200 is essentially similar to... Figure 1 The design of the context module 131 is described. When independent weights are applied to the input feature map of the context module, this type of context module can advantageously increase the receptive field and / or add context to the prediction. Experimental results show that this type of context module also helps improve accuracy.

[0070] As depicted, both input 205 and output 260 can have 256 channels. On the first branch from input 205 to output 260, a first portion of the channels of input 205 (e.g., 128 channels) can be processed only by the first convolutional filter 210. On the second and third branches, a second portion of the channels of input 205 (e.g., 128 channels) can be processed by the second convolutional filter 220. On the second branch, a first subgroup of these channels (e.g., 64 channels) can then be processed by the third convolutional filter 230; and on the third branch, a second subgroup of these channels (e.g., 64 channels) can then be processed by the fourth convolutional filter 240 and the fifth convolutional filter 250. In various embodiments, a modified linear unit (ReLU) can be applied after each convolutional filter (or layer).

[0071] Subsequently, output 260 can concatenate the outputs of the first convolutional filter 210 (e.g., 128 channels), the third convolutional filter 230 (e.g., 64 channels), and the fifth convolutional filter 250 (e.g., 64 channels). Therefore, output 260 can include a total of 256 channels from the respective branches through the individual convolutional filters.

[0072] The convolutional filters of the context module 200 are described as including 3×3 convolutional filters. Compared to larger convolutional filters, 3×3 convolutional filters can advantageously reduce the number of computations in the context module 200. In some embodiments, some or all of the convolutional filters of the context module 200 may include convolutional filters of other sizes.

[0073] Furthermore, although input 205 and output 260 are depicted as each having 256 channels, in various embodiments, input 205 and output 260 may have other numbers of channels. Similarly, although depicted as having a specific number of branches and a specific number of convolutional filters in a particular configuration, alternative embodiments may have different branches from input to output and / or different convolutional filter processing sequences in different configurations.

[0074] Figures 3 to 5 A flowchart illustrating a method for processing an input image to perform face detection is shown. Figure 3 As shown, method 300 may include a first part 310, a second part 330, a third part 330, a fourth part 340, a fifth part 350, a sixth part 360 and / or a seventh part 370.

[0075] In the first part 310, an input image can be fed to a neural network for face detection. The input image can be substantially similar to input image 105, and the neural network can have an architecture substantially similar to neural network architecture 100.

[0076] In the second part 320, the input image can be provided to an FPN having a set of feature maps. For example, the FPN can be substantially similar to the FPN of neural network architecture 100, and can have a first part including feature maps of the last layer (which can be substantially similar to feature maps of the last layer 111) and a second part including feature maps computed based on the feature maps of the last layer (which can be substantially similar to feature map 121). In various embodiments, the FPN can have a set of outputs corresponding to the inputs of a set of context modules (see below). For some embodiments, the set of feature maps can be computed based on a set of last layer feature maps with a stride power of 2.

[0077] In the third part 330, a set of multi-channel outputs of a set of context modules corresponding to a specific group of neural networks can be collected. This set of multi-channel outputs may correspond to the input image. The set of multi-channel outputs may be substantially similar to the output of the context module 131 of the neural network architecture 100. In some implementations, the context module may use a 3×3 filter.

[0078] In Part 4, 340, a set of multi-channel outputs can be provided to both the first head and the second head of the neural network. The first head can be substantially similar to First Head 140, and the second head can be substantially similar to Second Head 150. The first head can be a face detection head; and the second head can be a feature point localization head.

[0079] In section 5, 350, a set of bounding boxes of the input image can be predicted using a first head based on a set of multi-channel outputs. In some implementations, the first head can use a 1×1 filter (e.g., a convolutional filter). For example, as discussed herein, the prediction can use 1×1 convolutional shared weights applied to the multi-channel outputs of a set of context modules, and anchor box regression and classification as discussed herein can be used. In some implementations, the prediction of a set of bounding boxes can be completed in a single forward pass.

[0080] In Part 6, 360, a set of feature point indicators of the input image can be extracted using a second head based on a set of multi-channel outputs and a set of bounding boxes. In various implementations, the second head may include a RoIAlign layer. The second head can treat the bounding boxes from the first head as RoIs for feature extraction to perform feature point localization, and feature extraction can be performed using the RoIAlign layer, as discussed herein.

[0081] In Part 7, 370, an output including a set of feature point indicators can be provided. The output can be presented as a modified copy of the input image, i.e., another data file different from the input image, or as metadata embedded in the input image, as discussed herein.

[0082] Go to Figure 4 The method 400 for processing images using a neural network may include a first portion 410, a second portion 420, a third portion 430, a fourth portion 440, and / or a fifth portion 450. In the first portion 410, an input image may be provided to the FPN stage of the neural network. For example, the neural network may have an architecture substantially similar to neural network architecture 105, and the FPN may be substantially similar to the FPN of neural network architecture 100. The FPN may have a first portion including feature maps of the last layer (which may be substantially similar to feature maps of the last layer 111) and a second portion including feature maps computed based on the feature maps of the last layer (which may be substantially similar to feature map 121). In various embodiments, the FPN may have a set of outputs (see below) corresponding to inputs of a set of context modules, respectively. For some embodiments, a set of feature maps is computed based on a set of last layer feature maps with a stride power of 2. The FPN stage may be a set of outputs (see below) corresponding to a set of inputs of a context module stage, respectively.

[0083] In the second part 420, a set of multi-channel outputs from the context module stage of the neural network can be provided to the face detection stage and the face landmark localization stage of the neural network. The set of multi-channel outputs can be substantially similar to the output of the context module 131 of the neural network architecture 100. The face detection stage can be substantially similar to the first head 140, and the face landmark localization stage can be substantially similar to the second head 150. The set of multi-channel outputs can correspond to the input image of the neural network. In some implementations, the context module can use a 3×3 filter.

[0084] In Part 3, 430, a set of bounding boxes for the input image can be predicted during the face detection stage based on a set of multi-channel outputs from the context module stage. In some implementations, the face detection stage may use 1×1 filters (e.g., convolutional filters). For example, as discussed herein, prediction may use 1×1 convolutional shared weights applied to the multi-channel outputs of a set of context modules, and anchor box regression and classification may be used as discussed herein. In some implementations, the prediction of a set of bounding boxes may be completed in a single forward pass.

[0085] In Part 440, a set of face landmark indicators from the input image can be extracted during the face landmark localization stage based on a set of multi-channel outputs from the context module stage and a set of bounding boxes predicted during the face detection stage. In various implementations, the face landmark localization stage may include a RoIAlign layer. The face landmark localization stage can treat the bounding boxes from the first head as RoIs for feature extraction for landmark localization, and feature extraction can be performed using the RoIAlign layer, as discussed herein.

[0086] In Part 5, 450, an output including a set of facial landmark indicators can be provided. The output can be presented as a modified copy of the input image, i.e., another data file different from the input image, or as metadata embedded in the input image, as discussed herein.

[0087] Go to Figure 5 The method 500 for a neural network-based system for image processing may include a first part 510, a second part 520, a third part 530, a fourth part 540, a fifth part 550, a sixth part 560, a seventh part 570, and an eighth part 580.

[0088] In the first part 510, an image can be provided to the FPN of the neural network. The image can be substantially similar to the input image 105, and the neural network can have an architecture substantially similar to the neural network architecture 100.

[0089] In the second part 520, a set of feature maps C2 to CN of the final layer can be computed using the FPN based on the image. The FPN can be substantially similar to the FPN of neural network architecture 100 and can have a first part including the feature maps of the final layer (which can be substantially similar to the feature maps of the final layer 111). In various embodiments, the FPN can have a set of outputs corresponding to the inputs of a set of context modules (see below). For some embodiments, the set of feature maps is computed based on a set of final layer feature maps with a stride power of 2.

[0090] In Part 3, 530, a set of feature maps P2 to PN can be computed using the FPN based on a set of feature maps C2 to CN from the last layer. The FPN may have a second part that includes feature maps computed based on the feature maps from the last layer (which may be substantially similar to feature map 121).

[0091] In Part 4, 540, a set of inputs can be provided to a corresponding set of context modules, the set of inputs being based on a set of feature maps P2 to PN as discussed herein.

[0092] In section 550, a set of multi-channel outputs of the context module can be generated based on a set of inputs to the context module. This set of multi-channel outputs can be substantially similar to the output of the context module 131 of the neural network architecture 100. In some implementations, the context module may use a 3×3 filter.

[0093] In Part 6, 560, a set of bounding boxes for an image can be predicted at the first head of the neural network based on a set of multi-channel outputs of a context module. The first head can be a face detection head. In some implementations, the first head can use a 1×1 filter (e.g., a convolutional filter). For example, as discussed herein, prediction can use 1×1 convolutional shared weights applied to the multi-channel outputs of a set of context modules, and anchor box regression and classification as discussed herein can be used. In some implementations, the prediction of a set of bounding boxes can be completed in a single forward pass.

[0094] In Part 7, 570, a set of facial feature point indicators of an image can be extracted at a second head of the neural network based on a set of multi-channel outputs from the context module and a set of bounding boxes predicted at the first head. The second head can be a feature point localization head. In various implementations, the second head may include a RoIAlign layer. The second head can treat the bounding boxes from the first head as RoIs for feature extraction to perform feature point localization, and feature extraction can be performed using the RoIAlign layer, as discussed herein.

[0095] In Part 8, 580, an output including a set of facial landmark indicators can be provided. The output can be presented as a modified copy of the input image, i.e., another data file different from the input image, or as metadata embedded in the input image, as discussed.

[0096] Instructions for performing methods 300, 400, and / or 500 may be executed by a control unit having one or more processors based on instructions stored in the controller's memory (e.g., non-transitory memory such as magnetic storage media, optical storage media, or non-volatile storage media). The control unit and memory may be part of a computing system, which may be local or remote to the user. In some embodiments, the computing system may be located remotely from the user (e.g., in a cloud-based server), and the user may interact with the computing system (and thereby initiate one or more of methods 300, 400, and / or 500) through a suitable communication interface (e.g., a wired or wireless communication interface to the Internet).

[0097] The description of the embodiments has been presented for illustrative and descriptive purposes. Suitable modifications and changes to the embodiments may be made in light of the above description, or such suitable modifications and changes may be obtained by practical means. For example, unless otherwise indicated, suitable devices and / or combinations of devices (such as, regarding) may be used. Figures 1 to 5 The methods described may be executed by computing systems and / or cloud-based computing systems discussed. The methods may be executed by combining one or more logical devices (e.g., processors) with one or more additional hardware elements (such as storage devices, memories, image sensor / lens systems, light sensors, hardware network interfaces / antennas, switches, actuators, clock circuits, etc.) to execute stored instructions. The described methods and associated actions may also be executed in various orders other than those described in this application, in parallel, and / or simultaneously. The described systems are exemplary in nature and may include additional elements and / or omit elements. The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of the various systems and configurations disclosed with other features, functions, and / or properties.

[0098] As used in this application, an element or step described in the singular form and followed by the word "a (or an)" should be understood to not exclude multiple said elements or steps, unless such exclusion is specified. Furthermore, references to "an embodiment" or "an example" in this disclosure are not intended to be construed as excluding the existence of additional embodiments also incorporating the said features. Terms such as "first," "second," and "third" are used merely as labels and are not intended to impose numerical requirements or a particular order on their objects.

[0099] As used herein, references to “one implementation,” “some implementations,” or “various implementations” indicate that the described associated feature, structure, or characteristic appears in at least some, but not necessarily all, implementations. Furthermore, the appearance of these terms in various places does not necessarily refer to the same implementation. Moreover, terms using the “and / or” language to present elements in a list refer to any combination of the listed elements. For example, “A, B, and / or C” can mean any of the following: A alone; B alone; C alone; A and B; A and C; B and C; or A, B, and C.

[0100] The appended claims specifically state that subject matter derived from the above disclosure is considered novel and non-obvious.

Claims

1. A method for face detection using an image processing neural network, comprising: The input image is fed into the neural network for face detection; Collect a set of multi-channel outputs from a set of context modules corresponding to the neural network, wherein the set of multi-channel outputs corresponds to the input image; The set of multi-channel outputs is provided to both the first head and the second head of the neural network, wherein the first head is a face detection head; wherein the second head is a feature point localization head; and wherein the first head and the second head share a representation. Based on the set of multi-channel outputs, the first head is used to predict a set of boundary regions of the input image; Based on the set of multi-channel outputs and the set of boundary regions, the second head is used to extract a set of feature point indicators of the input image; as well as Provides an output that includes the set of feature point indicators.

2. The method of claim 1, wherein the second head comprises a RoIAlign layer.

3. The method of claim 1, wherein the prediction of the set of boundary regions is performed in a single forward pass.

4. The method according to claim 1, further comprising: The input image is provided to a feature pyramid network with a set of feature maps.

5. The method of claim 4, wherein the feature pyramid network has a set of outputs, each of which corresponds to an input of the set of context modules.

6. The method of claim 4, wherein the set of feature maps is calculated based on a set of final layer feature maps with a stride power of 2.

7. The method of claim 1, wherein the context module uses a 3×3 filter.

8. The method of claim 1, wherein the first head uses a 1×1 filter.

9. A method for processing images using a neural network, the method comprising: A set of multi-channel outputs from the context module stage of the neural network is provided to both the face detection stage and the face feature point localization stage of the neural network. The set of multi-channel outputs corresponds to the input image of the neural network. The face detection stage is a face detection head, the face feature point localization stage is a feature point localization head, and the face detection head and the feature point localization head share a representation. In the face detection stage, a set of bounding boxes of the input image is predicted based on the set of multi-channel outputs of the context module stage; In the face feature point localization stage, a set of face feature point indicators of the input image are extracted based on the set of multi-channel outputs of the context module stage and the set of bounding boxes predicted in the face detection stage. as well as Provides an output including the set of facial feature point indicators.

10. The method for processing images using a neural network according to claim 9, wherein the set of facial feature point indicators is extracted using a RoIAlign layer.

11. The method for processing an image using a neural network according to claim 9, wherein the prediction of the set of bounding boxes is performed in a single forward pass.

12. The method for processing images using a neural network according to claim 9, further comprising: The input image is provided to the feature pyramid network stage of the neural network, which has a set of feature maps.

13. The method for processing images using a neural network according to claim 12, wherein the feature pyramid network stage has a set of outputs, each of which corresponds to a set of inputs of the context module stage.

14. The method for processing images using a neural network according to claim 12, wherein the set of feature maps is calculated based on a set of final layer feature maps with a stride power of 2.

15. The method for processing images using a neural network according to claim 9, wherein the context module stage includes a 3×3 filter; and wherein the face detection head uses a 1×1 filter.

16. A neural network system for image processing, comprising: The control unit has one or more processors; as well as A non-transitory memory storing executable instructions that, when executed by the control unit, cause the control unit to: Provide images to the feature pyramid network of the neural network; The feature pyramid network is used to compute a set of final layer feature maps C2 to CN based on the image; The feature pyramid network is used to calculate a set of feature maps P2 to PN based on the set of last layer feature maps C2 to CN; Based on the set of feature maps P2 to PN, a set of inputs is provided to a set of corresponding context modules; The set of context modules generates a set of multi-channel outputs based on the set of inputs of the context modules; At the first head of the neural network, a set of boundary regions of the image are predicted based on the set of multi-channel outputs of the context module; At the second head of the neural network, a set of facial feature point indicators of the image are extracted based on the set of multi-channel outputs of the context module and the set of boundary regions predicted at the first head, wherein the first head is a face detection head; wherein the second head is a feature point localization head, and wherein the first head and the second head share a representation; and Provides an output including the set of facial feature point indicators.

17. The neural network system for image processing according to claim 16, wherein the set of facial feature point indicators is extracted using a RoIAlign layer.

18. The neural network system for image processing according to claim 16, wherein the prediction of the set of boundary regions is performed in a single forward pass.

19. The neural network system for image processing according to claim 16, wherein the stride of the last layer feature maps C2 to CN is a power of 2; wherein the context module uses a 3×3 filter; and wherein the first head uses a 1×1 filter.

Images