Convolutional Neural Network-Based Landmark Tracker

Abstract

Systems and methods for facial landmark detection using a convolutional neural network (CNN) are provided. The CNN includes a first stage and a second stage, wherein the first stage produces an initial heat map of the landmarks and initial corresponding positions of the landmarks. The second stage processes the heat map and performs region of interest based pooling while maintaining feature alignment to produce cropped features. Finally, the second stage predicts corresponding refined position offsets from the cropped features to each corresponding initial position. Each corresponding initial position is combined with its corresponding refined position offset to provide a corresponding final coordinate (x, y) for each corresponding landmark in the image. The two-stage positioning design facilitates fine-level alignment while maintaining computational efficiency. The resulting architecture is small enough in size and inference time to be suitable for real-time web applications such as product simulation and virtual reality.

Description

Technical Field

[0001] The present disclosure relates to improvements in computers and computer processing, particularly image processing and neural networks, and more particularly to convolutional neural network based landmark tracker systems and methods. Background Art

[0002] Facial landmark detection, the process of locating predefined landmarks on a person's face in an image, is a common desire in many image processing / computer vision applications. Interesting image processing applications that provide practical applications may include facial recognition, animation, and augmented reality uses, among others. An example of augmented reality image processing is a virtual try-on application, such as for applying cosmetics or other products to a person. The task of a virtual makeup try-on application is to render the cosmetics in the correct position under different lighting, poses, and face shape variations. Precise alignment, especially for frontal facial poses that are common in virtual try-on applications, is required to provide an accurate and enjoyable experience. Furthermore, for client-side web applications, loading times are extremely important and GPUs required to quickly execute large neural network architectures cannot be effectively utilized.

[0003] While these resource limitations are not a major concern for existing face alignment architectures [1][2][3] (see the list of references below, each of which is incorporated herein by reference), in order to strike a better balance for real-time applications, an ideal architecture would be expected to minimize overhead and execution time while maintaining or improving alignment accuracy. Summary of the invention

[0004] In the proposed architecture, the first stage makes initial predictions from which crops of shared convolutional features are extracted; these regions of interest are then processed by the second stage to produce refined predictions. This two-stage localization design helps achieve fine-level alignment while maintaining computational efficiency. The resulting architecture is small enough in both size and inference time to be suitable for real-time web applications.

[0005] In one aspect, a computing device is provided, comprising a processing unit and a storage device coupled to the processing unit, the storage unit storing instructions that, when executed by the processing unit, configure the computing device to process an image to determine a corresponding position of each of a plurality of landmarks by: processing the image using a convolutional neural network (CNN) having a first stage and a second stage, wherein: the first stage processes the image to generate an initial prediction of the corresponding position; and the second stage uses intermediate features generated from the image by the first stage and the initial prediction to generate a crop of shared convolutional features for a region of interest to define a crop feature, and further processes the crop feature to produce a corresponding refined position offset to the initial prediction of the corresponding position.

[0006] In one aspect, a computing device is provided, comprising a processing unit and a storage device coupled to the processing unit, the storage unit storing instructions that, when executed by the processing unit, configure the computing device to process an image to determine a corresponding position of each of a plurality of landmarks by: processing the image using a convolutional neural network (CNN) having a first stage and a second stage, wherein: the first stage receives the image and determines: in a first portion of the first stage, a volume of intermediate features from the image; and in a second portion of the first stage, using the volume of the intermediate features for a corresponding initial position of each landmark within the image; and the second stage receives the volume of the intermediate features in a first portion of the second stage and receives the corresponding initial position of each landmark in a second portion of the second stage, the second stage further: processes the volume to further refine the intermediate features; performs region of interest based pooling on the further refined intermediate features while maintaining feature alignment to produce a cropped feature for each of the plurality of landmarks; and uses the cropped features and determines, for each landmark, a corresponding refined position offset for the corresponding initial position of each landmark; and wherein the processing further operates to combine each corresponding initial position and the corresponding refined position offset to determine a final position coordinate in the image for each of the plurality of landmarks.

[0007] In one aspect, a computing device including a processor and a storage device is provided, the computing device being configured to process an image via a convolutional neural network (CNN) to detect corresponding positions of multiple landmarks in the image, the CNN comprising: a two-stage localization architecture, wherein a first stage employs a first heat map to determine initial coordinates of corresponding positions, and a second stage employs a second heat map to determine a refined offset of the initial coordinates, the second stage further using a region of interest to pool each individual landmark to reduce overlapping calculations to avoid irrelevant regions and guide the generation of relevant shared features; and wherein the CNN is trained with an auxiliary coordinate regression loss to minimize the size of the corresponding heat map and computing resource usage.

[0008] Methods, computer program products and other aspects will be apparent to one of ordinary skill in the art.A computer program product as used herein includes a non-transitory storage device storing instructions that, when executed by a processing unit, configure a computing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a network diagram showing a two-stage CNN.

[0010] Figure 2 Schematic diagram of the inverted residual block of the feature map / volume.

[0011] Figure 3 yes Figure 1Diagram of the prediction block for the feature map / volume of the second stage of the network.

[0012] Figure 4 and Figure 5 It is an operation flow chart.

[0013] The concepts of the present invention are best described by certain embodiments thereof, which are described herein with reference to the accompanying drawings, wherein like reference numerals refer to like features throughout. It should be understood that when the term invention is used herein, it is intended to imply the inventive concepts underlying the embodiments described below and not just the embodiments themselves. It should be further understood that the overall inventive concepts are not limited to the illustrative embodiments described below, and the following description should be read in light of such a view. More than one inventive concept may be shown and described, and unless otherwise stated, each inventive concept may be independent or combined with one or more other inventive concepts. DETAILED DESCRIPTION

[0014] 1. Context

[0015] 1.1 Facial landmark alignment

[0016] The facial landmark alignment problem has a long history of classical computer vision solutions. For example, algorithms based on fast ensemble trees [4] achieve reasonable accuracy and are widely used for real-time face tracking [5]. However, the model size required to achieve such accuracy is prohibitively large.

[0017] Convolutional neural network based methods have achieved state-of-the-art facial landmark alignment accuracy. To maximize the accuracy on challenging datasets [6][7][8], large neural networks are used which are non-real-time and have model sizes ranging from tens to hundreds of megabytes (MB) [3][9], resulting in unreasonable loading times for web applications.

[0018] 1.2 Efficient CNN Architecture

[0019] In order to apply the performance of convolutional neural networks to mobile vision applications, many architectures with efficient building blocks have been published recently, such as MobileNetV2

[10] , ShuffleNet

[11] , and ShuffleNet

[12] . The goal of these networks is to maximize the performance (e.g., classification accuracy) under a given computational budget, which consists of the number of required learnable parameters (model size) and multiplication-addition operations.

[0020] We focus on MobileNetV2, whose inverted residual blocks can be used for implementation of this design. MobileNetV2 uses depthwise convolutions on top of regular convolutions, which significantly reduces the number of multiplications-additions and learnable parameters, but with a slight performance penalty

[13] . In addition, the inverted design, based on the principle that the network's expressiveness can be separated from its capacity, allows for a significant reduction in the number of cross-channel computations within the network

[10] . Finally, the residual design drawn from ResNet

[14] alleviates the problem of gradient propagation in deep networks.

[0021] 1.3 Heatmap

[0022] Fully convolutional neural network architectures based on heatmap regression

[15]

[16]

[17]

[18] have been widely used in the task of human pose estimation. The use of heatmaps provides a high degree of accuracy while also providing an intuitive way to see the network's understanding and confidence in the landmark regression. This technique has also been used in recent face alignment algorithms such as the Stacked Hourglass architecture [3]. However, the Stacked Hourglass method [3] uses high-resolution heatmaps, which require a lot of computation in the decoding layer. There is room for optimization here because the heatmap only has non-negligible values ​​in a very concentrated and small part of the overall image. This observation motivates us to use region processing, which allows the network to focus processing on relevant regions (i.e., roughly the region of interest).

[0023] 1.4 Mask-RCNN

[0024] There are a range of flexible and robust frameworks for object detection and semantic segmentation, such as Fast R-CNN

[19] , Faster R-CNN

[20] , and Fully Constitutional Network

[21] . Faster R-CNN uses a multi-branch design to perform bounding box regression and classification in parallel. Mask-RCNN

[22] is an extension of Faster-RCNN and adds new branches for predicting segmentation masks based on each region of interest. Of particular interest is Mask-RCNN's use of RoIAlign

[22] (where RoI comes from the acronym for the term "region of interest"), which allows significant savings in computation time by extracting crops from shared convolutional features. By doing so, it avoids recomputing features for overlapping regions of interest.

[0025] 1.5 Verification

[0026] To keep the output face shape valid, a validation step can be performed before returning the final prediction, for example, to prevent returning strange shapes when there is no face, a partial face, or the face is over-rotated. To get a standard reference for face shapes, principal component analysis can be used to get the first 100 main clusters from the training dataset. The minimum distance between the transformed predicted shape and one of the cluster centers can be determined. This minimum distance is used as a score to verify whether the predicted shape is valid.

[0027] 2. Computing Devices, Systems, Methods, and Other Aspects

[0028] Here are some of the features described in this article:

[0029] -RoIAlign

[22] is used for each individual landmark to save potentially overlapping computations, allowing the network to avoid irrelevant regions, and forcing the network to learn to produce good shared features. In the example, the 8x8 heatmap from stage 1 indicates the coordinates of each (facial) landmark. These landmarks can be calculated using the masked mean method to form coordinates (x, y). RoIAlign uses the predicted coordinates from the first stage to crop the intermediate feature map of uniform size 4x4. For example, suppose there is a first landmark predicted to be at (0.5, 0.5) in normalized coordinates. Then a 32x32 feature map is cropped. The cropped box will be [(14.0, 14.0), (18.0, 18.0)][top_left_corner, bottom_right_corner].

[0030] - The proposed two-stage localization architecture together with the auxiliary coordinate regression loss allows the use of extremely small and computationally cheap heatmaps in both stages. Two losses can be combined together - the heatmap loss and the coordinate distance loss.

[0031] 2.1 Model Structure

[0032] like Figure 1 As shown, the CNN model has two stages and is trained end-to-end. Figure 1 A CNN 100 is shown comprising a stream of layers and / or blocks having an output volume (feature map) starting from an input layer 102 (e.g., an image of a face with an example of facial landmarks) of size 128x128x3 (height, width, and color). Other sizes may be used, such as 224x224x3. The size may be adjustable. Different sizes (resolutions) may be used for different purposes.

[0033] The flow includes two stages, a first stage 104 and a second stage 106. The first and second stage flows are defined according to groups of corresponding layers and / or blocks including first stage layers / blocks and second stage layers / blocks, each having a first portion and a second portion. As will be appreciated by those skilled in the art, the groups of these layers / blocks (e.g., 108, 110, 114, and 116) are represented by arrows between feature maps / volumes. The first stage 104 includes a group 108 in the first portion and a group 110 in the second portion, while the second stage 106 includes a group 114 in the first portion and a group 116 in the second portion. Groups 108 and 110 may also be referred to as first groups 108 and second groups 110 in the first stage. Groups 114 and 116 may also be referred to as first groups 114 and second groups 116 in the second stage. The first stage 104 further includes a layer 112, and the second stage 106 further includes a layer 118. These layers 112 and 120 are combined at layer 120 to provide the output of CNN 100, as further described.

[0034] Figure 1 The shading legend indicates the type of processing operation of each layer and / or block of CNN 100. In more detail, group 108 includes a convolutional layer 108A of size 64x64x8 and inverted residual blocks 108B and 108C of size 64x64x8 and 32x32x8, respectively. It should be understood that the size of each block or layer refers to the size of the output feature map. Figure 2 The general form of the extended inverted residual block according to

[10] is shown in . Group 110 includes inverted residual blocks 110A-110D of respective sizes 16x16x16, 8x8x32, 8x8x32 and 8x8x#L, where #L=number or count of multiple flags. After training and testing, #L=16. Other flag counts (sizes) (e.g., #L) (e.g., 65, 86, etc.) may be implemented. The value of #L may be adjustable. Following group 110 is layer 112, a get mask average layer of size #Lx2.

[0035] The output of group 108 (e.g., after 118C) is the intermediate feature map (or sometimes referred to as the volume of intermediate features) of the first stage 104, which is shared with (e.g., input to) the second stage 106 at group 114. Group 114 includes inverted residual blocks 114A-114C of respective sizes 32x32x8, 32x32x16, and 32x32x16.

[0036] The output of group 114 (e.g., the intermediate feature map further refined by the processing of blocks 114A-114C) is processed by group 116 together with the output of layer 112 (representing the initial position of the landmark). Group 116 includes #L RoI cropping+concatenation blocks for each of the #L landmarks (processed by block 116). 1 , 1162 ,…116 #L denoted by ), where each of the #L blocks has a size of 4x4x16, which when concatenated gives a 4x4x16 #L output feature map. The concatenated feature map is provided to predict a block 117 or group 116 having a size of 4×4×#L. Figure 3 The prediction block 117 is extended in the middle.

[0037] In turn, the output of the prediction block 117 is provided to layer 118, a second acquisition mask average layer of size #Lx2. The respective outputs of the two layers 112 and 118 represent the initial position of the #L marker and the refinement offset thereto. These are provided to an output layer 120, also of size #LX2, so that when combined, for each of the individual #L markers, an (x, y) coordinate relative to the input layer 102 is produced.

[0038] Thus, the first stage 104 shows a series of inverted residual blocks that predict 8×8 heatmaps via 110D, one heatmap for each facial landmark. The normalized activations on the heatmaps are interpreted as probability distributions, and the expected values ​​of these heatmaps are calculated to obtain x,y coordinates. This will be described in more detail below.

[0039] The second stage has several shared layers / blocks that branch off from parts of the first stage. Using the initial predictions from the previous stage (the intermediate feature maps from group 108 after block 108C, as further refined by group 114 after block 114C), RoIAlign

[22] is applied to the final shared convolutional features. Each cropped feature is input to a final convolutional layer (of prediction block 117) that has separate weights for each individual landmark. Prediction block 117 implements this in a straightforward way using group convolutions

[12] . The output of 117 is a heatmap for each landmark. The coordinates obtained from these heatmaps indicate the offsets needed for the initial “rough” predictions, i.e., if the heatmaps are perfectly centered at this stage, then there is effectively no refinement.

[0040] The region of interest based pooling performed by group 116 uses the predictions of the first stage (from layer 112) as the cropped centers with coordinates [x_c, y_c] for each landmark, which are obtained from applying the masked averaging layer at 112 to the coarse heatmap from 110D. Group 116 (via prediction block 117) uses these cropped features (e.g., from block 116 1 , 116 2 ,…116 #L The concatenated output of y_r (also first predict the heatmap, then use the masked average layer to get the refined shift distance [x_r, y_r]). The final prediction (output layer) adds the coarse prediction from the first stage and the refined prediction from the second stage.

[0041] 2.2 Coordinate regression from heatmaps

[0042] For the ground truth heatmap, a Gaussian distribution is used, with the mode corresponding to the location of the ground truth coordinates. Let x,y denote the coordinates of any pixel in the feature map, the value can be calculated using the following distribution:

[0043]

[0044] Where (x i ,y i ) are the corresponding marker coordinates. In the experiment, σ x ,σ y Both are set to 0.8.

[0045] (For example)

[0046] Based on obtaining the mask average layer (e.g., each of 112 and 120), regressing x pred ,y pred is the expected value of the pixel location according to the distribution computed from the heatmap predicted by the CNN (e.g., “predicted heatmap”). Let j index all pixels in the predicted heatmap, and w j Represents the heatmap value of the pixel:

[0047]

[0048] 2.3 Loss Function

[0049] The loss function uses pixel-wise sigmoid cross entropy

[23] to learn the heatmap.

[0050] In addition, to alleviate the problem of heatmaps of landmarks being cut off near the border, L 2 Distance loss:

[0051]

[0052]

[0053] in is the predicted value of the heat map in the lth channel at the pixel position (i, j) of the nth sample, and is the corresponding basic reality. is the weight at pixel position (i, j) for position , which is calculated by Equation 4. is the ground truth coordinate of the lth marker of the nth sample; and are the predicted coordinates of the same sign. Here L is the number of signs, and H and W are the height and width of the heatmap (e.g. 8x8). The auxiliary coordinate regression loss is the mean squared error loss in the second line of equation (3). A loss function comprising a combined pixel-wise sigmoid cross entropy loss and an L2 loss is applied to each corresponding stage so that each stage has its own loss determination during training. As mentioned earlier, the use of two stages facilitates smaller heatmaps and thus computational resource consumption.

[0054] 2.4 blocks

[0055] 2.4.1 Inverted Residual Block

[0056] refer to Figure 2 , shows a general extension of the inverted residual block 200 substantially according to reference

[10] . Figure 2 The difference is that reference

[10] uses ReLU6, while this example uses ReLU. This general approach can be applied to the inverted residual block of CNN100, although some dimensions may be different.

[0057] Experiments have found that an expansion ratio of 5 best suits the competing needs of performance and efficiency.

[0058] The processing of the inverted residual block 200 is performed in the following order:

[0059] a. A channel-wise convolution with a 1x1 kernel followed by BatchNorm and ReLU activation with an input layer 202 of size HxWxC, where C represents the channel rather than the color itself. The output is a feature map 204 of size HxWxC*5;

[0060] b. A depth-wise convolution of the feature map 204 with a 3x3 kernel followed by BatchNorm and ReLU activation, providing an output (feature map) 206 with a size of HxWxC*5; and

[0061] c. A channel-wise convolution with a 1x1 kernel followed by BatchNorm on the feature map 206 and an addition operation with layer 202 provides an output with size HxWxC.

[0062] 2.4.2 Prediction Block

[0063] After concatenating all RoIAlign cropped features, the number of channels is equal to the number of input channels (e.g. 16) multiplied by the number of landmarks (#L). Since the refined heatmap of each landmark is predicted independently, it can be implemented using 16-channel-channel convolution

[12] , such as Figure 3 shown. Figure 3An extension of block 117 is shown working on an input of cropped and concatenated features (input feature map 117A) having a size of 4x4x16*l.

[0064] The prediction block 117 is performed in the following order:

[0065] a. A group-wise convolution with a 3x3 kernel followed by BatchNorm and ReLU activation on the input feature map 117A to output a feature map 206 with size 4x4x16*#L; and

[0066] b. Channel-wise convolution with 1x1 kernel followed by BatchNorm to output feature map 117C with size 4x4x#L (4x4 heatmaps are defined for each landmark of #L).

[0067] 2.5 Data Augmentation

[0068] Several common methods are used to perform data augmentation. For example, random rotations, shifts, and horizontal flips of the input image are used. To better equip the model to handle common occlusions such as glasses or hands, these objects are also randomly pasted into the image around the face.

[0069] 3. Results and comparison

[0070] The new model is trained using a batch size of 8 and a learning rate of 8ε -5 and SGD optimizer with momentum=0.9. The new model was evaluated on the internal test set and the distance error was calculated by normalizing the distance between the eye centers. The normalized error of the first stage was 3.35%, and the error of the whole model was 2.89%. The running time of the new model on an iPhone7 web browser is about 40ms / frame, and its total learnable parameters are about 300KB.

[0071] Tables 1 and 2 show comparative data for the new model, including comparisons to larger in-house models RF Tracker and MobileNet_V2_0.35_128.

[0072]

[0073] Table 1: Performance compared with an internal RF tracker.

[0074]

[0075] Table 2: Performance comparison with MobileNetV2

[10] .

[0076] Figure 4 and Figure 5is an operational flow diagram illustrating aspects of a computer-implemented method. Figure 4 Operation 400 of a computer-implemented method is shown. At 402, an image is received for processing. The image may be a self-portrait image or a video-based self-portrait image. The image may be received from a camera that is a component of a computing device or system that performs the method. This may be a mobile device, a kiosk on a product counter, a tablet, etc. Other form factors and computing devices and systems will be apparent. A cloud-based or other service-based system may be envisioned in which a local computing device may receive an image via a local camera and provide the image to a remote computing device configured to perform as a service. The service may be provided via a local application or browser of the local computing device.

[0077] The image may be pre-processed, for example, by scaling to a particular size (step 404). At 406, an operation processes the image using a convolutional neural network (CNN) having a first stage and a second stage to determine a corresponding position of each of a plurality of landmarks. The first stage processes the image to generate an initial prediction of the corresponding position. The second stage generates a crop of the shared convolutional features for the region of interest using the intermediate features generated from the image by the first stage and the initial prediction to define a crop feature, and further processes the crop feature to produce a corresponding refined position offset to the initial prediction of the corresponding position.

[0078] The CNN may combine the initial predictions of the corresponding positions and the corresponding refined position offsets to provide a corresponding position for each of the plurality of markers. The corresponding position of the marker may include a final position determined by combining the initial predictions with the corresponding refined position offsets, or may include a position data set including the initial predictions and the corresponding refined position offsets. The position data may be provided in any form for use, such as modifying an image of at least one marker position (step 408).

[0079] The first stage of the CNN can generate and use intermediate features to produce an initial heat map, from which initial predictions are generated.

[0080] The second stage can generate a second heat map from the cropped features, one second heat map for each landmark, and use the second heat map to generate corresponding refined position offsets.

[0081] Figure 5 is a flow chart illustrating the operation 500 of a computer-implemented method. The operation may be performed by a computing device or system such as described herein (e.g., with respect to Figure 4 or others). Steps 502 and 504 are similar to steps 402 and 404. Step 510 is similar to step 408, wherein the image is modified.

[0082] Step 506 shows processing the image to determine the respective positions of each of the plurality of landmarks using a convolutional neural network (CNN) having a first stage and a second stage. The first stage receives the image and determines: in a first portion of the first stage, a volume of an intermediate feature from the image; and, in a second portion of the first stage, using the volume of the intermediate feature, a respective initial position for each landmark within the image. The second stage receives the volume of the intermediate feature in a first portion of the second stage, and receives the respective initial position for each landmark in a second portion of the second stage.

[0083] The second stage further operates to: process the volume to further refine the intermediate features; perform region of interest based pooling on the further refined intermediate features while maintaining feature alignment to produce cropped features for each of the multiple landmarks; and use the cropped features and determine, for each landmark, a corresponding refined position offset for each corresponding initial position of each landmark.

[0084] Operation 500 may further operate (eg, at 508 ) to combine each respective initial position and the respective refined position offset to determine a final position coordinate in the image for each marker of the plurality of markers.

[0085] In operation 500 , a second portion of the first stage may determine an initial heat map using the volume of the intermediate features and use the initial heat map to predict each corresponding initial position of each landmark.

[0086] exist Figure 4 or Figure 5 In the operation of, at least some of the first stage may include a series of inverted residual blocks, and at least some of the second stage may include a series of inverted residual blocks. The second stage may: perform region of interest based pooling using RoIAlign while maintaining feature alignment to produce cropped features; and concatenate cropped features.

[0087] exist Figure 4 or Figure 5 In the operation, the second stage may include a prediction block to process the cropped features, which is performed in the following order: channel-wise convolution with a 3×3 kernel, followed by BatchNorm and ReLU activation; and group-wise channel convolution with a 1×1 kernel, followed by BatchNorm; outputting each corresponding refined position offset.

[0088] exist Figure 4 or Figure 5In the operation of, the CNN model can be trained using a corresponding training image having a ground truth heat map of a corresponding landmark among a plurality of landmarks defined according to a Gaussian distribution, having a pattern corresponding to a corresponding coordinate position of the ground truth heat map in the corresponding training image. The Gaussian distribution can be defined as previously described above.

[0089] The CNN in operation 400 or 500 can be trained with a loss function defined by pixel-by-pixel sigmoid cross entropy for learning the heat map. The loss function may further include an L2 distance loss. The loss function may be as previously shown and described above.

[0090] In operation 400 or 500, modifying the image may include simulating a product applied to the image. The image may be a video image, and the method may present the modified image in real time to simulate virtual reality.

[0091] In operation 400 or 500, the logo may be a facial logo and the image may include a face. A corresponding operation may include updating the image with at least one product simulation using a corresponding position of the logo.

[0092] In addition to the computing device (or system) aspects and the method aspects, one of ordinary skill will understand that a computer program product aspect is disclosed in which instructions are stored in a non-transitory storage device (e.g., memory, CD-ROM, DVD-ROM, disk, etc.) to configure a computing device to perform any method aspect disclosed herein.

[0093] It will be appreciated that the CNN may provide corresponding landmark locations for further processing of the image.For example, a computing device may be configured via instructions to receive an image and perform landmark detection on the image using a CNN.

[0094] The instructions may configure the computing device to modify an image at or around at least one marker using the final coordinates of the at least one marker. The image (an example of modification) may be annotated at or around at least one marker, for example, to display a bounding box or region, to display a mask, etc. Modifying the image may include simulating a product applied to the image. The product may be a makeup product, for example when the image is a face and the marker is a facial marker. The image may be a video image, and the computing device may be configured via the instructions to modify and present the image in real time to simulate virtual reality. The computing device may further include a camera, and the video may be a selfie video taken by the camera.

[0095] Actual implementations may include any or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, devices, systems, components, program products, and other combinations of features described herein for performing functions. Many embodiments have been described. However, it will be understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided from the described processes, or steps may be eliminated, and other components may be added to the described systems, or removed from the described systems. Therefore, other embodiments are within the scope of the appended claims.

[0096] With respect to the embodiments of the present invention described above, they may be implemented at least in part using a software-controlled programmable processing device, such as a general-purpose processor or a special-purpose processor, a digital signal processor, a microprocessor or other processing device, a data processing apparatus or a computer system, and it will be understood that a computer program for configuring a programmable device, an apparatus or a system to implement the aforementioned method, apparatus and system is contemplated as an aspect of the present invention. The computer program may be embodied as any suitable type of code, such as source code, object code, compiled code, interpreted code, executable code, static code, dynamic code, etc. Instructions may be implemented using any suitable high-level, low-level, object-oriented, visual, compiled and / or interpreted programming language, such as C, C++, Java, BASIC, Perl, Matlab, Pascal, visual BASIC, Java, ActiveX, assembly language, machine code, etc. The skilled person will readily understand that the term "computer" in its most general sense includes programmable devices such as those mentioned above, as well as data processing apparatus and computer systems in any format in which they may appear, such as desktop personal computers, laptop personal computers, tablet computers, smart phones or other computing devices.

[0097] Suitably, the computer program is stored in a machine-readable form on a carrier medium, for example, the carrier medium may include a memory, a removable or non-removable medium, an erasable or non-erasable medium, a writable or rewritable medium, a digital or analog medium, a hard disk, a floppy disk, a compact disk read-only memory (CD-ROM), a recordable compact disk (CD-R), a rewritable compact disk (CD-RW), an optical disk, a magnetic medium, a magneto-optical medium, a removable memory card or disk, various types of digital versatile disk (DVD) user identification modules, a magnetic tape, a cassette solid-state memory. The computer program may be provided from a remote source contained in a communication medium, such as an electronic signal, a radio frequency carrier wave or an optical carrier wave. Such a carrier medium is also contemplated as an aspect of the present invention.

[0098] Throughout the description and claims of this specification, the words "comprise" and "comprising" and their variations mean "including but not limited to", and are not intended to (and do not) exclude other components, integers or steps. Throughout the specification, the singular encompasses the plural, unless the context requires otherwise. In particular, where the indefinite article is used, the specification should be understood as contemplating plural as well as singularity, unless the context requires otherwise.

[0099] Features, integer features, compounds, chemical moieties or groups described in conjunction with a particular aspect, embodiment or example of the invention are to be understood to be applicable to any other aspect, embodiment or example unless incompatible therewith. All features disclosed herein (including any accompanying claims, abstract and drawings) and / or all steps of any method or process so disclosed may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. The invention is not limited to the details of any foregoing examples or embodiments. The invention extends to any novel one or any novel combination of the features disclosed in this specification (including any accompanying claims, abstract and drawings), or to any novel one or any novel combination of the steps of any method or process disclosed.

[0100] As used herein, the terms "comprises," "includes," "including," "has," "having," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, "or" refers to an inclusive or and not to an exclusive or.

[0101] In addition, "a" or "an" is used to describe elements and components of the present invention. This is done only for convenience and to give a general sense of the present invention. This description should be understood to include one or at least one, and the singular also includes the plural, unless it is obvious that there is another meaning.

[0102] In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

[0103] The scope of the present disclosure includes any new feature or combination of features disclosed therein or any generalization thereof, unless incompatible therewith, whether or not it is relevant to the claimed invention or mitigates any or all of the problems solved by the invention. The applicants hereby give notice that new claims may be formulated to such features during the prosecution of the present application or of any such further application derived therefrom. In particular, with reference to the appended claims, features from dependent claims may be combined with features of the independent claims, and features from respective independent claims may be combined in any appropriate manner and not merely in the specific combinations recited in the claims.

[0104] References

[0105] The following publications are incorporated herein by reference.

[0106] [1] M.Kowalski, J.Naruniec, and T.Trzcinski, "Deep alignment network: Aconvolutional neural network for robust face alignment," CoRR, vol.abs / 1706.01789, 2017.

[0107] [2]Y.Sun,X.Wang,and

[0108] [3]K.Yuen and MMTrivedi, "An occluded stacked hourglass approach to facial landmark localization and occlusion estimation," CoRR, vol.abs / 1802.02137, 2018.

[0109] [4]V.Kazemi and J.Sullivan,“One millisecond face alignment with anensemble of regression trees,”2014 IEEE Conference on Computer Vision andPattern Recognition,pp.1867–1874,2014.

[0110] [5]D.E.King,“Dlib-ml:A machine learning toolkit,”Journal of MachineLearning Research,vol.10,pp.1755–1758,2009.

[0111] [6]P.N.Belhumeur,D.W.Jacobs,D.J.Kriegman,and N.Kumar,“Localizingparts of faces using a consensus of exemplars,”IEEE Transactions on PatternAnalysis and Machine Intelligence,vol.35,pp.2930–2940,Dec 2013.

[0112] [7]V.Le,J.Brandt,Z.Lin,L.Bourdev,and T.S.Huang,“Interactive facialfeature localization,”in Computer Vision–ECCV 2012(A.Fitzgibbon,S.Lazebnik,P.Perona,Y.Sato,and C.Schmid,eds.),(Berlin,Heidelberg),pp.679–692,SpringerBerlin Heidelberg,2012.

[0113] [8]G.Trigeorgis,P.Snape,M.A.Nicolaou,E.Antonakos,and S.Zafeiriou,“Mnemonic descent method:A recurrent process applied for end-to-end facealignment,”2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR),pp.4177–4187,2016.

[0114] [9]A.Newell,K.Yang,and J.Deng,“Stacked hourglass networks for humanpose estimation,”CoRR,vol.abs / 1603.06937,2016.

[0115]

[10] M.Sandler,A.G.Howard,M.Zhu,A.Zhmoginov,and L.Chen,“MobileNetV2:Inverted residuals and linear bottlenecks:Mobile networks for classification,detection and segmentation,”CoRR,vol.abs / 1801.04381,2018.

[0116]

[11] F.N.Iandola,M.W.Moskewicz,K.Ashraf,S.Han,W.J.Dally,and K.Keutzer,“Squeezenet:Alexnet-level accuracy with 50x fewer parameters and <1mb modelsize,”CoRR,vol.abs / 1602.07360,2016.

[0117]

[12] X.Zhang,X.Zhou,M.Lin,and J.Sun,“Shufflenet:An extremely efficientconvolutional neural network for mobile devices,”CoRR,vol.abs / 1707.01083,2017.

[0118]

[13] A.G.Howard,M.Zhu,B.Chen,D.Kalenichenko,W.Wang,T.Weyand,M.Andreetto,and H.Adam,“Mobilenets:Efficient convolutional neural networksfor mobile vision applications,”CoRR,vol.abs / 1704.04861,2017.

[0119]

[14] K.He,X.Zhang,S.Ren,and J.Sun,“Deep residual learning for imagerecognition,”CoRR,vol.abs / 1512.03385,2015.

[0120]

[15] A.Bulat and G.Tzimiropoulos,“Human pose estimation viaconvolutional part heat map regression,”CoRR,vol.abs / 1609.01743,2016.

[0121]

[16] S.Wei,V.Ramakrishna,T.Kanade,and Y.Sheikh,“Convolutional posemachines,”CoRR,vol.abs / 1602.00134,2016.

[0122]

[17] Y.Chen,C.Shen,X.Wei,L.Liu,and J.Yang,“Adversarial learning ofstructure-aware fully convolutional networks for landmark localization,”CoRR,vol.abs / 1711.00253,2017.

[0123]

[18] E.Insafutdinov,L.Pishchulin,B.Andres,M.Andriluka,and B.Schiele,“Deepercut:A deeper,stronger,and faster multi-person pose estimation model,”CoRR,vol.abs / 1605.03170,2016.

[0124]

[19] R.B.Girshick,“Fast R-CNN,”CoRR,vol.abs / 1504.08083,2015.

[0125]

[20] S.Ren,K.He,R.B.Girshick,and J.Sun,“Faster R-CNN:towards real-timeobject detection with region proposal networks,”CoRR,vol.abs / 1506.01497,2015.

[0126]

[21] J.Long,E.Shelhamer,and T.Darrell,“Fully convolutional networksfor semantic segmentation,”CoRR,vol.abs / 1411.4038,2014.

[0127]

[22] K.He,G.Gkioxari,P.Dollàr,and R.B.Girshick,“Mask R-CNN,”CoRR,vol.abs / 1703.06870,2017.

[0128]

[23] N.Zhang,E.Shelhamer,Y.Gao,and T.Darrell,“Fine-grained poseprediction,normalization,and recognition,”CoRR,vol.abs / 1511.07063,2015.

Claims

1. A computing device comprising a processing unit and a storage unit coupled to the processing unit, the storage unit storing instructions that, when executed by the processing unit, configure the computing device to process an image to determine a respective position of each of a plurality of landmarks by: The image is processed using a convolutional neural network (CNN) having a first stage and a second stage, wherein: The first stage processes the image to generate an initial prediction of the corresponding location; and The second stage generates a crop of shared convolutional features for a region of interest using the intermediate features generated from the image by the first stage and the initial prediction to define a cropped feature, and further processes the cropped feature to produce a corresponding refined position offset to the initial prediction for the corresponding position.

2. The computing device of claim 1, wherein: The first stage generates and uses the intermediate features to produce an initial heat map, from which the initial prediction is generated.

3. A computing device according to claim 1 or claim 2, wherein: The second stage generates a second heat map from the cropped features, one second heat map for each landmark, and uses the second heat map to generate the corresponding refined position offsets.

4. A computing device according to claim 1 or claim 2, wherein: The CNN combines the initial prediction of the corresponding position and the corresponding refined position offsets to provide the corresponding position for each of a plurality of landmarks.

5. A computing device comprising a processing unit and a storage unit coupled to the processing unit, the storage unit storing instructions that, when executed by the processing unit, configure the computing device to process an image to determine a respective position of each of a plurality of landmarks by: The image is processed using a convolutional neural network (CNN) having a first stage and a second stage, wherein: The first stage receives the image and determines: in a first part of said first stage, a volume of intermediate features from said image; as well as in a second part of said first stage, using said volume of intermediate features for a respective initial position of each of said landmarks within said image; and The second stage receives the volume of the intermediate feature in a first portion of the second stage and receives the respective initial position of each of the markers in a second portion of the second stage, the second stage further: processing the volume to further refine the intermediate features; performing region-of-interest based pooling on the further refined intermediate features while maintaining feature alignment to generate a cropped feature for each of the plurality of landmarks; as well as using the cropping features and determining, for each landmark, a corresponding refined position offset for each corresponding initial position of each landmark; and Wherein the processing further operates to combine each respective initial position and the respective refined position offset to determine a final position coordinate in the image for each of the plurality of the markers.

6. The computing device of claim 5, wherein: The second part of the first stage uses the volume of intermediate features to determine an initial heat map, and uses the initial heat map to predict each corresponding initial position of each of the landmarks.

7. The computing device of claim 1 or claim 5, wherein: At least some of the first stages include a series of inverted residual blocks, and wherein at least some of the second stages include a series of inverted residual blocks.

8. The computing device of claim 1 or claim 5, wherein: The second stage: Using RoIAlign for region of interest based pooling while maintaining feature alignment to produce the cropped features; and The cropping features are concatenated.

9. The computing device of claim 1 or claim 5, wherein: The second stage includes a prediction block for processing the cropped features, and the prediction block is executed in the following order: Channel-wise convolution with 3×3 kernel followed by BatchNorm and ReLU activation; and Group-wise channel-wise convolution with 1×1 kernel followed by BatchNorm; To output each of the corresponding refined position offsets.

10. The computing device of claim 1 or claim 5, wherein: The CNN model is trained using corresponding training images, wherein the corresponding training images have ground truth heat maps of corresponding landmarks among the plurality of landmarks defined according to a Gaussian distribution, and have patterns corresponding to corresponding coordinate positions of the ground truth heat maps in the corresponding training images.

11. The computing device of claim 10, wherein: The Gaussian distribution is defined as follows: where x,y represent the coordinates of any pixel in the training image, and (x i ,y i ) are the corresponding landmark coordinates.

12. The computing device of claim 11, wherein the regression x pred ,y pred is the expected value of the pixel position according to the distribution (1) computed from the corresponding predicted heatmap such that: where j is the index of all said pixels in the corresponding heatmap, and w j A heatmap value representing the pixel.

13. The computing device of claim 1 or claim 5, wherein: The CNN is trained using a loss function defined by pixel-wise sigmoid cross entropy to learn the heatmap.

14. The computing device of claim 13, wherein: The loss function further includes L2 distance loss.

15. The computing device of claim 13, wherein: The loss function includes: in: is the predicted value of the heat map in the lth channel at the pixel position (i, j) of the nth sample; is the ground truth corresponding to the nth sample; is the weight at pixel position (i, j), which is calculated by Equation 4; is the ground truth coordinate of the lth marker of the nth sample; and are the expected coordinates of the same sign.

16. The computing device of claim 1 or claim 5, further configured via instructions to receive the image and perform landmark detection on the image.

17. The computing device of claim 1 or claim 5, further configured via instructions to modify the image at or around at least one of the markers using the corresponding position.

18. The computing device of claim 17, wherein: Modifying the image includes simulating a product applied to the image.

19. The computing device of claim 17, wherein: The image is a video image, and the computing device is configured via the instructions to modify and render the image in real time to simulate virtual reality.

20. The computing device of claim 19, further comprising a camera, and wherein: The video is a selfie video shot by the camera.

21. The computing device of claim 16, wherein: The indicia is a facial indicia, the image includes a face, and further comprising using the corresponding position of the indicia to update the image with at least one product simulation.

22. A method based on a convolutional neural network, comprising: The image is processed using a convolutional neural network (CNN) having a first stage and a second stage to determine a corresponding position of each of a plurality of landmarks, wherein: The first stage processes the image to generate an initial prediction of the corresponding location; and The second stage generates a crop of shared convolutional features for a region of interest using the intermediate features generated from the image by the first stage and the initial prediction to define a cropped feature, and further processes the cropped feature to produce a corresponding refined position offset to the initial prediction for the corresponding position.

23. The method according to claim 22, wherein: The first stage generates and uses the intermediate features to produce an initial heat map, from which the initial prediction is generated.

24. A method according to claim 22 or claim 23, wherein: The second stage generates a second heat map from the cropped features, one second heat map for each landmark, and uses the second heat map to generate the corresponding refined position offsets.

25. The method of claim 22 or claim 23, wherein: The CNN combines the initial prediction of the corresponding position and the corresponding refined position offsets to provide the corresponding position for each of a plurality of landmarks.

26. A method based on a convolutional neural network, comprising: The image is processed using a convolutional neural network (CNN) having a first stage and a second stage to determine a corresponding position of each of a plurality of landmarks, wherein: The first stage receives the image and determines: In a first part of said first stage, a volume of intermediate features from said image; and in a second part of said first stage, using said volume of intermediate features for a respective initial position of each of said landmarks within said image; and The second stage, receiving the volume of the intermediate feature in a first part of the second stage and receiving the corresponding initial position of each of the marks in a second part of the second stage, the second stage further: processing the volume to further refine the intermediate features; performing region-of-interest based pooling on the further refined intermediate features while maintaining feature alignment to generate a cropped feature for each of the plurality of landmarks; and Using the cropping feature and determining, for each landmark, a respective refined position offset for each respective initial position of each landmark; and Wherein the processing further operates to combine each respective initial position and the respective refined position offset to determine a final position coordinate in the image for each of the plurality of the markers.

27. The method according to claim 26, wherein: The second part of the first stage uses the volume of intermediate features to determine an initial heat map, and uses the initial heat map to predict each corresponding initial position of each of the landmarks.

28. The method of claim 22 or claim 26, wherein: At least some of the first stages include a series of inverted residual blocks, and wherein at least some of the second stages include a series of inverted residual blocks.

29. The method of claim 22 or claim 26, wherein: The second stage: Using RoIAlign for region of interest based pooling while maintaining feature alignment to produce the cropped features; and The cropping features are concatenated.

30. The method of claim 22 or claim 26, wherein: The second stage includes a prediction block for processing the cropped features, and the prediction block is executed in the following order: Channel-wise convolution with 3×3 kernel followed by BatchNorm and ReLU activation; and Group-wise channel-wise convolution with 1×1 kernel followed by BatchNorm; To output each of the corresponding refined position offsets.

31. The method of claim 22 or claim 26, wherein: The CNN model is trained using corresponding training images, which have ground truth heat maps of corresponding landmarks among the plurality of landmarks defined according to a Gaussian distribution, and have patterns corresponding to corresponding coordinate positions of the ground truth heat maps in the corresponding training images.

32. The method according to claim 31, wherein: The Gaussian distribution is defined as follows: where x,y represent the coordinates of any pixel in the training image, and (x i ,y i ) are the corresponding landmark coordinates.

33. The method of claim 32, wherein the regression x pred ,y pred is the expected value of the pixel position according to the distribution (1) computed from the corresponding predicted heatmap such that: where j is the index of all said pixels in the corresponding heatmap, and w j A heatmap value representing the pixel.

34. The method of claim 22 or claim 26, wherein: The CNN is trained using a loss function defined by pixel-wise sigmoid cross entropy to learn the heatmap.

35. The method of claim 34, wherein: The loss function further includes L2 distance loss.

36. The method of claim 34, wherein: The loss function includes: in: is the predicted value of the heat map in the lth channel at the pixel position (i, j) of the nth sample; is the ground truth corresponding to the nth sample; is the weight at pixel position (i, j), which is calculated by Equation 4; is the ground truth coordinate of the lth marker of the nth sample; and are the expected coordinates of the same sign.

37. The method of claim 22 or claim 26, further comprising modifying the image at or around at least one of the corresponding locations of a plurality of the indicia.

38. The method of claim 37, wherein: Modifying the image includes simulating a product applied to the image.

39. The method of claim 37, wherein: The image is a video image, and wherein the method presents the image modified in real time to simulate virtual reality.

40. The method of claim 37, further comprising performing the method via a personal computing device having a camera, and wherein, The image is a selfie taken by the camera.

41. The method of claim 40, wherein the personal computing device is a smartphone or a tablet computer.

42. The method of claim 37, wherein: The indicia is a facial indicia, the image includes a face, and further comprising using the corresponding position of the indicia to update the image with at least one product simulation.

43. A non-transitory storage device storing instructions that, when executed by a processing unit, configure a computing device to process an image to determine a respective position of each of a plurality of markers by: The image is processed using a convolutional neural network (CNN) having a first stage and a second stage, wherein: The first stage receives the image and determines: in a first part of said first stage, a volume of intermediate features from said image; as well as in a second part of said first stage, using said volume of intermediate features for a respective initial position of each of said landmarks within said image; and The second stage, receiving the volume of the intermediate feature in a first part of the second stage and receiving the corresponding initial position of each of the marks in a second part of the second stage, the second stage further: processing the volume to further refine the intermediate features; performing region-of-interest based pooling on the further refined intermediate features while maintaining feature alignment to generate a cropped feature for each of the plurality of landmarks; as well as using the cropping feature and determining, for each landmark, a corresponding refined position offset of the corresponding initial position for each landmark; and Wherein the processing further operates to combine each respective initial position and the respective refined position offset to determine a final position coordinate in the image for each of the plurality of the markers.

44. A computing device comprising a processor and a memory device, the computing device being configured to process an image via a convolutional neural network (CNN) to detect respective locations of a plurality of landmarks in the image, the CNN comprising: A two-stage localization architecture, wherein the first stage uses a first heat map to determine initial coordinates of the corresponding position, and the second stage uses a second heat map to determine a refined offset of the initial coordinates, and the second stage further uses a region of interest to pool each individual landmark for reducing overlapping calculations to avoid irrelevant regions and guide the generation of relevant shared features; as well as Among them, the CNN is trained using auxiliary coordinate regression loss to minimize the size of the corresponding heat map and the usage of computational resources.

Images