Method and apparatus for object detection and stabilized rendering

Abstract

Systems, methods, and apparatuses for object detection and for stabilizing the rendering of effects, such as makeup effects, applied to facial images are provided. In one embodiment, a face tracker comprising one or more deep neural networks (DNNs) trained to locate facial features is used to locate faces in facial input images, and training images including the located faces are generated, the training images including either occluded training images in which an occluding object is rendered on the face, or unoccluded training images in which the face appears without a facial mask, the training images generated for occluded face DNN training. In one embodiment, rendering of the effect on a current frame of a video stream is responsive to stabilizing the position of the detected features in the stream.

Description

[Technical Field]

[0001] 《Cross reference》 This application claims the domestic benefit or other priority of an earlier U.S. patent application, U.S. Patent Application No. 63 / 435,837, filed December 29, 2022, entitled "Methods, Apparatus for Object Detection and Stabilized Rendering," the entire contents of which are incorporated herein by reference. This application claims priority to French Patent Application No. FR 2303057, filed March 30, 2023, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to image processing, for example using deep neural networks, and more particularly to methods and apparatus for object detection and for stabilized rendering. [Background technology]

[0003] Deep learning techniques are useful for processing images, including a series of video frames, to locate one or more objects within the images. In one example, the objects are facial features, including parts of a user's face. Image processing techniques are also useful for rendering effects associated with the objects, such as augmented reality for the user. One example of such augmented reality is providing a virtual try-on (VTO) that simulates the application of a product to an object. Product simulation in the beauty industry includes simulating makeup, hair, and nail effects. Another example might include locating the iris and simulating color changes thereto, such as with colored contact lenses. These objects and simulations are as obvious as any other.

[0004] In a frame of a video, the position of an object in a current frame may differ from the position of the object in an earlier frame, for example, as a result of motion. Localization of a particular object in two or more frames using a deep neural network may produce undesirable results when effects are applied due to inaccurate localization between frames.

[0005] To facilitate providing augmented reality, including VTO, experiences, improved techniques for determining object information from images are desirable. Summary of the Invention

[0006] Systems, methods, and apparatuses for object detection and for stabilizing the rendering of effects, such as makeup effects, applied to facial images are provided. In one embodiment, a face tracker comprising one or more deep neural networks (DNNs) trained to locate facial features is used to locate faces in facial input images, and training images including the located faces are generated, the training images including either occluded training images in which an occluding object is rendered on the face, or unoccluded training images in which the face appears without a face mask, the training images being generated for occluded face DNN training. In one embodiment, rendering of the effect on a current frame of a video stream is responsive to stabilizing the position of the detected features in the stream.

[0007] In embodiments, a computer-implemented method is provided that performs the steps of: one or more processors locating faces in face input images using a face tracker comprising one or more deep neural networks (DNNs) trained to locate facial features; and generating training images including the located faces, the training images including either occluded training images in which occluding objects are rendered on the faces or unoccluded training images in which the faces appear without a facial mask, the training images generated for occluded face DNN training.

[0008] In one embodiment, a system is provided that includes a face tracker engine comprising a deep neural network (DNN) for locating faces in face input images, and a training image generator for generating training images including the localized faces, wherein the training images include either occluded training images in which occluding objects are rendered on the faces or unoccluded training images in which the faces appear without occluding objects, and the training images are generated for occluded face DNN training.

[0009] In one embodiment, a computer-implemented method is provided in which one or more processors perform the steps of: locating a facial feature in a current frame of a set of frames of a video stream using a face tracking engine having one or more deep neural networks (DNNs) configured to process a current frame and predict tracker locations of the facial feature; generating a current stabilized location of the facial feature in the current frame, the current stabilized location being responsive to the tracker locations and prior stabilized locations of the facial feature in prior frames of the video stream; and rendering an effect in the current frame associated with the facial feature in response to the current stabilized location, the effect simulating a product to be tried on as a component of a virtual try-on experience.

[0010] In one embodiment, a system is provided that includes a face tracker engine having computational circuitry configured to locate a facial feature in a current frame of a set of frames of a video stream using one or more deep neural networks (DNNs) configured to process the current frame and predict a tracker position for the facial feature; a stabilizing component having computational circuitry configured to generate a current stabilized position of the facial feature in the current frame, the generation being responsive to the tracker position and a previous stabilized position of the facial feature in a previous frame of the video stream; and a rendering component having computational circuitry configured to render an effect in the current frame related to the facial feature in response to the current stabilized position, the effect simulating trying on a product as part of a virtual try-on experience.

[0011] These and other embodiments will be apparent to those skilled in the art. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 2 is a block diagram illustrating a training pipeline including components for generating synthetic training images, according to one embodiment. [Figure 2] 1A, 1B, 1C, 1D, and 1E are diagrams of representative images of faces, including a face image, a cropped face image, two synthetic training images, and a face image with face points, respectively, according to an embodiment. [Figure 3] 1 is a flowchart of the operation of a computer-implemented method, according to one embodiment. [Figure 4] FIG. 1 illustrates a computing environment, such as for performing a virtual try-on, according to one embodiment. [Figure 5] 1 is a flowchart of the operation of a computer-implemented method, according to one embodiment. [Figure 6]FIG. 1 illustrates a computing environment, such as for performing a virtual try-on, according to one embodiment. [Figure 7] 1 is a flowchart of the operation of a computer-implemented method, according to one embodiment.

[0013]

[0014]

[0015]

[0016]

[0017]

[0018] DETAILED DESCRIPTION OF THE INVENTION

[0019] According to embodiments herein, one or more methods, devices, and techniques for object localization and object stabilization are described, which may be used independently or together. In connection with object localization, methods, devices, and techniques are described for generating synthetic data and using such synthetic data to train network models.

[0020] In the embodiments described herein, classes of objects for classification or classification and location, etc., are various facial features. In one embodiment, such features include facial contours, nose, inner mouth, outer mouth, left eye, right eye, left brown, and right eyebrow. In an embodiment, an additional class of objects includes face masks worn to reduce the penetration of airborne particles, such as aerosols. The face mask can fully or partially occlude one or more of the facial features in the facial image. Examples of occluded facial features or portions thereof include the nose, inner mouth, outer mouth, and facial contours (e.g., jawline, chin, or both).

[0021] In embodiments herein, one or more deep neural networks, e.g., as components of a face tracker engine, classify and locate an input image to detect objects therein and determine the respective locations of at least some of the detected objects. In one embodiment, a first deep neural network determines whether a face is present and provides a bounding box, e.g., crops the input image to fit the face. In one embodiment, a second deep neural network classifies and locates multiple facial features (e.g., each detected object) in a facial image comprising the cropped input image. In one embodiment of such a network, the localization includes identifying face points that define a general outline of at least some of the detected facial features. In one embodiment, the deep neural networks further classify whether a facial mask is present on the cropped face, e.g., by working in parallel and processing the same cropped facial image. In one embodiment, the localization of the facial mask itself is not performed. In one embodiment, the facial mask is located.

[0022] In one embodiment, for example, a further deep neural network running in parallel processes the same cropped face image, but instead of classification, performs segmentation and provides a segmentation mask indicating where facial masks are or are not present in the cropped face.

[0023] The classification, localization, and / or segmentation of face masks is useful, for example, in a virtual makeup try-on application. Effects to be tried on can be rendered for classification and / or localization, e.g., not rendered if the rendering face is occluded in whole or in part. The results of face mask classification can be used to not render lipstick. The rendering can be related to segmentation, such as a segmentation mask for an occluding object. Thus, in one embodiment, the face tracking engine can output any one or more of classification results, localization results, and segmentation results for at least some of the objects. The localization results can include face points that provide a contour for the processed image. Figure 2E, further described herein below, illustrates face points.

[0024] In one embodiment, as described above, a rendering operation or component, as may be applicable to methods and apparatus, renders an output image with an effect applied to a face in response to at least some of the detected objects in the input image. In an embodiment, the applied effect is a makeup effect, for example, but not limited to, an eye, eyebrow, or lip makeup product effect. In some embodiments, the rendering is a component of a VTO experience for the user. In some examples, the effect is rendered to one or more of the detected objects. One example is rendering an eyebrow effect to each eyebrow, and another example is rendering a lip effect to each lip. Typically, makeup looks are rendered symmetrically, but this need not be the case. In some examples, the effect is rendered to an area adjacent to or located relative to one or more detected objects. One example is rendering an eye makeup effect to each eyelid, with each eye area typically being located at least partially between a detected eye and a detected pair of eyebrows. Another example is rendering blush or other cheek makeup, for example, positioned on the cheek relative to the eyes and facial contours.

[0025] In one embodiment, such as to provide a VTO experience from a "live" stream of video (e.g., selfie video), each frame of the video (e.g., each image) is processed to detect and locate objects and render at least one effect responsive to the location of the detected objects according to the product or service being virtually tried on. Thus, the input may include a selfie image contained in a selfie video frame.

[0026] In some embodiments, the respective positions of at least some of the detected objects are derived from contours (e.g., face points) generated by a face tracking engine (e.g., its one or more deep neural networks). In some embodiments, a stabilizing operation or component that may be applicable to the methods and apparatus, such as when the input image is the current frame of a sequence of frames of a video, stabilizes the respective positions of at least some of the detected objects prior to rendering the effect. Object stabilization is described further herein below.

[0027] Object Detection An example of a deep neural network for object detection is described in Sandler et al., "MobileNetV2: Inverted Residuals and Linear Bottlenecks" (2018), published January 13, 2018, at the 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition in Computer Science, the contents of which are incorporated herein by reference in their entirety. A copy of this publication is available at arxiv.org / abs / 1801.04381 ​​at the time of filing this application. A deep neural network constructed in accordance with this publication is referred to herein as a MobileNetV2 deep neural network. MobileNetV2 deep neural networks themselves build upon previous single-shot detectors; an example of a deep neural network for detecting objects in images is described in "Liu, Wei, et al. "Ssd: Single shot multibox detector." In European conference on computer vision, pp. 21-37. Springer, Cham, 2016." An earlier version was published on arxiv.org on December 8, 2015; at the time of filing, it is available at arxiv.org / abs / 1512.02325.

[0028] According to one embodiment herein, one or more deep neural networks are configured, through training, to detect facial features (in images) including faces wearing face masks or faces that are otherwise occluded by occluding objects, where the deep neural networks are adapted from the MobileNetV2 deep neural network. In one embodiment, the one or more deep neural networks are configured to at least one of classify, localize, or segment occluding objects (e.g., face masks or other objects as described herein).

[0029] 1 illustrates a training pipeline 100 according to an embodiment for defining a new deep neural network, where the pipeline 100 includes components for generating training images for training the new deep neural network. In one embodiment, images containing faces (e.g., facial images) 102, such as from a data store 103, are provided to a face tracker 104A that includes one or more deep neural networks 106A (each of which may have a MobileNetV2 deep neural network backbone). The data store 103 may comprise a database or other storage configuration that stores, for example, multiple different facial images used to train and / or test the deep neural networks. The data store 103 may include publicly available open data stores, closed, proprietary data stores, or both open and closed data stores.

[0030] The face tracker 104A is adapted to track (i.e., locate) a class of face-related objects, including the face object itself. The output from such a face tracker 104 includes a bounding box, mask, or other structure for deriving a cropped face image 108. For example, in a cropped face image, any background in the face image 102 is minimized. FIG. 2A shows a representative face image 102 including a face 202 and a background 204. The background may include other portions of the subject as well as non-subject portions. A bounding box 206, shown as a dotted box, indicates coordinates for defining a cropped image (e.g., 108) including the face 202 and minimized background content (a portion of the background 204). FIG. 2B shows a representative cropped face image 108.

[0031] Training image generator 110 receives cropped facial images, such as cropped facial image 108, and generates training images, such as image 114, according to its configuration. Training image generator 110, in one embodiment, generates multiple training images for each cropped facial image it receives. While not shown, the training images, in one embodiment, are stored in data store 103, such as for later use. In one embodiment, the training images are provided for training a deep neural network, as will be further described. While described as training images, such may, in one embodiment, include test images.

[0032] In one embodiment, training image generator 110 generates training images, such as by rendering an effect on at least some of the cropped facial images it receives. In one embodiment, the effect is the application of a facial mask, such as by applying an isolated facial mask image 112 (e.g., from data store 103) through a rendering operation. Data store 103 may store multiple isolated facial mask images. An isolated facial mask image, such as image 112, may comprise a portable network graphic (e.g., .png) or other usable image format image of a facial mask with a transparent background. That is, when a facial mask is applied over a cropped facial image, only the facial mask occludes the portion of the cropped image over which it is rendered, and the transparent background of the facial mask image allows other portions of the cropped facial image to remain visible.

[0033] In one embodiment, the training image generator 110 operates to use the predicted / labeled points coordinates to add a facial mask image to the cropped face image to generate synthetic data, resize the facial mask image before applying it to the cropped face, apply augmentations such as rotation and translation to the mask in at least some of the generated training images to ensure diversity in the training data, and generate some training images without a mask so that a target percentage of images include a mask, e.g., 55% of such training images include a mask.

[0034] In one embodiment, the face tracker 104A outputs facial feature localization data, e.g., face points relative to the face contour. An example is shown in FIG. 2E, which is described further herein. The face points and / or face bounding box may be useful for resizing a face mask image for the detected face.

[0035] In one embodiment, the training image generator defines multiple training images from a single cropped face image, where the multiple training images include any one or more of: a cropped face image with no facial mask added; multiple cropped face images each with a different facial mask added; and / or multiple cropped face images each with a facial mask added in a different manner (e.g., after a facial mask rotation or transition).

[0036] Figure 2C shows training image 114A with mask image 112A applied to the cropped image of Figure 2B. Figure 2D shows training image 114B with mask image 112A after rotation (e.g., flipped horizontally) and transition. Figure 2B can also define training images where the image does not include a facial mask.

[0037] FIG. 2E shows an example of output 200 from face tracker 104A, including a cropped face image 202 and groups of face points 204, such as face contour face points 204A, eyebrow face points 204B, and nose face points 204C. For illustrative purposes, the figure depicts an annotated cropped face image 202 with groups of face points. The output of the face tracker need not include an annotated image; the output can be separate data. The face points within each group are numbered (e.g., 0, 1, 2, ...) to help define the contours of detected objects. The face tracker assigns each point such that it is consistently positioned relative to the contour of the object it represents. For example, a particular point may always be located at the right corner of the mouth. In one example, face points are X,Y pixel coordinates for the cropped face image 202 and are associated with each detected object from (e.g., one of) networks 106B of tracker 104A.

[0038] Components above the dotted line 116, such as the face tracker 104A and the training image generator 110, are useful for defining training images, including facial mask training images and non-facial mask training images, from faces as they are located.

[0039] Training images, such as image 114, are applied to train a deep neural network, such as a component of face tracker 104B, which includes one or more deep neural networks 106B (each of which, like 106A, may have a MobileNetV2 backbone). Components 104B and 106B are similar to components 104A and 106A, but include applicable structures for classifying facial mask objects in facial images. Components 104B and 106B are also configured with applicable training structures, for example, as described in the above-mentioned publication, "MobileNetV2: Inverted Residuals and Linear Bottlenecks." The deep neural network trained for facial mask classification may be similar in structure to the deep neural network component of face tracker 104A that classifies (detects) other objects. In one embodiment, the deep neural network for facial mask detection does not need to localize the facial mask. In one embodiment, the deep neural network for facial mask detection localizes and / or segments the facial mask, and therefore is configured with an appropriate structure for training and generating output, as applicable. That is, a deep neural network can be configured to be trained to segment and generate a mask.

[0040] Following training, the resulting face tracker 104B with the trained deep neural network(s) 106B is tested, such as by using real images of faces with facial masks. Following the testing and training cycle, the resulting face tracker 104B with the deep neural network(s) 106B, optionally configured in real-time or live application use (not in a training configuration), is useful for classifying facial images to identify whether a mask is present and / or locating / segmenting facial masks, which may also be segmented. Preferably, the resulting face tracker 104B, along with its deep neural network(s) 106B, also classifies and locates other facial features. In one embodiment, the resulting face tracker 104B, along with its deep neural network(s) 106B, provides an engine for locating facial features, such as for use in applications providing a VTO experience, as further described herein below.

[0041] Although the embodiment of FIG. 1 is described with reference to one or more networks 106A, each having a MobileNetV2 deep neural network backbone, other neural network backbones defined for image localization tasks may form the backbone of the face tracker and may be similarly adapted, such as through training with synthetic images to detect the presence of a facial mask (e.g., classification, localization, and / or segmentation of occluding objects).

[0042] Although described with respect to detecting a face mask that occludes at least a portion of a face, the methods, apparatus, and techniques herein can be adapted to define synthetic data and to detect other types of face occlusions in which at least a portion of a face is occluded by another object, such as for training an occluded face detecting network. For example, synthetic data generation and training can be performed for occlusions by face masks, occlusions by sunglasses / dark glasses or other occluding eye glasses that occlude part of the face, occlusions by hair, occlusions by scarves, occlusions by hats, occlusions by hands / finger, occlusions by smartphones (e.g., when a selfie mirror has part of the smartphone covering the (reflected) face in the image), etc.

[0043] For greater diversity in the training data, it may be preferable to include a variety of examples of objects applied to faces, e.g., examples of different types of eye-obscuring glasses, hats, hands, (number of) fingers, smartphones, etc. In one embodiment, an occluded face detection network is configured to detect multiple classes of occluded objects and is trained with training images, the training images including occluded objects of each class. Such occluded face detection may also be configured and trained to localize such objects, including segmentation.

[0044] FIG. 3 is a flowchart of operations 300 of a computer-implemented method or the like. The method may include, for example, performing the steps illustrated in FIG. 3 by one or more processors. Embodiment 1: The method includes step 302 illustrating locating a face in a facial input image using a face tracker comprising at least one deep neural network (DNN) trained to locate facial features; and step 304 illustrating generating training images including the localized face, the training images comprising either occluded training images in which occluding objects are rendered on the face or non-occluded training images in which the face appears without occluding objects, the training images being generated for occluded face DNN training. Embodiment 2: Embodiment 1 may include operations such as step 306 illustrating training an occluded face detecting DNN using the training images. The training may include training for classification, localization, and / or segmentation of occluded objects.

[0045] Embodiment 3: The occlusion object in embodiment 1 or embodiment 2 covers at least a portion of the face and comprises any one of a face mask, occlusion glasses, a hat, a scarf, a hand or finger, hair, a smartphone, or a portion of any of these.

[0046] Embodiment 4: In any of embodiments 1 to 3, the occluded face detection DNN comprises a DNN pre-trained to classify and localize facial features such that when trained, the occluded face detection DNN detects the presence of at least one face occluding object and classifies and localizes the facial features.

[0047] Embodiment 5: In any of embodiments 1 to 4, steps 302 and 304 are repeated with multiple face input images of different faces (not shown) to generate multiple training images for occluded face DNN training.

[0048]

[0023] Embodiment 6: In any of embodiments 1 to 5, generating training images is randomly generating occluded training images instead of unoccluded training images according to a probability selected to maximize occluded face DNN training.

[0024] Embodiment 7: In embodiment 6, the probability of generating occluded training images instead of unoccluded training images is 55%.

[0049] Embodiment 8: In any of embodiments 1 to 7, the method includes cropping a face from the facial input image in response to the localization (e.g., between steps 203 and 304, not shown) and generating training images using the cropped face.

[0050]

[0023] Embodiment 9: In any of embodiments 1 to 8, the occluding objects for rendering include isolated occluding object images with a transparent background.

[0024] Embodiment 10: In embodiment 9, the method includes, before rendering the occluding objects as localized to the face, performing one or more of: resizing the occluding objects as localized to the face; and augmenting the occluding objects to maximize face mask DNN training.

[0051] Corresponding system embodiments are disclosed for each of embodiments 1 through 10, and it will be understood that, for example, the system comprises a respective component having computational circuitry configured to perform the operations of the computer-implemented method embodiment.

[0052] 1 illustrates, by way of example, a system, such as one or more processors and / or computing circuits, that provides components capable of performing any of method embodiments 1 to 10. For example, a system is provided that includes a face tracker engine that includes a deep neural network (DNN) for locating faces in face input images, and a training image generator for generating training images that include the localized faces, wherein the training images include either occluded face training images in which occluding objects are rendered on the faces, or unoccluded training images in which faces appear without occluding objects, and wherein the training images are generated for occluded face DNN training.

[0053] VTO ​​Applications FIG. 4 illustrates a computing environment 400 according to one embodiment, such as for implementing one or more method aspects. The computing environment 400 includes a user computing device 402, such as a smartphone, a communication network 404, a server 406, and a server 408. The communication network 404 may be a wired and / or wireless network, including a public or private network, such as the Internet. The server 406 may be a server computing device, such as for providing a website. The server 408 may be a server computing device, such as for providing e-commerce services. While shown separately, the servers 406 and 408 may be a single server device. The computing environment is simplified. For example, not shown are payment transaction gateways and other components, such as those for completing e-commerce transactions.

[0054] The computing device 402 includes a storage device 410 (e.g., a non-transitory device such as a memory and / or a solid-state drive) for storing instructions that, when executed by a processor (not shown), cause the computing device 402 to perform operations such as a computer-implemented method. The storage device 410 stores a virtual try-on application 412, which is a component that provides software modules such as a user interface 414, a face tracker 104B having one or more deep neural networks 106B trained according to FIG. 1, a VTO rendering pipeline component 416, a product recommendation component 418 having product data 420, and a purchasing component 422 having a shopping cart 424 (e.g., purchasing data).

[0055] In one embodiment, the VTO application is a web-based application obtained from server 406. Although not shown, user device 402 may store a web browser for executing web-based VTO application 412. In one embodiment, the VTO application is a native application that conforms to an operating system (not shown), and software development requirements may be imposed, for example, by the hardware manufacturer of user device 402. The native application may be configured for web-based or similar communication with servers 406 and 408, as is known.

[0056] 4 illustrates, for example, various input and output data or information associated with use of VTO application 412. Such may include user input images 426 to be processed for the VTO experience, output images 428 in which product effects are simulated to provide the VTO experience, VTO product selections 430 including user inputs selecting one or more product effects to be simulated, VTO product options 432 comprising options for products to virtually try on for selection by the user of device 402, for example, and purchase transaction information 434 including purchase information provided to and / or received from the user to purchase a product.

[0057] In one embodiment, via one or more of the user interfaces 414, VTO product options 432 are presented for selection for virtual try-on by simulating the effect on the input image 426. In one embodiment, the VTO product options 432 are obtained from or associated with product data 420. In one embodiment, the product data may be obtained from the server 406 and provided by the product recommendation component 418. Although not shown, user or other input may be received for use in determining the product recommendation. The user may be prompted, such as via one of the interfaces 414, to provide input for determining the product recommendation. In one embodiment, the product recommendation component 418 communicates with the server 406. The server 406, in one embodiment, determines the recommendation based on the input received via component 418 and provides the product data accordingly. The user interface 414 may present a selection of VTO products, with the same display of VTO products being updated in response to data received, for example, as the user browses or otherwise interacts with the user interface.

[0058] In one embodiment, one or more user interfaces provide command and control for acquiring an input image 426 and a VTO product selection input 430, such as identifying one or more recommended VTO products to try on. In one embodiment, the input image 426 is a facial image of a user, which can be a still image or a frame from a video. In one embodiment, the input image 426 can be received from a camera (not shown) of the device 402 or from a stored image (not shown). The input image 426 is provided to the face tracker 104B for processing, such as detecting objects in the facial image using one or more trained deep neural networks 106B. In one example, the network classifies, locates, or segments a facial mask (or other occluding object) in the image. For example, classifying the presence of a facial mask is useful for outputting a request (e.g., a command to the user, such as via the user interface 414) to lower or remove the facial mask. This is applicable to any occluding object for which the face tracking engine is trained.

[0059] In one embodiment, output (not shown) from face tracker 104B, such as classification, localization, or segmentation results of one or more detected objects, is provided to VTO rendering pipeline component 416. In one example, the output may comprise bounding boxes and face points for the detected objects as shown in FIG. 2E. Input images 426 are also provided (e.g., made available) to component 416. VTO product selection 430 is also provided to component 416 to determine which effects should be rendered. In an embodiment related to makeup simulation, one or more effects may be shown, such as for any one or more of product categories including lips, eye shadow, eye liner, blush, etc.

[0060] The VTO rendering pipeline component 416, in one embodiment, determines whether to render one or more product effects on the input image 426 to simulate a try-on. For example, in response to a facial mask classification output, the VTO rendering pipeline component 416 may determine, for example, not to render a product effect because a mask is detected. If a facial mask is detected, for example, the VTO rendering pipeline component 416 may trigger the user interface 414 to prompt the user to remove the facial mask. New images may be received and processed by the face tracker 104B. In one embodiment, images are continuously received as part of a live stream (e.g., a selfie video).

[0061] If the VTO rendering pipeline component 416 determines to render one or more product effects, in one embodiment, the VTO rendering pipeline component 416 renders the effects to the input image 426, such as by drawing (rendering) the effects in layers, to generate an output image 428, one layer per product effect. Some of the operations of the VTO rendering pipeline component 416 (e.g., drawing the layers) may be performed by a graphics processing unit in one embodiment. The rendering is responsive to the positions of the detected objects according to product data 420 selected by VTO product selection 430. For example, a VTO product selection of lipstick, lip gloss, or other lip-related product invokes the application of the effect to one or more detected mouth- or lip-related objects at their respective positions. Similarly, a product selection related to eyebrows invokes the application of the selected product effect to the detected eyebrow objects. Typically, for a symmetrical appearance, the same eyebrow effect is applied to each eyebrow, the same lip effect is applied to each lip, or the same eye effect is applied to each eye area, although this need not be the case. In one example, rendering is applied to areas related to detected objects, such as adjacent to one or more such detected objects. Some VTO product selections include multiple product selections, such as coordinated products for eyebrows and eyes, or other combinations of detected objects. The VTO rendering pipeline component 416 can render each effect, for example, one at a time, until all effects have been applied. The order of application can be defined by rules or in the selection of products, for example, lipstick before top gloss.

[0062] In embodiments where occluding objects are detected and positions are determined, for example, as represented in a segmentation mask, rendering may be responsive to such segmentation mask. Rendering of effects may be applied to unoccluded portions of the face. The segmentation mask may indicate pixels of the face that can (e.g., can) receive an effect, such as a makeup effect, and pixels that cannot receive the effect.

[0063] The user interface 414 provides output images 428. In one embodiment, the output images 428 are presented as part of a live stream of successive output images (each example 428), such as when selfie footage is augmented to present an augmented reality experience. In one embodiment, the output images 428 are presented along with the input images 426, such as displayed side-by-side for comparison. In one embodiment, the output images 428 can be saved (not shown), such as in the storage device 410, and / or shared (not shown) with another computing device.

[0064] In one embodiment, the input image (not shown) comprises an input image of a video conferencing session and the output image comprises a video to be shared with another participant(s) of the video conferencing session. In one embodiment, the VTO application is a component or plug-in of a video conferencing application (not shown) that allows a user of device 402 to wear makeup during a video conference with one or more other conference participants.

[0065] In one embodiment, as described further herein below, the VTO rendering pipeline component 416 is configured to apply object stabilization, for example, to stabilize the position of each detected object between successive frames of the video.

[0066] Figure 5 is a flowchart of operations 500 of a computer-implemented method or the like. The method may include, for example, performing the steps illustrated in Figure 5 by one or more processors. Embodiment 11: The method includes step 502 illustrating processing an input image using a face tracker engine having at least one deep neural network to i) determine facial features from the input image for rendering an effect and ii) the presence of an occluding object occluding at least a portion of the face, and step 504 illustrating avoiding rendering at least a portion of the effect on at least one of the detected facial features in response to the detected presence of the occluding object.

[0067] Embodiment 12: In embodiment 11, the method includes at least one of: i) providing a recommendation interface that recommends one or more makeup products for virtual try-on, where each product is associated with one or more effects that are rendered in association with one or more facial features; or ii) providing a purchase transaction interface to facilitate the purchase of makeup products.

[0068]

[0033] Embodiment 13: In embodiment 11 or embodiment 12, processing the input image by the face tracking engine provides a segmentation of the occluding object, and the step of preventing rendering of at least a portion of the effect is responsive to the segmentation such that at least a portion of the effect occluded by the occluding object is not rendered.

[0034] Embodiment 14: In embodiment 13, the method includes providing instructions via a user interface to remove the occluding object to facilitate full rendering of the effect.

[0069] Embodiment 15: In any of embodiments 11 to 14, the method includes providing instructions via a user interface to remove occluding objects to facilitate rendering (e.g., not rendering any effects after the step of avoiding rendering).

[0070] Embodiment 16: In any of embodiments 11 to 15, the method includes receiving and processing additional images using a face tracking engine for facial feature detection and occluding object detection, and rendering the effect after the presence of the occluding object is no longer detected.

[0071] Embodiment 17: In any of embodiments 11 to 16, the effect is a makeup effect, and the method is performed in the context of a computer operating to provide a virtual try-on experience.

[0072] Corresponding system embodiments are disclosed for each of embodiments 11 through 17, and it will be understood that, for example, the system comprises a respective component having computational circuitry configured to perform the operations of the computer-implemented method embodiments.

[0073] Object Stabilization Object localization using deep neural network processing can result in jitter or other instability between images. That is, the DNN's predicted location of an object in a first image may be perceptibly different from the DNN's predicted location of the same object in a second image. This is particularly noticeable when the first and second images are two consecutive frames of video and an effect is applied in response to the predicted location. The effect moves with the jitter. Tracking an object between consecutive frames and rendering the effect on the input frames may result in the effect jittering or moving in a way that does not appear to match the underlying input frames when viewed together.

[0074] In one embodiment, stabilization is applied to the localization of detected objects generated by the DNN processing the current frame. In one embodiment, each frame of the video (e.g., as a sequence of images) is processed to detect and locate objects and render effects according to the product or service being virtually tried on, such as to provide a VTO experience from a "live" stream of video (e.g., a selfie video). The effect is applied to one or more positions or regions of at least one of the detected objects.

[0075] In one embodiment, prior to rendering, the positions of detected objects (e.g., at least one object associated with an effect) are stabilized for smooth tracking. These stabilized positions are used to render the effect. The effect can be applied to the stabilized positions of the detected objects (e.g., stabilized eyebrow positions or lip positions, etc.) or to regions adjacent to one or more detected objects, such as eyelid regions adjacent to the stabilized positions of detected eyes. In some images, such as when a face mask is worn, not all objects are positioned.

[0076] In one embodiment, stabilization is performed using an optical flow technique (e.g., "optical flow tracking") that predicts the position of an object in the current frame. In one embodiment, the positions of the face points as output from the face tracker, as described above, may be stabilized.

[0077] The stabilization process is resource intensive. According to one embodiment, detected objects are grouped by their importance to the task, i.e., to the VTO experience. In one embodiment, the positions of detected objects related to the mouth and eyes are stabilized using a blending of tracker predictions from the current frame with optical flow predictions for the current frame that respond to stabilized positions in the previous frame, and the positions of detected objects related to the eyebrows, nose, and facial contours are stabilized using an exponential moving average filter that responds to the net velocity of the object's face points over the previous n frames.

[0078] The following are embodiments of stabilization operations Stab-1 to Stab-5b, which are:

[0079] Stab-1: Obtain a prediction of face points as trackerPt from the face tracker engine (e.g., 104A or 104B). The tracker prediction trackerPt relates to the current frame at time t of the video. The previous frame is at the previous time t-1. The tracker prediction includes the positions of various detected objects, e.g., a set of face points for each detected object, as in FIG. 2E. The stabilization attempts to generate a stabilized face point Pt for each object detected in the current frame. The stabilized face points (stabilized positions) for each object detected in the previous frame generated by the stabilization operation are denoted as Pt-1.

[0080] Stab-2: In one embodiment, face points received from a face tracker, such as those representing object contours as shown in FIG. 2E, are grouped for each object as the following groups of points (e.g., a subset of trackerPt for each object): left eye, right eye, left eyebrow, right eyebrow, nose, outer mouth, inner mouth, and face contour. In one embodiment, objects are assigned an importance rating, which in one embodiment is one of two ratings (e.g., higher / lower importance). In one embodiment, stabilization for an object is performed using one set of actions for objects of higher importance and another set of actions for objects of lower importance depending on the importance rating. In one embodiment, stabilization actions performed for objects of higher importance are more accurate but require more resources and / or processing than actions performed for objects of lower importance. Thus, objects are assigned an importance rating that balances accuracy with device performance criteria (e.g., processing time / memory usage, etc.), and in one embodiment, the left eye, right eye, outer mouth, and inner mouth objects are assigned higher importance ratings, and the left eyebrow, right eyebrow, nose, and facial contour objects are assigned lower importance ratings. In one embodiment, for example, eyes and lips are prioritized because many effects relate to eyes and lips.

[0081] Stab-3: For higher importance objects: optFlowPt is obtained by applying an optical flow function to Pt-1 for only the higher importance objects. The optFlow function calculates optical flow (e.g., image velocity) for a sparse feature set using the iterative Lucas-Kanade method with pyramids (a pyramid from the previous frame and a pyramid from the current frame) (see Bouguet, J.-Y. (1999). Pyramidal implementation of the Lucas Kanade feature tracker, available at semanticscholar.org at the time of filing). It will be understood that optFlowPt for a particular object represents the predicted face points of the object for the current frame in response to the stabilized face points (positions) Pt-1 generated for the object in the previous frame. In an embodiment implemented using the optical flow function from OpenCV, for example, points for all higher importance objects are provided together rather than processing each object separately.

[0082] Stab-4: For objects of higher importance: Blend using a blending factor, correcting if the distance between the tracker position and the optFlow position is above a threshold: At regular intervals, set blendingFactor=0.4 (starting value). The regular interval can be based on time or number of frames (e.g., a rough time conversion), e.g., every 1.3 seconds or every 40 frames. The number of frames to approximate time depends on the processing speed, so time may be preferable for consistency. At every frame, run blendingFactor*=0.080 (decay value). The blending factor is used to control the blending between trackerPt and optFlowPt, and is reset to a starting value (e.g., 0.4) to prevent optFlowPt from moving too far away from trackerPt. Over time, optFlow points and face tracker points may move apart. Blending can produce jarring results if the blending causes sudden changes.

[0083] For each blend group of points (left eye, right eye, inner mouth, outer mouth):

[0084] Stab-4.a: Blending based on blendingFactor:

[0085]

number

[0086] Stab-4.b: Distance-based blending - Compare pixel distances between corresponding face points in trackerPt and optFlowPt. For a mouth object, for example, compare the mouth corner face points from trackerPt with the same face points from optFlowPt. If trackerPt and optFlowPt are too far apart, blend towards trackerPt:

[0087]

number

[0088]

number

[0089] where distanceBlendingNorm is the normalization factor for point distances ( ) 6 is used to make small values ​​smaller. In one embodiment, distanceBlendingNorm is, for example, 5 pixels.

[0090] Stab-5: For objects of lower importance: Apply an exponential moving average filter to the left eyebrow, right eyebrow, nose and face contour points.

[0091] Stab-5.a: For each group, a net velocity v is calculated and averaged over the previous n frames. In one embodiment, the velocity calculation uses tracker points for both trackerPt and trackerPt-1 of the previous and current frames, and does not use stabilized points Pt-1 for the previous frame. These stabilized points are ultimately used when applying blending determined using the results of the velocity calculation. The reason for this is that using tracker points allows the motion to detect velocity changes more quickly, as opposed to using stabilized points.

[0092]

number

[0093]

number

[0094] Stab-5.b: The coordinates of the updated face points are calculated using the blending coefficient α:

[0095]

number

[0096]

number

[0097] where transitionSpeedFactor is a constant that controls the impact of v, with a default value of 1.5.

[0098] Therefore, in association with the blending operations Stab-4.a and Stab-4.b, a form of linear interpolation is performed for each of the eye and mouth groups (i.e., for each facial feature from the more significant group of facial features), respectively. Specifically, two positions of each face point in the current image (the tracker position and the optFlow position (second position)) are blended according to a blending coefficient. The blending coefficient weights the respective contributions of the tracker position and the optFlow position to generate a first blended result. The second blending operation generates a current stabilized location and is responsive to the distance between the two positions (e.g., the distance between the pixel coordinate of each face point at the tracker position and the corresponding each face point at the second tracker position) and a distance normalization factor for moving the first blended result toward the tracker position. Thus, the blending coefficients blend the tracker position and the optiflow position based on prior stabilized locations, initially favoring the optiflow position, and applying a correction if the two positions are far enough apart to generate a current stabilized position from the first blending result, which has been moved towards the tracker position.

[0099] In one embodiment, the blending coefficients for the first blending result vary (decay) from the maximum amount over a period of time (e.g., a series of frames or a predetermined time), after which the blending coefficients are reset to the maximum amount. As the blending coefficients decay, the optiflow position becomes increasingly favored in the blending. The reset helps readjust the blend in case of misalignment. For distance-based blending thresholds, in one embodiment, the distance normalization factor is 5 pixels.

[0100] Therefore, an exponential moving average filter is applied in conjunction with operations Stab-5.a and Stab-5.b, which are performed for each of the less important groups (bows, nose, and facial contours). In an exponential moving average filter, the operation uses only points from the previous and current frames. Due to the repeated application of stabilization between frames, points from previous frames implicitly include information from older frames. In an alternative approach (not shown), a window of previous position values ​​is determined and averaged. For example, points from the current frame and the previous N frames (e.g., N=3) can be used for a total for N+1 frames. The resulting point is calculated as the average of the points over N+1 frames. The average can be, for example, a weighted average, with a greater weight given to more recent frames. The weight can also be influenced by velocity. For example, a higher velocity can assign a greater weight to the most recent frames.

[0101] However, any method for smoothing time series data can be used instead. Another example is a Kalman filter, which attempts to estimate the current state by modeling the dynamics of the system (such as using past velocities to predict the current point) and combining that prediction with current measurements (tracker points).

[0102] 6 is a diagram of a computing environment 600 according to one embodiment. Computing environment 600 is similar to environment 400, but differs from VTO application 412 in that VTO application 602 includes a stabilization component 604. While shown as a component included in VTO rendering pipeline component 606, stabilization component 604 may be a separate component. VTO rendering pipeline component 606 is similar to component 416, but includes stabilization of the position of detected objections for rendering effects relative to the stabilized position.

[0103] In one embodiment, the operation of the stabilization component 604 is configured as described with reference to operations Stab-1 through Stab-5b above.

[0104] The VTO application 602 includes a face tracker 104B having one or more deep neural networks 106B configured for facial mask classification, localization, or segmentation to detect the presence of a facial mask (or other occluding object) in a facial image. In one embodiment, the VTO application may include a face tracker having one or more deep neural networks that localize facial features but do not detect the presence of a facial mask (or other occluding object), similar to, for example, face tracker 104A.

[0105] FIG. 7 is a flowchart of operations 700 of a computer-implemented method or the like. The method may include, for example, performing the steps illustrated in FIG. 7 by one or more processors. Embodiment 18: A method includes step 702 locating a facial feature in a current frame of a set of frames of a video stream using a face tracking engine having one or more DNNs configured to process the current frame and predict a tracker position for the facial feature; step 704 illustrating generating a current stabilized position for the facial feature in the current frame, the generation being responsive to the tracker position and a previous stabilized position of the facial feature in prior frames of the video stream; and step 706 illustrating rendering an effect in the current frame associated with the facial feature in response to the current stabilized position, the effect simulating trying on a product as a component of a virtual try-on experience. Although not shown, the operations may include providing the rendered current frame and effect for display (e.g., as an output image).

[0106] Embodiment 19: In embodiment 18, the method includes at least one of: i) providing a recommendation interface that recommends one or more makeup products for virtual try-on, where each product is associated with one or more effects that are rendered in association with one or more facial features; or ii) providing a purchase transaction interface to facilitate the purchase of makeup products.

[0107] Embodiment 20: In embodiment 18 or 19, the method locates a plurality of facial features, the plurality of facial features are grouped by importance ratings relevant to the virtual try-on experience, defining a more important group of facial features and a less important group of facial features, and the current stabilized position of each of the plurality of facial features is determined in response to the importance ratings that select between different stabilization operations to balance accuracy and device performance criteria.Embodiment 21: In embodiment 20, the plurality of facial features include a left eye object, a right eye object, and at least one mouth object grouped as more important facial features, and a left eyebrow object, a right eyebrow object, a nose object, and a face contour object grouped as less important facial features.

[0108] Embodiment 22: In any of embodiments 18 to 21, generating the current stabilized position includes one of the following operations: (a) blending the tracker position and a second position for the facial feature in the current frame using linear interpolation, where the second predicted position is responsive to an optical flow determined for the facial feature using the tracker position and a previous stabilized position for the facial feature in the immediately preceding frame; or (b) applying averaging to the tracker position and the previous stabilized position of the facial feature, where the averaging is responsive to an averaged velocity determined from the tracker position and each previous tracker position for the facial feature over the set of previous frames.

[0109] Embodiment 23: In embodiment 22, the method locates a plurality of facial features, the plurality of facial features are grouped according to an importance rating associated with the virtual try-on experience, defining a more important group of facial features and a less important group of facial features, for each individual facial feature from the more important group, a current stabilized position is generated according to operation (a), and for each individual facial feature from the less important group, a current stabilized position is generated according to operation (b), and rendering uses each current stabilized position to render one or more effects associated with at least some of the plurality of facial features.

[0110] Embodiment 24: In embodiment 21 or 22, operation (a) is configured to, for a particular facial feature to be stabilized across a set of frames including the current frame and the immediately preceding frame, blend each face point at the tracker position with a corresponding face point at the second position according to a blending coefficient that weights the respective contributions of the tracker position and the second position to generate a first blending result, and further blend the first blending result and each face point at the tracker position to generate a current stabilized position according to the distance between the pixel coordinates of each face point at the tracker position and the corresponding each face point at the second tracker position, and the blending further includes moving the first blending result toward the tracker position in response to the distance normalization coefficient.

[0111] Embodiment 25: In embodiment 24, the method includes initializing a blending coefficient to a maximum amount in each frame processed, attenuating the blending coefficient, using the attenuated blending coefficient when blending, and periodically resetting the blending coefficient to the maximum amount.

[0112] Embodiment 26: In any of embodiments 18 to 24, the method further includes performing occlusion detection by one or more neural networks on the facial feature and rendering an effect in response to the occlusion detection. The occlusion detection provides occlusion information indicating that the facial feature is occluded. In one embodiment, the occlusion information is granular and provides granular information about partial occlusions. For example, a segmentation mask from the one or more neural networks indicates which pixels of the facial feature are included (or not included). Occlusion is also described further herein below.

[0113] Corresponding system embodiments are disclosed for each of embodiments 18 through 26, and it will be understood that, for example, the system comprises a respective component having computational circuitry configured to perform the operations of the computer-implemented method embodiment.

[0114] In one implementation, an optical flow tracking operation is performed for each frame for the purpose of temporal stabilization of facial landmarks. Given images and landmarks for the previous and current frame images, optical flow can predict the location of landmarks for the current frame. The prediction is combined with the output of the facial landmark model according to a stabilization operation.

[0115] Those skilled in the art will appreciate that in addition to computing device and method aspects, computer program product aspects are disclosed in which instructions are stored on a non-transitory storage device (e.g., memory, CD-ROM, DVD-ROM, disk, etc.) and, when executed, cause a computing device to perform any of the method aspects stored herein.

[0116] Although computing devices are described with reference to processors and instructions that, when executed, cause the computing device to perform operations, it is understood that other types of circuitry other than programmable processors may be configured. Hardware components with specifically designed circuitry, such as, but not limited to, application-specific integrated circuits (ASICs) or other hardware designed to perform specific functions, may be used, which may be more efficient than a general-purpose central processing unit (CPU) programmed using software. Accordingly, apparatus aspects herein generally relate to systems or devices having circuitry (sometimes referred to as computational circuitry) configured to perform certain operations described herein, such as, but not limited to, those of method aspects described herein, regardless of whether the circuitry is configured through programming or through its hardware design.

[0117] A practical implementation may include any or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, apparatus, systems, means for performing functions, and other ways of combining the features described herein. Several embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. Additionally, other steps may be provided or steps may be eliminated from the described processes, and other components may be added to or removed from the described systems. Accordingly, other aspects are within the scope of the following claims.

[0118] Throughout the description and claims of this specification, the words "comprise" and "contain" and variations thereof mean "including but not limited to" and are not intended to exclude other elements, integers or steps. Throughout this specification, the singular encompasses the plural unless the context requires otherwise. In particular, where the indefinite article is used, it should be understood that the specification contemplates the plural as well as the singular, unless the context requires otherwise.

[0119] It should be understood that any feature, integer, property, compound, chemical moiety, or group described in connection with a particular aspect, embodiment, or example of the invention is applicable to any other aspect, embodiment, or example, except where incompatible therewith. All features disclosed herein (including any accompanying claims, abstract, and drawings), and / or all steps of any method or process so disclosed, can be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. The invention is not limited to the details of any of the foregoing examples or embodiments. The invention extends to any novel one or any novel combination of features disclosed herein (including any accompanying claims, abstract, and drawings), or any novel one or any novel combination of steps of any method or process disclosed. <Other> <Means> The computer-implemented method of Technical Idea 1 performs the steps of: one or more processors locating faces in face input images using a face tracker comprising one or more deep neural networks (DNNs) trained to locate facial features; and generating training images including the located faces, the training images including either occluded training images in which an occluding object is rendered on the face or unoccluded training images in which the face appears without a facial mask, the training images being generated for occluded face DNN training. The method of Technical Idea 2 includes, in the method described in Technical Idea 1, training an occluded face detection DNN using the training images. The method of Technical Idea 3, in the method described in Technical Idea 2, configures a face tracking engine trained with the occluded face detection DNN, and the face tracking engine classifies and locates facial features, and performs at least one of classifying or locating face-occluding objects. The method of Technical Idea 4 is a method according to any one of Technical Ideas 1 to 3, in which steps a. and b. are repeated using multiple face input images of different faces to generate multiple training images for occluded face DNN training. A method of technical idea 5 is a method according to any one of technical ideas 1 to 4, in which generating the training images comprises randomly generating the occluded training images instead of the unoccluded training images according to a probability selected to maximize occluded face DNN training. In the method of Technical Idea 6, in the method according to Technical Idea 5, the probability of generating the occluded face training images instead of the unoccluded training images is 55%. The method of Technical Idea 7 is the method of any one of Technical Ideas 1 to 6, including cropping the face from the facial input image in response to the location identification, and generating the training images using the cropped face. The method of technical idea 8 is the method of any one of technical ideas 1 to 7, wherein the occluding object for rendering includes an isolated occluding object image with a transparent background. The method of Technical Idea 9 includes, in the method of Technical Idea 8, performing one or more of resizing the occluding objects as located on the face before rendering the occluding objects as located on the face, and augmenting the occluding objects to maximize occluding face DNN training. A method of technical idea 10 is a method according to any one of technical ideas 1 to 9, wherein the occlusion object covers at least a portion of the face, and the occlusion object comprises any one of a face mask, occlusion glasses, a hat, a scarf, a hand or finger, a smartphone, hair, or a portion of any of these. The system of technical idea 11 includes a face tracker engine comprising a deep neural network (DNN) for locating a face in a face input image, and a training image generator for generating training images including the localized face, the training images including either occluded training images in which an occluding object is rendered on the face or unoccluded training images in which the face appears without the occluding object, and the training images are generated for occluded face DNN training. The system of Technical Idea 12 is the system described in Technical Idea 11, further comprising a training component for training the occluded face detection DNN with the training images. The system of Technical Idea 13 is the system described in Technical Idea 12, and includes components constituting a face tracking engine trained with the occluded face detection DNN, and the face tracking engine classifies and locates facial features, and performs at least one of classifying or locating face-occluding objects. The system of technical idea 14 is a system described in any one of technical ideas 11 to 13, which includes the face tracker engine and a data store that stores a plurality of face input images for use by the training image generator to generate a plurality of training images for occluded face DNN training. The system of Technical Idea 15 is a system according to any one of Technical Ideas 11 to 14, wherein the training image generator is configured to randomly generate the occluded training images instead of the unoccluded training images according to a probability selected to maximize facial mask DNN training. The system of Technical Idea 16 is the system of Technical Idea 15, in which the probability of generating the occluded training images instead of the unoccluded training images is 55%. The system of Technical Idea 17 is a system according to any one of Technical Ideas 11 to 16, further comprising a face cropping component configured to crop the face from the facial input image in response to the location identification, and the training image generator generates the training images using the cropped face. The system of Technical Idea 18 is the system according to any one of Technical Ideas 11 to 17, wherein the occlusion object for rendering includes an isolated occlusion object image with a transparent background. The system of Technical Idea 19 is the system described in Technical Idea 18, wherein before rendering the occluding objects as located on the face, the training image generator is configured to perform one or more of: a. resizing the occluding objects as located on the face; and b. augmenting the occluding objects to maximize face mask DNN training. The system of technical idea 20 is a system described in any one of technical ideas 11 to 19, wherein the occlusion object covers at least a portion of the face, and the occlusion object comprises any one of a face mask, occlusion glasses, a hat, a scarf, a hand or finger, a smartphone, hair, or a portion of any of these. The computer-implemented method of technical idea 21 performs the following steps: one or more processors use a face tracking engine having one or more deep neural networks (DNNs) configured to process a current frame and predict a tracker position for the facial feature to locate the facial feature in the current frame of a set of frames of a video stream; generate a current stabilized position of the facial feature in the current frame, the generation being responsive to the tracker position and a previous stabilized position of the facial feature in a previous frame of the video stream; and render an effect in the current frame associated with the facial feature in response to the current stabilized position, the effect simulating a product to be tried on as a component of a virtual try-on experience. The computer-implemented method of Technical Idea 22 is the computer-implemented method of Technical Idea 21, including at least one of: i) providing a recommendation interface that recommends one or more makeup products for virtual try-on, where each of the products is associated with one or more effects that are rendered in association with one or more facial features; or ii) providing a purchase transaction interface to facilitate the purchase of makeup products. A computer-implemented method of technical idea 23 is a computer-implemented method according to technical idea 21 or 22, wherein the method locates a plurality of facial features, the plurality of facial features are grouped by importance ratings relevant to the virtual try-on experience to define more important groups of facial features and less important groups of facial features, and a current stabilized position of each of the plurality of facial features is determined in response to the importance ratings to select between different stabilization operations to balance accuracy and device performance criteria. The computer-implemented method of Technical Idea 24 is the computer-implemented method described in Technical Idea 23, wherein the plurality of facial features includes a left eye object, a right eye object, and at least one mouth object, which are grouped as more important facial features, and a left eyebrow object, a right eyebrow object, a nose object, and a face contour object, which are grouped as less important facial features. A computer-implemented method of Technical Idea 25 is the computer-implemented method according to any one of Technical Ideas 21 to 24, wherein generating the current stabilized position includes one of the following operations: (a) blending the tracker position and a second position for the facial feature in the current frame using linear interpolation, wherein the second predicted position is responsive to an optical flow determined for the facial feature using the tracker position and a previous stabilized position for the facial feature in a immediately preceding frame; or (b) applying averaging to the tracker position and the previous stabilized position of the facial feature, wherein the averaging is responsive to an averaged velocity determined from the tracker position and each previous tracker position for the facial feature over a set of previous frames. A computer-implemented method of technical idea 26 is a computer-implemented method according to technical idea 25, wherein the method locates a plurality of facial features, the plurality of facial features are grouped according to an importance rating associated with the virtual try-on experience to define a more important group of facial features and a less important group of facial features, for each facial feature from the more important group, the current stabilized position is generated according to operation (a), and for each facial feature from the less important group, the current stabilized position is generated according to operation (b), and the rendering uses each current stabilized position to render one or more effects associated with at least some of the plurality of facial features. A computer-implemented method of Technical Idea 27 is the computer-implemented method according to Technical Idea 25 or 26, wherein operation (a) is configured to blend, with respect to a particular facial feature to be stabilized across a set of frames including the current frame and the immediately preceding frame, each face point of the tracker position with a corresponding face point of the second position according to a blending coefficient that weights respective contributions of the tracker position and the second position to generate a first blended result, further blend the first blended result and the each face point of the tracker position to generate the current stabilized position according to a distance between each face point of the tracker position and the corresponding each face point of the second tracker position, and further blending includes moving the first blended result toward the tracker position in response to a distance normalization factor. The computer-implemented method of Technical Idea 28 is the computer-implemented method of Technical Idea 27, including initializing the blending coefficient to a maximum amount for each frame processed, attenuating the blending coefficient, using the attenuated blending coefficient during blending, and periodically resetting the blending coefficient to the maximum amount. The system of Technical Idea 29 includes a face tracker engine having computational circuitry configured to locate a facial feature in a current frame of a set of frames of a video stream using one or more deep neural networks (DNNs) configured to process the current frame and predict a tracker position for the facial feature; a stabilization component having computational circuitry configured to generate a current stabilized position of the facial feature in the current frame, the generation being responsive to the tracker position and a previous stabilized position of the facial feature in a previous frame of the video stream; and a rendering component having computational circuitry configured to render an effect in the current frame related to the facial feature in response to the current stabilized position, the effect simulating trying on a product as a component of a virtual try-on experience. The system of technical idea 30 is the system described in technical idea 29, including at least one of: i) a recommendation component configured to provide a recommendation interface that recommends one or more makeup products for virtual try-on, where each of the products is associated with one or more effects that are rendered in association with one or more facial features; or ii) a purchase transaction component configured to provide a purchase transaction interface to facilitate the purchase of makeup products. The system of technical idea 31 is a system according to technical idea 29 or 30, wherein the face tracker engine is configured to locate a plurality of facial features, the stabilization component is configured to group the plurality of facial features according to an importance rating associated with the virtual try-on experience to define a more important group of facial features and a less important group of facial features, and the current stabilized position of each of the plurality of facial features is determined by the stabilization component in response to the importance rating to select between different stabilization operations to balance accuracy and device performance criteria. The system of Technical Idea 32 is the system described in Technical Idea 31, wherein the plurality of facial features includes a left eye object, a right eye object, and at least one mouth object, which are grouped as more important facial features, and a left eyebrow object, a right eyebrow object, a nose object, and a face contour object, which are grouped as less important facial features. The system of Technical Idea 33 is a system according to any one of Technical Ideas 29 to 32, wherein, to generate the current stabilized position, the stabilization component is configured to perform one of the following operations: (a) blending the tracker position and a second position for the facial feature in the current frame using linear interpolation, wherein the second predicted position is responsive to an optical flow determined for the facial feature using the tracker position and a previous stabilized position for the facial feature in a immediately preceding frame; or (b) applying averaging to the tracker position and the previous stabilized position of the facial feature, wherein the averaging is responsive to an averaged velocity determined from the tracker position and each previous tracker position for the facial feature over a set of previous frames. The system of Technical Idea 34 is the system described in Technical Idea 33, wherein the face tracking engine is configured to locate a plurality of facial features, the stabilization component is configured to group the plurality of facial features according to an importance rating relevant to the virtual try-on experience, define a more important group of facial features and a less important group of facial features, generate the current stabilized positions for each facial feature from the more important group according to operation (a), and generate the current stabilized positions for each facial feature from the less important group according to operation (b), and the rendering component is configured to render one or more effects associated with at least some of the plurality of facial features using each current stabilized position. The system of Technical Idea 35 is the system of Technical Idea 33 or 34, wherein operation (a) is configured to blend, with respect to a particular facial feature to be stabilized across a set of frames including the current frame and the immediately preceding frame, each face point of the tracker position with a corresponding face point of the second position according to a blending coefficient that weights contributions of the tracker position and the second position to generate a first blended result, further blend the first blended result and the each face point of the tracker position to generate the current stabilized position according to a distance between a pixel coordinate of each face point of the tracker position and a corresponding face point of the second tracker position, and further wherein the blending includes moving the first blended result toward the tracker position in response to a distance normalization factor. The system of Technical Idea 36 includes the system of Technical Idea 35, further including initializing the blending coefficient to a maximum amount for each frame processed, attenuating the blending coefficient, using the attenuated blending coefficient during blending, and periodically resetting the blending coefficient to the maximum amount. The computing device or computer program product of the technical concept 37 comprises a non-transitory storage device that stores computer-executable instructions that, when executed by a processor, perform the steps of a method according to any of the method concepts.

Claims

1. one or more processors locating a face in the facial input image using a face tracker comprising one or more deep neural networks (DNNs) trained to locate facial features; generating training images including the localized face, the training images including either occluded training images in which an occluding object is rendered on the face or unoccluded training images in which the face appears without a facial mask, the training images generated for occluded face DNN training; A computer-implemented method comprising the steps of:

2. The method of claim 1 , further comprising training an occluded face detection DNN using the training images.

3. 3. The method of claim 2, further comprising configuring a face tracking engine trained with the occluded face detection DNN, the face tracking engine classifying and locating facial features and at least one of classifying or locating face occluding objects.

4. 4. The method of claim 1, wherein steps a. and b. are repeated with multiple face input images of different faces to generate multiple training images for occluded face DNN training.

5. 5. The method of claim 1, wherein generating the training images comprises randomly generating the occluded training images instead of the unoccluded training images according to probabilities selected to maximize occluded face DNN training.

6. 6. The method of claim 5, wherein the probability of generating the occluded face training image instead of the unoccluded training image is 55%.

7. 7. The method of claim 1, further comprising cropping the face from the facial input image in response to the localization, and using the cropped face to generate the training images.

8. 8. The method of claim 1, wherein the occluding object for rendering comprises an isolated occluding object image with a transparent background.

9. before rendering the occluding object as located on the face; 10. The method of claim 8, further comprising: performing one or more of: resizing the occluding objects as localized to the face; and augmenting the occluding objects to maximize occluding face DNN training.

10. 10. The method of claim 1, wherein the occluding object covers at least a portion of the face, the occluding object comprising any one of a face mask, occluding glasses, a hat, a scarf, a hand or finger, a smartphone, hair, or a portion of any of these.

11. a face tracker engine comprising a deep neural network (DNN) for locating faces in a facial input image; a training image generator for generating training images including the localized faces; the training images include either occluded training images in which an occluding object is rendered over the face, or unoccluded training images in which the face appears without the occluding object; The system further characterized in that the training images are generated for occluded face DNN training.

12. The system of claim 11 further comprising a training component for training an occluded face detection DNN with the training images.

13. 13. The system of claim 12, further comprising components that comprise a face tracking engine trained with the occluded face detection DNN, the face tracking engine classifying and locating facial features and at least one of classifying or locating face occluding objects.

14. 14. The system of claim 11, further comprising a data store for storing a plurality of facial input images for use by the face tracker engine and the training image generator to generate a plurality of training images for occluded face DNN training.

15. 15. The system of claim 11, wherein the training image generator is configured to randomly generate the occluded training images instead of the unoccluded training images according to probabilities selected to maximize facial mask DNN training.

16. 16. The system of claim 15, wherein the probability of generating the occluded training image instead of the unoccluded training image is 55%.

17. 17. The system of claim 11, further comprising a face cropping component configured to crop the face from the facial input image in response to the localization, wherein the training image generator is configured to generate the training images using the cropped face.

18. 18. The system of claim 11, wherein the occluding object for rendering comprises an isolated occluding object image with a transparent background.

19. before rendering the occluding object as located on the face; The training image generator a. resizing the occluding object as located on the face; and b. Increasing the number of occluding objects to maximize face mask DNN training; 20. The system of claim 18, configured to perform one or more of the following:

20. 20. The system of any of claims 11 to 19, wherein the occlusion object covers at least a portion of the face, the occlusion object comprising any one of a face mask, occlusion glasses, a hat, a scarf, a hand or finger, a smartphone, hair, or a portion of any of these.

21. one or more processors locating the facial feature in a current frame of a set of frames of a video stream using a face tracking engine having one or more deep neural networks (DNNs) configured to process the current frame and predict tracker positions of the facial feature; generating a current stabilized position of the facial feature in the current frame, the generation being responsive to the tracker position and a previous stabilized position of the facial feature in a previous frame of the video stream; rendering an effect in the current frame associated with the facial feature in response to the current stabilized position, the effect simulating a product for trying on as a component of a virtual try-on experience; A computer-implemented method comprising the steps of:

22. 22. The computer-implemented method of claim 21, comprising at least one of: i) providing a recommendation interface that recommends one or more makeup products for virtual try-on, wherein each said product is associated with one or more effects that are rendered in relation to one or more facial features; or ii) providing a purchase transaction interface to facilitate the purchase of makeup products.

23. 23. The computer-implemented method of claim 21 or 22, wherein the method locates a plurality of facial features, the plurality of facial features being grouped by importance ratings relevant to the virtual try-on experience to define more important groups of facial features and less important groups of facial features, and a current stabilized position of each of the plurality of facial features is determined in response to the importance ratings to select between different stabilization operations to balance accuracy and device performance criteria.

24. 24. The computer-implemented method of claim 23, wherein the plurality of facial features includes a left eye object, a right eye object, and at least one mouth object grouped together as more important facial features, and a left eyebrow object, a right eyebrow object, a nose object, and a face contour object grouped together as less important facial features.

25. generating the current stabilized position, operation (a): blending, using linear interpolation, the tracker position and a second position for the facial feature in the current frame, wherein the second predicted position is responsive to an optical flow determined for the facial feature using the tracker position and a previous stabilized position for the facial feature in a immediately preceding frame; or operation (b): applying averaging to the tracker position and the previous stabilized position of the facial feature, wherein the averaging is responsive to an averaged velocity determined from the tracker position and each previous tracker position for the facial feature over a set of previous frames; 25. The computer-implemented method of claim 21, comprising one of:

26. the method locating a plurality of facial features, the plurality of facial features being grouped by an importance rating associated with the virtual try-on experience to define a more important group of facial features and a less important group of facial features; For each facial feature from the more important group, the current stabilized position is generated according to operation (a); For each facial feature from the less important group, the current stabilized position is generated according to operation (b); 26. The computer-implemented method of claim 25, wherein the rendering uses the respective current stabilized positions to render one or more effects associated with at least some of the plurality of facial features.

27. Operation (a) includes, for a particular facial feature to be stabilized across a set of frames including the current frame and the immediately preceding frame: blending each face point at the tracker location with a corresponding face point at the second location according to a blending coefficient that weights the respective contributions of the tracker location and the second location to generate a first blending result; 27. The computer-implemented method of claim 25 or 26, further blending the first blended result and the respective face points of the tracker positions to generate the current stabilized position according to a distance between pixel coordinates of each face point of the tracker positions and corresponding respective face points of the second tracker positions, and further configured to blend including moving the first blended result towards the tracker positions in response to a distance normalization factor.

28. 28. The computer-implemented method of claim 27, further comprising initializing the blending coefficient to a maximum amount for each frame processed; attenuating the blending coefficient; using the attenuated blending coefficient in blending; and periodically resetting the blending coefficient to the maximum amount.

29. a face tracker engine having computational circuitry configured to locate the facial features in a current frame of a set of frames of a video stream using one or more deep neural networks (DNNs) configured to process the current frame and predict tracker positions of the facial features; a stabilization component having computational circuitry configured to generate a current stabilized position of the facial feature in the current frame, the generation being responsive to the tracker position and a previous stabilized position of the facial feature in a previous frame of the video stream; a rendering component having computational circuitry configured to render an effect in the current frame associated with the facial feature in response to the current stabilized position, the effect simulating trying on a product as a component of a virtual try-on experience.

30. 30. The system of claim 29, comprising at least one of: i) a recommendation component configured to provide a recommendation interface that recommends one or more makeup products for virtual try-on, wherein each of the products is associated with one or more effects that are rendered in association with one or more facial features; or ii) a purchase transaction component configured to provide a purchase transaction interface to facilitate the purchase of makeup products.

31. 31. The system of claim 29 or 30, wherein the face tracker engine is configured to locate a plurality of facial features, and the stabilization component is configured to group the plurality of facial features by importance ratings relevant to the virtual try-on experience to define more important groups of facial features and less important groups of facial features, and wherein the current stabilized position of each of the plurality of facial features is determined by the stabilization component in response to the importance ratings to select between different stabilization operations to balance accuracy and device performance criteria.

32. 32. The system of claim 31, wherein the plurality of facial features includes a left eye object, a right eye object, and at least one mouth object grouped together as more important facial features, and a left eyebrow object, a right eyebrow object, a nose object, and a face contour object grouped together as less important facial features.

33. To generate the current stabilized position, the stabilization component: operation (a): blending, using linear interpolation, the tracker position and a second position for the facial feature in the current frame, wherein the second predicted position is responsive to an optical flow determined for the facial feature using the tracker position and a previous stabilized position for the facial feature in a immediately preceding frame; or operation (b): applying averaging to the tracker position and the previous stabilized position of the facial feature, wherein the averaging is responsive to an averaged velocity determined from the tracker position and each previous tracker position for the facial feature over a set of previous frames; 33. The system of any of claims 29 to 32, configured to perform one of the following:

34. the face tracking engine configured to locate a plurality of facial features; The stabilizing component comprises: grouping the plurality of facial features by importance ratings associated with the virtual try-on experience to define more important groups of facial features and less important groups of facial features; generating the current stabilized positions for each facial feature from the more important group according to operation (a); and configured to generate the current stabilized positions for each facial feature from the less important group according to operation (b); 34. The system of claim 33, wherein the rendering component is configured to render one or more effects associated with at least some of the plurality of facial features using their respective current stabilized positions.

35. Operation (a) includes, for a particular facial feature to be stabilized across a set of frames including the current frame and the immediately preceding frame: blending each face point at the tracker location with a corresponding face point at the second location according to a blending coefficient that weights the respective contributions of the tracker location and the second location to generate a first blending result; 35. The system of claim 33 or 34, further configured to blend the first blended result and the respective face points of the tracker positions to generate the current stabilized position according to a distance between pixel coordinates of each face point of the tracker positions and corresponding respective face points of the second tracker positions, and further configured to blend the first blended result towards the tracker positions in response to a distance normalization factor.

36. 36. The system of claim 35, further comprising: initializing the blending coefficient to a maximum amount for each frame processed; attenuating the blending coefficient; using the attenuated blending coefficient in blending; and periodically resetting the blending coefficient to the maximum amount.

37. A computing device or computer program product comprising a non-transitory storage device storing computer-executable instructions which, when executed by a processor, perform the steps of the method according to any method claim.

Images