Method and apparatus for virtual makeup try-on

Abstract

Methods, systems, and techniques are provided (e.g., in embodiments) for rendering effects, such as makeup effects. In one embodiment, facial shaping of facial features located by a face tracking engine is performed by mapping and warping. Rendering involves rendering the facial features as they are distorted. In one embodiment, an input image is processed to locate facial features of a face using a face tracking engine with one or more deep neural networks, and an output image is rendered using a rendering pipeline, where the output image includes makeup effects at locations associated with at least some of the facial features to simulate a virtual try-on of a makeup product, the effects having 2D shapes obtained from a predefined 2D mask image adjusted using the 3D shapes of the locations.

Description

[Technical Field]

[0001] 《Cross reference》 This application claims priority to U.S. Provisional Application No. 63 / 435,603, filed December 28, 2022, the entire contents of which are incorporated herein by reference. This application also claims priority to French Application No. FR 2303587, filed April 11, 2023, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to image processing, such as using neural networks, to apply simulation effects to one or more objects detected in an input image to generate an output image for a virtual try-on of a product or service. [Background technology]

[0003] Deep learning techniques are useful for processing images, including a series of video frames, to locate one or more objects within the images. In one example, an object is a facial feature that comprises part of a user's face. Image processing techniques are also useful for rendering object-related effects, such as augmented reality, for a user. One example of such augmented reality is providing a virtual try-on (VTO) that simulates the application of a product (or service) to an object. Product simulation in the beauty industry includes simulating makeup, hair, and nail effects. Another example may include iris localization and simulating color changes thereto, such as with colored contact lenses. These objects and simulations are as obvious as any other.

[0004] In many VTO scenarios, users generate input images using their own camera-equipped computing device, such as a smartphone, tablet, or other computing device with a camera (e.g., a webcam), and do so under a variety of uncontrolled lighting conditions.

[0005] For example, to better simulate the effects of real products, or if not for augmented reality, it may be desirable to increase the complexity of simulated effects. For example, real product effects may include glitter effects or special lighting effects. Augmented reality simulations may include shaping effects that simulate changing the shape of facial features (e.g., distortion) or applying beauty filters. Shape changes may be the result of simulating a professional service (e.g., a cosmetologist or plastic surgeon) or a personal activity such as self-care or entertainment. Examples include brow shaping, nose shaping (nostril slimming, bridge slimming), facial contour changes, eye changes, or eyelid changes (e.g., vertical or horizontal eye enlargement), etc.

[0006] Improved techniques for processing images are desirable to facilitate providing augmented reality, including VTO, experiences. Summary of the Invention

[0007] Methods, systems, and techniques are provided (e.g., in embodiments) for rendering effects such as makeup effects. In one embodiment, face shaping of facial features located by a face tracking engine is performed by mapping and warping. Rendering involves rendering the facial features as they are distorted. In one embodiment, an input image is processed to locate facial features of a face using a face tracking engine with one or more deep neural networks, and an output image is rendered using a rendering pipeline, where the output image includes makeup effects at locations associated with at least some of the facial features to simulate a virtual try-on of a makeup product, the effects having 2D shapes obtained from a predefined 2D mask image adjusted using the 3D shapes of the locations.

[0008] In one embodiment, a computer-implemented method is provided that performs the steps of: one or more processors processing an input image using a face tracking engine having one or more deep neural networks to locate facial features and generate face points defining a contour for each of the located facial features; and rendering an output image obtained from the input image using a rendering pipeline, the output image obtained by applying one or more shape changes to the particular facial features, the one or more shape changes being determined by: i) mapping a grid of spaced grid points to pixels of the particular facial feature and any associated facial features; and ii) warping at least some of the spaced grid points using respective shape changing functions, the warping changing positions of at least some of the spaced grid points to change positions of the face points of the particular facial feature; and rendering determining output pixels of the particular facial feature and associated facial features for the output image in response to the warping.

[0009] In one embodiment, there is provided a system comprising at least one processor and a memory storing instructions executable by the at least one processor, the system processing an input image using a face tracking engine having one or more deep neural networks to locate facial features and generate face points defining a contour for each of the located facial features, and rendering an output image obtained from the input image using a rendering pipeline, the output image obtained by applying one or more shape transformations to the particular facial features, the one or more shape transformations being determined by: i) mapping a grid of spaced grid points to pixels of the particular facial feature and any associated facial features; and ii) warping at least some of the spaced grid points using respective shape transformation functions, the distortions changing positions of at least some of the spaced grid points to change positions of the face points of the particular facial feature, and the rendering determining output pixels for the particular facial feature and associated facial features for the output image in response to the distortions.

[0010] In one embodiment, a computer-implemented method is provided in which one or more processors perform the following steps: process an input image to locate facial features of a face using a face tracking engine having one or more deep neural networks; render an output image obtained from the input image using a rendering pipeline; the output image includes makeup effects at positions associated with at least some of the facial features to simulate a virtual try-on of a makeup product, the effect having a 2D shape obtained from a pre-defined 2D mask image adjusted using the 3D shape of the positions.

[0011] These and other method and system aspects, among other types of aspects, will be apparent. [Brief explanation of the drawings]

[0012] [Figure 1]FIG. 1 is a diagram of an input image showing a typical scene for a VTO experience. [Figure 2] FIG. 1 is a diagram of an eye providing the sclera as a reference, according to one embodiment. [Figure 3] FIG. 1 illustrates a computing environment, according to one embodiment, for implementing one or more method aspects. [Figure 4] 1 is a screenshot of a user interface (UI) for parameter setting, according to one embodiment. [Figure 5] 4 is a flowchart of operations according to one embodiment, such as for the computing device of FIG. 3. [Figure 6] FIG. 1 illustrates an example of a face image (e.g., a cropped image of a face) annotated with a group of face points. [Figure 7] 7A and 7B are diagrams of a face distortion structure 700 for distorting eyebrows, according to one embodiment. [Figure 8] 4A and 4B are a flowchart of operations according to one embodiment, such as for the computing device of FIG. [Figure 9] 1A, 1B, and 1C are diagrams of 3D polygon meshes comprising eye meshes and lip meshes for eye- and lip-related facial features, according to one embodiment. [Figure 10] FIG. 1 is a diagram of an eye area 1000 with an eye mask 1002 according to one embodiment. [Figure 11] 4 is a flowchart of operations according to one embodiment, such as for the computing device of FIG. 3.

[0013]

[0014]

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022] DETAILED DESCRIPTION OF THE INVENTION

[0023] Lighting Adaptation

[0024] In one embodiment, a VTO application executed by a computing device includes a tracking engine and a rendering engine. The input image provided to the virtual try-on application is considered a scene. This scene typically consists of a face, a surrounding environment (such as a background), and several other objects. The scene also includes one or more light sources that alter the color and brightness of the scene. How these light sources alter the scene depends on their nature: natural (direct light from the sun) or artificial (such as incandescent or neon lights). This alteration can be a hue shift, or a difference in brightness / darkness, especially for artificial lights, and sometimes some artifacts such as shadows. In one embodiment, the VTO application can improve the realism and accuracy of the makeup applied to the user's face by taking these light sources into account.

[0025] Correcting illumination for makeup or skin effects applied to an image is described in U.S. Patent No. 10,892,166 B2 (granted January 12, 2021), entitled "System and Method for Light File Correction of Color Surfaces in an Image," the entire contents of which are incorporated herein by reference (hereinafter the "'166 Patent"). In one embodiment, a VTO application utilizes color correction as described in the '166 Patent and further adapted as described herein.

[0026] It is very difficult to estimate the hue and brightness shifts in an image without reference to the light source used or any external references. Humans tend to do this naturally when the image is disturbed, making this seem like a simple problem to solve. However, to a computer, such a task may appear very complex. In one example, FIG. 1 shows an input image 100 in which cold natural light 102 originates from a light source in the upper left and is directed diagonally toward a central face 104. In the same image, warm artificial light 106 originates from a light source in the lower right and is directed diagonally upward toward the face 104. As a result, there is a hue shift in the skin tone of the face, etc., as well as a brightness shift, as represented by the diagonal dotted line 108. While the shift is represented by a straight line 108, such a shift need not be linear or abrupt.

[0027] Recoloring algorithms can have difficulty detecting hue shifts and rendering realistic colors—there is often not enough data to infer a general tone, and using external references (such as the background) can introduce errors. For example, the background (e.g., the area 110 outside the area of ​​the human subject) can be noticeably shifted toward cooler colors, while the face 104 can be generally lit with warmer colors.

[0028] While challenging, inferring hue shifts is not impossible, and techniques exist. In one prior art example, this technique requires the use of machine learning models trained using a large database of reference images with known lighting conditions. By comparing an image to this reference set, the device can then approximate the lighting in the scene and adapt its makeup rendering using that output. This approach is similar to how the human brain works, comparing the lighting of a scene to a memory set and adapting perception. However, these models are expensive to implement and typically do not scale to real-world experiences. In one embodiment, the goal of a VTO application is for a web browser-based VTO application to run at 20 frames per second (FPS) or higher on a mobile device, such as a smartphone or tablet. Such a goal is typically incompatible with the above-mentioned techniques that use machine learning approaches to color.

[0029] Other techniques for adapting brightness and hue shifts require the use of an external reference, such as a known color grid to which colors have been pre-calibrated. This is known as a color checker, but is impractical in scenarios where many VTOs are used. While such techniques work very well and are fast to compute, they require the color grid to be available during the entire experience (as lighting changes) and, more importantly, require the distribution of such a checker to all users. Because the VTO experience is run by any user, this type of color grid is not a viable option.

[0030] Instead of relying on color checkers and machine learning models, in one embodiment according to the novel teachings herein, color operations use a standard available to any user: the sclera. Providing an example of an eye 200, as shown in Figure 2, the sclera 202 is the white part of a user's eye outside the iris 204. The sclera is fairly consistent across ethnicities, but darker skin tones, such as those of Black African descent, have a slight yellow tint.

[0031] The '166 patent describes an illumination adaptation operation that involves sampling different parts of the face to find an average skin color. In one embodiment, the illumination adaptation operation also samples for average skin color, but performs additional skin or scleral evaluation with the goal of improving results while maintaining a reasonable "cost" associated with computing device resource usage and processing time.

[0032] The input image may include the face of an individual subject whose eyes are obscured, such as by hair or dark glasses. In one embodiment, to mitigate this problem, a pre-step is included to detect whether the eyes are obscured, for example, by an Eye Coverage Detector. In such cases, evaluation of the sclera may be bypassed so that the illumination adaptation operation is not performed entirely. In one embodiment, illumination adaptation, such as that described in the '169 patent, may be performed depending on the sample skin color.

[0033] VTO ​​Applications

[0034] FIG. 3 illustrates a computing environment 300, according to one embodiment, for implementing one or more method aspects. The computing environment 300 includes a user computing device 302, such as a smartphone, a communication network 304, a server 306, and a server 308. The communication network 304 may be a wired and / or wireless network, including a public or private network, such as the Internet. The server 306 includes a server computing device, such as for providing a website. The server 308 includes a server computing device, such as for providing e-commerce services. While shown separately, the servers 306 and 308 may be a single server device. The computing environment is simplified. For example, not shown are a payment transaction gateway and other components, such as those for completing e-commerce transactions.

[0035] The computing device 302 includes a storage device 310 (e.g., a non-transitory device such as a memory and / or a solid-state drive) for storing instructions that, when executed by a processor (not shown), such as a central processing unit (CPU), a graphics processing unit (GPU), or both, cause the computing device 302 to perform operations, such as a computer-implemented method. The storage device 310 stores components providing software modules: a virtual try-on application 312, a user interface 314, a face tracker 315 having a deep neural network (DNN) (e.g., a convolutional neural network (CNN)), a face detector 315A providing face points 315B, a VTO rendering pipeline component 316, a product recommendation component 318 having product data 318A, and a purchase component 322 having a shopping cart 324 (e.g., purchase data). In one embodiment (not shown), the VTO application does not include the recommendation component and / or the purchase component, e.g., providing a product selection component for selecting products to visualize as a VTO effect.

[0036] In one embodiment, the VTO application is a web-based application retrieved from server 306. In one embodiment, the VTO application, such as a native application, is provided by a content delivery network. Product data 318A can be retrieved from content management system 307 associated with server 307A. Content management system 307 includes data store 307B that stores product-related data, such as swatch data and rendering effect data. Swatch data typically includes product images that can be used to illustrate the product. In one embodiment, swatch data (e.g., images) can be processed to extract data therefrom. Swatch data in the form of extracted data can include, for example, the color or other light-related characteristics (e.g., brightness, etc.) of the swatch data. Product data (e.g., color, etc.) can be provided to the server in other manners, such as color or other parameters. For example, a slider can provide an input that maps to a data value. In one embodiment, product data is provided to the user device (e.g., 302) by inclusion in a native application bundle and can be provided in the form of updates to the native application by server 306, etc. Thus, product data can be provided from different sources. Rendering effect data can include data for rendering effects such as simulating makeup or other properties such as a matte look, gloss look, metallic look, vinyl look, or glitter look for makeup effects, as described further herein. UI 307C (see also, e.g., FIG. 4 ) to content management system 307 can be provided to product suppliers, such as brand owners, to provide inputs such as uploading sample images and defining sample data, product data, and / or rendering effect data. In one embodiment, UI 307B is web-based.

[0037] Although not shown, user device 302 may store a web browser for executing web-based VTO application 312. In one embodiment (not shown), VTO application 312 is a native application that conforms to an operating system (not shown), and software development requirements may be imposed, for example, by the hardware manufacturer of computing device 302. The native application may be configured for web-based or similar communication to servers 306, 307A, and / or 308, as is known.

[0038] 3 illustrates, for example, various input and output data or information associated with use of VTO application 312. Such input and output data may include user input images 326 that are processed for the VTO experience, output images 328 in which product effects are simulated to provide the VTO experience, VTO product selections 320 that include user input selecting one or more product effects to be simulated, VTO product options 322 that include options for products to be virtually tried on for selection by the user of device 302, for example, and purchase transaction information 324 that includes purchase information provided to and / or received from the user to purchase a product. As noted above, not all VTO application embodiments include e-commerce functionality.

[0039] In one embodiment, via one or more of the user interfaces 314, VTO product options 322 are presented for selection for virtual try-on by simulating the effect on an input image 326. In one embodiment, the VTO product options 322 are obtained from or associated with product data 318A. In one embodiment, the product data is obtained from the server 306 and provided by the product recommendation component 318, which in one embodiment may be a product data parser that does not itself make user-based recommendations. Rather, any available product data is made available for use selection. Although not shown, user or other input may be received for use in determining product recommendations. The user may be prompted, such as via one of the interfaces 314, to provide input for determining product recommendations. In one embodiment, the product recommendation component 318 communicates with the server 306. The server 306, in one embodiment, determines recommendations based on the input received via component 318 and provides product data accordingly. The user interface 314 may present a selection of VTO products, and the display of VTO products may be updated, for example, in response to data received as the user browses or otherwise interacts with the user interface.

[0040] In one embodiment, one or more user interfaces provide command and control for acquiring an input image 326 and a VTO product selection input 320, such as identifying one or more recommended VTO products to try on. In one embodiment, the input image 326 is a facial image of a user, which may be a still image or a frame from a video. In one embodiment, the input image 326 may be received from a camera (not shown) of the device 302 or from a stored image (not shown). The input image 326 is provided to a face tracker 315 for processing, such as to locate features (e.g., objects) within the facial image using a deep neural network 315A.

[0041] In one embodiment, location output from face tracker 315 may include object classification results, segmentation masks, or one or more other detected location data and is provided to VTO rendering pipeline component 316. In one embodiment, the location output includes face points 315B from face tracker 315. Input image 326 is also provided (e.g., made available) to pipeline component 316. VTO product selection 320 is also provided to pipeline component 316 to determine which effects should be rendered. In an embodiment related to makeup simulation, one or more effects may be shown, such as for any one or more of product categories including lips, eye shadow, eye liner, blush, etc.

[0042] The VTO rendering pipeline component 316 renders effects on the input image 326, such as by drawing (rendering) the effects in layers, one layer per product effect, to generate an output image 328. Rendering occurs according to product data 318A selected by VTO product selection 320 and depending on the location of the detected objects. For example, a VTO product selection of lipstick, lip gloss, or other lip-related product invokes application of the effect to one or more detected mouth- or lip-related objects at their respective locations. Similarly, an eyebrow-related product selection invokes application of the selected product effect to the detected eyebrow objects. Typically, for a symmetrical look, the same eyebrow effect is applied to each eyebrow, the same lip effect is applied to each lip, or the same eye effect is applied to each eye area, but this is not necessarily the case. Some VTO product selections involve the selection of multiple products, such as coordinated products for eyebrows and eyes or other combinations of detected objects, which may be labeled as "product looks." The VTO rendering pipeline component 316 can render each effect, e.g., one product effect per layer, until all effects are applied. The order of application can be defined by rules or in the selection of products, e.g., lipstick before top gloss.

[0043] The user interface 314 provides an output image 328. In one embodiment, the output image 328 is presented as part of a live stream of successive output images (each instance 328), such as when a selfie video is augmented to present an augmented reality experience. In one embodiment, the output image 328 can be displayed with the input image 326, such as side-by-side for comparison. In one embodiment, it can be a before / after "in place" comparison interface, where the user moves a slider to reveal more initial or processed images. In one embodiment, the output image 328 can be saved (not shown), such as to a storage device 310, and / or shared with another computing device (not shown).

[0044] In one embodiment, the input image comprises an input image of a video conferencing session, and the output image comprises a video to be shared with another participant(s) of the video conferencing session (not shown). In one embodiment, the VTO application (which may have another name) is a component or plug-in of a video conferencing application (not shown) that allows a user of device 302 to wear makeup during a video conference with one or more other conference participants.

[0045] 3, VTO rendering pipeline component 316 is configured to apply lighting adaptation to the effect to be rendered. The components of VTO rendering pipeline component 316 are illustrated as a process flow illustrating operations (e.g., method steps) performed by one or more processors (e.g., CPU, GPU, or both) of computing device 302, for example. It should be understood that corresponding modules or software components may implement such a flow, for example. It should be understood that other components for other adaptations that may be applied before rendering are not illustrated for simplicity.

[0046] In this embodiment, for at least some purposes, color is modeled using a hue, saturation, and value / brightness (HSV or HSB) model. Operations at 316A perform skin average color detection to determine average skin color. In one embodiment, operations are performed in accordance with the '166 patent. For example, operations evaluate left and right skin (cheeks), facial skin (cheeks and forehead), left and right eye color, left and right eye sclera, eye brightness minimum / maximum, and lips. Operations at 316B perform blindfold detection to determine if the eyes are obscured such that the sclera are unavailable.

[0047] In 316C, the operation adjusts the brightness and saturation of the product data for the effect to be applied to the input image in response to product selection 320. The brightness and saturation are adjusted in response to the average skin color and minimum / maximum eye brightness values ​​detected in 316A. If multiple product effects are to be applied, step 316C is performed for each one.

[0048] In 316D, the operation adapts the hue of the applied effect, if available, based on the white color of the eye detected from the sclera in step 316A (availability detected in step 316B). In one embodiment, the operation determines whether the current color is shifted toward a warmer or colder tone. The operation interpolates between different H, S, and V values ​​based on the current color of the sclera. Thus, the hue of the effect is pushed toward a warmer or colder tone based on the detected condition. The HSV value of the sclera is examined to determine whether it is a warmer or colder tone. More detailed examples of the operation are described further below.

[0049] At 316E, the adapted product effects are rendered in the layer associated with the input image 326 to define an output image 328. In one embodiment, steps 316D and 316E are repeated for each effect that is applied, such as when multiple effects are applied. Upon completion of the steps of the pipeline component 316, the output image is provided, such as via a user interface 314.

[0050] In one embodiment, steps 316A-316C are performed by a central processor or CPU (not shown), and steps 316D and 316E are performed by a graphics processor or GPU (not shown), as indicated by the dotted boxes.

[0051] Below is a more detailed example of the operation of interpolating according to the warm or cool color of the sclera. The following inputs are used: skin refers to the HSV color at the current skin pixel, where the value of each HSV channel ranges from 0 to 1. sclera refers to the average HSV color at the sclera of the eye (either the left or right eye is used depending on which half of the face the current pixel is in), and the value of each HSV channel ranges from 0 to 1. And c makeup refers to the HSV color of the target makeup color, where the value of each HSV channel ranges from 0 to 1.

[0052] The smoothstep(e0,e1,x) function is used throughout and is defined as a function that smoothly interpolates between 0 and 1, starting at x = e0 and ending at x = e1. An example implementation can be found at en.wikipedia.org / wiki / Smoothstep at the time of filing. Also, rotationalMix(h0,h1,t) is used as a function that linearly interpolates between hue h0 and hue h1 based on t. This function takes into account that hue wraps around. For example, at t = 0.5, interpolating between 0.9 and 0.1 produces an output of 0.0.

[0053] To determine if skin is warm, the following calculations are performed: condition skin_h_1 = smoothstep(0.03,0.2,c skin,h ) condition skin_h_2 = 1.0 - smoothstep(0.3,0.35,c skin,h ) condition skin_s = smoothstep(0.3,0.4,c skin,s ) condition skin_v = smoothstep(0.3,1.5,c skin,v )

[0054] To determine if the sclera is warm, the following calculation is performed: condition eye_h_1 = smoothstep(0.06,0.7,c sclera,h ) condition eye_h_2 = 1.0 - smoothstep(0.4,0.45,c sclera,h ) condition eye_s = smoothstep(0.1,0.15,c sclera,s ) condition eye_v = smoothstep(0.6,0.7,c sclera,v )

[0055] To determine whether to apply a warming adjustment, multiply all the conditions together and use the condition warm The adjustment is done using: c makeup,h,new = rotationalMix(c makeup,h ,0.2,condition warm )

[0056] A similar process is applied to check for cool tones. The following calculations are made to determine if the skin is in the cool tones: condition skin_h_1 = smoothstep(0.5,0.8,c skin,h ) condition skin_h_2 = 1.0 - smoothstep(0.8,0.85,c skin,h ) condition skin_s = smoothstep(0.01,0.5,c skin,s ) condition skin_v_1 = smoothstep(0.6,0.7,c skin,v ) condition skin_v_2 = 1.0 - smoothstep(0.9,1.0,c skin,v )

[0057] To determine if the sclera is cool, the following calculation is performed: condition eye_h_1 = smoothstep(0.5,0.85,c sclera,h ) condition eye_h_2 = 1.0 - smoothstep(0.85,0.9,c sclera,h ) condition eye_s = smoothstep(0.1,0.15,c sclera,s ) condition eye_v = smoothstep(0.6,0.7,c sclera,v )

[0058] To determine whether to perform a cooler adjustment, multiply all the conditions together and use the condition cold The adjustment is done using: c makeup,h,new = rotationalMix(c makeup,h ,0.75,condition cold )

[0059] In method aspects, the following hue-related embodiments are provided: Hue Embodiment 1: A computer-implemented method in which one or more processors perform the steps of processing facial sclera of an input image to determine a reference hue for the face, adjusting the hue of the makeup effect according to the reference hue to render the adjusted makeup effect in the input image, and presenting an output image defined from the input image and the adjusted makeup effect as components of a virtual try-on experience via a user interface.

[0060] Hue Embodiment 2: In Hue Embodiment 1, the hue of the makeup effect is pushed towards warmer or cooler colors depending on the reference hue.

[0061] Hue Embodiment 3: In Hue Embodiment 1 or 2, the processing adjusts the brightness and saturation of the makeup effect according to an average skin color determined by processing the skin of the face and an eye brightness (e.g., minimum / maximum value) determined by processing one or more eyes of the face.

[0062] Hue Embodiment 4: In any of Hue Embodiments 1 to 3, adjusting the hue is responsive to detecting the presence of sclera to adjust the hue. For example, if sclera are not present, the hue is not adjusted. The brightness and saturation are adjusted.

[0063] Hue Embodiment 5: In any of Hue Embodiments 1-4, the method includes processing the image using a deep neural network to locate facial features on the face, the sclera being processed, and makeup effects being rendered to be located according to the facial features.

[0064] Hue Embodiment 6: In any of Hue Embodiments 1 through 5, the makeup effect is associated with a makeup product, and the method includes providing, via a user interface, recommendations of a plurality of makeup products to virtually try on, and receiving, via the user interface, a selection input selecting the makeup products for the virtual try-on experience.

[0065] Hue Embodiment 7: In Hue Embodiment 6, the selection input selects a plurality of products, and the method includes adjusting the hue of each associated makeup effect according to a reference hue to render a plurality of adjusted makeup effects in the input image.

[0066] Hue Embodiment 8: In any of Hue Embodiments 1-7, the method provides a purchasing service for conducting a purchasing transaction (e.g., via e-commerce) to purchase one or more makeup products via a user interface.

[0067] It will be understood that system aspects and computer program product aspects are respectively disclosed corresponding to each of Hue Embodiments 1 through 8.

[0068] For example, in system aspects, the following hue-related embodiments are provided: Hue Embodiment 9: A system including a rendering pipeline (e.g., via a circuit) is configured to process facial sclera of an input image to determine a reference hue for the face, adjust the hue of the makeup effect according to the reference hue to render the adjusted makeup effect on the input image, and provide an output image defined from the input image and the adjusted makeup effect for presentation via a user interface as components of a virtual try-on experience.

[0069] Makeup Effect with Glitter

[0070] Glitters are used in many different makeup products such as eye shadow, lipstick, but also blush or eyeliner. Glitters typically require many input variables and calculations to calculate realistic reflections, so simulating the effect is difficult to perform accurately.

[0071] At the core of any gloss effect is the calculation of a set of particles spread across the surface where they should shine. Calculating the position and reflectivity (e.g., particle properties) of each particle represented in a texture map can require too much computing power to run in live mode on a mobile device.

[0072] According to one embodiment, to off-load computation during live mode rendering, texture maps (position and particle properties) are pre-generated before rendering takes place and can be reused during rendering of glossy products.

[0073] According to the techniques and empirical values ​​herein, pre-generated textures provide improved gloss effects when used in real time.

[0074] Particle Texture: The first step is to calculate the particle texture, which serves to position the particles and store the properties of each particle (e.g., size, reflectance, base color, orientation, etc.). In one embodiment, the texture is calculated using a Voronoi diagram (structure). An example of a Voronoi diagram can be found at en.wikipedia.org / wiki / Voronoi_Diagram.

[0075] Voronoi Particle Textures have specific properties that are useful for defining glitter particles for rendering. Voronoi textures are created by generating random 2D points and creating boundaries between the points, where the boundaries have equal distances to the nearest point. These Voronoi regions formed from the boundaries define the locations of the glitter particles. It has been found that techniques for generating random 2D points can have a perceptible effect. In one embodiment, the random 2D points are generated according to a Poisson disk sampling technique. Poisson disk sampling produces more uniformly distributed samples in an image than uniform sampling. A discussion and examples can be found in the paper "Visualizing Algorithms," Bostock, M, June 26, 2014, available at bostock.ocks.org / mike / algorithms at the time of filing and incorporated herein by reference.

[0076] In one embodiment, for example, for a product swatch image, there are two texture images (maps) used to store particle information: one stores particle normals and the other stores particle centers. Normals are normalized vectors with their angles relative to the +z direction randomly sampled (e.g., using a Poisson disk) over an empirically determined, hard-coded range. Particle centers are coordinates normalized by the image dimensions. The texture map defines the gloss particles at their locations in 3D space, for example, using particle centers and normal vectors. In one embodiment, the normal vectors are 3D directions and include x, y, and z values ​​(where the x and y values ​​are different from the pixel or other grid-related (x, y) location of the particle center).

[0077] These textures are sampled at a specific size depending on the glitter size parameter in the product and are configured to be repeatedly sampled across boundaries. In one embodiment, the textures are sampled using mipmapping, where different resolutions of the texture are pre-generated. Mip-mapping is a graphics technique that scales an original high-resolution texture image, or map and filter, and scales it into multiple smaller resolution texture maps. During rendering, the operation dynamically selects which texture to use based on various factors, such as the output resolution relative to the glitter texture size. The texture is then scaled based on the glitter size.

[0078] Environment Mapping

[0079] A pre-generated environment map is used to define the lighting of the environment according to image-based lighting (IBL) techniques. It can be seen that instead of using an actual environment map, control over how "sparkly" the gloss is achieved by generating an environment map that looks as if there is a spotlight from a direction and concentric rings of light around the central spot. Note that this environment map can be grayscale, as it is only concerned with the brightness of the environment and not color. In one embodiment, the environment map is stored as a cubemap (e.g., six textures that can be wrapped / folded to define a cube), with each cube face stacked vertically into one image. Each cube face is what a spotlight from a particular direction would look like if the light was seen from a particular face.

[0080] Glitter Rendering

[0081] To render the gloss effect, the operation finds the gloss particle normals, which may be transformed by the rotation of the face and surface (if provided). In one embodiment, a face tracking engine may provide localization information for objects such as eyes, lips, and / or facial contours (e.g., face points). The points (localization information) may be used to determine the rotation of the face (e.g., relative to a standard). A 3D or other model may provide additional surface information, for example, regarding the surface of the area where the gloss is applied. 3D shape information is described further herein.

[0082] In one embodiment, rendering is applied using physically based rendering (PBR) and image-based lighting (IBL) techniques. Physically based rendering material properties (e.g., roughness, F0, etc.) and view direction are hard-coded (e.g., pre-calculated and stored), while the environment brightness is found from an environment map (cubemap and its texels) using a normal vector. These data are used in the PBR image-based lighting calculation to determine the specular brightness of particles as described in Learn Graphics Programming "specular-IBL," de Vries, Joe y, June 17, 2020, available at learnopengl.com / PBR / IBL / specular-IBL at the time of filing this application, and incorporated herein by reference. Note that in one embodiment, the diffuse term (one of the IBL elements) is ignored, as particles are expected to be mostly reflective.

[0083] As described herein, a makeup effect simulates the application of a product to a portion of the face, such as lipstick on the lips or eyeshadow on the eyelid area around the eyes. The portion of the face with the effect is overlaid on an input image, such as by a GPU. The portion / effect thus comprises pixel data to be determined for the overlay operation, such as using a texture map, an environment map, and rendering data. The pixels of the effect can then be considered to represent glitter particles and non-glitter particles. Whether a particular pixel, e.g., the current pixel, is a glitter particle pixel can be determined using various parameters, such as the distance to the center of the glitter particle (in a texture map) and the size of the particular glitter particle. The center of the particular glitter particle can serve as a seed for determining particle parameters, such as the particle's size (e.g., in a number generator that determines a size within a normalized range) or color (e.g., in a number generator that determines a value within a color model range).

[0084] In one embodiment, the glitter particles are circular and the size distance to the center is used to describe the particle's circular shape and glow alpha (only the center is fully lit and the light fades out towards the edge of the particle).

[0085] The results of these calculations are combined to render the particle lighting.

[0086] One limitation of Voronoi textures is that they do not allow overlap. In one embodiment, to allow the glossy effect to appear more dense and have particle overlap, a rendering operation is repeated using texture maps to define multiple overlapping texture map layers. The operation repeatedly overlays the glossy rendering with particle textures (i.e., texture maps) rotated 90 degrees, 180 degrees, and 270 degrees. In one embodiment, the operation varies the density by assigning layers an existence probability of a particular glossy particle in the texture map, and whether the particle is present depends on a random value greater or less than the existence probability. That is, if a layer's existence probability is 0.8 and a particle's value is 0.6, it is present, but if its value is 0.9, it is not present (e.g., its glow alpha is set to 0).

[0087] 4 is a screenshot of UI 400 for gloss parameter settings, for example, to define rendering effect data. UI 400 can be made available to a brand owner or other entity to define characteristics for makeup effects, such as a VTO experience. The UI can be configured to receive product data, such as defining the underlying makeup effect, as well as any special rendering effects, such as special lighting or gloss effects, makeup effect shapes (e.g., mask images), etc. In one embodiment, UI 400 is an example of UI 307B.

[0088] The UI 400 is useful for adjusting a gloss effect (e.g., a glossy look or "sparkle") such as that provided by sparkle / gloss particles in a makeup effect. Exemplary makeup effects are eye effects such as those provided by eye shadow, lip effects such as those provided by lipstick, cheek effects such as those provided by face powder, etc.

[0089] UI 400 includes a ribbon of icons 402, such as for toggling the user interface (not shown) for each of the different products, product effects, and / or associated appearances. Icon 402A is for shine ("sparkle") and invokes UI 400. UI 400 includes multiple input controls (e.g., 404-416) for receiving input for defining respective shine look properties / parameters. Controls 404-416 can take a variety of forms, with multiple controls included herein being slider controls. Other types of input controls for inputting text or similar values ​​are known and useful. Voice-activated controls can be used. Controls 404-416 include:

[0090] Color 404: Gloss base color (modified by reflectance, intensity, and color variation); color control 404 can be invoked to present further color segmentation controls (not shown), such as a color wheel and / or red, green, and blue (RGB) input value controls, for defining colors using RGB additive color values.

[0091] Reflectivity 406: How reflective the gloss is.

[0092] Color Variation 408: 0 means no variation, 100% means variation across the entire color spectrum.

[0093] Density 410: How much gloss does it produce?

[0094] Intensity 412: Amount of gloss visible.

[0095] Size 414: Gloss base size (can be changed depending on size fluctuations).

[0096] Size Variation 416: 0 means no variation, 100% means all glitter particles vary in size (eg, within the normalized range).

[0097] Thus, in one embodiment, the rendering operation first renders a base product (e.g., with a shape applied to an area of ​​the face associated with one or more facial features). The rendering operation then applies one or more, preferably multiple, gloss layers of glitter particles on top (e.g., four layers total). Multiple layers are preferably used to allow for particle overlap. For the random size of the glitter particles in the layer of glitter particles, the rendering operation generates random sizes (affected by the glitter size) between hard-coded minimum and maximum values, and linearly interpolates between the base glitter size and the random size based on a parameter called "gloss size variation." A value of 0 for this parameter indicates no randomness, and a value of 1 indicates completely random size. For the random color of the glitter particles, the operation generates random colors within a specific hue, saturation, and lightness (HSL) range (e.g., in one embodiment, limited to only colors with high saturation and high lightness). Similar to the glitter size, the operation linearly interpolates between the base glitter color and the random color based on a parameter called "gloss color variation." A value of 0 for this parameter indicates no randomness, and a value of 1 indicates completely random color. Pixel values ​​for glossy makeup effects are assigned according to their distance to the center of a glossy particle. If the current pixel's distance to the center is within the circle size, then the current pixel is treated as a glossy pixel; otherwise, nothing is changed for that pixel (as the base product has already been rendered). Each pixel in the pre-generated texture map contains the center of the nearest particle, so only one texture lookup needs to be done for the current pixel to determine the nearest particle.

[0098] 5 is a flowchart of operations 500 according to one embodiment, such as a computing device (e.g., device 302) of FIG. 3. The operations 500 define a computer-implemented method that includes performing the method steps by one or more processors. For example, in a method aspect, the following gloss-related embodiments are provided:

[0099] Glossy Embodiment 1: A computer-implemented method in which one or more processors process faces in input images to determine the location of facial features, the input images being processed by a face tracking engine comprising at least one deep neural network to locate the facial features (step 502), rendering makeup effects associated with the facial features, including a glossy effect defined from pre-computed textures and a pre-computed lighting environment map for determining the location and lighting of the glossy effect (step 504), and providing output images defined from the input images and the makeup effects for presentation via a user interface as components of a virtual try-on experience (step 506). The face tracking engine may, for example, comprise a face tracker 315 as described above. Rendering (and provision of the output image) may be provided by a rendering pipeline 316 adapted to the glossy effect teachings herein, for example.

[0100] Gloss embodiment 2: In gloss embodiment 1, the makeup effect includes lip makeup effect, eye area makeup effect or cheek area makeup.

[0101] Glossy Embodiment 3: Glossy In embodiment 1 or 2, the rendering step defines the glossy effect according to physically based rendering techniques.

[0102] Gloss Embodiment 4: In any of Gloss Embodiments 1 to 3, the pre-calculated texture map defines a gloss position and a gloss reflection angle for each of a plurality of gloss particles, the light environment map simulates a light source (e.g., in three-dimensional space) to determine the specular brightness of the specific gloss particle according to the reflection angle of the specific gloss particle, and the rendering defines a pixel value of a pixel of the makeup effect and adjusts the lighting of the current pixel according to the distance between the current pixel and the gloss position of the specific particle, the specular brightness of the specific gloss particle, and the size of the specific gloss particle. Gloss Embodiment 5: In Gloss Embodiment 4, the specular brightness is further determined according to the three-dimensional shape of the makeup effect and the rotation of the face.

[0103] Glitter Embodiment 6: In glitter embodiment 4 or 5, the size of the specific glitter particles is randomly assigned within a normalized range.

[0104] Glitter Embodiment 7: In any of Glitter Embodiments 4 to 6, the lighting is further responsive to the glow alpha of a particular glitter particle to define a fade of the light from the center.

[0105] Glitter Embodiment 8: In any of Glitter Embodiments 4 to 7, the rendering further colors the makeup effect according to the glitter color of the particular glitter particle.

[0106] Glossy Embodiment 9: In any of Glossy Embodiments 4 to 8, multiple glossy particles are randomly spaced without overlap within the texture map, and rendering overlaps the glossy particles within the glossy effect by repeating the definition of pixel values ​​using the texture map within multiple overlapping texture map layers.

[0107] Glossy Embodiment 10: In Glossy Embodiment 9, the rendering varies the gloss particle density of the gloss effect, randomly determining whether a particular particle in the texture map is present in any one of the overlapping texture map layers.

[0108] Glossy Embodiment 11: Glossy In any of embodiments 4 through 10, the normal (eg, defined as a vector) of the specular angle is used in conjunction with the lighting environment map to determine the spectral brilliance.

[0109] Gloss Embodiment 12: In any of Gloss Embodiments 4 to 11, each of the gloss locations of the plurality of gloss particles is randomly assigned using Poisson disk sampling.

[0110] Glossy Embodiment 13: Glossy Any one of embodiments 1 to 12, wherein the rendering is responsive to any one or more rendering effect parameters including color, reflectance, color variation, density, intensity, size, and size variation parameters.

[0111] It will be understood that system aspects and computer program product aspects are disclosed corresponding to each one of the gloss embodiments 1-13, respectively.

[0112] Further gloss-related method aspects are disclosed, such as for computing devices configured to pre-calculate maps for use during rendering. For example, Gloss Embodiment 14: A computer-implemented method in which one or more processors pre-calculate textures and a light environment map to determine the position and lighting of a gloss effect as a component of a makeup effect to be rendered relative to facial features located by a face tracking engine of a virtual makeup try-on application, and provide the pre-calculated texture and light environment map to a rendering pipeline of the virtual makeup try-on application to render the makeup effect with the gloss effect. Gloss Embodiment 15: In Gloss Embodiment 14, the pre-calculated texture and light environment map are defined to generate the gloss effect according to a physically based rendering technique that models particles in a lit environment. Gloss Embodiment 16: In Gloss Embodiments 14 or 15, a texture map is pre-calculated according to a Voronoi diagram technique to provide random positions for multiple gloss particles for the gloss effect, and the texture map is further calculated to comprise a reflection angle for each of the multiple gloss particles. Glossy Embodiment 17: In Glossy Embodiment 16, a light environment map is defined to model the light source, and upon rendering, a specular gloss depending on the reflection angle is provided for each of multiple glossy particles in the glossy effect. Glossy Embodiment 18: In Glossy Embodiment 16 or 17, the positions of the glossy particles are assigned using a Poisson disk sampling distribution.

[0113] It will be understood that additional method-related embodiments adapted as may be applicable from any of Gloss Embodiments 1 through 13 are applicable to Gloss Embodiments 14 through 18. It will be understood that system aspects and computer program product aspects are each disclosed corresponding to, or adapted as described above for, a respective one of Gloss Embodiments 14 through 18. Any of the Gloss embodiments can be combined with any one or more of the hue, shape, and mesh embodiments, e.g., combining method aspects thereof, defining corresponding system aspects, etc.

[0114] Shaping a Facial Feature

[0115] Typically, a makeup VTO applies product effects to facial features as they are identified by a face tracking engine, such as face tracker 315. The VTO typically uses only the original image and applies a "painting" color or texture on top to cover the original color of the detected facial feature, as regular makeup does. However, with regard to the brow category, some beauty products sold include shaping tools and products that allow the user to change the overall or local shape of the eyebrows. It may be desirable to simulate the reshaping. In addition to detecting and rendering the detected shape, an additional set of transformations, such as those described herein, can be used to perform, for example, brow warping.

[0116] In one embodiment, eyebrow distortion is an image manipulation technique that transforms (deforms) the pixels of the original image. Pixel transformation (deformation) is preferable to replacing the original eyebrows with synthetic eyebrows. Removing and replacing extremes can be difficult and can provide results that are not sufficiently realistic.

[0117] Some examples of eyebrow modifications include a global arch raise, a reduction or increase in inner thickness, and top and bottom cleaning. An eyebrow shape can be characterized as having an inner portion closer to the nose, a middle portion, and an outer portion farthest from the nose. The inner, middle, and outer portions lie along or define the arch of the brow, which has an upper line (closest to the forehead) and a lower line (closest to the eyes).

[0118] In one embodiment, eyebrow values ​​(e.g., forehead parameters) that can be transformed include outer portion: horizontal and vertical alignment, thickness, inner portion: horizontal and vertical alignment, thickness, arch: local and global increase / decrease, arch pointiness, middle portion: thickness, cleaning: top, bottom, inner portion, and global: horizontal and vertical shift.

[0119] In one embodiment, a face tracker, such as tracker 315, can locate facial features and provide face points as shown in Figure 6. Figure 6 shows an example of a face image 600 (e.g., an image with a cropped face 602) annotated with a group of face points 604. The depiction of points on the face is for illustrative purposes. The output of face tracker 315 need not include an annotated image; for example, the output of face points and cropped face may be separate data, or the cropped face itself may not be provided.

[0120] In one embodiment, the group of face points 604 includes a face outline face point 604A, right and left eyebrow face points 604B, 604C, left and right eye face points 604D, 604E, a nose face point 604F, and two lip face points 604G, 604H. In one embodiment, the lip face points include an outer lip group 604G (around the mouth) and an inner lip group 604H (between the lips). The face points within each group are numbered (e.g., 0, 1, 2, ...) to help define the contours of detected objects. In one embodiment, the face tracker assigns each point so that it is consistently positioned relative to the contour of the object it represents. For example, a particular point may always be located at the right corner of the mouth. In one example, the face points are X,Y pixel coordinates relative to the cropped face image 602 and are associated with each detected object from (e.g., one of) the network of trackers 315.

[0121] In one embodiment, operation for the face tracker and rendering pipeline components of the virtual try-on application performs a computer-implemented method comprising one or more steps such as the following.

[0122] 1. A face tracker is used to process the input image and provide one or more neural networks to locate facial features to obtain eye and eyebrow facial points.

[0123] 2. Figures 7A and 7B are diagrams of a face distortion structure 700 for warping eyebrows (an example of a facial feature), according to one embodiment. As shown in Figure 7A, a rectangular distortion grid (704) is defined over each eyebrow (e.g., left eyebrow 702) and centered on the area of ​​each eyebrow. The size of this box (grid) is calculated from the size and distance of various facial features. The grid is also rotated (via an affine transformation) to match the rotation of the face. For example, the face contour face points 604A can be used to determine the rotation of the face relative to a reference position. A coordinate system (e.g., as represented by points within grid box 704) is defined within the grid, with coordinates ranging from (0,0) to (1,1). This simplifies warping calculations because there is no need to worry about face rotation or eyebrow size. The eyebrow face point group and eye face point group for the right or left eye are mapped to this coordinate system (respective grids determined for each pair of these facial features) for use during deformation calculations.

[0124] 3. The distortion grid is discretized into a grid of 2D points (e.g., the points illustrated in box 704, including points 704A, 704B, 704C, 704D, and 704E, where 704C is located within eyebrow 702 between points 704A and 704D). Moving a grid point (e.g., 704D) similarly moves facial image pixels at or near that grid point and its surrounding pixels (e.g., the neighborhood of grid point 704E). In FIG. 7B, the movement of the grid point and the resulting movement of eyebrow pixels to reshape eyebrow 702 occurs. A linear interpolation of such distortion is performed between the points. In one embodiment, the GPU provides this linear interpolation as a built-in function. There is a trade-off: choosing a denser grid results in smoother deformations but higher (GPU) processing time. For example, through experimentation, it was found that for GPUs provided in commonly available mobile devices such as smartphones at the time of filing, a 25x50 point grid (1,250 points total) provided a good balance between quality and speed. Going higher resulted in diminishing gains in quality. With more powerful hardware, denser grids could be used.

[0125] 4. Each point on the grid is then distorted, the distortion being calculated using the applicable face points and distortion parameters, i.e. the distortion (movement) of a predefined grid point is defined in relation to a particular eyebrow deformation (e.g. parameters).

[0126] 5. The face image (e.g., its pixels) is then distorted based on the distortion grid. For example, eyebrow shaping involves shaping both eyebrows, each with its own grid. Eyebrow shaping can be performed with other beauty filter shaping, such as eye or eyelid shaping, lip shaping, nose bridge shaping, nostril shaping, face contouring (e.g., chin slimming), etc., where each feature to be shaped has its own grid.

[0127] Distortion Calculation

[0128] Although the distortion calculations used for each eyebrow parameter are different, there are some shared techniques between them: each parameter has an associated distortion function.

[0129] p out = f param (p in ,p brows ,p eyes ,v param )

[0130] where p in are the input grid points,

[0131] p brows is the set of eyebrow points,

[0132] p eyes is a set of eye points, which are an example of associated facial features, and are useful for defining facial features (e.g., associated facial features) in the eye region between the eyebrows and the eye points along the curve of the top of the eye; initially, the set of eye points and the eyebrow points are provided by the face tracker.

[0133] v param is the value of that parameter,

[0134] p out are the output grid points.

[0135] In one embodiment, the distortion is performed by iteratively applying each distortion function (e.g., where the shape parameters (params) indicate the changes in the eyebrows) on a distortion grid. The overall process is:

[0136] For each parameter:

[0137] f param The grid is distorted by applying to each grid point.

[0138] The eyebrow point (and eye point) may have changed due to distortion, so f param p brows and p eyes Apply it to the new eyebrow point p' brows (and the eye point p' eyes ) and use the new points for the next iteration.

[0139] Although not shown, for example, a user interface similar to user interface 400 may be provided to receive input of eyebrow parameters. In one embodiment, sliders or other controls may be provided to allow input of how much change to make to the parameters.

[0140] Distortion Technology

[0141] There are several techniques that are common among distortion functions. This section describes these techniques:

[0142] Curve Matching: An A-curve is calculated for the current eyebrow (using the eyebrow points from the face tracker to fit a curve along the center of the eyebrow), and a target curve is calculated from the parameters. This is useful for changing the overall curvature of the eyebrow. The eyebrow is deformed (e.g., within a grid) so that the current curve is distorted to match the target curve. This is done by matching points on the source curve with the target curve and distorting them based on the delta. For points far from the curve within the grid, a 2D Gaussian attenuation function is applied so that points closer to the curve are distorted more than points farther away. The σx and σy of this Gaussian function can be modified to adjust the "area of ​​affect." For example, σy needs to be large enough to affect the entire thickness of the eyebrow. In some cases, the Gaussian function can be replaced with a different function, for example, an asymmetric version of the Gaussian, or one with a "shorter tail" or "longer tail."

[0143] Point Matching: This is similar to curve matching, except that instead of matching several curves, several discrete points are matched instead. This is useful in cases such as adjusting the alignment of a particular part of an eyebrow.

[0144] Region Dilation / Compression: The region along a curve or around a point can be "compressed" or "expanded" (in a specific direction, such as vertically, or uniformly). This is done by moving nearby points closer / farther apart, or by attenuating them with a Gaussian function. This is useful for operations such as varying the thickness of eyebrows or sharpening the edges of eyebrows.

[0145] Implementation: The distortion grid is rendered by using the grid points as vertices, such as to define a polygon mesh. Triangles are fitted to the vertices and the image is UV-mapped to the vertices. UV mapping in this embodiment is a 2D modeling process that projects the surface of a 2D model onto a 2D image for texture mapping. The letters "U" and "V" refer to the axes of the 2D texture.

[0146] Beauty Filter: Similar to the operation that performs eyebrow distortion, the beauty filter operation is a set of transformations that change the shape of other facial features. In one embodiment, the transformations include eye widening (vertical and horizontal), jaw slimming, nostril slimming, and nose bridge slimming. The implementation of the beauty filter is the same as the implementation of the eyebrow filter, although the affected areas are different, if applicable.

[0147] 8A and 8B are flowcharts of operations 800 and 804 of a computing device (e.g., device 302) such as that of FIG. 3, according to one embodiment. Operation 800 defines a computer-implemented method that includes performing the method steps by one or more processors. Operation 804 is further defined as steps 804A-804C in one embodiment.

[0148] For example, in the method aspect, the following formulation-related embodiments are provided: Formulation Embodiment 1: A computer-implemented method in which one or more processors process an input image using a face tracking engine having one or more deep neural networks to locate facial features and generate face points defining a contour for each of the located facial features (step 802), and then render an output image obtained from the input image using a rendering pipeline, the output image being obtained by applying one or more shape transforms to the particular facial features (step 804), the one or more shape transforms including: i) mapping a grid of spaced grid points to pixels of the particular facial feature and any associated facial features (804A); and ii) warping at least some of the spaced grid points using respective shape transform functions, the warping changing positions of at least some of the spaced grid points to change positions of the face points of the particular facial feature (804B), and rendering determining output pixels for the particular facial feature and any associated facial features for the output image in response to the distortion (804C).

[0149] The face tracking engine may, for example, comprise a face tracker 315 as described above. Rendering (and providing an output image) may, for example, be provided by a rendering pipeline 316 adapted to the gloss effect teachings herein. Formulation Example 2: In Formulation Example 1, the face tracking engine and rendering pipeline are components of a VTO application for simulating the effect of makeup products applied to facial features. Formulation Example 3: In Formulation Example 2, the rendering pipeline renders makeup effects to particular reshaped facial features such that the output image includes the reshaped particular facial features and the makeup effects.

[0150] Formulation embodiment 4: In any of formulation embodiments 1 to 3, the method includes providing a user interface for receiving input for defining shape parameters for one or more shape changes, and the rendering is responsive to the user input.

[0151] Formulation embodiment 5: In formation embodiment 4, at least some of the shape change functions perform one or more of the following: curve matching to match an intermediate curve along the center of a particular facial feature contour to a target curve defined by one or more shape change shape parameters, and attenuating position changes to spaced grid points according to distance to the target curve; point matching to match discrete points of a particular facial feature contour to target points defined by one or more shape change shape parameters; and region expansion or region compression to expand or compress a region along a facial point curve or around a particular pixel according to the shape parameters, where the region expansion or region compression is attenuated by a decay function responsive to the distance from the facial point curve or particular pixel.

[0152] Formulation Embodiment 6: In any of formation embodiments 1 to 5, determining the pixels includes fitting triangles to vertices defined by the warped spaced grid points and UV-mapping pixels of the particular facial feature and any associated features to the vertices.

[0153] Formulation embodiment 7: In any of formulation embodiments 1 to 6, a particular facial feature defines a first facial feature, and the rendering step is repeated with respect to a second facial feature to generate an output image having at least two shape-modified facial features.

[0154] Formation embodiment 8: In any of formation embodiments 1 to 7, the shape changes applied to a particular facial feature include any one of eyebrow shaping, nose shaping such as nostril slimming or bridge slimming, facial contour changes such as chin slimming, eye changes or eyelid changes such as vertical or horizontal eye widening, or lip changes such as lip fulling.

[0155] It will be understood that system aspects and computer program product aspects are disclosed corresponding to, for example, each of the formulation embodiments 1 through 7. Any formulation embodiment can be combined with any one or more of the hue, gloss, and mesh embodiments described above.

[0156] 3D Mesh from 2D Landmark

[0157] In one embodiment, to facilitate rendering of more realistic facial features, rendering operations (e.g., in pipeline 316) may be configured to utilize 3D polygon meshes, whether for applying shaping effects or makeup effects (e.g., makeup looks). In one embodiment, 3D polygon meshes are used to render special lighting effects (e.g., metallic, vinyl) for makeup effects.

[0158] A mesh is a collection of vertices, edges, and faces that define (e.g., model) the shape of a 3D object. In addition to being able to achieve complex lighting effects, using polygon meshes has many advantages: many commercially available GPUs, such as those used in consumer smartphones and tablets at the time of filing, are optimized for processing polygons and can render polygon meshes efficiently, resulting in faster rendering speeds.

[0159] In a VTO application, user input can indicate which product should apply one or more makeup effects associated with at least some of the facial features located by the face tracker. The effects can be applied to the image using a mask (e.g., a 2D mask image) that provides a shape for the makeup effect in a position relative to at least some of the facial features. For example, to provide a realistic effect according to the 3D shape of the facial features, the mask for the effect can be warped to realistically fit a 3D mesh model of the facial features. The warped 2D mask image can be UV mapped to be used to form the effect during rendering.

[0160] 3D mesh generation

[0161] In one embodiment, the face tracker 315 detects 2D face points. A rendering operation uses the 2D face points to generate a 3D mesh. In one embodiment, the operations to generate the 3D mesh are:

[0162] 1. Estimate 3D face points using the detected 2D face points and an existing 3D face model. Obtain 3D face points by projecting the 2D face points into 3D space. Typically, cameras and human eyes see through perspective projection. When 3D points are projected into 2D space, a perspective projection matrix is ​​applied. In one embodiment, the inverse of the projection matrix can be applied to go from 2D points to 3D points. However, because the depth is unknown, the result is actually a 3D ray rather than a 3D point. To obtain the 3D points, the operation approximates the face with a 3D plane and intersects the 3D ray with the plane to obtain the 3D points. A hard-coded offset is added to each point to account for the fact that the face is not actually flat. For example, for lip points, points near the corners of the mouth are deeper from the camera than points in the center.

[0163] 2. Use the 3D facial points to interpolate spline curves and use these smoothed curves as the contours of facial features.

[0164] 3. Generate a 3D grid (e.g., of spaced grid points) and deform the grid points to fit the curvature of the 3D contour lines.

[0165] 4. Extract 3D facets (triangles) from the deformed grid and group them together to form a 3D facial mesh.

[0166] 9A, 9B, and 9C are diagrams of a 3D polygon mesh 900 comprising an eye mesh 902 and a lip mesh 904 for eye- and lip-related facial features, according to one embodiment. FIG. 9A shows a front view, while FIGS. 9B and 9C show partial side-to-side views. FIG. 9A is enlarged relative to FIGS. 9B and 9C for convenience. It is clear that the eye-related meshes in this embodiment include the area around each eye, including the associated eyebrows.

[0167] Map 2D Mask Image onto 3D Mesh

[0168] In one embodiment, known techniques for mapping can be used. For example, to "warp" a 2D mask image onto the surface of a 3D mesh, an operation utilizes a technique called UV mapping. In this embodiment, the mask image references shapes for makeup effects, such as eyeshadow shapes, that are placed adjacent to the upper contour of the eye and extend toward the lower contour of the eyebrow, and that may extend outside the eye. FIG. 10 is a diagram of an eye region 1000 with an eye mask 1002, according to one embodiment. This mapping process can be interpreted as a paper 3D model of an object, such as a sphere, that is placed flat on a table, and each of the object's 3D coordinates can be mapped to 2D coordinates on the flat image. The purpose of unwrapping the 3D coordinates is to map these 3D coordinates to an image / photograph so that the 3D images can have a realistic-looking surface with textures derived from these images.

[0169] Using an eye shadow effect as an example, the general process for mapping a mask image to a 3D mesh is as follows: use an eye template image (e.g., mask image 1002 that generates the eye makeup effect) as a texture image, unwrap the 3D mesh to 2D, and adjust the 2D UV vertices so that the texture appears properly on the 3D mesh. In one embodiment, the 3D mesh can be unwrapped, such as by using available third-party software. The 2D vertices can be aligned with reference points in the mask image (e.g., which gives the shape of the makeup effect).

[0170] Figure 11 is a flowchart of operations 1100 according to one embodiment of a computing device (e.g., device 302) of Figure 3. The operations 1100 define a computer-implemented method that includes performing method steps by one or more processors.

[0171] For example, in method aspects, the following mesh-related embodiments are provided: Mesh Embodiment 1: A computer-implemented method in which one or more processors perform the following steps: process an input image to locate facial features of a face using a face tracking engine having one or more deep neural networks (step 1102); render an output image obtained from the input image using a rendering pipeline, the output image including makeup effects at positions associated with at least some of the facial features to simulate a virtual try-on of a makeup product, the effect having a 2D shape obtained from a pre-defined 2D mask image adjusted using the 3D shape of the positions (step 1104).

[0172] Mesh Embodiment 2: In Mesh Embodiment 1, the face trackers each generate face points defining a shape outline for each located facial feature, and the pipeline generates a 3D mesh using 3D models of at least some of the facial features and the face points to define the 3D shape of the location, warps a pre-defined 2D mask image to the 3D mesh to adjust the 2D mask image, and maps (e.g., unwraps) the 2D mask image as warped to provide the 2D shape of the makeup effect at the location.

[0173] Mesh embodiment 3: In mesh embodiment 2, the pipeline performs UV mapping to map onto the warped 2D mask image using a 3D mesh.

[0174] Mesh Embodiment 4: In any of Mesh Embodiments 1 to 3, the makeup effect is associated with a makeup product, and the method includes providing, via a user interface, recommendations of a plurality of makeup products for virtual try-on; and receiving, via the user interface, a selection input selecting the makeup products for the virtual try-on experience.

[0175] Mesh Embodiment 5: In any of Mesh Embodiments 1 to 4, the selection input selects one or more products for rendering one or more makeup effects at two or more locations associated with at least some facial features, and the rendering renders the first makeup effect at the first location using a first predefined 2D mask image adjusted using the 3D shape of the first location, and the rendering renders the second makeup effect at the second location using a second predefined 2D mask image adjusted using the 3D shape of the second location. In one example, the one or more products comprise eye shadow, the one or more makeup effects include an eye shadow effect, the first and second locations are left and right eye regions associated with the facial features of eyes and eyebrows, respectively, and the first and second predefined 2D mask images are left and right eye mask images, which may be an image of one eye and its mirror image.

[0176] Mesh Embodiment 6: In any of Mesh Embodiments 1 to 5, the method includes providing, via a user interface, a purchasing service for conducting a purchase transaction (e.g., via e-commerce) to purchase a makeup product.

[0177] It will be understood that system aspects and computer program product aspects are disclosed corresponding to, for example, each of Mesh Embodiments 1 through 6. Any one or more of the Mesh Embodiments can be combined with any one or more of the Hue, Gloss, and Shape embodiments, as described above.

[0178] Those skilled in the art will appreciate that in addition to computing device and method aspects, computer program product aspects are disclosed in which instructions are stored on a non-transitory storage device (e.g., memory, CD-ROM, DVD-ROM, disk, etc.) and, when executed, cause a computing device to perform any of the method aspects stored herein.

[0179] A practical implementation may include any or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, apparatus, systems, means for performing functions, and other ways of combining the features described herein. Several embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. Additionally, other steps may be provided or steps may be eliminated from the described processes, and other components may be added to or removed from the described systems. Accordingly, other aspects are within the scope of the following claims.

[0180] Throughout the description and claims of this specification, the words "comprise" and "contain" and variations thereof mean "including but not limited to" and are not intended to exclude other elements, integers or steps. Throughout this specification, the singular encompasses the plural unless the context requires otherwise. In particular, where the indefinite article is used, it should be understood that the specification contemplates the plural as well as the singular, unless the context requires otherwise.

[0181] It should be understood that any feature, integer, property, compound, chemical moiety, or group described in connection with a particular aspect, embodiment, or example of the invention is applicable to any other aspect, embodiment, or example, except where incompatible therewith. All features disclosed herein (including any accompanying claims, abstract, and drawings), and / or all steps of any method or process so disclosed, can be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. The invention is not limited to the details of any preceding example or embodiment. The invention extends to any novel one or any novel combination of features disclosed herein (including any accompanying claims, abstract, and drawings), or any novel one or any novel combination of steps of any method or process disclosed. <Other> <Means> The computer-implemented method of Technical Idea 1 performs the following steps: one or more processors process an input image using a face tracking engine having one or more deep neural networks to locate facial features; generate face points defining a contour for each of the located facial features; render an output image obtained from the input image using a rendering pipeline; the output image is obtained by applying one or more shape transformations to the particular facial feature, the one or more shape transformations being determined by mapping a grid of spaced grid points to pixels of the particular facial feature and any associated facial features and distorting at least some of the spaced grid points using respective shape transformation functions; the distortions change positions of at least some of the spaced grid points to change positions of the face points for the particular facial feature; and the rendering determines output pixels for the particular facial feature and associated facial features for the output image in accordance with the distortions. The method of Technical Idea 2 is the method of Technical Idea 1, wherein the face tracking engine and rendering pipeline are components of a VTO application for simulating the effect of makeup products applied to facial features. The method of Technical Idea 3 is the method of Technical Idea 2, wherein the rendering pipeline renders the makeup effect onto the reshaped specific facial feature such that the output image includes the reshaped specific facial feature and the makeup effect. A method of technical idea 4 is a method according to any one of technical ideas 1 to 3, wherein the method includes providing a user interface for receiving input for defining shape parameters for the one or more shape changes, and the rendering is responsive to the user input. The method of Technical Idea 5 is the method described in Technical Idea 4, wherein at least some of the shape change functions perform one or more of the following: curve matching to match an intermediate curve along the center of the contour of the particular facial feature to a target curve defined by the shape parameters of the one or more shape changes; attenuating position changes to spaced grid points according to the distance to the target curve; or point matching to match discrete points of the contour of the particular facial feature to target points defined by the shape parameters of the one or more shape changes; or region growing or region compressing to expand or compress a region along a facial point curve or around a particular pixel according to the shape parameters, wherein the region growing or region compressing is attenuated by a decay function responsive to the distance from the facial point curve or the particular pixel. A method of technical idea 6 is a method according to any one of technical ideas 1 to 5, wherein determining the pixels includes fitting triangles to vertices defined by the distorted spaced grid points, and UV-mapping the pixels of the particular facial feature and any associated features to the vertices. A method of technical idea 7 is a method according to any one of technical ideas 1 to 6, wherein the specific facial feature defines a first facial feature, and the rendering step is repeated for a second facial feature to generate an output image having at least two shape-modified facial features. The method of Technical Idea 8 is a method according to any one of Technical Ideas 1 to 7, wherein the shape change applied to the specific facial feature includes any one of eyebrow shaping, nose shaping such as nostril slimming or bridge slimming, facial contour change such as chin slimming, eye change or eyelid change such as vertical or horizontal eye widening, or lip change such as lip fulling. A system of technical idea 9 includes at least one processor and a memory storing instructions executable by the at least one processor, and uses a face tracking engine having one or more deep neural networks to process an input image to locate facial features, and generates face points defining a contour for each of the located facial features; and uses a rendering pipeline to render an output image obtained from the input image, the output image being obtained by applying one or more shape transformations to the particular facial features, the one or more shape transformations being determined by mapping a grid of spaced grid points to pixels of the particular facial feature and any associated facial features and distorting at least some of the spaced grid points using respective shape transformation functions, the distortions changing positions of at least some of the spaced grid points to change positions of the face points for the particular facial feature; and the rendering determines output pixels for the particular facial feature and associated facial features for the output image in response to the distortions. The computer-implemented method of technical idea 10 performs the following steps: one or more processors process input images to locate facial features of a face using a face tracking engine having one or more deep neural networks; render output images obtained from the input images using a rendering pipeline; the output images include makeup effects at positions associated with at least some of the facial features to simulate a virtual try-on of a makeup product, the effects having 2D shapes obtained from a pre-defined 2D mask image adjusted using the 3D shapes of the positions. The method of Technical Idea 11 is the method of Technical Idea 10, wherein the face trackers each generate face points defining a shape outline for each of the located facial features, and the pipeline generates a 3D mesh using 3D models of at least some of the facial features and the face points to define the 3D shape at the location, warps the pre-defined 2D mask image onto the 3D mesh to adjust the 2D mask image, and maps the 2D mask image as warped to provide the 2D shape of the makeup effect at the location. The method of Technical Idea 12 is the method of Technical Idea 11, wherein the pipeline performs UV mapping to map the warped 2D mask image using the 3D mesh. A method of technical idea 13 is a method according to any one of technical ideas 10 to 12, wherein the makeup effect is associated with a makeup product, and the method includes providing, via a user interface, recommendations of a plurality of makeup products for virtual try-on, and receiving, via the user interface, a selection input for selecting the makeup product for the virtual try-on experience. A method of Technical Idea 14 is a method according to any one of Technical Ideas 10 to 13, wherein the selection input selects one or more products for rendering one or more makeup effects at two or more locations associated with the at least some facial features, the rendering including rendering a first makeup effect at the first location using a first predefined 2D mask image adjusted using a 3D shape of the first location, and rendering a second makeup effect at the second location using a second predefined 2D mask image adjusted using a 3D shape of the second location. The method of Technical Idea 15 is the method of Technical Idea 14, wherein the one or more products comprise eye shadow, the one or more makeup effects include an eye shadow effect, the first and second locations are left eye and right eye regions associated with eye and eyebrow facial features, respectively, and the first and second predefined 2D mask images are left eye and right eye mask images. A method of technical idea 16 is a method according to any one of technical ideas 10 to 15, wherein the method includes providing a purchasing service via the user interface for conducting a purchase transaction to purchase a makeup product. The computing device or computer program product of concept 17 comprises a memory storing instructions for execution by a processor, causing the processor to perform a method according to any of the preceding concepts.

Claims

1. one or more processors processing an input image using a face tracking engine having one or more deep neural networks to locate facial features and generate respective face points defining a contour for each of the located facial features; using a rendering pipeline to render an output image derived from the input image, the output image derived from applying the one or more shape transformations to specific facial features; the one or more shape transformations are determined by mapping a grid of spaced grid points to pixels of the particular facial feature and any associated facial features and distorting at least some of the spaced grid points using respective shape transformation functions, the distortion changing positions of at least some of the spaced grid points thereby changing positions of the face points of the particular facial feature; the rendering determines output pixels for the particular facial feature and associated facial features for the output image in response to the distortion. A computer-implemented method comprising the steps of:

2. 10. The method of claim 1, wherein the face tracking engine and rendering pipeline are components of a VTO application for simulating the effects of makeup products applied to facial features.

3. 3. The method of claim 2, wherein the rendering pipeline renders the makeup effect onto the reshaped particular facial feature such that the output image includes the reshaped particular facial feature and the makeup effect.

4. 4. The method of claim 1, wherein the method includes providing a user interface for receiving input for defining shape parameters for the one or more shape changes, and wherein the rendering is responsive to the user input.

5. At least some of the shape-changing functions curve matching to match an intermediate curve along the center of the contour of the particular facial feature to a target curve defined by the shape parameters of the one or more shape changes, attenuating position changes to spaced grid points according to distance to the target curve; or point matching for matching discrete points of the contour of the particular facial feature to target points defined by the shape parameters of the one or more shape variations; or region growing or region compressing to expand or compress a region along a face point curve or around a particular pixel according to the shape parameters, the region growing or region compressing being attenuated by a decay function responsive to a distance from the face point curve or the particular pixel; 5. The method of claim 4, further comprising:

6. 6. The method of claim 1, wherein determining the pixels comprises fitting a triangle to vertices defined by the distorted spaced grid points and UV-mapping the pixels of the particular facial feature and any associated features to the vertices.

7. 7. The method of claim 1, wherein the particular facial feature defines a first facial feature and the step of rendering is repeated for a second facial feature to generate an output image having at least two reshaped facial features.

8. 8. The method of any one of claims 1 to 7, wherein the shape changes applied to the particular facial feature include any one of eyebrow shaping, nose shaping such as nostril slimming or bridge slimming, facial contour changes such as chin slimming, eye changes or eyelid changes such as vertical or horizontal eye widening, or lip changes such as lip fulling.

9. 1. A system comprising at least one processor and a memory storing instructions executable by the at least one processor, processing an input image using a face tracking engine having one or more deep neural networks to locate facial features and generate respective face points defining a contour for each of the located facial features; using a rendering pipeline to render an output image derived from the input image, the output image derived from applying the one or more shape transformations to specific facial features; the one or more shape transformations are determined by mapping a grid of spaced grid points to pixels of the particular facial feature and any associated facial features and distorting at least some of the spaced grid points using respective shape transformation functions, the distortion changing positions of at least some of the spaced grid points thereby changing positions of the face points of the particular facial feature; The rendering determines output pixels for the particular facial feature and associated facial features for the output image in response to the distortion.

10. one or more processors processing the input image to locate facial features of the face using a face tracking engine having one or more deep neural networks; Rendering an output image obtained from the input image using a rendering pipeline, the output image including a makeup effect at a location associated with at least some of the facial features to simulate a virtual try-on of a makeup product, the effect having a 2D shape obtained from a predefined 2D mask image adjusted using the 3D shape of the location. A computer-implemented method comprising the steps of:

11. the face trackers each generate face points defining a shape contour for each of the located facial features; The pipeline comprises: generating a 3D mesh using 3D models of at least some of the facial features and the facial points to define the 3D shape of the location; warping the predefined 2D mask image onto the 3D mesh to adjust the 2D mask image; 11. The method of claim 10, further comprising mapping the 2D mask image as warped to provide the 2D shape of the makeup effect at the location.

12. The method of claim 11, wherein the pipeline performs UV mapping to map onto the warped 2D mask image using the 3D mesh.

13. 13. The method of claim 10, wherein the makeup effect is associated with a makeup product, the method comprising: providing, via a user interface, recommendations of a plurality of makeup products for virtual try-on; and receiving, via the user interface, a selection input selecting the makeup product for the virtual try-on experience.

14. 14. The method of claim 10, wherein the selection input selects one or more products for rendering one or more makeup effects at two or more locations associated with the at least some facial features, and wherein the rendering renders a first makeup effect at the first location using a first predefined 2D mask image adjusted using a 3D shape of the first location, and the rendering renders a second makeup effect at the second location using a second predefined 2D mask image adjusted using a 3D shape of the second location.

15. 15. The method of claim 14, wherein the one or more products comprise eye shadow, the one or more makeup effects include an eye shadow effect, the first and second locations are left and right eye regions associated with eye and eyebrow facial features, respectively, and the first and second predefined 2D mask images are left and right eye mask images.

16. 16. The method of any of claims 10 to 15, wherein the method includes providing, via the user interface, a purchasing service for conducting a purchase transaction to purchase makeup products.

17. A computing device or computer program product comprising a memory storing instructions for execution by a processor, said computing device or computer program product causing said processor to carry out a method according to any preceding claim.

Images