3D VIRTUAL HAIR TRY-ON METHOD AND SYSTEM

Abstract

METHOD AND SYSTEM FOR 3D VIRTUAL HAIR TRY-ON The methods, systems, devices, and techniques in accordance with embodiments herein are directed to providing a new approach to virtual hair try-on by replacing the user's natural hair with a 3D representation (e.g., mesh, point cloud, implicit function, etc.) that mimics the behavior of real hair. In one embodiment, a physical simulation of at least one force deforms the 3D hairstyle. In one embodiment, the 3D representation includes detail and texture information independent of color to enable accurate calculation of the resulting color. In one embodiment, hair color is applied in response to estimated ambient light conditions.The 3D representation can be segmented into groups of hair strands, allowing for vertical recoloring such as highlights, contouring, or balayage (a highlighting technique applied to real hair by retouching the hair color to create a graduated, more natural effect). Figure for abstract: none.

Description

Title of the invention: METHOD AND SYSTEM FOR VIRTUAL 3D HAIR FITTING FIELD OF THE INVENTION

[0001] The present application relates to image processing, including applying an effect to an object in an input image to define an output image, and more particularly to applying a three-dimensional (3D) effect to hair and including a method and system for virtual try-on (VTO) of 3D hair. CONTEXT

[0002] Realistic hair recoloring in the context of real-time VTO is a challenging topic due to the organic nature of hair and the large amount of detail required to achieve good realism. "Real-time" in this context refers to a VTO experience where a live video, such as a self-portrait video capture by a smartphone, tablet, or webcam (which examples are not limiting), is processed and an effect is applied in real-time to simulate a product or service effect such as a hair coloring effect upon displaying the live video. Traditional techniques usually involve detecting the hair area in the image (hair segmentation) and recoloring the pixels using color matching. While effective, this technique has some drawbacks, for example, it is highly dependent on the quality of the input.If the input lacks texture and detail (in the case of very dark hair for example), these details are not reconstructed during recoloring and the results look poor or worse for any lighter shade tried, as these lighter shades usually have more texture and detail.

[0003] Another disadvantage is that consumers want to try different types of hair coloring, including highlights, balayage, contouring, and other recoloring styles. These types of effects treat components or portions of hair differently, for example, by applying different colors to different portions. Traditional hair tracking and coloring yields unsatisfactory results. Current technology makes it virtually impossible to reliably track the same strand of hair across a sequence of frames (such as in a video stream).

[0004] Prior work for a 3D hair VTO includes patent references KR101997702B1 entitled “3D simulation System for hair-styling”, issued on October 1, 2019 (published as KR20190052832A on May 17, 2019) and PCT / KR2008 / 005109 filed on September 1, 2008 (published as WO2010024486A1 on March 4, 2010). These references describe a system for simulating 3D hair on a user in real time.

[0005] However, these patents focus on changing the user's hairstyle rather than applying an effect to a retained hairstyle. ABSTRACT

[0006] The methods, systems, devices, and techniques in accordance with embodiments herein are directed to providing a novel approach to virtual hair try-on by replacing the user's natural hair with a 3D representation (e.g., mesh, point cloud, implicit function, etc.) that mimics the behavior of real hair. In one embodiment, a physical simulation of at least one force deforms the 3D hairstyle. In one embodiment, the 3D representation includes detail and texture information independent of color to enable accurate calculation of the resulting color. In one embodiment, hair color is applied in response to estimated ambient light conditions.The 3D representation can be segmented into groups of hair strands, allowing for vertical recoloring such as highlights, contouring, or balayage (a highlighting technique applied to real hair by retouching the hair color to create a graduated, more natural effect).

[0007] In one embodiment, user input is received to segment (i.e., group) the hair strands according to the input, e.g., allowing a user to select adjacent hair strands and define multiple groups.

[0008] In accordance with embodiments herein, the hairstyle of the input frame(s) is preserved and the hair color is changed to match a hair coloring product. In this hairstyle preservation, a machine learning model is used to infer the characteristics of the user's hair and choose the closest matching hair model.

[0009] In accordance with embodiments herein, a set of intuitive parameters is used to control color placement, brightness and intensity as well as the ability to use multiple hair colors simultaneously.

[0010] In accordance with embodiments herein, a physical simulation approach is used in which the hair strands react in a physically plausible manner in response to movements of the user's head, gravity, and collisions of the strands against the user's head and body.

[0011] To further enhance realism, in accordance with embodiments herein, a machine learning model is used to estimate the environment map of the user's environment and the map is used to illuminate the 3D hair mesh in a way that matches the environmental lighting.

[0012] The following statements provide various aspects and features disclosed in the embodiments herein. These and other aspects and features will be readily understood by those skilled in the art, including aspects of computer program products. It is also understood that aspects / features of computer devices or systems may have corresponding method aspects / features and vice versa.

[0013] Statement 1: A system comprising: at least one processor; and at least one memory device storing computer-readable instructions that, when executed by the at least one processor, cause the system to: process an input image using a three-dimensional (3D) face detector array to determine a 3D representation of a face of a head, the input image comprising the face and the head; determine a 3D hairstyle model of a 3D hairstyle to be applied to the input image; render the 3D hairstyle from the 3D hairstyle model in response to the 3D representation of the face to define an output image, wherein the rendering applies a hair color to recolor the hair of the 3D hairstyle model, the hair color being refined in accordance with an estimate of ambient light conditions; and provide the output image for display by a display device.

[0014] Statement 2: The system of statement 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to perform a physical simulation of the hair movement and hair position of the 3D hairstyle to be rendered, the physical simulation being responsive to the 3D representation of the face and the physical simulation modeling at least one force and deforming the 3D hairstyle in accordance with the at least one force; and wherein the rendering is responsive to the deformed 3D hairstyle.

[0015] Statement 3: The system of Statement 1, wherein the computer-readable instructions that, when executed by the at least one processor, cause the system to provide one or more interfaces for at least: receiving input identifying one or more of the 3D hairstyle or hair color for rendering; recommending VTO options including one or more of the 3D hairstyles or hair colors to be selected for identification for rendering; and performing a transaction to purchase a product associated with the 3D hairstyle or hair color. Brief description of the drawings

[0016] [Fig.l] [Fig.l] is a block diagram of a system, in accordance with one embodiment.

[0017] [Fig.2] [Fig.2] is a representation of a graphical user interface, in accordance with one embodiment.

[0018] [Fig.3] [Fig.3] is an image of a colored strand, in grayscale for convenience and in accordance with one embodiment.

[0019] [Fig 4A-4B] Figures 4A and 4B are illustrations of components of a hair model, in accordance with one embodiment.

[0020] [Fig.5] [Fig.5] is a flowchart of operations in accordance with one embodiment.

[0021] [Fig.6] [Fig.6] is an illustration of a computing environment comprising one or more systems, in accordance with one embodiment. DETAILED DESCRIPTION System Overview

[0022] [Fig. 1] is a block diagram of a device 100 having one or more storage devices 102 storing a plurality of components (e.g., software and / or data components), in accordance with one embodiment, that operate to provide a 3D hair VTO application. The device 100 is typically a consumer user device such as a smartphone, tablet, laptop, desktop computer, etc. Often, such types of devices have or are coupled to a camera to produce video comprising a sequence of video images / frames for use in a VTO application. In one embodiment, an input image 104A is processed to render an output image 104B. The image 104A may be a single “still” image (e.g., provided by a user (not shown) or a frame extracted from a camera feed (not shown) or input video.The output image 104B is typically, but not necessarily, of the same type, meaning that the image 104B is a corresponding still image or an image from a video stream for display, etc.

[0023] The image 104 is processed by a plurality of components including: a 3D face detector 106; a hair segmentation component 108, an environment map estimation component 110 and a hairstyle detector component 112.

[0024] The 3D face detector 106 operates to detect a 3D representation of a face, providing a face with a 3D representation (e.g., a mesh) and associated data as described in more detail. Preferably, one or both of a face position and a face rotation are also determined. The hair segmentation component 108 detects where hair is located on the image, providing a segmentation mask indicating hair and not hair pixels. The environment map estimation component 110 estimates an environment map primarily related to lighting. The lighting environment map includes an HDR environment panorama (map) with data (e.g., colored pixels) for locating major directional light sources and estimating the color of ambient light. Although a mesh is often described herein as one type of 3D shape representation, other modes of 3D shape representation may be employed such as point cloud, implicit function, etc.

[0025] In the embodiment, the hairstyle detector component 112 detects a hairstyle in the image 104A and locates (e.g., performs a match) a matching 3D style definition (e.g., as defined by a 3D artist) stored in the hairstyle data store 112A (e.g., a database 112A) for use as described in more detail. Although not shown in [Fig. 1], a mismatched hairstyle (e.g., a different style, that can be selected (by the user) from the store 112A can be used to replace the hair in the image 104A.

[0026] A plurality of rendering components (e.g., which define a rendering engine 120) process the outputs of components 106, 108, 110, and 112 / 112A. A hair removal filter component removes detected hair from image 104A in response to the hair mask and performing retouching on the image. The result of the retouching technique is enhanced by applying a bald filter to the head (i.e., a 3D mesh of a scalp oriented and positioned using the output of (2) and recolored with the user's skin color). This bald filter enhances rendering by masking the user's actual hair. In another embodiment, not shown, a bald filter is not employed.Instead, users with long hair or another hairstyle with a large hair volume (e.g., spiked hair, among others) that interferes with a 3D hairstyle to be applied are prompted to apply to the input image a tight ponytail or other tight hairstyle that preferentially conforms to the shape of the user's head to conceal and / or minimize the user's hair (i.e., hair volume) to improve rendering of the 3D hairstyle to be applied. Although not shown, in one embodiment, a hair detector function is provided to process the input image, e.g., by evaluating a hair segmentation mask or using a hair volume classifier, to determine that a user should apply the hair concealment / minimization style to improve results.In one embodiment, the user may decline and continue in any case.

[0027] An occlusion detector component 124 determines occlusions and renders a depth map.

[0028] A physics simulation component 126 models the physics of the hair for movement, gravity, etc., using guided strands to adjust the hair mesh that models the 3D hairstyle. From the face mesh and hair mesh data, an approximation of the user's head, neck, and shoulders is made to serve as a collision mesh. The collision mesh (e.g., a collision 3D shape representation) is used with a set of "guided strands" to calculate the physics simulation of the hair's movement and position. The guided strands are used to deform the hair model (previously designed by a 3D artist).

[0029] A hair rendering engine component 128 renders the hair according to the deformed 3D model, etc. That is, the hair is recolored using state-of-the-art techniques and the rendering is further enhanced using the environment map. The output image 104B is provided, for example, sent to a display device such as a display screen for display. The output image 104B may be saved to a storage device or communicated (e.g., shared) with another device, for example, via email, social media, etc. Components Face and hair detection

[0030] In some embodiments, the rendering engine 120 uses information about the 3D structure of the face in the form of a 3D mesh. To facilitate this, a 3D face mesh detector is used. In one embodiment, the 3D face mesh includes the frontal portion of the face, most of the forehead, as well as the sides of the face extending to the ears. Simulation accuracy can be optimized when the mesh provided by the detector also includes the scalp, neck, and shoulders, but these can be added to the mesh as a post-processing step if they are not available.

[0031] In some embodiments, the hair removal filter component 122 uses the 2D segmentation mask of the hair. To do this, a machine learning model may be used to estimate the segmentation mask given in an input image.

[0032] For the respective face and hair detectors 106 and 108, an “off-the-shelf” solution or a custom-made detector may be used. Here, “off-the-shelf” refers to a pre-existing component that does not need to be entirely created from scratch. Some degree of refinement may be used for the specific purpose(s) at hand, e.g., training to provide desired results or a desired shape.

[0033] thereof. In one embodiment, with a custom detector, the machine learning models for 3D face mesh detection and 2D hair segmentation can be fused for optimal speed. Additionally, the scalp, neck, and shoulders can be included in the mesh output, which is typically not the case in a “standard” 3D face detector. Hairstyle detection

[0034] In one embodiment, to maintain the user's hairstyle, a machine learning model is used on the hairstyle detector 112 to detect characteristics of the user's hair through processing an image. In one embodiment, the detected characteristics include color (including any information about highlights or multiple colors), texture (afro, straight, curly, wavy), length, visibility (visible, covered, bald), and whether the hair is knotted, braided, or has bangs.

[0035] In one embodiment, the features are used to choose the closest matching hairstyle from the 3D hair model database 112A, as well as to adjust the physical parameters to better match the physical behavior of the style. Hair removal

[0036] In one embodiment, for better realism, the user is rendered virtually bald by the hair removal filter 112 before the new hair model is rendered on top. That is, the hair in the image is removed, for example by masking. In one embodiment, the filter 122 performs the following steps:

[0037] Removal-1: The hair segmentation mask is expanded to increase its coverage area.

[0038] Removal-2: A retouching technique according to a known technique (e.g., the fast marching method (Telea, 2003)) is used to erase and reconstruct the erased region, using as the region the mask from the previous step. In one embodiment, as a performance optimization, the mask may be miniaturized before using the algorithm.

[0039] Removal-3: A scalp mesh is aligned with the face mesh if the face mesh does not include the scalp. If the face mesh already includes the scalp, then the scalp can be extracted from the face mesh.

[0040] Removal-4: The scalp mesh is rendered onto the image, along with skin material of a similar skin tone to the face using physically based rendering (PBR) techniques. In one embodiment, the skin tone is estimated by making the average of all skin pixels in the face. In one embodiment, Poisson blending is also applied to smooth the transition between the original face and the rendered scalp.

[0041] In an alternative embodiment, existing hair is removed in another manner. For example, a GAN model (not shown) may be provided to process the input image and remove the hair. In such an embodiment, a hair segmentation mask may not be used. 3D hair rendering Hair model representation

[0042] In one embodiment, the system 100 supports two different hair model representations. The first is a hair map model where hair is modeled using maps (3D rectangular planes) where each map contains a texture with hairs grouped together and a transparent background. The advantage of this representation is that it is efficient in terms of memory usage and simulation time, but it is less realistic because the system 100 does not have fine granular control over individual strands. For example, the system 100 can only simulate the physics for the hair map as a whole and not individual strands. Such a model is inefficient for hair with complex geometry, such as curly hair.

[0043] The second representation is with hair curves. The hair model consists of a set of Bézier curves, each curve representing a strand of hair. This representation offers more realism and control over individual strands, but requires more memory and simulation time.

[0044] Memory-efficient hair curve representation

[0045] In one embodiment, to reduce both the storage and RAM requirements of the hair curve representation, a number of optimizations have been developed. Normally, cubic Bézier curves are used to model a strand of hair which requires two control points per curve. As an optimization, quadratic Bézier curves (requiring one control point) are used instead when a hair curve can be sufficiently approximated using the quadratic Bézier curve, otherwise reverting to a cubic Bézier curve.

[0046] Additionally, for symmetrical hairstyles, only one side of the hairstyle is recorded and the other side is produced as a mirror image when the hair model is loaded.

[0047] To reduce storage requirements, curve points may be stored in binary format and compressed using a standard lossless compression algorithm. Hair Map Direction Map Editor

[0048] When using a hair map representation, a direction map texture is defined that specifies the 2D strand direction for each pixel on the hair map. The strand direction information is required by the shading model.

[0049] However, it is difficult to define this direction map texture using available standard image editing tools. In one embodiment, to facilitate this, a custom tool allows an artist or developer to define a direction map intuitively by manipulating a vector field using a mouse or touch controls. The vector field is displayed over the hair strand texture as a grid of arrows (e.g., vectors), and the user can drag their mouse or finger to align the arrows with the direction of the strands. [Fig. 2] shows a portion of a graphical user interface 200 (e.g., which may be displayed on a display screen) presenting a vector field 202 presented over a plurality of hair strands, e.g., hair strands of groups 204A, 204B, 204C, 204D, and 204E, as well as over a plurality of GUI controls 206.The vector field 200 includes a grid of short vectors represented as short line segments (e.g., 206A) each having a direction. The segments may each be represented by an arrow construction such as a line segment with a point for compactness in the interface. In one embodiment, the point at one end shows an origin of the arrow, the line slopes away from it to show the direction. The direction of these vectors is manipulated using gesture input such as pulling / sliding along the hair using a pointing tool (e.g., a mouse, stylus, etc.) or a finger, etc. In one embodiment, the display screen is a touch-sensitive input device. The map is defined from the values ​​of the vectors in field 202. Commands 206 are provided to define a brush (e.g., dimensions) that is actuated by gesture input (e.g., tool or finger, etc.) as well as to import the hair image. Reset the vector values ​​of field 202 or export the values ​​of field 202 to define the map. In one embodiment, the vector field is used to form a direction map texture that is mapped onto the hair map. Multiple textures are mapped onto each hair map, where each texture contains different information (e.g., one texture contains the direction of the strands, and another texture contains the color and opacity of the strands). Within each texture are multiple groups of hairs. When a texture is mapped onto a map, only a subregion of the texture containing a specific group of hairs is used. Hair shading model

[0050] To render the hair model, in one embodiment, a Marschner hair shading model is used with some modifications to allow real-time execution. One optimization used is to pre-compute the azimuthal and longitudinal scattering functions for different angle inputs and store the result in a look-up table (LUT). In addition, the Marschner model requires performing a mathematical integration of the scattering equation over all light directions. In one embodiment, this integration can be simplified to a summation over the different light sources by using discrete light sources (such as point light sources or directional light sources). This will be discussed in more detail in the next section.

[0051] Environment map lighting approximation

[0052] To enhance realism, an environment map is used (e.g., at component 110) to model light coming from all directions in the user's environment. This can, for example, be represented using a high dynamic range (HDR) panoramic image. However, using the environment map with the Marschner shading model requires integrating the diffusion equation across the entire environment map, which cannot be done in real time. In one embodiment, as an approximation, the environment map is converted into a set of discrete light sources. This conversion is done by detecting the brightest points on the environment map and finding the average color for each point to determine the light color and intensity. Each light point is converted into a directional light source. Indirect lighting by double diffusion

[0053] For realistic hair rendering, indirect illumination is simulated in accordance with one embodiment. Indirect illumination refers to light rays from light sources that have undergone more ray-surface interaction (the surface in this case being the hair) before reaching the virtual camera. A ray-surface interaction can be either a reflection (where the light ray bounces off the surface) or a transmission (where the light ray enters the surface).

[0054] For example, a ray of light emitted by a light source may interact with many strands of hair before reaching the camera. For example, a light source located behind the hair. Whenever a ray-surface interaction occurs, the direction, color, and intensity of the ray of light change. The effect of indirect lighting is most noticeable for lighter hair colors (such as blonde hair).

[0055] Accurately simulating all these light rays in real time can involve millions of light rays and hundreds of thousands of strands of hair. The computational cost also grows exponentially with the number of bounces / transmissions per ray, since a single ray is scattered into multiple rays at each surface interaction.

[0056] In one embodiment, the double scattering technique is employed to solve this problem. By approximating the aggregate behavior of light rays using statistical models, the system does not need to simulate each ray and ray-surface interaction, thereby approximating indirect illumination in real time. In one embodiment, the hair shading model incorporates this technique. Recolorable dual-diffusion LUTs

[0057] In the dual diffusion technique, different LUTs are pre-generated for optimization. However, these LUTs depend on the hair color and the LUTs are recalculated if the hair color changes. If multiple hair colors are applied simultaneously, multiple sets of LUTs must be generated, which consumes more memory and increases the loading time.

[0058] To optimize double scattering and LUT generation, a modification was made to the double scattering technique where an additional dimension was added to the LUT input for one channel of the input color. For example, when using an RGB hair color, the LUT would be applied separately to each channel of the RGB color. This allows the hair color to be changed without having to recalculate the LUTs. Occlusion management

[0059] In a 3D model of hair for application to a head, portions of the hair are occluded by the head, neck, and shoulders when the head is viewed from a particular viewpoint. In one embodiment, a depth map is used to track the depth of all rendered pixels during rendering operations. This depth map is initialized with the depth of the face, neck, and shoulders, in accordance with a standard approach in computer graphics for handling occlusion.

[0060] When rendering hair, the hair geometry (whether hair maps or curves) is rasterized into pixels by the GPU. Before rendering each pixel, the depth of that pixel is checked against the depth of the depth map at the current frame position. If the depth is greater than that of the depth map, the pixel is not rendered. Otherwise, the pixel is rendered (overwriting any existing pixels at the same frame position) and the depth map is updated.

[0061] When using hair maps, the rendering process must also check if the current pixel is transparent. If it is transparent, it will not render the current pixel regardless of depth.

[0062] Hair strand color transfer

[0063] To support rendering of actual hair color products such as to provide a virtual try-on of a specific product, the hair rendering engine 120 (e.g., at component 128) is configured to receive a strand of hair representing a hair color product. [Fig. 3] shows a representative strand 300. Although shown in a grayscale image for deposition, the image is typically a color-based image representing the product color (e.g., an RGB image).

[0064] A strand is capable of capturing a broad color distribution from the product, as opposed to using a single color. Each hair model as used in the system 100 includes a hair texture that is matched onto the mesh or strands. The color distribution of the hair strand is transferred to the hair texture using a histogram matching process. This process is further described in U.S. Patent Application No. 18 / 109,310 filed on February 14, 2023.For example, in one embodiment, a computer-implemented method includes performing on a processor one or more steps including: matching gray levels from a strand image and gray levels from a hair portion of an input image by matching their respective frequencies to establish a mapping relationship between the strand image and the hair portion, wherein the gray levels of the strand image are associated with respective strand color values; and coloring a pixel in the hair portion based on a strand color value determined using a gray level of the pixel and the mapping relationship.

[0065] In one embodiment, the respective frequencies are probabilities of the occurrence of the gray levels respectively in the strand image or in the hair portion. In one embodiment, the frequencies are represented by histograms or by cumulative distribution functions. In one embodiment, the method further comprises calculating a lookup table that maps the gray levels of the hair portion to the strand color values. In one embodiment, the coloring of the pixel of the hair portion is performed by a graphics processing unit (GPU) using a shading element. In one embodiment, the lookup table is provided to the shading element using a 1D texture. In one embodiment, the method includes preprocessing the strand image or the input image using a deep neural network to improve the accuracy of coloring the pixel in the hair portion. In one embodiment, an output image may be displayed including the input image and the colored hair portion. The output image may be processed using a guided filter. Wick variation

[0066] In a real-life situation, hair color is usually not uniform. This is especially true for light-colored hair. Using the hair strand color transfer technique above, the recolored hair texture provides this variation. For better control of the variation, there are adjustable parameters for the frequency and scale of the variation. The frequency will control the amount of detail in the variation, while the scale will determine how closely similarly colored strands appear next to each other.

[0067] Frequency is implemented by blurring the hair texture to reduce detail, while scale is achieved by adjusting the texture mapping so that the texture is more compressed or stretched relative to the hair surface.

[0068] Additionally, for blonde hair, it is common for the top portion of the hair near the roots to be more faded. The renderer (component 128) exposes parameters to control the fading and color of the roots. Smoothing and antialiasing

[0069] In one embodiment, smoothing and anti-aliasing techniques are used to augment the rendering of 3D hair. In particular, when the 3D hair mesh is rendered into the scene, the higher resolution texture of the mesh appears very pixelated. This is particularly noticeable on loose strands of hair and when the hair is in motion. To make the experience less stark, a Box Blur and Poisson Blending algorithm is applied to the edge of the 3D hair mesh, which smooths out the pixelation of the loose strands at the edges, which are most noticeable. As noted, the 3D hair mesh could be hair maps, in one embodiment. As noted, the 3D hair mesh could be hair curves, in one embodiment. When rendering hair, in one embodiment, each pixel has an opacity value in addition to its RGB color.Since edges typically have an opacity greater than 0 but less than 1 due to antialiasing, these values ​​can be used as an easy way to identify edges. Similarly, a multi-sample antialiasing (MSAA) technique is used in one embodiment. to reduce the pixelation of all hair strands in the mesh, so that a more natural looking result is achieved that blends better with the video resolution.

[0070] In one embodiment, the hair strands may be smaller than one pixel in thickness. To achieve this, the hair is rendered at a higher resolution than the resolution of the output image and then miniaturized. The resulting strands appear smaller than one pixel and also appear smoother due to the interpolation used in the downsampling. Multicolored investments

[0071] In one embodiment, the hair rendering by component 128 is capable of applying multiple colors simultaneously to the hair mesh. These color segments may be arranged horizontally, vertically, or as a root recolor on the mesh.

[0072] For horizontal placement, in one embodiment, the mesh is segmented into multiple sections based on the length of the hair strands and the colors applied to each of those sections. For example, a color may be applied only to a section near the root of the hair strand on the scalp. For example, a color (e.g., a different color) may be applied only to the section at the tip of the hair strands. Everywhere else, boundaries may be defined for the horizontal sections and colored accordingly. To smooth the color transition, a color gradient is applied between each of the sections.

[0073] For vertical placement, in one embodiment, the 3D representation (e.g., the 3D mesh) segments the hair strands into multiple groups, and colors are applied to each of these groups. Note that in this case, the groups may include non-contiguous clumps of strands. For example, in one case, a color may be applied to the group of strands at the front of the head, creating a face-framing effect, or in another case, a color may be applied to small clumps of strands throughout the mesh, to create a streaked, highlighted effect. To smooth the color transition, the color of adjacent hair strands that do not belong to the same group is adjusted, and the color of the different groups is blended to produce a visual transition color for that strand.In one embodiment, user input is received to segment (i.e., group) the hair strands according to the input, e.g., allowing a user to select adjacent hair strands and define multiple groups. Coloring may be applied to these interactively defined groups.

[0074] For root recoloring, in one embodiment, there is control over the amount of the underlying user's hair root color to be exposed. A root color parameter controls the amount of the original root color that is to be exposed and a root migration parameter controls the amount of the original root color that is to migrate into the new hair color. Shadows

[0075] In one embodiment, the hair rendering engine 120 is enabled to apply two types of shadows. The first type of shadows are those that the head and strands of hair cast on the hair model itself. The second type of shadows are those that the hair casts on the head. For both shadows, two types of depth maps are rendered from the perspective of the light sources: one depth map is that of the entire hair and the head; and one depth map is that of the hair only. These depth maps are used to apply shadow effects to the hair and the head respectively. The depth value of each pixel is compared to that of the corresponding depth value on the depth map; if the pixel's depth is greater than the depth map value, the pixel is considered to be in shadow.

[0076] For hair, the shadow value is used to determine if the current pixel on the hair will be in shadow. In response to it being in shadow, the shadow color is blended into the final hair color. For the head, the shadow color is applied to the pixel in shadow and rendered onto a blank image. As each pixel of the head is rendered, the shadow appears in the shape of the head in the blank image.

[0077] To address the existence of aliasing artifacts for hair shadows, a standard technique called Percentage Proximity Filter (PCF) is employed, where a pixel that is in shadow is determined by the average of its neighbors. Rather than depending on a binary yes / no outcome for "in shadow" or "not in shadow" as determined by a depth comparison of the current pixel against the depth map, neighboring pixels are also respectively compared using the depth map and a weighted average (e.g., between 0 and 1) is determined. For example, a pixel may be considered 40% in shadow. For head shadows, a Gaussian blur is applied to the rendering of the shadows that were applied to the blank image, which smooths out the shadows at the edges. Finally, these blurred shadows are rendered onto the head. Physical Simulation Simulation Overview

[0078] This section explains in broad outline the technique used to simulate hair movements as they appear in real life, for example in through the operations of the physics simulation component 126. In one embodiment, to achieve this, a programmatic frame is incorporated into the hair mesh to guide the movement of the hair strands during rendering. The frame includes several guide strands, which are distributed throughout the hair mesh model. In one embodiment, the guide strands are placed by the hairstyle designer, and are chosen to give the most realistic physical behavior for that hairstyle. For example, 10 to 20 relatively uniformly distributed guided strands are defined. Like other hair strands (e.g., non-guiding), these guide strands start at the hairline and grow following the shape of the hair mesh. Each hair strand in the hair mesh is mapped to these guide strands in order to follow the movements of the guide strands. [Fig.4A] shows an image 400 having hair 402 and a representative guide strand 404 for a hair mesh (not shown).

[0079] The guide strands comprise particles (e.g., 404A, 404B, 404C, and 404D) that are connected to each other by line segments (e.g., 404E). The particles in the guide strands are separated into two categories: anchored particles (shown as dotted lines such as 404A and 404B) and unanchored particles (shown as solid lines such as 404C and 404D). An anchored particle is one that only follows the position of the head, so that it appears "anchored" to the head, while unanchored particles are free to move. Using these properties, it is possible to define which portion of the hair should remain close to the head and which portion should be free.

[0080] In one embodiment, the movement of the non-guiding hair strands is determined relative to the movement of the associated guided hair strands. The operations associate each non-guided hair strand with two nearest guided hair strands.

[0081] For each hair strand, a representative vertex on the hair strand at a location that is 3 / 4 (75%) along the hair strand from the root to the tip is determined. The number 3 / 4 is empirically determined to be farther from the root because it has been determined that the guide strands are close together toward the root and may cause inaccurate matching if a vertex closer to the root was chosen. The length can be between 50 and 80%, with a preferred length of 75% (i.e., 3 / 4). The two closest guide strands to serve as references are located for each particular hair strand. For example, the physical distance (e.g., Euclidean distance) to each of the guide strands is determined and the two closest are determined using these distances. Each vertex on the non-guiding strand of hair is matched to a corresponding segment on each of the two guiding strands. This is also done by finding the shortest distance. During the physical computation, the vertex movements (of a non-guiding strand) will be influenced by the line segments assigned from the two nearest guiding strands, with the nearest guiding strand having more influence than the farthest guiding strand.

[0082] In one embodiment, the hair simulation avoids purely uniform hair motion, as this is not what occurs in real life. Each particle in the guide strand is also assigned a configurable mass parameter, which affects the acceleration it experiences in the physical calculation. This mass is also influenced by the number of non-guiding hair strands assigned to it, with more strands adding to the mass of each of the particles in that guide strand. In accordance with the assignment of non-guiding strands to guide strands, tufts of hair are defined, with the size of a tuft being related to the number of co-assigned hair strands, thus defining tufts of different sizes. The tufts move differently depending on the amount of hair contained in those tufts.Independent movement of individual parts of the hair is achieved, which is closer to what we see in reality. Simulation forces

[0083] In one embodiment, the hair physics simulation is stabilized. A reference strand for each guide strand is defined to help stabilize the physical rendering. A reference strand is a strand of unanchored particles that only reacts to simulated forces such as gravity and centrifugal force. Each reference strand defines the movement and bending of the strand as a whole, and the corresponding guide strand would move around it. In one embodiment, initially, each reference strand is defined to have the same position as its associated guide strands. In one embodiment, for gravity, it is assumed to be an acceleration of 9.8 m / sA2 in the downward direction. In one embodiment, the downward direction would be the same as the actual downward direction (which is possible if the orientation of the camera / 3D device is known).Otherwise, the downward direction is relative to the camera / device (e.g., if the camera / device were tilted 45 degrees, the direction of gravity would also be tilted by the same angle relative to the actual downward direction). In one embodiment, the centrifugal force is calculated based on the average angular velocity of the head, estimated from the change in head rotation over time. In one embodiment, . head rotation is estimated using the vector from the nose to the center of the head. In one embodiment, the direction of the centrifugal force is from the center of the head to the end of the beam, but only the perpendicular (to the beam) component of the force is considered.

[0084] In one embodiment, the bending of each reference strand is modeled as deflections on straight bendable cantilever beams, with one end attached to an anchored particle and the other end free. Two beams are modeled for each reference strand, one to model bending due to gravity and one to model bending due to centrifugal force. The beams are constructed from the final anchored particle of the guide strand (reference strand particles are unanchored particles that come after this anchored particle), and the reference strand particles are then associated with the beam deflection at some point on the beam. Each cantilever beam is configured to be straight, starting with the anchored particle closest to the root.

[0085] For the beam that models bending due to gravity, the free end of the beam is located directly below the anchored end which is twice the length halfway down the reference strand, so that the initial shape of the guide strands and hair strands is preserved when the head is at its initial orientation. That is, the length of the beam is twice the distance from the midway top of the reference strand to the anchored end. The initial orientation is assumed to be untilted. The beam deflection due to gravity is calculated as if the beam experiences a constant gravitational force along the entire beam, with the beam having physical properties approximated by 30 hair strands tightly packed along the beam.

[0086] For the beam that models bending due to centrifugal force, this is a beam that is twice the length of a beam that extends from the anchor to the center point of all unanchored particles in the reference strand. The beam deflection due to centrifugal force is calculated as if the beam were experiencing a single centrifugal force at the end of the beam, with the beam having physical properties approximated by 30 tightly packed strands of hair along the beam. The single centrifugal force is used as an approximation to facilitate calculations.

[0087] After calculating the deflections on the two beams due to gravity and centrifugal force, the unanchored particles in the reference strands are then moved from their respective original positions based on the addition of the two deflections on the two beams. For each unanchored particle, its motion due to the deflection of a beam is calculated from the deflection at the point on the beam where the original position of the unanchored particle is projected onto the beam. [Fig.4B] is a representation of the simulation on an inclined face 450. The thick black line 452 represents a beam 452. The downward arrows (e.g., 454) represent forces. A reference strand is shown in two positions, namely as strand 456A, before application of forces, and as strand 456B, after application.

[0088] In one embodiment, in order for the guide strands to move around their respective reference strands, modeled springs are present between the guide strands and the respective reference strands and dampers on the guide strand particles. In one embodiment, multiple iterations of Verlet integration calculations are performed per frame to infer the next position of the guide strand particles due to the forces acting on them by the springs and dampers. In one embodiment, a position modifier is added that moves each of the guide strand particles by the head movement but only along the direction of the strand to make the strand non-extensible and non-compressible.

[0089] Once the guided strands are moved, the tufts / strands of hair associated with a respective guided strand are moved, for example, by applying the influence of the two guided strands associated with a non-guided strand of hair. In one embodiment, each non-guiding strand has two closest guiding strands associated with it. In addition, each particle within the non-guiding strand also contains the closest curve position (e.g., the parametric value "t" of the curve) along the two closest guiding strand curves. A non-guiding strand is deformed by the guiding strands as follows: 1. For each particle in the non-guiding strand, a global closest position is calculated across the two guiding curves.For these closest global positions, the original global position (before physics is applied) is subtracted from them to give a displacement vector representing the change in position due to physics. The global position refers to the (x, y, z) position in global space, the space that contains all objects in the scene. 2. The global position of the non-guiding wick particle is set to its original position offset by the weighted average of the two displacement vectors. The weight is determined by the relative proximity of the non-guiding particle to the nearest guiding wicks, with the closest guiding wick receiving more weight.

[0090] In order to make the hair strands less uniformly distributed as they follow the guide strand, in one embodiment, each hair strand is moved toward the nearest or second nearest guide strand to achieve a clumping effect. In one embodiment, the amount of movement depends on one or more of the following parameters: - the speed of the guide strand; the higher the speed, the greater the clumping effect will be; - the distance between the hair strand particle and the root, where the further it is from the root, the greater the clumping effect; - a user-selected amount for the clumping effect that affects all hair strands; - the number of hair strands that are closest to the guide strands; the more hair strands there are around the guide strand, the less clumping effect movement towards that guide strand will be so that the resulting clumping of hair strands around that guide strand will appear larger;- the randomly generated size factor of the guide strand which affects the resulting clump size of the hair strands around the guide strand by affecting the clumping effect movement of the hair strands towards the guide strand (see point above); and - a randomly generated ratio which determines how much the hair strand should move towards the nearest guide strand relative to the second nearest guide strand. ; Collision management

[0091] In one embodiment, collision detection is determined for collisions between hair strands and objects that are convex hull meshes of triangles. To improve performance, in one embodiment, collision detection is approximated by precomputing the triangular position of the collision mesh corresponding to each corner with an approximate center of the convex hull. In one embodiment, each object in the scene (such as the head, shoulders, neck) is represented as a mesh composed of triangles, e.g., using known techniques. Hair strands are always represented as 3D points instead of triangles. Typically, collision detection algorithms only work with convex meshes. Intuitively, a convex mesh can be thought of as a mesh that has no "indentation" or holes.Since some of the above meshes are not convex, a convex approximation (called a convex hull) of these meshes is used during collision detection. During collision detection, hairline points are checked against triangles in the convex hull meshes. This approach is called "spherical approximation" in which the approximation causes particles to interact with the convex hull "approximately" as if it were a sphere.

[0092] In order to mitigate false positives and false negatives, a padding element is added during collision detection to make the computation more robust. For this To do this, we use a scaling matrix to enlarge the convex hull meshes.

[0093] In situations where a hair particle would collide with multiple objects in close proximity, in one embodiment, only the first object the particle came into contact with is recorded. This ensures that the particle does not jump back and forth due to simultaneous collisions.

[0094] When a hair particle is detected as colliding with another object, the particle is repositioned to be on the surface of the colliding object. A configurable damping factor for the particle is added to simulate friction on the object's surface. This damping factor is also applied to any hair particles in the same strand that are below that colliding particle (toward the free end), so that segments below it will not move as if it were free, which would appear unrealistic. Lighting Estimation

[0095] In one embodiment, the system uses a machine learning model to predict the environment map given by a single camera image. The environment map is an HDR panoramic image that models light from all directions. This map is transferred to the environment map lighting step in rendering as described above. A model may be trained using supervised learning to predict an HDR environment, e.g., trained using paired portrait image and HDR environment map data. Furthermore, there are known models used to estimate the lighting scene of images that can be classified into two main categories: regression models that estimate low-dimensional panorama lighting parameters and generative models that generate non-parametric illumination maps or light probes.With the growing popularity of generative models, there are also frame-by-frame generative models that directly generate an HDR environment map, such as styleLight. Previous work has explored ways to infer lighting information in real-time augmented reality applications, including that of LeGender et al., who first proposed models that predict low-resolution HDR light probes based on LDR images without the constraint of limited field of view, and later extended their work to focus on predicting light probes based on limited field-of-view portrait LDR images. These models are small models that can infer in real-time on mobile applications.However, because light probes have low resolution and only capture lighting information behind the camera, this limits realism when using light probes to render images. To address this problem, Somanath et al. presented a model that infers high-resolution HDR panorama maps based on unconstrained LDR images with limited field of view. The datasets are also publicly available. The generated map is provided for use as described above here as an illumination estimate of the original scene and from which shadows and other light parameters associated with light direction and / or color can be determined.

[0096] [Fig. 5] is a flowchart of operations in accordance with a respective embodiment. The operations are performed by a system, e.g., at least one computing device having at least one processor; and at least one memory device storing computer-readable instructions that, when executed by the at least one processor, cause the system to perform the operations. With reference to [Fig. 5], operations 500 show, at 502, processing an input image using a three-dimensional (3D) face detector array to determine a position, rotation, and 3D representation of a face, the input image comprising the face of a head having existing hair. At 504, removing the existing hair from the head. At 506, determining a 3D hairstyle model of a 3D hairstyle to be applied to the head having the existing hair removed.At 508, rendering the 3D hairstyle onto the head to define an output image. The rendering is responsive to one or more of: i) a physical simulation of hair movement and hair position as applied to the head, the physical simulation modeling at least one force and deforming the 3D hairstyle model in response; or ii) a hair color to recolor the hair of the 3D hairstyle model, the hair color being refined in accordance with an estimation of ambient light conditions. And at 510, providing the output image for display by a display device. In one embodiment, the modeling is responsive to at least some of face position, face rotation, 3D representation of the face.It will be understood that operations similar to operations 500, such as in one embodiment, may be performed when rendering using the 3D hairstyle model reacts to a physical simulation or hair coloring that changes the color of a 3D hairstyle model. Further, hair coloring, if performed, with or without physical simulation, may or may not react to ambient lighting conditions.

[0097] The system may, in one embodiment, comprise a laptop, smartphone, tablet, desktop computer, server, or other computing device.

[0098] [Fig.6] is an illustration of a computing environment 600 comprising one or more systems, in accordance with one embodiment, such as for practicing one or more process aspects in which operations are performed such as those of [Fig.5].

[0099] The computing environment 600 shows a user computing device 602 (e.g., a system), such as a smartphone, a communications network 604, a server 606, and a server 608. The communications network 604 includes wired and / or wireless networks, which may be public or private and may include, for example, the Internet. The server 606 includes a server computing device such as for providing a website. The server 608 includes a server computing device such as for providing e-commerce transaction services. Although shown separately, the servers 606 and 608 may comprise a single server device. The computing environment is simplified. For example, payment transaction gateways and other components such as for performing an e-commerce transaction are not shown.

[0100] The computing device 602 includes a storage device 610 (e.g., a non-transitory device such as memory and / or a solid-state disk, etc.) for storing instructions that, when executed by a processor (not shown) such as a central processing unit (CPU), a graphics processing unit (GPU), or both, cause the computing device 602 to perform operations such as a computer-implemented method. The storage device 610 stores a virtual try-on application 612 including components such as software modules providing a user interface 614, a VTO processing pipeline 616, a VTO recommendation component 618 with VTO data 618A, and a shopping component 622 with a shopping cart 624 (e.g., shopping data).In one embodiment, not shown, the VTO application lacks a recommendation component and / or a purchase component, e.g., providing a VTO selection component for selecting VTO options to be visualized as VTO effects. In one embodiment, the VTO application is a 3D hair VTO using physical simulation. The VTO data 618A may include hair color data (e.g., as one or more strands such as strand 300), associated product data, 3D hairstyle model data for trying on a 3D hairstyle, etc. This data may be obtained and / or stored in a database or other data store.

[0101] In one embodiment, the VTO application 612 is a web application as obtained from the server 606. In one embodiment, the VTO application 612 such as for a native application is provided by a content delivery network. The VTO data 618A may be obtained from a content management system 607, associated with the server 607A. The content management system Content 607 includes a data store 607B storing VTO-related data, e.g., strand data and rendering effects data, 3D hairstyle model data, etc. In one embodiment, the VTO data, particularly color-related data, is associated with real-world products. Such a VTO experience allows a user to simulate trying on a real-world product, e.g., using a desired 3D hairstyle. The VTO data (e.g., color, etc.) may be provided to a server in other ways, e.g., as color or other parameters. For example, sliders may provide input that is mapped to data values.In one embodiment, the VTO data may be provided to a user device (e.g., 602) by inclusion in a native application package and provided as updates to a native application such as by the server 606. The VTO data may thus be provided from various sources. The rendering effect data may include data for rendering an effect such as simulating a try-on property such as a matte appearance, a glossy appearance, etc. A UI 607C to the content management system 607 may be provided to a product provider such as a brand owner, to upload VTO data (e.g., hair strand images) and provide input such as to define hair strand data, product data and / or rendering effect data, 3D hairstyle model data, etc. In one embodiment, the UI 607B is web-based.

[0102] Although not shown, the user device 602 may store a web browser for execution of the web-based VTO application 612. In one embodiment (not shown), the VTO application 612 is a native application in accordance with an operating system (also not shown) and software development requirements that may be imposed by a hardware manufacturer, e.g., of the computing device 602. The native application may be configured for web-based or similar communications with the servers 606, 607A, and / or 608, as is known.

[0103] [Fig. 6] shows various input and output data or information associated with a use of the VTO application 612, for example. These input and output data include an input image 626 of the user to be processed for a VTO experience, an output image 628 on which the product effects are simulated providing a VTO experience, a VTO selection 620 comprising, in one embodiment, a user input selecting one or more color effects to be simulated and a selected 3D hairstyle on which the color(s) are rendered, the VTO options 622 comprising color and hairstyle options to be virtually tried on, for example for selection by a user of the device 602, and the purchase transaction information 624 comprising the purchase information provided to and / or received from a user to purchase a product. As noted, not all VTO application embodiments include e-commerce capabilities. Although some embodiments refer to a strand as a format for providing a color to be applied, alternatively or in addition, the color may be identified in other ways. For example, the color may be determined from the user's existing hair in the input image. The user may provide a second input image—for example, a color identification image, which may include a face with hair—and the VTO application may process the color identification image to determine the hair and extract one or more colors to be used to color the 3D hairstyle to be rendered.Color options may be provided in other ways such as a color chart presenting a set of colors, a color value input interface (e.g., text) providing values ​​according to a color representation scheme, etc.

[0104] In one embodiment, via one or more user interfaces 614, VTO product options 622 are presented for selection to be virtually tried on by simulating effects on an input image 626. In one embodiment, the VTO options 622 are derived from or associated with VTO data 618A which may be product data. In one embodiment, the VTO data (e.g., product data) may be obtained from the server 606 and provided by the VTO recommendation component 618, which in one embodiment may be a product data parser where user-based recommendations per se are not made. Instead, all available product data is made available for selection for use. Although not shown, user or other input may be received for use in determining the VTO recommendations.The user may be prompted, for example via one of the interfaces 614, to provide input to determine recommendations. In one embodiment, the VTO recommendation component 618 communicates with the server 606. The server 606, in one embodiment, determines the recommendation based on the input received via the component 618 and provides VTO data accordingly. The user interface 614 may present the VTO options 622, for example, by updating the display of the options, in response to the received data as the user navigates or otherwise interacts with the user interface.

[0105] In one embodiment, the one or more user interfaces provide instructions and commands for obtaining the input image 626, and the VTO selection input 620 such as an identification of one or more VTO products recommended to try. In one embodiment, the input image 326 is an image of a user's face (e.g., a portrait) typically having hair, which may be a still image or a frame of a video. In one embodiment, the input image 626 may be received from a camera (not shown) of the device 602 or from a stored image (not shown). The input image 626 is provided to the VTO processing pipeline 616 such as for processing in accordance with the operations of [Fig. 5] and the components shown in the device 102 of [Fig. 1] to produce an output image 628 for the VTO. In one embodiment, the VTO selection input 620 includes an input for selecting strands of hair from the 3D hairstyle model with which to define groups for coloring.More than one color can be selected to be tried at the same time, applied to different groups or to different places in the same group for horizontal or vertical coloring effects, for example.

[0106] It is understood that the output image 628 may comprise a frame of a video sequence. The user interfaces 614 provide the output image 628. The output image 628, in one embodiment, is presented as a portion of a live stream of successive output images (each 628 for example), such as when a selfie video is augmented to present an augmented reality experience. In one embodiment, the output image 628 may be presented with the input image 626, such as in a side-by-side display for comparison purposes. In one embodiment, this may be an "in-place" before / after comparison interface where the user moves a slider to reveal more of the initial or processed image. In one embodiment, the output image 628 may be saved (not shown) such as on the storage device 610 and / or shared (not shown) with another computing device.

[0107] In one embodiment, the input images (not shown) comprise input images of a video conference session and the output images comprise video shared with another (or more) participant(s) of a video conference session. In one embodiment, the VTO application (which may have another name) is a component or plug-in of a video conference application (not shown) allowing the user of the device 602 to present VTO results (e.g., a new hairstyle or color) during a video conference with one or more other conference participants.

[0108] It will be apparent to those skilled in the art that numerous aspects and features are disclosed by the embodiments herein. The following numbered statements relate to at least some of these aspects and features.

[0109] Statement 1: A system comprising: at least one processor; and at least one memory device storing computer-readable instructions which, when are executed by the at least one processor, cause the system to: process an input image using a three-dimensional (3D) face detector network to determine a 3D representation of a face of a head, the input image comprising the face and the head; determine a 3D hairstyle model of a 3D hairstyle to be applied to the input image; render the 3D hairstyle from the 3D hairstyle model to define an output image, wherein the rendering is responsive to one or more of: a physical simulation of hair movement and hair position of the 3D hairstyle to be rendered, the physical simulation responsive to the 3D representation of the face and the physical simulation modeling at least one force and deforming the 3D hairstyle in accordance with the at least one force; or a hair color to recolor the hair of the 3D hairstyle model, the hair color being refined in accordance with an estimate of ambient light conditions;and providing the output image for display by a display device. ;

[0110] Statement 2: The system of Statement 1, wherein the computer-readable instructions that, when executed by the at least one processor, cause the system to provide one or more interfaces for at least: receiving input identifying one or more of the 3D hairstyle or hair color for rendering; recommending VTO options including one or more of the 3D hairstyles or hair colors to be selected for identification for rendering; and performing a transaction to purchase a product associated with the 3D hairstyle or hair color.

[0111] Statement 3: The system of Statement 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to represent the head as a 3D representation on which the 3D hairstyle is rendered.

[0112] Statement 4: The system of Statement 3, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to remove existing hair from the head; and wherein the rendering renders the head with the existing hair removed.

[0113] Statement 5: The system of Statement 4, wherein removing existing hair comprises: processing the input image using a hair segmentation network to provide a hair segmentation mask for the existing hair; and removing existing hair from the head using the hair segmentation mask.

[0114] Statement 6: The system of Statement 4, wherein removing existing hair comprises retouching a scalp portion of the head using a 3D scalp filter to reconstruct a shape of the head, the 3D scalp filter being responsive to at least some of a facial position, a face rotation and 3D representation of the face and head; and in which the 3D hairstyle is to be applied to the reconstructed head.

[0115] Statement 7: A system according to Statement 6, wherein the scalp retouching uses a skin color determined from the input image.

[0116] Statement 8: The system of Statement 7, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to downsample and dilate an area of ​​existing hair detected by the hair segmentation network to remove existing hair.

[0117] Statement 9: The system of Statement 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to process the input image to determine at least one of a face position or a face rotation; and wherein the physical simulation is responsive to at least one of the face position or the face rotation.

[0118] Statement 10: The system of Statement 9, wherein the computer-readable instructions that, when executed by the at least one processor, cause the system to define a collision 3D shape representation for at least some objects of the input image, the at least some objects being selected from a head, a neck, and a shoulder; and wherein the physical simulation of hair movement and hair position responds to one or more collisions of hair strands with the collision 3D shape representation.

[0119] Statement 11: The system of Statement 10, wherein the collision 3D shape representation is responsive to at least some of face position, face rotation, 3D representation of the face, or a 3D representation of hair according to the 3D hairstyle model.

[0120] Statement 12: The system of Statement 9, wherein the 3D hairstyle model models a plurality of hair strands defining the hairstyle, the model including an embedded frame for guiding movement of the hair strands, the frame including a set of guided strands distributed throughout the model, each hair strand being matched to at least one of the guided strands, each individual hair strand and each individual guided strand extending from a respective individual hair root outwardly in accordance with a shape of the hair model; and wherein the physical simulation determines hair movement and hair position of the guided strands according to the at least one force and determines hair movement and hair position of the hair strands according to the match to the guided strands.

[0121] Statement 13: A system according to Statement 12, wherein each guided wick comprises a plurality of particles bonded to each other by segments of line, the particles comprising i) anchored particles that follow only a movement or position of the head and ii) unanchored particles, the movement or position of which responds to at least some of: the position of the face, the rotation of the face, the 3D representation of the face, or at least one force.

[0122] Statement 14: The system of Statement 13, wherein the computer-readable instructions that, when executed by the at least one processor, cause the system to match each hair strand with two nearest guided strands, wherein the two nearest guided strands for a particular hair strand are determined in response to an evaluation of the physical distance between a vertex of the hair strand and a vertex of the guided strand, and wherein the vertex of the hair strand is between 50% and 80% or preferably 75% of the distance along the hair strand and the vertex of the guided strand is between 50% and 80% or preferably 75% along the guided strand.

[0123] Statement 15: The system of Statement 14, wherein, for a particular strand of hair, matching comprises matching each vertex on the particular strand of hair to a respective corresponding segment on each of the two nearest guided strands.

[0124] Statement 16: A system according to Statement 13, wherein each of the particles in a guided wick is associated with a respective mass parameter and wherein an acceleration of any one of the particles determined by the physical simulation responds to the respective mass parameter of one of the particles.

[0125] Statement 17: A system according to Statement 16, wherein a mass of the guided strand further responds to the number of hair strands matched to the guided strand.

[0126] Statement 18: The system of Statement 13, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to: define a plurality of reference strands comprising unanchored particles subjected to motion by at least one force; assign each guided strand to a respective one of the reference strands; for each respective reference strand: model the at least one force; and apply the at least one modeled force to the respective reference strand, moving the unanchored particles in response; and move the respective guided strands about the respective reference strands.

[0127] Statement 19: The system of Statement 18, wherein the at least one force comprises one or both of a gravitational or centrifugal force.

[0128] Statement 20: The system of Statement 18, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to model each force of the at least one force as deflections on a respective bendable straight cantilever beam, wherein a fixed end of the beam begins at a location of an anchored particle closest to the root of the associated guided strand.

[0129] Statement 21: The system of Statement 20, wherein the at least one force comprises two or more forces, and wherein applying the at least one force comprises moving the unanchored particles according to the addition of deflections of the two or more forces.

[0130] Statement 22: The system of Statement 18, wherein the at least one force comprises two or more forces, and wherein applying the at least one force comprises moving the unanchored particles according to the addition of deflections of the two or more forces.

[0131] Statement 23: The system of Statement 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to process the input image to determine a 3D environmental map defining one or more light sources estimating ambient light to refine hair color.

[0132] Statement 24: The system of Statement 23, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to process the input image using a deep neural network to define the 3D environmental map.

[0133] Statement 25: A system according to Statement 23, wherein the environmental map is a panoramic HDR image that models light from all directions.

[0134] Statement 26: The system of Statement 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to render in accordance with a Marschner hair shading model.

[0135] Statement 27: The system of Statement 26, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to render in accordance with the Marschner hair shading model adapted to operate in real time, using at least some pre-computed diffusion functions for different angle inputs.

[0136] Statement 28: The system of Statement 26, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to perform an approximation of a mathematical integration of a scattering equation on all light directions by performing simplified operations using a plurality of discrete light sources determined from a 3D environmental map generated from the input image that estimates the plurality of discrete light sources.

[0137] Statement 29: The system of Statement 28, wherein the computer-readable instructions that, when executed by the at least one processor, cause the system to process the input image to determine a 3D environmental map defining one or more light sources estimating ambient light; process the map to detect the brightest points on the map, find the average color for each point to determine the color and intensity of light, and convert each point to a directional light source.

[0138] Statement 30: The system of Statement 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to render in accordance with an indirect lighting simulation, approximating the aggregate behavior of light rays using statistical models in accordance with a double scattering technique.

[0139] Statement 31: The system of Statement 30, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to adapt to operate in real time using pre-computed recolorable double diffusion lookup tables having an additional color dimension for a hair color channel.

[0140] Statement 32: The system of Statement 1, wherein the computer-readable instructions that, when executed by the at least one processor, cause the system to define a rendered pixel depth map for pixels rendered for the output image, initialize the depth map in response to pixel depths for the face, neck, and at least one shoulder of the input image, and update the depth map as pixels are rendered on the output image; and use the rendered pixel depth map to determine hair pixel occlusion.

[0141] Statement 33: The system of Statement 32, wherein the computer-readable instructions that, when executed by the at least one processor, cause the system to skip rendering a hair pixel at a location in the output image in response to an evaluation of a depth value of the hair pixel and a depth value from the rendered pixel depth map at the location of the hair pixel, the evaluation indicating that the hair pixel is occluded.

[0142] Statement 34: A system according to Statement 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to define the 3D hairstyle model as a hair map model using 3D rectangular planes; and to skip rendering a transparent hair pixel on the output image defined from the 3D hairstyle model.

[0143] Statement 35: A system according to Statement 1, wherein the 3D hairstyle model comprises: a hair map model defined using 3D rectangular planes; or a hair curve model defined using a set of Bézier curves, each curve representing a strand of hair.

[0144] Statement 36: The system of Statement 1, wherein the 3D hairstyle model comprises a hair curve model defined using a set of Bézier curves, each curve representing a strand of hair, and wherein the 3D hairstyle model is defined or stored using one or more optimization techniques including: representing at least some of the Bézier curves as a quadratic curve shape rather than a cubic curve where the quadratic curve shape approximates the cubic curve shape; storing only one of the two sides of a symmetrical hairstyle, producing a mirror image of the other side when the symmetrical hairstyle is loaded; or storing curve data in a compressed binary format with a lossless compression technique.

[0145] Statement 37: The system of Statement 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to: receive a strand of hair representing a desired hair coloring result, the strand comprising a color distribution.

[0146] Statement 38: The system of Statement 37, wherein the computer-readable instructions, when executed by the at least one processor, cause the system to transfer the color distribution of the strand to a hair texture of the 3D hairstyle model using a histogram matching process.

[0147] Statement 39: The system of Statement 38, wherein the computer-readable instructions that, when executed by the at least one processor, cause the system to provide control of a variation of hair color, to control a frequency of variation by scaling the hair texture to reduce detail, or to control a scale of variation by adjusting a hair texture mapping of the hair texture to a hair surface, which mapping compresses or stretches the hair texture relative to the hair surface.

[0148] Statement 40: The system of Statement 38, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to provide a hair root coloring control to control one or both of a color fade or a color choice.

[0149] Statement 41: The system of Statement 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to soften the pixelation of edges of loose hair strands from the 3D hairstyle model by applying blurring or blending or both blurring and blending.

[0150] Statement 42: The system of Statement 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to reduce the pixelation of all hair strands from the 3D hairstyle model using a multi-sample anti-aliasing (MSAA) technique.

[0151] Statement 43: The system of Statement 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to render hair at a higher resolution than the output image and miniaturize to reduce the size of the strands and provide smoothing.

[0152] Statement 44: The system of Statement 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to apply two or more hair colors to respective hair segments from the 3D hairstyle model, wherein the hair segments are arranged horizontally, vertically, or as a root recolor.

[0153] Statement 45: The system of Statement 44, wherein the 3D hairstyle model segments the hair strands into multiple groups, and computer-readable instructions that, when executed by the at least one processor, cause the system to apply the two or more hair colors to at least some of the multiple groups.

[0154] Statement 46: The system of Statement 45, wherein the respective hair segments are grouped to define non-contiguous tufts of hair strands and the computer-readable instructions which, when executed by the at least one processor, cause the system to at least: apply a first color to a majority of the strands and a second color to a group of strands at the front of the head, creating a face-framing effect; or apply a first color to a majority of tufts of strands throughout the hair and a second color to second tufts of strands distributed throughout the hair, creating a streaked highlighted effect.

[0155] Statement 47: The system of Statement 45, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to smooth the color transition, adjust the colors of adjacent hair strands that do not belong to the same group, and blend the color of the different groups to produce a visually transitional color for the adjacent strands.

[0156] Statement 48: The system of Statement 44, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to: segment hair into multiple groups interactively in accordance with user input to identify the multiple groups; and apply the two or more hair colors to at least some of the multiple groups.

[0157] Statement 49: The system of Statement 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to render shadow pixels simulating the shadow cast by the hair from the hair model, the shadow pixels responsive to the estimation of ambient light conditions.

[0158] Statement 50: The system of Statement 49, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to: define a first shadow depth map and a second shadow depth map, each defined from the perspective of one or more light sources determined from the estimation of ambient light conditions, the first shadow depth map being defined for the hair and the head and the second shadow depth being defined for the hair only;and applying shadow effects to the hair and head respectively, by evaluating, for each of the shadow depth maps, a depth value of a pixel against that of a corresponding depth value in the shadow depth maps and if the depth value of the pixel is greater than the depth value in the depth maps, then the pixel is considered a shadow pixel. ;

[0159] Statement 51: The system of Statement 49, wherein for a shadow pixel cast onto a hair pixel, computer-readable instructions that, when executed by the at least one processor, cause the system to blend a shadow color of the shadow pixel with a final hair color of the hair pixel onto which the shadow is cast.

[0160] Statement 52: System according to Statement 51, in which the shadow pixel is determined by the average of its neighbors.

[0161] Statement 53: The system of Statement 49, wherein for a shadow pixel projected onto a head pixel, computer-readable instructions which, when executed by the at least one processor, cause the system to apply a shadow color of the shadow pixel in a head shape on a blank image.

[0162] Statement 54: The system of Statement 53, wherein for a shadow pixel cast on a head pixel, the computer-readable instructions which, when executed by the at least one processor, cause the system to apply a blur Gaussian to the blank image to smooth the shadow pixels on the edges, and render the smoothed shadow pixels on the head in the output image.

[0163] Statement 55: A computer-implemented method comprising: obtaining a 3D hairstyle model of a 3D hairstyle to be rendered on the input image to produce an output image, the 3D hairstyle model comprising a plurality of hair strands; responsive to a 3D representation of i) a face, or ii) a face and a head in the input image, simulating and applying at least one force to deform at least some of the hair strands; and rendering the output image for display using the deformed 3D hairstyle model.

[0164] Statement 56: A method according to Statement 55, comprising determining the 3D representation of i) the face, or ii) the face and head in the input image using one or more deep neural networks.

[0165] Statement 57: A method according to Statement 55, comprising applying a hair color to the deformed 3D hairstyle model, the application responding to ambient light conditions determined from the input image.

[0166] Statement 58: Method according to Statement 57: comprising determining a 3D environmental map defining one or more light sources to estimate ambient light conditions to refine hair color.

[0167] Statement 59: The method of Statement 57, wherein the rendering is responsive to one or both of a hair shading model or a light scattering model to refine the hair color.

[0168] Statement 60: A method according to Statement 55, comprising modeling one or more collisions of the hair strands with the face or the face and the head and wherein the rendering reacts to the one or more collisions.

[0169] Statement 61: A method according to Statement 55, comprising modeling one or more occlusions of the hair strands with the face or the face and the head according to a pixel depth map and wherein rendering is responsive to the one or more occlusions.

[0170] Statement 62: A method according to Statement 55, comprising modeling one or more shadows and wherein rendering is responsive to the one or more shadows.

[0171] Statement 63: A method according to Statement 55, wherein the 3D hairstyle model defines the hair strands into two or more groups and wherein the method comprises applying two or more hair colors to the hair strands, wherein at least one of the groups is colored a different color than another or other groups.

[0172] Statement 64: A method according to Statement 55, wherein the 3D hairstyle model defines the hair strands into a plurality of hair segments and wherein the method comprises applying a hair color in accordance to hair segments to apply hair color horizontally, vertically or for root coloring.

[0173] Statement 65: A system comprising: at least one processor; and at least one memory device storing computer-readable instructions that, when executed by the at least one processor, cause the system to: process an input image using a three-dimensional (3D) face detector array to determine a 3D representation of a face of a head, the input image comprising the face and the head; determine a 3D hairstyle model of a 3D hairstyle to be applied to the input image; render the 3D hairstyle from the 3D hairstyle model in response to the 3D representation of the face to define an output image, wherein the rendering applies a hair color to recolor the hair of the 3D hairstyle model, the hair color being refined in accordance with an estimate of ambient light conditions; and provide the output image for display by a display device.

[0174] Statement 66: The system of Statement 65, wherein the computer-readable instructions that, when executed by the at least one processor, cause the system to perform a physical simulation of the hair movement and hair position of the 3D hairstyle to be rendered, the physical simulation being responsive to the 3D representation of the face and the physical simulation modeling at least one force and deforming the 3D hairstyle in accordance with the at least one force; and wherein the rendering is responsive to the deformed 3D hairstyle.

[0175] The features of any of Statements 2 through 54 may apply to either or both of Statements 65 or 66, with modifications as necessary.

[0176] A practical implementation may include some or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, apparatus, systems, means for performing functions, program products, and other ways combining the features described herein. A number of embodiments have been described. Nevertheless, it is understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided, or steps may be eliminated, from the described process, and other components may be added to or removed from the described systems. Accordingly, other embodiments fall within the scope of the claims that follow.

[0177] Throughout the description and claims of this patent specification, the terms "include", "contain" and variations thereof mean "including but not limited to" and are not intended to (and do not) exclude other components, integers, or steps. References to an operation or component involving "one or more" thing(s) as presented do not require that a component or operation be configured to process a plurality of thing(s) as presented, but include components or operations configured to process a single thing or one of the things.

[0178] Features, integers, characteristics or groups described in conjunction with a particular aspect, embodiment or example of the invention are to be understood as being applicable to any other aspect, embodiment or example, unless inconsistent therewith. Any features disclosed herein (including any accompanying claims, abstracts and drawings), and / or any steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of these features and / or steps are mutually exclusive. The invention is not limited to the details of the foregoing examples or embodiments.The invention extends to any new feature, or any new combination, of the features disclosed in this patent specification (including any accompanying claims, abstracts and drawings) or to any new step, or any new combination, of the steps of any disclosed method or process.

Claims

Claims

1. A three-dimensional (3D) virtual hair try-on system comprising: - at least one processor; and - at least one memory device storing computer-readable instructions which, when executed by the at least one processor, cause the system to: • process an input image using a three-dimensional (3D) face detector array to determine a 3D representation of a face of a head, the input image comprising the face and the head; • determine a 3D hairstyle model of a 3D hairstyle to be applied to the input image; • render the 3D hairstyle from the 3D hairstyle model in response to the 3D representation of the face to define an output image, wherein the rendering applies a hair color to recolor the hair of the 3D hairstyle model, the hair color being refined in accordance with an estimation of ambient light conditions;and • provide the output image for display by a display device.;

2. The system of claim 1, wherein the computer-readable instructions, when executed by the at least one processor, cause the system to perform a physical simulation of the hair movement and hair position of the 3D hairstyle to be rendered, the physical simulation being responsive to the 3D representation of the face and the physical simulation modeling at least one force and deforming the 3D hairstyle in accordance with the at least one force; and wherein the rendering is responsive to the deformed 3D hairstyle.

3. The system of claim 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to provide one or more interfaces for at least: - receiving input identifying one or more of the 3D hairstyle or hair color for rendering; - recommend VTO options including one or more 3D hairstyles or hair colors to be selected for identification for rendering; - complete a transaction to purchase a product associated with the 3D hairstyle or hair color.

4. The system of claim 1, wherein the computer-readable instructions, when executed by the at least one processor, cause the system to process the input image to determine a 3D environmental map defining one or more light sources estimating ambient light to refine hair color.

5. The system of claim 1, wherein the computer-readable instructions, when executed by the at least one processor, cause the system to render in accordance with a Marschner hair shading model.

6. The system of claim 1, wherein the computer-readable instructions, when executed by the at least one processor, cause the system to render in accordance with an indirect lighting simulation, approximating the aggregate behavior of light rays using statistical models in accordance with a double scattering technique.

7. The system of claim 1, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to: receive a strand of hair representing a desired hair coloring result, the strand comprising a color distribution.

8. The system of claim 1, wherein the computer-readable instructions, when executed by the at least one processor, cause the system to render shadow pixels simulating the shadow cast by the hair from the hair model, the shadow pixels being responsive to the estimation of ambient light conditions.

9. The system of claim 8, wherein the computer-readable instructions which, when executed by the at least one processor, cause the system to: define a first shadow depth map and a second shadow depth map, each defined from the perspective of one or more light sources determined from estimating ambient light conditions, wherein the first shadow depth map is defined for the hair and the head and the second shadow depth is defined for the hair only; and applying the shading to the hair and the head respectively, by evaluating, for each of the shadow depth maps, a depth value of a pixel compared to that of a corresponding depth value in the shadow depth maps and if the depth value of the pixel is greater than the depth value in the depth maps, then the pixel is considered a shadow pixel.

10. The system of claim 1, wherein the computer-readable instructions, when executed by the at least one processor, cause the system to: i) smooth pixelation of edges of loose hair strands of the 3D hairstyle model by applying blurring or blending or both blurring and blending; ii) reduce pixelation of all hair strands of the 3D hairstyle model by using a multi-sample anti-aliasing (MSAA) technique; or render the hair at a higher resolution than the output image and miniaturize to reduce the size of the strands and provide smoothing.

Images