3D VIRTUAL HAIR TRY-ON METHOD AND SYSTEM

Abstract

METHOD AND SYSTEM FOR VIRTUAL 3D HAIR TRY-ON The methods, systems, devices, and techniques described herein aim to provide a new approach to virtual hair try-on by replacing the user's natural hair with a 3D representation (e.g., mesh, point cloud, implicit function, etc.) that mimics the behavior of real hair. In one embodiment, a physical simulation of at least one force deforms the 3D hairstyle. In another embodiment, the 3D representation includes details and texture information independently of color to allow for accurate calculation of the resulting color. In yet another embodiment, the hair color is applied in response to estimated ambient lighting conditions.The 3D representation can be segmented into groups of hair strands, allowing for vertical recoloring such as highlights, contouring, or balayage (a highlighting technique applied to real hair by retouching the hair color to create a graduated and more natural effect). Figure for the abbreviation: none.

Description

Title of the invention: METHOD AND SYSTEM FOR VIRTUAL 3D HAIR TRY-ON FIELD OF INVENTION

[0001] The present application relates to image processing, including the application of an effect to an object in an input image to define an output image, and more particularly the application of a three-dimensional (3D) effect to hair and including a method and a virtual try-on (VTO) system for 3D hair. CONTEXT

[0002] Realistic hair recoloring in the context of real-time VTO is a difficult subject due to the organic nature of hair and the large amount of detail required for good realism. “Real-time” in this context refers to a VTO experience where live video, such as a self-portrait video capture by a smartphone, tablet, or webcam (these examples are not limiting), is processed and an effect is applied in real time to simulate a product or service effect, such as a hair coloring effect, on the live video display. Traditional techniques usually involve detecting the hair area in the image (hair segmentation) and recoloring the pixels using color matching. While effective, this technique has some drawbacks; for example, it is highly dependent on the quality of the input.If the input lacks texture and detail (in the case of very dark hair, for example), these details are not reconstructed during recoloring, and the results appear poor or worse for any lighter shade tried, as these lighter shades usually have more texture and detail.

[0003] Another drawback is that consumers want to try different types of hair coloring, including highlights, balayage, contouring, and other recoloring styles. These types of effects treat the components or portions of hair differently, for example, by applying different colors to different sections. Traditional hair tracking and coloring give unsatisfactory results. Current technology makes it virtually impossible to reliably track the same strand of hair across a sequence of frames (as in a video stream).

[0004] Prior work for a 3D hair VTO includes patent references KR101997702B1 entitled "3D simulation System for hair-styling", issued on October 1, 2019 (published as KR20190052832A on May 17, 2019) and PCT / KR2008 / 005109 filed on September 1, 2008 (published as WO2010024486A1 on March 4, 2010). These References describe a 3D hair simulation system on a user in real time.

[0005] However, these patents focus on changing the user's hairstyle rather than applying an effect to an existing hairstyle. SUMMARY

[0006] The methods, systems, devices, and techniques according to the embodiments herein aim to provide a new approach to virtual hair trying-on by replacing the user's natural hair with a 3D representation (e.g., mesh, point cloud, implicit function, etc.) that mimics the behavior of real hair. In one embodiment, a physical simulation of at least one force deforms the 3D hairstyle. In another embodiment, the 3D representation includes details and texture information independently of color to allow for accurate calculation of the resulting color. In yet another embodiment, the hair color is applied in response to estimated ambient lighting conditions.The 3D representation can be segmented into groups of hair strands, allowing for vertical recoloring such as highlights, contouring, or balayage (a highlighting technique applied to real hair by retouching the hair color to create a graduated and more natural effect).

[0007] In one embodiment, user input is received to segment (i.e. group) the strands of hair according to the input, for example, allowing a user to select adjacent strands of hair and define multiple groups.

[0008] In accordance with the embodiments herein, the hairstyle of the input hair strand(s) is preserved and the hair color is changed to match a hair coloring product. As part of this hairstyle preservation, a machine learning model is used to infer the characteristics of the user's hair and select the closest matching hair model.

[0009] In accordance with the embodiments herein, a set of intuitive parameters is used to control color placement, brightness and intensity as well as the ability to use multiple hair colors simultaneously.

[0010] In accordance with the embodiments herein, a physical simulation approach is used in which the hair strands react in a physically plausible manner in response to the movements of the user's head, gravity, and collisions of the strands against the user's head and body.

[0011] To further improve realism, in accordance with the embodiments herein, a machine learning model is used to estimate the environment map of the user's environment, and the map is used to illuminate the 3D hair mesh in a way that matches the ambient lighting.

[0012] The following statements describe various aspects and features disclosed in the embodiments herein. These aspects and features, as well as others, will be readily understood by a person skilled in the art, particularly the aspects relating to computer program products. It is also understood that aspects / features of computer devices or systems may have corresponding process aspects / features, and vice versa.

[0013] Declaration 1: System comprising: at least one processor; and at least one memory device storing computer-readable instructions which, when executed by at least one processor, cause the system to: process an input image using a three-dimensional (3D) face detector array to determine a 3D representation of a face or head, the input image comprising the face and head; determine a 3D hairstyle model to be applied to the input image; render the 3D hairstyle from the 3D hairstyle model in response to the 3D representation of the face to define an output image, wherein the rendering applies a hair color to recolor the hair of the 3D hairstyle model, the hair color being refined according to an estimate of the ambient lighting conditions; and provide the output image for display by a display device.

[0014] Declaration 2: System according to Declaration 1, wherein the computer-readable instructions, when executed by at least one processor, cause the system to perform a physical simulation of the movement and position of the hair of the 3D hairstyle to be rendered, the physical simulation reacting to the 3D representation of the face and the physical simulation modeling at least one force and deforming the 3D hairstyle in accordance with the at least one force; and wherein the rendering responds to the deformed 3D hairstyle.

[0015] Declaration 3: System according to Declaration 1, wherein the computer-readable instructions, when executed by at least one processor, cause the system to provide one or more interfaces to at least: receive input identifying one or more of the 3D hairstyle or hair color for rendering; recommend VTO options including one or more of the 3D hairstyles or hair colors to be selected for identification for rendering; conduct a transaction to purchase a product associated with the 3D hairstyle or hair color. Brief description of the drawings

[0016] [Fig.1] The [Fig.1] is a schematic diagram of a system, according to an embodiment.

[0017] [Fig.2] The [Fig.2] is the representation of a graphical user interface, according to one embodiment.

[0018] [Fig.3] The [Fig.3] is an image of a colored strand, in greyscale for convenience and in accordance with one embodiment.

[0019] [Fig 4A-4B] Figures 4A and 4B are illustrations of components of a hair model, according to one embodiment.

[0020] [Fig.5] The [Fig.5] is an operational flowchart according to an embodiment.

[0021] [Fig. 6] [Fig. 6] is an illustration of a computer environment comprising one or more systems, according to an embodiment. DETAILED DESCRIPTION System overview

[0022] Figure 1 is a schematic diagram of a device 100 having one or more storage devices 102 storing a plurality of components (e.g., software and / or data components), according to one embodiment, which function to provide a 3D hair VTO application. The device 100 is typically a consumer device such as a smartphone, tablet, laptop, desktop computer, etc. Often, such types of devices have or are coupled to a camera to produce video comprising a sequence of video images / frames for use in a VTO application. In one embodiment, an input image 104A is processed to render an output image 104B. The image 104A can be a single "still" image (e.g., provided by a user (not shown)) or a frame extracted from a camera stream (not shown) or from an input video.The 104B output image is typically, but not necessarily, of the same type, meaning that the 104B image is either a corresponding still image or an image from a video stream for display, etc.

[0023] The image 104 is processed by a plurality of components including: a 3D face detector 106; a hair segmentation component 108, an environment map estimation component 110 and a hairstyle detector component 112.

[0024] The 3D face detector 106 operates to detect a 3D representation of a face, providing a face with a 3D representation (e.g., a mesh) and the associated data as described in more detail. Preferably, one or both of a face position and a face rotation are also determined. The hair segmentation component 108 detects where the hairs are located. on the image, providing a segmentation mask indicating hairs, not individual hair pixels. The environment map estimation component 110 estimates an environment map primarily related to lighting. The lighting environment map includes an HDR environment panorama (map) containing data (e.g., colored pixels) to locate the main directional light sources and estimate the ambient light color. Although a mesh is often described here as a type of 3D shape representation, other 3D shape representation methods can be used, such as point clouds, implicit functions, etc.

[0025] In this embodiment, the hairstyle detector component 112 detects a hairstyle in the image 104A and locates (e.g., performs a match search) a corresponding 3D style definition (e.g., as defined by a 3D artist) stored in the hairstyle data store 112A (e.g., a database 112A) for use as described in more detail. Although not shown in [Fig. 1], a non-matching hairstyle (e.g., a different style) that can be selected (by the user) from the store 112A can be used to replace the hair in the image 104A.

[0026] A plurality of rendering components (e.g., which define a rendering engine 120) process the outputs of components 106, 108, 110, and 112 / 112A. A hair removal filter component removes detected hair from image 104A in response to the hair mask and retouches the image. The result of the retouching technique is improved by applying a baldness filter to the head (i.e., a 3D mesh of a scalp oriented and positioned using the output of (2) and recolored with the user's skin tone). This baldness filter improves the rendering by masking the user's actual hair. In another embodiment, not shown, a baldness filter is not used.Instead, users with long hair or other hairstyles with a large volume of hair (e.g., spiky hair, among others) that interfere with a 3D hairstyle to be applied are advised to apply a tight ponytail or other tight hairstyle to the input image. This hairstyle should preferably conform to the shape of the user's head to conceal and / or minimize the user's hair (i.e., hair volume) to improve the rendering of the 3D hairstyle to be applied. Although not shown, in one embodiment, a hair detection function is provided to process the input image, for example, by evaluating a hair segmentation mask or using a hair volume classifier, to determine that a user should apply the hair concealment / minimization style to improve the results.In one embodiment, the user can refuse and proceed in all cases.

[0027] An occlusion detector component 124 determines occlusions and renders a depth map.

[0028] A physics simulation component 126 models the physics of hair for movement, gravity, etc., using guided strands to adjust the hair mesh that models the 3D hairstyle. From the face mesh and hair mesh data, an approximation of the user's head, neck, and shoulders is made to serve as a collision mesh. The collision mesh (e.g., a 3D collision shape representation) is used with a set of "guided strands" to compute the physics simulation of the hair's movement and position. The guided strands are used to deform the hair model (previously designed by a 3D artist).

[0029] A hair rendering engine component 128 renders the hair according to the deformed 3D model, etc. That is, the hair is recolored using advanced techniques and the rendering is further enhanced using the environment map. The output image 104B is provided, for example, sent to a display device such as a display screen for display. The output image 104B can be saved on a storage device or communicated (for example, shared) with another device, for example by email, on social networks, etc. Components Face and hair detection

[0030] In certain embodiments, the rendering engine 120 uses information about the 3D structure of the face in the form of a 3D mesh. To facilitate this, a 3D face mesh detector is used. In one embodiment, the 3D face mesh includes the frontal portion of the face, most of the forehead, and the sides of the face extending to the ears. Simulation accuracy can be optimized when the mesh provided by the detector also includes the scalp, neck, and shoulders, but these can be added to the mesh as a post-processing step if they are not available.

[0031] In some embodiments, the hair removal filter component 122 uses the 2D hair segmentation mask. To do this, a machine learning model can be used to estimate the segmentation mask given in an input image.

[0032] For the respective face and hair detectors 106 and 108, a "standard" solution or a custom-designed detector can be used. Here, "standard" refers to a pre-existing component that does not need to be entirely created from scratch. A certain degree of refinement can be used for the specific purpose(s) to be pursued, for example, training to provide the desired results or a desired shape.

[0033] of these. In one embodiment, with a custom detector, the machine learning models for 3D face mesh detection and 2D hair segmentation can be merged for optimal speed. Furthermore, the scalp, neck, and shoulders can be included in the mesh output, which is not typically the case in a "standard" 3D face detector. Hairstyle detection

[0034] In one embodiment, to preserve the user's hairstyle, a machine learning model is used on the hairstyle detector 112 to detect the user's hair characteristics by processing an image. In one embodiment, the detected characteristics include color (including any information about highlights or multiple colors), texture (afro, straight, curly, wavy), length, visibility (visible, covered, bald), and whether the hair is tied, braided, or has bangs.

[0035] In one embodiment, the features are used to choose the closest matching hairstyle in the 112A database of 3D hair models, as well as to adjust the physical parameters to better match the physical behavior of the style. Hair removal

[0036] In one embodiment, for greater realism, the user is virtually bald by the hair removal filter 112 before the new hair model is rendered over it. That is, the hair in the image is removed, for example by masking. In one embodiment, the filter 122 performs the following steps:

[0037] Removal-1: The hair segmentation mask is expanded to increase its coverage area.

[0038] Removal-2: A retouching technique according to a known technique (e.g., the fast marching method, Telea, 2003) is used to erase and reconstruct the erased region, using the mask from the previous step as the region. In one embodiment, as a performance optimization, the mask can be miniaturized before using the algorithm.

[0039] Removal-3: A scalp mesh is aligned with the face mesh if the face mesh does not include the scalp. If the face mesh already includes the scalp, the scalp can then be extracted from the face mesh.

[0040] Removal-4: The scalp mesh is rendered on the image, with skin material of a skin tone similar to the face using physically realistic rendering (PBR) techniques. In one embodiment, the skin tone is estimated by making the average of all facial skin pixels. In one embodiment, Poisson blending is also applied to smooth the transition between the original face and the rendered scalp.

[0041] In one embodiment, the existing hair is removed in a different way. For example, a GAN model (not shown) may be provided to process the input image and remove the hair. In such an embodiment, a hair segmentation mask may not be used. 3D Hair Rendering Hair pattern representation

[0042] In one embodiment, the system 100 supports two different hair model representations. The first is a hair map model where the hair is modeled using maps (3D rectangular planes) where each map contains a texture with hairs grouped together and a transparent background. The advantage of this representation is that it is efficient in terms of memory usage and simulation time, but it is less realistic because the system 100 does not have fine-grained control over individual strands. For example, the system 100 can only simulate the physics for the hair map as a whole and not the individual strands. Such a model is inefficient for hair with complex geometries, such as curly hair.

[0043] The second representation uses hair curves. The hair model consists of a set of Bézier curves, each curve representing a strand of hair. This representation offers greater realism and control over individual strands, but requires more memory and simulation time.

[0044] Memory-efficient hair curve representation

[0045] In one embodiment, to reduce both the storage and RAM requirements of the hair curve representation, a number of optimizations have been developed. Normally, cubic Bézier curves are used to model a strand of hair, which requires two control points per curve. As an optimization, quadratic Bézier curves (requiring one control point) are used instead when a hair curve can be sufficiently approximated using the quadratic Bézier curve, otherwise reverting to a cubic Bézier curve.

[0046] In addition, for symmetrical hairstyles, only one side of the hairstyle is recorded and the other side is produced as a mirror image when the hair model is loaded.

[0047] To reduce storage requirements, curve points can be stored in binary format and compressed using a standard lossless compression algorithm. Hair map direction editor

[0048] When using a hair map representation, a direction map texture is defined that specifies the 2D strand direction for each pixel on the hair map. The strand direction information is required by the shading model.

[0049] However, defining this direction map texture is difficult using the standard image editing tools available. In one embodiment, to facilitate this, a custom tool allows an artist or developer to define a direction map intuitively by manipulating a vector field using a mouse or touch controls. The vector field is displayed over the hair strand texture as a grid of arrows (e.g., vectors), and the user can drag their mouse or finger to align the arrows with the direction of the strands. Figure 2 shows a portion of a graphical user interface 200 (e.g., one that can be displayed on a display screen) showing a vector field 202 presented over a plurality of hair strands, e.g., hair strands from groups 204A, 204B, 204C, 204D, and 204E, as well as over a plurality of GUI controls 206.The vector field 200 comprises a grid of short vectors represented as short line segments (e.g., 206A), each with a direction. Each segment can be represented by an arrow construction, such as a line segment with a dot for compactness within the interface. In one embodiment, the dot at one end indicates the origin of the arrow, and the line curves from this point to show the direction. The direction of these vectors is manipulated using gesture input, such as dragging / swiping along the vectors with a pointing tool (e.g., a mouse, stylus, etc.) or a finger, etc. In one embodiment, the display screen is a touch input device. The map is defined from the values ​​of the vectors in field 202. The controls 206 are intended to define a brush (e.g., dimensions) that is actuated by gesture input (e.g., tool or finger, etc.).as well as to import the hair image. Reset the vector values ​​of field 202 or export the values ​​of field 202 to define the map. In one embodiment, the vector field is used to form a direction map texture that is mapped onto the hair map. Multiple textures are mapped onto each hair map, where each texture contains different information (for example, one texture contains the direction of the strands, and another texture contains the color and opacity of the strands). Within each texture are multiple groups of hairs. When a texture is mapped onto a map, only a subregion of the texture containing a specific group of hairs is used. Hair shading pattern

[0050] To render the hair model, in one embodiment, a Marschner hair shading model is used with certain modifications to allow for real-time execution. One optimization used involves pre-calculating the azimuthal and longitudinal dispersion functions for different angle inputs and storing the result in a lookup table (LUT). Furthermore, the Marschner model requires mathematical integration of the diffusion equation over all light directions. In one embodiment, this integration can be simplified to a summation over the different light sources by using discrete light sources (such as point or directional light sources). This will be discussed in more detail in the following section.

[0051] Environmental map lighting approximation

[0052] To improve realism, an environment map is used (for example, at component 110) to model light coming from all directions in the user's environment. This can, for example, be represented using a high dynamic range (HDR) panoramic image. However, using the environment map with the Marschner shading model requires integrating the diffusion equation over the entire environment map, which cannot be done in real time. In one embodiment, as an approximation, the environment map is converted into a set of discrete light sources. This conversion is done by detecting the brightest points on the environment map and finding the average color for each point to determine the color and intensity of the light. Each bright point is converted into a directional light source. Indirect lighting by double diffusion

[0053] For realistic hair rendering, indirect lighting is simulated according to one embodiment. Indirect lighting refers to light rays from light sources that have undergone more than one ray-surface interaction (the surface being the hair in this case) before reaching the virtual camera. A ray-surface interaction can be either a reflection (where the light ray bounces off the surface) or a transmission (where the light ray enters the surface).

[0054] For example, a beam of light emitted from a light source may interact with many strands of hair before reaching the camera. For example, a light source located behind the hair. Each time a ray-surface interaction occurs, the direction, color, and intensity of the light beam change. The effect of indirect lighting is most noticeable for lighter hair colors (such as blonde hair).

[0055] Accurately simulating all these light rays in real time can involve millions of light rays and hundreds of thousands of strands of hair. The computational cost also grows exponentially with the number of reflections / transmissions per ray, since a single ray is scattered into multiple rays at each surface interaction.

[0056] In one embodiment, the double diffusion technique is used to solve this problem. By approximating the aggregate behavior of the light rays using statistical models, the system does not need to simulate each ray and ray-surface interaction, thus enabling real-time approximation of indirect lighting. In one embodiment, the hair shading model incorporates this technique. Recolorable double diffusion LUTs

[0057] In the double diffusion technique, different LUTs are pre-generated for optimization purposes. However, these LUTs are dependent on hair color and are recalculated if the hair color changes. If multiple hair colors are applied simultaneously, multiple sets of LUTs must be generated, which consumes more memory and increases loading time.

[0058] To optimize double diffusion and LUT generation, a modification was made to the double diffusion technique where an additional dimension was added to the LUT input for a channel of the input color. For example, when using an RGB hair color, the LUT would be applied separately to each RGB color channel. This allows the hair color to be changed without having to recalculate the LUTs. Occlusion management

[0059] In a 3D hair model for application to a head, parts of the hair are occluded by the head, neck, and shoulders when the head is viewed from a particular point of view. In one embodiment, a depth map is used to track the depth of all rendered pixels during rendering operations. This depth map is initialized with the depth of the face, neck, and shoulders, in accordance with a standard approach in computer graphics for handling occlusion.

[0060] During hair rendering, the hair geometry (whether hair maps or curves) is rasterized into pixels by the GPU. Before rendering each pixel, the depth of that pixel is checked against the depth of the depth map at the current frame position. If the depth is greater than that of the In the depth map, the pixel is not rendered. Otherwise, the pixel is rendered (overwriting any existing pixels at the same image position) and the depth map is updated.

[0061] When using hair maps, the rendering process must also check if the current pixel is transparent. If it is transparent, it will not render the current pixel regardless of the depth.

[0062] Hair strand color transfer

[0063] To support the rendering of real hair coloring products, such as providing a virtual try-on of a specific product, the hair rendering engine 120 (for example, at component 128) is configured to receive a strand of hair representing a hair coloring product. Figure 3 shows a representative strand. Although shown as a grayscale image for deposition, the image is typically a color-based image representing the product color (for example, an RGB image).

[0064] A strand is capable of capturing a wide color distribution from the product, as opposed to using a single color. Each hair model as used in the 100 system includes a hair texture that is matched to the mesh or strands. The color distribution of the hair strand is transferred to the hair texture using a histogram matching process. This process is described in more detail in US patent application No. 18 / 109,310 filed on February 14, 2023.For example, in one embodiment, a computer-implemented process includes executing on a processor one or more steps comprising: matching gray levels from a strand image to gray levels from a portion of hair in an input image by matching their respective frequencies to establish a mapping relationship between the strand image and the portion of hair, wherein the gray levels of the strand image are associated with respective strand color values; and coloring a pixel in the portion of hair based on a strand color value determined using a gray level of the pixel and the mapping relationship.

[0065] In one embodiment, the respective frequencies are probabilities of the occurrence of gray levels in the strand image or in the hair portion, respectively. In one embodiment, the frequencies are represented by histograms or by cumulative distribution functions. In one embodiment, the method further comprises calculating a lookup table that maps the gray levels of the hair portion to the strand color values. In one embodiment, the coloring of the hair portion pixel is performed by a graphics processing unit (GPU) using a shading element. In one embodiment, the lookup table is The shading element is provided using a 1D texture. In one embodiment, the method includes preprocessing the strand image or input image using a deep neural network to improve pixel coloring accuracy in the hair portion. In one embodiment, an output image can be displayed comprising the input image and the colored hair portion. The output image can be processed using a guided filter. Wick variation

[0066] In real-life situations, hair color is generally not uniform. This is especially true for light hair. By using the above hair strand color transfer technique, the recolored hair texture exhibits this variation. For better control of the variation, there are adjustable parameters for the frequency and scale of the variation. The frequency controls the amount of detail in the variation, while the scale determines how closely similar colored strands appear next to each other.

[0067] Frequency is implemented by blurring the hair texture to reduce detail, while scaling is achieved by adjusting the texture matching so that the texture is more compressed or stretched relative to the hair surface.

[0068] Furthermore, for blond hair, it is common for the upper part of the hair near the roots to be more shaded. The rendering engine (component 128) exposes the parameters to control the shading and color of the roots. Smoothing and anti-aliasing

[0069] In one embodiment, smoothing and anti-aliasing techniques are used to enhance the rendering of 3D hair. In particular, when the 3D hair mesh is rendered in the scene, the higher-resolution texture of the mesh appears very pixelated. This is especially noticeable on loose strands of hair and when the hair is in motion. To make the experience less jarring, a Box Blur and Poisson Blending algorithm is applied to the edge of the 3D hair mesh, which softens the pixelation of loose strands at the edges, which are the most visible. As noted, the 3D hair mesh could be hair maps, in one embodiment. As noted, the 3D hair mesh could be hair curves, in one embodiment. When rendering hair, in one embodiment, each pixel has an opacity value in addition to its RGB color.Since edges typically have an opacity greater than 0 but less than 1 due to anti-aliasing, these values ​​can be used as an easy way to identify edges. Similarly, a multi-sample anti-aliasing (MSAA) technique is used in one embodiment. to reduce the pixelation of all the hair strands in the mesh, so that a more natural-looking result is obtained that blends better with the video resolution.

[0070] In one embodiment, the hair strands can be smaller than one pixel in thickness. To achieve this, the hair is rendered at a higher resolution than the output image resolution and then miniaturized. The resulting strands appear smaller than one pixel and also appear smoother due to the interpolation used in the downsampling. Multicolored placements

[0071] In one embodiment, the hair rendering by component 128 is capable of applying multiple colors simultaneously to the hair mesh. These color segments can be arranged horizontally, vertically, or as root recoloring on the mesh.

[0072] For horizontal placement, in one embodiment, the mesh is segmented into multiple sections based on the length of the hair strands and the colors applied to each of these sections. For example, a color may be applied only to a section near the root of the hair strand on the scalp. For example, a color (e.g., a different color) may be applied only to the section at the tips of the hair strands. Everywhere else, boundaries may be defined for the horizontal sections and colored accordingly. To smooth the color transition, a color gradient is applied between each of the sections.

[0073] For vertical placement, in one embodiment, the 3D representation (for example, the 3D mesh) segments the hair strands into multiple groups, and colors are applied to each of these groups. Note that in this case, the groups may include non-contiguous tufts of hair. For example, in one case, a color may be applied to the group of hair at the front of the head, creating a face-framing effect, or in another case, a color may be applied to small tufts of hair throughout the mesh to create a highlighted effect. To smooth the color transition, the color of adjacent hair strands that do not belong to the same group is adjusted, and the colors of the different groups are blended to produce a visual transition color for that strand.In one embodiment, user input is received to segment (i.e., group) the hair strands according to the input, for example, allowing a user to select adjacent hair strands and define multiple groups. Coloring can be applied to these interactively defined groups.

[0074] For root recoloring, in one embodiment, there is a control over the amount of the user's underlying root hair color to be exposed. A root color parameter controls the amount of original root color to be exposed, and a root migration parameter controls the amount of original root color to migrate into the new hair color. Shades

[0075] In one embodiment, the Hair Renderer 120 is activated to apply two types of shadows. The first type of shadows are those cast by the head and strands of hair onto the hair model itself. The second type of shadows are those cast by the hair onto the head. For both shadows, two types of depth maps are rendered from the perspective of the light sources: one depth map is for the entire hair and head; and one depth map is for the hair only. These depth maps are used to apply shadow effects to the hair and the head, respectively. The depth value of each pixel is compared to the corresponding depth value on the depth map; if the pixel's depth is greater than the depth map value, the pixel is considered to be in shadow.

[0076] For hair, the shadow value is used to determine whether the current pixel on the hair will be in shadow. In response to the fact that it is in shadow, the shadow color is blended with the final hair color. For the head, the shadow color is applied to the pixel in shadow and rendered onto a blank image. As each pixel of the head is rendered, the shadow appears in the shape of the head in the blank image.

[0077] To address the issue of aliasing artifacts for shadows on hair, a standard technique called Percentage Proximity Filtering (PCF) is used, where a pixel that is in shadow is determined by the average of its neighbors. Rather than relying on a binary yes / no result for "in shadow" or "not in shadow" as determined by comparing the depth of the current pixel against the depth map, neighboring pixels are also compared using the depth map, and a weighted average (e.g., between 0 and 1) is determined. For example, a pixel might be considered 40% in shadow. For shadows cast on the head, a Gaussian blur is applied to the rendered shadows that were applied to the blank image, smoothing the shadows at the edges. Finally, these blurred shadows are rendered onto the head. Physical Simulation Simulation Overview

[0078] This section explains in broad outline the technique used to simulate hair movements as they appear in real life, for example at through the operations of the physical simulation component 126. In one embodiment, to achieve this, a programmatic frame is incorporated into the hair mesh to guide the movement of the hair strands during rendering. The frame includes several guide strands, which are distributed throughout the hair mesh model. In one embodiment, the guide strands are placed by the hairstyle designer and are chosen to give the most realistic physical behavior for that hairstyle. For example, 10 to 20 relatively evenly distributed guided strands are defined. Like other hair strands (e.g., non-guiding), these guide strands start at the hair root and grow along the shape of the hair mesh. Each hair strand in the hair mesh is mapped to these guide strands in order to follow the movements of the guide strands. The [Fig.4A] shows an image 400 having hairs 402 and a representative guide strand 404 for a hair mesh (not shown).

[0079] The guide strands comprise particles (e.g., 404A, 404B, 404C, and 404D) that are connected to each other by line segments (e.g., 404E). The particles in the guide strands are divided into two categories: anchored particles (shown as dashed lines such as 404A and 404B) and unanchored particles (shown as solid lines such as 404C and 404D). An anchored particle is one that only follows the position of the head, so that it appears "anchored" to the head, while unanchored particles are free to move. Using these properties, it is possible to define which part of the hair should remain close to the head and which part should be free.

[0080] In one embodiment, the movement of the non-guiding hair strands is determined relative to the movement of the associated guided hair strands. The operations associate each non-guided hair strand with the two nearest guided hair strands.

[0081] For each strand of hair, a representative vertex on the strand is determined at a location that is 3 / 4 (75%) along the strand from root to tip. The number 3 / 4 is determined empirically as being farther from the root because it has been determined that the guide strands are close together near the root and could result in an inaccurate match if a vertex closer to the root were chosen. The length can be between 50% and 80%, with a preferred length of 75% (i.e., 3 / 4). The two nearest guide strands to serve as references are located for each particular strand of hair. For example, the physical distance (e.g., Euclidean distance) from each of the guide strands is determined, and the two nearest are determined using these distances. Each vertex on the non-guiding strand is mapped to a corresponding segment on each of the two guiding strands. This is done by finding the shortest distance. During the physical calculation, the movements of the vertexes (of a non-guiding strand) will be influenced by the line segments assigned from the two nearest guiding strands, with the nearest guiding strand having more influence than the farthest.

[0082] In one embodiment, the hair simulation avoids purely uniform hair movement, as this is not what occurs in real life. Each particle in the guide strand is also assigned a configurable mass parameter, which affects the acceleration it undergoes in the physical calculation. This mass is also influenced by the number of non-guide hair strands assigned to it, with more strands adding to the mass of each particle in that guide strand. According to the assignment of non-guide strands to guide strands, tufts of hair are defined, with the size of a tuft related to the number of co-assigned hair strands, thus defining tufts of different sizes. The tufts move differently depending on the amount of hair contained in those tufts.Independent movement of the different parts of the hair is achieved, which is closer to what we see in reality. Simulation forces

[0083] In one embodiment, the physical simulation of hair is stabilized. A reference strand is defined for each guide strand to help stabilize the physical rendering. A reference strand is a strand of unanchored particles that reacts only to simulated forces such as gravity and centrifugal force. Each reference strand defines the movement and bending of the strand as a whole, and the corresponding guide strand would move around it. In one embodiment, initially, each reference strand is defined to have the same position as its associated guide strands. In one embodiment, gravity is assumed to be an acceleration of 9.8 m / s² in the downward direction. In one embodiment, the downward direction would be the same as the actual downward direction (which is possible if the orientation of the camera / 3D device is known).Otherwise, the downward direction is relative to the camera / device (for example, if the camera / device were tilted at 45 degrees, the direction of gravity would also be inclined at the same angle relative to the actual downward direction). In one embodiment, the centrifugal force is calculated based on the average angular velocity of the head, estimated from the change in the head's rotation over time. In one embodiment, . Head rotation is estimated using the vector from the nose to the center of the head. In one embodiment, the direction of the centrifugal force is from the center of the head to the end of the beam, but only the component perpendicular (to the beam) of the force is taken into account.

[0084] In one embodiment, the deflection of each reference wick is modeled as deflections on straight, flexible cantilever beams, one end of which is fixed to an anchored particle and the other end is free. Two beams are modeled for each reference wick, one to model the deflection due to gravity and the other to model the deflection due to centrifugal force. The beams are constructed from the final anchored particle of the guide wick (the reference wick particles are unanchored particles that follow this anchored particle). The reference wick particles are then associated with the beam deflection at a certain point on the beam. Each cantilever beam is configured to be straight, starting with the anchored particle closest to the root.

[0085] For the beam that models the bending due to gravity, the free end of the beam is located directly below the anchored end, which has twice the mid-downward length of the reference strand, so that the initial shape of the guide strands and the hair strands is preserved when the head is in its initial orientation. That is, the length of the beam is twice the distance between the mid-downward apex of the reference strand and the anchored end. The initial orientation is assumed to be straight. The beam deflection due to gravity is calculated as if the beam were subjected to a constant gravitational force along its entire length, the beam having approximate physical properties of 30 tightly packed hair strands.

[0086] The beam modeling the deflection due to centrifugal force is twice the length of a beam extending from the anchor point to the midpoint of all unanchored particles in the reference strand. The beam deflection due to centrifugal force is calculated as if the beam were subjected to a single centrifugal force at its tip, the beam having approximate physical properties of 30 tightly packed strands of hair. This single centrifugal force is used as an approximation to simplify the calculations.

[0087] After calculating the deflections on the two beams due to gravity and centrifugal force, the unanchored particles in the reference wicks are then displaced from their respective original positions according to the sum of the two deflections on the two beams. For each unanchored particle, its movement due to the deflection of a beam is calculated from the deflection at the point on the beam where the original position of the unanchored particle is projected onto the beam. Figure 4B is a representation of the simulation on an inclined face 450. The thick black line 452 represents a beam 452. The downward arrows (e.g., 454) represent forces. A reference wick is shown in two positions, namely as wick 456A, before application of forces, and as wick 456B, after application.

[0088] In one embodiment, in order for the guide wicks to move around their respective reference wicks, modeled springs are present between the guide wicks and the respective reference wicks, and dampers are placed on the guide wick particles. In one embodiment, multiple iterations of Verlet integration calculations are performed per frame to deduce the subsequent position of the guide wick particles due to the forces acting on them by the springs and dampers. In one embodiment, a position modifier is added that displaces each of the guide wick particles by the movement of the head, but only along the direction of the wick, to render the wick non-extensible and non-compressible.

[0089] Once the guided strands are moved, the tufts / strands of hair associated with a respective guided strand are displaced, for example, by applying the influence of the two guided strands associated with an unguided strand of hair. In one embodiment, each unguided strand has two nearest guide strands associated with it. In addition, each particle within the unguided strand also contains the nearest curve position (for example, the parametric value "t" of the curve) along the two nearest curves of the guide strand. An unguided strand is deformed by the guide strands as follows: 1. For each particle in the unguided strand, an overall nearest position is calculated on the two guide curves.For these closest global positions, the original global position (before the application of physics) is subtracted to give a displacement vector representing the change in position due to physics. The global position refers to the (x, y, z) position in global space, the space that contains all the objects in the scene. 2. The global position of the non-guiding wick particle is fixed at its original position offset by the weighted average of the two displacement vectors. The weight is determined by the relative proximity of the non-guiding particle to the nearest guiding wicks, with the nearest guiding wick receiving more weight.

[0090] In order to make the hair strands less uniformly distributed as they follow the guide strand, in one embodiment, each hair strand is moved towards the nearest guide strand or the next nearest to obtain an agglomeration effect. In one embodiment, the quantity The movement depends on one or more of the following parameters: - the speed of the guide strand; the higher the speed, the greater the agglomeration effect; - the distance between the hair strand particle and the root, where the further it is from the root, the greater the agglomeration effect; - a user-selected amount for the agglomeration effect that affects all hair strands; - the number of hair strands that are closest to the guide strands; the more hair strands there are around the guide strand, the less the agglomeration effect movement towards that guide strand, so the resulting agglomeration of hair strands around that guide strand appears greater;- the randomly generated size factor of the guide strand which affects the resulting agglomeration size of the hair strands around the guide strand by affecting the agglomeration effect movement of the hair strands towards the guide strand (see point above); and - a randomly generated ratio which determines how much the hair strand must move towards the nearest guide strand relative to the second nearest guide strand. Collision Management

[0091] In one embodiment, collision detection is performed for collisions between strands of hair and objects that are convex shell meshes of triangles. To improve performance, in one embodiment, collision detection is approximated by pre-calculating the triangular position of the collision mesh corresponding to each angle with an approximate center of the convex shell. In one embodiment, each object in the scene (such as the head, shoulders, neck) is represented as a mesh composed of triangles, for example, using known techniques. Strands of hair are always represented as 3D points instead of triangles. Typically, collision detection algorithms only work with convex meshes. Intuitively, a convex mesh can be considered as a mesh that has neither "indentation" nor holes.Since some of the meshes above are not convex, a convex approximation (called a convex shell) of these meshes is used during collision detection. During collision detection, hairline points are checked against triangles in the convex shell meshes. This approach is called the "spherical approximation" in which the approximation causes the particles to interact with the convex shell "approximately" as if it were a sphere.

[0092] In order to mitigate false positives and false negatives, a padding element is added during collision detection to make the calculation more robust. For this To do this, we use a scaling matrix to enlarge the convex hull meshes.

[0093] In situations where a hair particle collides with multiple very close objects, in one embodiment, only the first object with which the particle came into contact is recorded. This ensures that the particle does not jump back and forth due to simultaneous collisions.

[0094] When a hair particle is detected as colliding with another object, the particle is repositioned to be on the surface of the colliding object. A configurable damping factor is added to the particle to simulate friction on the object's surface. This damping factor is also applied to all hair particles in the same strand that are below this colliding particle (towards the free end), so that the segments below will not move as if it were free, which would appear unrealistic. Lighting estimation

[0095] In one embodiment, the system uses a machine learning model to predict the environment map given by a single camera image. The environment map is a panoramic HDR image that models light coming from all directions. This map is transferred to the environment map lighting stage in the rendering as described above. A model can be trained using supervised learning to predict an HDR environment, for example, trained using matched portrait image and HDR environment map data. Furthermore, there are known models used to estimate the lighting scene of images that can be classified into two main categories: regression models that estimate low-dimensional panoramic lighting parameters and generative models that generate nonparametric illumination maps or light probes.With the increasing popularity of generative models, there are also frame-by-frame generative models that directly generate an HDR environment map, such as styleLight. Previous work has explored ways to infer lighting information in real-time augmented reality applications, including that of LeGender et al., who initially proposed models that predict low-resolution HDR light probes based on LDR images without a limited field of view constraint, and later extended their work to focus on predicting light probes based on portrait LDR images with a limited field of view. These models are small and can infer information in real time on mobile applications.However, because light probes have low resolution and only capture lighting information behind the camera, this limits realism when using light probes to render images. To address this issue, Somanath et al. presented a model that derives high-resolution HDR panorama maps from unconstrained, field-of-view LDR images. The datasets are also publicly available. The generated map is provided for use as described above, serving as an estimation of the original scene's lighting, from which shadows and other light parameters related to the direction and / or color of the light can be determined.

[0096] Figure 5 is a flowchart of operations according to a respective embodiment. The operations are performed by a system, for example, at least one computing device having at least one processor; and at least one memory device storing computer-readable instructions which, when executed by the at least one processor, cause the system to perform the operations. With reference to Figure 5, operations 500 show, in 502, the processing of an input image using a three-dimensional (3D) face detector array to determine a position, rotation, and 3D representation of a face, the input image comprising the face of a head with existing hair. In 504, the removal of the existing hair from the head. In 506, the determination of a 3D hairstyle model to be applied to the head with the existing hair removed.In 508, the rendering of the 3D hairstyle on the head defines an output image. The rendering reacts to one or more of the following: i) a physical simulation of the hair movement and position as applied to the head, the physical simulation modeling at least one force and deforming the 3D hairstyle model in response; or ii) a hair color to recolor the hair of the 3D hairstyle model, the hair color being refined according to an estimate of the ambient lighting conditions. And in 510, the provision of the output image for display by a display device. In one embodiment, the modeling reacts to at least some of the following: face position, face rotation, and the 3D representation of the face.It will be understood that operations similar to operations 500, such as in one embodiment, can be performed when rendering using the 3D hairstyle model reacts to a physical simulation or hair coloring that changes the color of a 3D hairstyle model. Furthermore, hair coloring, if performed with or without physical simulation, may or may not react to ambient lighting conditions.

[0097] The system may, in one embodiment, include a laptop, a smartphone, a tablet, a desktop computer, a server or other computing device.

[0098] Figure 6 is an illustration of a computer environment comprising one or more systems, according to one embodiment, such as for practicing one or more process aspects in which operations are carried out such as those in [Fig.5].

[0099] The computing environment 600 shows a user computing device 602 (for example, a system), such as a smartphone, a communication network 604, a server 606, and a server 608. The communication network 604 includes wired and / or wireless networks, which may be public or private and may include, for example, the Internet. The server 606 includes a server computing device, such as one for providing a website. The server 608 includes a server computing device, such as one for providing e-commerce transaction services. Although shown separately, servers 606 and 608 may comprise a single server device. The computing environment is simplified. For example, payment transaction gateways and other components, such as those used to perform an e-commerce transaction, are not shown.

[0100] The computing device 602 includes a storage device 610 (for example, a non-transient device such as memory and / or an integrated circuit disk, etc.) for storing instructions which, when executed by a processor (not shown) such as a central processing unit (CPU), a graphics processing unit (GPU), or both, cause the computing device 602 to perform operations such as a computer-implemented process. The storage device 610 stores a virtual fitting application 612 comprising components such as software modules providing a user interface 614, a VTO processing pipeline 616, a VTO recommendation component 618 with VTO data 618A, and a shopping component 622 with a shopping cart 624 (for example, shopping data).In one embodiment, not shown, the VTO application lacks a recommendation component and / or a purchase component, for example, providing a VTO selection component to choose VTO options to view as VTO effects. In one embodiment, the VTO application is a 3D hair VTO using physical simulation. The VTO 618A data can include hair color data (for example, such as one or more strands like strand 300), associated product data, 3D hairstyle model data for trying on a 3D hairstyle, etc. This data can be retrieved and / or stored in a database or other data store.

[0101] In one embodiment, the VTO 612 application is a web application as obtained from the server 606. In another embodiment, the VTO 612 application, as for a native application, is provided by a content delivery network. The VTO 618A data can be obtained from a content management system 607, associated with the server 607A. The content management system Content 607 includes a 607B data store storing VTO-related data, such as strand data and rendering effect data, 3D hairstyle model data, etc. In one embodiment, VTO data, particularly color-related data, is associated with real-world products. Such a VTO experience allows a user to simulate trying on a real-world product, for example, using a desired 3D hairstyle. VTO data (e.g., color, etc.) can be provided to a server in other ways, such as as color or other parameters. For example, sliders can provide input that is mapped to data values.In one embodiment, VTO data can be provided to a user device (e.g., 602) by inclusion in a native application package and delivered as updates to a native application, such as via the 606 server. VTO data can thus be provided from various sources. Rendering effect data can include data for rendering an effect, such as simulating a try-on property like a matte finish, a glossy finish, etc. A 607C UI to the 607 content management system can be provided to a product provider, such as a brand owner, to upload VTO data (e.g., strand images) and provide input for defining strand data, product data, and / or rendering effect data, 3D hairstyle model data, etc. In one embodiment, the 607B UI is web-based.

[0102] Although not shown, the 602 user device can store a web browser for running the web-based VTO 612 application. In one embodiment (not shown), the VTO 612 application is a native application conforming to an operating system (also not shown) and software development requirements that may be imposed by a hardware manufacturer, for example, of the 602 computing device. The native application can be configured for web-based or similar communication with the 606, 607A, and / or 608 servers, as is known.

[0103] Figure 6 shows various input and output data or information associated with the use of the VTO application 612, for example. This input and output data includes an input image 626 of the user to be processed for a VTO experience, an output image 628 on which the product effects are simulated providing a VTO experience, a VTO selection 620 comprising, in one embodiment, a user input selecting one or more color effects to be simulated and a selected 3D hairstyle on which the color(s) is / are rendered, and VTO options 622 comprising color and hairstyle options to be virtually tried on, for example, for a user selection. of device 602, and purchase transaction information 624, including purchase information provided to and / or received from a user to purchase a product. As noted, not all VTO application embodiments include e-commerce capabilities. While some embodiments refer to a strand as a format for providing a color to be applied, alternatively or in addition, the color can be identified in other ways. For example, the color can be determined from the user's existing hair in the input image. The user can provide a second input image—for example, a color identification image, which might include a face with hair—and the VTO application can process the color identification image to determine the hair and extract one or more colors to use for coloring the 3D hairstyle to be rendered.Color options can be provided in other ways such as a color chart showing a set of colors, a color value input interface (e.g., text) providing values ​​according to a color representation scheme, etc.

[0104] In one embodiment, via one or more user interfaces 614, the VTO product options 622 are presented for selection to be virtually tried out by simulating effects on an input image 626. In one embodiment, the VTO options 622 are derived from or associated with VTO data 618A, which may be product data. In one embodiment, the VTO data (e.g., product data) may be obtained from the server 606 and provided by the VTO recommendation component 618, which in one embodiment may be a product data analyzer where user-based recommendations are not made per se. Instead, all available product data is made available for selection. Although not shown, user or other input may be received for use in determining the VTO recommendations.The user can be prompted, for example via one of the 614 interfaces, to provide input to determine recommendations. In one embodiment, the VTO recommendation component 618 communicates with the server 606. The server 606, in one embodiment, determines the recommendation based on the input received via the 618 component and provides VTO data accordingly. The 614 user interface can present the VTO options 622, for example, by updating the display of options in response to the data received when the user navigates or otherwise interacts with the user interface.

[0105] In one embodiment, one or more user interfaces provide instructions and commands to obtain the input image 626, and the VTO selection input 620 such as an identification of one or more VTO products recommended to try. In one embodiment, the input image 326 is an image of a user's face (e.g., a portrait), typically with hair, which can be a still image or a frame from a video. In one embodiment, the input image 626 can be received from a camera (not shown) of the device 602 or from a stored image (not shown). The input image 626 is provided to the VTO processing pipeline 616 for processing according to the operations in [Fig. 5] and the components shown in the device 102 of [Fig. 1] to produce an output image 628 for the VTO. In one embodiment, the VTO selection input 620 includes a selection input for strands of hair from the 3D hairstyle model with which to define groups for coloring.More than one color can be selected to be tried at the same time, applied to different groups or to different places within the same group for horizontal or vertical coloring effects, for example.

[0106] It is understood that the output image 628 may include a frame from a video sequence. The user interfaces 614 provide the output image 628. The output image 628, in one embodiment, is presented as a portion of a live stream of successive output images (each 628, for example), as when a self-portrait video is augmented to present an augmented reality experience. In one embodiment, the output image 628 may be presented with the input image 626, as in a side-by-side display for comparison purposes. In one embodiment, this may be an "in-place" before / after comparison interface where the user moves a slider to reveal more of the original or processed image. In one embodiment, the output image 628 can be saved (not shown) as such on the storage device 610 and / or shared (not shown) with another computing device.

[0107] In one embodiment, the input images (not shown) include input images from a videoconference session, and the output images include a video shared with another (or several other) participant(s) in a videoconference session. In one embodiment, the VTO application (which may have another name) is a component or extension module of a videoconferencing application (not shown) that allows the user of device 602 to present the VTO results (for example, a new hairstyle or color) during a videoconference with one or more other conference participants.

[0108] It will be obvious to a person skilled in the art that many aspects and features are disclosed by the embodiments herein. The following numbered statements relate to at least some of these aspects and features.

[0109] Declaration 1: System comprising: at least one processor; and at least one memory device storing computer-readable instructions which, when they are executed by at least one processor, causing the system to: process an input image using a three-dimensional (3D) face detector array to determine a 3D representation of a face and head, the input image including the face and head; determine a 3D hairstyle model to be applied to the input image; render the 3D hairstyle from the 3D hairstyle model to define an output image, in which the rendering reacts to one or more of the following: a physical simulation of the movement and position of the hair of the 3D hairstyle to be rendered, the physical simulation reacting to the 3D representation of the face, and the physical simulation modeling at least one force and deforming the 3D hairstyle according to the at least one force; or a hair color to recolor the hair of the 3D hairstyle model, the hair color being refined according to an estimate of the ambient lighting conditions;and provide the output image for display by a display device.

[0110] Declaration 2: System according to Declaration 1, wherein the computer-readable instructions, when executed by at least one processor, cause the system to provide one or more interfaces to at least: receive input identifying one or more of the 3D hairstyle or hair color for rendering; recommend VTO options including one or more of the 3D hairstyles or hair colors to be selected for identification for rendering; conduct a transaction to purchase a product associated with the 3D hairstyle or hair color.

[0111] Declaration 3: System according to Declaration 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to represent the head as a 3D representation on which the 3D hairstyle is rendered.

[0112] Declaration 4: System according to Declaration 3, wherein the computer-readable instructions, when executed by at least one processor, cause the system to remove existing hair from the head; and wherein the rendering renders the head with the existing hair removed.

[0113] Declaration 5: System according to Declaration 4, wherein the removal of existing hair includes: processing the input image using a hair segmentation network to provide a hair segmentation mask for existing hair; and removing existing hair from the head using the hair segmentation mask.

[0114] Declaration 6: System according to Declaration 4, wherein the removal of existing hair includes retouching a portion of the scalp using a 3D scalp filter to reconstruct a head shape, the 3D scalp filter reacting to at least certain of a facial position, a rotation of the face and the 3D representation of the face and head; and in which the 3D hairstyle is to be applied to the reconstructed head.

[0115] Declaration 7: System according to Declaration 6, wherein scalp retouching uses a skin color determined from the input image.

[0116] Declaration 8: System according to Declaration 7, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to downsample and expand an area of ​​existing hair detected by the hair segmentation network to remove existing hair.

[0117] Declaration 9: System according to Declaration 1, wherein the computer-readable instructions, when executed by at least one processor, cause the system to process the input image to determine at least one of a face position or a face rotation; and wherein the physical simulation reacts to at least one of the face position or the face rotation.

[0118] Declaration 10: System according to Declaration 9, wherein the computer-readable instructions, when executed by at least one processor, cause the system to define a 3D collision shape representation for at least some objects in the input image, the at least some objects being selected from the head, a neck, and a shoulder; and wherein the physical simulation of hair movement and hair position reacts to one or more collisions of hair strands with the 3D collision shape representation.

[0119] Declaration 11: System according to Declaration 10, wherein the 3D collision shape representation reacts to at least some of the face position, face rotation, 3D face representation or a 3D hair representation according to the 3D hairstyle model.

[0120] Declaration 12: System according to Declaration 9, wherein the 3D hairstyle model models a plurality of hair strands defining the hairstyle, the model comprising an incorporated frame to guide the movement of the hair strands, the frame comprising a set of guided strands distributed throughout the model, each hair strand being matched with at least one of the guided strands, each individual hair strand and each individual guided strand extending outwards from a respective individual hair root in accordance with a shape of the hair model; and wherein the physical simulation determines the hair movement and hair position of the guided strands according to at least one force and determines the hair movement and hair position of the hair strands according to the matching with the guided strands.

[0121] Declaration 13: System according to Declaration 12, wherein each guided wick comprises a plurality of particles linked to each other by segments of line, the particles comprising i) anchored particles that only follow a movement or position of the head and ii) unanchored particles, whose movement or position reacts to at least some of the following: face position, face rotation, 3D representation of the face, or at least one force.

[0122] Declaration 14: System according to Declaration 13, wherein the computer-readable instructions, when executed by at least one processor, cause the system to match each hair strand with two nearest guided strands, wherein the two nearest guided strands for a particular hair strand are determined in response to an evaluation of the physical distance between a hair strand crest and a guided strand crest, and wherein the hair strand crest is between 50% and 80% or preferably 75% of the distance along the hair strand and the guided strand crest is between 50% and 80% or preferably 75% along the guided strand.

[0123] Declaration 15: System according to Declaration 14, wherein, for a particular strand of hair, the matching includes matching each vertex on the particular strand of hair with a corresponding respective segment on each of the two nearest guided strands.

[0124] Declaration 16: System according to Declaration 13, wherein each of the particles in a guided wick is associated with a respective mass parameter and wherein an acceleration of any one of the particles determined by the physical simulation reacts to the respective mass parameter of one of the particles.

[0125] Declaration 17: System according to Declaration 16, wherein a mass of the guided strand further reacts to the number of hair strands matched with the guided strand.

[0126] Declaration 18: System according to Declaration 13, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to: define a plurality of reference wicks comprising unanchored particles subjected to motion by at least one force; assign each guided wick to one of the respective reference wicks; for each respective reference wick: model the at least one force; and apply the at least one modeled force to the respective reference wick, moving the unanchored particles in response; and move the respective guided wicks around the respective reference wicks.

[0127] Declaration 19: System according to Declaration 18, wherein at least one force comprises one or both of a gravitational or centrifugal force.

[0128] Declaration 20: System according to Declaration 18, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to model each force of the at least on the force as deflections on a respective right-folding cantilever beam, wherein a fixed end of the beam starts at a location of an anchored particle nearest to the root of the associated guided wick.

[0129] Declaration 21: System according to Declaration 20, wherein the at least one force comprises two or more forces, and wherein the application of the at least one force comprises the displacement of the unanchored particles according to the addition of deflections of the two or more forces.

[0130] Declaration 22: System according to Declaration 18, wherein the at least one force comprises two or more forces, and wherein the application of the at least one force comprises the displacement of the unanchored particles according to the addition of deflections of the two or more forces.

[0131] Declaration 23: System according to Declaration 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to process the input image to determine a 3D environmental map defining one or more light sources estimating ambient light to refine hair color.

[0132] Declaration 24: System according to Declaration 23, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to process the input image using a deep neural network to define the 3D environmental map.

[0133] Declaration 25: System according to Declaration 23, wherein the environmental map is a panoramic HDR image that models light coming from all directions.

[0134] Declaration 26: System according to Declaration 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to perform a rendering in accordance with a Marschner hair shading pattern.

[0135] Declaration 27: System according to Declaration 26, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to perform a rendering in accordance with the Marschner hair shading model adapted to operate in real time, using at least some pre-calculated diffusion functions for different angle inputs.

[0136] Declaration 28: System according to Declaration 26, wherein the computer-readable instructions, when executed by at least one processor, cause the system to perform an approximation of a mathematical integration of a diffusion equation over all directions of light by performing simplified operations using a plurality of discrete light sources determined from a 3D environmental map generated from the input image which estimates the plurality of discrete light sources.

[0137] Declaration 29: System according to Declaration 28, wherein the computer-readable instructions, when executed by at least one processor, cause the system to process the input image to determine a 3D environmental map defining one or more light sources estimating ambient light; process the map to detect the brightest points on the map, find the average color for each point to determine the color and intensity of light, and convert each point into a directional light source.

[0138] Declaration 30: System according to Declaration 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to perform a rendering in accordance with an indirect lighting simulation, approximating the aggregate behavior of light rays using statistical models in accordance with a double diffusion technique.

[0139] Declaration 31: System according to Declaration 30, wherein the computer-readable instructions, when executed by at least one processor, cause the system to adapt to operate in real time using precomputed recolorable double-diffusion search tables having an additional color dimension for a hair color channel.

[0140] Declaration 32: System according to Declaration 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to define a rendered pixel depth map for pixels rendered for the output image, initialize the depth map in response to pixel depths for the face, neck and at least one shoulder of the input image, and update the depth map as pixels are rendered on the output image; and use the rendered pixel depth map to determine hair pixel occlusion.

[0141] Declaration 33: System according to Declaration 32, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to skip a rendering of a hair pixel at a location in the output image in response to an evaluation of a depth value of the hair pixel and a depth value from the rendered pixel depth map at the location of the hair pixel, the evaluation indicating that the hair pixel is occluded.

[0142] Declaration 34: System according to Declaration 1, wherein the computer-readable instructions which, when executed by at least one processor, lead the system to define the 3D hairstyle model as a hair map model using 3D rectangular planes; and to skip a rendering of a transparent hair pixel on the output image defined from the 3D hairstyle model.

[0143] Declaration 35: System according to Declaration 1, wherein the 3D hairstyle model comprises: a hair map model defined using 3D rectangular planes; or a hair curve model defined using a set of Bézier curves, each curve representing a strand of hair.

[0144] Declaration 36: System according to Declaration 1, wherein the 3D hairstyle model comprises a hair curve model defined using a set of Bézier curves, each curve representing a strand of hair, and wherein the 3D hairstyle model is defined or stored using one or more optimization techniques consisting of: representing at least some of the Bézier curves in a quadratic curve shape rather than a cubic curve where the quadratic curve shape approximates the cubic curve shape; storing only one of the two sides of a symmetrical hairstyle, producing a mirror image of the other side when the symmetrical hairstyle is loaded; or storing curve data in a binary format compressed with a lossless compression technique.

[0145] Declaration 37: System according to Declaration 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to: receive a strand of hair representing a desired hair coloring result, the strand comprising a color distribution.

[0146] Declaration 38: System according to Declaration 37, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to transfer the color distribution of the strand to a hair texture of the 3D hairstyle model using a histogram matching process.

[0147] Declaration 39: System according to Declaration 38, wherein the computer-readable instructions, when executed by at least one processor, cause the system to provide a command for a variation in hair color, to command a frequency of variation by compressing the hair texture to reduce detail, or to command a scale of variation by adjusting a hair texture matching of the hair texture on a hair surface, which matching compresses or stretches the hair texture relative to the hair surface.

[0148] Declaration 40: System according to Declaration 38, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to provide a command for hair root coloring to command one or both of a color fade or a color choice.

[0149] Declaration 41: System according to Declaration 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to soften the pixelation of the edges of loose strands of hair from the 3D hairstyle model by applying a blur or a blend or both a blur and a blend.

[0150] Declaration 42: System according to Declaration 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to reduce the pixelation of all hair strands from the 3D hairstyle model using a multi-sample anti-aliasing (MSAA) technique.

[0151] Declaration 43: System according to Declaration 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to perform hair rendering at a resolution higher than the output image and miniaturization to reduce the size of the strands and provide smoothing.

[0152] Declaration 44: System according to Declaration 1, wherein the computer-readable instructions, when executed by at least one processor, cause the system to apply two or more hair colors to respective hair segments from the 3D hairstyle model, wherein the hair segments are arranged horizontally, vertically, or as root recoloring.

[0153] Declaration 45: System according to Declaration 44, wherein the 3D hairstyle model segments the hair strands into multiple groups, and computer-readable instructions which, when executed by at least one processor, cause the system to apply two or more hair colors to at least some of the multiple groups.

[0154] Declaration 46: System according to Declaration 45, wherein the respective hair segments are grouped to define non-contiguous tufts of hair and computer-readable instructions which, when executed by at least one processor, cause the system to at least: apply a first color to a majority of the strands and a second color to a group of strands at the front of the head, creating a face-framing effect; or apply a first color to a majority of tufts of strands throughout the hair and a second color to second tufts of strands distributed throughout the hair, creating a striated highlight effect.

[0155] Declaration 47: System according to Declaration 45, wherein the computer-readable instructions, when executed by at least one processor, cause the system to smooth the color transition, adjust the colors of adjacent hair strands that do not belong to the same group, and blend the color of the different groups to produce a visually transitional color for adjacent strands.

[0156] Declaration 48: System according to Declaration 44, wherein the computer-readable instructions, when executed by at least one processor, cause the system to: segment the hair into multiple groups interactively in accordance with user input to identify the multiple groups; and apply the two or more hair colors to at least some of the multiple groups.

[0157] Declaration 49: System according to Declaration 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to render shadow pixels simulating the shadow cast by the hair from the hair model, the shadow pixels reacting to the estimation of ambient light conditions.

[0158] Declaration 50: System according to Declaration 49, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to: define a first shadow depth map and a second shadow depth map, each being defined from the point of view of one or more light sources determined from the estimation of ambient light conditions, the first shadow depth map being defined for the hair and head and the second shadow depth map being defined for the hair only;and apply shadow effects to the hair and head respectively, by evaluating, for each of the shadow depth maps, a depth value of a pixel against a corresponding depth value in the shadow depth maps, and if the depth value of the pixel is greater than the depth value in the depth maps, then the pixel is considered a shadow pixel.

[0159] Declaration 51: System according to Declaration 49, wherein for a shadow pixel projected onto a hair pixel, the computer-readable instructions which, when executed by at least one processor, cause the system to blend a shadow color from the shadow pixel with a final hair color from the hair pixel onto which the shadow is projected.

[0160] Declaration 52: System according to Declaration 51, in which the shadow pixel is determined by the average of its neighbors.

[0161] Declaration 53: System according to Declaration 49, wherein for a shadow pixel projected onto a head pixel, the computer-readable instructions which, when executed by at least one processor, cause the system to apply a shadow color from the shadow pixel in a head shape onto a blank image.

[0162] Declaration 54: A system according to Declaration 53, wherein, for a shadow pixel projected onto a head pixel, the computer-readable instructions which, when executed by at least one processor, cause the system to apply a blur Gaussian to the blank image to smooth shadow pixels on the edges, and render the shadow pixels smoothed on the head in the output image.

[0163] Declaration 55: A computer-implemented method comprising: obtaining a 3D hairstyle model of a 3D hairstyle to be rendered on the input image in order to produce an output image, the 3D hairstyle model comprising a plurality of hair strands; in response to a 3D representation i) of a face, or ii) of a face and a head in the input image, simulating and applying at least one force to deform at least some of the hair strands; and rendering the output image for display using the deformed 3D hairstyle model.

[0164] Declaration 56: A method according to Declaration 55, comprising determining the 3D representation i) of the face, or ii) of the face and head in the input image using one or more deep neural networks.

[0165] Declaration 57: Method according to Declaration 55, comprising applying a hair color to the deformed 3D hairstyle model, the application reacting to ambient light conditions determined from the input image.

[0166] Declaration 58: Process according to Declaration 57: comprising the determination of a 3D environmental map defining one or more light sources to estimate ambient light conditions to refine hair color.

[0167] Declaration 59: A process according to Declaration 57, wherein the rendering reacts to one or both of a hair shading pattern or a light diffusion pattern to refine hair color.

[0168] Declaration 60: A method according to Declaration 55, comprising modeling one or more collisions of the hair strands with the face or the face and the head and wherein the rendering reacts to one or more collisions.

[0169] Declaration 61: A method according to Declaration 55, comprising modeling one or more occlusions of the hair strands with the face or the face and head in accordance with a pixel depth map and wherein the rendering reacts to one or more occlusions.

[0170] Declaration 62: A method according to Declaration 55, comprising modeling one or more shadows and wherein the rendering reacts to one or more shadows.

[0171] Declaration 63: A method according to Declaration 55, wherein the 3D hairstyle model defines the hair strands into two or more groups and wherein the method includes the application of two or more hair colors to the hair strands, wherein at least one of the groups is colored a different color from another group or groups.

[0172] Declaration 64: A method according to Declaration 55, wherein the 3D hairstyle model defines the hair strands into a plurality of hair segments and wherein the method comprises the application of a hair color in accordance to hair segments to apply hair color horizontally, vertically or for root coloring.

[0173] Declaration 65: System comprising: at least one processor; and at least one memory device storing computer-readable instructions which, when executed by the at least one processor, cause the system to: process an input image using a three-dimensional (3D) face detector array to determine a 3D representation of a face and a head, the input image comprising the face and the head; determine a 3D hairstyle model to be applied to the input image; render the 3D hairstyle from the 3D hairstyle model in response to the 3D representation of the face to define an output image, wherein the rendering applies a hair color to recolor the hair of the 3D hairstyle model, the hair color being refined according to an estimate of the ambient lighting conditions; and provide the output image for display by a display device.

[0174] Declaration 66: System according to Declaration 65, wherein the computer-readable instructions, when executed by at least one processor, cause the system to perform a physical simulation of the movement and position of the hair of the 3D hairstyle to be rendered, the physical simulation responding to the 3D representation of the face and the physical simulation modeling at least one force and deforming the 3D hairstyle in accordance with the at least one force; and wherein the rendering responds to the deformed 3D hairstyle.

[0175] The features of any one of Declarations 2 to 54 may apply to one or both of Declarations 65 or 66, with modifications if necessary.

[0176] A practical implementation may include all or part of the features described herein. These features, characteristics, and various combinations thereof, as well as others, may be expressed in the form of processes, apparatus, systems, means for performing functions, program products, and other means, combining the features described herein. A number of embodiments have been described. Nevertheless, it is understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. Furthermore, other steps may be proposed, or steps may be eliminated, from the described process, and other components may be added to or removed from the described systems. Accordingly, other embodiments fall within the scope of the following claims.

[0177] Throughout the description and claims of this patent application, the terms "include," "contain," and variants thereof mean "including but not limited to" and are not intended to exclude (and do not exclude) other components, integers, or steps. References to an operation or component involving "one or more" thing(s) as presented do not require that a component or operation be configured to handle a plurality of thing(s) as presented, but include components or operations configured to handle only one thing or one of the things.

[0178] The features, integers, characteristics, or groups described in conjunction with a particular aspect, embodiment, or example of the invention shall be understood as applicable to any other aspect, embodiment, or example, unless they are inconsistent with them. All features disclosed herein (including any attached claim, abstract, and drawing), and / or all steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of these features and / or steps are mutually exclusive. The invention is not limited to the details of the preceding examples or embodiments.The invention extends to any new feature, or any new combination, of features disclosed in this patent memorandum (including any attached claim, abstract, and drawing) or to any new step, or any new combination, of steps of any disclosed method or process.

Claims

Demands

1. A three-dimensional (3D) virtual hair fitting system comprising: - at least one processor; and - at least one memory device storing computer-readable instructions which, when executed by at least one processor, cause the system to: • process an input image using a three-dimensional (3D) face detector array to determine a 3D representation of a face and head, the input image comprising the face and head; • determine a 3D hairstyle model to be applied to the input image; • render the 3D hairstyle from the 3D hairstyle model in response to the 3D representation of the face to define an output image, wherein the rendering applies a hair color to recolor the hair of the 3D hairstyle model, the hair color being refined according to an estimate of the ambient lighting conditions;and • provide the output image for display by a display device.;

2. A system according to claim 1, wherein the computer-readable instructions, when executed by at least one processor, cause the system to perform a physical simulation of the movement and position of the hair in the 3D hairstyle to be rendered, the physical simulation reacting to the 3D representation of the face and the physical simulation modeling at least one force and deforming the 3D hairstyle in accordance with the at least one force; and wherein the rendering responds to the deformed 3D hairstyle.

3. System according to claim 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to provide one or more interfaces to at least: - receive an input identifying one or more of the 3D hairstyle or hair color for rendering; - recommend VTO options including one or more 3D hairstyles or hair colors to select for identification for rendering; - make a transaction to purchase a product associated with the 3D hairstyle or hair color.

4. System according to claim 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to process the input image to determine a 3D environmental map defining one or more light sources estimating ambient light to refine hair color.

5. System according to claim 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to perform a rendering in accordance with a Marschner hair shading pattern.

6. System according to claim 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to perform a rendering in accordance with an indirect lighting simulation, approximating the aggregate behavior of light rays using statistical models in accordance with a double diffusion technique.

7. System according to claim 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to: receive a strand of hair representing a desired hair coloring result, the strand comprising a color distribution.

8. System according to claim 1, wherein the computer-readable instructions which, when executed by at least one processor, cause the system to render shadow pixels simulating the shadow cast by the hair from the hair model, the shadow pixels being reactive to the estimation of ambient light conditions.

9. A system according to claim 8, wherein the computer-readable instructions, when executed by at least one processor, cause the system to: define a first shadow depth map and a second shadow depth map, each defined from the point of view of one or more light sources determined from the estimation of ambient lighting conditions, the first shadow depth map being defined for the hair and head and the second shadow depth being defined for the hair only; and applying the shading to the hair and head respectively, by evaluating, for each of the shadow depth maps, a depth value of a pixel compared to that of a corresponding depth value in the shadow depth maps and if the depth value of the pixel is greater than the depth value in the depth maps, then the pixel is considered to be a shadow pixel.

10. A system according to claim 1, wherein the computer-readable instructions, when executed by at least one processor, cause the system to: i) soften the pixelation of the edges of untied hair strands in the 3D hairstyle model by applying blurring or blending, or both blurring and blending; ii) reduce the pixelation of all hair strands in the 3D hairstyle model using a multi-sample anti-aliasing (MSAA) technique; or perform hair rendering at a higher resolution than the output image and miniaturization to reduce the size of the strands and provide smoothing.

Images