VIRTUAL HAIR COLOR METHODS AND SYSTEMS

Abstract

METHODS AND SYSTEMS FOR VIRTUAL HAIR COLOR Methods and systems for virtual hair try-in are provided. Gray levels of a strand image and gray levels of a hair portion of an input image are matched by matching their respective frequencies to establish a correspondence relationship between the strand image and the hair portion, the gray levels of the strand image being associated with respective strand color values. A pixel in the hair portion is colored based on a strand color value determined using a gray level of the pixel and the correspondence relationship. Figure for abstract: none

Description

Title of the invention: METHODS AND SYSTEMS FOR VIRTUAL HAIR COLOR FIELD OF THE INVENTION

[0001] The present disclosure relates to image processing and augmented reality and more particularly to methods and systems for virtual hair coloring. CONTEXT

[0002] Virtual hair coloring aims to give users the ability to recolor their hair virtually on a device. Users can choose the target shade they want to apply, and the device produces an image of the user with the user's hair colored in the chosen shade. This can help users decide on a shade to color their hair. Hair, like any other real, organic object, tends to have a lot of details and color variations that must be preserved during rendering in order to maintain realism. Any recoloration that eliminates these details would not appear realistic. Furthermore, representing a target color for virtual hair dye is not just a matter of identifying a single target color, but defining a range of all the shades that that specific target hair dye would have once applied to real hair.

[0003] Improved techniques for virtually coloring hair are desired. ABSTRACT

[0004] A computer-implemented method is provided comprising executing on a processor one or more steps. The method comprises matching gray levels from a strand image and gray levels from a hair portion of an input image by matching their respective frequencies to establish a correspondence relationship between the strand image and the hair portion, wherein the gray levels of the strand image are associated with respective strand color values. The method comprises coloring a pixel in the hair portion based on a strand color value determined using a gray level of the pixel and the correspondence relationship.

[0005] The method may further comprise determining the hair portion from the input image via a deep neural network.

[0006] The respective frequencies may be probabilities of the gray levels appearing respectively in the strand image or in the hair portion. The frequencies may be represented by histograms or by cumulative distribution functions.

[0007] The method may further include calculating a lookup table that maps the gray levels of the hair portion to the strand color values.

[0008] Coloring the pixel of the hair portion may be performed by a graphics processing unit (GPU) using a shader. A shader is software code executed by a GPU to render or otherwise modify an image. For example, a shader may be used to modify one or more of lightness, darkness, and color of a pixel. The method may further include providing the lookup table to the shader using a 1D texture. The method may further include interpolating the lookup table.

[0009] The method may further include preprocessing the strand image or the input image using a deep neural network to improve the accuracy of the coloring of the pixel in the hair portion.

[0010] The method may further comprise displaying an output image comprising the input image and the portion of hair as colored.

[0011] The method may further comprise processing the output image using a guided filter.

[0012] The method may further comprise calculating the frequencies of each of the gray levels in the strand image, and calculating the frequencies of each of the gray levels in the hair portion of the input image.

[0013] According to another aspect of the disclosure, a system is provided. The system includes a virtual try-on (VTO) rendering pipeline having computer circuitry for coloring respective pixels in a hair portion of an input image based on respective color values from a strand image, wherein the respective color values are selected from the strand image using, for each respective pixel, a gray value of the respective pixel and a mapping relationship to a gray value of the strand image associated with a respective color value. The system further includes a user interface component for presenting an output image including the input image and the hair portion as colored.

[0014] The system may further include a hair detection engine including computer circuitry to determine the hair portion from the input image via a deep neural network.

[0015] The VTO may include a shader for coloring the respective pixels in the hair portion of the input image.

[0016] The system may further include a preprocessing component including computer circuitry to modify properties of the wick image using of a deep neural network.

[0017] The system may further include a post-processing component including computer circuitry to apply a guided filter to the output image.

[0018] According to another aspect of the disclosure, there is provided a computer-implemented method comprising executing on a processor one or more steps. The method comprises coloring in a virtual try-on (VTO) rendering pipeline respective pixels in a hair portion of an input image based on respective color values from a strand image, wherein the respective color values are selected from the strand image using, for each respective pixel, a gray value of the respective pixel and a mapping relationship to a gray value of the strand image associated with a respective color value. The method further comprises presenting in a user interface an output image comprising the input image and the hair portion as colored.

[0019] The method may further include determining the hair portion from the input image via a deep neural network of a hair detection engine.

[0020] The VTO may include a shader for coloring the respective pixels in the hair portion of the input image.

[0021] According to another aspect of the disclosure, there is provided a computer-implemented method comprising executing on a processor one or more steps. The method comprises selecting a strand gray value from a strand image based on a probability of occurrence of a hair gray value in a hair portion of an input image. The method further comprises coloring a pixel in the hair portion of the input image based on a color value of the strand image associated with the strand gray value. Brief description of the drawings

[0022] [Fig-1] [Fig.l] is an illustration of a computer environment, in accordance with one embodiment, such as for performing a virtual fitting.

[0023] [Fig.2] [Fig.2] is a flowchart of operations such as for a process im computer-implemented in accordance with one embodiment.

[0024] [Fig.3] [Fig.3] is an illustration of a representative histogram mapping, in accordance with one embodiment.

[0025] [Fig.4A] [Fig.4A] is an illustration of a representative image of a facial input image including a portion of hair, in accordance with one embodiment.

[0026] [Fig.4B] [Fig.4B] is an illustration of a representative image of a mask ca- pillar, in accordance with one embodiment.

[0027] [Fig.5] [Fig.5] is a flowchart of operations such as for a computer-implemented method in accordance with one embodiment. DETAILED DESCRIPTION

[0028] In accordance with the present embodiments, one or more methods, systems, apparatuses and techniques for virtual hair coloring in a virtual try-on (VTO) application are described.

[0029] The hair VTO may provide users with the ability to virtually color their hair on a device using either the camera (live video mode) or a snapshot (photo mode). The user may choose the target shade they wish to apply, and the VTO processes the input (video stream or image) to apply a recoloring process and produce the processed image that is shown to the user. In live mode, this may occur multiple times per second with a target of 30 frames per second (FPS) on a mobile device.

[0030] At a high level, the hair VTO may comprise the following steps: input, hair detection, hair recoloring. At the input step, the image may be provided to be recolored. In the case of a direct mode, this image may be extracted from the camera's video stream. Hair detection may represent the process used to segment the hair in the image from the rest (face, background, etc.) and export a hair mask that will be used to apply the recoloring. Hair recoloring may be the process of replacing each original pixel in the image with a new value from the target color.

[0031] Hair, like any other real, organic object, tends to have a lot of detail and color variation that needs to be preserved during rendering in order to maintain realism. Any recoloring that eliminates these details would not look realistic. Furthermore, representing a target color for a virtual hair dye is not just about identifying a specific target color, but defining the range of all the shades that this specific target hair dye would have when applied to real hair. Usually, hair dye brands represent this variety using images of hair strands. These hair strand images represent hair (real, synthetic, or virtual) to which a recolor is applied.Unfortunately, there is no market-wide standard for hair dye color representation, and each brand is free to use its own classification. Within the L'Oréal group, a standard has been defined for evaluating the color of a target shade. This standard uses a notation such as "7.43." The first number (here 7) represents the tone level in a range from 1 (brown) to 10 (blonde). The second number (here 4) is optional and re . presents the primary highlight and defines the overall tone of the color, in a range from 0 to 7. The last number (here 3) is optional and represents the secondary highlight, which is an optional additional color that will enhance, shade, or tone down the intensity of the primary highlight and is in a range from 0 to 7.

[0032] However, this target color notation itself cannot capture all the details of a resulting color. Hair has a lot of variety in the resulting colors. The strand image conveys much more information than just the tone, primary and secondary highlight. The variety in the resulting color comes from the nature of the hair to reflect or capture light from the environment and is visible as dark areas and light areas in the image. Failure to represent these areas in a natural way, which includes respecting the direction of the natural hair strands, will break the illusion of recoloring.

[0033] Hair VTO is typically used to showcase products and simulate what would be a realistic and accurate result if users dyed their hair with the product. Consumer hair dyes should be distinguished from salon hair dyes. Consumer hair dyes are designed to be applied at home by non-specialists and result in a color that will change depending on the user's base color. The product packaging typically shows the resulting color applied to different base colors. In contrast, salon hair dyes are expected to be applied by hairdressers who know how to treat the hair before applying the dye and achieve the specific target color (e.g., by bleaching the hair before application).A successful VTO must be able to deliver both results: color variation based on the user's natural hair color and coverage of the natural hair to achieve a consistent color.

[0034] A hair VTO is subject to a number of constraints, including the following. The hair VTO must only recolor the hair portion of the input image. The produced output image must be realistic. The hair VTO must be able to recolor hair in real-time (more than 20 TPS) on a mobile device. This last constraint has the greatest impact on the techniques available for recoloring the image. Although many techniques exist for recoloring images, none of them are fast enough to be applied in real-time on a mobile device.

[0035] A known technique is to shift the hue (H), saturation (S), and brightness (L) of the input image. Although this technique is easy to implement, it cannot significantly change the brightness of the image, so the degree to which a person's hair color can be changed is limited. It also does not capture the full range of colors in the target color, because it uses only HSL values and not a wick image.

[0036] Another known technique uses the application of a semi-transparent colored layer over the hair. In this technique, the hair mask is used to define a shape that is filled with a semi-transparent texture representing the target color. Although this technique is quick to implement and execute, it is difficult to preserve the natural texture of the hair and it requires lowering the intensity of the layer, which prevents this technique from rendering vivid colors. VTO Application

[0037] [Fig.l] is an illustration of a computing environment 100, in accordance with one embodiment, such as for practicing one or more method aspects. The computing environment 100 shows a user computing device 102, such as a smartphone, a communications network 104, a server 106, and a server 108. The communications network 104 includes wired and / or wireless networks, which may be public or private and may include, for example, the Internet. The server 106 includes a server computing device such as for providing a website. The server 108 includes a server computing device such as for providing e-commerce transaction services. Although shown separately, the servers 106 and 108 may comprise a single server device. The computing environment 100 is simplified.For example, payment transaction gateways and other components such as to complete an e-commerce transaction are not shown.

[0038] The computing device 102 includes a storage device 110 (e.g., a non-transitory device such as memory and / or a solid-state disk, etc.) for storing instructions that, when executed by a processor (not shown), cause the computing device 102 to perform operations such as a computer-implemented method. The storage device 110 stores a virtual try-on application 112 including components such as software modules providing a user interface 114, a face tracker 104B with one or more deep neural networks 106B, a VTO rendering pipeline component 116, a product recommendation component 118 with product data 120, and a shopping component 122 with a shopping cart 124 (e.g., shopping data).

[0039] In one embodiment, the VTO application is a web application as obtained from the server 106. Although not shown, the user device 102 may store a web browser for running the VTO application 112 on the web. In one embodiment, the VTO application is a native application in accordance with an operating system (also not shown) and the requirements software development requirements that may be imposed by a hardware manufacturer, for example, of the user device 102. The native application may be configured for web-based or similar communications to the servers 106 and 108, as is known.

[0040] [Fig.l] shows various input and output data or information associated with a use of the VTO application 112, for example. This includes an input image 126 of the user to be processed for a VTO experience, an output image 128 to which product effects are simulated providing a VTO experience, a VTO product selection 130 comprising a user input selecting one or more product effects to be simulated, VTO product options 132 comprising options for products to be virtually tried on, for example for selection by a user of the device 102, and purchase transaction information 134 comprising purchase information provided to and / or received from a user to purchase a product.

[0041] In one embodiment, via one or more user interfaces 114, VTO product options 132 are presented for selection for virtual try-on by simulating effects on an input image 126. In one embodiment, the VTO product options 132 are derived from or associated with the product data 120. In one embodiment, the product data may be obtained from the server 106 and provided by the product recommendation component 118. Although not shown, user or other input may be received for use in determining the product recommendations. The user may be prompted, for example via one of the interfaces 114, to provide input to determine the product recommendations. In one embodiment, the product recommendation component 118 communicates with the server 106.The server 106, in one embodiment, determines the recommendation based on the input received via the component 118 and provides product data accordingly. The user interface 114 may present the VTO product choices, for example, by updating the display thereof in response to the received data as the user navigates or otherwise interacts with the user interface.

[0042] In one embodiment, the one or more user interfaces provide instructions and commands for obtaining the input image 126, and the VTO product selection input 130 such as an identification of one or more VTO products to try on. In one embodiment, the products may be recommended to a user. In one embodiment, the products may be selected by a user without having been recommended as such. In other words, instances of the products may be presented, for example from a product data store, and the user selects an instance for virtual try-on. In one embodiment, the input image 126 is an image of a user's face, e.g., including hair for a hair-related VTO, which may be a still image or a frame of a video. In one embodiment, the input image 126 may be received from a camera (not shown) of the device 102 or from a stored image (not shown). The input image 126 is provided to the face tracker 104B, e.g., for processing to detect objects in the input image 126 using one or more deep neural networks 106B. In one example, the network classifies, localizes, or segments a portion of the hair in the image.

[0043] In one embodiment, the output (not shown) of the face tracker 104B, such as classification results, localization results, or segmentation results for one or more detected objects, is provided to the VTO rendering pipeline component 116. In one example, the output may include a hair mask. The input image 126 is also provided (e.g., made available) to the component 116. The VTO product selection 130 is also provided to the component 116 to determine which effects are to be rendered. In an embodiment related to makeup simulation, one or more effects may be indicated such as for one or more of the product categories including: lips, eyeshadow, eyeliner, eye shadow, etc. In a hair-only VTO embodiment, only a hair effect is applied.

[0044] The VTO rendering pipeline component 116, in one embodiment, determines whether to render one or more product effects on the input image 126 to simulate a try-on. In an embodiment such as one relating to makeup, for example, in response to the face mask classification output, the VTO rendering pipeline component 116 may determine not to render a product effect on all or part of a face, for example, because a face mask is detected. When a face mask is detected, for example, the VTO rendering pipeline component 116 may trigger the user interface 414 to request the user to remove the face mask. A new image may be received and processed by the face tracker 104B. In one embodiment, images are continuously received as a component of a live stream (e.g., a selfie video).

[0045] For example, in an embodiment where more than one product effect is to be applied to the input image, the VTO rendering pipeline component 116 may render effects (e.g., onto or to) for the input image 126 such as by drawing (rendering) effects in layers, one layer for each product effect, to produce the output image 128. The layering may be assisted by the use of layers in some examples. Some examples may modify pixel values of the input image itself without applying a layer, per se.

[0046] Portions of the operations of the VTO rendering pipeline component 116 (e.g., such as drawing the layers) may be performed by a graphics processing unit, in one embodiment. Rendering is in accordance with the product data 120 as selected by the VTO product selection 130 and is responsive to the location of detected objects. For example, a VTO product selection of a lipstick, lip gloss, or other lip-related product calls for an effect to be applied to one or more detected mouth- or lip-related objects at respective locations. Similarly, an eyebrow-related product selection calls for a selected product effect to be applied to the detected eyebrow objects. Typically, for symmetrical appearances, the same eyebrow effects are applied to each eyebrow, the same lip effect, or the same eye effect to each eye region, but this is not required.In one example, rendering is applied to a region that is relative to the detected objects, such as one or more adjacent detected objects. Some VTO product selections include a selection of more than one product such as coordinated eyebrow and eye products or other combinations of detected objects. The VTO rendering pipeline component 416 may render each effect, for example, one at a time until all effects are applied. The order of application may be defined by rules or in the product selection, e.g., lipstick before lip gloss.

[0047] In an embodiment where an occluding object is detected and the location is determined, for example, as represented in a segmentation mask, rendering may be responsive to such a segmentation mask. Effect rendering may be applied to portions of the face that are not occluded. A segmentation mask may indicate which pixels of the face are available to (e.g., can) receive an effect such as a makeup effect and which pixels are not available to receive an effect.

[0048] The user interfaces 114 provide the output image 128. The output image 128, in one embodiment, is presented as a portion of a live stream of successive output images (each such as example 128) such as when a selfie video is augmented to present an augmented reality experience. In one embodiment, the output image 128 is presented in conjunction with the input image 126, such as in a side-by-side display for comparison purposes. In one embodiment, the output image 128 may be saved (not shown) such as on the storage device 110 and / or shared (not shown) with another computing device.

[0049] In one embodiment, the input images (not shown) comprise input images of a videoconferencing session and the output images comprise video shared with another participant (or more than one) of a videoconferencing session. conference. In one embodiment, the VTO application is a component or plug-in of a video conferencing application (not shown) allowing the user of the device 102 to wear makeup during a video conference with one or more other conference participants. Hair VTO

[0050] Reference is now made to [Fig.2], which shows a method 200 implemented by computer comprising executing on a processor one or more steps. The method 200 comprises matching gray levels from a strand image 136 and gray levels from a hair portion of an input image 126 by matching their respective frequencies to establish a correspondence relationship between the strand image 136 and the hair portion, wherein the gray levels of the strand image 136 are associated with the respective strand color values 210.

[0051] The gray levels of the strand image 136 and the hair portion of the input image 126 may be matched by their frequency (i.e., the most frequent gray level of the strand image 136 may be matched to the most frequent gray level of the input image 126). A matching relationship may be established between the two images. The respective frequencies may be probabilities that gray levels appear in the strand image 136 or the hair portion, respectively. A color mapping may be established to associate any color in an image with a gray level. As such, the gray level distribution of the input image 126 may be used as a lookup index to retrieve a corresponding color in the strand image 136.

[0052] Frequencies may be represented by histograms or by cumulative distribution functions. A cumulative distribution function (CDF) is the cumulative probability that a gray value (between 0 and 255) appears in the image, normalized over the total number of pixels in the image to obtain a result between 0.0 and 1.0. Colors may be represented using RGB, where each color is represented by three values between 0 and 255. Other methods of representing colors may also be used. The CDF may be represented as a histogram. The method 200 may include calculating the frequencies of the gray levels in the hair portion of the input image 126. The method 200 may include calculating the frequencies of the gray levels in the strand image 136.The method 200 may include calculating the frequencies of the strand color values in the strand image 136 and associating the strand color values with the gray values in the strand image 436 based on the frequencies.

[0053] The FDCmèche may be the cumulative gray histogram of the wick image 136, normalized between 0.0 and 1.0. The FDUser can be the cumulative gray histogram of the hair portion of the input image 126, normalized between 0.0 and 1.0. Hstrand>c can be the per-channel histogram of the strand, where c is {r, g, b} and each bin is normalized between 0 and 255.

[0054] The objective may be to find the gray value in the strand image 136 (Gstrand) that corresponds to the gray value in the hair portion of the input image 126 (Gutiiisateur) - That is, the objective is to find Gstrand such that FDUser(Gutiiisateur) = FDmèche(Gmèche). This can be done by calculating Gmèche=FD1mèche(FDUser(Gutiiisateur)). Mathematically, the inverse does not exist unless the FD is strictly monotonically increasing, but an inverse can always be defined by choosing the lowest value of X if a value of Y corresponds to more than one value of X. Now refer to [Fig.3], which shows the correspondence relationship between the two histograms. The first histogram 500 represents the distribution of gray values in the hair portion of the input image 126. The second histogram 520 represents the distribution of gray values in the strand image 4136.Arrow 510 represents the correspondence relationship for a particular gray value. The gray value with a particular frequency or probability in the hair portion input image 126 can be matched to the gray value with the same frequency or probability in the strand image 136. G strand can then be used to index into Hstrand,r, Hstrand,g, Hstrand,b, which gives the color value associated with that gray value in the strand image 436. The function Hn^hXCDF-^echXCDF^^^ provides the correspondence relationship. between gray values in the hair portion of the input image 426 and color values in the strand image 136.

[0055] The method 200 further includes coloring a pixel in the hair portion based on a strand color value determined using a gray level of the pixel and the correspondence relationship 220. For each pixel in the hair portion of the input image 126, the correspondence relationship Hstrand(CDF 1strand(GDFuser(Guser))) may be used to look up a strand color in the strand image 136. This strand color may then be used to color the pixel in the hair portion of the input image 126. This strand color may be blended with the initial pixel color (depending on the desired result and lighting conditions). As a result, the hair portion of the input image 126 may be recolored to resemble the hair color in the strand image 136.

[0056] The method 200 may further comprise determining the hair portion from the input image 126 via a deep neural network 106B. Reference is now made to [Fig.4A], which shows a representative example of an input image 126 showing a face of a user. The input image 126 may include a face portion 402, a neck portion 404, a hair portion 406, and a background portion 408. Referring now to [Fig. 4B], which shows an example of a hair mask 412 produced by the neural network 106B. The neural network 106B takes the input image 126 as input and identifies the hair portion 406 from the other portions of the input image 126 to produce a hair mask 412. The hair mask 412 may include two portions: a hair portion 416 and all other portions 418. The hair mask 412 may distinguish the two portions 416 and 418 by coloring them different colors, such as white and black. The hair mask 412 can be used to identify the pixels in the input image 126 that belong to the hair portion 406.

[0057] The method 200 may further include calculating a lookup table that maps the gray levels of the hair portion 406 to the strand color values. The lookup table may map gray values to strand color values. The lookup table may be calculated by identifying a strand color value for each possible gray value using the mapping relationship. For example, if RGB colors are used, the lookup table may be an array containing RGB elements and indexed by a gray value. That is, the array may contain 256 elements, each element containing an RGB strand color value indexed by a gray value.Coloring a pixel from the hair portion 406 of the input image 126 may involve determining the gray value of the pixel from the hair portion 406, using the gray value as a lookup index in the lookup table, and using the strand color value at the specified index of the lookup table to recolor the pixel in the hair portion 106.

[0058] Coloring the pixel of the hair portion may be performed by a graphics processing unit (GPU) using a shader. Using a GPU may improve the efficiency of coloring the hair portion 406, for example for real-time coloring of the hair portion 406 in a video stream. The method 200 may further include providing the lookup table to the shader using a 1D texture. The table may be stored in a 1D texture and passed into the shader as a uniform. A uniform is a type of variable in a shader that may be available at any stage of the rendering pipeline in the GPU. Since some devices do not support array entries of the size of the lookup table, this allows the method 200 to be used on those devices. The method 200 may further include interpolating the lookup table.Since the GPU . can interpolate textures, the lookup table can be interpolated to smooth the resulting color of each pixel, providing a more accurate and realistic 128 output image.

[0059] The method 200 may further comprise preprocessing the strand image 436 or the input image 126 using a deep neural network 106B to improve the accuracy of the coloring of the pixel in the hair portion 106. The accuracy of the resulting color in the output image 128 may depend on the quality of the input strand image 136. Unfortunately, there is no quality standard for the generation of the strand image 136, and each brand is free to modify the image for its own purposes. A number of parameters of the strand image 136 may be modified to achieve a more consistent result: • Brightness Range: This controls an overall brightness effect that will lighten or darken all pixels in the image. • Brightness Enhancement: This controls a brightness enhancement that makes light pixels brighter, but does not change dark pixels. • Contrast scale: This allows you to control an overall contrast effect. • Saturation Boost: This allows you to control the saturation of the image by further boosting saturated colors, without changing unsaturated ones. • Brightness Cutoff: This controls the cutoff of the brightest values in the histogram data (like a low-pass filter). • Darkness Cutoff: This controls the cutoff of the darkest values in the histogram data (like a high-pass filter). • Hue Rotation: Slightly rotates the hue of the original image.

[0060] Adjusting these parameters before extracting the histogram data from the strand image 136 may improve the accuracy of the hair rendering. These parameters may be adjusted manually. Alternatively, a neural network (e.g., one of the networks 106B, or another) may be trained to adjust these parameters.

[0061] The method 200 may further include displaying an output image 128 comprising the input image 126 and the hair portion 406 as colored. The output image 128 may be displayed, for example, on a user interface 114 of the user computing device 102.

[0062] The method 200 may further comprise processing the output image 128 using a guided filter. When extracting the pixels corresponding to the hair portion 406 in the input image 126 to create the hair mask 412, the neural network 106B may include rough edges with a high amount of alpha transparency. This problem may cause the color to bleed onto the immediate environment around the hair, resulting in a halo effect. To resolve To address this problem, several image processing techniques can be used, however most of them introduce other artifacts and / or are too slow to apply to processing video streams at a frame rate consistent with a live VTO. One solution is to use a post-processing step (after running the neural network model 106B) to recalculate better edges using a guided filter as a matting solution. By using the original input image 126 as a guide for the hair mask image 412, the guided filter can smooth the edges of the hair mask 412 while limiting color bleeding outside the initial hair position.

[0063] The method 200 may further include calculating the frequencies of each of the gray levels in the strand image 136, and calculating the frequencies of each of the gray levels in the hair portion 406 of the input image 126. The frequencies may include the histogram data or the FDC. The frequencies may be converted to probabilities in a range of 0 to 1. The frequencies of the strand image 436 may be calculated once each time a strand image 136 is selected. In a real-time video stream, the frequencies of the input image 126 may be calculated for each frame.

[0064] The method 200 may be implemented on a system 100. The system 100 may include a user computing device 102, which may include a VTO rendering pipeline 116 having computing circuitry for coloring respective pixels in a hair portion 406 of an input image 126 based on respective color values from a strand image 136, the respective color values being selected from the strand image 136 using, for each respective pixel, a gray value of the respective pixel and a correspondence relationship to a gray value of the strand image 136 associated with a respective color value. The user computing device 102 of the system 100 may further include a user interface component 114 for presenting an output image 128 including the input image 126 and the hair portion 406 as colored.

[0065] The system 100 may further include a hair detection engine including computing circuitry to determine the hair portion 406 from the input image 126 via a deep neural network 106B. The hair detection engine may be a component of the user computing device 102 or the server 106 / 108 (not shown).

[0066] The user computing device 102 may include a GPU, and at least a portion of the VTO rendering pipeline 116 may be executed by the GPU. In particular, a shader of the GPU may color the respective pixels in the hair portion 406 of the input image 126.

[0067] The system 100 may further include a pre-processing component comprising computing circuitry for modifying properties of the strand image 136 using a deep neural network 106B. The pre-processing component may be a component of the user computing device 102 or the server 106 / 108.

[0068] The system 100 may further include a post-processing component including computer circuitry to apply a guided filter to the output image 128.

[0069] The server 108 may include a CMS data store for storing the wick image 136 and the frequency or histogram data 138 for the wick image 136. The server 106 / 108 may calculate the frequency or histogram data for the wick image 136. Alternatively, the user computing device 102 may calculate the frequency or histogram data for the wick image 136 and the input image 126. The server 106 / 108 may calculate the frequency or histogram data for the input image 126.

[0070] Reference is now made to [Fig. 5], which shows a computer-implemented method 300 comprising executing on a processor one or more steps. The method 300 may comprise coloring 310 in a VTO rendering pipeline 116 respective pixels in a hair portion 406 of an input image 126 based on respective color values from a strand image 136, the respective color values being selected from the strand image 136 using, for each respective pixel, a gray value of the respective pixel and a correspondence relationship to a gray value of the strand image 136 associated with a respective color value. The method 300 may further comprise presenting 320 in a user interface 114 an output image 128 comprising the input image 126 and the hair portion 406 as colored.

[0071] The method 300 may further comprise determining the hair portion 406 from the input image 126 via a deep neural network 106B of a hair detection engine.

[0072] In another embodiment, a computer-implemented method includes executing on a processor one or more steps including selecting a strand gray value from a strand image 136 based on a probability that a hair gray value occurs in a hair portion 406 of an input image 126. The method further includes coloring a pixel in the hair portion 406 of the input image 126 based on a color value of the strand image 136 associated with the strand gray value.

[0073] It will be understood that corresponding system embodiments are disclosed for each of the method embodiments disclosed herein, e.g. where the system comprises respective components having computer circuitry configured to perform the operations of the computer-implemented method embodiments.

[0074] In addition to the computer device and method aspects, a person of ordinary skill in the art will understand that computer program product aspects are disclosed, where instructions are stored in a non-transitory storage device (e.g., memory, CD-ROM, DVD-ROM, disk, etc.), and that, when executed, the instructions cause a computer device to perform any of the method aspects stored therein.

[0075] Although the computing devices are described with reference to processors and instructions that, when executed, cause the computing devices to perform operations, it is understood that other types of circuitry than programmable processors may be configured. Hardware components including specifically designed circuitry may be used, such as, but not limited to, an application-specific integrated circuit (ASIC) or other hardware designed to perform specific functions, which may be more efficient than a general-purpose central processing unit (CPU) programmed using software.Thus, an apparatus aspect herein generally refers to a system or device having circuitry (sometimes references to computer circuitry) that is configured to perform certain operations described herein, such as, but not limited to, those of a method aspect herein, whether the circuitry is configured via programming or via its hardware design.

[0076] The practical implementation may include some or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, apparatus, systems, means for performing functions, program products, and other ways, combining the features described herein. A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided, or steps may be eliminated, from the described process, and other components may be added to or removed from the described systems. Accordingly, other embodiments fall within the scope of the following claims.

[0077] Throughout the description and claims of this specification, the terms "include" and "contain" and variations thereof mean "including but not limited to" and are not intended to (and do not) exclude other components, integers, or steps.

[0078] The particularities, whole numbers, characteristics, compounds, chemical fractions or groups described in conjunction with a particular aspect, embodiment, or example of the invention are to be understood as applicable to any other aspect, embodiment, or example, unless inconsistent therewith. Any features disclosed herein (including the claims, abstract, and accompanying drawings), and / or any steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and / or steps are mutually exclusive. The invention is not limited to the details of the foregoing examples or embodiments.The invention extends to any novel feature, or any novel combination, of the features disclosed in this specification (including any accompanying claim, abstract and drawing) or to any novel feature, or any novel combination, of the steps of any disclosed method or process.

Claims

Claims

1. A computer-implemented method of virtually coloring hair comprising performing on a processor one or more steps comprising: matching gray levels from a strand image and gray levels from a hair portion of an input image by matching their respective frequencies to establish a correspondence relationship between the strand image and the hair portion, wherein the gray levels of the strand image are associated with the respective strand color values; and coloring a pixel in the hair portion based on a strand color value determined using a gray level of the pixel and the correspondence relationship.

2. The method of claim 1, further comprising determining the hair portion from the input image via a deep neural network.

3. A method according to claim 1, wherein the respective frequencies are probabilities that the gray levels appear in the strand image or the hair portion respectively.

4. The method of claim 1, wherein the frequencies are represented by histograms or by cumulative distribution functions.

5. The method of claim 1, further comprising calculating a lookup table that maps the gray levels of the hair portion to the strand color values.

6. Method according to claim 5, in which the coloring of the pixel of the portion of hair is carried out by a graphics processing unit, called GPU, using a shader.

7. The method of claim 6, further comprising providing the lookup table to the shader using a 1D texture.

8. The method of claim 1, further comprising preprocessing the strand image or the input image using a deep neural network to improve the accuracy of the pixel coloring in the hair portion.

9. The method of claim 1, further comprising displaying an output image comprising the input image and the portion of hair as colored.

10. The method of claim 1, further comprising calculating the frequencies of each of the gray levels in the strand image; and calculating the frequencies of each of the gray levels in the hair portion of the input image.

Images