Method and system for virtual hair coloring

Abstract

A method and system for virtual hair fitting are provided. A mapping relationship between the sample image and the hair portion is established by matching the frequencies of gray levels from the sample image and the gray levels from the hair portion of the input image, where the gray levels of the sample image are associated with their respective sample color values. The pixels of the hair portion are colored based on the gray levels of the hair portion pixels and the sample color values ​​determined using the mapping relationship.

Description

[Technical Field]

[0001] Cross-reference / priority information This disclosure claims priority to U.S. Patent Application No. 18 / 109,310, filed on 14 February 2023, the entire contents of which are incorporated herein by reference. This disclosure also claims priority to FR 2305815, filed on 9 June 2023, the entire contents of which are incorporated herein by reference.

[0002] This disclosure relates to image processing and augmented reality, and more particularly to methods and systems for virtual hair coloring. [Background technology]

[0003] Virtual hair coloring aims to provide users with the ability to virtually recolor their hair on their device. Users can select a target shade they wish to apply, and the device outputs an image of the user with their hair colored in the selected shade. This can help users decide on a shade to color their own hair. Hair, like any other real organic object, tends to have many details and variations in color that need to be preserved during rendering to maintain realism. Color changes that remove such details will not appear realistic. Furthermore, representing a target color for virtual hair dye involves not only identifying a single target color, but also defining the entire range of shades that this particular target hair dye would have when applied to real hair.

[0004] Improved technology for virtual hair coloring is desired. [Overview of the Initiative]

[0005] A computer implementation method is provided which involves performing one or more steps on a processor. The method establishes a map relationship between a swatch image and hair by mapping the respective frequencies of gray levels from a swatch image to gray levels from hair in an input image, where the gray levels of the swatch image are associated with their respective swatch color values. The method then includes coloring the pixels of the hair based on the gray levels of the hair pixels and the swatch color values ​​determined using the map relationship.

[0006] This method may further include determining hair regions from an input image via a deep neural network.

[0007] Each frequency may represent the probability of each gray level in the sample image or hair area. The frequencies can also be represented as a histogram or a cumulative distribution function.

[0008] The method may further include calculating a mapping table that maps gray levels from the hair portion to sample color values.

[0009] Coloring the pixels of hair can be performed by a graphics processing unit (GPU) using a shader. A shader is software code executed by the GPU to render or modify an image. For example, a shader can be used to change any one or more of the light, darkness, and color levels of a pixel. The method may further include providing a mapping table to the shader using a 1D texture. The method may further include interpolating the mapping table.

[0010] The method may further include preprocessing a sample image or input image using a deep neural network to improve the accuracy of coloring the pixels in the hair area.

[0011] The method may further include displaying an output image that includes the input image and the colored hair portion.

[0012] The method may further include processing the output image using a guided filter.

[0013] The method may further include calculating the frequency of each gray level in the sample image and calculating the frequency of each gray level in the hair portion of the input image.

[0014] A further aspect of this disclosure provides a system comprising a virtual try-on (VTO) rendering pipeline having computational circuitry, which colors each pixel of the hair portion of an input image based on the respective color values ​​from the sample image, where each color value is selected from the sample image for each pixel using a mapping relationship between the gray value of each pixel and the gray value of the sample image associated with the respective color value. The system further comprises a user interface component for presenting an output image including the input image and the colored hair portion.

[0015] The system may further include a hair detection engine that has computational circuits for determining hair regions from an input image via a deep neural network.

[0016] VTO ​​can include shaders for coloring each pixel of the hair in the input image.

[0017] The system can further include a preprocessing component having a computing circuit for modifying the characteristics of a sample image using a deep neural network.

[0018] The system can further include a postprocessing component having a computing circuit for applying a guided filter to an output image.

[0019] According to a further aspect of the present disclosure, a computer-implemented method is provided that includes executing one or more steps on a processor. The method includes coloring each pixel of a hair portion of an input image based on respective color values from a sample image in a virtual try-on (VTO) rendering pipeline, where each color value is selected from the sample image using a mapping relationship between the gray value of each pixel and the gray value of the sample image associated with each color value for each pixel. The method further includes presenting an output image including the input image and the colored hair portion to a user interface.

[0020] The method can further include determining a hair portion from the input image via a deep neural network of a hair detection engine.

[0021] VTO can include a shader for coloring each pixel of a hair portion of an input image.

[0022] According to a further aspect of the present disclosure, a computer-implemented method is provided that includes executing one or more steps on a processor. The method includes selecting a swatch gray value from a sample image based on a probability of a hair gray value in a hair portion of an input image. The method further includes coloring pixels of the hair portion of the input image based on color values from the sample image associated with the swatch gray value.

[0023] These and other embodiments will be apparent to those skilled in the art. Such embodiments include: a) a system comprising at least one computing device having at least one processor and at least one memory, wherein at least one memory stores computer-executable instructions that, when executed by at least one processor, cause the system to perform the method according to any of the claims of the above-described method; and / or b) a computer program product comprising at least one non-temporary storage device, for example, at least one memory, wherein at least one non-temporary storage device includes a computer program product that, when executed by at least one processor of the system, causes the system to perform the method according to any of the claims of the above-described method. [Brief explanation of the drawing]

[0024] [Figure 1] This is a diagram of a computing environment according to one embodiment for performing virtual try-on. [Figure 2] This is a flowchart illustrating the operation of a computer implementation method according to one embodiment. [Figure 3] This is a typical histogram mapping diagram according to one embodiment. [Figure 4] A is a representative image of a face input image including hair according to one embodiment, and B is a representative image of a hair mask according to one embodiment. [Figure 5] This is a flowchart illustrating the operation of a computer implementation method according to one embodiment.

[0025]

[0026]

[0027]

[0028] [Modes for carrying out the invention]

[0029] Embodiments of this specification describe one or more methods, systems, apparatus and techniques for virtual hair coloring in virtual try-on (VTO) applications.

[0030] Hair VTO can provide users with the ability to virtually recolor their hair color on a device using either a camera (live video mode) or a picture (photo mode). Users can select a target shade, and the VTO processes the input (video stream or image) to apply the color change, outputting the processed image shown to the user. In live mode, this can occur several times per second on a mobile device, targeting 30 frames per second (FPS).

[0031] At a high level, a hair VTO may include the following steps: input, hair detection, and hair color modification. In the input step, an image may be provided to be color modified. In live mode, this image may be extracted from a camera's video stream. Hair detection can represent a method used to segment the hair in the image from the rest of the image (face, background, etc.) and export a hair mask to which the color modification will be applied. Hair color modification may also be a process of replacing each original pixel in the image with a new value from the target color.

[0032] Hair, like any other real organic material, tends to have many details and variations in color that need to be preserved during rendering to maintain realism. Color changes that remove such details will not look realistic. Furthermore, representing a target color for a virtual hair dye involves not only identifying a specific target color but also defining the entire range of shades that this particular target hair dye would have when applied to real hair. Typically, hair dye brands represent this type by using hair swatch images. These hair swatch images represent hair (real, synthetic, or virtual) to which the color change has been applied. Unfortunately, there is no market-wide standard for color representation of hair dyes, and each brand is free to use its own classification. The L'Oréal® Group has defined a standard for evaluating target shades. This standard uses notation such as "7,43". The first digit (7 here) represents a tone level ranging from 1 (dark) to 10 (blonde). The second number (here, 4) is optional and represents the primary reflect, defining the overall tone of the color on a scale of 0 to 7. The last number (here, 3) is optional and represents the secondary reflect, which is an additional color that can enhance, nuance, or reduce the intensity of the primary reflect, on a scale of 0 to 7.

[0033] However, this target color notation itself cannot capture all the details of the resulting color. Hair exhibits a great deal of variation in the resulting color. More information can be gleaned from the sample image than just tone, primary and secondary reflections. This variation in the resulting color stems from the properties of hair to reflect or capture ambient light, which are visible as dark and light areas in the image. Failure to represent these areas in a natural way will disrupt the illusion of color change, including following the direction in which natural hair strands are braided.

[0034] Hair VTOs are typically used to showcase a product and simulate realistic and accurate results when a user dyes their hair with the product. Mass-market hair dyes and salon dyes should be distinguished. Mass-market hair dyes are designed for application at home by non-professionals and result in shades that vary depending on the user's base color. Product packaging usually shows the resulting color when applied to different base colors. In contrast, salon dyes mean that the hair is treated before the dye is applied and applied by a hairdresser who knows how to achieve a specific target color (e.g., bleaching the hair before application). A successful VTO should be able to render both results: color variation depending on the user's natural hair color and a homogeneous color that covers natural hair.

[0035] Hair VTO has several constraints, including the following: Hair VTO should only change the color of the hair portion of the input image. The generated output image should be realistic. Hair VTO must be able to change hair color in real time (over 20 FPS) on mobile devices. This last constraint has the greatest impact on the techniques available for changing image color. While many techniques exist for changing image color, none of them are fast enough to be applied in real time on mobile devices.

[0036] One known technique involves shifting the hue (H), saturation (S), and lightness (L) of an input image. While this technique is easy to implement, it cannot significantly alter the brightness of the image, and therefore its ability to change the color of human hair is limited. Furthermore, because it uses only HSL values ​​and not a reference image, it cannot capture the full range of the target color.

[0037] Another known technique involves applying a translucent colored layer over the hair. This technique uses a hair mask to define a shape filled with a translucent texture representing the target color. While this technique is quick to implement and run, it is difficult to maintain a natural hair texture and requires reducing the layer's intensity, which prevents this technique from rendering vibrant colors.

[0038] VTO ​​application Figure 1 shows a computing environment 100 according to one embodiment for carrying out one or more methodological aspects. The computing environment 100 includes a user computing device 102 such as a smartphone, a communication network 104, a server 106, and a server 108. The communication network 104 may be public or private and may include, for example, the Internet, and includes wired and / or wireless networks. Server 106 comprises a server computing device for providing a website, etc. Server 108 comprises a server computing device for providing e-commerce services, etc. Although shown separately, servers 106 and 108 can be configured as a single server device. The computing environment 100 is simplified. For example, components not shown include a payment transaction gateway and other components for completing e-commerce transactions, etc.

[0039] The computing device 102 includes a storage device 110 (e.g., a memory and / or a non-transient device such as a solid-state drive) for storing instructions that cause the computing device 102 to perform operations such as computer implementation methods when executed by a processor (not shown). The storage device 110 stores a virtual try-on application 112, which is a component that provides software modules such as a user interface 114, a face tracker 104B having one or more deep neural networks 106B, a VTO rendering pipeline component 116, a product recommendation component 118 having product data 120, and a purchase component 122 having a shopping cart 124 (e.g., purchase data).

[0040] In one embodiment, the VTO application is a web-based application, such as one retrieved from server 106. Although not shown, user device 102 may store a web browser for running the web-based VTO application 112. In one embodiment, the VTO application is a native application according to an operating system (not shown), and software development requirements may be imposed, for example, by the hardware manufacturer of user device 102. The native application may be configured for web-based or similar communication with servers 106 and 108, as is known.

[0041] Figure 1 shows, for example, various input and output data or information related to the use of the VTO application 112. Such include a user input image 126 processed for the VTO experience, an output image 128 with simulated product effects to provide the VTO experience, a VTO product selection 130 including user input to select one or more product effects to simulate, a VTO product options 132 with options for products to be virtually tried on for selection by the user of device 102, and purchase transaction information 134 including purchase information provided to and / or received from the user for purchasing the product.

[0042] In one embodiment, VTO product options 132 are presented for selection for virtual try-on by simulating the effect on an input image 126 through one or more of the user interfaces 114. In one embodiment, the VTO product options 132 are obtained from or associated with product data 120. In one embodiment, product data may be obtained from a server 106 and provided by a product recommendation component 118. Although not shown, user or other input may be received for use in determining product recommendations. The user may be prompted to provide input for determining product recommendations, for example, through one of the interfaces 114. In one embodiment, the product recommendation component 118 communicates with the server 106. In one embodiment, the server 106 determines recommendations based on the input received through the component 118 and provides product data accordingly. The user interface 114 can present a selection of VTO products, and the same display of VTO products is updated, for example, in response to data received when the user browses the user interface or otherwise interacts with the user interface.

[0043] In one embodiment, one or more user interfaces provide commands and controls for acquiring an input image 126 and a VTO product selection input 130, such as the identification of one or more VTO products to try on. In one embodiment, products may be recommended to the user. In one embodiment, products may be selected by the user without being recommended themselves. That is, instances of products are presented from a product data store, etc., and the user can select an instance to virtually try it on. In one embodiment, the input image 126 may be, for example, a still image or a frame from a video, a user's face image including hair for a hair-related VTO. In one embodiment, the input image 126 may be received from a camera (not shown) of device 102 or from a stored image (not shown). The input image 126 is provided to a face tracker 104B for processing such as detecting objects in the input image 126 using one or more deep neural networks 106B. In one example, the network classifies, localizes, or segments the hair portion of the image.

[0044] In one embodiment, output from the face tracker 104B (not shown), such as classification, localization, or segmentation results for one or more detected objects, is provided to the VTO rendering pipeline component 116. In one example, the output may include a hair mask. An input image 126 is also provided to component 116 (e.g., made available). A VTO product selection 130 is also provided to component 116 to determine which effects should be rendered. In embodiments relating to makeup simulation, one or more effects may be shown, such as for any one or more product categories including lips, eyeshadow, eyeliner, blush, etc. In embodiments of a hair-only VTO, only hair effects are applied.

[0045] In one embodiment, the VTO rendering pipeline component 116 determines whether to render one or more product effects on the input image 126 to simulate trying on clothes. In embodiments such as those related to makeup, for example, depending on the facemask classification output, the VTO rendering pipeline component 116 may decide, for example, that a mask has been detected and therefore not to render the product effects on all or part of the face. If a facemask is detected, the VTO rendering pipeline component 116 may, for example, trigger the user interface 414 to prompt the user to remove the facemask. The new image may be received and processed by the face tracker 104B. In one embodiment, images are received sequentially as components of a live stream (e.g., a selfie video).

[0046] For example, in an embodiment where multiple product effects are applied to an input image, the VTO rendering pipeline component 116 renders the effects on the input image 126 (for example, on or over it) by drawing (rendering) the effects in layers, thereby generating an output image 128. Here, each product effect consists of one layer. Layering can be assisted by the use of overlays in some examples. In some examples, the pixel values ​​of the input image itself can be modified without overlaying.

[0047] Some of the operations of the VTO rendering pipeline component 116 (e.g., drawing layers) can be performed by a graphics processing unit in one embodiment. Rendering responds to the location of detected objects according to the product data 120 selected by the VTO product selection 130. For example, a VTO product selection for lipstick, lip gloss, or other lip-related products invokes the application of effects to one or more detected mouth or lip-related objects at their respective locations. Similarly, a product selection related to eyebrows invokes the application of selected product effects to detected eyebrow objects. Typically, for a symmetrical appearance, the same eyebrow effect is applied to each eyebrow, the same lip effect to each lip, or the same eye effect to each eye area, but this is not necessarily the case. In one example, rendering is applied to areas related to the detected objects, such as adjacent to one or more such detected objects. Some VTO product selections include selections of multiple products, such as coordinated products for eyebrows and eyes, or other combinations of detected objects. The VTO rendering pipeline component 416 can render each effect one at a time, for example, until all effects have been applied. The order of application can be defined by rules or by the product, for example, in the selection of lipstick before top gloss.

[0048] In embodiments where an occluding object is detected and its position is determined to be represented, for example, by a segmentation mask, rendering can respond to such a segmentation mask. Rendering of an effect can be applied to the unoccluded portion of the face. The segmentation mask can indicate which face pixels can receive (for example, potentially receive) an effect, such as a makeup effect, and which pixels cannot receive the effect.

[0049] The user interface 114 provides an output image 128. In one embodiment, the output image 128 is presented as part of a live stream of consecutive output images (each example 128), such as when a selfie video is augmented to present an augmented reality experience. In one embodiment, the output image 128 is presented together with the input image 126, such as when it is displayed side by side for comparison. In one embodiment, the output image 128 can be stored in a storage device 110 or the like (not shown) and / or shared with another computing device (not shown).

[0050] In one embodiment, the input image (not shown) includes an input image of a video conferencing session, and the output image includes a video shared with another participant(s) of the video conferencing session. In one embodiment, the VTO application is a component or plug-in of a video conferencing application (not shown) that allows the user of device 102 to wear makeup during a video conference with one or more other meeting participants.

[0051] Hair VTO Refer to Figure 2, which shows a computer implementation method 200 that includes performing one or more steps on a processor. Method 200 includes establishing a map relationship between the sample image 136 and the hair portion by mapping the respective frequencies of gray levels from the sample image 136 to the gray levels from the hair portion of the input image 126, where the gray levels of the sample image 136 are associated with their respective sample color values.

[0052] The gray levels of the hair portion in the sample image 136 and the input image 126 can be mapped by their frequencies (i.e., the most frequent gray level in the sample image 136 can be mapped to the most frequent gray level in the input image 126). A mapping relationship is established between the two images. Each frequency may be the probability of each gray level in the sample image 136 or the hair portion. A color mapping is established that associates any color in the image with a gray level. Thus, the distribution of gray levels in the input image 126 can be used as a lookup index to find the corresponding color in the sample image 136.

[0053] The frequency can be represented as a histogram or a cumulative distribution function. The cumulative distribution function (CDF) is the cumulative probability of a gray value (between 0 and 255) appearing in an image, normalized to the total number of pixels in the image to obtain a result between 0.0 and 1.0. Color is represented using RGB, where each color is represented by three values ​​between 0 and 255. Other methods of representing color can also be used. The CDF can be represented as a histogram. Method 200 may include calculating the frequency of gray levels in the hair portion of the input image 126. Method 200 may include calculating the frequency of gray levels in the sample image 136. Method 200 may include calculating the frequency of the sample color values ​​in the sample image 136 and associating the sample color values ​​of the sample image 136 with gray values ​​based on that frequency.

[0054] CDF swatch This may also be a cumulative gray histogram of sample image 136, normalized between 0.0 and 1.0. CDF user This may also be a cumulative gray histogram of the hair portion of input image 126, normalized between 0.0 and 1.0. swatch,cIt may be a histogram for each channel of the sample, where c is {r, g, b}, and each bin is normalized between 0 and 255.

[0055] The objective is to find the gray value in the reference image 136(G user ) corresponding to the gray value in the hair part of the input image 126(G swatch ). That is, the objective is to find G user such that CDF user (G swatch ) = CDF swatch (G swatch ). This can be done by calculating G swatch = CDF -1 swatch (CDF user (G user )). Mathematically, unless the CDF is strictly monotonically increasing, there is no inverse function. However, when a Y value maps to multiple X values, the inverse function can be defined by selecting the smallest X value among them. Refer to FIG. 3, which shows the mapping relationship between two histograms. The first histogram 300 represents the distribution of gray values in the hair part of the input image 126. The second histogram 320 represents the distribution of gray values in the reference image 136. The arrow 310 represents the mapping relationship for a specific gray value. The gray value with a specific frequency or probability in the hair part of the input image 126 can be mapped to the gray value with the same frequency or probability in the reference image 136. Then, G swatch can be used to index H swatch,r , H swatch,g , H swatch,b , thereby obtaining the color value associated with that gray value in the reference image 136. The function H swatch (CDF -1 swatch (CDF user (G user ))) provides the mapping relationship between the gray value in the hair part of the input image 426 and the color value in the reference image 136.

[0056] Method 200 further includes coloring the pixels of the hair portion based on a sample color value determined using the gray level and mapping relationship of the pixels of the hair portion. For each pixel of the hair portion in the input image 126, mapping relationship H swatch (CDF -1 swatch (CDF user (G user Using ), the sample color of sample image 136 can be looked up. Then, this sample color can be used to color the pixels of the hair portion of input image 126. This sample color can be blended with the initial pixel color (depending on the desired result and lighting conditions). As a result, the hair portion of input image 126 may be colored to resemble the hair color in sample image 136.

[0057] Method 200 may further include determining hair portions from an input image 126 via a deep neural network 106B. Next, we refer to Figure 4A, which shows a representative example of an input image 126 representing a user's face. The input image 126 may comprise a face portion 402, a neck portion 404, a hair portion 406, and a background portion 408. Next, we refer to Figure 4B, which shows an example of a hair mask 412 generated by the neural network 106B. The neural network 106B takes the input image 126 as input and identifies the hair portion 406 from the rest of the input image 126 to generate a hair mask 412. The hair mask 412 comprises two parts, namely the hair portion 416 and all other parts 418. The hair mask 412 can be distinguished by coloring the two parts 416 and 418 with different colors, such as white and black. The hair mask 412 can be used to identify pixels of the input image 126 that belong to the hair portion 406.

[0058] Method 200 may further include calculating a mapping table that maps gray levels from hair portion 406 to sample color values. The mapping table can map gray values ​​to sample color values. The mapping table may be calculated by using mapping relationships to identify a sample color value for each possible gray value. For example, if RGB colors are used, the mapping table may be an array containing RGB elements and indexed by gray values. That is, the array contains 256 elements, each containing an RGB sample color value indexed by a gray value. Coloring pixels from hair portion 406 of input image 126 may include determining the gray values ​​of pixels from hair portion 406, using the gray values ​​as lookup indexes in the mapping table, and changing the color of pixels in hair portion 106 using the sample color values ​​corresponding to the indexes specified in the mapping table.

[0059] Coloring the pixels of the hair can be performed by a graphics processing unit (GPU) using a shader. Using a GPU can improve the efficiency of coloring the hair 406, for example, for real-time coloring of the hair 406 in a video stream. Method 200 may further include providing a mapping table to the shader using a 1D texture. The table can be stored in a 1D texture and passed to the shader as a uniform. A uniform is a type of variable in the shader and is available at any stage of the rendering pipeline in the GPU. This makes it possible to use Method 200 on some devices that do not support array input of the size of the mapping table. Method 200 may further include interpolating the mapping table. Since the GPU can interpolate textures, the lookup table is interpolated to smooth the resulting color of each pixel, providing a more accurate and realistic output image 128.

[0060] Method 200 may further include preprocessing the sample image 136 or input image 126 using a deep neural network 106B to improve the accuracy of pixel coloring in the hair portion 106. The resulting color accuracy in the output image 128 may depend on the quality of the input sample image 136. Unfortunately, there is no quality standard for generating the sample image 136, and each brand is free to modify the image for its own purposes. To obtain more consistent results, several parameters of the sample image 136 can be modified: Brightness Scale: Controls the overall brightness effect, making all pixels in an image brighter or darker. Brightness Boost: Controls the brightness boost to brighten bright pixels but not dark pixels. Contrast Scale: Controls the overall contrast effect. Saturation Boost: Controls the saturation of an image by further enhancing saturated colors without changing unsaturated colors. Brightness Cut-off: Controls the cutoff of the brightest value in the histogram data (similar to a low-pass filter). Darkness Cut-off: Controls the cutoff of the darkest values ​​in the histogram data (similar to a high-pass filter). Hue Rotate: Slightly rotates the hue of the original image. The accuracy of hair rendering can be improved by adjusting these parameters before extracting histogram data from sample image 136. These parameters can be adjusted manually. Alternatively, a neural network (e.g., one of the networks 106B, or another) can be trained to adjust these parameters.

[0061] Method 200 may further include displaying an output image 128 that includes an input image 126 and a colored hair portion 406. The output image 128 may be displayed, for example, on the user interface 114 of a user computing device 102.

[0062] Method 200 may further include processing the output image 128 using a guided filter. When extracting pixels corresponding to the hair portions 406 of the input image 126 to create a hair mask 412, the neural network 106B may include rough edges with a large amount of alpha transparency. This problem can result in color bleeding into the immediate environment surrounding the hair and can produce a halo effect. Several image processing techniques can be used to correct this problem, but most of them introduce other artifacts and / or are too slow to be applied to video stream processing at frame rates compliant with live VTO. One solution is to recalculate better edges by using a guided filter as a matting solution as a post-processing step (after the neural network model 106B has been executed). By using the original input image 126 as a guide for the hair mask image 412, the guided filter can smooth the edges of the hair mask 412 while limiting color bleeding outside the original hair positions.

[0063] Method 200 may further include calculating the frequency of each gray level in the sample image 136 and the frequency of each gray level in the hair portions 406 of the input image 126. The frequencies may include histogram data or CDF. The frequencies may be converted into probabilities ranging from 0 to 1. The frequencies of the sample image 136 may be calculated once each time the sample image 136 is selected. In a real-time video stream, the frequencies of the input image 126 may be calculated frame by frame.

[0064] Method 200 may be implemented on System 100. System 100 comprises a user computing device 102 having a VTO rendering pipeline 116 having a calculation circuit for coloring each pixel of the hair portion 406 of the input image 126 based on each color value from the sample image 136, where each color value is selected from the sample image 136 for each pixel using the gray value of each pixel and a mapping relationship to the gray value of the sample image 136 associated with each color value. The user computing device 102 of System 100 may further comprise a user interface component 114 for presenting an output image 128 including the input image 126 and the colored hair portion 406.

[0065] System 100 may further include a hair detection engine comprising a computational circuit for determining hair portions 406 from an input image 126 via a deep neural network 106B. The hair detection engine may be a component of a user computing device 102 or a server 106 / 108 (not shown).

[0066] The user computing device 102 may be equipped with a GPU, and at least some of the VTO rendering pipeline 116 may be executed by the GPU. In particular, the GPU's shaders can color each pixel of the hair portion 406 of the input image 126.

[0067] System 100 may further include a preprocessing component comprising a computational circuit for modifying the characteristics of a sample image 136 using a deep neural network 106B. The preprocessing component may be a component of a user computing device 102 or a server 106 / 108.

[0068] The system 100 may further include a post-processing component comprising a calculation circuit for applying a guided filter to the output image 128.

[0069] Server 108 may be equipped with a CMS data store for storing sample images 136 and frequency or histogram data 138 for sample images 136. Servers 106 / 108 can calculate the frequency or histogram data of sample images 136. Alternatively, user computing device 102 may calculate the frequency or histogram data of sample images 136 and input images 126. Servers 106 / 108 can calculate the frequency or histogram data of input images 126.

[0070] Herein, we refer to Figure 5, which shows a computer implementation method 500 that includes performing one or more steps on a processor. Method 500 includes coloring each pixel of the hair portion 406 of the input image 126 in a VTO rendering pipeline 116 based on the respective color values ​​from a sample image 136, where each color value is selected from the sample image 136 for each pixel using the gray value of each pixel and the mapping relationship to the gray value of the sample image 136 associated with each color value. Method 500 may further include presenting an output image 128, including the input image 126 and the colored hair portion 406, to a user interface 114.

[0071] Method 500 may further include determining hair portions 406 from an input image 126 via a deep neural network 106B of a hair detection engine.

[0072] In another embodiment, the computer implementation method includes performing one or more steps on a processor to select a sample gray value from a sample image 136 based on the probability of the hair gray value in the hair portion 406 of the input image 126. The method further includes coloring the pixels of the hair portion 406 of the input image 126 based on the color values ​​from the sample image 136 associated with the sample gray value.

[0073] The corresponding system embodiments are disclosed for each embodiment of the method disclosed herein, and it will be understood that the system comprises, for example, components having computing circuits configured to perform the operation of the computer implementation method embodiment.

[0074] Those skilled in the art will understand that, in addition to embodiments of computing devices and methods, embodiments of computer program products are disclosed in which instructions are stored in a non-temporary storage device (e.g., memory, CD-ROM, DVD-ROM, disk, etc.) and, when executed, cause the computing device to execute one of the embodiments of the methods stored therein.

[0075] Computing devices are described with reference to processors and instructions that cause the computing device to perform operations when executed, but it is understood that other types of circuits other than programmable processors may be configured. Hardware components with specifically designed circuits, such as application-specific integrated circuits (ASICs) or other hardware (not limited to these), designed to perform specific functions, may be used, and these may be more efficient than general-purpose central processing units (CPUs) programmed with software. Thus, in general, embodiments of devices in this specification relate to systems or devices having circuits (sometimes referred to as computing circuits) configured to perform some of the operations described herein, whether the circuits are configured through programming or through hardware design, etc., including but not limited to embodiments of the methods described herein.

[0076] Practical implementations may include any or all of the features described herein. These and other embodiments, features, and various combinations may be represented as methods, apparatus, systems, means, and other ways of combining the features described herein for performing a function. Several embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps can be provided or steps can be excluded from the described processes, and other components can be added to or removed from the described systems. Thus, other embodiments are within the scope of the claims.

[0077] Throughout this specification and in the claims, the words “comprise” and “contain” and their variations mean “including but not limited to” and are not intended to exclude other components, integers, or steps. Throughout this specification, the singular form includes the plural form unless the context requires otherwise. In particular, where the indefinite article is used, it should be understood that both singular and plural are intended unless the specification requires otherwise.

[0078] Features, integer properties, compounds, chemical parts, or groups described in relation to specific aspects, embodiments, or examples of the present invention should be understood to be applicable to any other aspects, embodiments, or examples unless they are incompatible therewith. All features disclosed herein (including any appended claims, abstract, and drawings), and / or all steps of any method or process so thus disclosed, can be combined in any combination except for any combination in which at least some of such features and / or steps are mutually exclusive. The present invention is not limited to the details of any examples or embodiments described herein. The present invention extends to any novel one or any novel combination of features disclosed herein (including the appended claims, abstract, and drawings), or any novel one or any novel combination of any steps of any method or process disclosed. <Other> <Means> The computer implementation method of technical idea 1 is a computer implementation method that performs one or more steps on a processor, which establishes a map relationship between the sample image and the hair portion by mapping the respective frequencies of gray levels from a sample image and gray levels from the hair portion of an input image, where the gray levels of the sample image are associated with their respective sample color values, and the pixels of the hair portion are colored based on the gray levels of the pixels of the hair portion and the sample color values ​​determined using the map relationship. The method of technical idea 2 further includes determining the hair portion from the input image via a deep neural network, in the method described in technical idea 1. The method of technical idea 3 is the method described in technical idea 1, wherein each of the frequencies is the probability of each of the gray levels in the sample image or the hair portion. In the method of technical idea 4, as described in technical idea 1, the frequency is represented by a histogram or a cumulative distribution function. The method of technical idea 5 further includes, in the method described in technical idea 1, calculating a mapping table that maps the gray level from the hair portion to the sample color value. In the method of technical idea 6, as described in technical idea 5, the coloring of the pixels of the hair portion is performed by a graphics processing unit (GPU) using a shader. The method of technical idea 7 further includes providing the mapping table to the shader using a 1D texture, in the method described in technical idea 6. The method of technical idea 8 further includes interpolating the mapping table in the method described in technical idea 7. The method of technical idea 9 further includes, in the method described in technical idea 1, preprocessing the sample image or the input image using a deep neural network in order to improve the accuracy of the coloring of the pixels in the hair portion. The method of technical idea 10 further includes displaying an output image that includes the input image and the colored hair portion, in the method described in technical idea 1. The method of technical idea 11 further includes processing the output image using a guided filter in the method described in technical idea 10. The method of technical idea 12 further includes, in the method described in technical idea 1, calculating the frequency of each of the gray levels in the sample image and calculating the frequency of each of the gray levels in the hair portion of the input image. The system of technical idea 13 comprises a virtual try-on (VTO) rendering pipeline having a calculation circuit that colors each pixel of the hair portion of an input image based on each color value from a sample image, where each color value is selected from the sample image for each pixel using a mapping relationship between the gray value of each pixel and the gray value of the sample image associated with each color value, and further comprises a user interface component for presenting an output image including the input image and the colored hair portion. The system of technical concept 14 further comprises a hair detection engine, which is a computational circuit for determining the hair portion from the input image via a deep neural network, in the system described in technical concept 13. The system of technical idea 15, in the system described in technical idea 13, includes a shader that colors each of the pixels of the hair portion of the input image. The system of technical concept 16 further comprises a preprocessing component that includes a computational circuit for modifying the characteristics of the sample image using a deep neural network, in addition to the system described in technical concept 13. The system of technical concept 17 further comprises a post-processing component in the system described in technical concept 13, which includes a calculation circuit for applying a guided filter to the output image. A computer implementation method of technical idea 18 is a computer implementation method that performs one or more steps on a processor, further comprising: in a virtual try-on (VTO) rendering pipeline, coloring each pixel of the hair portion of an input image based on each color value from a sample image, wherein each color value is selected from the sample image for each pixel using a mapping relationship between the gray value of each pixel and the gray value of the sample image associated with each color value, and presenting an output image including the input image and the colored hair portion to a user interface. The method of technical idea 19 further includes determining the hair portion from the input image via a deep neural network of a hair detection engine, in the method described in technical idea 18. The method of technical idea 20, in the method described in technical idea 18, comprises a shader that colors each of the pixels of the hair portion of the input image. The system of technical idea 21 is a system comprising at least one computing device having at least one processor and at least one memory, wherein the at least one memory stores computer executable instructions that, when executed by the at least one processor, cause the system to perform a method according to any of the technical ideas of the methods described above. The computer program product of technical idea 22 is a computer program product comprising at least one non-temporary storage device, preferably at least one memory, wherein the at least one non-temporary storage device, when executed by at least one processor of the system, stores computer executable instructions causing the system to perform a method according to any of the technical ideas of the methods described above.

Claims

1. A computer implementation method for performing one or more steps on a processor, The frequency of gray levels from the sample image and the gray levels from the hair portion of the input image are matched and mapped to establish a mapping relationship between the sample image and the hair portion, where the gray levels of the sample image are associated with their respective sample color values. A computer implementation method characterized by coloring the pixels of the hair portion based on a sample color value determined using the gray level of the pixels of the hair portion and the map relationship.

2. The method according to claim 1, further comprising determining the hair portion from the input image via a deep neural network.

3. The method according to claim 1, characterized in that each of the frequencies is the probability of each of the gray levels in the sample image or the hair portion.

4. The method according to claim 1, characterized in that the frequency is represented by a histogram or a cumulative distribution function.

5. The method according to claim 1, further comprising calculating a mapping table that maps the gray level from the hair portion to the sample color value.

6. The method according to claim 5, characterized in that the coloring of the pixels of the hair portion is performed by a graphics processing unit (GPU) using a shader.

7. The method according to claim 6, further comprising providing the mapping table to the shader using a 1D texture.

8. The method according to claim 7, further comprising interpolating the mapping table.

9. The method according to claim 1, further comprising preprocessing the sample image or the input image using a deep neural network in order to improve the accuracy of coloring the pixels of the hair portion.

10. The method according to claim 1, further comprising displaying an output image including the input image and the colored hair portion.

11. The method according to claim 10, further comprising processing the output image using a guided filter.

12. Calculating the frequency of each of the aforementioned gray levels in the aforementioned sample images, The method according to claim 1, further comprising calculating the frequency of the gray level in each of the hair portions of the input image.

13. It is a system, The system includes a virtual try-on (VTO) rendering pipeline having a calculation circuit that colors each pixel of the hair portion of the input image based on each color value from the sample image, where each color value is selected from the sample image for each pixel using the mapping relationship between the gray value of each pixel and the gray value of the sample image associated with each color value. The system further comprises a user interface component for presenting an output image that includes the input image and the colored hair portion.

14. The system according to claim 13, further comprising a hair detection engine that includes a computational circuit for determining the hair portion from the input image via a deep neural network.

15. The system according to claim 13, wherein the VTO comprises a shader that colors each of the pixels of the hair portion of the input image.

16. The system according to claim 13, further comprising a preprocessing component that includes a computational circuit for modifying the characteristics of the sample image using a deep neural network.

17. The system according to claim 13, further comprising a post-processing component that includes a calculation circuit for applying a guided filter to the output image.

18. A computer implementation method for performing one or more steps on a processor, In the virtual try-on (VTO) rendering pipeline, each pixel of the hair portion of the input image is colored based on the respective color values ​​from the sample image, where each color value is selected from the sample image for each pixel using the mapping relationship between the gray value of the respective pixel and the gray value of the sample image associated with that color value. A computer implementation method further characterized by presenting an output image, which includes the input image and the colored hair portion, to a user interface.

19. The method according to claim 18, further comprising determining the hair portion from the input image via a deep neural network of a hair detection engine.

20. The method according to claim 18, characterized in that the VTO includes a shader that colors each of the pixels of the hair portion of the input image.

21. A system comprising at least one computing device having at least one processor and at least one memory, wherein the at least one memory stores computer-executable instructions that, when executed by the at least one processor, cause the system to perform the method according to any of the claims of the above-described method.

22. A computer program product comprising at least one non-temporary storage device, preferably at least one memory, wherein the at least one non-temporary storage device, when executed by at least one processor of the system, stores computer executable instructions causing the system to perform the method according to any of the claims of the above-described method.

Images