HAIR COLOR SIMULATION USING A GUIDED HAIR COLOR CLASSIFICATION NETWORK

Abstract

HAIR COLOR SIMULATION USING A GUIDED HAIR COLOR CLASSIFICATION NETWORK Aspects of hair simulation and corresponding networks are provided, including aspects regarding the training of such networks. A generative hair simulation model is provided and guided during its training by a hair classifier model. The generative model, in one embodiment, is provided for use in a virtual try-on (VTO) pipeline such as for virtually trying on hair color products. A color mapping network is also provided for processing an input image and a target hair color for the generative model to define the hair simulation (e.g., as an output image with simulated hair color). Figure for abstract: none

Description

Title of the invention: HAIR COLOR SIMULATION USING A GUIDED HAIR COLOR CLASSIFICATION NETWORK FIELD OF THE INVENTION

[0001] The present disclosure relates to computer image processing and artificial intelligence, including systems and methods for hair color simulation and hair color classification and, more particularly, to a method, device and system for hair color simulation using a guided hair color classification network. BACKGROUND

[0002] Due to the growing popularity of online shopping, e-commerce companies are exploring ways to improve the customer experience. Virtual try-on technology (VTO) has been developed to help customers search for and experience products virtually, for example, to find the best fit. In some examples of VTO, it may be useful to determine information, such as from a user-provided image. For example, in a VTO related to hair color, a facial image (e.g., a portrait) may be processed to determine, by classification, the hair color in the portrait. [Fig. 1] represents a block diagram of a representative processing flow 100 that shows a general model in the form of a classifier 102 that receives a color image of the face 104 (e.g., a portrait) and identifies a hair color 106, in accordance with the prior art.The output, along with the color image of the face, can then be used for a virtual hair dye trial.

[0003] Several challenges arise when developing a hair color classification model. For example, color labels collected from hair experts are subject to human bias and labeled data is expensive to obtain. It is desired to have an improved hair classification model that can identify hair color.

[0004] A VTO pipeline may simulate hair color, for example, using a generative neural network. Improvements in simulation using generative networks are desired. SUMMARY

[0005] Aspects of hair simulation and corresponding networks are proposed, including aspects concerning the training of such networks. A model A generative hair simulation model is provided and guided during its training by a hair classifier model. The generative model, in one embodiment, is provided for use in a virtual try-on pipeline (VTO), e.g., for virtually trying on hair color products. A color mapping network is also provided for processing an input image and a target hair color for the generative model to define the hair simulation (e.g., as an output image with hair color simulation).

[0006] Aspects of VTO pipelines and corresponding networks are provided, including aspects regarding training such networks. Further, in accordance with one embodiment, a classifier model is provided to mitigate the impact of human biases where modeling of the actual label distribution and annotator biases are separated by incorporating annotator confusion matrices into a baseline model. To further improve the model's performance by leveraging unlabeled data, the model was trained using a consistency-based semi-supervised learning framework. Using only 1,000 labeled data, the final classifier model achieved 20% better classification accuracy than a professional human annotator.The trained model can be used for a wide range of downstream tasks, including as a color classifier to train generative models for hair color translation.

[0007] The following statements present various aspects and features disclosed in the embodiments herein. These and other aspects and features will be readily understood by those skilled in the art, including aspects of computer program products. It is also understood that aspects / features of computer devices or systems may have corresponding method aspects / features and vice versa.

[0008] Embodiment 1: A computing device comprising a processor coupled to a storage device that stores instructions executable by the processor to cause the computing device to: process an input image and a target hair color using a virtual try-on (VTO) pipeline to produce a VTO experiment that simulates the target hair color with the input image to produce an output image; wherein the VTO pipeline comprises a color refining neural network for generating the output image combining the target hair color and the input image, the color refining neural network comprising a generative neural network trained under the supervision of a hair classification network comprising a hue classifier that outputs a hair hue value and a reflectance classifier that outputs a value hair reflectance to determine training loss information from output images to train the generative neural network.

[0009] Embodiment 2: A computing device according to embodiment 1, wherein the instructions are executable by the processor to cause the computing device to provide an interface with one or both of: i) a color recommendation engine for recommending a target hair color and ii) an e-commerce service with which to purchase a product or service or both.

[0010] Embodiment 3: A computing device according to embodiment 1, wherein the VTO pipeline comprises a color mapping network trained to produce a color map for input to the generative neural network to produce the output image for the VTO experiment, the color map produced from features of the image, image hair color data determined from the image, and the target hair color.

[0011] Embodiment 4: A computing device according to embodiment 3, wherein the color mapping network comprises an encoder for determining features from the input image; a concatenator for combining the features, the target hair color, and the image hair color data for processing by a first fully connected block, and a second block for processing an intermediate map from the first block with the image hair color data to produce the color map for input to the generative neural network.

[0012] Embodiment 5: A computing device according to embodiment 4, wherein the VTO pipeline comprises a hair analysis engine for determining image hair color data from the input image.

[0013] Embodiment 6: A computing device according to embodiment 1, wherein the input image and the target hair color are defined using RGB values, such that the generative neural network is trained to produce an output image using particular RGB values ​​as guided during training by the hair classification network using hair hue values ​​and reflectance values.

[0014] Embodiment 7: A computing device according to embodiment 6, wherein the hair classification network is defined and trained according to an industry standard for hair color classification comprising a respective plurality of classes for hair hue and hair reflectance.

[0015] Embodiment 8: A computing device according to embodiment 1, wherein the hair classification network comprises a coding skeleton and the hair hue classifier and the hair reflectance classifier each comprise a respective linear classifier.

[0016] Embodiment 9: A computing device according to embodiment 8, wherein the hair classification network has been defined by training with a sum of hue and reflectance cross-entropy losses determined from i) outputs of hair hue values ​​and hair reflectance values, and ii) target labels for hair hue and hair reflectance, the target labels being prepared for the training images of the hair classification network from respective expert votes by a plurality of experts.

[0017] Embodiment 10: A computing device according to embodiment 9, wherein the target labels comprise soft labels for at least some of the training images, the soft label being determined from an empirical distribution of expert votes on the respective classes.

[0018] Embodiment 11: Computing device according to embodiment 9, wherein one of the following: a hair classification network having been trained with respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of reflectance annotator confusion matrices, wherein n is defined from a total number of experts providing the expert votes, and wherein the output of the hue classifier is multiplied by a respective one of the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective one of the plurality of matrices to predict the vote of a respective one of the plurality of experts and wherein the neural network and the respective annotator confusion matrices are trained together or,The hair classification network was trained according to a Mean Teacher framework.

[0019] Embodiment 12: A method of configuring a neural network that classifies hair hue and hair reflectance in an input image, the method comprising: providing the neural network, the neural network comprising a coding skeleton coupled to i) a hue classifier that outputs a hair hue value and ii) a reflectance classifier that outputs a hair reflectance value, each classifier comprising a linear classifier; providing a plurality of training images associated with respective training labels for each of the hair hue and the hair reflectance, the target labels being prepared from respective expert votes by a plurality of experts;and training the neural network using the training images, the training being performed according to a sum of hue and reflectance cross-entropy losses determined from i) the classifier outputs of the hair hue values ​​and the hair reflectance values ​​and ii) the target labels. ;

[0020] Embodiment 13: The method of embodiment 12, wherein the training comprises training the neural network using a Mean Teacher framework.

[0021] Embodiment 14: The method of embodiment 12, wherein the training comprises training the neural network using the respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of reflectance annotator confusion matrices, wherein n is defined from a total number of experts providing the expert votes, and wherein the output of the hue classifier is multiplied by a respective one of the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective one of the plurality of reflectance confusion matrices to predict the vote of a respective one of the plurality of experts and wherein the neural network and the respective annotator confusion matrices are trained together.

[0022] Embodiment 15: A method comprising: providing a generative neural network that generates an output image combining a target hair color with an input image; providing a hair classification network as a component of a discriminator of the generative neural network, the hair classification network comprising i) a hue classifier that determines an output hair hue and ii) a reflectance classifier that determines an output hair reflectance; determining training loss information using the target hair color and the output hair hue, and the output hair reflectance; and training the generative neural network under the direction of the hair classification network using the training loss information.

[0023] Embodiment 16: The method of embodiment 15, wherein the target hair color, the input image, and the output image are defined using RGB type data and the output hair hue and the output hair reflectance are defined in accordance with an industry standard for hair color classification comprising a respective plurality of classes for hair hue and hair reflectance.

[0024] Embodiment 17: A method according to embodiment 15 comprising, prior to training the generative neural network, pre-training a color mapping network configured to process the input image and image hair color data from hair pixels of the input image and the target hair color to provide a color map to the generative neural network to generate the output image from the input image. Brief description of the drawings

[0025] [Fig-1] [Fig. 1] is a block diagram showing a classifier model according to the prior art.

[0026] [Fig.2] [Fig.2] is a block diagram showing a classifier model of reference in accordance with an embodiment.

[0027] [Fig.3] [Fig.3] is a block diagram showing the classifier model reference of [Fig.2] as adapted with annotator confusion matrices in accordance with one embodiment.

[0028] [Fig.4] [Fig.4] is a diagram of an annotator confusion matrix representative.

[0029] [Fig.5] [Fig.5] is a block diagram showing the classifier model of reference [Fig.2] as adapted with a Mean Teacher frame in accordance with one embodiment.

[0030] [Fig.6] [Fig.6] is a graph, in accordance with one embodiment, showing a comprehensive comparison of model prediction accuracy results between all models and an average human expert.

[0031] [Fig.7A] [Fig.7A] is a graph showing, according to one embodiment, an accuracy comparison of different numbers of training selfie images

[0032] [Fig.7B] [Fig.7B] is a graph showing, in accordance with one embodiment, a comparison of accuracy of different numbers of annotators ([Fig.7B]).

[0033] [Fig.8] [Fig.8] is a block diagram of a computer device, in accordance with one embodiment.

[0034] [Fig.9A] [Fig.9A] is a block diagram of network systems, in accordance with one embodiment.

[0035] [Fig.9B] [Fig.9B] is a block diagram of network systems, in accordance with one embodiment.

[0036] [Fig.9C] [Fig.9C] is a block diagram of network systems, in accordance with one embodiment. DETAILED DESCRIPTION

[0037] The systems, methods, and techniques herein are intended to improve existing hair color testing technology by leveraging artificial intelligence (AI). Many current hair simulation efforts are primarily based on traditional computer vision engineering techniques, and sometimes the hair texture in simulated images appears synthetic—e.g., overtly simulated and poorly simulated. The current applicant seeks to using AI to improve the photorealism of simulated hair images while maintaining color accuracy on the simulated hair. A primary objective is color accuracy, and a hair color classification model is described herein in one or more embodiments, for identifying hair color in selfies.

[0038] According to one embodiment, hair color is defined based on L'Oréal's hair color definition for hair dye products and classifies hair color based on 1) hue and 2) reflectance, namely primary and secondary reflectance. Hair shades are divided into 10 classes, with 1 being the lightest and 10 being the darkest. In one embodiment, there are 9 reflectance classes for each of primary reflectance and secondary reflectance, where primary reflectance indicates the dominant reflectance in the hair, and secondary reflectance indicates the second dominant reflectance. Hair may have only primary reflectance or no reflectance. Considering no reflectance as a separate reflectance class, there are 100 combinations of primary and secondary reflectances.Examples0: no primary or secondary reflectance, gold primary reflectance without secondary reflectance, ash primary reflectance without secondary reflectance, gold primary reflectance with ash secondary reflectance and gold primary reflectance with mahogany secondary reflectance, etc.

[0039] Labeled datasets from hair color experts exist or can be obtained for hair color. However, such datasets, particularly their use, present challenges for building a hair color identification model. For example, hair colors are defined based on the judgment of professional hair experts instead of scientific definitions of color, such as the average RGB color. Therefore, the existing definition of color is subject to human bias. Since the annotations were provided by hair experts, the labeled data is expensive to obtain and may result in a limited number of labeled examples with which to perform training and verification, etc.

[0040] It was desirable to develop a hair color classifier to produce color predictions at least as accurate as those of an average human hair expert. Annotations of 1,000 selfies from 10 hair experts were received to define a dataset to be used to develop the model - the "expert dataset". Model performance was evaluated based on whether it matched the majority of the hair experts' annotations or whether the predictions deviated from the expert annotations.

[0041] The expert dataset was relatively small, especially since the data were obtained gradually as the experts performed their tasks at different times. Furthermore, the combined results illustrated the differences in expert opinion. 1,000 labeled images is a relatively small dataset given that there were over 100 hair colors. After gradually receiving labels from other hair experts, it was determined that the labels from different hair experts often did not agree with each other. In some cases, there was no clear majority vote on hair color, and all 10 experts believed the hair was different colors. Therefore, one of the goals of the activities was to mitigate the impact of label noise.

[0042] To address these two challenges, a final model according to one embodiment includes two improvements on a standard (prior art) classifier (e.g. of [Fig.l]), by adding annotator confusion matrices and by using a Mean Teacher semi-supervised learning framework. The addition of annotator confusion matrices aims to mitigate the problem of label noise by modeling the biases of individual experts, while the Mean Teacher framework is a semi-supervised learning technique that mitigates the problem of limited labeled data by using unlabeled data during training. Label noise

[0043] In the real world, data are often labeled by multiple annotators. Since biases in human annotators are inevitable, these labels often contain noise, and annotators may disagree with each other. Studies show that the rate of corrupted labels in real datasets was in the range of 8% to 39%

[19]

[16] [9] [8]. A classic way to deal with label noise is to aggregate annotations using majority voting. However, the majority voting method has some limitations, for example, it only uses the class with the largest number of votes as the aggregated label and ignores the distribution of votes, and there may be scenarios where there is no clear majority vote. Another way to aggregate annotations is to use soft labels in which the values ​​of soft labels are based on the empirical distribution of votes.

[0044] Model-based approaches have also been explored to handle label noise, including using a translation matrix to measure noise and reweighting labels based on confidence in them. The transition matrix is ​​widely used to model the relationship between the true distribution and the observed distribution [5]

[13] . However, these methods often do not take into account that different annotators may have different different biases. On the other hand, loss reweighting has also been studied to improve model robustness by assigning more weight to labels or models with higher confidence

[10] . In particular, Weighted Doctor Net [4] proposed to model individual annotators separately and then combine the predictions of individual models using training-resulting weights. This method takes into account the overall skill level of annotators across all data classes and ignores the scenario that annotators may have different skill levels when labeling different classes. Tanno et al.

[17] took inspiration from this and combined the use of a transition matrix with loss reweighting and proposed to model annotator labels and real data separately using annotator-specific confusion matrices. Annotator confusion matrix

[0045] The annotator confusion model

[17] is a probabilistic model based on assumptions that annotators are independent and label noises are independent of input images. Based on these two assumptions, the joint probability of observing n labels for an image from n annotators can be expressed as:

[0046] / -v) = n* jix) Eq 1

[0047] Where ÿ(n) is the annotation of the nth annotator and % is the image, y is the actual color and a(1) y"1,y is a value in the confusion matrix that indicates the probability that the annotator indicates an image as being of class y given that the actual class is y. During training, the annotator-specific confusion matrices are trained together with the classifier, in which the classifier learns to predict the actual distribution of labels and each confusion matrix learns the observed distribution of labels based on the actual labels for a particular annotator. Semi-supervised learning

[0048] Labeled data is often expensive to obtain, whereas unlabeled data is usually abundant and provides additional information about the distribution of the data. To take advantage of the information contained in unlabeled data, semi-supervised learning (SSL) [3]

[21] is a research area that explores ways to train models using both labeled and unlabeled data. Typical semi-supervised learning techniques include consistency regularization and pseudo-labeling [7].

[0049] Consistency regularization was first introduced by the F-model

[14] and suggests that the model should give similar data points with consistent outputs. To encourage models to do so, the F-model applies a consistency loss between data points with and without noise. Following this, Temporal Assemblage [6] proposes to improve the model by maintaining an exponential moving average (EMA) prediction of the training data. Mean Teacher

[18] is an extension of the F-model and Temporal Assemblage, and suggests maintaining an exponential moving average of the model weights across a student-teacher framework. The student model is trained based on a supervised classification loss and an unsupervised consistency loss with the teacher model. The consistency loss is applied to minimize the differences between the results of the student and teacher models.The teacher model weights are updated based on the student model's EMA. The Mean Teacher framework has achieved excellent results for semi-supervised learning image classification benchmarks.

[0050] Recent state-of-the-art models in SSL often combine several SSL techniques, including combining consistency regularization with pseudolabeling [2][ 1][ 15]. FixMatch

[15] is based on the idea that data augmentation has a significant impact on consistency regularization

[20] , and proposes to train the classifier to process consistent output for various augmentations of the same output with the use of pseudo-labels. This method produces excellent results in image classification tasks. However, since it relies on strong augmentations that change the color of images, it is not applicable to color classification. Model architecture Reference model

[0051] [Fig.2] shows a block diagram of a processing flow 200 in accordance with to an embodiment in which a reference model 202 receives an image 204 as input and outputs predictions 206 including hair hue 206A and reflectance 206B. In one embodiment, the reference model 202 uses ResNet18 202A as a backbone. The output of the backbone 202A is passed to two linear classifiers 202B and 202C which separately predict hue 206A and reflectance 206B. In one embodiment, each linear classifier consists of two fully connected linear layers with a ReLu activation layer between the two layers. In one embodiment, the stream 200 represents a training stream at a training time. The components 202 are retained for inference time use.

[0052] In one embodiment, the reference model 202 was trained using a sum of hue and reflectance cross-entropy losses (e.g., 208) between the model outputs 206A / 206B and the target colors (e.g., soft labels 210) prepared from the expert labels. Hard labels and soft labels

[0053] When training the reference model in [Fig.2], both hard and soft labels are tested. The hard labels for hue and reflectance are calculated separately based on the class with the highest number of votes. The soft labels are calculated based on the empirical distribution of labels from all annotations. For example, if nine experts label the image as class 1 and one expert labels it as class 2, the soft label will be 0.9 for class 1, 0.1 for class 2, and 0 for all other classes. Regardless of soft or hard labels, the reference models are trained based on the cross-entropy loss (e.g., 208).

[0054] Classifier with annotator confusion matrices

[0055] [Fig. 3] is a block diagram of a processing flow 300 showing a classifier 302 in accordance with an embodiment of [Fig. 2]. The classifier 302 is adapted by training using annotator confusion matrices 304 as described in more detail. The processing flow 300 is useful at a training time and the components 302 are useful at an inference time.

[0056] In one embodiment, components 302A-302C of classifier 302 are structured in the same manner as components 202A-202C of classifier 202 but are adapted, for example, by training with the annotator confusion matrices 304 as described. Output hue 306A and reflectance 306B are also similar to outputs 206A and 206B, but will depend on the trained classifier 302. The outputs of the respective classifiers for the same input image may vary in value.

[0057] According to one embodiment, the annotator confusion matrices 304 comprise a plurality (n) of hue annotator confusion matrices 304A and a plurality (n) of reflectance annotator confusion matrices 304B. Here, "n" refers to the number of annotators.

[0058] [Fig. 4] is an illustration of a confusion matrix of representative hue annotators 400, in accordance with one embodiment. The rows represent actual classes (“Actual Values”) and the columns represent observed annotations (“Expert Annotations”).

[0059] In one embodiment, the classifier 302 with the annotator confusion matrices 304 is built based on the reference model 202, having annotator-specific confusion matrices 304 after the classification layers (202B / 202C) of the reference model 202 to predict the observed labels 306A / 206B from the annotators. Assuming there are "n" annotators, n 10*10 confusion matrices are added after the hue classifier and n 100*100 confusion matrices are added after the reflectance classifier. The outputs 306A / 306B of the hue and reflectance classifiers 302B / 302C are multiplied by the respective confusion matrices 304A / 304B to predict the annotations of individual annotators. For example, the outputs of the hue classifier 302B are multiplied by the first hue confusion matrix (e.g., 307) to predict the hue annotation for the first annotator.

[0060] During training, for each training data, the cross-entropy loss sums of n hues and n reflectances (e.g., 308) are calculated to update the model. Since an annotator has only one hue label and one reflectance label for each training data, each cross-entropy loss was calculated using hard labels (e.g., 310) instead of soft labels (e.g., 210). The confusion matrices are first randomly assigned with values ​​such that the diagonal values ​​dominate and such that the matrices are approximately identical. The matrices are trained with the rest of the model based on the classification loss 308, and a regularization term is added to encourage the trained matrices to converge to the actual confusion matrices (e.g., according to the example in [Fig. 4]). Mean Teacher Frame

[0061] [Fig. 5] is a block diagram of a processing flow 500 showing the reference classifier model 202 of [Fig. 2] adapted to a Mean Teacher framework for training. The flow 500 includes a student model 502A and a teacher model 502B, according to one embodiment. The models 502A and 502B are configured as instances of the model 202 so that the details are not shown in [Fig. 5]. The student model 502A produces an output including the hue output 506A and the reflectance output 506B and the teacher model 502B produces an output including the hue output 506C and the reflectance output 506D, respectively.

[0062] The student model 502A and the teacher model 502B are first pre-trained using the labeled data. During the semi-supervised learning process, two augmented versions of the same image (e.g., 504A and 504B) are fed separately to each of the models 502A / 502B. The student model 502B is trained using the (supervised) cross-entropy loss 508 and the (unsupervised) consistency loss 512. The cross-entropy loss (classification loss) is calculated based on the labeled data (e.g., 510), and the consistency loss is calculated as the mean square error of the softmax outputs (506A, 506B and 506C, 506D) between the student model 502B and the teacher model 502A. The consistency loss 512 based on the augmented images 504A, 504B forces the student model 502A to provide consistent outputs at similar data points. After updating the student model 502A, the teacher model 502B is updated based on the exponential moving average of the weights of the student model 502A. According to one embodiment, during inference, only the teacher model 502B is retained and used for evaluation. Experiences Dataset

[0063] The selfie dataset used during the experiments contained 1,000 labeled images and 2,500 unlabeled images. For the labeled data, the selfie hair was annotated by 10 professional hair experts and each label contained a hue, a primary reflectance and a secondary reflectance. In one embodiment, the range of hue values ​​is from 1 to 10 where 1 corresponded to the darkest hue, and the range of primary and secondary reflectance values ​​is from 0 to 8. The hair color may also have no secondary reflectance or primary reflectance.

[0064] Labels from different annotators may not match. However, for most labeled images, there were at least two annotations that matched. Of the 1,000 images, only 40 did not match. When looking at hue, primary, and secondary reflectance values ​​individually, at least two annotators always agreed. To assess annotator quality, Table 1 lists the accuracy and average absolute hue difference between individual annotator labels and the majority vote. The overall agreement on color was about 30%, with experts disagreeing more on reflectance values ​​(49%) than on hues (64%). In particular, experts agreed on secondary reflectance values ​​only 42% of the time.

[0065] [Table 1]

[0066] [Tables 1] Annotator Avg. 1 2 3 4 5 6 7 8 9 10 Overall Accuracy (%) 30 33 20 31 21 41 39 34 30 28 21 Hue Accuracy (%) 64 81 55 68 53 70 73 66 59 61 55 Reflectance Accuracy (%) 49 45 52 18 18 59 65 67 63 58 49 Primary reflectance accuracy (%) 48 56 47 26 29 59 56 56 52 57 41 Secondary reflectance accuracy (%) 42 33 31 38 32 54 55 49 49 51 31 Configuring experiments

[0067] The reference model 202 and the classifier 302 with the confusion matrices 304 were trained using the stochastic gradient descent (SGD) optimizer, the learning rate was constant for the first 20 iterations and then decreased linearly to zero over the next 65 iterations. Simple data augmentations, including random horizontal and vertical flips, rotations, and translations were used during training. The batch size was set to 128.

[0068] The teacher model of the Mean Teacher 502B framework was trained using the SGD optimizer, the learning rate was brought to zero using simulated annealing based on the cosine function

[12] . For data augmentation, random data augmentations, including horizontal and vertical flip, rotation, translation, and shear, were used. The batch size was set to 256, with 128 labeled data and 128 unlabeled data. Evaluation metrics

[0069] In one embodiment, the model's performance was evaluated based on the acceptance of the model by the majority of experts and the magnitude of the deviation between the predictions and the annotations. Three main metrics were used for evaluation, accuracy, hue within ±1, and mean absolute difference in hue. For hue, reflectance, and overall color, accuracy was measured in terms of the model's prediction corresponding to the color that received the highest number of votes from the ten annotators.Since hues are ordinal data, two additional metrics were used: hue at ±1 measures the probability that the model's predictions fall within one hue difference from the hue with the highest number of votes, and mean absolute difference measures the average distance between predicted hues and the average hue annotations for each image. The results of the experiment were based on 5-fold cross-validation, and in each run, the metrics from the best iteration were reported. Soft label / hard label results

[0070] Table 2 compares the prediction performance between an average human expert, the reference model trained with hard labels and the reference model trained with soft labels, according to one embodiment. In case of disagreements Among experts, training the model with soft labels allowed for learning more about the distribution of votes and making better predictions. The prediction accuracies of the model trained with soft labels were 9% higher than those of the model trained with majority votes, with a 7% increase in hue accuracy and a 6% increase in reflectance accuracy. Another observation is that the model trained with majority votes performed slightly worse than an average human expert, whereas the model trained with soft labels performed better than the average human expert.

[0071] [Table 2]

[0072] [Tables2] Human Expert Majority Soft labels Overall accuracy (%) 30+7 29+4 38+4 Hue accuracy (%) 64+9 62+6 69+6 Reflectance accuracy (% 49+18 41+5 47+4 ) Reflectance accuracy pr 48+12 60+6 66+6 imary Classifier with confusion matrices

[0073] In one embodiment, stepwise experiments were conducted to examine the impacts of adding confusion matrices to the model and training the model using the Mean Teacher framework. In brief, either model was better than the performance of an average human expert. As shown in Table 3, adding the confusion matrices to the model resulted in an improvement in overall color accuracy (2%). It is noted that the baseline model trained with soft labels is a special case of the model with confusion matrices where all confusion matrices were identity matrices. The improvement in color classifications is primarily due to the improvement in reflectance prediction (5%), and the accuracy of hue prediction is similar to the baseline model. It is understood that this results from more disagreement on reflectance labels among hair experts.

[0074] [Table 3]

[0075] [Tables3] Human Expert Reference Confusion Average Overall Accuracy (%) 30+7 38+4 40+6 50+5 Hue Accuracy (%) 64+9 69+6 67+3 73+4 Reflectance accuracy (%) 49+18 47+4 53+6 58+5 Primary reflectance accuracy (%) 48+12 66+6 68+6 73+2 Secondary reflectance accuracy 42+10 54+3 60+5 63+3

[0076] Although the addition of annotator confusion matrices did not improve hue accuracy, a more detailed analysis showed that its prediction deviated less from expert annotations. Table 4 shows a comparison of hue prediction performance between human annotators, the baseline model with soft labels, the classifier with annotator confusion matrices, and the Mean Teacher framework. As shown in Table 4, after adding the confusion matrices, the percentage of hue within ±1 of the hue with the highest number of votes was 1% higher, and the mean absolute differences were 0.04 lower. This revealed that hue prediction was also improved.

[0077] [Table 4]

[0078] [Tables4] Reference model Confusion matrix Mean Teacher Hue at +1 (%) 97+0 98+0 98+2 Mean abs. diff. hue 0.44+0.07 0.40+0.07 0.32+0.07 Cadre Mean Teacher

[0079] After integrating the Mean Teacher framework, the overall performance of the model was significantly improved. Table 3 and Table 4 show that the framework improved the prediction accuracy by 10% compared to the confusion matrix model, and also reduced the difference between the model prediction and the experts' annotation. The average absolute difference was reduced by 0.08. Furthermore, when comparing the classifier based on the Mean Teacher framework and an average human expert, there was a 20% improvement in the overall color accuracy with a 9% and 11% improvement in the hue and reflectance accuracy. [Fig. 6] is a graph 600, according to one embodiment, showing a comprehensive comparison of the model prediction accuracy results between all models and an average human expert.Chart 600 shows that there is a significant improvement between the accuracy of an average human expert and the Mean Teacher framework model. Ablation study

[0080] To facilitate better planning of data collection, an experiment was conducted to study the impact of having additional labeled selfies and annotators. When conducting the following experiments, the baseline model with soft labels was trained to evaluate the impact of having more images and more labels per image. When examining the impact of additional labeled images, the number of labeled data varied from 100 to 900. When examining the impact of additional labels per image, labels from a subset of experts were used to train the model.

[0081] The model's performance improved when it had more selfies or more annotators than expected. However, as shown in Figures 7A and 7B, the model improved more when it had more labeled images. [Fig.7A] is a graph 700 showing an accuracy comparison of different numbers of training selfies and [Fig.7B] is a graph 710 showing an accuracy comparison of different numbers of annotators.

[0082] It was observed that a consistent increase in accuracy is achieved when increasing the training images from 100 to 900, where the overall, hue, and reflectance accuracies increased in parallel. No downward trend was observed in the improvement even after using 900 images. In contrast, minimal improvement was observed in the overall accuracy between 5 and 10 annotators, in particular, less improvement was observed in the reflectance accuracy after having 8 annotators. This showed that having additional labeled images was more effective in improving the model performance. Hair simulation application

[0083] [Fig. 8] is a block diagram of a computing device 800 in accordance with one embodiment. In the embodiment, a color classifier in accordance with one embodiment herein is integrated with a VTO application to provide hair color simulation. The computing device 800 provides a user computing device, such as a smartphone, tablet, laptop, or other computing device, for use by a user such as a consumer of hair dye products or a salesperson assisting such a user. The device may include a component of a larger form factor such as a kiosk for placement in a retail environment, for example. The device 800 is non-limiting and simplified for brevity.

[0084] The device 800 comprises a storage device 802, a processing unit 806, a camera 808, a microphone / speaker 810, a display screen 812 and a communication subsystem 814. In one embodiment, the storage device 802 includes a memory device, for example, one or more types of memory such as RAM, ROM, etc. The storage device 802 may include a long-term storage device such as a solid-state drive (SSD) or other type of drive for storing non-volatile data, etc. The storage device 802 stores computer-readable instructions for execution by the processing unit, such that once executed, the instructions cause the computing device to perform operations such as one or more methods. The processing unit 806 includes one or more central processing units (e.g., CPUs), and / or graphics processing units (e.g., GPUs) having one or more processors / microprocessors, controllers / microcontrollers, etc. Other types of processors may be used.GPUs can be particularly useful for accelerating graphics processing tasks and / or AI processing tasks (e.g., training and / or inference).

[0085] The camera 808 may be used to take selfies. The microphone and speaker 810 are generally separate devices, but labeled together here for convenience and represent some of the input (I), output (O), or I / O devices that may be available. Other devices may include a light, buzzer, vibrator, button, keyboard, pointing device (e.g., mouse or touchpad, etc.), etc.

[0086] The display screen 812 presents images such as components of a graphical user interface, camera images, etc. In one embodiment, the display screen is a touchscreen device, a type of I / O device, configured to receive gesture inputs (e.g., swiping, tapping, etc.) that interact with the region(s) of the screen and in association with the user interface components (e.g., controls) presented by an application executed by the processing unit 806.

[0087] The communication subsystem 814, in one embodiment, is configured to manage communications between device components and / or between the computing device and external devices such as a remotely located computing device (e.g., web service, cellular network component, printer, etc.). A subcomponent thereof in one embodiment is an antenna for wireless communication. A subcomponent thereof in one embodiment is a wired communication interface (e.g., Ethernet, USB A, USB C, Thunderbolt (TM from Intel Corporation), etc.) to be coupled to a cable suitable for wired communication.

[0088] The storage device 804 stores the components of a VTO application and corresponding data (e.g., 820). Representative components are shown. The VTO application 820 includes the user interface component 822 (e.g., screens, instructions, icons, controls, etc.). The user interface provides outputs to a user and receives inputs such as inputs for the application workflow, user selections of color choices, etc. A color classifier 824 is provided and includes one of classifiers 202, 302, and 502 as previously described for determining hair color information (e.g., hue and reflectance). A VTO pipeline 826 is provided for simulating hair color in association with (e.g., on or in) an input image. A color recommendation engine 828 and associated data store 830 are provided.In one embodiment, the data store 830 stores hair color data and representative images thereof such as in the form of a color swatch image or colored hair, hair reflectance images, etc. The user interface may present one or more choices that a user can select via user input, e.g., sorting or filtering examples and presenting them in groups / pages, etc. In one embodiment, the color recommendation engine 828 may include an interface to a chatbot or live agent to discuss a recommendation.

[0089] In one embodiment, recommendations may be made, for example, based on recommendation factors such as a user's personal information, including the user's current hair color and age; hair color trend data; availability of a product and / or service local to the user (e.g., within a given radius); cost information, etc. In one embodiment, rules or another manner of recommending may be used to determine which recommendation (or recommendations) to present to a user.

[0090] In one embodiment, a user may provide an input image with hair (e.g., 832) similar to the input image 204, for classification. Hue and reflectance data 834 are generated by the classifier 824. The hue and reflectance data may be made available to the color recommendation engine 828 for processing (e.g., using color matching rules, color combination rules, etc.) to select one or more colors from the color data store 830 to recommend to a user via the user interface component 822. The recommendation may be displayed on the screen 812. The user may invoke the VTO engine (via input to a command) to have the engine simulate the color and / or color and reflectance (e.g., target hair) using image 832 and target hair data as input to produce an output image with simulated hair 836.

[0091] The output image with simulated hair 836 may be presented via the user interface component 822 on the display screen 812. A before and after display may be provided for comparison. The comparison between at least two simulated colors (e.g., two different output images) may be displayed for comparison. The products or services or both may be purchased via the interface 838. Such an interface 838 may direct the user (e.g., the computing device) to a web-based e-commerce service (e.g., a website (not shown)) to make the purchase, reservation, or the like.

[0092] In one embodiment, the VTO pipeline 826 includes a generative neural network (e.g., a model) configured to generate a simulated hair image using the input image 832 and the target hair data as input. Refer to Figure 9 discussed below.

[0093] Other components stored in the storage device 804 include an operating system 840, a browser 842 (e.g., for browsing web pages), an email and / or messaging application 844 (e.g., SMS or other type), and a social networking application 846. The output image with simulated hair 836 may be shared (e.g., communicated) via the applications 844 and / or 846, for example.

[0094] In another embodiment of using the color classifier, not shown in [Fig. 8], the color classifier 824 processes an output image from a generative model configured to generate a simulated hair image using an input image having hair and target color data as input. The classifier determines the color and / or reflectance data such as for comparison with the target hair data to confirm the accuracy of the generative model.

[0095] The following aspects and characteristics cited in the numbered embodiments (class 1 embodiment, class 2 embodiment... class 20 embodiment) will emerge clearly, in particular, from the present disclosure.

[0096] Embodiment - Class 1: A computing device comprising a processor coupled to a storage device that stores instructions that, when executed by the processor, cause the computing device to: classify hair hue and hair reflectance in an input image using a neural network, the neural network comprising an encoding skeleton coupled to a pair of classifiers comprising i) a hue classifier that outputs a hair hue value and ii) a reflectance classifier that outputs a hair reflectance value, each of the two classifiers comprising a linear classifier, the a neural network having been defined by training with a sum of hue and reflectance cross-entropy losses determined from i) the results of hair hue values ​​and hair reflectance values, and ii) the target labels for hair hue and hair reflectance, the target labels prepared for the training images from the respective expert votes by a plurality of experts.

[0097] Embodiment -class 2: A computing device according to embodiment-class 1, wherein the neural network is configured to classify hair hue and hair reflectance according to a plurality of respective classes to follow an industry standard for hair color.

[0098] Embodiment -class 3: Computer device according to embodiment -class 1, in which the reflectance of the hair comprises a primary reflectance component and a secondary reflectance component

[0099] Embodiment - class 4: Computer device according to embodiment - class 1, in which the target labels comprise soft labels for at least some of the training images, the soft label being determined from an empirical distribution of expert votes on the respective classes.

[0100] Embodiment -class 5: A computing device according to embodiment-class 1, wherein the neural network has been trained with respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of reflectance annotator confusion matrices, wherein n is defined from a total number of experts providing the expert votes, and wherein the output of the hue classifier is multiplied by a respective hue confusion matrix from among the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective reflectance confusion matrix from among the plurality of matrices to predict the vote of a respective expert from among the plurality of experts and wherein the neural network and the respective annotator confusion matrices are trained together.

[0101] Embodiment -class 6: Computing device according to embodiment-class 1, in which the neural network has been trained in accordance with a Mean Teacher framework.

[0102] Embodiment -class 7: Computer device according to embodiment-Class 6, in which the neural network comprises a teacher network obtained from the Mean Teacher framework.

[0103] Embodiment -class 8: A method comprising: receiving an input image and classifying hair shade and hair reflectance in the input image using a neural network, the neural network comprising an encoding skeleton coupled to a pair of classifiers comprising i) a hue classifier that outputs a hair hue value and ii) a reflectance classifier that outputs a hair reflectance value, each element of the pair of classifiers comprising a linear classifier, the neural network having been defined by training with a sum of hue and reflectance cross-entropy losses determined from i) outputs of the hair hue values ​​and the hair reflectance values, and ii) target labels for the hair hue and the hair reflectance, the target labels being prepared for image training from respective expert votes by a plurality of experts; providing the hair hue and the hair reflectance.

[0104] Embodiment-class 9: Method according to embodiment-class 8, wherein hair hue and hair reflectance are provided to train a generative model to simulate hair color.

[0105] Embodiment -class 10: The method of embodiment-class 9, wherein the neural network is configured to classify hair hue and hair reflectance according to a plurality of respective classes to follow an industry standard for hair color, and wherein the hair reflectance comprises a primary reflectance component and a secondary reflectance component.

[0106] Embodiment -class 11: A computing device according to embodiment-class 9, wherein the neural network having been trained with respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of reflectance annotator confusion matrices, wherein n is defined from a total number of experts providing the expert votes, and wherein the output of the hue classifier is multiplied by a respective matrix from among the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective matrix from among the plurality of reflectance confusion matrices to predict the vote of a respective expert from among the plurality of experts and wherein the neural network and the respective annotator confusion matrices are trained together.

[0107] Embodiment -class 12: Method according to embodiment-class 9, in which the neural network has been trained in accordance with a Mean teacher framework.

[0108] Embodiment - class 13: Computer device according to embodiment - class 12, in which the neural network comprises a teacher network obtained from the Mean Teacher framework.

[0109] Embodiment -class 14: Method according to embodiment -class 9, in which the target labels comprise soft labels for at least some of the images training, the soft label being determined from an empirical distribution of expert votes on the respective classes.

[0110] Embodiment -class 15: Method according to embodiment-class 14, in which the input image is associated with at least one of the target labels and in which step b. is carried out to train the neural network using the input image.

[0111] Embodiment -class 16: A method according to embodiment-class 15, comprising training the neural network with respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of reflectance annotator confusion matrices, wherein n is defined from a total number of experts providing the expert votes, and wherein the output of the hue classifier is multiplied by a respective matrix from the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective matrix from the plurality of reflectance confusion matrices to predict the vote of a respective expert from the plurality of experts and wherein the neural network and the respective annotator confusion matrices are trained together.

[0112] Embodiment - Class 17: Computer software product comprising a non-transitory storage device that stores computer-executable instructions that, when executed by a processor of a computing device, cause the computing device to: classify hair hue and hair reflectance in an input image using a neural network, the neural network comprising an encoding skeleton coupled to a pair of classifiers comprising i) a hue classifier that outputs a hair hue value and ii) a reflectance classifier that outputs a hair reflectance value, each of the classifiers comprising a linear classifier, the neural network having been defined by training with a sum of hue and reflectance cross-entropy losses determined from i) the outputs of the hair hue values ​​and the hair reflectance values,and ii) target labels for hair tone and hair reflectance, the target labels prepared for the training images from the respective expert votes by a plurality of experts.

[0113] Embodiment -class 18: Computer software product according to embodiment -class 17, wherein hair hue and hair reflectance are provided to train a generative model to simulate hair color.

[0114] Embodiment -class 19: Computer software product according to embodiment -class 17, wherein execution of the instructions causes the computing device to classify hair shade and hair reflectance in accordance with a plurality of respective classes to follow an industry standard for hair color and wherein the hair reflectance comprises a primary reflectance component and a secondary reflectance component.

[0115] Embodiment -class 20: Computer software product according to embodiment -class 17, wherein one or both of the following: execution of the instructions causes the computing device to provide the neural network for classification wherein the network having been trained with respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of reflectance annotator confusion matrices, wherein n is defined from a total number of experts providing the expert votes,and wherein the output of the hue classifier is multiplied by a respective hue confusion matrix of the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective one of the plurality of reflectance matrices to predict the vote of a respective expert of the plurality of experts; and wherein the neural network and the respective annotator confusion matrices are trained together; or execution of the instructions causes the computing device to output the neural network for classification wherein the network has been trained in accordance with a Mean Teacher framework.

[0116] Classifier-guided training of a color-refining neural network

[0117] In one embodiment, a classifier may guide the training of a color refinement neural network that is configured to produce rendered images with a target hair color. It is difficult to assess the accuracy of hair rendering based solely on RGB values ​​or simple metrics. In one embodiment, there is a feedback loop in a rendering network system to use a color classifier trained in accordance with an embodiment herein to guide the rendering network. [Fig. 9A] is a block diagram of a network system 900A in accordance with an embodiment providing a training system. [Fig. 9B] is a block diagram showing additional details of a training network system 900B linked to the system 900A, in accordance with one embodiment.The method, computer program product and other aspects will be readily understood by those skilled in the art from an understanding of the training aspects and inference aspects shown and described.

[0118] The network system 900A displays the input image 104 (e.g., image I) and a target hair color 902 provided to a hair analysis engine 904 to determine hair information for the hair pixels ... e.g., from hair segmentation), e.g., hair histogram color data for image I. In one embodiment, the image I, target hair 902 (e.g., as a hair sample image), and hair histogram color data comprise RGB data (e.g., 256*3) commonly used for images. In one embodiment, the hair analyzer engine 904 includes or communicates with a hair analyzer such as a hair classifier network to determine the hair pixels in the input image 104. In one embodiment, the I-image and color information (e.g., histogram data) for the target hair color 902 and the hair in the I-image (e.g., in the form of a histogram (not shown as such in [Fig. 9A])) are provided to a color refinement neural network 906 to produce a rendered image 908.In one embodiment, as shown in [Fig.9B], the color refinement neural network 906 includes a color mapping network and a generative network configured to modify features of an image it processes, namely hair color. The generative network in one embodiment includes a model (e.g., generator (G)) that is defined by training guided by the hue and reflectance classifier 910.

[0119] Referring again to [Fig. 9A], the rendered image 908 is provided to the color classifier 910 defined (e.g., trained) in accordance with one embodiment herein to classify the hue and reflectance properties to produce the color prediction 912 for the image 908. The loss 914 represents a loss determined from the color prediction 912 and the target hair color 902 as used to train the color refining neural network 906 as further described with [Fig. 9B]. In one embodiment, the system 900A is configured with one or more computing devices (not shown) to provide the computing components (e.g., 904, 906, 910, etc.) and to store data (e.g., 104, 902, 908, 912, 914, etc.).A display device (not shown) may be included to display the image 104, 908 and any of the other data, including the output (not shown) of the hair analyzer engine 904, provided to the color refinement neural network 906.

[0120] In one embodiment, network system components 900A and 900B may be configured as an inference time system (e.g., following training) to provide a virtual try-on experience for simulating a hair color applied to an image. For example, components 904 and 906 are useful for defining a VTO pipeline (e.g., 920) for simulating hair color, for processing an input image to simulate a target hair color and produce a simulated image. In one embodiment, the color refining neural network 906 comprises a neural network defined by training as guided by the color classifier 910 as described in more detail below.

[0121] According to Figures 9A and 9B, a color mapping network 922 is trained to predict correct rendering parameters based on the classification results of the color classifier (910). As shown in [Fig.9B], the pipeline uses the color mapping network 922 and a generative adversarial network (GAN) 924 with a generator (G) 924A to simulate the hair rendering 908 (namely (G} )).

[0122] A hair segmentation model (e.g., as a component of the hair analyzer engine 904, and not shown in [Fig.9B]) is used to extract the hair masks and the engine 904 provides RGB histogram data of hair pixels (Hl) in (i.e., from) the image / 104. The color mapping network 922 includes an encoder 926 encodes image features (F) from the image / . The encoder 926 encodes features including, but not limited to, color and lighting of the hair representation and in the generator is used to regenerate the same hair rendering.

[0123] A concatenator 928 of the color mapping network 922 receives inputs: i) RGB histogram of the sample (i.e., target hair color) (HT)

[0124]

[0125]

[0126]

[0127] 902, ii) source RGB hair histogram (Hl) 930 (frame I) and iii) image features (F) from encoder 926. The three inputs are concatenated and passed through a block of fully connected layers 932 for the output color mapping (Mp). The output color mapping Mp and the source RGB hair histogram Hl 930 are concatenated and used as input and passed through another block of fully connected layers (FC Block 2 (934)) to obtain the output RGB histogram (H0) to be used by the generator (G) 924A to produce its output from frame I. Therefore, the output rendered image G1 908 is produced by replacing the hair pixels with the color mapping. The internal color maps (M c ) and RGB histograms (HG) are used as real data labels to train the color mapping network 922 in a supervised manner. In one embodiment, the following losses are used to train the color mapping network 922: PcrtC — L Color-mapping P RGB histogram L2 loss between the real data color mapping (M c) and the network predicted color mapping: 2 ^Color-mapping ~ ln \^G ~ Mp)

[0128] L1 loss between the histograms of the real data RGB histogram (H c) and the output RGB histogram (HO):

[0129] j ij i ^RGB histogram ~ Z—y I ■“ G " O i

[0130] With the trained color mapping network 922, the GAN 924 may be trained, as follows, according to one embodiment.

[0131] The output RGB histogram (H 0 ) from the color mapping network 922 is used as a condition with the source image 104 as inputs to the generator (G)924A of the GAN 924. In one embodiment, the GAN 924 is defined as a StarGAN (in accordance with the teaching of Choi, Y., et al., “StarGAN: Unified Generative Adversarial Networks for Multi-Domain Histogram-to-Image Translation,” IEEE Conference on Pattern Recognition and Computer Vision (CVPR), 2018, pp. 8789-8797), such that G1 = G(I, H0).

[0132] It is noted that the previously trained color mapping network 922 is frozen during the training of the GAN. That is, the network 922 is not trained or co-trained with the training of the GAN 924.

[0133] The discriminating part (discriminator (D) 924B) of the GAN 924 uses two classifiers, one (936) which classifies whether the image is fake or real ((G; ) or I); and the other (910) is the pre-trained classifier which classifies the two parts of the color: the hue and the reflectance.

[0134] The pre-trained classifier guides and enhances the generative capabilities of the GAN network and can be used independently with any GAN architecture for advisory purposes.

[0135] The losses used for training the generator 924A and discriminator 924B parts are the same as those defined in the StarGAN document referenced above.

[0136] In one embodiment, the color mapping + GAN is labeled as a color refinement network because it uses the instructions from the pre-trained hair classification model and the losses indicated above in order for the generative model to refine its outputs.

[0137] As shown in the simplified block diagram of [Fig.9C] illustrating an application of VTO, in one embodiment, at inference time (e.g., when training is complete and the refinement network is provided to generate output images, in particular from user input), the discriminator component 924B with its two classifiers 910 and Conclusion

[0138] In this work, techniques, etc. are provided to address the problem of limited availability of labeled data and label noise caused by human bias, to classify the color of attributes in images, through a combination of a semi-supervised learning framework and annotator confusion matrices. The final model achieved 20% higher accuracy than an average human expert.

[0139] The experiments described herein have shown that the confusion matrix is ​​useful for improving color prediction, especially in scenarios where there is more disagreement among experts. The approach primarily considers label noise caused by individual expert biases and assumes that biases depend only on the actual class (e.g., without considering other factors, such as lighting and contrast with the background). Due to the noisy nature of subjective annotation, in one embodiment, soft labels are used but not majority votes because the training labels and confusion matrices are adapted to represent different ideas of the experts.

[0140] The experiments as described here have also shown that the Mean Teacher framework helps to significantly improve color prediction. The effectiveness of the current framework depends on the teacher model providing good labels for training the student model.

[0141] Unlike normal RGB value prediction, the classification model is defined to classify according to a color standard defined specifically for hair. In one embodiment, such a standard is three digits, with the first digit representing natural hair tones and the second and third digits representing the primary and secondary reflectance of the hair. Such color prediction adapts to different lighting conditions and therefore accurately predicts dark hair in bright light and light hair in dim light. The color representation is light independent. The color number prediction can be directly related to how hair products are color coded (e.g., using the same color standard), which generally cannot be expressed in RGB or simple text information.

[0142] The model structure jointly uses a branch for the natural hue (first digit) and a branch for the two reflectance determinations in order to obtain better performance.

[0143] A practical implementation may include some or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, apparatus, systems, means for performing functions, program products, and in other ways, combining the features described herein. A number of embodiments have been described. Nevertheless, it is understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided, or steps may be eliminated, from the described process, and other components may be added to or removed from the described systems.

[0144] Throughout the description of this patent specification, the terms "include", "contain" and their variations mean "including but not limited to" and are not intended to (and do not) exclude other components, integers or steps. Throughout this patent specification, the singular includes the plural, unless the context otherwise requires. In particular, when the indefinite article is used, this patent specification is to be understood as contemplating plurality as well as singularity, unless the context otherwise requires.

[0145] Features, integers, characteristics or groups described in conjunction with a particular aspect, embodiment or example of the invention are to be understood as being applicable to any other aspect, embodiment or example, unless they are inconsistent therewith. Any features disclosed herein and / or any steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of these features and / or steps are mutually exclusive. The invention is not limited to the details of the foregoing examples or embodiments. The invention extends to any novel feature, or any novel combination, of the features disclosed in this patent specification or to any novel feature, or any novel combination, of the steps of any method or process disclosed. REFERENCES

[0146] The following documents are cited in the respective title in their respective integrity:

[0147] Berthelot, D., Carlini, N., Cubuk, E.D., Kurakin, A., Sohn, K., Zhang, H., Raffel, C.: Remix-match: Semi-supervised leaming with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785 (2019)

[0148] Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, CA: Mixmatch: A holistic approach to semi-supervised leaming. Advances in neural information processing Systems 32 (2019)

[0149] Chapelle, O., Schôlkopf, B., Zien, A.: Introduction to semi-supervised leaming (2006)

[0150] Guan, M., Gulshan, V., Dai, A., Hinton, G.: Who said what: Modeling individual labelers improves classification. In: Proceedings of the AAAI conference on artificial intelligence, vol. 32 (2018)

[0151] Hendrycks, D., Mazeika, M., Wilson, D., Gimpel, K.: Using trusted data to train deep networks on labels corrupted by severe noise. Advances in neural information Processing Systems 31 (2018)

[0152] Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 (2016)

[0153] Lee, D.H., et al.: Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on challenges in représentation learning, ICML. vol. 3, p. 896 (2013)

[0154] Lee, K.H., He, X., Zhang, L., Yang, L.: Cleannet: Transfer learning for scalable image classifier training with label noise. In: Proceedings of the IEEE conférence on computer vision and pattern récognition, pp. 5447-5456 (2018)

[0155] Li, W., Wang, L., Li, W., Agustsson, E., Van Gool, L.: Base de données Webvision : Visual learning and understanding from web data. arXiv preprint arXiv: 1708.02862 (2017)

[0156] Liu, T., Tao, D.: Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence 38(3), 447-461 (2015)

[0157] Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conférence on Computer Vision (ICCV) (December 2015)

[0158] Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 1608.03983 (2016)

[0159] Patrini, G., Rozza, A., Krishna Menon, A., Nock, R., Qu, L.: Making deep neural networks robust to label noise: A loss correction approach. In: Proceedings of the IEEE conférence on computer vision and pattern récognition, pp. 1944-1952 (2017)

[0160] Rasmus, A., Berglund, M., Honkala, M., Valpola, H., Raiko, T.: Semi-supervised learning with ladder networks. Advances in neural information processing Systems 28 (2015)

[0161] Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C.A., Cubuk, E.D., Kurakin, A., Li, C.L.: Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing Systems 33, 596-608(2020)

[0162] Song, H., Kim, M., Lee, J.G.: Selfie: Refurbishing unclean samples for robust deep learning. In: International Conférence on Machine Learning. pp. 5907-5915. PMLR (2019)

[0163] Tanno, R., Saeedi, A., Sankaranarayanan, S., Alexander, D.C., Silberman, N.: Learning from noisy labels by regularized estimation of annotator confusion pp. 11244-11253(2019)

[0164] Tarvainen, A., Valpola, H.: Mean teachers are better rôle models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing Systems 30 (2017)

[0165] Xiao, T., Xia, T., Yang, Y., Huang, C., Wang, X.: Learning from massive noisy labeled data for image classification. In: Proceedings of the IEEE conférence on computer vision and pattern récognition, pp. 2691-2699 (2015)

[0166] Xie, Q., Dai, Z., Hovy, E., Luong, T., Le, Q.: Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems 33, 6256-6268 (2020)

[0167] Zhu, X., Goldberg, A.B.: Introduction to semi-supervised learning. Synthesis lectures on artificial intelligence and machine learning 3(1), 1-130 (2009)

Claims

Claims

1. A computing device comprising a processor coupled to a storage device containing instructions that, when executed by the processor, cause the computing device to: (a) process an input image and a target hair color using a virtual try-on (VTO) pipeline to produce a VTO experience that simulates the target hair color with the input image to produce an output image;(b) wherein the VTO pipeline comprises a color refinement neural network for generating the output image combining the target hair color and the input image, the color refinement neural network comprising a generative neural network trained under the direction of a hair classification network comprising a hue classifier that outputs a hair hue value and a reflectance classifier that outputs a hair reflectance value to determine training loss information from output images for training the generative neural network.;

2. The computing device of claim 1, wherein the instructions are executed by the processor to cause the computing device to provide an interface to one of: i) a color recommendation engine for recommending a target hair color; and ii) an e-commerce service with which to purchase a product or service, or both.

3. The computing device of claim 1, wherein the VTO pipeline comprises a color mapping network trained to produce a color map for input to the generative neural network to produce the output image for the VTO experiment, the color map produced from features of the image, image hair color data determined from the image, and the target hair color.

4. The computing device of claim 3, wherein the color mapping network comprises an encoder for determining features from the input image; a concatenator for combining the features, the color of target hair and image hair color data for processing by a first fully connected block, and a second block for processing an intermediate map from the first block with the image hair color data to produce the color map for input to the generative neural network.

5. The computing device of claim 4, wherein the VTO pipeline comprises a hair analysis engine for determining image hair color data from the input image.

6. The computing device of claim 1, wherein the input image and the target hair color are defined using RGB values, such that the generative neural network is trained to produce an output image using particular RGB values ​​as directed during training by the hair classification network using hue values ​​and hair reflectance values.

7. The computing device of claim 6, wherein the hair classification network is defined and trained according to an industry standard for hair color classification comprising a respective plurality of classes for hair hue and hair reflectance.

8. The computing device of claim 1, wherein the hair classification network comprises a coding skeleton and the hair hue classifier and the hair reflectance classifier each comprise a respective linear classifier.

9. The computing device of claim 8, wherein the hair classification network has been defined by training with a sum of hue and reflectance cross-entropy losses determined from i) outputs of hair hue values ​​and hair reflectance values, and ii) target labels for hair hue and hair reflectance, the target labels being prepared for the training images of the hair classification network from respective expert votes by a plurality of experts.

10. A computing device according to claim 9, wherein: (a) the hair classification network having been trained with respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of reflectance annotator confusion matrices, wherein n is defined from a total number of experts providing the expert votes, and wherein the output of the hue classifier is multiplied by a respective matrix from among the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective matrix from among the plurality of reflectance matrices to predict the vote of a respective expert from among the plurality of experts and wherein the neural network and the respective annotator confusion matrices are trained together or (b) the hair classification network trained according to a Mean Teacher framework.

Images