HAIR COLOR CLASSIFICATION USING ANNOTATION CONSISTENCY AND CONFUSION REGULARIZATION MATRICES

Abstract

HAIR COLOR CLASSIFICATION USING CONSISTENCY REGULARIZATION AND ANNOTATION CONFUSION MATRICES Aspects of hair classification and networks for this purpose are provided, including aspects for training such networks. A classifier model is provided to mitigate the impact of human bias where modeling the actual label distribution and annotator biases are separated by incorporating annotator confusion matrices into a baseline model. To further improve the model performance by leveraging unlabeled data, the model was trained using a consistency-based semi-supervised learning framework. Using only 1000 labeled data, the final classifier model achieved 20% better classification accuracy than a professional human annotator.The trained model can be used for a wide range of downstream tasks, including as a color classifier to train generative models for hair color translation. Figure for abstract: none.

Description

Title of the invention: HAIR COLOR CLASSIFICATION USING ANNOTATION CONSISTENCY AND CONFUSION REGULARIZATION MATRICES FIELD OF THE INVENTION

[0001] The present disclosure relates to computer image processing and artificial intelligence, including systems and methods for hair color classification and, more particularly, to improving hair color classification using consistency regularization and annotation confusion matrices. The present disclosure also relates to hair color simulation guided by hair color classification. BACKGROUND

[0002] Due to the growing popularity of online shopping, e-commerce companies are exploring ways to improve the customer experience. Virtual try-on technology (VTO) has been developed to help customers search for and experience products virtually, for example, to find the best fit. In some examples of VTO, it may be useful to determine information, such as from a user-provided image. For example, in a VTO related to hair color, a facial image (e.g., a portrait) may be processed to determine, by classification, the hair color in the portrait. [Fig. 1] represents a block diagram of a representative processing flow 100 that shows a general model in the form of a classifier 102 that receives a color facial image 104 (e.g., a portrait) and identifies a hair color 106, in accordance with the prior art.The output, along with the color facial image, can then be used for a virtual hair dye trial.

[0003] Several challenges arise when developing a hair color classification model. For example, color labels collected from hair experts are subject to human bias and labeled data is expensive to obtain. It is desired to have an improved hair classification model that can identify hair color.

[0004] A VTO pipeline may simulate hair color, for example, using a generative neural network. Improvements in simulation using generative networks are desired. SUMMARY

[0005] Aspects of hair color classification, VTO pipelines, and corresponding networks are provided, including aspects regarding training such networks. Further, in accordance with one embodiment, a classifier model is provided to mitigate the impact of human biases where modeling of the distribution of actual labels and annotator biases are separated by incorporating annotator confusion matrices into a baseline model. To further improve the model's performance by leveraging unlabeled data, the model was trained using a consistency-based semi-supervised learning framework. Using only 1000 labeled data, the final classifier model achieved 20% better classification accuracy than a professional human annotator.The trained model can be used for a wide range of downstream tasks, including as a color classifier to train generative models for hair color translation.

[0006] The following statements present various aspects and features disclosed in embodiments herein. These and other aspects and features will be readily understood by those skilled in the art, including aspects of computer program products. It is also understood that aspects / features of computer devices or systems may have corresponding method aspects / features and vice versa.

[0007] Statement 1: A computing device comprising a processor coupled to a storage device that stores instructions that, when executed by the processor, cause the computing device to: classify hair hue and hair reflectance in an input image using a neural network, the neural network comprising a coding skeleton coupled to a pair of classifiers comprising i) a hue classifier that outputs a hair hue value and ii) a reflectance classifier that outputs a hair reflectance value, each of the classifier pair members comprising a linear classifier, the neural network having been defined by training with a sum of hue and reflectance cross-entropy losses determined from i) the outputs of the hair hue values ​​and the hair reflectance values,and ii) the target labels for hair tone and hair reflectance, the target labels prepared for the training images from the respective expert votes by a plurality of experts.

[0008] Statement 2: A computing device according to Statement 1, wherein the neural network is configured to classify hair hue and hair reflectance according to a plurality of respective classes to follow an industry standard for hair color.

[0009] Statement 3: A computing device according to Statement 1, wherein the hair reflectance comprises a primary reflectance component and a secondary reflectance component.

[0010] Statement 4: A computing device according to Statement 1, wherein the target labels comprise soft annotations for at least some of the training images, the soft annotation being determined from an empirical distribution of expert votes on the respective classes.

[0011] Statement 5: A computing device according to Statement 1, wherein the neural network having been trained with respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of reflectance annotator confusion matrices, wherein n is defined from a total number of experts providing the expert votes, and wherein the output of the hue classifier is multiplied by a respective hue confusion matrix from among the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective reflectance confusion matrix from among the plurality of reflectance confusion matrices to predict the vote of a respective expert from among the plurality of experts and wherein the neural network and the respective annotator confusion matrices are trained together.

[0012] Statement 6: A computing device according to Statement 1, wherein the neural network has been trained in accordance with a Student-Teacher framework.

[0013] Statement 7: A computing device according to Statement 6, wherein the neural network comprises a teacher network obtained from the Student-Teacher framework.

[0014] Statement 8: A method comprising: receiving an input image and classifying hair hue and hair reflectance in the input image using a neural network, the neural network comprising a coding skeleton coupled to a pair of classifiers comprising i) a hue classifier that outputs a hair hue value and ii) a reflectance classifier that outputs a hair reflectance value, each member of the pair of classifiers comprising a linear classifier, the neural network having been defined by training with a sum of hue and reflectance cross-entropy losses determined from i) outputs of the hair hue values ​​and the hair reflectance values, and ii) target labels for the hair hue and hair reflectance, the target labels being prepared for image training from respective expert votes by a plurality of experts;the provision of hair tint and hair reflectance.;

[0015] Statement 9: The method of Statement 8, wherein hair hue and hair reflectance are provided to train a generative model to simulate hair color.

[0016] Statement 10: The method of Statement 9, wherein the neural network is configured to classify hair hue and hair reflectance according to a plurality of respective classes to follow an industry standard for hair color, and wherein the hair reflectance comprises a primary reflectance component and a secondary reflectance component.

[0017] Statement 11: The method of Statement 9, wherein the neural network having been trained with respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of reflectance annotator confusion matrices, wherein n is defined from a total number of experts providing the expert votes, and wherein the output of the hue classifier is multiplied by a respective hue confusion matrix from among the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective reflectance confusion matrix from among the plurality of reflectance confusion matrices to predict the vote of a respective expert from among the plurality of experts and wherein the neural network and the respective annotator confusion matrices are trained together.

[0018] Statement 12 .-A method according to Statement 9, wherein the neural network has been trained in accordance with a Student-Teacher framework.

[0019] Statement 13: A computing device according to Statement 12, wherein the neural network comprises a teacher network obtained from the Student-Teacher framework.

[0020] Statement 14: A method according to Statement 9, wherein the target labels comprise soft annotations for at least some of the training images, the soft annotation being determined from an empirical distribution of expert votes on the respective classes.

[0021] Statement 15: Method according to Statement 14, wherein the input image is associated with at least one of the target labels and wherein step b is performed to train the neural network using the input image.

[0022] Statement 16: A method according to Statement 15, comprising training the neural network with respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of reflectance annotator confusion matrices, wherein n is defined from a total number of experts providing the expert votes, and wherein the output of the hue classifier is multiplied by a confusion matrix of respective hue from among the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective reflectance confusion matrix from among the plurality of reflectance confusion matrices to predict the vote of a respective expert from among the plurality of experts and wherein the neural network and the respective annotator confusion matrices are trained together.

[0023] Statement 17: A computer software product comprising a non-transitory storage device that stores computer-executable instructions that, when executed by a processor of a computing device, cause the computing device to: classify hair hue and hair reflectance in an input image using a neural network, the neural network comprising a coding skeleton coupled to a pair of classifiers comprising i) a hue classifier that outputs a hair hue value and ii) a reflectance classifier that outputs a hair reflectance value, each of the pairs of classifiers comprising a linear classifier, the neural network having been defined by training with a sum of hue and reflectance cross-entropy losses determined from i) the outputs of the hair hue values ​​and the hair reflectance values,and ii) the target labels for hair tone and hair reflectance, the target labels being prepared for the training images from the respective expert votes by a plurality of experts.

[0024] Statement 18: A computer software product according to Statement 17, wherein execution of the instructions causes the computer to provide hair shade and hair reflectance to train a generative model to simulate hair color.

[0025] Statement 19: A computer software product according to Statement 17, wherein execution of the instructions causes the computing device to classify hair shade and hair reflectance in accordance with a plurality of respective classes to follow an industry standard for hair color and wherein the hair reflectance comprises a primary reflectance component and a secondary reflectance component.

[0026] Statement 20: A computer software product according to Statement 17, wherein one or both of: execution of the instructions causes the computing device to provide the neural network for classification wherein the network having been trained with respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of reflectance annotator confusion matrices, wherein n is defined from a total number of experts providing the expert votes, and wherein the output of the hue classifier is multiplied by a respective hue confusion matrix from the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective reflectance confusion matrix of the plurality of reflectance confusion matrices to predict the vote of a respective expert of the plurality of experts; and wherein the neural network and the respective annotator confusion matrices are trained together; or execution of the instructions causes the computing device to output the neural network for classification wherein the network has been trained in accordance with a Student-Teacher framework. Brief description of the drawings

[0027] [Fig. 1] [Fig. 1] is a block diagram showing a classifier model according to the prior art.

[0028] [Fig.2] [Fig.2] is a block diagram showing a reference classifier model according to the prior art.

[0029] [Fig.3] [Fig.3] is a block diagram showing the reference classifier model of [Fig.2] as fitted with annotator confusion matrices in accordance with one embodiment.

[0030] [Fig.4] [Fig.4] is a diagram of a representative annotator confusion matrix.

[0031] [Fig.5] [Fig.5] is a block diagram showing the reference classifier model of [Fig.2] as adapted with a Student-Teacher framework in accordance with one embodiment.

[0032] [Fig.6] [Fig.6] is a graph, according to one embodiment, showing a comprehensive comparison of model prediction accuracy results between all models and an average human expert.

[0033] [Fig.7A] [Fig.7A] is a graph showing, according to one embodiment, an accuracy comparison of different numbers of training selfie images.

[0034] [Fig.7B] [Fig.7B] is a graph showing, in accordance with another embodiment, a comparison of accuracy of different numbers of annotators.

[0035] [Fig.8] [Fig.8] is a block diagram of a computing device according to one embodiment.

[0036] [Fig.9A] [Fig.9A] is a block diagram of a network system according to one embodiment.

[0037] [Fig.9B] [Fig.9B] is a block diagram of a network system in accordance with another embodiment.

[0038] [Fig.9C] [Fig.9C] is a block diagram of a network system according to another embodiment. DETAILED DESCRIPTION

[0039] The systems, methods, and techniques herein are directed toward improving existing hair color testing technology by leveraging artificial intelligence (AI). Many current hair simulation efforts are primarily based on traditional computer vision engineering techniques, and sometimes hair texture in simulated images appears synthetic—e.g., overtly simulated and poorly simulated. The current applicant seeks to use AI to improve the photorealism of simulated hair images while maintaining color accuracy on the simulated hair. A primary objective is color accuracy, and a hair color classification model is described herein in one or more embodiments, for identifying hair color in selfies.

[0040] According to one embodiment, hair color is defined based on L'Oréal's hair color definition for hair dye products and classifies hair color based on 1) hue and 2) reflectance, namely primary and secondary reflectance. Hair shades are divided into 10 classes, with 1 being the lightest and 10 being the darkest. In one embodiment, there are 9 reflectance classes for each of primary reflectance and secondary reflectance, where primary reflectance indicates the dominant reflectance in the hair, and secondary reflectance indicates the second dominant reflectance. Hair may have only primary reflectance or no reflectance. Considering no reflectance as a separate reflectance class, there are 100 combinations of primary and secondary reflectances.Examples: no primary or secondary reflectance, gold primary reflectance without secondary reflectance, ash primary reflectance without secondary reflectance, gold primary reflectance with ash secondary reflectance and gold primary reflectance with mahogany secondary reflectance, etc.

[0041] Labeled datasets from hair color experts exist or can be obtained for hair color. However, such datasets, particularly their use, present challenges for building a hair color identification model. For example, hair colors are defined based on the judgment of professional hair experts instead of scientific color definitions, such as the average RGB color. Therefore, the existing color definition is subject to human bias. Since the annotations were provided by hair experts, the labeled data is expensive to obtain and may result in a limited number of labeled examples to train and verify with, etc.

[0042] It was desirable to develop a hair color classifier to produce color predictions at least as accurate as those of an average human hair expert. Annotations of 1000 selfies from 10 hair experts were received to define a dataset to be used to develop the model - the "expert dataset". The performance of the model was evaluated based on whether it matched the majority of the hair experts' annotations or how far the predictions deviated from the expert annotations.

[0043] The expert dataset was relatively small, especially since the data were obtained gradually as the experts performed their tasks at different times. Furthermore, the combined results illustrated the differences in expert opinion. 1000 labeled images is a relatively small dataset given that there were over 100 hair colors. After gradually receiving labels from other hair experts, it was determined that the labels from different hair experts often did not agree with each other. In some cases, there was no clear majority vote on hair color, and all 10 experts believed the hair was different colors. Therefore, one of the goals of the activities was to mitigate the impact of label noise.

[0044] To address these two challenges, a final model according to one embodiment includes two improvements on a standard (prior art) classifier (e.g. of [Fig.l]), by adding annotator confusion matrices and by using a semi-supervised Student-Teacher (Student-Teacher) learning framework. The addition of annotator confusion matrices aims to mitigate the label noise problem by modeling the biases of individual experts, while the Student-Teacher framework is a semi-supervised learning technique that mitigates the problem of limited labeled data by using unlabeled data during training. Label noise

[0045] In the real world, data are often labeled by multiple annotators. Since biases in human annotators are inevitable, these labels often contain noise, and annotators may disagree with each other. Studies show that the rate of corrupted labels in real datasets was in the range of 8% to 39%

[19]

[16] [9] [8]. A classic way to deal with label noise is to aggregate annotations using majority voting. However, the majority voting method has some limitations, for example, it only uses the class with the highest number of votes as the aggregated label and ignores the distribution of votes, and there may be scenarios where there is no clear majority vote. Another way to aggregate annotations is to use soft annotations (or "soft labels" in English) in which the values ​​of the soft annotations are based on the empirical distribution of votes.

[0046] Model-based approaches have also been explored to handle label noise, including using a translation matrix to measure noise and reweighting labels based on confidence in them. The transition matrix is ​​widely used to model the relationship between the true distribution and the observed distribution [5]

[13] . However, these methods often do not take into account that different annotators may have different biases. On the other hand, loss reweighting has also been studied to improve model robustness by assigning more weight to labels or models with higher confidence

[10] . In particular, Weighted Doctor Net [4] proposed to model individual annotators separately and then combine the predictions of individual models using weights resulting from training.This method takes into account the overall skill level of annotators across all data classes and ignores the scenario that annotators may have different skill levels when labeling different classes. Tanno et al.

[17] were inspired by this and combined the use of a transition matrix with a novel loss weighting and proposed to model annotator labels and real data separately using annotator-specific confusion matrices. Annotator confusion matrix

[0047] The annotator confusion model

[17] is a probabilistic model based on assumptions that annotators are independent and label noises are independent of input images. Based on these two assumptions, the joint probability of observing n labels for an image from n annotators can be expressed as: [00481

[0049] where y(n) is the annotation of the nth annotator and x is the image, y is the actual color and a(1) y"',yc is a value in the confusion matrix that indicates the probability that the annotator indicates an image as being of class y given that the actual class is y. During training, the annotator-specific confusion matrices are trained together with the classifier, in which the classifier learns to predict the actual distribution of labels and each confusion matrix learns the observed distribution of labels based on the actual labels for a particular annotator. Semi-supervised learning

[0050] Labeled data is often expensive to obtain, whereas unlabeled data is usually abundant and provides additional information about the distribution of the data. To take advantage of the information contained in unlabeled data, semi-supervised learning (SSL) [3]

[21] is a research area that explores ways to train models using both labeled and unlabeled data. Typical semi-supervised learning techniques include consistency regularization and pseudo-labeling [7].

[0051] Consistency regularization was first introduced by the F-model

[14] and suggests that the model should give similar data points with consistent outputs. To encourage models to do so, the F-model applies a consistency loss between data points with and without noise. Following this, Temporal Assemblage [6] proposes to improve the model by maintaining an exponential moving average (EMA) prediction of the training data. Student-Teacher

[18] is an extension of the F-model and Temporal Assemblage, and suggests maintaining an exponential moving average of the model weights across a student-teacher framework. The student model is trained based on a supervised classification loss and an unsupervised consistency loss with the teacher model. The consistency loss is applied to minimize the differences between the results of the student and teacher models.The teacher model weights are updated based on the EMA of the student model. The Student-Teacher framework has achieved excellent results for semi-supervised learning image classification benchmarks.

[0052] Recent state-of-the-art models in SSL often combine several SSL techniques, including combining consistency regularization with pseudolabeling [2][ 1][ 15]. FixMatch

[15] is based on the idea that data augmentation has a significant impact on consistency regularization

[20] , and proposes to train the classifier to process consistent output for various augmentations of the same output with the use of pseudo-labels. This method produces excellent results in image classification tasks. However, since they rely on strong augmentations that change the color of images, they are not applicable for color classification. Model architecture Reference model

[0053] [Fig. 2] shows a block diagram of a processing flow 200 in accordance with one embodiment in which a reference model 202 receives an image 204 as input and outputs predictions 206 including hair hue 206A and reflectance 206B. In one embodiment, the reference model Reference 202 uses ResNetl8 202A as a skeleton. The output of skeleton 202A is passed to two linear classifiers 202B and 202C which separately predict hue 206A and reflectance 206B. In one embodiment, each linear classifier consists of two fully connected linear layers with a ReLu activation layer between the two layers. In one embodiment, stream 200 represents a training stream at a training time. Components 202 are retained for inference time use.

[0054] In one embodiment, the reference model 202 was trained using a sum of hue and reflectance cross-entropy losses (e.g., 208) between the model outputs 206A / 206B and the target colors (e.g., soft annotations 210) prepared from the expert labels.

[0055] “Soft” and “hard” annotations (“Soft” and “hard” labels in English)

[0056] When training the reference model of [Fig.2], the “soft” annotations » and « hard » annotations are tested. Hard annotations for hue and reflectance are calculated separately based on the class with the highest number of votes. Soft annotations are calculated based on the empirical distribution of labels across all annotations. For example, if nine experts label the image as class 1 and one expert labels it as class 2, the soft annotation will be 0.9 for class 1, 0.1 for class 2, and 0 for all other classes. Regardless of soft or hard annotations, the baseline models are trained based on the cross-entropy loss (e.g., 208).

[0057] Classifier with annotator confusion matrices

[0058] [Fig. 3] is a block diagram of a processing flow 300 showing a classifier 302 in accordance with an embodiment of [Fig. 2]. The classifier 302 is adapted by training using annotator confusion matrices 304 as described in more detail. The processing flow 300 is useful at a training time and the components 302 are useful at an inference time.

[0059] In one embodiment, components 302A-302C of classifier 302 are structured similarly to components 202A-202C of classifier 202 but are adapted, for example, by training with the annotator confusion matrices 304 as described. Output hue 306A and reflectance 306B are also similar to outputs 206A and 206B, but will depend on the trained classifier 302. The outputs of the respective classifiers for the same input image may vary in value.

[0060] According to one embodiment, the annotator confusion matrices 304 comprise a plurality (n) of hue annotator confusion matrices 304A and a plurality (n) of reflectance annotator confusion matrices 304B. Here, "n" refers to the number of annotators.

[0061] [Fig. 4] is an illustration of a confusion matrix of representative hue annotators 400, in accordance with one embodiment. The rows represent actual classes (“Current Values”) and the columns represent observed annotations (“Expert Annotations”).

[0062] In one embodiment, the classifier 302 with the annotator confusion matrices 304 is built on top of the reference model 202, having annotator-specific confusion matrices 304 after the respective classification layers (202B / 202C) of the reference model 202 to predict the observed labels 306A / 206B from the annotators. Assuming there are "n" annotators, n 10*10 confusion matrices are added after the hue classifier and n 100*100 confusion matrices are added after the reflectance classifier. The outputs 306A / 306B of the hue and reflectance classifiers 302B / 302C are multiplied by the respective confusion matrices 304A / 304B to predict the annotations of individual annotators. For example, the outputs of the hue classifier 302B are multiplied by the first hue confusion matrix (e.g. 307) to predict the hue annotation for the first annotator.

[0063] During training, for each training data, the cross-entropy loss sums of n hues and n reflectances (e.g., 308) are calculated to update the model. Since an annotator has only one hue label and one reflectance label for each training data, each cross-entropy loss was calculated using hard annotations (e.g., 310) instead of soft annotations (e.g., 210). The confusion matrices are first randomly assigned with values ​​such that the diagonal values ​​dominate and the matrices are approximately identical. The matrices are trained with the rest of the model based on the classification loss 308, and a regularization term is added to encourage the trained matrices to converge to the actual confusion matrices (e.g., according to the example in [Fig. 4]). Student-teacher framework

[0064] [Fig. 5] is a block diagram of a processing flow 500 showing the reference classifier model 202 of [Fig. 2] adapted to a Student-Teacher framework for training. The flow 500 includes a student model 502A and a teacher model 502B, according to one embodiment. The models 502A and 502B are configured as instances of the model 202 so that details are not shown in [Fig. 5]. The student model 502A produces an output including the hue output 506A and the reflectance output 506B and the teacher model 502B produces an output including the hue output 506C and the reflectance output 506D, respectively.

[0065] The student model 502A and the teacher model 502B are first pre-trained using the labeled data. During the semi-supervised learning process, two augmented versions of the same image (e.g., 504A and 504B) are fed separately to each of the models 502A / 502B. The student model 502B is trained using the cross-entropy loss (supervised) 508 and the consistency loss (unsupervised) 512. The cross-entropy loss (classification loss) is calculated based on the labeled data (e.g., 510), and the consistency loss is calculated as the mean square error of the softmax outputs (506A, 506B, and 506C, 506D) between the student model 502B and the teacher model 502A. The consistency loss 512 based on the augmented images 504A, 504B forces the student model 502A to provide consistent outputs at similar data points.After updating the student model 502A, the teacher model 502B is updated based on the exponential moving average of the weights of the student model 502A. According to one embodiment, during inference, only the teacher model 502B is retained and used for evaluation. Experiences Dataset

[0066] The selfie dataset used during the experiments contained 1000 labeled images and 2500 unlabeled images. For the labeled data, the selfie hair was annotated by 10 professional hair experts and each label contained a hue, a primary reflectance, and a secondary reflectance. In one embodiment, the hue values ​​range from 1 to 10 where 1 corresponded to the darkest hue, and the primary and secondary reflectance values ​​range from 0 to 8. The hair color may also have no secondary reflectance or primary reflectance.

[0067] Labels from different annotators may not match. However, for most labeled images, there were at least two annotations that matched. Of the 1000 images, only 40 had no agreement. When looking at hue, primary, and secondary reflectance values ​​individually, at least two annotators always agreed. To assess annotator quality, Table 1 lists the accuracy and average absolute hue difference between individual annotator labels and the majority vote. The overall agreement on color was about 30%, with experts disagreeing more on reflectance values ​​(49%) than on hues (64%). In particular, experts agreed on secondary reflectance values ​​only 42% of the time.

[0068] [Table 1]

[0069] Table 1

[0070] Experiment Configuration Annotator Avg. 1 2 3 4 5 6 7 8 9 10 Overall Accuracy (%) 30 33 20 31 21 41 39 34 30 28 21 Hue Accuracy (%) 64 81 55 68 53 70 73 66 59 61 55 Reflectance Accuracy (%) 49 45 52 18 18 59 65 67 63 58 49 Primary Reflectance Accuracy (%) 48 56 47 26 29 59 56 56 52 57 41 Secondary Reflectance Accuracy (%) 42 33 31 38 32 54 55 49 49 51 31

[0071]

[0072]

[0073]

[0074] The baseline model 202 and classifier 302 with 304 confusion matrices were trained using the stochastic gradient descent (SGD) optimizer, the learning rate was constant for the first 20 epochs and then decreased linearly to zero over the next 65 epochs. Simple data augmentations, including random horizontal and vertical flips, rotations, and translations were used during training. The batch size was set to 128. The teacher model of the Student-Teacher 502B framework was trained using the SGD optimizer, the learning rate was reduced to zero using cosine cancellation

[12] . For data augmentation, random data augmentations including horizontal and vertical flip, rotation, translation, and shear were used. The batch size was set to 256, with 128 labeled data and 128 unlabeled data. Evaluation metrics In one embodiment, model performance was evaluated based on the majority of experts' acceptance of the model and the magnitude of the discrepancy between predictions and annotations. Three main metrics were used for evaluation: accuracy, hue within ±1, and mean absolute difference in hue. For hue, reflectance, and overall color, accuracy was measured in terms of the model prediction corresponding to the color that received the highest number of votes from the ten annotators.Since hues are ordinal data, two additional metrics were used: hue ±1 measures the probability that the model's predictions fall within one hue difference from the highest-voted hue, and mean absolute difference measures the average distance between predicted hues and the average hue annotations for each image. The results . of the experiment were based on 5-fold cross-validation and, at each run, the metrics of the best period were reported. Results Soft annotations / hard annotations

[0075] Table 2 compares the prediction performance between an average human expert, the reference model trained with hard annotations, and the reference model trained with soft annotations, in accordance with one embodiment. In the case of disagreements between experts, training the model with soft annotations allowed for learning more about the distribution of votes and making better predictions. The prediction accuracies of the model trained with soft annotations were 9% higher than those of the model trained with majority votes, with a 7% increase for hue accuracy and a 6% increase for reflectance accuracy. Another observation is that the model trained with majority votes was slightly worse than an average human expert, whereas the model trained with soft annotations was better than the average human expert.

[0076] [Table 2]

[0077] Table 2 Classifier with confusion matrices Human Expert Majority Soft Annotations Overall Accuracy (%) 30±7 29+4 38+4 Hue Accuracy (%) 64±9 62+6 69+6 Reflectance Accuracy (%) 49±18 41+5 47+4 Primary Reflectance Accuracy 48±12 60+6 66+6

[0079] In one embodiment, stepwise experiments were conducted to examine the impacts of adding confusion matrices to the model and training the model using the Student-Teacher framework. In brief, either model was better than the performance of an average human expert. As shown in Table 3, adding the confusion matrices to the model resulted in an improvement in overall accuracy for colors (2%). It is noted that the baseline model trained with soft annotations is a special case of the model with confusion matrices where all confusion matrices were identity matrices. The improvement in color classifications is primarily due to the improvement in reflectance prediction (5%) and the prediction accuracy shades is similar to the reference model. It is understood that this results from more disagreements on reflectance labels among hair experts.

[0080] [Table 3]

[0081] [Tables3] Human Expert Reference Confusion Average Overall Accuracy (%) 30+7 38+4 40+6 50+5 Hue Accuracy (%) 64+9 69+6 67+3 73+4 Reflectance Accuracy (%) 49+18 47+4 53+6 58+5 Primary Reflectance Accuracy (%) 48+12 66+6 68+6 73+2 Secondary Reflectance Accuracy 42+10 54+3 60+5 63+3

[0082] Although the addition of annotator confusion matrices did not improve hue accuracy, a more detailed analysis showed that its prediction deviated less from expert annotations. Table 4 shows a comparison of hue prediction performance between human annotators, the baseline model with soft annotations, the classifier with annotator confusion matrices, and the Student-Teacher framework. As shown in Table 4, after adding the confusion matrices, the percentage of hue within ±1 of the hue with the highest number of votes was 1% higher, and the mean absolute differences were 0.04 lower. This revealed that hue prediction was also improved.

[0083] Student-Teacher Confusion Matrix Reference Model

[0084] [Table 4]

[0085] Table 4 Student-teacher framework Reference model Confusion matrix Student-teacher Hue at +1 (%) 97+0 98+0 98+2 Diff. Abs. Mean Hue 0.44+0.07 0.40+0.07 0.32+0.07

[0087] After integrating the Student-Teacher framework, the overall performance of the model was significantly improved. Table 3 and Table 4 show that the framework improved the prediction accuracy by 10% compared to the confusion matrix model, and it also reduced the difference between the model prediction and expert annotation. The average absolute difference was reduced by 0.08. Furthermore, when comparing the Student-Teacher framework-based classifier and an average human expert, there was a 20% improvement in overall color accuracy with a 9% and 11% improvement in hue and reflectance accuracy. [Fig. 6] is a graph 600, according to one embodiment, showing a comprehensive comparison of model prediction accuracy results between all models and an average human expert. Graph 600 shows that there is a significant improvement between the accuracy of an average human expert and the Student-Teacher framework model. Ablation study

[0088] To facilitate better planning of data collection, an experiment was conducted to study the impact of having additional labeled selfies and annotators. When conducting the following experiments, the baseline model with soft annotations was trained to evaluate the impact of having more images and more labels per image. When examining the impact of additional labeled images, the number of labeled data varied from 100 to 900. When examining the impact of additional labels per image, labels from a subset of experts were used to train the model.

[0089] The model's performance improved when it had more selfies or more annotators than expected. However, as shown in Figures 7A and 7B, the model improved more when it had more labeled images. [Fig.7A] is a graph 700 showing an accuracy comparison of different numbers of training selfies and [Fig.7B] is a graph 710 showing an accuracy comparison of different numbers of annotators.

[0090] It was observed that a consistent increase in accuracy is achieved when increasing the training images from 100 to 900, where the overall, hue, and reflectance accuracies increased in parallel. No downward trend was observed in the improvement even after using 900 images. In contrast, minimal improvement was observed in the overall accuracy between 5 and 10 annotators, in particular, minimal improvement was observed in the reflectance accuracy after having 8 annotators. This showed that having additional labeled images was more effective in improving the model performance. Hair simulation application

[0091] [Fig. 8] is a block diagram of a computing device 800 in accordance with one embodiment. In the embodiment, a color classifier in accordance with one embodiment hereof is integrated with a computer application VTO ​​for providing a hair color simulation. Computing device 800 provides a user computing device such as a smartphone, tablet, laptop, or other computing device for use by a user such as a consumer of hair dye products or a salesperson assisting such a user. The device may include a component of a larger form factor such as a kiosk for placement in a retail environment, for example. Device 800 is non-limiting and is simplified for brevity.

[0092] The device 800 includes a storage device 802, a processing unit 806, a camera 808, a microphone / speaker 810, a display screen 812, and a communication subsystem 814. In one embodiment, the storage device 802 includes a memory device, for example, one or more types of memory such as RAM, ROM, etc. The storage device 802 may include a long-term storage device such as a solid-state drive (SSD) or other type of drive for storing non-volatile data, etc. The storage device 802 stores computer-readable instructions for execution by the processing unit, such that once executed, the instructions cause the computing device to perform operations such as one or more methods.The processing unit 806 includes one or more central processing units (e.g., CPUs), and / or graphics processing units (e.g., GPUs) having one or more processors / microprocessors, controllers / microcontrollers, etc. Other types of processors may be used. GPUs may be particularly useful for accelerating graphics processing tasks and / or AI processing tasks (e.g., training and / or inference).

[0093] The camera 808 may be used to take selfies. The microphone and speaker 810 are generally separate devices, but labeled together here for convenience and represent some of the input (I), output (O), or I / O devices that may be available. Other devices may include a light, buzzer, vibrator, button, keyboard, pointing device (e.g., mouse or touchpad, etc.), etc.

[0094] The display screen 812 presents images such as components of a graphical user interface, camera images, etc. In one embodiment, the display screen is a touchscreen device, a type of I / O device, configured to receive gesture inputs (e.g., swipe, tap, etc.) that interact with the region(s) of the screen and in association with the user interface components (e.g., controls) presented by an application executed by the processing unit 806.

[0095] The communication subsystem 814, in one embodiment, is configured to manage communications between device components and / or between the computing device and external devices such as a remotely located computing device (e.g., web service, cellular network component, printer, etc.). A subcomponent thereof in one embodiment is an antenna for wireless communication. A subcomponent thereof in one embodiment is a wired communication interface (e.g., Ethernet, USB A, USB C, Thunderbolt (TM from Intel Corporation), etc.) to be coupled to a cable suitable for wired communication.

[0096] The storage device 804 stores components of a VTO application and corresponding data (e.g., 820). Representative components are shown. The VTO application 820 includes the user interface component 822 (e.g., screens, instructions, icons, controls, etc.). The user interface provides outputs to a user and receives inputs such as inputs for the application workflow, user selections of color choices, etc. A color classifier 824 is provided and includes one of classifiers 202, 302, and 502 as previously described for determining hair color information (e.g., hue and reflectance). A VTO pipeline 826 is provided for simulating hair color in association with (e.g., on or in) an input image. A color recommendation engine 828 and an associated data store 830 are provided.In one embodiment, the data store 830 stores hair color data and images representative thereof, such as in the form of a color strand or colored hair image, hair reflectance images, etc. The user interface may present one or more choices that a user can select via user input, e.g., sorting or filtering examples and presenting them in groups / pages, etc. In one embodiment, the color recommendation engine 828 may include an interface to a chatbot or live agent to discuss a recommendation.

[0097] In one embodiment, recommendations may be made, for example, based on recommendation factors such as a user's personal information, including the user's current hair color and age; hair color trend data; availability of a product and / or service local to the user (e.g., within a radius); cost information, etc. In one embodiment, rules or another manner of recommending may be used to determine which recommendation (or recommendations) to present to a user.

[0098] In one embodiment, a user may provide an input image with hair (e.g., 832) similar to the input image 204, for classification. Hue and reflectance data 834 are generated by the classifier 824. The hue and reflectance data may be made available to the color recommendation engine 828 for processing (e.g., using color matching rules, color compliment rules, etc.) to select one or more colors from the color data storage 830 to recommend to a user via the user interface component 822. The recommendation may be displayed on the screen 812.The user may invoke the VTO engine (via input to a command) to have the engine simulate color and / or color and reflectance (e.g., target hair) using the image 832 and the target hair data as input to produce an output image with simulated hair 836.

[0099] The output image with simulated hair 836 may be presented via the user interface component 822 on the display screen 812. A before and after display may be provided for comparison. The comparison between two or more simulated colors (e.g., two different output images) may be displayed for comparison. The products or services or both may be purchased via the interface 838.

[0100] Such an interface 838 may direct the user (e.g., the computing device) to a web-based e-commerce service (e.g., a website (not shown)) to make the purchase, reservation, or the like.

[0101] In one embodiment, the VTO pipeline 826 includes a generative neural network (e.g., a model) configured to generate a simulated hair image using the input image 832 and the target hair data as input. Refer to Figure 9 discussed below.

[0102] Other components stored in the storage device 804 include an operating system 840, a browser 842 (e.g., for browsing web pages), an email and / or messaging application 844 (e.g., SMS or other type), and a social networking application 846. The output image with simulated hair 836 may be shared (e.g., communicated) via the applications 844 and / or 846, for example.

[0103] In another embodiment of using the color classifier not shown in [Fig. 8], the color classifier 824 processes an output image from a generative model configured to generate a simulated hair image using an input image having hair and target color data as input. The classifier determines the color and / or reflectance data such as for comparison with target hair data to confirm the accuracy of the generative model.

[0104] Classifier-Guided Training of a Color Refining Neural Network

[0105] In one embodiment, a classifier may guide the training of a color refining neural network that is configured to produce rendered images with a target hair color. It is difficult to assess the accuracy of hair rendering based solely on RGB values ​​or simple metrics. In one embodiment, there is a feedback loop in a rendering network system to use a color classifier trained in accordance with an embodiment herein to guide the rendering network. [Fig. 9A] is a block diagram of a network system 900A in accordance with an embodiment providing a training system. [Fig. 9B] is a block diagram showing additional details of a training network system 900B linked to the system 900A, in accordance with one embodiment.The method, computer program product and other aspects will be readily understood by those skilled in the art from an understanding of the training aspects and inference aspects shown and described.

[0106] The network system 900A displays the input image 104 (e.g., image 1) and a target hair color 902 provided to a hair analysis engine 904 to determine hair information for the hair pixels (e.g., from hair segmentation), e.g., hair histogram color data for image 1. In one embodiment, the image color data 1, target hair 902 (e.g., as a hair strand image), and hair histogram data comprise RGB data (e.g., 256*3) commonly used for images. In one embodiment, the hair analyzer engine 904 comprises or communicates with a hair analyzer such as a hair classifier network to determine the hair pixels in the input image 104.In one embodiment, the image I and color information (e.g., histogram data) for the target hair color 902 and the hair in the image I (e.g., as a histogram (not shown as such in [Fig. 9A])) are provided to a color refinement neural network 906 to produce a rendered image 908. In one embodiment, as shown in [Fig. 9B], the color refinement neural network 906 includes a color mapping network and a generative network configured to modify features of an image it processes, namely the hair color. The generative network in one embodiment includes a model (e.g., generator (G)) that is defined by training guided by the hue and reflectance classifier 910.

[0107] Referring again to [Fig. 9A], the rendered image 908 is provided to the color classifier 910 defined (e.g., trained) in accordance with one embodiment herein to classify the hue and reflectance properties to produce the color prediction 912 for the image 908. The loss 914 represents a loss determined from the color prediction 912 and the target hair color 902 as used to train the color refinement neural network 906 as described in more detail with [Fig. 9B]. In one embodiment, the system 900A is configured with one or more computing devices (not shown) to provide the computing components (e.g., 904, 906, 910, etc.) and to store data (e.g., 104, 902, 908, 912, 914, etc.).A display device (not shown) may be included to display the image 104, 908 and any of the other data, including the output (not shown) of the hair analyzer engine 904 provided to the color refinement neural network 906.

[0108] In one embodiment, network system components 900A and 900B may be configured as an inference time system (e.g., following training) to provide a virtual try-on experience for simulating a hair color applied to an image. For example, components 904 and 906 are useful for defining a VTO pipeline (e.g., 920) for simulating hair color, processing an input image to simulate a target hair color, and producing a simulated image. In one embodiment, color refinement neural network 906 includes a neural network defined by training as guided by color classifier 910 as described in more detail below.

[0109] According to Figures 9A and 9B, a color mapping network 922 is trained to predict correct rendering parameters based on the classification results of the color classifier (910). As shown in [Fig.9B], the pipeline uses the color mapping network 922 and a generative adversarial network (GAN) 924 with a generator (G)924A to simulate hair rendering 908 (i.e., (G} )).

[0110] A hair segmentation model (e.g., as a component of hair analyzer engine 904, and not shown in [Fig.9B]) is used to extract hair masks and engine 904 provides RGB histogram data of hair pixels (Hl) in (i.e., from) image 1 104. Color mapping network 922 includes an encoder 926 that encodes image features (F) from image 1. Encoder 926 encodes features including, but not limited to, color and lighting of the hair representation and, in the generator, is used to re-generate the same hair rendering. [YES]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117]

[0118]

[0119]

[0120]

[0121] A concatenator 928 of the color mapping network 922 receives inputs: i) sample (i.e., target hair color) RGB histogram (H r) 902, ii) source (image 7) hair RGB histogram (777 ) 930 and iii) image features (F) from the encoder 926. The three inputs are concatenated and passed through a block of fully connected layers 932 for output color mapping (Mp). The output color map Mp and the source RGB hair histogram H1 930 are concatenated and used as input and passed through another block of fully connected layers (FC Block 2 (934)) to obtain the output RGB histogram (770) to be used by the generator (G)924A to produce its output from frame 7. Thus, the output rendered image G i 908 is produced by replacing the hair pixels with the color map. The internal color maps (Mc ) and RGB histograms (77G ) are used as real data labels to train the color mapping network 922 in a supervised manner. In one embodiment, the following losses are used to train the color mapping network 922: PcrtC — L Color Mapping + P RGB Histogram L2 loss between the real data color mapping (M c) and the network predicted color mapping: Color lcatorgraphy — G " ^p) L1 loss between histograms of real data RGB histogram (77c) and output RGB histogram (770): ^Histogram-RGB ~ E,»c- HJ with the trained color mapping network 922, the GAN 924 may be trained, including as follows, according to one embodiment. The output RGB histogram (Ho ) from the color mapping network 922 is used as a condition with the source image 104 as inputs to the generator (G) 924A of the GAN 924. In one embodiment, the GAN 924 is defined as a StarGAN (in accordance with the teaching of Choi, Y., et al., “StarGAN: Unified Generative Adversarial Networks for Multi-Domain Histogram-to-Image Translation,” IEEE Conference on Pattern Recognition and Computer Vision (CVPR), 2018, pp. 8789-8797), such that G7 = G(I, Ho). It is noted that the previously trained 922 color mapping network is frozen during GAN training. That is, the 922 network is not trained or co-trained with the 924 GAN training. The discriminating part (discriminator (D) 924B) of GAN 924 uses two classifiers, one (936) which classifies whether the image is fake or real ((G7 ) or 7 ); and the other (910) is the pre-trained classifier that classifies the two parts of color: hue and reflectance.

[0122] The pre-trained classifier guides and enhances the generative capabilities of the GAN network and can be used independently with any GAN architecture for guidance.

[0123] The losses used for training the generator 924A and discriminator 924B parts are the same as those defined in the StarGAN document referenced above.

[0124] In one embodiment, the color mapping + GAN is labeled as a color refinement network because it uses the instructions from the pre-trained hair classification model and the losses indicated above in order for the generative model to refine its outputs.

[0125] As shown in the simplified block diagram of [Fig.9C] illustrating an application of VTO, in one embodiment, at inference time (e.g., when training is complete and the refinement network is provided to generate output images in particular from user input), the discriminator component 924B with its two classifiers 910 and Conclusion

[0126] In this work, techniques, etc. are provided to address the problem of limited availability of labeled data and label noise caused by human biases, to classify color attributes in images, through a combination of a semi-supervised learning framework and annotator confusion matrices. The final model achieved 20% higher accuracy than an average human expert.

[0127] The experiments described herein have shown that the confusion matrix is ​​useful for improving color prediction, especially in scenarios where there is more disagreement between experts. The approach primarily considers label noise caused by individual expert biases and assumes that biases depend only on the actual class (e.g., without considering other factors, such as lighting and contrast with the background). Due to the noisy nature of subjective annotation, in one embodiment, soft annotations are used but not majority votes because the training labels and confusion matrices are adapted to represent different ideas of the experts.

[0128] The experiments described herein also showed that the Student-Teacher framework helps to significantly improve color prediction. The effectiveness of the current framework depends on the teacher model providing good labels for training the student model.

[0129] Unlike normal RGB value prediction, the classification model is defined to classify according to a color standard defined specifically for hair. In one embodiment, such a standard is three digits, with the first digit representing natural hair tones and the second and third digits representing the primary and secondary reflectance of the hair. Such color prediction adapts to different lighting conditions and therefore accurately predicts dark hair in bright light and light hair in dim light. The color representation is light independent. The color number prediction can be directly related to how hair products are color coded (e.g., using the same color standard), which generally cannot be expressed in RGB or simple text information.

[0130] The model structure uses one branch for natural hue (first digit) and one branch for both reflectance determinations together to achieve better performance.

[0131] The following aspects and characteristics cited in the numbered declarations (Declaration-Sim 1, Declaration-Sim 2... Declaration-Sim 17) will emerge clearly from the disclosure herein, in particular:

[0132] Statement-Sim 1: A computing device comprising a processor coupled to a storage device that stores instructions executable by the processor to cause the computing device to: process an input image and a target hair color using a virtual try-on (VTO) pipeline to produce a VTO experience that simulates the target hair color with the input image to produce an output image;wherein the VTO pipeline comprises a color refinement neural network for generating the output image combining the target hair color and the input image, the color refinement neural network comprising a generative neural network trained under the supervision of a hair classification network comprising a hue classifier that outputs a hair hue value and a reflectance classifier that outputs a hair reflectance value to determine training loss information from output images for training the generative neural network. ;

[0133] Statement-Sim 2: A computing device according to Statement-Sim 1, wherein the instructions are executable by the processor to cause the computing device to provide an interface with one or both of: i) a color recommendation engine for recommending a target hair color and ii) an e-commerce service with which to purchase one or both of a product or service.

[0134] Statement-Sim 3: A computing device according to Statement-Sim 1, wherein the VTO pipeline comprises a color mapping network trained to produce a color map for input to the generative neural network to produce the output image for the VTO experiment, the color map produced from features of the image, image hair color data determined from the image, and the target hair color.

[0135] Statement-Sim 4: A computing device according to Statement-Sim 3, wherein the color mapping network comprises an encoder for determining features from the input image; a concatenator for combining the features, the target hair color, and the image hair color data for processing by a first fully connected block, and a second block for processing an intermediate map from the first block with the image hair color data to produce the color map for input to the generative neural network.

[0136] Statement-Sim 5: A computing device according to Statement-Sim 4, wherein the VTO pipeline includes a hair analysis engine for determining image hair color data from the input image.

[0137] Declaration-Sim 6.'Computer device according to Declaration-Sim 1, in which the input image and the target hair color are defined using RGB values, such that the generative neural network is trained to produce an output image using particular RGB values ​​as guided during training by the hair classification network using hue values ​​and hair reflectance values.

[0138] Statement-Sim 7: A computing device according to Statement-Sim 6, wherein the hair classification network is defined and trained according to an industry standard for hair color classification comprising a respective plurality of classes for hair hue and hair reflectance.

[0139] Statement-Sim ^.'A computing device according to Statement-Sim 1, wherein the hair classification network comprises a coding skeleton and each of the hue classifier and the hair reflectance classifier comprises a respective linear classifier.

[0140] Statement-Sim 9: A computing device according to Statement-Sim 8, wherein the hair classification network has been defined by training with a sum of hue and reflectance cross-entropy losses determined from i) outputs of hair hue values ​​and hair reflectance values, and ii) target labels for hair hue and hair reflectance, the target labels being prepared for the training images of the hair classification network from respective expert votes by a plurality of experts.

[0141] Statement-Sim 70,-A computing device according to Statement-Sim 9, wherein the target labels comprise soft annotations for at least some of the training images, the soft annotation being determined from an empirical distribution of expert votes on the respective classes.

[0142] Statement-Sim 11: A computing device according to Statement-Sim 9, wherein one of: the hair classification network having been trained with respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of reflectance annotator confusion matrices, wherein (n) is defined from a total number of experts providing the expert votes,and wherein the output of the hue classifier is multiplied by a respective hue confusion matrix from among the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective reflectance confusion matrix from among the plurality of reflectance confusion matrices to predict the vote of a respective expert from among the plurality of experts and wherein the neural network and the respective annotator confusion matrices are trained together; or the hair classification network has been trained according to a Student-Teacher framework. ,

[0143] Statement-Sim 12: A method of configuring a neural network that classifies hair hue and hair reflectance on an input image, the method comprising: providing the neural network, the neural network comprising a coding skeleton coupled to i) a hue classifier that outputs a hair hue value and ii) a reflectance classifier that outputs a hair reflectance value, each classifier comprising a linear classifier; providing a plurality of training images associated with respective training labels for each of hair hue and hair reflectance, the target labels being prepared from respective expert votes by a plurality of experts;and training the neural network using the training images, wherein the training is performed according to a sum of hue and reflectance cross-entropy losses determined from i) the classifier outputs of the hair hue values ​​and the hair reflectance values, and ii) the target labels. ;

[0144] Statement-Sim 13: The Method according to Statement-Sim 12, wherein the training comprises training the neural network using a Student-Teacher framework.

[0145] Statement-Sim 14: The method of Statement-Sim 12, wherein training comprises training the neural network using respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of confusion matrices of reflectance annotators, wherein n is defined from a total number of experts providing the expert votes, and wherein the output of the hue classifier is multiplied by a respective hue confusion matrix from among the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective reflectance confusion matrix from among the plurality of reflectance confusion matrices to predict the vote of a respective expert from among the plurality of experts; and wherein the neural network and the respective annotator confusion matrices are trained together.

[0146] Statement-Sim 15: A method comprising: providing a generative neural network that generates an output image combining a target hair color with an input image; providing a hair classification network as a component of a discriminator of the generative neural network, the hair classification network comprising i) a hue classifier that determines an output hair hue and ii) a reflectance classifier that determines an output hair reflectance; determining training loss information using the target hair color and the output hair hue, and the output hair reflectance; and training the generative neural network under the direction of the hair classification network using the training loss information.

[0147] Statement-Sim 16: A method according to Statement-Sim 15, wherein the target hair color, the input image and the output image are defined using RGB type data and the output hair hue and the output hair reflectance are defined in accordance with an industry standard for hair color classification comprising a respective plurality of classes for hair hue and hair reflectance.

[0148] Declaration-Sim 7 7.-Method according to Declaration-Sim 15 comprising, before training the generative neural network, pre-training a color mapping network configured to process the input image and image hair color data from hair pixels of the input image and the target hair color to provide a color map to the generative neural network to generate the output image from the input image.

[0149] A practical implementation may include some or all of the features described herein. These and other aspects, features, and various combinations thereof may be expressed as methods, apparatus, systems, means for performing functions, program products, and other ways, combining the features described herein. A number of embodiments have been described. Nevertheless, it is understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other Steps may be provided, or steps may be eliminated, from the described process, and other components may be added to or removed from the described systems. Accordingly, other embodiments fall within the scope of the following claims.

[0150] Throughout the description and claims of this specification, the terms "include", "contain" and variations thereof mean "including but not limited to" and are not intended to (and do not) exclude other components, integers, or steps. Throughout this specification, the singular includes the plural, unless the context otherwise requires. In particular, when the indefinite article is used, this specification is to be understood as contemplating plurality as well as singularity, unless the context otherwise requires.

[0151] Features, integers, characteristics or groups described in conjunction with a particular aspect, embodiment or example of the invention are to be understood as being applicable to any other aspect, embodiment or example, unless inconsistent therewith. Any features disclosed herein (including the claims, abstract and accompanying drawings), and / or any steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of these features and / or steps are mutually exclusive. The invention is not limited to the details of the preceding examples or embodiments.The invention extends to any new feature, or any new combination, of the features disclosed in this specification (including any accompanying claim, abstract and drawing) or to any new feature, or any new combination, of the steps of any disclosed method or process. REFERENCES .

[0152] Berthelot, D., Carlini, N., Cubuk, ED, Kurakin, A., Sohn, K., Zhang, H., Raffel, C.: Remix match: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785 (2019)

[0153] Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, CA: Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing Systems 32 (2019)

[0154] Chapelle, O., Schôlkopf, B., Zien, A.: Introduction to semi-supervised learning (2006)

[0155] Guan, M., Gulshan, V., Dai, A., Hinton, G.: Who said what: Modeling individual labelers improves classification. Dans : Actes du congrès de l’AAAI sur l’intelligence artificielle, vol. 32 (2018)

[0156] Hendrycks, D, Mazeika, M, Wilson, D, Gimpel, K : Using trusted data to train deep networks on labels corrupted by severe noise. Advances in neural information Processing Systems 31 (2018)

[0157] Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 (2016)

[0158] Lee, D.H. et al.: Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. Dans : Workshop on challenges in représentation learning, ICML. vol. 3, p. 896 (2013)

[0159] Lee, K.H., He, X., Zhang, L., Yang, L. : Cleannet : Transfer learning for scalable image classifier training with label noise. Dans : Proceedings of the IEEE conférence on computer vision and pattern récognition, pp. 5447-5456 (2018)

[0160] Li, W., Wang, L., Li, W., Agustsson, E., Van Gool, L : Base de données Webvision : Visual learning and understanding from web data. arXiv preprint arXiv: 1708.02862 (2017)

[0161] Liu, T., Tao, D. : Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence 38(3), 447-461 (2015)

[0162] Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. Dans : Proceedings of International Conférence on Computer Vision (ICCV) (Décembre 2015)

[0163] Loshchilov, I., Hutter, F. : Sgdr : Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 1608.03983 (2016)

[0164] Patrini, G., Rozza, A., Krishna Menon, A., Nock, R., Qu, L. : Making deep neural networks robust to label noise: A loss correction approach. Dans : Proceedings of the IEEE conférence on computer vision and pattern récognition, pp. 1944-1952 (2017)

[0165] Rasmus, A., Berglund, M., Honkala, M., Valpola, H., Raiko, T. : Semi-supervised learning with ladder networks. Advances in neural information processing Systems 28 (2015)

[0166] Fils, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C.A., Cubuk, E.D., Kurakin, A., Li, C.L.: Fixmatch : Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing Systems 33, 596-608(2020)

[0167] Song, H., Kim, M., Lee, J.G.: Selfie: Refurbishing unclean samples for robust deep learning. Dans : International Conférence on Machine Learning. pp. 5907-5915. PMLR (2019)

[0168] Tanno, R., Saeedi, A., Sankaranarayanan, S., Alexander, D.C., Silberman, N.: Learning from noisy labels by regularized estimation of annotator confusion pp. 11244-11253(2019)

[0169] Tarvainen, A., Valpola, H.: Mean teachers are better rôle models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing Systems 30 (2017)

[0170] Xiao, T., Xia, T., Yang, Y., Huang, C., Wang, X.: Learning from massive noisy labeled data for image classification. Dans : Proceedings of the IEEE conférence on computer vision and pattern récognition, pp. 2691-2699 (2015)

[0171] Xie, Q., Dai, Z., Hovy, E., Luong, T., Le, Q.: Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems 33, 6256-6268 (2020)

[0172] Zhu, X., Goldberg, A.B.: Introduction to semi-supervised learning. Synthesis lectures on artificial intelligence and machine learning 3(1), 1-130 (2009)

Claims

Claims

1. A computing device comprising a processor coupled to a storage device containing instructions that, when executed by the processor, cause the computing device to: classify hair hue and hair reflectance on an input image using a neural network, the neural network comprising a coding skeleton coupled to a pair of classifiers comprising i) a hue classifier that outputs a hair hue value and ii) a reflectance classifier that outputs a hair reflectance value, each member of the pair of classifiers comprising a linear classifier, the neural network having been defined by training with a sum of hue and reflectance cross-entropy losses determined from i) the outputs of the hair hue values ​​and the hair reflectance values, and ii) the target labels for the hair hue and hair reflectance,the target labels being prepared for the training images from respective expert votes by a plurality of experts.,

2. The computing device of claim 1, wherein the neural network is configured to classify hair hue and hair reflectance according to a plurality of respective classes to follow an industry standard for hair color.

3. The computing device of claim 1, wherein the reflectance of the hair comprises a primary reflectance component and a secondary reflectance component.

4. The computing device of claim 1, wherein the target labels comprise soft annotations for at least some of the training images, the soft annotation being determined from an empirical distribution of expert votes on the respective classes.

5. The computing device of claim 1, wherein the neural network having been trained with respective annotator confusion matrices comprising a plurality (n) of hue annotator confusion matrices and a plurality (n) of reflectance annotator confusion matrices, wherein n is defined from a total number of experts providing the expert votes, and wherein the output of the hue classifier is multiplied by a respective hue confusion matrix from the plurality of hue confusion matrices and wherein the output of the reflectance classifier is multiplied by a respective reflectance confusion matrix from the plurality of reflectance confusion matrices to predict the vote of a respective expert from the plurality of experts and wherein the neural network and the respective annotator confusion matrices are trained together.

6. The computing device of claim 1, wherein the neural network has been trained in accordance with a Student-Teacher framework.

7. The computing device of claim 6, wherein the neural network comprises a teacher network obtained from the Student-Teacher framework.

Images