Machine image color extraction and machine image construction using the extracted colors
Through deep learning network and pixel clustering technology, the problem of difficulty in color extraction in cosmetic sample images is solved, and high-accuracy color recognition and extraction under different conditions is achieved, which is suitable for applications such as virtual makeup trials.
Patent Information
- Application Number
- CN202080046007.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-04-23
- Filing Date
- 2020-04-22
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-04-22
AI Technical Summary
The prior art has difficulty in automatically extracting colors from cosmetic sample images, especially in the case of changes in image style and lighting conditions, resulting in difficulty in color recognition and extraction, and lack of common data sets and robust methods.
Deep learning network and pixel clustering technology are adopted to identify and extract colors by converting image pixels from RGB color space to LAB color space, ignoring the brightness channel, using k-mean clustering and heuristic methods, combining deep learning classifiers to identify and extract images Features are clustered and classified to determine the intensity of color representation and candidate colors.
It realizes high-accurate color extraction under different image styles and lighting conditions, and can automatically and unbiasedly extract cosmetic colors from sample images. It is suitable for virtual makeup trials and other applications, improving the robustness and efficiency of color recognition.
Smart Images

Figure CN114144813B_ABST
Abstract
Description
Technical Field
[0001] The following relates to image processing such as for a machine, and more particularly, to extraction of colors from an image such as a sample, and color image construction using the extracted colors. Background Art
[0002] A swatch is a small sample of a product that is designed to show what the product will look like. One feature of the appearance that may be of interest is color.
[0003] In a machine environment, samples are often presented in the form of images. In some examples, the image may consist entirely of the sample, and in some examples, a portion of the image may contain the sample, such as in a region of interest.
[0004] While sighted people can generally easily identify colors from arbitrary samples, programmatically identifying colors for machines is a difficult task that remains unsolved. In addition to extracting colors, it may be desirable to apply or otherwise use the extracted colors to construct a new image. In some examples, for example, a new image is constructed by applying the extracted colors to the surface of an object in the new image. Summary of the invention
[0005] Systems and methods are provided for performing color extraction from sample images and defining new images using the extracted colors. A source image can be classified using a deep learning network (e.g., CNN) to indicate color representation intensity and drive color extraction. A clustering classifier is trained to use the feature vectors extracted by the network. Additionally, pixel clustering is useful in extracting colors. Cluster counts can vary based on the classification. In another approach, heuristics (with or without classification) are useful in extraction. The resulting clusters are evaluated against a (ordered) set of desired colors to determine a match. An instance of a standardized sample image can be defined from a template sample image and the corresponding extracted colors using image processing. The extracted colors can be presented in an augmented reality GUI, such as a virtual try-on application, and applied to user images such as selfies using image processing.
[0006] In practice, there are different styles or types of sample images for the same type of product, which results in the colors of different styles being presented in different ways. Therefore, in some sample image styles, it may be more challenging to identify the product color in a particular sample image. Different images (of the same product type) can be processed using clustering techniques applied to the feature vectors of the image. Feature vector clustering techniques can be used in a classifier to determine how strongly a particular sample image represents the product color. The classifier can assign (label) a particular sample image to one of 3 classes, such as strong, weak, and ignore. The labeled sample images can be processed separately using a color extraction operation including image pixel clustering techniques. Those labeled as ignore may be ignored. Those labeled as strong can be processed to identify a single product color. Images labeled as weak can be processed using image pixel clustering techniques to identify candidate product colors, such as three candidate colors. Manual selection can distinguish a single product color from the candidate colors.
[0007] An instance of a standardized sample image can be defined by using image processing from a template sample image and corresponding extracted colors. The extracted colors can be presented in an augmented reality graphical user interface (GUI), such as a virtual try-on application, and applied to user images such as selfies using image processing.
[0008] In one aspect, a method is provided, comprising: classifying a source color image showing a color of an item of interest to indicate an intensity of a representation of the color in the source color image, the classification using a deep learning network model to extract image features for determining the intensity; in response to the classification, extracting color information from the source color image by clustering pixels of the source color image according to their corresponding colors and identifying at least one candidate color as an extracted color in response to the clustering of the pixels; and providing the extracted color to define a new image using image processing.
[0009] Classification may identify the source color image as strongly representing color. Therefore, extracting color information responsively generates a candidate color from the primary colors identified by the clustering of pixels. Classification may identify the source color image as weakly representing color. Therefore, extracting color information responsively generates multiple candidate colors identified by the clustering of pixels.
[0010] Classification may use cluster-based classification in response to image feature vectors extracted by a deep learning network model.
[0011] Clustering the pixels may define a plurality of clusters k, and determining at least one candidate color from the plurality of clusters, wherein the number k varies in response to the classification.
[0012] The classification may perform cluster-based classification, which: measures an image feature vector of a source color image generated by a deep learning network model for a plurality of image feature clusters; and provides, as a classification of the source color image, a corresponding class associated with one of the plurality of image feature clusters that is closest to the image feature vector for the source color image, the corresponding class indicating the intensity of the color representation. The plurality of image feature clusters may be generated from training data that includes training images representing colors processed by a deep neural network model to generate image feature vectors for defining the plurality of image feature clusters. Thus, each training image is respectively assigned to one of the corresponding classes indicating the intensity of the color representation.
[0013] The network model may include a convolutional neural network (CNN) that is pre-trained to process images of objects to define feature vectors of the images of the objects.
[0014] The method may include obtaining a starting image for defining a new image, and defining the new image from the starting image using the extracted colors to be applied to an object displayed within the new image. The starting image may include a first selfie image, the new image includes the new selfie image including the object, and the object includes a product for simulation on the first selfie image. The product may include a cosmetic product for virtual try-on using augmented reality, and the source color image includes a sample image of the cosmetic product. The method may include presenting the new image in a graphical user interface (GUI), the graphical user interface (GUI) providing a cosmetic product for virtual try-on using an augmented reality experience, the GUI further configured to: receive a user's selfie image and a selection of the new image, and apply the extracted colors to a portion of the selfie image to virtually try on the cosmetic product represented by the new image.
[0015] In one aspect, a computing machine is provided, comprising a processor coupled to a storage device, the storage device storing instructions that, when executed by the processor, configure the computing machine to: obtain a source color image comprising pixels, the source color image having an area of color to be extracted; cluster the pixels into a plurality of clusters without regard to intensity, each of the plurality of clusters being characterized by a corresponding characteristic color; evaluate the plurality of clusters against a set of color expectations, the color expectations comprising respective expected colors to be present in the image, search the plurality of clusters for color matching using the expected colors to define an extracted color; and provide the extracted color using image processing to define a new image.
[0016] Clustering the pixels may include applying k-means clustering to the pixels without considering intensity.
[0017] The pixel may be defined in a first color space, and the instructions may configure the computing machine to convert the pixel to a second color space that is independent of intensity. The first color space may include a red, green, blue (RGB) color space or an equivalent. The second color space may include a luminance (L), green-red (a), blue-yellow (b) (LAB) color space or an equivalent.
[0018] Searching for a color match may include determining a similarity of a corresponding feature color of one of the plurality of clusters to a corresponding expected color of one of the color expected sets. When the corresponding feature color is similar to the corresponding expected color in brightness and direction, the search may locate a color match according to a corresponding threshold requirement for the corresponding expected color. The instructions may configure the computer to determine a color match by: (a) calculating a brightness difference between the brightness of the feature color and the brightness of the corresponding expected color, and comparing the brightness difference to a brightness difference threshold of a corresponding threshold requirement for the corresponding expected color; and (b) calculating a direction difference between a cosine angle of a direction of the feature color and a cosine angle of a direction of the corresponding expected color, and comparing the direction difference to a direction difference threshold of a corresponding threshold requirement for the corresponding expected color.
[0019] The computing machine may define the color desired set as an ordered set according to a priority order. The evaluating may include searching the color desired set using the plurality of clusters according to the priority order, stopping when a color match is located.
[0020] The instructions may configure the computing machine to: in response to the search, if each of the two or more clusters provides a color match, cluster two or more of the k clusters into a combined cluster; and define the extracted color using the combined cluster.
[0021] The instructions may configure a computing machine to obtain a starting image for defining a new image, and to define a new image from the starting image using extracted colors to be applied to an object displayed within the new image. The starting image may include a first selfie image, and the new image may include the new selfie image including the object. The object may include a product for simulation on the first selfie image. The computing machine may include a mobile device, and the product may include a makeup product for virtual try-on using augmented reality.
[0022] The source color image may include a sample image of a cosmetic product.
[0023] The new image may include a standardized sample image defined from the template image and the extracted colors. The instructions may configure the computing machine to present the new image in a GUI that provides makeup products for virtual try-on using an augmented reality experience. The GUI may be further configured to: receive a user's selfie image and a selection of the new image, and apply the extracted colors of the new image to a portion of the selfie image to virtually try on the makeup product represented by the new image.
[0024] For any of the method aspects, the computing machine and computer program product aspects will be apparent. Similarly, for any of the computing machine aspects, the method and computer program product aspects will be apparent. Other aspects will also be apparent. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 is a block diagram of a machine environment for color extraction and generating a color image using the extracted colors according to an example.
[0026] Figure 2 is a block diagram of a representative computing machine.
[0027] Figure 3A and 3B is a flow chart of operations for machine color extraction and generation of a color image using the extracted colors, according to an example, and also illustrates data structures.
[0028] Figure 4A showing a template smear image of a first color, and Figure 4B Shown from Figure 4A An example of a normalized sample image updated with a second color.
[0029] Figures 5A-5C , 6A-6C and 7A-7B show different sample images classified according to the intensity of the product color representation, where: Figure 5A , 5B and 5C is classified as showing a strong product color representation; Fig. 6A , 6B and 6C is classified as showing weak product color representation; and Fig. 7A and 7B Product color representation that is classified as showing negligible.
[0030] Figure 8 is a block diagram of components of a class-driven color extractor (eg, in a memory or other storage device) of a computing machine.
[0031] Fig. 9 is a flow chart of the operations used for class driven color extraction.
[0032] The concepts of the present invention are best described by certain embodiments thereof, which are described herein with reference to the accompanying drawings, wherein like reference numerals refer to like features throughout. It should be understood that when the term invention is used herein, it is intended to imply the inventive concepts underlying the embodiments described below and not just the embodiments themselves. It should be further understood that the overall inventive concepts are not limited to the illustrative embodiments described below, and the following description should be read in light of such a view. More than one inventive concept may be shown and described, and unless otherwise stated, each inventive concept may be independent or combined with one or more other inventive concepts. DETAILED DESCRIPTION
[0033] In a machine environment, a digital image includes data in the form of pixels that are usually associated with physical points in a raster image. The intensity of each pixel is variable, and in a color image, the color is usually represented by the intensity of three or four components, depending on the color model (e.g., data scheme). Pixel data is usually defined using the RGB (red, green, blue) color model. RGB is an additive color model. RGB is usually used for images from RGB input devices and RGB output devices. RGB input devices include cameras, image scanners, and digital cameras. RGB output devices include monitors of various display screen devices (e.g., television screens, and monitors including computer monitors and mobile phone displays of various technologies (e.g., CRT, LCD, plasma, OLED), video projectors, etc.). Other color models include CMYK (cyan, magenta, yellow, key (black)), which is a subtractive color model and is more often used in color printing, such as depositing ink on a substrate such as paper. Therefore, in a machine environment, images that are usually acquired from most sources for processing are at least initially defined according to the RGB color model.
[0034] The RGB color model has alternative color representations, such as the HSL (hue, saturation, brightness) and HSV (hue, saturation, and value) color spaces. Other color spaces include the CIELAB (International Commission on Illumination (CIE) Lightness (L), Green-Red (A), and Blue-Yellow (B)) color space. CIELAB is also known as CIE L*a*b* and is sometimes abbreviated to LAB or Lab.
[0035] In a variety of industries, including the beauty industry, swatch images are ubiquitous in online stores and are often the only visual indication a customer has of what a product will look like. Machine-based color extraction, where a program extracts color from an arbitrary swatch to identify a color, is a difficult task.
[0036] In some applications, it is useful to programmatically extract color from a sample image. Using the extracted colors to machine construct a new image is one such example. For example, augmented reality uses constructed images to give a virtual view of one or more objects.
[0037] With the development of augmented reality technology, virtual try-on has become very popular in online beauty stores. Rendering methods for virtual makeup require some information about the target product, the most important of which is its color. Currently, this problem is solved by manually creating a large database of products and their corresponding colors, and adjusting these color values until the virtual rendering result is similar enough to the real result. This process has several drawbacks. First, manually adjusting the color values is time-consuming and subjective. Second, the process may require extensive coordination between the developers of the virtual try-on software and the cosmetics brands, which may not be feasible if the virtual try-on software is to be embedded in an online store selling products from third-party retailers. Instead, shown in this paper is a solution that utilizes already existing sample images of each product to programmatically extract colors directly from them in an unbiased and objective manner.
[0038] Another common application of sample color extraction is in data collection. The ability to programmatically extract makeup colors from online sample images or from images of makeup applied to people in the real world can have many benefits. The collected data can be used for fashion trend analysis, advertising popular color shades, product recommendations based on a user-selected "look" in an image, metadata for more accurate online searches, and more.
[0039] Other uses in automatic color extraction can include using the extracted colors as a proxy for locating and tracking an object (or multiple objects) in a series of frames. This can be useful in situations where large amounts of data to train neural networks for object tracking are not available. However, the more complex the object image (e.g., not a simple sample), the less likely this approach will be successful, since it assumes that there are only a few dominant colors for such an object, as can be seen in the method described below.
[0040] challenge
[0041] Despite the many potential benefits, extracting cosmetic colors from samples remains an unsolved problem for a number of reasons. (Note that the problem is not limited to cosmetics, but can be applied to other products, such as paint colors for homes or cars, fabric colors for decoration, etc. Colors extracted from paint or fabric samples can be used when constructing a 2D or 3D room, where the surface is colored using the extracted color or a color derived from it. The colors thus derived can be complementary colors.
[0042] In recent years, the most popular approach to trying to solve difficult computer vision tasks is to use deep learning. However, a major pitfall of deep learning is the need for large amounts of annotated data. Constructing such datasets is resource intensive. In the makeup context, no public dataset exists for automatically extracting colors from makeup sample images. Furthermore, annotated datasets tend to be static and outdated unless regularly updated, such as with new colors.
[0043] Another significant challenge for machine learning approaches, and a challenge for any approach, is the extreme variance in the distribution of sample data. For example, the distribution of sample images in lipstick alone includes images of the lipstick applied on a piece of paper, images of it applied on a patch of skin, images of it applied on the lips, images of it fitted with the cap off, and invalid images covered in its box, to name a few. Each lipstick brand has its own style of sample images, which means that the variance increases with the number of brands sold in an online store, and styles also change over time. This huge variance makes it very difficult to develop a single solution that works reliably in most situations.
[0044] Not only do the styles of images vary widely, but also the distribution of colors within images varies widely. Samples rarely consist of pure colors on a flat background. Instead, they usually consist of multiple colors distributed across the image, many of which can be possible colors (e.g., possible lipstick colors in a lipstick makeup image). Therefore, a successful machine color extraction method must not only identify the primary color palette in the image, but also have a robust method to select the correct color among the colors in the image. For example, imagine having a sample of lipstick that consists of an image of a lipstick case with the cap removed and some words (text) on the background. Finding the color palette in this sample may yield four colors: the color of the lipstick, the color of the case, the color of the text, and the color of the background. These four primary colors in the image can define the color palette of the image. After extracting the four color palettes, the problem of finding the color of the lipstick is not solved because it is still unknown which color corresponds to the actual lipstick. The goal of solving the second problem is to have a robust method to select the correct color among the palette colors, where the word "correct" refers to the color of the object of interest (e.g., the lipstick, not the case).
[0045] Finally, samples are almost always real-world images, which means that the lighting conditions in which they were taken are never completely controlled. As a result, the colors in the image vary greatly in intensity between well-lit areas and areas in shadow. This makes it more difficult to specify the correct color intensity and locate that color in the first place, since there is no single RGB value to identify the makeup in the image.
[0046] Figure 11 is a block diagram of a machine environment for color extraction and generating a color image using the extracted colors according to an example. A mobile device 102 (an example of a computing machine) is shown in association with a user 104. The mobile device 102 (reference Figure 2 Further described) can communicate with one or more (e.g., N) online stores (store 1 (108A)...store N (108B)), for example, via a browser or a local application using Web technology. Each store (store 1 (108A)...store N (108B)) is shown as a computing machine, such as one or more servers associated with each online store data storage (e.g., databases 110A and 110B, respectively) that stores Web data and sample images. It should be understood that these e-commerce environments are simplified. The N stores can receive products and other data including sample data from multiple (e.g., M) product sources. In this example, the product sources are cosmetics source 1 (112A)...cosmetics source M (112B) represented as computing machines such as one or more servers. Each product source (cosmetics source 1 (112A)...cosmetics source M (112B)) is associated with a corresponding data storage (e.g., databases 114A and 114B, respectively) that stores data such as product-related data (descriptions, etc.) and sample images. The M product sources may generate their respective product data, including sample data such as from respective cameras 116A, 116B or other image generating devices. Typically, as described, images are stored in RGB color space. It should be understood that these product sources are also simplified.
[0047] Each online store and product source may be associated with a legal entity operating a corresponding business. The corresponding computing machine may be operated by such an entity or by other persons on their respective devices (e.g., service contacts). The N online stores typically sell other people's products, but may also sell their own products or white-label products. The corresponding M product sources share (communicate) corresponding product data with the corresponding online stores that sell their products. Each of the N online stores can provide different product data and may have some of the same product data as another online store. Online stores can also generate their own product data, including sample images (not shown), or obtain the same data from others (third parties).
[0048] The user device 102 includes a display screen 118, such as a GUI for presenting an application. The application can be configured to virtually try on makeup. The GUI enables the capture and presentation of a selfie image 120 and the selection of a color (e.g., represented by an icon, GUI control type) 122A, 122B, 122C, and 122D of makeup (e.g., lipstick) to be virtually applied to the selfie image 120. In some examples, the selfie image is a real-time image such as from a video to allow the user to turn their head (or move the camera or both actions) to view the appearance from different angles and / or different lights. The corresponding colors associated with icons 122A, 122B, 122C, and 122D can be determined from sample images from corresponding online stores or other sources.
[0049] In this example, the display screen 118 is a gesture-based (e.g., touchscreen) enabled I / O device, and the corresponding color to be tried can be selected by touching / swiping icons representing the colors. Voice control or other interfaces can be used (in conjunction with or in lieu of use). Other cosmetics, nail care, hair, or other products can be virtually tried.
[0050] Figure 2 1 is a block diagram of a user device 102. The user device 102 includes one or more processors 202 (e.g., one or more CPUs, GPUs, ASICs, FPGAs, etc.), input devices 204 including cameras, one or more communication units 206 (e.g., for wired or wireless communication), one or more output devices 208 (e.g., speakers, lights, etc.), and gesture-based I / O devices such as a touch screen (display screen 118) to receive input and visually present output. The user device 102 also includes one or more storage devices 210 (e.g., memory, disk, etc.) for storing data, including a browser 212 for web-based communication, a makeup simulation application 214 for virtual makeup try-on, and lower-level functions such as a communication module 216 for facilitating wired and / or wireless communication, and an operating system 218. User data, such as color and image data, is also stored, such as for self-portrait images, sample images, web data, colors extracted from samples, etc.
[0051] Although shown as a stand-alone or local application, the makeup simulation application 214 can be configured in other ways, such as a web-based application. For example, running in a browser environment of a user's computing device. The makeup simulation application 214 is configured to receive a sample image associated with a corresponding product in a virtual try-on application and extract a color for simulating the corresponding product on the selfie image. Although a selfie is typical, other images can also be used. The makeup simulation application 214 receives a first image (e.g., a selfie), determines the product and color to be applied thereto, and generates (e.g., using image processing techniques) a second image using the first image, the product, and the color to virtually try on the product.
[0052] Machine color extraction method
[0053] You can refer to Figure 3A and 3B , wherein, according to one aspect, method operation 300 extracts makeup colors from a sample image without using any training data or machine learning. The method was developed and tested specifically for lipstick images, although it can be extended or modified to apply to other types of makeup and other types of products. At a high level, the method works by finding intensity-independent color clusters and then selecting the color cluster that is most likely to correspond to a product (e.g., lipstick) based on a priority heuristic designed from prior expectations.
[0054] The operation initially obtains a pixel (eg, P i,j ), and converting the image pixels from RGB space to an intensity-independent color space (at 304) to define image I'. If the image is not defined in RGB and is already defined in an intensity-independent color space, the conversion may not be performed.
[0055] The candidate sample images have a small number of colors. This is because the operation here specifies the number of expected colors, which corresponds to the number of clusters specified when performing k-means clustering. The color of the product of interest must actually exist in the image (i.e., it cannot be occluded). It will be easy to understand that to extract a color, it must first be in the image. Given these two constraints, the types of samples processed by the operation are very diverse: for example, the types of samples may include smudges on a piece of paper, products applied to the face (although it is more likely to work when cropped to the lips (e.g., the area where the product is applied), reducing the number of colors), images that expose product containers (e.g., tubes), etc. An example where the operation may not be successful without preprocessing may include a full-body picture of a person with all of their clothes, hair, and background shown, as this would only add too many colors to the source image to be able to confidently identify the lip color. In this case, it may be indicated to preprocess the image to limit the image to the portion where the color to be extracted is located and minimize other colors. Those skilled in the art will appreciate that preprocessing operations can be used to locate objects of interest, such as lips on a face.
[0056] Therefore, some assumptions about the source images can be considered to define the operation 300. In this example, no preprocessing is performed, and it is assumed that the data source will satisfy the constraints in the vast majority of cases (e.g., based on the fact that the source images input to the operation correspond to lipstick product pictures from an e-commerce website, so they are (almost) all simple samples).
[0057] Since the sample images are taken under real lighting conditions, individual objects in the images may vary greatly in color intensity. For this reason, trying to cluster colors directly in RGB space often results in multiple incorrect clusters being separated for the same object color. For example, in a simple sample image of red lipstick applied on a piece of paper, clustering in RGB space may produce an incorrect dark red cluster from the shadowed parts, and another incorrect bright red cluster from the parts with incident light and specular reflections. This will make it difficult to determine which cluster is closer to the true lipstick color, while the average between them is usually closer to the true color. To address this issue and find clusters in a more lighting-independent way, images in RGB color space are converted to LAB color space, and the L channel encoding intensity (brightness) is ignored.
[0058] While LAB defines its color space with a specific intensity (brightness) channel, operations do not require the use of LAB. Another option is to simply split RGB into two components: 1) normalize the color ratios to a constant sum, which would be the intensity-independent part; 2) sum of all components, which would be the intensity part. So, in short, LAB itself is not necessary, and another color space can be used that allows examination without regard to intensity (e.g., color components can be examined separately from intensity).
[0059] The operation at 306 clusters the colors using k-means clustering with random center initialization (e.g., using standard techniques for such k-means clustering). K-means clustering is a vector quantization method that seeks to partition n observations into k clusters, where each pixel is assigned to the corresponding cluster with the closest mean. In the current example, when the image is represented in the LAB color space, the i×j=n pixels P of the image may be partitioned using only the a and b color data values without considering the L intensity / brightness values. i,j Each one of them.
[0060] Once each pixel is assigned to one of the k clusters according to the clustering technique (e.g., after convergence), the center (mean) and weight of each cluster are calculated, where the weight is the proportion of image pixels in the cluster (e.g., pixel count / n in the cluster) (step 308). The mean is used as the cluster center (e.g., the prototype or characteristic pixel of the corresponding pixel cluster) that provides the characteristic color (CC) for the cluster. By clustering the colors, the main color components of the image can be located in an unsupervised manner, and it is almost always the case that one of these components corresponds to the correct lipstick color. K-means clustering is an iterative process that utilizes starting centers and then gradually improves them. First, the centers are initialized by taking k random pixels. Other clustering methods can be used as alternatives. According to the literature, k-means is generally the most stable and is relatively fast. One disadvantage is that it requires specifying the number of desired clusters, which is one of the reasons for operating on 300 target sample images, which typically have very few colors. In this example, k=3.
[0061] Additionally, the operation at 309 removes typical background image colors from consideration based on the fact that the background color is unlikely to be a true lipstick color. The operation checks the k colors, searching for a "white" cluster close to RGB = (255, 255, 255), and if it is found, removes it as a candidate color for the product. This approach is based on the fact that a background is usually present in a typical sample image, and is almost certainly white. White is an atypical lipstick color. Therefore, in almost all cases, even if the operation extracts k = 3 clusters, one color is removed to leave two clusters after filtering out the white background. However, this strategy (the number of clusters and what to filter out) was designed with lipstick samples in mind, and can be adjusted or configured for other types of product samples (e.g., paint) and their typical characteristics. The class-driven color extraction method that does not use heuristics is further described below.
[0062] The final general step is to predict which cluster center corresponds to the correct color. This is generally an ill-posed problem; however, leveraging prior expectations about the color of a product (e.g., lipstick) is a very effective heuristic. Lipsticks are not produced in random, uniformly distributed colors. Instead, there are different patterns where certain colors are more or less common. To this end, at step 310, a prioritized list (e.g., set) of 1...q color expectations (e.g., which may be referred to as "expected colors" (EC)) is defined. At 312, the list / set is evaluated (e.g., in-order traversal), and stops if the EC is matched by one or more cluster centers described further. The following description provides detailed information on the definition of a color expectation and the conditions for a cluster center to satisfy that expectation.
[0063] Color expectations are simply the range of RGB values. Color can be broken down into a combination of two components: 1) brightness / intensity and 2) the relative magnitude of the individual channels. Brightness is described by the length of the RGB vector, while the relative magnitude is described by the direction of the vector. The qualitative interpretation of brightness is self-explanatory, while the direction of the vector can be thought of as encoding the color. For example, blue has an orientation of [0,0,1] in RGB space, while red has a perpendicular orientation of [1,0,0], and both colors can have varying degrees of brightness.
[0064] For a color cluster center to satisfy a given color expectation, it satisfies both the brightness similarity requirement and the orientation similarity requirement. The brightness difference is simply calculated as the magnitude difference between the cluster center vector and the color expectation vector, while the orientation difference is calculated as the cosine angle difference between the two vectors. Each color expectation has different threshold requirements for these two components, such as the brightness difference threshold (B) and the orientation difference threshold (D). For example, red lipsticks have a wide range of hues, while purple lipsticks tend to be closer to a specific hue, so the red expectation has more relaxed brightness and orientation thresholds.
[0065] In this operation 300, color expectations are evaluated one by one in turn, and once one or more cluster centers from the image meet the threshold requirements for color expectations, the operation ends. The color expectation list is static and is sorted in descending order of popularity in the lipstick market to minimize errors. For example, the color expectation "red" is detected before the color expectation "green" because red lipstick is more likely to have some green elsewhere in the image (because green lipstick is very rare to begin with).
[0066] When a single cluster center matches (eg, meets a threshold requirement for color desirability), that color is used as the extracted color to represent the sample color and can be used as further described.
[0067] It is also worth noting that in some cases, for a given color expectation, multiple color cluster centers may meet the same threshold requirement. In this case, the cluster centers are combined (e.g., step 314) as a weighted average based on their relative pixel volume within the image (these weight values are also calculated in step 308). Thus, each cluster center is compared to the color expectation being evaluated. More than one cluster center may match (as described above), and these more than one cluster centers can be combined as a weighted average based on their relative pixel volume to define the extracted color.
[0068] Finally, in the case where no color cluster center meets the threshold requirements for any color desirability and the list has been exhausted, the cluster center closest to "red" is assumed to be the lipstick color, although the confidence of the prediction must be low (not shown).
[0069] In this example, a GUI may be provided via a mobile device (or other computing machine) to present a virtual try-on or simulation of makeup on a selfie or other image using augmented reality methods (step 316).
[0070] result
[0071] The example method achieved high accuracy on a test set of 160 sample images of online retailer product pages from selected online stores. The test set consisted of 100 "easy" images (which were randomly selected and represented the true distribution of online store samples) and 60 "difficult" images (which were specifically sought for their unusual lipstick colors).
[0072] The ground truth RGB values of the image were created by manually annotating a bounding box over the lipstick region and averaging the pixel values inside. The accuracy of the predicted color vectors was evaluated separately for direction and intensity. For color vector direction, a successful extraction was defined as a cosine similarity of the predicted color to the ground truth vector within 8 degrees. For color vector intensity, a successful extraction was defined as a difference of less than 20 between the predicted color and the ground truth color in grayscale color space at 255 scale. Table 1 shows the various accuracy measures:
[0073]
[0074] On the "easy" test set, which represents the true distribution of online store samples, the accuracy is extremely high. On the "hard" test set, which represents extreme cases and unusual lipstick colors, the accuracy drops, but remains high. The true color (direction and intensity) is correctly detected in almost all cases. In fact, most errors come from images where the correct color cannot be extracted, such as when the lipstick is in a box, the lipstick has a cap, or there are multiple lipsticks of different colors in the image.
[0075] Diversified sample images - intensity of (product) color representation
[0076] The sample images may show items of interest (e.g., products) in various ways or styles. The intensity with which each corresponding image represents the color of the product may help improve color extraction. Certain styles may be more challenging than others in terms of identifying products and extracting colors.
[0077] Prior to performing the color extraction operation, a network may be trained to process each sample image to classify the intensity of the color representation. For example, the sample images may be initially classified (e.g., using a deep learning classifier). In response to the classification, a color extraction operation is performed to extract color information from the sample image. The color operation uses an image pixel clustering technique (which may be a k-means or similar clustering technique) to extract color information, but does not use a heuristic.
[0078] The deep learning classifier can be configured to use a (separate) clustering technique that clusters image features represented by image feature vectors. These feature vectors can be generated by applying deep learning techniques, such as a convolutional neural network (CNN) trained to recognize image features. The classifier is further described below.
[0079] Color extraction using heuristics may not work well for all product types: for example, restricting or biasing color information to a set of possible candidate colors for a product while applying a heuristic may not work for all product types, especially those with a broad and uniform color distribution. Compared to many other types of products (which may be in any color of the spectrum and are more evenly distributed), the colors of lipsticks are usually restricted to a small palette range, dominated by red. A more scalable treatment without heuristics supports more categories (product types) than heuristic-based methods. In the makeup domain, other categories may include "lip liner", "foundation", "concealer", "blush", "highlighter", "contour", "eyeshadow", "eyeliner", "mascara", "eyebrow pencil". In principle, the method can be extended to support different types of products, while HE only specifically supports lipsticks because the heuristics need to be hand-crafted.
[0080] A single classifier can be defined (e.g., a corresponding deep learning neural network can be trained) for all types of items (e.g., all product types). Any corresponding product can be represented by different sample image types, and the classifier can be trained using the different images to label (classify) the image into one of multiple classes. The classes can be defined in response to the strength of the sample image's representation of the product's color. A single classifier may be more convenient rather than defining and / or using product-specific (separate) classifiers. In this way, a single classifier and color extraction tool or function can be defined and / or used across different products.
[0081] The method does not assume product-specific color heuristics (i.e., it does not rank colors that appear most likely to be lipstick colors).
[0082] Features in the corresponding images are identified and clustered using deep learning methods. The corresponding clusters can be ranked in response to the representation of the product color. Therefore, the classifier is trained to rank or label each sample image in response to the identified and clustered feature vectors. In the example here, the classifier is trained to classify the sample images as "strong", "weak" or "ignore".
[0083] Classifier definition and training
[0084] The classifier for classifying different sample image types includes a deep learning model (e.g., a neural network such as a CNN) configured and trained for performance objectives. It is often convenient to start defining a new network using an existing CNN-based model that is previously defined and pre-trained to process images, many of which are publicly available. An example is ResNet-152, a deep residual learning network (i.e., Res network) for image recognition, which can have a depth of up to 152 layers. ResNet-152 is described by K. He, X. Zhang, S. Ren, and J. Sun in "Deep residual learning for image recognition". ABS / 1512.03385, 2015, which is incorporated herein by reference and can be obtained from arXIV:1512.03385 of Cornell University. ResNet is trained on publicly available image datasets, such as ImageNet provided by Image-Net.org operated by Stanford University and Princeton University.
[0085] In order to further refine the network model for the proposed goal of classifying sample images based on how strongly the image represents the color of the product, the starting network model can be used to generate feature information (feature vectors, including data generated after the neural network analyzes the image). Each vector represents a feature of the corresponding image, similar to a fingerprint. During training, a dataset of random product sample samples (images) is collected. In order to train (define) a single classifier, the training data can include samples from multiple product types to ensure the breadth or diversity of the data. ResNet (i.e., the starting model) is applied to the dataset of random product sample images to generate corresponding feature vectors based on the deep learning applied by the network model. That is, the images are run through a pre-trained ResNet-152 model to obtain a feature vector for each image.
[0086] Extracting a feature vector for each training image, akin to a fingerprint, can make it easier to define the final classifier using clustering techniques than through other techniques, such as configuring and training a ResNet as an end-to-end classifier for this goal.
[0087] The resulting feature vectors are clustered (e.g., using a k-means clustering technique) into a certain number (n) of clusters. In this example, n=50. It is important to repeat that this clustering process is different from the clustering process described for color extraction. The clustering at this classifier stage is performed on image feature vectors. The clustering at the color extraction stage is performed on image pixels.
[0088] Perform manual inspection and classification. An operator manually inspects each cluster and classifies each cluster into three classes: strong, weak, and ignore, which help drive the color extraction operation.
[0089] Figures 5A-5C , 6A-6C and 7A-7B show different sample images classified according to the intensity of the product color representation, where: Figure 5A , 5B and 5C is classified as showing a strong product color representation; Fig. 6A , 6B and 6C is classified as showing weak product color representation; and Fig. 7A and 7B Product color representation that is classified as showing negligible.
[0090] Strong clusters usually have images that feature a preview of the product’s color. This image usually has only a white background and shows the product’s color in an easily extractable form. Figure 5A , 5B and examples 500 , 502 , and 504 in 5C , which may depict product packaging, but with the color preview prominent and not obscured.
[0091] according to Fig. 6A , 6B and examples 600, 602, and 604 in 6C, the weak cluster typically has an image that still shows the product color, but it can contain more packaging that can obscure part of the color preview.
[0092] The ignored clusters typically include anything that cannot be easily used for color extraction purposes. Fig. 7A and 7B In examples 700 and 702, some examples include product packaging that does not display color, display of legal disclaimers or text, and images showing models wearing the product.
[0093] After clustering and sorting the feature vectors, corresponding cluster centers are defined for each cluster, along with their corresponding categories (strong, weak, ignore (or similar classifications, the specific term / word or code itself is not important)). The cluster center is the mean of the cluster of grouped observations (in this example, the observations are feature vectors), and it serves as the prototype observation for the cluster. Future observations (e.g., feature vectors from sample images analyzed at inference time) are measured against the mean, the nearest mean is determined (e.g., based on Euclidean distance), and are thereby associated with the corresponding cluster.
[0094] Figure 8A block diagram of a class-driven color extractor 800 for sample images of the same product type according to an embodiment is shown. The class-driven color extractor 800 includes a deep learning classifier 802 with a deep learning feature vector generator 804 and a cluster-based classification component 806 using feature vectors. The class-driven color extractor 800 also includes a cluster-based color extractor component 808 using image pixels. It should be understood that although the components are shown separately, there may be no bright line distinction, and some components may be configured to perform multiple functions. The class-driven color extractor 800 itself can be divided into more different components, such as a classifier and a cluster-based color extractor using image pixels. The cluster-based color extractor component 808 can operate or be used in response to the classification generated by the deep learning classifier 802. That is, the classification can be provided to the cluster-based color extractor component 808 and color information can be returned in response to the classification. Alternatively, the cluster-based color extractor component 808 can only provide an image (source color image) and return candidate extracted color information, which can be sorted (e.g., by strongest match), and other operations determine the extracted color in response to the class. For ignored classes generated for the image, the extracted color cannot be extracted. It is expected that for most embodiments in commercial or practical environments, the extracted color is the product color.
[0095] The cluster-based color extractor component 808 can be similar to the reference Figure 3A and 3B The described operation. Figure 3A and Figure 3B In the example of , a heuristic is used and the ordered set of colors is evaluated at step 310 and subsequently. Figure 8 In the current example of , there is no need to perform an operation to compare the clustered colors to the ordered set. The cluster-based color extractor component 808 can be configured to use a small number of clusters for colors, using the understanding that images with strong representation colors will have the product color as their primary color. When defining the number of color clusters in the class-driven color extraction example, in k-means clustering, for images classified as strong, k can be equal to 2, and when classified as weak, k can be equal to 5, however, other values can be used.
[0096] The classifier can use the pixel count of each cluster as a measure of the dominant color. In weak class images, the product color may be a secondary or even tertiary color. The cluster-based color extractor component 808 can generate and provide more than one candidate color, particularly for weakly classified images.
[0097] That's right Figure 3A and 3BWith respect to the modification of operation 300, color extraction driven by classification may be performed similarly, but may be substituted into operations 310-312. Fig. 9 Operations of class driven color extraction for a corresponding source color image (eg, a sample image) are shown 900. Operations when multiple images are available for the same project and color are further described below.
[0098] At 902, the image is classified using a deep learning classifier (e.g., 802). A strong, weak, or ignored class is obtained. As previously described, other labels may be used. At 904, a decision is made based on the class (e.g., an operation branch). Via the strong branch to 904, the execution from Figure 3A Operations 302-308 and optional 309 are used to process the image to cluster pixels, preferably without regard to intensity. The number of clusters (k) may be 2 to help select the dominant color. In step 906, the dominant color is extracted using the pixel count and returned.
[0099] Via the weak branch from 904, at step 910, execute Figure 3A Operations 302-308 and optional 309 are performed to process the image to cluster pixels, preferably without regard to intensity. The number of clusters (k) may be 5 to help select the dominant colors. At 912, at least 3 dominant colors extracted using pixel counts are returned. Via the ignore branch to 914, these operations do not return a color.
[0100] although Fig. 9 900. If multiple strong images are processed and classified, the dominant color of each strong image is averaged and returned. Weak images from the same branch can be ignored.
[0101] If a strong image is not classified and a weak image is classified, multiple primary colors (e.g., a set of candidate colors) are returned. Since multiple candidate colors can be generated per image, similar colors from multiple images can be used, for example to sort colors based on commonality or reduce color counts. Manual processing can be used to parse the extracted colors in the weak case (e.g., present multiple colors and receive a selection), or other methods can be utilized.
[0102] As described above, the processed source color image may be an image in which the item of interest is a product and its color is a product color. Thus, the extracted color or multiple candidate colors are product colors. The extracted color may be used, for example, using image processing techniques to define a new image containing that color. The product may be a cosmetic product, and the new image may be a simulation of the product being used.
[0103] Makeup Simulation-Augmented Reality
[0104] As described above, the makeup simulation application 214 receives a first image, such as a selfie, and determines the product and its color to be virtually tried out from the sample image. The color data may be extracted and provided to the makeup simulation application, such as from a computing machine in communication with the user device 102, or the user device 102 may be configured to determine the color data. The product sample may be provided to the class-driven color extractor (e.g., 800), and the returned color or product sample may be analyzed by the component using a clustering-based approach and heuristics (e.g., in Figure 3A and 3B ) to obtain the color.
[0105] The makeup simulation application 214 generates (e.g., using image processing techniques) a second image using the first image, the product, and the color to virtually try on the product. The second image is presented, for example, via the display screen 118. In another example, the user device can be configured to transmit the image along with the product and color selection to a remote computing machine, and receive a second image (showing a virtual application of the product) in return. An example of a makeup simulation application for a user device such as a tablet, smartphone (mobile device), etc. is ModiFace Live Streaming, available from ModiFace Inc. TM .
[0106] Defines a standardized color sample image
[0107] In another use example, it may be desirable to present different product colors in a standardized manner, for example, presenting candidate lipstick colors as smears, where each smear is represented as a template image of a lipstick smear having the same texture, brightness, and other characteristics. Different instances of the lipstick smear image can be generated from the template image using corresponding colors determined by extracting sample colors from the respective sample images. In this way, different samples from different production lines and / or different product producers, etc. (as examples) can be presented in a standard or consistent manner. In one embodiment, the operation of the computing device can perform sample color extraction, such as reference Figure 3A and 3B As described, to determine the extracted color. In one embodiment, Figure 8 The widget can be used based on class driven color extraction.
[0108] A template image may be obtained. Image processing may be performed to define an instance of a standardized sample using the template image updated with the extracted colors (eg, replacing any colors in the template where applicable). Figure 4A A template smear image 400 of a first color (eg, red) is shown, and Figure 4BAn example of a standardized sample image 402 updated from an image 400 of a second color (blue) is shown. Images 400 and 402 are representative and are described herein in grayscale. They are not actually shown in color, but the effect will be understood by those skilled in the art. Example 402 may be displayed on a computer such as Figure 1 122A or used in another manner. Instance 402 may be stored (e.g., to data storage 3.g. 110A, 110B, or another (not shown)) or generated "dynamically" for the GUI. Operations for generating instances may be performed by or on behalf of device 102.
[0109] In addition to computing device aspects, one of ordinary skill will appreciate that computer program product aspects are disclosed wherein instructions are stored in a non-transitory storage device (e.g., memory, CD-ROM, DVD-ROM, optical disk, etc.) to configure a computing device to perform any method aspects stored herein.
[0110] Actual implementations may include any or all of the features described herein. These and other aspects, features, and various combinations may be expressed as methods, devices, systems, components, program products, and other combinations of features described herein for performing functions. Many embodiments have been described. However, it will be understood that various modifications may be made without departing from the spirit and scope of the processes and techniques described herein. In addition, other steps may be provided from the described processes, or steps may be eliminated, and other components may be added to the described systems, or removed from the described systems. Therefore, other embodiments are within the scope of the appended claims.
[0111] With respect to the embodiments of the present invention described above, they may be implemented at least in part using a software-controlled programmable processing device, such as a general-purpose processor or a special-purpose processor, a digital signal processor, a microprocessor or other processing device, a data processing apparatus or a computer system, and it will be understood that a computer program for configuring a programmable device, an apparatus or a system to implement the aforementioned method, apparatus and system is contemplated as an aspect of the present invention. The computer program may be embodied as any suitable type of code, such as source code, object code, compiled code, interpreted code, executable code, static code, dynamic code, etc. Instructions may be implemented using any suitable high-level, low-level, object-oriented, visual, compiled and / or interpreted programming language, such as C, C++, Java, BASIC, Perl, Matlab, Pascal, visual BASIC, Java, ActiveX, assembly language, machine code, etc. The skilled person will readily understand that the term "computer" in its most general sense includes programmable devices such as those mentioned above, as well as data processing apparatus and computer systems in any format in which they may appear, such as desktop personal computers, laptop personal computers, tablet computers, smart phones or other computing devices.
[0112] Suitably, the computer program is stored in a machine-readable form on a carrier medium, for example, the carrier medium may include a memory, a removable or non-removable medium, an erasable or non-erasable medium, a writable or rewritable medium, a digital or analog medium, a hard disk, a floppy disk, a compact disk read-only memory (CD-ROM), a recordable compact disk (CD-R), a rewritable compact disk (CD-RW), an optical disk, a magnetic medium, a magneto-optical medium, a removable memory card or disk, various types of digital versatile disk (DVD) user identification modules, a magnetic tape, a cassette solid-state memory. The computer program may be provided from a remote source contained in a communication medium, such as an electronic signal, a radio frequency carrier wave or an optical carrier wave. Such a carrier medium is also contemplated as an aspect of the present invention.
[0113] Throughout the description and claims of this specification, the words "comprises" and "comprising" and their variants mean "including but not limited to", and are not intended to (and do not) exclude other parts, wholes or steps. Throughout the specification, the singular encompasses the plural, unless the context otherwise requires. In particular, where indefinite articles are used, this specification should be understood to contemplate both the plural and the singular, unless the context otherwise requires. For example, the term "and / or" associated with "A and / or B" herein means A, B, and either A or B.
[0114] Features, integer features, compounds, chemical moieties or groups described in conjunction with a particular aspect, embodiment or example of the invention are to be understood to be applicable to any other aspect, embodiment or example unless incompatible therewith. All features disclosed herein (including any accompanying claims, abstract and drawings) and / or all steps of any method or process so disclosed may be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. The invention is not limited to the details of any foregoing examples or embodiments. The invention extends to any novel one or any novel combination of the features disclosed in this specification (including any accompanying claims, abstract and drawings), or to any novel one or any novel combination of the steps of any method or process disclosed.
[0115] As used herein, the terms "comprises," "includes," "including," "has," "having," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, "or" refers to an inclusive or and not to an exclusive or.
[0116] In addition, "a" or "an" is used to describe elements and components of the present invention. This is done only for convenience and to give a general sense of the present invention. This description should be understood to include one or at least one, and the singular also includes the plural, unless it is obvious that there is another meaning.
[0117] In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.
[0118] The scope of the present disclosure includes any new feature or combination of features disclosed therein or any generalization thereof, unless incompatible therewith, whether or not it is relevant to the claimed invention or mitigates any or all of the problems solved by the invention. The applicants hereby give notice that new claims may be formulated to such features during the prosecution of the present application or of any such further application derived therefrom. In particular, with reference to the appended claims, features from dependent claims may be combined with features of the independent claims, and features from respective independent claims may be combined in any appropriate manner and not merely in the specific combinations recited in the claims.
Claims
1. A method comprising: Through the computing device's processor: obtaining a source color image comprising pixels, the source color image having a region of color to be extracted; clustering the pixels into a plurality of clusters without regard to intensity, each cluster in the plurality of clusters being characterized by a corresponding characteristic color; evaluating each of the characteristic colors of a corresponding plurality of clusters against a set of color expectations, the color expectations including a corresponding expected color to be present in the image, searching the plurality of clusters for a color match using the expected color to define the extracted color, the expected colors being a predefined list of expected colors for a type of cosmetic product present in the source color image; and Image processing is used to provide the extracted colors to define a new image that is different from the source color image and is a simulation of the use of the type of makeup on a user's face based on the extracted colors.
2. The method according to claim 1, wherein: The pixels are defined in a first color space, and wherein the method comprises converting the pixels to a second color space that is intensity independent.
3. The method of claim 1 , comprising obtaining a starting image for defining the new image, and defining the new image from the starting image using the extracted colors to apply to an object displayed within the new image, and wherein: The starting image comprises a first selfie image, and the new image comprises a new selfie image containing the object, the object comprising a product for simulation on the first selfie image, and the product comprising a makeup product for virtual try-on using augmented reality.
4. The method according to claim 1, wherein: The source color image includes a sample image of a cosmetic product.
5. The method according to claim 1 includes presenting the new image in a graphical user interface (GUI), the GUI providing a cosmetic product for virtual try-on using an augmented reality experience, the GUI further configured to: receive a user's selfie image and a selection of the new image, and apply the extracted color of the new image to a portion of the selfie image to virtually try on the cosmetic product represented by the new image.
6. A method comprising: Through the computing device's processor: obtaining a source color image comprising pixels, the source color image having a region of color to be extracted; clustering the pixels into a plurality of clusters without regard to intensity, each cluster in the plurality of clusters being characterized by a corresponding characteristic color; evaluating the plurality of clusters against a set of color expectations, the color expectations including corresponding expected colors to be present in the image, searching the plurality of clusters for a color match using the expected colors to define an extracted color; using image processing to provide the extracted colors to define a new image; as well as The color desired set is defined as an ordered set according to a priority order, and wherein searching the color desired set using the plurality of clusters according to the priority order for evaluation stops when the color match is located.
7. A method comprising: Through the computing device's processor: obtaining a source color image comprising pixels, the source color image having a region of color to be extracted; clustering the pixels into a plurality of clusters without regard to intensity, each cluster in the plurality of clusters being characterized by a corresponding characteristic color; evaluating the plurality of clusters against a set of color expectations, the color expectations including corresponding expected colors to be present in the image, searching the plurality of clusters for a color match using the expected colors to define an extracted color; using image processing to provide the extracted colors to define a new image; as well as If each of two or more clusters provides a color match in response to the search, combining the two or more clusters of the plurality of clusters into a combined cluster; and defining the extracted color using the combined cluster.
8. A computing machine comprising: a processor coupled to a storage device storing instructions that, when executed by the processor, configure the computer to: obtaining a source color image comprising pixels, the source color image having a region of color to be extracted; clustering the pixels into a plurality of clusters without regard to intensity, each cluster in the plurality of clusters being characterized by a corresponding characteristic color; evaluating each of the characteristic colors of a corresponding plurality of clusters against a set of color expectations, the color expectations including a corresponding expected color to be present in the image, searching the plurality of clusters for a color match using the expected color to define the extracted color, the expected colors being a predefined list of expected colors for a type of cosmetic product present in the source color image; and Image processing is used to provide the extracted colors to define a new image that is different from the source color image and is a simulation of the use of the type of makeup on a user's face based on the extracted colors.
9. The computing machine of claim 8, wherein: Clustering the pixels includes applying k-means clustering to the pixels without considering intensity.
10. The computing machine of claim 8, wherein: Searching for the color match includes determining a similarity of the corresponding characteristic color of one of the plurality of clusters to a corresponding desired color of one of the desired sets of colors.
11. The computing machine of claim 8, wherein: The instructions configure the computing machine to obtain a starting image for defining the new image, and to define the new image from the starting image using the extracted colors to be applied to an object displayed within the new image, the starting image comprising a first selfie image, and the new image comprising a new selfie image containing the object, the object comprising a product for simulation on the first selfie image.
12. The computing machine of claim 11, the computing machine comprising a mobile device, and wherein: The products include makeup products to be virtually tried on using augmented reality.
13. A computing machine comprising: a processor coupled to a storage device storing instructions that, when executed by the processor, configure the computer to: obtaining a source color image comprising pixels, the source color image having a region of color to be extracted; clustering the pixels into a plurality of clusters without regard to intensity, each cluster in the plurality of clusters being characterized by a corresponding characteristic color; evaluating the plurality of clusters against a set of color expectations, the color expectations including corresponding expected colors to be present in the image, searching the plurality of clusters for a color match using the expected colors to define an extracted color; and using image processing to provide the extracted colors to define a new image; wherein searching for the color match comprises determining a similarity between the corresponding characteristic color of one of the plurality of clusters and a corresponding desired color of one of the color desired sets, and Wherein, when the corresponding characteristic color is similar to the corresponding expected color in both brightness and direction, the search locates the color match according to the corresponding threshold requirement of the corresponding expected color.
14. A computing machine comprising: a processor coupled to a storage device storing instructions that, when executed by the processor, configure the computer to: obtaining a source color image comprising pixels, the source color image having a region of color to be extracted; clustering the pixels into a plurality of clusters without regard to intensity, each cluster in the plurality of clusters being characterized by a corresponding characteristic color; evaluating the plurality of clusters against a set of color expectations, the color expectations including corresponding expected colors to be present in the image, searching the plurality of clusters for a color match using the expected colors to define an extracted color; and using image processing to provide the extracted colors to define a new image; Wherein the instructions configure the computing machine to define the color desired set as an ordered set according to a priority order, and wherein searching the color desired set using the plurality of clusters according to the priority order to evaluate stops when the color match is located.
