Apparatus and method for color matching and recommendation

Abstract

This application relates to devices and methods for color matching and recommendations. An image or spectrum of a surface can be acquired by a computing device, which in some instances can be included in a mobile device. The computing device can extract a measured spectrum from the image and generate a corrected spectrum of the surface. In some instances, the corrected spectrum can be generated to compensate for ambient light effects. The corrected spectrum can be analyzed to provide a result, such as a diagnosis or product recommendation. In some instances, the result is based at least in part on a comparison of the corrected spectrum to a reference spectrum. In some instances, the result is based at least in part on an inference of a machine learning model.

Description

Technical Field

[0001] This application relates to devices and methods for color matching and recommendation. Background Technology

[0002] Surface color analysis is used across a variety of industries. For example, surface color can be analyzed to recommend colors for cosmetics or paint. For instance, any consumer's choice of foundation is based on visual contrast. However, this requires the user to apply several different cosmetics, which can be cumbersome and time-consuming. In another instance, to identify the appropriate paint for a wall, a consumer might need to obtain a paint sample from a store, manually compare the sample to the wall, and then return to the store to purchase the suitable paint, or obtain additional samples if previous samples do not provide an acceptable match. Acquiring data to quantitatively analyze surface color and providing product recommendations based on the analysis may require specialized equipment and a well-controlled environment to minimize light interference. This can reduce usability and / or increase the cost of analyzing surface color. Therefore, it may be desirable to utilize techniques for analyzing surface color using cheaper and / or more readily available equipment. Summary of the Invention

[0003] On one hand, this application provides an apparatus comprising: a light source configured to illuminate a surface; a camera configured to acquire an image of the illuminated surface; and a processor configured to: extract a measurement spectrum from the image; generate a corrected spectrum by at least partially compensating for the influence of ambient light on the measurement spectrum; compare the corrected spectrum with a reference spectrum in a database of reference spectra to produce a result; and provide at least one of a product recommendation or a diagnosis based at least in part on the result.

[0004] On the other hand, this application provides a method comprising: illuminating a surface using a light source of a mobile device; acquiring an image of the illuminated surface; extracting a measurement spectrum representing the color of the surface from the image; correcting the measurement spectrum, wherein correcting the measurement spectrum includes adjusting the effect of ambient light on the measurement spectrum to generate a corrected spectrum; comparing the corrected spectrum with a database of reference spectra to determine the spectrum in the reference spectra that best matches the corrected spectrum; and providing a result based at least in part on the best-matching spectrum.

[0005] In another aspect, this application provides an apparatus comprising: a light source configured to illuminate a surface; a camera configured to acquire an image of the surface; an interface configured to couple a spectrometer configured to generate a measurement spectrum from the surface; a processor configured to: analyze the measurement spectrum to compensate for the influence of ambient light from the measurement spectrum and generate a correction spectrum; and a cloud computing device communicating with the processor, which includes a database of reference spectra, wherein the cloud computing device is configured to: compare the correction spectrum with the database of reference spectra to generate a result; and provide at least one of a product recommendation or a diagnosis, at least in part based on the result. Attached Figure Description

[0006] Figure 1 This is a schematic diagram of a system arranged according to an example of this disclosure.

[0007] Figure 2 This is a schematic diagram of a computing system arranged according to an example of this disclosure.

[0008] Figure 3 This is an example of a mobile device based on the present disclosure.

[0009] Figure 4 An exemplary spectrum is shown according to an example of this disclosure.

[0010] Figure 5 A method according to an embodiment of an example of this disclosure is shown.

[0011] Figure 6 This is a flowchart of a method based on an example of this disclosure.

[0012] Figure 7 This is a flowchart of a method based on an example of this disclosure. Detailed Implementation

[0013] The following description of certain embodiments is merely exemplary in nature and is not intended to limit the scope of this disclosure or its application or use. In the following detailed description of embodiments of the devices, systems, and methods, reference is made to the accompanying drawings, which form part of the invention, and illustrate specific embodiments of the described devices, systems, and methods by way of illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice the currently disclosed devices, systems, and methods, and it should be understood that other embodiments may be utilized, and structural and logical changes may be made without departing from the spirit and scope of this disclosure. Furthermore, for clarity, detailed descriptions of certain features will not be discussed where they are obvious to those skilled in the art, so as not to obscure the description of embodiments of this disclosure. Therefore, the following detailed description should not be considered limiting, and the scope of this disclosure is defined only by the appended claims.

[0014] Surface color analysis is an emerging area of ​​focus for providing improved color matching, such as in cosmetics and coatings. For example, in cosmetics, lighting conditions and / or skin sheen can lead to misjudgments of skin tone, resulting in poor product selection. A store clerk might recommend foundation products based on a customer's skin tone appearance. However, skin tone can vary as the customer leaves the store due to changes in lighting. If a clerk and customer meet outside the store under different lighting conditions, the clerk might recommend different foundation products. Existing technologies for analyzing surface color often require specialized equipment, which can limit the ability to acquire surface color data and / or provide recommendations based on surface color analysis. Therefore, more economical and / or reliable surface color analysis technologies are desired.

[0015] According to examples of this disclosure, a mobile device can be used to acquire surface color data and extract a measured spectrum from the data. In some instances, the mobile device can further analyze the measured spectrum to compensate for the influence of ambient light on the measured spectrum and generate a calibrated spectrum. In some instances, the mobile device can provide product recommendations and / or diagnoses based at least in part on the calibrated spectrum. Where the surface is a user's skin, product recommendations may include cosmetics (e.g., primer, foundation, setting powder, concealer) and diagnoses may include conditions (e.g., redness, pores, pigmentation, oily or dry skin). In another instance, the surface may be a wall and product recommendations may include paint colors that match the color or are complementary to the hue of the surface color (e.g., warm, refreshing, neutral). In yet another instance, the surface may be a garment and product recommendations may include matching garment items and / or accessories (e.g., shoes, sunglasses, necklaces, etc.). In some instances, product recommendations and / or diagnoses may be generated by a machine learning model based at least in part on the measured spectrum and / or user feedback on the product recommendations and diagnoses. As used in this article, machine learning is collectively referred to as machine learning, deep learning, and / or other artificial intelligence techniques used to infer from data.

[0016] Figure 1This is an illustration of a system 100 arranged according to an example of the present disclosure. System 100 may include a processor 102, a computing device 114, a spectrometer and / or camera 104, and a light source 106. In some instances, system 100 may also include a cloud computing device 110. System 100 may use the camera and / or spectrometer 104 to acquire images and / or spectra of surface 112. In some instances, surface 112 may be illuminated by ambient light Ia. In some instances, surface 112 may be illuminated alternatively or additionally by light Is from light source 106, and the spectrometer and / or camera 104 may acquire data associated with surface 112I. The data may include images and / or spectra of surface 112. The data may be processed by processor 102 to extract measurement spectrum I. For example, processor 102 may extract measurement spectrum from images acquired by camera 104, or it may acquire measurement spectrum directly from spectrometer 104. However, in some applications, for example, the measured spectrum may not accurately reflect the color of surface 112 due to the influence of ambient light 108. Ambient light 108 may have a non-uniform and / or unknown spectrum, which may affect the apparent color of surface 112. Ambient light 108 may be reflected by surface 112 and / or other surfaces and collected by camera and / or spectrometer 104. Therefore, the measured spectrum I extracted by processor 102 may be a mixture of the spectrum of surface 112 and ambient light 108. Therefore, processor 102 may analyze the measured spectrum to eliminate the influence of ambient light Ia from the measured spectrum, thereby generating a corrected spectrum I0 for surface 112. That is, in some instances, the corrected spectrum containing the spectrum of surface 112 may be expressed as I0 = I - Ia. In instances where light source 106 is used to illuminate surface 112, the corrected spectrum may be expressed as I0 = I - Ia - Is.

[0017] The computing device 114 can compare the calibrated spectrum I0 with a database of reference spectra to produce a result. The result may be the spectrum from the database of reference spectra that best matches the calibrated spectrum of surface 112. The computing device 114 may provide product recommendations and / or surface diagnoses based at least in part on the result.

[0018] In some instances, computing device 114 may optionally be communicatively coupled to cloud computing device 110. For example, processor 102 may be communicatively coupled to cloud computing device 110. In some instances, cloud computing device 110 may store data including images and / or spectra received from computing device 114, measured spectra, calibrated spectra generated by processor 102, one or more databases of reference spectra, results generated by processor 102, and / or user feedback. In some instances, cloud computing device 110 may implement one or more machine learning models to make inferences based on data provided by computing device 114. For example, processor 102 may provide calibrated spectra to cloud computing device 110, and cloud computing device 110 may return product recommendations to computing device 114 based on the inferences of the one or more machine learning models.

[0019] Figure 2 This is a schematic diagram of a computing system 200 arranged according to an example of the present disclosure. The computing system 200 may include a mobile device 222, a computing device 218, and / or a light source 208. Optionally, in some examples, the computing system 200 may include a display 204, a camera 212, a spectrometer 210, and / or a cloud computing device 230. In some examples, the computing device 218 may include one or more processors 206, a computer-readable medium (or multiple media) 224, a memory controller 220, a memory 216, and / or a user interface 214. In some examples, the computing system 200 may be used to implement system 100. In some examples, the computing device 218 may be used to implement computing device 114. In some examples, the processor 206 may be used to implement processor 102. In some examples, the cloud computing device 230 may implement cloud computing device 110. In some examples, the light source 208 may be implemented... Figure 1 The light source 106 shown is shown.

[0020] In some instances, computing device 218 may be included in mobile device 222 (such as a smartphone, cellular phone, gaming device, or tablet). In some instances, computing device 218 may be implemented wholly or partially using a computer, server, television, or laptop. In some instances, spectrometer 210 may be an accessory device connected to computing device 218 and / or mobile device 222. In other instances, spectrometer 210 may be an integrated element of computing device 218 and / or mobile device 222, such as a sensor on a smartphone. In some instances, such as Figure 2 In the example shown, the light source 208, camera 212, and / or display 204 may be integrated elements of a mobile device 222 that communicates with a computing device 218, such as the camera, flash, and touchscreen of a smartphone.

[0021] In some other instances, processor 206 may be implemented using one or more central processing units (CPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or other processor circuitry systems. In some instances, processor 206 may communicate with memory 216 via memory controller 220. In some instances, memory 216 may be volatile memory, such as dynamic random access memory (DRAM). In some instances, memory 216 may provide information to and / or receive information from processor 206 and / or computer-readable medium 224 via memory controller 220. Although a single memory 216 and a single memory controller 220 are shown, any number may be used. In some instances, memory controller 220 may be integrated with processor 206.

[0022] Computing device 218 may include computer-readable medium 224. Computer-readable medium 224 may be implemented using any suitable medium, including non-transitory computer-readable media. Examples include memory, random access memory (RAM), read-only memory (ROM), volatile or non-volatile memory, hard disk drive, solid-state drive, or other storage devices. Processor 206 and / or memory controller 220 may access computer-readable medium 224. Computer-readable medium 224 may be encoded using executable instructions 228. Executable instructions 228 may be executed by processor 206. For example, executable instructions 228 may cause processor 206 to analyze acquired images to extract measurement spectra from the images. In some instances, for example, executable instructions 228 may cause processor 206 to correct measurement spectra extracted from images or acquired from spectrometer 210 to correct for the effects of ambient light. In some instances, executable instructions 228 may enable processor 206 to provide commands or other control signals to light source 208, camera 212, display 204, spectrometer 210 and / or other components of computing device 218, such as memory controller 220.

[0023] Computer-readable medium 224 may store data 226. In some instances, data 226 may include images received from camera 212 and / or measured spectra extracted from the images, spectra acquired by spectrometer 210, calibration spectra generated by processor 206, a database of reference spectra, product recommendations, diagnostics, and / or user feedback. Although Figure 2 A single medium is shown, but multiple media can be used to implement computer-readable medium 224.

[0024] The computing device 218 may communicate with the display 204 as a separate component (e.g., using a wired and / or wireless connection), or the display 204 may be integrated with the mobile device 222. In some instances, the display 204 may display data, such as output (e.g., results) generated by the processor 206 and / or images acquired by the camera 212. Any number or type of display may be present, including one or more LED, LCD, plasma, or other display devices.

[0025] In some instances, user interface 214 may receive input from user 202. User input may include, but is not limited to, desired resolution, selection of one or more relevant areas, tastes and preferences, and desired output (e.g., diagnostics, product recommendations). Examples of user interface components include a keyboard, mouse, touchpad, touchscreen, and microphone. In some instances, display 204 may be included in user interface 214. In some instances, processor 206 may implement a graphical user interface (GUI) via user interface 214 including display 204. For example, user 202 may view an image of an object's face or other surface on display 204, which in some instances may be a touchscreen, and circle relevant areas on the image. In some instances, such as Figure 2 In the examples shown, the subject can be user 202. In some instances, processor 206 can transfer information between one or more components of computing device 218 and an external device, such as spectrometer 210, that can generate a measured spectrum of a surface; this information may include user input, data, images, and / or commands. In other instances, user interface 214 can be coupled with a recommendation engine based on a recommendation model to provide product recommendations. Product recommendations may be based on user tastes and preferences, and a definitive match of the calibrated spectrum generated by processor 206.

[0026] In some instances, light source 208 may emit light for image acquisition using camera 212 and / or spectrometer 210. In some instances, light source 208 may comprise a visible light source, an infrared light source, an infrared floodlight illuminator, an ultraviolet light source, and / or a broad-spectrum light source (e.g., comprising two or more of visible, infrared, and ultraviolet light). In some embodiments, the light source may be polarized. In some instances, light source 208 may comprise an LED, a xenon flash tube, and / or other light sources. In some instances, light source 208 may be included in camera 212. In some instances, light source 208 may emit light in response to commands provided by camera 212 and / or one or more commands provided by processor 206.

[0027] In some instances, camera 212 can be used to acquire data from surfaces (such as...). Figure 1The image of surface 112 shown. In some instances, camera 212 may include a charge-coupled device (CCD), a complementary metal-oxide-semiconductor (CMOS) sensor, and / or other optical sensors. See reference... Figure 6 In more detail, processor 206 can extract the spectrum from the image acquired by camera 212. In some instances, in addition to acquiring images using camera 212, or instead of acquiring images using the camera, spectrometer 210 can acquire the spectrum of a surface. In some instances, spectrometer 210 may include a CCD, photoresistor, photodiode, phototransistor, and / or other light sensors.

[0028] In some instances, user 202 can use camera 212 to capture images of the surface. Processor 206 can extract the measurement spectrum from the image and generate a corrected spectrum to eliminate the influence of ambient light on the measurement spectrum. In some instances, the influence of ambient light can be eliminated through white balance, which will refer to... Figure 6 A more detailed description follows. In some instances, the user interface 214 may prompt the user 202 to capture a second image of the white card or other "controls" under the same lighting conditions as when acquiring the surface image. The processor 206 may extract a second measurement spectrum from the second image and analyze the second measurement spectrum to compensate for the influence of ambient light from the second measurement spectrum and generate a second corrected spectrum. In some instances, white balance may be applied to the second image to account for the "color temperature" of the light source 208 and / or ambient light.

[0029] In another example, camera 212 can acquire multiple images of the illuminated surface simultaneously. Processor 206 can compare each measured spectrum of the multiple acquired images and generate a corrected spectrum for each measured spectrum. Processor 206 can generate an average spectrum of the corrected spectra of the multiple images.

[0030] A calibration spectrum (or the average calibration spectrum in an instance where multiple spectra are acquired) generated by processor 206 from a measurement spectrum (or multiple spectra) via white balance and / or other methods can be compared with one or more reference spectra in a database of reference spectra. In some instances, the database of reference spectra can be stored as data 226 on a computer-readable medium 224. In some instances, one or more of the spectra may each correspond to a different product (e.g., different paint colors, different foundation colors). In other instances, one or more of the spectra may each correspond to a condition (e.g., rosacea, melanoma). In some instances, processor 206 can determine the reference spectrum that best matches the calibration spectrum. Figure 4The techniques used for comparison and / or determination of the best match are described in more detail. Based on the comparison, processor 206 can output a product and / or condition (e.g., condition diagnosis) corresponding to the best-matching reference spectrum as a result. In some instances, product recommendations and / or diagnoses may be based at least in part on results, user preferences, user feedback, or a combination thereof.

[0031] In some instances, processor 206 may implement a machine learning model trained to provide product recommendations and / or diagnoses based on a calibrated spectrum and / or a best-matching reference spectrum. In some instances, the machine learning model may provide product recommendations and / or diagnoses based at least in part on received user feedback regarding previous product recommendations and / or diagnoses. In some instances, the machine learning model may be implemented by executable instructions 228 that execute on processor 206. In some instances, the machine learning model, or a portion thereof, may be implemented in the hardware included with processor 206.

[0032] In some instances, processor 206 may be communicatively coupled to cloud computing device 230 via user interface 214. Processor 206 may provide images and / or spectra (e.g., measured, calibrated) to cloud computing device 230. Cloud computing device 230 may generate calibrated spectra, compare the calibrated spectra with a database of reference spectra stored on cloud computing device 230, provide results based on the comparison (e.g., product recommendations, diagnostics), and / or make inferences based at least in part on a machine learning model implemented by cloud computing device 230 to produce results. Cloud computing device 230 may provide processor 206 with calibrated spectra, one or more reference spectra, diagnostics, and / or recommended products. In some instances, calibrated spectra, product recommendations, and / or diagnostics may be provided on display 204. In some instances, cloud computing device 230 may contain a database of products and / or conditions in computer-readable media (not shown). This arrangement may be desirable in some applications, for example, when computing device 218 may have limited computing power, such as if computing device 218 is contained in compact mobile device 222. This arrangement may be more convenient when machine learning models are dynamically trained and / or spectral databases are dynamically updated.

[0033] Figure 3This is an example of a mobile device 300 according to embodiments of the present disclosure. In some embodiments, mobile device 300 may be used to implement mobile device 222. Mobile device 300 may include a display 302. Display 302 may provide a GUI implemented by mobile device 222. For example, display 302 may provide diagnostics 306, measurement spectra 308, and / or calibration spectra 310. In some embodiments, mobile device 300 provides and displays on display 302 a diagnostic representing a comparison between calibration spectra 310 and a spectral database. Display 302 may include product recommendations 304 at least in part based on diagnostics 306.

[0034] Optionally, in some instances, the display 302 may provide an image 312 of a surface (e.g., surface 112) acquired by a camera (e.g., camera 212) of the mobile device 300. In some instances, a user (e.g., user 202) may be able to select a region of relevance (ROI) 314 within the image 312. For example, if the display 302 is a touchscreen, the user may tap or circle a portion of the image 312 to select the ROI 314. In some instances, the measurement spectrum may be extracted only from the data in the image 312 within the ROI 314.

[0035] Figure 4 An exemplary spectrum 400 according to an example of this disclosure is shown. The spectrum 400 may be generated and / or stored in a computing device (such as computing device 110 and / or 218), and may be included on a mobile device (such as mobile device 222 and / or mobile device 300). In some embodiments, the spectrum 400 may be displayed to a user, for example, on a display 204 and / or 302. Figure 4 In the image, spectrum (a) shows the corrected reflectance spectrum 404 within a wavelength range. Examples of wavelength ranges include ultraviolet (UV), visible, and / or infrared spectra. UV spectra typically encompass a wavelength range from about 10 nm to about 400 nm. Visible spectra typically encompass a wavelength range from about 380 nm to about 760 nm. Infrared spectra typically encompass a wavelength range from about 700 nm to about 1 mm.

[0036] As described in this article, for example, refer to Figure 1 and 2A computing device (such as computing devices 110 and / or 218) can compare a calibrated spectrum (e.g., a calibrated spectrum calculated by processor 102 and / or processor 206) with one or more reference spectra from a database of reference spectra. In some instances, each spectrum may correspond to a product (e.g., paint color, foundation hue). Spectra (b) and (c) are examples of comparisons between calibrated reflectance spectrum 404 and reference spectra 402, 406 from a database of reference spectra. The comparison method may include regression analysis, such as least mean square, least squares, or total least squares. In some instances, the reference spectrum that best matches the calibrated spectrum of a surface can be used to identify candidate products and provide product recommendations. In this example, spectrum (b) shows a comparison of calibrated spectrum 404 with a first reference spectrum 402, and spectrum (c) is a comparison of calibrated spectrum 404 with a second reference spectrum 406. Figure 4 In the example shown, the first spectrum 402 matches the corrected spectrum 404 better than the second spectrum 406. Therefore, a product corresponding to spectrum 402 can be selected as a recommendation. In an example where a user seeks product recommendations for their skin (e.g., foundation), the recommended product could have a spectrum that best matches the user's skin tone spectrum.

[0037] Alternatively, instead of finding the best-matching reference spectrum across the entire wavelength range, comparisons can be made on a subset of the wavelengths. When only a portion of the spectrum affects the product's appearance, it may be appropriate to seek the best match only on that subset. For example, the spectral characteristics of a product in the UV and / or far IR regions may have little or no impact on how the product appears to the user. Therefore, in some instances, these regions of the spectrum can be ignored when calculating the best-matching reference spectrum to the calibration spectrum 404.

[0038] In other instances, users may seek decorating recommendations based on the color of a wall. Based on the corrected spectrum of the wall, candidate products could include fabrics and / or other products in colors that match the wall. In these instances, as described above, the corrected spectrum can be compared to a reference spectrum. Alternatively or additionally, instead of providing a matching (e.g., identical) color, product recommendations could offer aesthetically pleasing colors or colors that complement the wall color. For example, if a wall is identified as an eye-catching bright hue due to its strong hue and / or brightness (e.g., red, yellow, orange), the recommendation could be a color that balances the strong hue and / or brightness (e.g., brown, gray). In these instances, the best-matching reference spectrum could be associated with a database of complementary colors and / or complementary color products. Alternatively, the corrected spectrum can be analyzed to determine relative or absolute intensity in different spectral ranges (e.g., red, blue, green). These intensities can be compared to an intensity reference table associated with a complementary color database. In another instance where a user is seeking clothing item recommendations, the user can obtain an image of a clothing item and receive recommendations for other clothing or accessories in the image that might match the clothing item, based on color. An example recommendation could include a pair of shoes, earrings, a tie, sunglasses, etc.

[0039] Figure 5 This is a flowchart of method 500 according to an example of this disclosure. Figure 5 The method described herein illustrates a method when user feedback is included in a database of product recommendations and / or diagnostics updated based on user feedback. In some instances, all or part of method 500 may be generated by a computing device (e.g., Figure 1 The computing device 114 and / or Figure 2 The computational device 218 in the middle executes the method. In some instances, all or part of method 500 can be executed by... Figure 1 Processor 102 in Figure 2 The machine learning model implemented by the processor 206, cloud computing device 110 and / or cloud computing device 230 is used to execute.

[0040] At block 502, the "Receive User Feedback" function can be executed. In some instances, the display (e.g., display 204 and / or display 302) can prompt the user for feedback regarding product recommendations. In some instances, user feedback can be provided by a user interface (such as...) Figure 2User feedback is received by the user interface 214 shown. In some instances, user feedback may include text, image, and / or sound data. In some instances, user feedback may be received from a mobile device such as mobile device 222 and / or a computing device such as computing device 114 or computing device 218. User interface 214 may transmit user input to a processor (e.g., processor 102 and / or processor 206) and / or a cloud computing device (e.g., cloud computing device 110 and / or cloud computing device 230).

[0041] At block 504, you can perform a "Grade User Feedback" operation. In some instances, the grade can indicate the user's satisfaction with the product recommendations and / or diagnoses provided. At block 506, you can perform a "Categorize User Feedback" operation. In some instances, categorization can indicate one or more factors included in the user feedback. For example, a user might be satisfied with the accuracy of a diagnosis but dissatisfied with the price range of a product recommendation. In some instances, categorization can be rule-based. In some instances, block 506 can be performed before block 504. In some instances, blocks 504 and 506 can be performed simultaneously.

[0042] At block 508, you can execute "Provide user feedback to the machine learning model". In some instances, the user feedback provided to the machine learning model can include tiered user feedback. That is, the user feedback provided to the machine learning model can be modified from the data initially received at block 502. In some instances, the user feedback can be used as a training dataset to train the machine learning model. Optionally, at block 510, you can execute "Train the machine learning model". The machine learning model can be trained using a training dataset.

[0043] In this way, user feedback can be reflected in future product recommendations and / or diagnoses. In some instances, recommendations for products in a lower price range can be offered as alternatives to users who find the results accurate but the recommended products too expensive. In other instances, the user interface can periodically ask users to complete surveys. Surveys might include questions such as ratings of the accuracy of product recommendations and / or diagnoses, and user preferences for additional features of products (e.g., SPF products, high-coverage foundations, hypoallergenic paints, etc.). Therefore, users can train machine learning models to recommend products using the feedback they provide to the user interface.

[0044] In some instances, a machine learning model may contain a neural network. In some instances, the neural network may be a convolutional network with two-dimensional or three-dimensional layers. The neural network may contain input nodes that receive input data (e.g., a calibrated spectrum, a reference spectrum, a combination thereof, and / or a portion thereof). In some instances, the input nodes may be organized within a single layer of the neural network. Input nodes may be coupled to one or more hidden units via one or more weights. In some instances, hidden units may perform operations on one or more inputs from input nodes, at least partially based on the association weights between nodes. In some instances, hidden units may be coupled to one or more hidden units via weights. Hidden units may perform operations on one or more outputs from hidden units, at least partially based on the weights. The outputs of the hidden units may be provided to output nodes to provide results (e.g., product recommendations, diagnostics). Of course, neural networks are provided only as examples, and one or more other machine learning models (e.g., decision trees, support vector machines) may be used. In some instances, a machine learning model may contain different machine learning models for different applications. For example, there may be a machine learning model for cosmetic applications (e.g., determining cosmetic recommendations) and separate machine learning models for other applications (e.g., determining coordinated clothing / accessories).

[0045] In some instances, a machine learning model can be trained by providing one or more training datasets. In some instances, the machine learning model can be trained by a computing device (e.g., computing devices 114, 222, cloud computing devices 110, 230) used for inference using the machine learning model. In some instances, the machine learning model can be trained by another computing device to determine the appropriate machine learning model to be implemented, weights, node arrangement, or other model configuration information, and the trained machine learning model can be provided to the computing device used for inference.

[0046] In some instances, supervised learning techniques can be used to train machine learning models. In some instances, the training data may contain a set of inputs x, each associated with a desired outcome y (e.g., labeled). Each input x may contain one or more values ​​of one or more spectra. For example, an input x might contain the spectrum associated with the outcome y, which is a reference spectrum associated with paint color. Based on the training dataset, the machine learning model can adjust one or more weights, layers, or other components of the model. The trained machine learning model can then be used to infer the outcome y from inputs x (not associated with the desired outcome). In some instances, semi-supervised and / or unsupervised techniques can be used to train machine learning models. In these instances, the dataset may not contain the desired outcome associated with each input.

[0047] In some instances, such as Figure 5 The examples described demonstrate how machine learning models can be trained dynamically. That is, even if a machine learning model was initially trained on a different device, it can continue to adjust based on new data. For example, new products and / or user feedback may lead to adjustments in the machine learning model.

[0048] Figure 6 This is a flowchart of a method according to an example of this disclosure. At block 602, the moving device utilizes a light source (e.g., Figure 1 The light source 106 and / or Figure 2 The light source 208 in the middle illuminates the surface. At block 604, the moving device utilizes, as shown in... Figure 1 Camera 104 and Figure 2 The camera 212 shown in the figure acquires an image of the illuminated surface.

[0049] At block 606, processors such as processor 106 and / or processor 206 extract a measured spectrum representing the color of the surface from the image. In some instances, the processor can extract the measured spectrum from the image by decoding the surface color using red-green-blue (RGB) weighting and assigning an RGB color code to the surface color. Histograms of red, green, and blue pixels can be generated for the image. The surface color can be determined at least in part based on the concentration of pixels. For example, if approximately equal numbers of red, green, and blue pixels exist, the processor can assign an RGB color code indicating that no single color is dominant.

[0050] Additionally or alternatively, the processor can extract the measurement spectrum by applying MPEG 7 global descriptors. MPEG 7 global descriptors define various color descriptors representing different aspects of color characteristics, including representing color, basic color distribution, global spatial distribution of color, local spatial distribution of color, and the perceived temperature of the image. Five color descriptors exist to represent different aspects of color characteristics: DominantColor, for representing color; ScalableColor, for basic color distribution; ColorLayout, for global spatial distribution of color; ColorStructure, for local spatial distribution of color; and ColorTemperature, describing the perceived temperature of the image. Additionally, the GoFGoPColor descriptor is defined as an extension of ScalableColor to a group of frames or images. Three supporting tools are defined: ColorSpace, ColorQuantization, and IlluminationInvariantColor. All descriptors and tools are applicable to regions of arbitrary shape. When calibrating the measurement spectrum, the signal-to-noise ratio of the measurement spectrum can be optimized by using filters and / or averaging the measurement spectrum.

[0051] Therefore, in some instances, RGB color codes, histograms, and / or MPEG 7 global descriptors can be used to generate the measured spectrum. In other instances, this can be performed using machine learning. Machine learning models can be similar to those described above. Figure 5 Training can be performed as described. For example, a machine learning model can be provided with RGB color codes, histograms, and / or MPEG7 global descriptors associated with the spectra acquired by the spectrometer. However, in other instances, other methods can be used to generate spectra from RGB color codes, histograms, and / or MPEG 7 global descriptors. In other instances, measured and calibrated spectra are saved in RGB color code, histogram, and / or MPEG 7 global descriptor format, and the spectra of a database of reference spectra are also saved in RGB color code, histogram, and / or MPEG 7 global descriptor format.

[0052] At block 608, the processor adjusts the measured spectrum by eliminating the influence of ambient light on the measured spectrum to generate a calibrated spectrum. In some instances, the influence of ambient light can be eliminated through white balance. In some instances, the user can be prompted to take an image of a white card under the same lighting conditions as the surface as a white reference. The processor can extract the measured spectrum of the white card, representing the ambient light influence Ia, and the measured spectrum of the surface image. To generate the calibrated spectrum of the surface, the processor subtracts the measured spectrum of the white card from the measured spectrum of the surface image.

[0053] At block 610, the processor can compare the calibrated spectrum with a database of reference spectra to determine the reference spectrum that best matches the calibrated spectrum. For example, if the reference... Figure 4 As discussed, at block 612, results can be provided at least partially based on the best-matching spectrum. In some instances, results can be provided via a GUI on a display (e.g., display 204 and / or display 302).

[0054] Optionally, in some instances, a floodlight or other light source can illuminate the surface, acquiring additional images of the surface to detect additional surface properties, including texture and sub-attributes. Texture analysis can be performed to adjust the texture based on the surface's color. Examples of textures can include skin sheen. White balance can be applied to the additional images to account for the "color temperature" of the light source. Color chromatograms can be analyzed based on the surface image and the additional images taken with the floodlight. Product diagnostics or recommendations can be provided based on the color chromatograms.

[0055] Optionally, in some instances, a calibrated spectrum can be analyzed to determine the gloss of a surface. In instances where the surface is skin cells, gloss can indicate oiliness. A matte foundation is recommended for oily skin. In some instances, the gloss of a surface can be determined by specular reflection based on the ratio of the intensity of the light source used to illuminate the surface to the light reflected from the surface. For example, gloss can be determined by the ratio of the intensity of the reflected beam to the intensity of an unpolarized light source. If the intensity of the reflected light is greater than a threshold, the surface can be determined to be glossy. In some instances, the threshold can be a percentage of the light source intensity. If the intensity of the reflected light is less than the threshold, the surface can be determined to be matte.

[0056] Figure 7 This is a flowchart of a method based on an example of this disclosure. Figure 7 The method in the text shows that when such as Figure 1 Spectrometer 104 and / or Figure 2 The method for obtaining surface spectra using the spectrometer 210 in the middle.

[0057] At block 702, the moving device uses a light source (e.g., Figure 1 The light source 106 and / or Figure 2 The light source 208 illuminates the surface having the desired color. The light source can be a moving device (such as...) Figure 2 Mobile device 222 or Figure 3 The mobile device 300 is provided.

[0058] At block 704, as shown by processors 102 and / or 206, the processor of the mobile device can obtain the spectrum of the signal reflected from the surface having the desired color, as referenced. Figure 1 and 2 As explained. In some instances, the spectrum of the signal can be obtained by a spectrometer coupled to the mobile device, such as the reference. Figure 2 The explanation given.

[0059] At block 706, the mobile device can access a stored representation of the candidate product sample. This stored representation can be a local memory implemented by memory 216, or a remote memory including cloud storage, etc. An instance of cloud storage can be... Figure 1 The cloud computing device 110 and / or Figure 2 The cloud computing device 230 is shown in the diagram.

[0060] At block 708, the processor compares the signal spectrum obtained by the spectrometer and selects at least one candidate product sample as a match for the desired color based on the comparison between the spectrum and the stored representation. In an instance where the candidate product sample includes foundation, the surface with the desired color can be skin. In another instance where the candidate product sample includes a paint sample, the surface with the desired color can be part of an object (e.g., a wall, a car) or an image. Optionally, in some instances, the user can provide user feedback on the candidate product samples, such as... Figure 5 As described above. In some instances, user feedback may influence the future selection of candidate product samples.

[0061] As disclosed herein, in various applications (e.g., cosmetics, interior design, fashion), mobile devices can analyze surface color and provide surface diagnostics and / or product recommendations based on the surface analysis. Therefore, the devices, systems, methods, and apparatuses of this disclosure can allow users to access data about surface color and receive recommendations, at least in part, based on surface color, without requiring expensive and / or specialized equipment.

[0062] Of course, it should be understood that any of the examples, embodiments or processes described herein may be combined with one or more other examples, embodiments and / or processes, or may be performed separately and / or between separate devices or device parts, based on this system, apparatus and method.

[0063] Finally, the above discussion is merely illustrative and should not be construed as limiting the appended claims to any particular embodiment or group of embodiments. Therefore, although various embodiments of this disclosure have been described in particular detail, it should be understood that many modifications and alternative embodiments can be devised by those skilled in the art without departing from the broader and contemplated spirit and scope of this disclosure as set forth in the appended claims. Thus, the specification and drawings are to be considered illustrative and are not intended to limit the scope of the appended claims.

Claims

1. An apparatus comprising: a light source configured to illuminate a surface; a camera configured to: acquire an image of the illuminated surface; and acquire a second image of a white reference under the same lighting environment as the illuminated surface; and a processor configured to: extract a measured spectrum from the image; extract a second measured spectrum from the second image; produce a corrected spectrum by at least partially compensating for effects of ambient light on the measured spectrum, wherein the corrected spectrum is produced by subtracting the second measured spectrum of the second image from the measured spectrum of the image; compare the corrected spectrum to reference spectra in a database of reference spectra to produce a result; and provide at least one of a product recommendation or a diagnosis based at least in part on the result.

2. The apparatus of claim 1, wherein the comparison of the corrected spectrum to the reference spectra in the database of reference spectra includes using a regression analysis method comprising at least one of least mean square, least squares, or total least squares.

3. The apparatus of claim 1, wherein the result includes a spectrum in the reference spectra that best fits the corrected spectrum, a group of spectra in the reference spectra that best fit at different spectral ranges, or a combination thereof.

4. The apparatus of claim 1, further comprising a user interface configured to receive preferences from a user, wherein the user interface is coupled to a recommendation engine configured to provide the product recommendation based on the user’s preferences and the result.

5. The apparatus of claim 4, wherein the user interface is further configured to prompt the user to move the apparatus to a new area of the surface, and wherein the camera is further configured to take a second image of the new area, wherein the processor is further configured to extract a second measured spectrum from the second image of the new area, analyze the second measured spectrum to compensate for ambient light effects from the second measured spectrum and produce a second corrected spectrum, compare the second corrected spectrum to the database to produce a second result, and provide at least one of a second product recommendation or a second diagnosis based at least in part on the comparison between the corrected spectrum and the database and the second comparison of the second corrected spectrum to the database.

6. The apparatus of claim 1, wherein the camera is further configured to acquire a plurality of images of the illuminated surface, and wherein the processor is further configured to compute an average spectrum of the measured spectra of the plurality of images, and produce a second corrected spectrum based on the average spectrum, compare the second corrected spectrum to the database to produce a second result, and provide at least a second product recommendation or a second diagnosis based at least in part on the comparison between the second corrected spectrum and the database.

7. The apparatus of claim 1, wherein the database is stored in a cloud computing device in communication with the processor or a local memory included with the apparatus in communication with the processor. ​ ​ ​ 8. The apparatus of claim 1, wherein the processor is configured to implement a machine learning model to provide at least one of a product recommendation or a diagnosis, wherein the machine learning model is based at least in part on user feedback.

9. The apparatus of claim 1, wherein the surface comprises skin cells, and wherein the product recommendation comprises a foundation that matches a skin tone of the skin cells.

10. The apparatus of claim 1, wherein the surface comprises a wall, and wherein the product recommendation comprises a decoration that is at least complementary to the wall.

11. The apparatus of claim 1, further comprising a display, wherein the display provides at least one of the image of the surface, the product recommendation, or the diagnosis.

12. The apparatus of claim 1, wherein the light source is a visible light source or an infrared (IR) flood illuminator.

13. The apparatus of claim 1, wherein the apparatus is a mobile device.

14. A method comprising: illuminating a surface with a light source of a mobile device; acquiring an image of the illuminated surface; acquiring a second image of a white reference under the same lighting environment as the illuminated surface; extracting a measured spectrum representative of a color of the surface from the image; correcting the measured spectrum, wherein correcting the measured spectrum comprises adjusting an effect of ambient light on the measured spectrum to produce a corrected spectrum, wherein adjusting the effect of the ambient light comprises subtracting a spectrum of the white reference from the measured spectrum; comparing the corrected spectrum to a database of reference spectra to determine a most matching spectrum of the reference spectra to the corrected spectrum; and providing a result based at least in part on the most matching spectrum.

15. The method of claim 14, wherein extracting comprises decoding a color with a red-green-blue (RGB) weighting; and providing an RGB color code of the color of the surface.

16. The method of claim 14, wherein extracting comprises analyzing a color spectrum further comprises applying an MPEG7 global descriptor.

17. The method of claim 14, wherein correcting the measured spectrum further comprises optimizing a signal-to-noise ratio of the measured spectrum.

18. The method of claim 14, further comprising analyzing the corrected spectrum to determine a sheen of the surface, wherein the sheen of the surface is determined by a ratio of an intensity of a reflected light beam to an intensity of an unpolarized light, and wherein the result is based at least in part on the sheen of the surface.

19. The method of claim 14, further comprising analyzing at least one of the corrected spectrum or the most matching spectrum with a machine learning model to make an inference; and providing the result based at least in part on the inference.

20. The method of claim 14, wherein comparing the corrected spectrum to the reference spectra comprises comparing the corrected spectrum to the reference spectra within a range comprising ultraviolet (UV) spectra, visible spectra, and infrared spectra.

21. The method of claim 14, wherein the result comprises at least one of a product recommendation or a diagnosis. ​ 22. The method of claim 21, wherein the product recommendation is for a product having a spectrum that is complementary to the spectrum that most closely matches the corrected spectrum, and wherein the product recommendation is based on a database of products based at least in part on user feedback.

23. The method of claim 14, further comprising: acquiring additional images of the surface with a flood illuminator to detect texture and subsurface; performing texture analysis to adjust the texture according to the color of the surface; applying white balancing on the additional images; analyzing a color spectrum based on the images and the additional images; and providing a recommendation or diagnosis of a product based at least in part on the analyzed color spectrum.

24. An apparatus comprising: a light source, wherein the light source is configured to illuminate a surface; a camera configured to acquire an image of the surface and a second image of a white reference under the same lighting environment as the surface; an interface configured to couple a spectrometer, wherein the spectrometer is configured to produce a measured spectrum from the surface; a processor configured to: analyze the measured spectrum to compensate for the effects of ambient light from the measured spectrum and produce a corrected spectrum, wherein compensating for the effects of ambient light includes subtracting a spectrum of the white reference from the measured spectrum; and a cloud computing apparatus in communication with the processor including a database of reference spectra, wherein the cloud computing apparatus is configured to: compare the corrected spectrum to the database of reference spectra to produce a result; and provide at least one of a product recommendation or a diagnosis based at least in part on the result.

25. The apparatus of claim 24, further comprising a display configured to provide at least one of: the image of the surface, the spectrum of the surface, and at least one of the product recommendation or the diagnosis.

26. The apparatus of claim 24, wherein the cloud computing apparatus is further configured to implement a machine learning model to provide the product recommendation or the diagnosis, and wherein at least one of the product recommendation and the machine learning model is updated with user feedback.

27. The apparatus of claim 24, wherein the processor is further configured to determine a specular reflectance of the illuminated surface based at least in part on an intensity of a reflected light beam and an intensity of non-polarized light.

28. The apparatus of claim 27, wherein the surface is determined to be glossy if the intensity of the reflected light beam is greater than the intensity of the non-polarized light. ​

Images