System and method for skin analysis using electronic device

Abstract

The present application relates to systems and methods for skin analysis using an electronic device. A skin analysis device for attachment to an electronic device of a human user, the electronic device having at least one skin feature measurement device, the skin analysis device comprising a housing, the housing comprising a housing body, the housing configured to be detachably connected to the electronic device; and one or more skin feature measurement aids connected to the housing and configured to assist the one or more skin feature measurement devices in collecting a skin feature sample of a skin feature of the human user.

Description

[0001] This application is a divisional application of Chinese Patent Application 2017800310917 (PCT / CA2017 / 050503) entitled "System and Method for Skin Analysis Using Electronic Device" filed on April 21, 2017.

[0002] Cross Reference to Related Applications

[0003] This application claims priority to U.S. Provisional Application No. 62 / 438,835 filed on December 23, 2016, U.S. Provisional Application No. 62 / 344,287 filed on June 1, 2016, and U.S. Provisional Application No. 62 / 326,558 filed on April 22, 2016, the entire contents of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0004] The present invention relates to measurement and analysis of skin features, and in particular to a system and method for measuring skin features using a skin analysis device attached to a smartphone, and sharing the measurement results in order to provide recommended skin care and makeup based on the skin analysis. BACKGROUND

[0005] Skin care manufacturers create skin care products in order to assist users in maintaining healthy and beautiful skin. However, one of the biggest problems in the consumer skin care and makeup industry is the difficulty in assessing skin types, inability to quantitatively and objectively prove that skin care products are effective, and difficulty in matching makeup colors to a consumer's specific skin tone.

[0006] Various attempts exist to solve at least some of these challenges. However, the limitations and failures of these solutions are numerous. For example, existing solutions have one or more of the following limitations:

[0007] (a) Inaccurate measurements. For example, caused by technical limitations, unrealistic requirements by users, etc.

[0008] (b) Too limited measurement range. For example, measuring only one or two skin care features when two or more skin care features are needed in order to effectively recommend skin care or provide skin analysis.

[0009] (c) Cost. For example, most solutions are standalone devices that cannot take advantage of existing technology and high-end components. As a result, the solutions include all components required for various measurements, and all product designs required. This makes such solutions very expensive.

[0010] (d) Logical challenges in deploying the solution. Specialized hardware is difficult to deploy, especially when it is expensive. Hardware that is difficult to calibrate, maintain, or use is equally difficult to deploy in a manner that will be used and used accurately.

[0011] Accordingly, there is a need in the art for an improved method and system for skin analysis that can be performed using an electronic device such as a smartphone. SUMMARY

[0012] In a first embodiment of the present invention, a skin analysis device for attachment to an electronic device of a user having at least one skin feature measurement device is disclosed, the skin analysis device comprising:

[0013] a housing including a housing body configured to be detachably connected to the electronic device;

[0014] one or more passive skin feature measurement aids connected to the housing and configured to assist the one or more skin feature measurement devices in collecting skin feature samples of skin features of the user.

[0015] In an aspect of the first embodiment, the at least one skin feature measurement device is a camera and the one or more passive skin feature measurement aids include a lens disposed in front of the camera when the housing is connected to the electronic device.

[0016] In another aspect of the first embodiment, the at least one skin feature measurement device is a camera and the one or more passive skin feature measurement aids include a color calibrator assembly disposed in front of the camera when the housing is connected to the electronic device.

[0017] In another aspect of the first embodiment, the color calibrator assembly includes a sleeve through which the camera takes pictures, a color calibrator disposed on an inner surface of the sleeve, and a skin contact ring.

[0018] In another aspect of the first embodiment, the at least one skin feature measurement device is a camera and the one or more passive skin feature measurement aids include a light source assembly, wherein the light source assembly includes:

[0019] a light source board having one or more individually controllable light sources that illuminate the user when the camera takes an image of the user;

[0020] a diffuser to diffuse the one or more light sources passing therethrough, the light source board including one or more light source apertures thereon that allow the light sources to pass through without diffusion;

[0021] a set of first light sources disposed on the light source board such that the first light sources do not pass through the light source apertures and are diffused by the diffuser.

[0022] In another aspect of the first embodiment, the light source assembly further comprises a set of second light sources providing second light sources disposed on the light source plate such that the second light sources pass through the light source apertures without being diffused.

[0023] In another aspect of the first embodiment, the first set of light sources are light emitting diodes providing light emitting diode light and the second set of light sources are ultraviolet lights providing ultraviolet light.

[0024] In another aspect of the first embodiment, the skin analysis device further comprises:

[0025] one or more active skin feature measurement aids connected to the housing and configured to assist the one or more skin feature measurement devices in collecting skin feature samples of a skin feature of a user; and

[0026] a skin analysis device processor in communication with the electronic device and the one or more active skin feature measurement aids, the skin analysis device processor configured to:

[0027] receive a sampling signal from the electronic device; and

[0028] control the one or more active skin feature measurement aids to assist the one or more skin feature measurement devices in collecting the skin feature samples.

[0029] In another aspect of the first embodiment, the one or more active skin feature measurement aids comprise a light source assembly.

[0030] In another aspect of the first embodiment, the light source assembly further comprises a first light source and a second light source, wherein the first light source is a light emitting diode (LED) light source and the second light source is an ultraviolet (UV) light source.

[0031] In another aspect of the first embodiment, the skin analysis device further comprises a diffuser configured to be positioned between the light source plate and the user and configured to diffuse the first light source without diffusing the second light source.

[0032] In another aspect of the first embodiment, the skin analysis device processor further comprises a Bluetooth transceiver and receives the sampling signal from a Bluetooth transceiver of the electronic device.

[0033] In another aspect of the first embodiment, the skin analysis device further comprises:

[0034] a second skin feature measurement device connected to the housing and configured to collect second skin feature samples of a skin feature of a user; and

[0035] a skin analysis device processor in communication with the electronic device and the second skin feature measurement device, the skin analysis device processor configured to:

[0036] obtaining a second skin characteristic sample from a second skin characteristic measuring device; and

[0037] providing the second skin characteristic sample to the electronic device.

[0038] In another aspect of the first embodiment, the skin analysis device processor is further configured to:

[0039] receiving a sampling signal from the electronic device; and

[0040] controlling the second skin characteristic measuring device to collect the second skin characteristic sample.

[0041] In another aspect of the first embodiment, the second skin characteristic measuring device comprises a moisture sensor.

[0042] In another aspect of the first embodiment, the skin analysis device processor further comprises a Bluetooth transceiver, and the sampling signal is received from a Bluetooth transceiver of the electronic device.

[0043] In another aspect of the first embodiment, the housing further comprises a cylindrical sleeve assembly aperture configured to be disposed in front of a camera of the electronic device when the skin analysis device is attached to the electronic device.

[0044] In another aspect of the first embodiment, the cylindrical sleeve assembly aperture is further configured to receive a skin characteristic measuring aid.

[0045] In another aspect of the first embodiment, the skin analysis device further comprises an application installed on and running on the electronic device, the application configured to:

[0046] communicate with the skin analysis device and the electronic device to facilitate obtaining the skin characteristic sample from the at least one skin characteristic measuring device.

[0047] In a second embodiment of the present invention, a system for performing one or more skin characteristic analyses is disclosed, the system comprising:

[0048] a skin characteristic application running on an electronic device having a camera capable of taking images of a user, the skin characteristic application configured to:

[0049] control the camera to take a set of images of the user;

[0050] perform a set of skin care processes on the set of images to obtain a set of skin analysis measurements; and

[0051] communicate with a skin analysis device:

[0052] send an activation signal to a moisture sensor to initiate a moisture sensor reading; and

[0053] receive the moisture sensor reading from the moisture sensor.

[0054] A skin analysis device connected to the electronic device, comprising:

[0055] a processor configured to:

[0056] receive an activation signal from the skin feature application;

[0057] in response to the activation signal, obtain a moisture sensor reading from the moisture sensor; and

[0058] send the moisture sensor reading to the skin feature application;

[0059] a moisture sensor configured to:

[0060] send the moisture sensor reading to the processor.

[0061] In one aspect of the second embodiment, a skin care treatment includes a sunscreen treatment and a color match.

[0062] In another aspect of the second embodiment, a skin care treatment includes a line treatment, and the line treatment further includes:

[0063] converting the image to L*ab color;

[0064] eliminating hair;

[0065] applying a high pass filter;

[0066] converting the image to a black and white image;

[0067] identifying possible lines and wrinkles using a morphological skeleton;

[0068] applying a Hough transform; and

[0069] inverting the image so that lines in the image are white.

[0070] In another aspect of the second embodiment, a skin care treatment includes a pore treatment, and the pore treatment further includes:

[0071] converting the image to a grayscale image having a grayscale matrix;

[0072] calculating a mean value of the grayscale matrix:

[0073] subtracting each grayscale matrix element from the mean value to obtain a new grayscale matrix.

[0074] for each pixel of the new grayscale matrix having a pixel color:

[0075] considering a pixel window around the pixel;

[0076] counting the pixels in the pixel window having pixel color values within a specified range;

[0077] setting the pixel color to white if the count of pixels in the pixel window exceeds a first number, and increasing a white pixel counter by one, otherwise setting the pixel color to black; and

[0078] setting the white pixel counter as the pore score.

[0079] In another aspect of the second embodiment, the skin feature application is further configured to activate the ultraviolet light source when taking the set of images;

[0080] the set of images includes an image before the sunscreen is applied and one or more images after the sunscreen is applied; and

[0081] wherein the set of skincare treatments includes a sunscreen treatment, and the sunscreen treatment further includes:

[0082] calculating a blue score for the set of images;

[0083] comparing the blue score to a threshold blue score; and

[0084] indicating to reapply sunscreen if the blue score exceeds the threshold blue score.

[0085] In another aspect of the second embodiment, the set of images further includes an image before the sunscreen is applied, and the comparing further includes determining a difference between a blue score of the image before the sunscreen is applied and a blue score of the image after the sunscreen is applied, and the indicating further includes indicating whether the difference between the blue score of the image before the sunscreen is applied and the blue score of the image after the sunscreen is applied exceeds a reduction threshold.

[0086] In a third embodiment of the invention, a skin analysis device for attachment to an electronic device of a user is disclosed, the skin analysis device comprising:

[0087] a housing comprising a housing body configured to detachably connect to the electronic device; and a cylindrical sleeve assembly aperture configured to receive a cylindrical sleeve assembly and disposed in front of a camera of the electronic device when the skin analysis device is attached to the electronic device;

[0088] a lens attachedly inserted into the cylindrical sleeve assembly aperture closer to the camera in a radial direction than the first light source;

[0089] a cylindrical sleeve assembly attachedly inserted into the cylindrical sleeve assembly aperture, comprising:

[0090] a cylindrical sleeve;

[0091] a moisture sensor disposed on the cylindrical sleeve assembly such that the moisture sensor can measure a moisture quality of the user's surface when the skin analysis device is in a measurement mode, the moisture sensor in communication with the skin analysis device processor; and

[0092] a light source assembly to diffuse light from a first light source positioned thereon, the light source assembly disposed in front of the camera when the skin analysis device is connected to the electronic device, the light source assembly in communication with the skin analysis device processor;

[0093] the skin analysis device processor in communication with the electronic device, the moisture sensor, and the light source assembly, the skin analysis device processor configured to facilitate one or more skin analyses; and

[0094] a battery to provide power to the skin analysis device processor.

[0095] In a fourth embodiment of the present invention, there is a system for collecting, transmitting, and using skin characteristic samples from a group of users and obtained from the users' electronic devices, the system comprising:

[0096] a group of skin characteristic applications installed on each of a group of electronic devices, each skin characteristic application configured to:

[0097] facilitate collection of skin characteristic samples of a user from one or more skin characteristic measurement devices;

[0098] create a skin characteristic sample data structure from the skin characteristic samples; and

[0099] allow for a first skin analysis based on the skin characteristic sample data structure to produce a result.

[0100] In an aspect of another embodiment, the skin characteristic application is further configured to: display the result of the first skin analysis.

[0101] In another aspect of the fourth embodiment, the skin characteristic application is further configured to:

[0102] execute a product recommendation algorithm based on the first skin analysis; and

[0103] recommend one or more recommended products according to the product recommendation algorithm.

[0104] In another aspect of the fourth embodiment, the skin characteristic application is further configured to:

[0105] provide one or more screens to purchase the recommended products.

[0106] In another aspect of the fourth embodiment, the skin characteristic application is further configured to:

[0107] transmitting the skin characteristic sample data structure to a skin analysis server; and

[0108] wherein the system further comprises a skin analysis server configured to:

[0109] receiving the skin characteristic sample data structure from a set of electronic devices; and

[0110] storing the skin characteristic sample data structure in a storage device.

[0111] In another aspect of the fourth embodiment, the skin analysis server is further configured to:

[0112] computing one or more relative scores from the skin characteristic sample; and

[0113] transmitting the one or more relative scores to the user's electronic device.

[0114] In another aspect of the fourth embodiment, the skin characteristic application is further configured to:

[0115] transmitting the skin characteristic sample data structure to a product owner server of a product owner; and

[0116] wherein the system further comprises a product owner server configured to:

[0117] receiving the skin characteristic sample data structure from a set of electronic devices; and

[0118] storing the skin characteristic sample data structure in a storage device.

[0119] In another aspect of the fourth embodiment, the skin characteristic sample data structure further comprises a user referrer, and the skin characteristic application is further configured to communicate if the product owner is the user referrer.

[0120] In another aspect of the fourth embodiment, the user referrer is established upon installation of the skin characteristic application on the user's electronic device, or is established based on a skin analysis device identifier of a skin analysis device attached to the user's electronic device.

[0121] In another aspect of the fourth embodiment, the product owner server is further configured to:

[0122] obtaining a first skin analysis;

[0123] executing a product recommendation algorithm based on the first skin analysis;

[0124] recommending one or more recommended products according to the product recommendation algorithm; and

[0125] providing the one or more recommended products to the user's electronic device.

[0126] In a fifth embodiment of the invention, there is a method for matching a user's face color to a suggested product color of a hypothetical lighting according to a total color option, the method comprising:

[0127] obtaining a set of images of a user's face, wherein each image of the set of images includes a user portion and a color calibration portion, wherein the color calibration portion includes at least one color quadrant of a color calibrator having a known color;

[0128] for each image of the set of images:

[0129] performing a color correction transform on the user portion based on the color calibration portion;

[0130] removing outlier pixels from the user portion;

[0131] determining an average color of the user portion;

[0132] averaging each average color of the user portion to derive a measured color of the user;

[0133] comparing the measured color to the total color option to derive a suggested product color;

[0134] communicating the suggested product color.

[0135] In one aspect of the fifth embodiment, the obtaining is from a camera of an electronic device having the camera with a skin analysis device attached thereto, the color calibrator disposed on the skin analysis device.

[0136] In another aspect of the fifth embodiment, the comparing further comprises:

[0137] deriving a color difference between the measured color and a set of colors of the total color option; and

[0138] selecting the color having the least color difference as the suggested product color.

[0139] In another aspect of the fifth embodiment, the deriving further comprises applying a hue filter to the colors of the total color option to obtain the set of colors.

[0140] In another aspect of the fifth embodiment, the method further comprises:

[0141] designating one or more alternative lighting or one or more ambiances;

[0142] determining whether the one or more alternative lighting or the one or more ambiances would change the suggested product color, if so:

[0143] deriving another situational suggested product color; and

[0144] Transmitting contextually suggested product colors.

[0145] In a sixth embodiment of the invention, there is a system for measuring the elasticity of a user's skin, the user's skin having an original position when the user's skin is at rest, the system comprising:

[0146] an electronic device proximate to the user's skin such that the user's skin moves in response to a vibrating motor, the electronic device comprising:

[0147] a camera capable of taking a video and controllable through a skin feature application;

[0148] a vibrating motor that vibrates the electronic device and is controllable through the skin feature application;

[0149] a skin feature application configured to:

[0150] activate the vibrating motor when the camera is positioned to take a video of the user's skin;

[0151] cause the camera to take the video;

[0152] deactivate the vibrating motor after a period of time; and

[0153] process the video to calculate an elasticity score.

[0154] In one aspect of the sixth embodiment, the skin feature application is further configured to process the video by measuring an amount of time for the user's skin to return to the original position.

[0155] In another aspect of the sixth embodiment, the skin feature application is further configured to activate a set of pulses of the vibrating motor and process an amount of time for the user's skin to return to the original position.

[0156] In another aspect of the sixth embodiment, the system further comprises a skin analysis device for attachment to the electronic device, the skin analysis device comprising a sleeve having a lens disposed on an inner surface of the sleeve through which the camera takes the video; and a skin contact ring that is in contact with the user's skin when the camera is positioned to take the video of the user's skin.

[0157] In a seventh embodiment of the invention, there is a color calibrator assembly for ensuring that a user image taken by a camera of an electronic device having a field of view has a known color, the color calibrator assembly comprising:

[0158] a sleeve that attaches to the electronic device and is disposed on top of the camera, the camera taking a photo of the field of view axially through the sleeve;

[0159] a color calibrator disposed on an inner surface of the sleeve and occluding a portion of the field of view.

[0160] a skin contact ring configured to contact a user at a user contact point when an image is taken.

[0161] In one aspect of the seventh embodiment, the color calibrator assembly of claim 46, wherein the color calibrator further comprises a color ring having one or more color quadrants, each of the one or more color quadrants having a known color.

[0162] In another aspect of the seventh embodiment, the color calibrator assembly further comprises a color calibrator assembly accessory that detachably attaches the color calibrator assembly to the electronic device at an attachment point.

[0163] In another aspect of the seventh embodiment, the sleeve, the attachment point, and the user contact point all block light from entering the color calibrator assembly.

[0164] In another aspect of the seventh embodiment, the color calibrator assembly further comprises a light source disposed axially along the sleeve and configured to add light of a known color to the field of view when an image is taken.

[0165] In another aspect of the seventh embodiment, the color calibrator assembly accessory is configured to cover a flash of the electronic device and prevent light from the flash of the electronic device from entering the image.

[0166] In an eighth embodiment of the present invention, a system for calibrating a moisture sensor is disclosed, where a user can depress the moisture sensor into contact with the user's skin with an unknown pressure when reading a moisture sensor reading, where the pressure affects the moisture sensor reading, the system comprising:

[0167] an electronic device comprising:

[0168] a camera having a variable focal length, the camera having a focal length when taking an image; and

[0169] a skin feature application configured to:

[0170] facilitate taking a first photo of a test subject under a high pressure and recording i) a first focal length when the first photo is taken and ii) a first moisture sensor reading from a moisture sensor;

[0171] facilitate taking a second photo of the test subject under a low pressure and recording i) a second focal length when the first photo is taken and ii) a second moisture sensor reading from the moisture sensor;

[0172] calculating a pressure-based moisture adjustment slope;

[0173] applying the pressure-based moisture adjustment factor to a future moisture sensor reading obtained at a future focal distance to obtain an adjusted moisture sensor reading; and

[0174] a moisture sensor configured to:

[0175] provide a moisture sensor reading to a skin feature application.

[0176] In one aspect of the eighth embodiment, the skin feature application is further configured to:

[0177] determine the pressure-based moisture adjustment slope having a first (x / y) point (first focal distance / first moisture sensor reading) and a second (x / y) point (second focal distance / second moisture sensor reading); and

[0178] quantify an average focal distance of the first focal distance and the second focal distance.

[0179] In another aspect of the eighth embodiment, the application further comprises:

[0180] solving a slope point form equation of the form Y - Y1 = m(X - X1), where

[0181] Y is the adjusted moisture sensor reading;

[0182] Y1 is the future moisture sensor reading;

[0183] m is the pressure-based moisture adjustment slope;

[0184] X is the average focal distance; and

[0185] X1 is the future focal distance.

[0186] In one aspect of the eighth embodiment, the skin feature application is further configured to:

[0187] prompt a user to select a first focal point near the middle of a display of an electronic device that is previewing a first photograph; and

[0188] accept an input of the user selecting the focal point as a trigger to take the photograph.

[0189] In one aspect of the eighth embodiment, the skin feature application is further configured to:

[0190] request a user to select a first focal point near the middle of a display of an electronic device that is previewing a first photograph; and

[0191] receive an input of the user selecting the focal point as a trigger to take the photograph.

[0192] In a ninth embodiment of the invention, a system for safe use of a harshness device that assists skin feature measurement devices in collecting skin feature samples of a user but can be dangerous to the user if used in an incorrect manner is disclosed, the system comprising:

[0193] An electronic device having a skin feature measurement device for taking images and a skin feature application configured to:

[0194] If a safe use indicator signal is received, sending an activation signal to the harshness device to allow the harshness device to be turned on; and

[0195] Activating a camera to take an image;

[0196] A skin analysis device comprising:

[0197] A harshness device connected to the skin analysis device and configured to:

[0198] Assist one or more skin feature measurement devices in collecting skin feature samples of a user's skin features in response to an activation signal from the skin feature application;

[0199] A safe use indicator device configured to:

[0200] Send a safe use indicator signal to a processor; and

[0201] A processor configured to:

[0202] Obtain a safe use indicator device signal; and

[0203] Send the safe use indicator device signal to the skin feature application.

[0204] In an aspect of the ninth embodiment, the harshness device is an ultraviolet light and the safe use indicator is a moisture sensor. BRIEF DESCRIPTION OF DRAWINGS

[0205] The invention is illustrated in the accompanying drawings, which are schematic and not limiting, wherein identical or corresponding parts are denoted by the same reference signs, and wherein:

[0206] Figure 1 Aspects of exemplary electronic devices and skin analysis devices and related systems according to embodiments of the invention are illustrated;

[0207] Figure 2 And Figure 3 A front view of an exemplary skin analysis device according to embodiments of the invention is illustrated;

[0208] Figures 4a to 4b and Figure 5 Aspects of a sleeve assembly of a skin analysis device according to embodiments of the application are shown;

[0209] Figure 6 A skin analysis device circuit board of a skin analysis device according to embodiments of the application is shown;

[0210] Figure 7 and Figure 8 A rear view of an exemplary skin analysis device according to embodiments of the application is shown

[0211] Figure 9 A method for assembling an exemplary skin analysis device according to an aspect of the application is shown;

[0212] Figure 10 A method of using a skin analysis device according to an aspect of the application is shown;

[0213] Figure 11 A method of performing a skin care measurement according to an aspect of the application is shown;

[0214] Figures 12a to 12m A screenshot of an application for an electronic device according to an aspect of the application is shown;

[0215] Figures 13a to 13b A method for color matching for different lighting according to an aspect of the application is shown;

[0216] Figures 14a to 14d A screenshot of an application for an electronic device according to an aspect of the application is shown;

[0217] Figure 15 A screenshot of an application for an electronic device according to an aspect of the application is shown; and

[0218] Figures 16a to 16f A screenshot of an application for an electronic device according to an aspect of the application is shown, with a large original image and resulting processed image in the screenshot. DETAILED DESCRIPTION

[0219] In general, the present application is a skin analysis device that attaches to a mobile device and performs one or more skin analysis actions, such as taking a picture of a person's face to assess moisture, pores, etc. The images and other measurements are stored, compared to other samples, and used to recommend products to assist with skin care.

[0220] As used and further described herein, the following terms have the following meanings:

[0221] (a) Back: Side of the electronic device that is typically facing away from the user when the electronic device is in use by the user, and which can have a rear-facing camera.

[0222] (b) Electronic device: A device to which a skin analysis device can be attached that has a camera, which can preferably be mobile (such as mobile phones and tablets), exemplary electronic devices include smartphones, tablets, digital cameras, personal computers, televisions, etc.

[0223] (c) Front: Side of the electronic device that is typically facing the user when the electronic device is in use by the user, and which can have a front-facing camera.

[0224] (d) Image: Reference to an image herein, unless otherwise specified, refers to a digital image that can be represented by digital data and can be manipulated and processed by electronic devices and computers (such as application 18, SAS 200, etc.).

[0225] (e) Skin feature: One or more features of the skin (including hair), such as color, moisture, oiliness, elasticity, etc.

[0226] (f) Skin feature measurement aid: A component that assists a skin feature measurement device in making a skin care measurement. These can generally be considered passive (i.e., not requiring control by the electronic device or skin analysis device, such as lens 34, contact ring, sleeve 24, etc.) or active (requiring control by the electronic device, such as light source board and vibration motor, etc.).

[0227] (g) Skin feature measurement device: A component that makes a skin analysis action on its own control or under the control of another component. These can be on or separate from electronic device 10, skin analysis device 20. Each can take part in or conduct multiple skin analysis actions, skin analysis, skin analysis calibration, and skin analysis measurements.

[0228] (h) Skin analysis / resolution: One or more analyses of one or more skin features.

[0229] (i) Skin analysis action: An action that results in or causes certain parts of a skin analysis to occur, such as a skin analysis measurement, exemplary skin analysis actions include taking a photograph, reading a moisture sensor, testing elasticity through an image or recording, etc.

[0230] (j) Skin analysis calibration: Calibration of one or more aspects of the skin analysis device and / or electronic device to allow for accurate skin analysis, exemplary skin analysis calibrations include light standardization of exposure and color temperature (both of which can be done using a transform function to color correct).

[0231] (k) Skin analysis device: a device according to aspects of the application, which is detachably connected to the electronic device 10.

[0232] (l) Skin analysis measurement: a measurement of one or more skin features, including obtaining a moisture reading, a texture / wrinkle / pore image, an elasticity reading, a skin tone, etc., any of which can be expressed as a "score" - an absolute score, a relative score, or an average score, i.e. a pore score, a relative texture score, a color, etc.

[0233] (m) Skin analysis processing: processing one or skin analysis measurements, e.g. by performing image processing on an image.

[0234] (n) Skin analysis recommendation: recommending a suitable skin care product to a user using one or more results of one or more skin analysis and one or more characteristics of one or more skin care products.

[0235] (o) Skin care product: a product that contributes to one or more skin care properties, e.g. a moisturizer, an anti-wrinkle cream, a cosmetic (e.g. a foundation and a blush), etc.

[0236] (p) Skin care product manufacturer: a maker, manufacturer, distributor, brand, and brand owner of a skin care product.

[0237] (q) Skin care product characteristic: a property of a particular skin product, which can include a color of the product, a moisturizing ability, a texture reduction ability, etc.

[0238] (r) Skin care update: these can include information related to skin care that can come from external sources. For example, weather (clouds, sun, high UV, snow) can come from a weather source (not shown, but known to those skilled in the art).

[0239] (s) Skin feature: a feature of the skin or a body part, such as a pore, a spot, a sensitivity (which can be shown using polarized or other light spectrum to show blood vessels close to the skin surface, which can result in an "redder" image, a texture, an elasticity, a moisture, an oil, an acne, and a skin tone.

[0240] (t) User / human user / subject, person: a person who uses the skin analysis device and / or a subject of a skin feature sample, as the case can be.

[0241] (u) User referral source: an entity that causes a user to start using the system 1 - e.g. by encouraging a download of an application (which can be tracked using application download technology) or by distributing a skin analysis device (which can be tracked using a skin analysis device identifier).

[0242] Figure 1Aspects of a system 1 are shown with an exemplary electronic device 10 and skin analysis device 20a capable of connecting with a skin analysis server 200 ("SAS"), one or more skin care product manufacturers / owners ("product owners") 300, and one or more e-commerce vendors / sites ("vendors") 400 via a network 100.

[0243] System

[0244] The system 1 can allow a user to measure and obtain / capture, store, transmit, track, and operate or use various skin characteristics and samples.

[0245] Measuring and obtaining can mean one or more data collections (per skin characteristic sample) using one or more skin characteristic measurement devices, with or without assistance from one or more skin characteristic aids. The electronic device 10 can have one or more skin characteristic measurement devices (e.g., camera 12) and one or more skin characteristic aids (e.g., vibration motor). The skin analysis device 20 can also have one or more skin characteristic measurement devices (e.g., moisture sensor 36) and one or more skin characteristic aids (e.g., lens 34 or

[0246] Storing can mean storing one or more skin characteristic samples locally or remotely for a particular user or a larger group of users.

[0247] Tracking can allow a user to compare themselves to their own previous skin characteristic samples (i.e., "my skin is more hydrated than it was a week ago") or to other people's skin characteristic samples (i.e., "my skin is more oily than other people who can be comparable to me").

[0248] Operating can mean a user buying a foundation that matches their skin color so that they will see the shine they will see, buying and using a recommended moisturizer, or going to a doctor to check for spots on their skin.

[0249] All of this can be described more fully herein. As described herein, all of this can occur between and among devices connected via one or more different types of networks 100.

[0250] Skin analysis server 200

[0251] As described herein, the SAS 200 can be a server that stores and processes skin characteristic measurements or samples. As known to those skilled in the art, the SAS 200 can be any combination of a web server, an application server, and a database server. Each such server can include typical server components including a processor, volatile memory storage devices, and non-volatile memory storage devices, and software instructions executable thereon. The SAS 200 can be the central communication point for the application 18 to perform the functions described herein including exchanging skin analysis measurement samples, product recommendations, e-commerce capabilities, etc. Of course, the skin characteristic application 18 can also perform these functions separately or in combination with the SAS 200.

[0252] The SAS 200 can include a database server that receives all skin characteristic samples from all users and stores them into a user profile for each registered and guest user. These can be received from one or more electronic devices 10, but the application 18 can be configured to only store skin characteristic samples locally (although this can exclude some result information based on demographic and population comparisons).

[0253] The SAS 200 (and / or the application 18) can share user profiles (and any skin characteristic samples received therefrom) with user referral sources, e.g., when the user profile includes a user referral source. This can be accomplished, for example, by providing the user profile to a server of a product owner.

[0254] The SAS 200 can provide various analysis functions as described herein (e.g., calculating a historical score for a user or a comparative histogram for comparison with peers), and can provide various display functions as described herein (e.g., providing a website that can present various analyses, providing links or functional links to other websites for accessing and displaying such results, etc.

[0255] Product owner 300

[0256] The product owner 300 can be an entity as defined above with respect to interest in skin care products, and can also have one or more product owner servers, as known to those skilled in the art, including web servers, application servers, and database servers. Each such server can include typical server components including a processor, volatile memory storage devices, and non-volatile memory storage devices, and software instructions executable thereon. The product owner 300 can be a communication point for the application program 18 (directly or through the SAS 200) for skin analysis measurement samples (such as those obtained by users of a skin analysis device 20 provided by such product owner 300) and for storing and executing product recommendation algorithms. For example, the SAS 200 can store and own one or more generic product recommendation algorithms for each product recommendation type, and the product owner can own and implement their own proprietary product recommendation algorithms (e.g., the product owner 300 receives the required data to execute the product recommendation algorithm, and returns the recommended product). The product owner 300 can also directly provide e-commerce services, can suggest vendors such as Amazon TM , etc. (either alone or with the recommended product), or can be independent of how the user purchases the recommended product.

[0257] E-commerce provider 400

[0258] The vendor 400 can provide one or more e-commerce websites or screens (e.g., separate from or embedded in screens in the application program 18) that facilitate business or commercial transactions involving the transmission of information over the network 100 (e.g., the Internet). Types of e-commerce websites include, but are not limited to: retail websites, auction websites, and business-to-business websites. Exemplary vendors 400 that can facilitate the purchase of skin care products can include Amazon TM , eBay TM , and Overstock TM . Of course, the product owner 300 can have their own e-commerce site as part of their general website, or the SAS 200 can be such a vendor.

[0259] Electronic device

[0260] The electronic device 10 can be, for example, an iPhone TMsuch as an iPhone® 6 or 6s. The electronic device 10 also has components such as a camera 12, and other components that are common to such devices, such as a flash 8 for the camera 12, power and volume buttons, a motor to cause vibration ("vibration motor", inside the electronic device 10), a screen, a processor, a storage device / memory, etc. Some such components can have visual or physical aspects that can be visible or present on the outer surface of the electronic device 10 ("visible components"). Other components can be inside the electronic device 10, such as a GPS transceiver ("internal components"), but can require that the outer surface of the electronic device 10 function normally (e.g. not block GPS or other wireless signals).

[0261] The electronic device 10 can include expensive and powerful components (including but not limited to a processor, a storage device / memory, the camera 12, etc.). However, without other skin feature measurement devices or skin feature measurement aids, these components can not be suitable as skin feature measurement devices or skin feature measurement aids. For example, the camera 12 can have good resolution, but can not have sufficient characteristics (e.g. optical zoom or magnification) to take images suitable for skin features and skin feature samples. Thus, as described herein, skin feature measurement aids can be employed (on the skin analysis device 20 and / or the electronic device 10), and other skin feature measurement devices can be employed.

[0262] As is known, the electronic device 10 can have software (e.g. "applications") located thereon that can be obtained from an "application store" and installed. The application described in accordance with embodiments of the present application can be accessed on the electronic device 10 to perform functions as described herein. The application can also access storage located on the electronic device 10, such as storing data as described herein. The application can also perform communications between the electronic device 10 and the skin analysis device 20 and the network 100 (and thus with the SAS 200, the product owner 300 and the supplier 400).

[0263] The electronic device 10 can also have an operating system that provides access to various application programming interfaces ("APIs"). Such APIs allow applications on the electronic device 10 to "call" the APIs, and thus access various functions of the electronic device 10 (e.g. the camera 12, control the vibration motor, turn on an electronic device light source such as the flash 8, or control its operation when performing a skincare action, etc.).

[0264] The electronic device 10 can have one or more applications 18 (also referred to as skin feature applications) installed thereon. The applications 18 can perform the various functions mentioned herein, and can be the primary means by which a human user interacts with the electronic device 10 (in terms of accessing the functions described herein) and the skin analysis device 20. The applications 18 can access various features and components of the electronic device 10. In particular, the applications 18 can receive input from the user and provide activation signals to the skin analysis device 20 (e.g., to read moisture sensor readings, to turn on a light source, etc.).

[0265] The electronic device 10 can have an owner / user, and can have one or more access users.

[0266] Skin analysis device 20a

[0267] The example skin analysis device 20a can be one embodiment of the skin analysis device 20. The skin analysis device 20a can include an example housing 22a and a sleeve 24a, both of which can include additional components as described herein. The sleeve 24a can be introduced and removably connected to the housing 22a, for example by moving the sleeve 24a along the attachment line 6 to introduce them together.

[0268] The skin analysis device 20a can be introduced and removably attached to the electronic device 10, for example by snapping the skin analysis device 20a along the securing line 2 / 4 onto the back of the electronic device 10. It should be appreciated that the shape of the skin analysis device 20, and in particular the shape of the housing 30, can be varied so as to allow attachment to various electronic devices 20 (e.g., iPhones TM , iPads TM , various Android TM phones, personal computers, etc.) in various ways (e.g., by snapping snugly to protrusions or edges of the electronic device 10 as part of the housing 30, thereby allowing the electronic device 10 to slide into the electronic device aperture 48). The shape of the housing 22 or other aspects of the skin analysis device 20 can be readily varied as needed to accommodate attachment to various electronic devices 10.

[0269] The housing 22 and other aspects of the skin analysis device 20 can also be varied as needed so as not to block or impair the functionality of the electronic device 10. By way of example, visible components can include volume controls, power or wake-up buttons, the camera 12, the flash 8, etc., and can be exposed through the component aperture 46 when attached to the electronic device 10 if these components have been covered or otherwise affected by the skin analysis device 20. It should also be appreciated that the shape of the housing 22 and other aspects of the skin analysis device 20 can be readily varied to accommodate exposure of various visible components or to facilitate operation of internal components for various electronic devices 10.

[0270] In one embodiment, as shown in Figure 1 , the sleeve 24a can include several components of the skin analysis device 20, such as a lens, a color calibrator, a light source board, a diffuser, a moisture sensor, etc. (all of which are mostly as described herein but not visible in Figure 1 ), which can be detachable from the housing 22a. In other embodiments, various components of the skin analysis device 20 can be separate from other components, such as a standalone color calibrator (e.g., having one or more sleeves 24, a calibration ring 84, optionally having one or more light sources 72 / 74, optionally having a diffuser 38), and a contact ring 28 including the standalone color calibrator 80, can be created as described herein.

[0271] As shown in Figure 1 , the skin analysis device 20 can be located at the upper end 16 of the back 14 of the electronic device 10, and thus can cover the top of the back of the electronic device 10, such as no more than one-third of the vertical height of the electronic device 10 (although the size of this upper portion can be changed as desired). Again as shown in Figure 1 , the skin analysis device 20 can be attached to the electronic device 10 such that the front of the electronic device 10 is rarely blocked.

[0272] Figure 2 and Figure 3 shows a front view of an exemplary skin analysis device 20 according to an embodiment of the present application, Figure 7 and 8 shows a rear view thereof. The skin analysis device 20 generally includes a housing 50 having a housing body 52, a cylindrical sleeve assembly aperture 54, a board housing 96, and an electronic device housing 98, which further includes a top housing 94 and a side edge 92, the board housing and the electronic device housing 98 are respectively operatively connected with a cover 140 to house a PCB 120 and an electronic device 10, a lens 34, a cylindrical sleeve assembly 60 including a cylindrical sleeve 24, a moisture sensor 36a having a moisture sensor lead 36b, a light source board 70 including a first light source 72, a second light source 74, and a light source lead 76, a diffuser 38 including a light source aperture 42 and a camera lens aperture 40, and a color calibration ring 80 including an adhesive strip 86, a calibration ring 84 having one or more color quadrants 82, a skin analysis device circuit board (PCB) 120 processor (which can also include a Bluetooth TM transceiver) 122, a connector 126, a charging indicator 128, a reset button 124, and a battery 130 (not shown, on the reverse side of the PCB 120), and the cover 140 including a reset aperture 144 and a charging aperture 142.

[0273] Housing

[0274] The housing 50 includes a housing body 52 configured to be removably connected to the electronic device 10; and a cylindrical sleeve assembly aperture 54 configured to receive the cylindrical sleeve assembly 60 and be disposed in front of the camera 12 of the electronic device 10 when the skin analysis device 20 is attached to the electronic device 10. When the attached housing or other portion of the skin analysis device 20 can cover the flash 8, such that any light from the light source 70 that is part of the user's skin image.

[0275] The cylindrical sleeve assembly aperture 54, also referred to as the sleeve housing 54, can also include a base plate 102, a lens plate 104, and a wire aperture 106. The base plate 102 can be proximate to a surface of the electronic device 10 and optionally form a portion of the electronic device housing 98. The base plate 102 can be a surface that the lens 34 and / or the sleeve assembly 60 adhere and / or rest against when inserted into the sleeve assembly aperture 50. The lens plate 104 can be a portion of the base plate 102 that conforms to the shape of the lens 34 to facilitate the lens being placed in position when the skin analysis device 20 is attached to the electronic device 10. The wire aperture 106 can be an opening on the sleeve housing 50 configured to allow electrical connection from one or more of the moisture sensor 36a (e.g., moisture sensor lead 36b) and the light source assembly (e.g., lead 76) to the PCB 120.

[0276] The sleeve housing 54 can also include a contact surface 28 that can sufficiently contact a subject when analyzing skin. The contact surface 28 can be a material that a user finds comfortable, such as plastic or rubber, and can be integral to the housing 50 or can be a separate component that can be connected thereto.

[0277] The sleeve housing 54 can also include one or more lips or cradles 62. The cradles 62 can be a portion of the sleeve housing 24, optionally cylindrical, to which the moisture sensor 36 and / or color calibrator are attached. The lips 62 can be radially inward from the contact surface 28 such that the contact surface substantially encloses the lips 62.

[0278] The electronic device 10 can be slid into the electronic device housing 98 such that the electronic device 10 is in contact with the cover 140 and held in place by the cover 140, the top housing 94, and the side edges 92. This connection can be sufficiently tight, for example due to the relative dimensions of the electronic device housing 98 and / or the material properties of the housing 50 and the electronic device 10, that a user is unlikely to be able to attempt to remove the skin analysis device 20 from the electronic device 10.

[0279] The housing has a front surface 22 on which an advertisement or other information can be printed, affixed, or otherwise provided.

[0280] Cover

[0281] The cover 140 can be slidably inserted or placed within the plate housing 96 to protect the PCB from the external environment of the skin analysis device 20. The cover 140 can be shaped to fit the plate housing 96 and otherwise cooperate with the internal layout of the PCB 120 (thus the reset hole 144 and the charging hole 142 allow access to the reset button 124 and the charging button 128, respectively). The cover 140 can form part of the electronics device housing 98 when inserted.

[0282] Lens

[0283] The skin analysis device 20 also includes a lens 34 that is attached inserted into the cylindrical sleeve assembly hole 54 radially closer to the camera than the first light source. The working distance of the lens 34 to the skin / surface can be 16 mm and the working distance of the camera 12 lens can be 2 mm. The lens 34 can be glass coated. There can also be one or more polarizers (which can be thin and colored sheets that polarize the one or more light sources passing through the polarizer and window to assist in the sensitivity analysis, and which can be located between the lens 34 and the skin). The lens 34 can be a magnifying lens with a magnification factor appropriate for the skin surface being imaged (e.g., a 30X lens 34 for skin analysis and another magnification for hair analysis).

[0284] Sleeve assembly

[0285] The cylindrical sleeve assembly 60 can be attached inserted into the cylindrical sleeve assembly hole 50 and includes a cylindrical sleeve 24; a moisture sensor 36 disposed on or in the cylindrical sleeve assembly 60 and optionally located axially distal to the sleeve of the camera, such that the moisture sensor 36 can measure the moisture mass of the surface of the human user when the skin analysis device 20 is in the measurement mode; and a light diffuser assembly that diffuses light from the first light source located thereon, disposed in front of the camera 12 when the skin analysis device is attached to the electronic device, located axially proximal to the sleeve of the camera 12.

[0286] Cylindrical Sleeve

[0287] The sleeve 24 can be a cylinder sized to be insertable into the housing 50. The sleeve 24 can rest on the base plate 102 when the sleeve 24 is attached to an interior that is substantially black. The sleeve 24 can have an inner surface and an outer surface. The inner surface of the sleeve 24 can be light absorbing, e.g., black, so that the properties of the light directed toward the subject (e.g., the skin of the user) are known.

[0288] Moisture Sensor

[0289] Moisture sensor 36 can include sensor 36a and sensor lead 36b. Moisture sensor 36 can measure the moisture content of the skin and provide such measurements to skin analysis device processor 122. Moisture sensor 36 can be capacitive or resistive. In one embodiment, moisture sensor 36 is capacitive and includes two circular / cylindrical electrodes, tracking a thickness of 0.3 mm to 1 mm, with a gap of 0.1 mm to 1.5 mm therebetween. Moisture sensor 36 can be disposed and attached to lip 62 of sleeve 24, and can be shaped so that they do not interfere with camera 12 (and thus can be located radially outward of the interior / inner surface of sleeve 24 through which camera 12 can take images.

[0290] In testing, it was found that a moisture sensor with a 0.9 mm track and a 0.1 mm gap produced acceptable, reliable, and repeatable capacitance values for various readings (facial skin, skin on an arm, etc.) and a good range of capacitance such that several moisture gradients or degrees could be attributed to the skin or surface of a subject.

[0291] Moisture sensor 36 can be in communication with the skin analysis device processor via moisture sensor lead 36b, which can travel along sleeve bore 54, through wire bore 106, and connect to PCB 120.

[0292] In use, skin analysis device 20, particularly moisture sensor 36 and contact ring 28, can contact a user’s face or other surface. The amount of pressure applied to skin analysis device 20 to contact a user’s face or be pressed into contact therewith can affect the readings of moisture sensor 36 in a measurable and predictable manner. However, it can be difficult to determine the amount of pressure with which a user is pushing during a given use. To address this, a sensor focus calibration can be performed. This can involve asking the user to push the device with a strong pressure and allowing auto-focus to determine a first lens movement distance at which a first moisture reading is taken, then asking the user to push the skin analysis device gently with a light pressure and allowing auto-focus to determine a second lens movement distance at which a second moisture reading is taken. Application 18 can then interpolate between these values and calculate a best-fit equation of correlation with focus distance and moisture reading range as described herein. Then, each future time a moisture reading can be taken, possibly using an unknown pressure, the future focus distance can be captured, compared to the first and second lens movement distances, to determine a moisture factor value that can be applied to the moisture reading. Of course, this approach can be limited to camera 12 being a camera 12 with a variable focus.

[0293] Light source board

[0294] The light source board 70 can include a first light source 72, a second light source 74, and a light source lead 76. The light source board 70 can be in communication with the skin analysis device processor, for example, through the light source lead 76. The light source board 70 can be combined with the diffuser 38, referred to as a light source assembly.

[0295] First light source: The first light source can be light emitting diode (LED) light, for example, through Vishay Semiconductors TM VLMU3100 (Power SMD LED PLCC-2), which can have known light characteristics (e.g., luminous intensity, luminous intensity vs. angular displacement, chromaticity, etc.), which can be published in technical literature related thereto. The first light source being LED light can mean that such light needs to be diffused so as not to create a white spot on the captured image. As such, each LED light can be directly behind the diffuser 38, and its light can not pass through the light source aperture 42. Some LED lights can be configured with one or more polarizers to be polarized.

[0296] Second light source: The second light source can be ultraviolet light (UV), for example, through Vishay Semiconductors TM VLMU1610-365-135, which can have known light characteristics (e.g., luminous intensity, luminous intensity vs. angular displacement, chromaticity, etc.), which can be published in technical literature related thereto. The second light source being UV light can mean that such light does not need to be diffused when capturing an image. As such, each UV light can be directly behind the light source aperture 42, such that the UV light passes through such light source aperture 42. UV lights can be considered to be ill-affected devices, as they can be dangerous if used improperly (e.g., pointed at a user’s eyes for a long time). Thus, additional precautions can be taken in controlling the activation of the UV light, as described herein.

[0297] The light source board 70 can specifically have multiple lights from multiple light sources, which can be individually controllable and mounted so as to vary the angle of light that illuminates the skin surface. Various angles can illuminate the skin to show specific textures in 3D-type effects. This can aid one or more skin analyses, for example, by providing a measure of the depth of lines or wrinkles.

[0298] In use, because the spectrum of light of one or more light sources (e.g., the first and second light sources, or any other light sources that can be added) is known, the flash 8 can be blocked or disabled (e.g., by the application 18), the contact ring 28 can be in close proximity to the user's face (forming a user contact point in which at least a portion of the skin analysis device 10 and / or the electronic device 10 is proximate to or in contact with the user's skin), and the sleeve 24 blocks external light, the spectrum of light applied to the user's skin can be known. Optionally, in combination with the color calibrator 80, this can allow one or more color matching functions as described herein.

[0299] It can be desirable to use the light source board 70 (in combination with other components described herein to block other light sources) rather than the flash 8, not only because the spectrum of light is known and consistent (possibly more consistent than from the flash 8) but also because shadows and other anomalies that can cause various processing difficulties can be eliminated.

[0300] Of course, the light source board can have any number of light sources, and various other light sources can be used. For example, infrared light can be used, which can measure skin temperature and thus can be used for different dermatological assessments.

[0301] Diffuser

[0302] The diffuser 38 can be a component shaped and sized to fit inside the sleeve 24, and thus can be circular or cylindrical. The diffuser can be made of any material of any color, so long as it has the desired diffusing properties required based on the light source to be diffused. In one example, such a diffuser 38 can be white plastic. The diffuser can include one or more light source apertures 42 arranged to allow the desired light to pass from the light source board, and the camera lens aperture 40 to allow the camera 12 to work with the lens 34 to take an image.

[0303] Colour calibrator

[0304] The color calibrator 80 can be an example passive skin feature measurement aid. The color calibrator 80 can be configured so that when the camera 12 takes an image, the color calibrator 80 or at least a portion of the calibration ring 84 having at least one color quadrant 82 can be a portion of the image, e.g., a radial portion of the field of view of the camera 12.

[0305] The color calibrator 80 can include an adhesive strip 86 that can assist in adhering the color calibrator 80 to the skin analysis device 20, for example by folding the adhesive strip 86, such that it is axially inserted into the sleeve hole 54 and the calibration ring 84 is at least partially in the field of view of the camera 12 (e.g., as a radially outer portion of the field of view - thus creating a human user portion and a color calibration portion when an image of the human user is taken). Of course, the color calibrator 80 can have a central hole 88 that is sized such that the camera 12 can take a picture through the lens 34 as described herein and still require a sufficiently large sample of the user's skin to be effective. The color quadrants 82 can include any number of colors of known characteristics. In one embodiment, there can be a unique color quadrant with a unique known color that is a gray color with RGB values of 122 / 122 / 121. In this way, when the camera 12 takes an image, the color of the user's skin can be adjusted based on the adjustment of the color of the color quadrants 82 compared to its known color (as described further herein).

[0306] When combined with other components, the color calibrator 80 can form a color calibrator assembly. The color calibrator assembly can substantially keep all light that is not from the flash 8 out of the image taken by the camera 12.

[0307] Such a color calibrator assembly can include the color calibrator 80, the sleeve 24, and the skin contact ring 28. In such a configuration, the color calibrator assembly can remain a passive skin feature measurement aid. In another embodiment, the color calibrator assembly can also include the light source panel and / or the diffuser 38, which can make it a more effective and flexible color calibrator assembly and make it an active skin feature measurement aid. As an active feature measurement aid, the color calibrator assembly can require the processor 24 or another way to control the light source panel 70.

[0308] The color calibrator assembly can be configured to attach to the electronic device 10 using the skin analysis device 20 or portions thereof, or can have a simple attachment mechanism (a "color calibrator assembly attachment"), such as an adhesive ring on the end of the sleeve 24, or a simple mechanical element that snaps or hooks the electronic device 10 at the point of attachment. The color calibrator assembly attachment can allow the flash 8 to direct light into the color calibrator assembly attachment, particularly when used as a passive feature measurement aid.

[0309] Although the color calibrator is described herein as an assembly that can be part of the skin analysis device 20, the color calibrator can also be part of a standalone device that attaches to the electronic device 10.

[0310] PCB

[0311] Assembly

[0312] PCB 120 can include a sleeve cutout 132, a processor (which can also include a Bluetooth TM transceiver) 122, a connector 126, a charge indicator 128, a reset button 124, a battery 130 (not shown, on the reverse of PCB 120), and one or more I / O connections 134.

[0313] PCB 120 can be a generally configured circuit board with standard connections between components. PCB 120 can be shaped to be insertable into the board housing 96, including having a sleeve cutout 132 to fit around the sleeve housing 50.

[0314] Connector 126 can allow the PCB to be connected to a charging and / or data transfer line, such as a micro or mini USB as known in the art. Charge indicator 128 can simply indicate that the battery 130 is charging. Reset button 124 can allow the PCB 120, and in particular the processor 122, to be reset.

[0315] PCB also includes a battery 130 (on the reverse of board 120) to provide power to the PCB 120, and thus to the power-attached components (such as moisture sensor 36 and light source board 70). Battery 130 can be charged by plugging into connector 126, which can be a micro or mini USB port.

[0316] PCB 120, and more particularly processor 132, is in communication with electronic device 10 (e.g., using a Bluetooth transceiver that can be part of processor 132, or through another wired or wireless connection), and moisture sensor 36 and light source board 70, such as through I / O connections 134.

[0317] Processor 132 can be a microprocessor with various composite functions, as described herein, including assisting in obtaining skin analysis samples (e.g., by controlling the functions of moisture sensor 36 and light source board 120) and processing data (e.g., skin analysis samples), communication (or control of communication) with components of the skin analysis device that are in communication with processor 132 or that are capable of being controlled by processor 132 can be considered active components. Processor 132 can include a Bluetooth transceiver to enable it to communicate with substantially any Bluetooth device, but preferably at least with electronic device 10 in order to facilitate the functions described herein.

[0318] Processor 132 can be a Bluetooth integrated microprocessor from Silicon Labs TM (BGM111) microprocessor, but can have custom firmware.

[0319] The custom firmware can include primarily off-the-shelf software (OTSS) instructions for controlling typical functions. However, custom software instructions can be written to implement the functions described herein, and to ameliorate deficiencies of typical firmware. In one embodiment, the custom firmware can simply be written to remove unnecessary code that causes slower response of the processor 132.

[0320] Operation

[0321] The processor 132 can perform at least the following operations:

[0322] (a) with respect to the light source board 70: control turning on and off any light source or individual light therein for a specified time.

[0323] (b) with respect to the moisture sensor 36: control reading and receiving readings.

[0324] (c) with respect to the functionality of the Bluetooth transceiver (embedded in the processor 132 or separate therefrom): advertise its Bluetooth signal to connect to the electronic device 10, communicate over Bluetooth as needed, to establish and maintain the connection and perform functions as described herein.

[0325] (d) with respect to the electronic device 10:

[0326] (i) receive signals from the electronic device 10 to control one or more of the light source board 70 or the moisture sensor 36 (e.g., receive a sampling signal from the electronic device 10 indicating that a sample is to be taken),

[0327] (ii) provide signals and / or skin feature samples to the electronic device 10, such as moisture sensor readings and light information (e.g., spectral properties of one or more light sources turned on when an image is taken, etc.).

[0328] (e) with respect to and related to the battery 130: monitor the battery voltage, shut down one or more components of the PCB 120 if the battery 130 voltage drops below a set and configurable threshold, monitor charging of the battery 130.

[0329] Skin analysis device - assembly

[0330] Figure 9 Method 900 for assembling a skin analysis device 20 according to an aspect of the present application.

[0331] Method 900 begins at 902, where various components are produced, for example, by a 3d printer. Such printed parts can include the housing 20, the sleeve 24, and the housing cover 140. As part of such production / printing, such printed parts can be trimmed to remove support material.

[0332] In step 904, the lens 34 can be bonded to the housing 24. This can be done by bonding the lens 34 to the substrate 102 and to the lens plate 104.

[0333] In step 906, the light source plate 70 may be bonded to one end of the sleeve 24. The end having the light source plate 70 may be the end near the electronic device 10 (“proximal end”). As described herein, bonding may include bonding the light source plate 70 to the sleeve 24 such that the light source is radially located within the sleeve 24 and allows light to pass through the sleeve 24 toward the distal end of the sleeve 24.

[0334] In step 908, the moisture sensor 36 may be bonded to one end of the sleeve 24 (the opposite end to which the light source plate 70 may be bonded, which may be the end of the sleeve 24 away from the electronic device (“remote end”). Bonding may include bonding the light source plate 70 to the sleeve 24.

[0335] In step 910, the battery can be attached to the skin analysis device processor board 70, and leads (such as moisture sensor lead 36b and light source board lead 76) can be soldered to provide power to it.

[0336] As described herein, in step 912, processor 122 can perform flash memory processing using custom firmware.

[0337] In step 914, the sleeve 24 can be slid into and bonded to the housing 22, for example by applying glue to the inner surface of the sleeve housing 54.

[0338] In step 916, the skin analysis device processor board can be inserted into the housing 22, and the wires for the light source board 70 and the moisture sensor 36 can be soldered to the skin analysis device processor board 120.

[0339] In step 918, the cover 140 can be inserted into and adhered to the housing cover hole 96, for example, by applying glue to it. Optionally, the housing cover hole can be designed such that the connection causes the cover 140 to snap into place.

[0340] Various tests can then be performed with or without attachment to the electronic device 10 to ensure proper functioning of the skin analysis device 20, but it can be separated from the components. Exemplary tests may include:

[0341] (a) Lens focus: The camera 12 is focused on the test pattern through the lens 34 to ensure that the captured image is clear.

[0342] (b) Bluetooth operation: Check if the Bluetooth serial number has been transmitted.

[0343] (c) LED function: Check whether the light source can be controlled and operate normally.

[0344] (d) Moisture sensor reading check: Test moisture sensor 36 on open air and wet towel to ensure proper values are obtained.

[0345] Of course, it should be understood that various methods of assembling skin analysis device 20 can be followed, and various adjustments to method 900 can be made.

[0346] Figure 10 Method 1000 is a method of using skin analysis device 20, in accordance with an aspect of the present application.

[0347] Method 1000 can be implemented by various elements of system 1, alone or in combination. Portions of method 1000 can be implemented or performed together and in various orders, alone (even if depicted as part of method 1000 and in a particular order). Various portions of method 1000 can be depicted in screenshots 1200a-1200m, 1400a-1400d, and 1500a-1500b.

[0348] Method 1000 begins at 1002, where a skin measurement is initiated. This can be, for example, by selection 1202, 1204, 1232, or 1234. While initiation can be by skin analysis device 20 or other aspect of system 1, in typical use, initiation is by application 18 and its user. As shown and described herein, initiation can be one or more skin feature measurements, primarily any combination.

[0349] At 1004, a query is made as to whether skin analysis device 20 and electronic device 10 are calibrated. The query can be answered via application 18, for example, which can store information or flags indicating whether one or more calibrations have been completed (e.g., autofocus-based moisture sensor calibration flag, which indicates whether such a calibration has occurred). The calibration subject to the query can be any and all calibrations, or can be only those calibrations that can be required for the skin feature measurement that has been initiated.

[0350] If any required calibration has not occurred, method 1000 proceeds to 1006, where such calibration is performed. Exemplary calibrations include:

[0351] (a) Focus-based moisture sensor calibration:

[0352] (i) The user takes a photo when they lightly touch skin analysis device 20 to their skin (sample A), and then when they very firmly touch skin analysis device 20 to their skin (sample B).

[0353] (ii) To take a picture, the user is asked to touch the middle of the screen / image (as a prompt the application 18 controls the API to instruct the camera 12 to take a picture of the trigger), which will tell the camera 12 to focus on that place as the focal point. The camera 12 will then adjust the focus, which will result in a focal length parameter that is a value between 0 and 1 (for at least some electronic devices, the numerical range can vary). At the same time, the moisture sensor 36 will be read to obtain a moisture value.

[0354] (iii) After obtaining both the depth of focus and moisture values, the degree to which pressure from the user affects the moisture (pressure-based moisture adjustment factor) can be determined. A simple (two-point) X vs. Y plot can be generated, and a line drawn between the two data points. The slope of the line can then be determined, and assumed to be the pressure-based moisture adjustment slope for all measurements made using the moisture sensor 36, where the pressure applied differs.

[0355] (iv) Averaging the two depth of focus values also produces an "average pressure" depth of focus value or average focal length. Each subsequent moisture sensor reading will be normalized just as if the depth of focus was the average value.

[0356] (v) Typically, this can be accomplished by solving for Y - Y1 = m(X - X1) in the form of a slope point-form equation, where

[0357] (A) Y is the adjusted moisture sensor reading;

[0358] (B) Y1 is the future moisture sensor reading;

[0359] (C) m is the pressure-based moisture adjustment slope;

[0360] (D) X is the average focal length; and

[0361] (E) X1 is the future focal length.

[0362] (vi) By way of example, Sample A can have a moisture level of 1000 and a depth of focus of 0.2. Sample B can have a moisture level of 1500 and a depth of focus of 0.8. Thus, the slope of the line is 833. The average depth of focus is 0.5.

[0363] (vii) A new measurement is then taken of a new surface or skin area (Sample C) (i.e., an actual sample is collected, calibration is complete when determining the slope). Sample C has a depth of focus of 0.3 and a moisture level of 600. Using the point-slope form, the moisture level can be adjusted to simulate a depth of focus of 0.5, resulting in an adjusted moisture level of 766.

[0364] (viii) The calibration can be performed once or multiple times - e.g., at first use of the skin analysis device 20, at first use of the skin analysis device 20 with a particular electronic device 10, first use of the electronic device 10 by a new user (including access user).

[0365] (b) Image calibration (exposure and color temperature): A conversion function is used to scale the RGB values read from one or more known colors on the color calibrator 80 to scale them to their target (known) values. The same scaling factors are applied to each pixel in the skin image. This can be as described herein.

[0366] If no calibration is needed, the method 1000 continues at 1008, where a skin measurement occurs. For each skin feature measurement taken, there can be one or more skin feature measurement devices and one or more skin feature aids. These can all need to work together to properly perform such a skin feature measurement. In one embodiment, the skin feature measurement can be based on one or more moisture sensor readings and / or an image of the user.

[0367] Step 1008 can be described more fully in Figure 11 but in one simple example, the skin feature can be a pore analysis, and step 1008 can include:

[0368] (a) providing input to the electronic device to take a picture (e.g., by the user pressing a button, such as a volume up button), which is received by the application 18;

[0369] (b) the application 18 communicates with the processor 122 over Bluetooth, causing the processor 122 to turn on the first light source 72;

[0370] (c) the application 18 uses the camera API to take a picture while the first light source 72 is on;

[0371] (d) the application 18 communicates with the processor 122 over Bluetooth, causing the processor 122 to turn off the first light source 72.

[0372] For example, screen shots 1200b / 1200d / 1200e can be used to perform or initiate 1008 portion and Figure 11 .

[0373] The method 1000 then continues to 1010, where a skin analysis measurement sample is obtained. This can involve the application 18 receiving an image from the camera 12 (such as the images 1610, 1630, 1650), or receiving a moisture level from the moisture sensor 36, which can be received by the processor 122 and then transmitted to the application 18 by, for example, Bluetooth. It will be appreciated that generally step 1010 involves obtaining generally unprocessed captured data from the skin feature measurement device to the application 18.

[0374] The method 1000 then continues to 1012, where processing of the skin feature measurement sample (i.e. one or more skin analysis processes) can occur. Of course, such skin analysis processes will depend on the skin feature measurement taken. However, exemplary skin analysis processes can include various image processing techniques (as described herein and illustrated in Figure 10 、 Figure 11 and Fig. 13), which can take the raw image (from the camera 12) and apply various techniques to it.

[0375] As a few examples, the following skin care processes can be undertaken:

[0376] (a) Line / Wrinkle Processing - the following steps are performed:

[0377] (i) Image conversion from RGB to L*ab color space, providing luminance, green-red and blue-yellow components;

[0378] (ii) Hair Elimination - filtering the L*ab image by removing very dark pixels (if the luminance value of a pixel is significantly darker than the average luminance of the image);

[0379] (iii) Application of a high-pass filter to remove slight noise without reducing sharpness (as with Gaussian blur) or contrast;

[0380] (iv) Conversion of the image to a black and white image;

[0381] (v) Use of morphological skeletons to identify possible structures in the darker construct (such as possible lines, possible wrinkles and possible pores);

[0382] (vi) Application of a probabilistic Hough transform, which can result in a list of lines that fill in the empty spaces after the previous filter, the resulting lines can be rendered as a separate monochrome black image with the lines as white lines;

[0383] (vii) Application of an inversion to convert dark to light and vice versa;

[0384] (viii) Apply a probabilistic Hough transform, for example as a second pass, which results in a list of lines in the original image that are very likely to become wrinkles. Then, filter the list by length and group the remaining lines. The resulting list of lines is used to compute a score for the original image - where the score can be influenced by the number of lines and the characteristics of the lines (length, width, etc.).

[0385] (ix) Exemplary results can be as shown in image 1640.

[0386] (b) Pore treatment

[0387] (i) Generally, this method uses edge detection to identify darker areas of the image surrounded by brighter pixels. Then, a second pass can be made to eliminate areas larger than, for example, 500 microns, as these are less likely to be pores. In more detail:

[0388] (ii) Take the image as a matrix of pixels (rgb - for example, each pixel has a pixel color in rgb units) and convert it to a grayscale image (each pixel has r=g=b);

[0389] (iii) Create an empty matrix with zeros (0) at each position, which is equal in width and height, both of which are 1;

[0390] (iv) Calculate the average of the grayscale matrix:

[0391] (v) Subtract each element of the grayscale matrix from the average to obtain a new matrix.

[0392] (A) Example: Grayscale matrix [x][y] = average - grayscale matrix [x][y]. Thus, the new grayscale matrix will be the old matrix subtracted from the average.

[0393] (vi) For each pixel of the new grayscale matrix:

[0394] (A) Employ a 7x7 window around the pixel, with a smaller window if the pixel is near an edge (pixel window).

[0395] (B) In this square of 49 pixels, count the pixels whose color value is greater than 23 and less than 90 (for example, a specified range).

[0396] (C) If more than 20 of the 49 pixels qualify in step B) (for example, 20 out of 49 is a first number of pixels), make the center pixel white (i.e., the pixel we started with in step vi). Otherwise, the relevant pixel will be set to black.

[0397] (vii) At this point, we have a matrix of pixels with only black or white values (after iteration of step vi).

[0398] (viii) The fraction will be the total number of white pixels from the image, which can be achieved by counting at the end or incrementing a white pixel counter when the pixel color is set to white.

[0399] (ix) An exemplary result can be as shown in image 1620.

[0400] (c) Oil treatment

[0401] (i) System 1 looks for Corynebacterium and Propionibacterium fluorescence These bacteria thrive on sebum (oil). The more oily the face, the more bacteria can be seen.

[0402] (ii) Therefore, the user's skin is exposed to UVB (370 nm wavelength), for example by the second light source, and an image is obtained.

[0403] (iii) The image is converted to L*ab color space.

[0404] (iv) The pixels are searched for any pixels in the specific color range of fluorescence, and these pixels are counted.

[0405] (v) The total number of pixels in this color range generates a measure of the bacterial surface area.

[0406] (vi) An exemplary result can be as shown in image 1660.

[0407] (d) Elasticity treatment

[0408] (i) Two methods can be used, typically based on the capabilities of the camera 12 and the electronic device 10:

[0409] (A) Method 1, for example for electronic devices 10 with more limited capabilities, for example not capable of shooting high frame rate high definition video in slow motion:

[0410] (I) Correlation is performed using the user's age and moisture to determine an elasticity estimate using one of several equations, but generally the higher the moisture, the greater the elasticity; the older the age, the lower the elasticity.

[0411] (B) Method 2, for electronic devices 10 with suitable features:

[0412] (I) The user's skin is vibrated (using a vibration motor) at the same time a video is taken with the camera, for example at 240 frames per second. The length of time of the vibration and the intensity of the vibration (for example, the speed / revolutions per minute of the motor) can be configured, and the length of time of the vibration can be varied for various electronic devices 10.

[0413] (II) Then, the application 18 determines the amount of movement of the skin. More specifically, the application 18 measures the amount of time it takes for the skin to "rebound" or return to its original position after the vibration stops. The more elastic the skin, the faster it returns to its original position. This can be based on, for example, tracking the motion of various pixels, groups of pixels, or areas of skin, or tracking the number of pixels entering and / or leaving the field of view. For example, the application 18 can activate one or more starts and stops ("pulses") of the vibrating motor in one sampling, and then process a set of such recordings of the amount of time. One or more positions can be measured and averaged to obtain a score.

[0414] (III) One or more measurements can be obtained or calculated from the measurements.

[0415] a. Absolute score: An absolute score can be obtained from a particular combination of elastometry parameters (motion speed, time, field of view size, electronic device 10 model / version, skin analysis device 20 model / version, for example, which can not be needed for the elastometry). For example, the absolute score can be between 0 and 1 million. The absolute score can be normalized to a "normal" set of elastometry parameters to allow for a better relative score.

[0416] b. Relative score: A relative score can be obtained by comparing the user to other users (e.g. with similar demographic characteristics), as described herein.

[0417] (e) Sunscreen treatment

[0418] (i) Before applying sunscreen, we have the user take a reference image (unless the second light source also has UVA light, it can be necessary to block UVB light for this function to work) under UV light (e.g. from a second light source).

[0419] (ii) Then take an image of the user's skin after applying sunscreen. This can be at a time the user decides to take another image, or the application 18 can be configured with a reminder schedule.

[0420] (iii) The image shows a green color shift (because the UVB light cannot penetrate and reflects back with a yellow color tint, making the image appear green). This green color gradually disappears to blue as the sunscreen wears off and the second light fades. As the image becomes more blue, more sunscreen needs to be applied. Thus, a green color score and a blue color score can be calculated, for example, by considering the pixels in the image. Thresholds can be set to indicate to the user by the application 18 to reapply sunscreen. These can include absolute thresholds (such as a threshold blue score) and relative thresholds (such as a decrease threshold, indicating a decrease in the blue score in the image before and after applying sunscreen).

[0421] (f) Color matching process:

[0422] (i) Color matching broadly consists of two steps - color determination (determining the user's color has a specific user facial color - "determined color") and color matching (taking the determined color and matching it to an available skin tone, which can use a skin color guide that can have color values for each product tone).

[0423] (A) Color determination can follow the following steps:

[0424] (I) Typically, one or more images of one or more areas of the user's face are obtained. For example, three images can be taken using the same procedure as skin analysis (as described in methods 1000 and Figure 12e 2000), except that the images are of the neck, cheek, and forehead. As described, these three images can be averaged to produce one color value.

[0425] (II) For each of the 3 images:

[0426] a. Take a picture;

[0427] b. If normalization is available (as described herein, for example using color calibrator 80), normalize the color, for example using RGB values;

[0428] c. Perform a color correction transform to correct for any discrepancies. For example, if color quadrant 84 is black (0 / 0 / 0) and in the image is 1 / 2 / 2 / 2, then each color's RGB value in the image needs to be adjusted down by 1 / 2 / 2;

[0429] (III) Apply a Gaussian filter to slightly blur the image;

[0430] (IV) Convert each pixel from RGB to L*ab color space;

[0431] (V) Calculate the average L value of the entire image;

[0432] (VI) For each pixel, compare its L value to the average L value of the image;

[0433] a. If the pixel's L value is in the top 25% darkest part (i.e. within the dark threshold) or the top 25% brightest part (i.e. within the light threshold), consider eliminating this pixel (eliminate the effects of skin color anomalies such as hair, blemishes, freckles, etc.) (commonly referred to as removing outlier pixels);

[0434] (VII) Calculate the average L value, average A value, and average B value of the remaining pixels (average color);

[0435] (VIII) Repeat the above for images 2 and 3.

[0436] (B) Obtain an average of the 3 L*ab values to obtain one L*ab value and convert it to RGB (determined color).

[0437] (C) Of course, it should be understood that there can be different numbers of images, from different locations and different sites. Also, the threshold can be changed to accommodate.

[0438] (D) Compare to base color image:

[0439] (I) Optionally perform a hue filtering, including:

[0440] a. Calculate the hue angle of the image from the determined color (“image hue angle”);

[0441] b. Specify a skin care product, so as to select between certain product types (“total color option”, e.g. “foundation” — “foundation total color option”).

[0442] i. This can be done by the application 1, the user, or a combination thereof.

[0443] ii. Options for total color options include:

[0444] a) All color options known to the application 18;

[0445] b) All color options known to the application 18 owned by one or more vendors 300;

[0446] c) All color options in a particular palette or collection, e.g. one vendor 300;

[0447] d) Colors in one or more skin care color guides (can be from one or more vendors 300).

[0448] iii. Note that the hue angle of each skin care image can be pre-loaded into the application, downloaded from the SAS 200 or vendors 300, or even entered into the application 18 using the camera and skin analysis techniques described herein.

[0449] c. Compare the image hue angle to the hue angle of each total color option.

[0450] d. Identify the top 5 (or any number) based on the hue angle comparison (“candidate hue matches”).

[0451] (II) Calculate the color difference between the user’s skin and each candidate hue match using a color difference formula (such as CIE Delta E 2000).

[0452] (III) The lowest score of the color difference formula is the best match color for the ambient lighting that matches a sunny day (approximately D50, and optionally converted for increased accuracy). Recall that the skin image can be standardized to this light.

[0453] (IV) The user can be asked what lighting they intend to use the skin care product under (e.g., night, day, beach, office, as in 1502 in screen shot 1500a, which can allow the application 18 to present features for the applicable lighting), and / or the atmosphere or makeup look they want (e.g., casual, rock and roll, sophisticated, as in 1502 in screen shot 1500a, which can allow the application 18 to assume factors in determining color match or adjustment match, such as boldness). This can, for example, change the recommended color by applying method 1300a. Such user interaction can be as shown in screen shots 1500a and 1500b. Figure 15 and screen shots 1500a and 1500b.

[0454] Steps 1010 and 1012 can be performed primarily on or by the electronic device 10 and / or the skin analysis device 20. However, other aspects of the system 1 can be involved, such as the SAS 200, for example, if greater processing power or storage space is needed.

[0455] At this point, the method 1000 can substantially complete processing of the skin analysis measurement sample for one or more skin analysis measurements. This can result in a skin analysis sample. Such a sample can have one or more raw images, one or more processed images, a user identifier, a date and time stamp, and other relevant information. The images can be sent from the electronic device 10, uploaded in real time to the SAS 200. The user's gender, age, GPS coordinates, and other possible non-personally identifying information (or personally identifying as needed / desired in accordance with applicable privacy requirements) can be stored. Processing can occur on the device and / or on the SAS 200 and / or on the product owner 300, depending on, for example, the intensive degree of processing, what will happen after processing, and which entity / entities own copies of the data when all data is complete, as described herein.

[0456] In one example of a skin feature sample, the sample (or skin feature sample data structure) can include one or more of the following:

[0457] (a) Six image files (unprocessed, color, two from three positions or configurations), exemplary images include 1610, 1630, and 1650;

[0458] (b) Optionally, one or more processed versions of the unprocessed image, exemplary images include 1620, 1640, and 1660;

[0459] (c) absolute scores for all skin analyses;

[0460] (d) RGB values for skin color;

[0461] (e) one or more moisture readings (absolute and / or normalized);

[0462] (f) personal information (name, age, ethnicity or race, allowed GPS coordinates, gender, etc.); and

[0463] (g) past sample or trend information.

[0464] Thereafter, the method 1000 continues to 1014 to begin the calculations and processing required for the analysis to be presented. The steps performed at 1014 will depend greatly on the analysis, however, at 1014 the collected samples can be compared to one or more sample sets or subsets stored at the SAS 200 that can be relevant to the current user (for comparison to themselves over time (e.g., in Figure 14c ), or to compare the current user to all users in the user's demographic in the SAS 200 (or product owner 300) (e.g., in Figure 12g , UI element 1260).

[0465] The method 1000 then continues to 1016, where the results can be displayed to the user, for example on one or more screenshots of the application 18 (e.g., as shown in Figure 12g and Figures 14a to 14d .

[0466] The method 1000 then continues to 1018, where a recommendation algorithm is executed. Of course, 1018 can occur before or concurrently with 1014 / 1016. There can be a recommendation algorithm for each skin feature, and even for various combinations of skin features. The recommendation algorithm can be substantially the general recommendation algorithm described herein, which can be executed by the application 18 and / or the SAS 200. In this case, the application 18 and / or the SAS 200 have the data required for the recommendation algorithm, and can simply apply the recommendation algorithm. Alternatively, and for example in the case where the product owner 300 has its own recommendation algorithm, the recommendation algorithm can be executed by the product owner 300. In this case, the application 18 and / or the SAS 200 can provide the product owner 300 with the data required for the recommendation algorithm ("recommendation required data"), and the product owner 300 can communicate the recommended product back to the application 18 (all data required or a skin product identifier, for identifying a product for which information is stored in the application 18).

[0467] Method 1000 then continues to 1020, where one or more recommended products are displayed for the user to review and consider for purchase, e.g., as shown in Figure 12h

[0468] Steps 1018 and 1020 can be somewhat iterative in that they can also include prompting the user to provide additional information to better perform the recommendation algorithm. For example, the user can be undergoing a moisture analysis, and their skin is dry. At 1018, a moisturizer can be recommended. However, after displaying such a recommended product, at 1020 or before, method 1000 can ask the user if they plan to use the product during the day and / or outside. If so, the recommendation algorithm can change the moisturizer to a moisturizer with UV protection, especially if the skin care update or external source indicates that the user is in a location with high UV (which can be located by the GPS of electronic device 10).

[0469] Figure 11 Method 1100 is a method for performing skin analysis measurements according to an aspect of the present application. Method 1100 substantially addresses the method for performing the recording or obtaining skin analysis measurements (bringing them into application 18, processing them, separating them after they are transmitted to application 18).

[0470] For example, after 1008, method 1100 begins at 1102. At 1102, a query is made as to whether skin analysis device 20 (and / or electronic device 10) is in place. This can mean that skin analysis device 20 is held against the user’s face or body. Such a step can be needed in the case of one or more skin feature measurement devices and / or skin feature measurement aids that can be damaging (badly affect the device - can assist a skin feature measurement device in taking a sample of a user’s skin feature, but can be dangerous to the user if used in an incorrect manner). For example, in one embodiment, second light source 74 can be ultraviolet light that can damage the eyes, or other light sources that can damage the eyes (laser, infrared, etc.). At 1102, moisture sensor 36 (in such an example, a safe to use indicator device) can be queried, and if the reading from moisture sensor 36 indicates that the user has skin analysis device 20 in place (in this case, such a reading is a safe to use indicator signal), then the method can proceed, particularly before turning on the ultraviolet light in safe to use. Of course, it can be necessary to ensure that skin analysis device 20 is in place before performing method 1100, e.g., to assist in preventing incorrect images from being taken. Or, such queries can be performed before activating skin feature measurement devices that are particularly problematic / dangerous.

[0471] If skin analysis device 20 is not in place, then at 1104, method 1100 can wait. ​

[0472] Continuing, the method 1100 reaches 1106, where a loop (of 1106-1120 or 1122) can be started for each skin feature measurement to be taken. Broadly, there can be imaging and moisture sensor readings. However, in imaging, the performance of 1106 to 1120 can vary slightly based on what equipment and aids can be used, e.g. lines / wrinkles, oil, elasticity, etc.

[0473] Returning to 1108, for a given skin feature measurement, are the required skin feature measurement equipment ready. For example, the camera 12 or moisture sensor 36 can need to be turned on or warmed up, especially if they have not been used recently (as known by the application 18). If they are not ready, the method 1100 waits and prepares them at 1110.

[0474] At 1112, for a given skin feature measurement, is queried whether the required skin feature measurement aids are ready. This is very similar to the skin feature measurement equipment in 1108 and 1110. For example, the first light source 72 can preferably be warm before use (making its spectrum more consistent).

[0475] At 1116, having ensured that the required components are ready, each skin feature measurement aid (one or more for a given skin feature measurement) can be activated. For passive skin feature measurement aids, this can not be necessary (i.e. the lens 34 can already be in place). For active skin feature measurement aids, this can be:

[0476] (a) the application 18 communicates with the processor 132 (e.g. over Bluetooth):

[0477] (i) turn on the first light source (unpolarized) to prepare for taking an image for lines processing;

[0478] (ii) turn on the second light source to prepare for taking an image for oil;

[0479] (b) the application 18 communicates with the aid, e.g. over an API:

[0480] (i) turn on the vibration motor to prepare for taking an image (e.g. a video) for elasticity.

[0481] At 1118, each skin feature measurement equipment (one or more for a given skin feature measurement) can be activated after a delay to ensure the aids are adequately prepared (e.g. 1 second after activating a particular light source). This can involve:

[0482] (a) the application 18 uses the API to prepare the camera 12, then accepts input from the user (e.g., the volume up button is being pressed) or a timer to automatically take an image;

[0483] (b) the application 18 uses the API to turn on "live preview" (e.g., lines / pores / moisture / oil);

[0484] (c) the application 18 communicates with the processor 132 (e.g., over Bluetooth) to start communicating with the moisture sensor 36.

[0485] At 1120, the activated skin feature measurement device and skin feature measurement aid can be deactivated (e.g., the first light source 72 is turned off).

[0486] At 1122, the method 1100 can confirm that the run is normal. This can be through logic on the application 18 and / or through input from the user (e.g., indicating acceptable image quality). For example, this can be performed using the screenshot 1200f.

[0487] The method 1100 can then return to 1010, as described herein.

[0488] It should be appreciated that although 1106 through 1122 are described as "for each skin feature measurement," several can be combined in quick succession. For example, one or more can be started at the same time, such that the above controls can be extracted from the user (i.e., pre-emptively as well as turning on and off various light sources between taking various images). The user or logic in the application 18 can select one or more skin feature measurements, and once the user places the skin analysis device at the first skin location, the required steps can be taken without further user input.

[0489] In one example embodiment, lines, oiliness, and moisture are the skin feature measurements. Assuming the method 1100 proceeds to 1116 (i.e., the camera 12, lens 34, first light source 72, and moisture sensor 36 are ready), the following can occur:

[0490] (a) the first light source is turned on;

[0491] (b) an image is taken;

[0492] (c) the first light source is turned off;

[0493] (d) the second light source is turned on;

[0494] (e) an image is taken;

[0495] (f) the second light source is turned off;

[0496] (g) the moisture sensor is activated;

[0497] (h) the processor obtains a moisture sensor reading;

[0498] (i) the moisture sensor is turned off.

[0499] (j) the app 18 ensures that both images and the moisture sensor reading look acceptable (e.g., in combination with a user so indicated) and the method 1100 ends.

[0500] Figures 12a to 12m A screen shot of the app 18 for the electronic device 10 is shown in accordance with one aspect of the application.

[0501] The screen shots 1200a-1200m show various screens of the app 18 that can be displayed to a user to allow the user to access the functionality described herein. Each such screen can include one or more user interface (UI) elements (buttons, text, widgets, icons, pictures, drop down lists, tabs, infographics, etc.). It should be understood that the screen shots 1200a-1200m and the UI elements shown are merely exemplary - many designs, layouts, orders, and screen styles, UI elements, and features can be conceived to implement the functionality and aspects of the application as described herein.

[0502] Starting with screen shot 1200a, the user can be presented with buttons 1202 or 1204 to select what they want to do first. Of course, all of the buttons here can be accessed using user input (e.g., pressing a button on the screen or using a pointer device). If the user selects button 1202, then they can start a skin analysis as described herein. If the user selects button 1204, then they can start looking for products as described herein.

[0503] In screen shot 1200b, the user can be presented with the following UI elements:

[0504] (a) tab selectors 1206-1214: the user can switch between areas of the app 18, e.g.:

[0505] (i) "User Info" 1206: the user can specify and view stored information about them, e.g., as shown in screen shot 1200c and as shown in user info area 1226.

[0506] (ii) Analysis 1208: the user can be shown several screen shots (e.g., 1200d-1200f) to initiate and take steps to perform a skin analysis.

[0507] (iii) Results 1210: the user can be shown multiple screen shots (e.g., 1200g and 1400a-1400d) to view the results of an analysis.

[0508] (iv) Product 1212: where several screenshots (e.g. 1200h) can be displayed to the user to view recommended products, and optionally initiate purchase.

[0509] (v) Assistance 1214: where several screenshots (e.g. 1200i to 1200m) can be displayed to the user to provide tutorial information on how to perform an analysis step or otherwise use the system 1 and its components as described herein.

[0510] (b) UI element 1216 can be a collection of UI elements that show various aspects of the user's skin (skin features), allowing the user to specify aspects they are interested in. When the user later switches to "analyze", these selections can pre-select or initiate steps or functions needed to analyze the selected skin features. Others can be added over time, and some can not be displayed but can be included, e.g. sunscreen as described herein. As an alternative to UI element 1216, UI element 1420 can be displayed on screenshot 1200b; such UI element 1420 can show rewards or prizes the user has earned, e.g. based on historical skin analysis results. For example, this can provide free or discounted skincare, or provide recognition, e.g. in a peer or skincare "leaderboard".

[0511] (c) UI element 1218 can display the user's skin history. For example, a graph or summary of moisture levels over time, e.g. Figure 14b the content shown in 1422 of screenshot 1400b or Figure 14c a summary of the content shown in 1432 / 1430 of screenshot 1400c.

[0512] (d) UI element 1220 can display skincare updates and can include links to recommended products (such as "high UV tomorrow, so apply sunscreen", and the push UI element 1220 can take the user to a screenshot, e.g. 1200h, that can pre-select a sunscreen according to the user and their skin features.

[0513] (e) UI element 1222 can display current feedback to the user, e.g. highlighting trends about their skin feature samples or comparisons to others.

[0514] (f) UI element 1224 can show an icon for adding a user profile (a first user profile for the application 18 or an additional user profile).

[0515] (g) UI elements 1226 and 1228( Figure 12c) can display user information and allow it to be entered. Such user information can allow results to be displayed based on demographics (i.e. age, gender, ethnicity) and can make results more accurate and provide quality control checks (i.e. if ethnicity is "coloured" and the skin analysis measurements indicate very white skin then a confirmation can be sought).

[0516] In screenshot 1200d, the following UI elements can be presented to the user:

[0517] (a) UI element 1229: can allow the application 18 to switch between obtaining a sample of the logged-in user and obtaining a sample of a visitor, so that a particular electronic device 10 (with skin analysis device 20) can be shared between friends.

[0518] (b) UI element 1230: can allow the user to jump to a tutorial on how to use the application 18 and skin analysis device 20.

[0519] (c) UI elements 1232 to 1236: can allow the user to initiate one or more analyses of one or more skin features, such as lines / pores / moisture / elasticity 1232, oil 1234, and acne 1236. One or more such UI elements can be disabled (not selectable). This can be the case, for example, if a required skin analysis measurement device or skin analysis device aid is not present on the skin analysis device 20 or electronic device 10. Other such UI elements can be added, for example for sunscreen, and other analyses can be added later.

[0520] In screenshot 1200e, the user can be presented with a photo of the user 1240 and various sample locations 1242, 1244, 1246 that the user is to apply the skin analysis device 20 to in order to perform the intended analysis.

[0521] In screenshot 1200f, the user can be presented with the image 1248 taken (note that this can be post-processed in the Figure 12f button reset 1250, save and take next image 1252 (i.e. save a copy of the image on the electronic device 10) and go to next 1254 (i.e. use the image but do not save in the photo album on the electronic device 10).

[0522] In screenshot 1200g, the user can be presented with the results of their skin analysis. Such results can be described based on the type of results 1256 and the aspect of the type of results 1258. In this example, the type is peer and the aspect is same gender, same age range, and same country. The type can also be a comparison to some aspect of the user in the past. The results can be shown in a results summary infographic 1260, which can include percentages of results for lines 1262, pores 1264, moisture 1266, elasticity 1268, oil 1270, and acne 1272.

[0523] In 1260, a percentile can be assigned to each result (note that in Figure 12g 100% percentiles are more desirable in 1260, although any nomenclature can be used). For various skin features, these percentiles can generally be calculated as follows:

[0524] (a) Lines 1262: More and / or coarser lines are generally worse. The surface area in the image and the total number of lines can be calculated. The lines / unit area can then be calculated. This "lines score" can then be compared to users in the same demographic, with lower percentiles for lower scores.

[0525] (b) Pores 1264: More and / or larger pores are generally worse. The total number of pores detected in the image can be calculated. The surface area represented in the image can be determined. The pores / unit area can then be calculated. This "pores score" can then be compared to users in the same demographic, with lower percentiles for lower scores.

[0526] (c) Moisture 1266: Generally, the more moisturized the skin is, the better. A moisture score, generally a numerical value (possibly normalized as described herein) can be compared to users in the same demographic, with lower percentiles for higher scores.

[0527] (d) Elasticity 1268: Generally, the more elastic the skin is, the better. An elasticity score can be compared to users in the same demographic, with lower percentiles for higher scores.

[0528] (e) Oil 1270: Generally, the less oily the skin is, the better. An oil score can be compared to users in the same demographic, with lower percentiles for lower scores.

[0529] (f) Acne 1272: Generally, the less acne on the skin, the better. An acne score can be compared to users in the same demographic, with lower percentiles for lower scores.

[0530] (g) Sunscreen (see 1400a): Generally, the more effective the sun protection, the better. This means the greener the image, the better, with blue indicating the sunscreen's wear-off. A blue score can be applied and the results displayed to indicate whether the user has sufficient sunscreen, especially compared to the current weather / UV conditions (as shown in 1402).

[0531] As described above, screenshot 1200 is an exemplary screenshot that can display results. Figure 14d Another exemplary screenshot can be seen in screenshot 1400d, which shows the results without comparison. In this screenshot 1400d, UI element 1446 may display various skin features (1442) and their results / scores (1444).

[0532] In screenshot 1200h, the recommended product 1274 can be presented to the user, along with information about the recommended product 1276 (e.g., ratings, reviews and links, quantity information, and price). It is noteworthy that ratings and reviews from other users may be specific to users with similar skin types and / or skin tones—thus increasing the value of the ratings and reviews, for example, by filtering ratings and reviewers. Information about the recommended product may also include how it works 1278 and instructions for use 1280. The user can select button 1282 to begin purchasing the recommended product, for example, by adding it to a shopping cart (as known in the art), which could be a shopping cart on application 18 and / or an application of supplier 400.

[0533] In screenshots 1200i to 1200m, various images 1284, 1286, 1288, 1290, and 1292 can be presented to the user to provide instructions. Of course, these can be accompanied by text if needed. Typically, and as further described herein, the user will wash their face, ensure they look “normal” (i.e., without dirt or makeup abnormalities), properly position the skin analysis device, and press a button (e.g., a volume button) on the electronic device 10 to initiate image capture by the camera 12 (as shown in 1288), and wait for a response from the application 18 indicating that they can move to another location or that image capture is complete (e.g., via an audible signal or vibration).

[0534] Figures 13a to 13b This is an example of method 1300a according to one aspect of the present invention and its use for color matching under different lighting conditions.

[0535] The goal of the method 1300a can be to determine, based on the light other than ambient light (which can be specified by the user, e.g. outdoors, very bright TV, darker than indoors, etc.) and the user's desired ambience or makeup (e.g. rock music - an "ambience" in 1502) whether a shade other than mindE is suitable for the desired user, lighting conditions, and makeup. As described, the method 1300a considers the image (and hence mindE) to be based on the ambient lighting at the time of capture (optionally normalized to such lighting but referred to as "capture lighting") and then the user can want a recommended product / shade for outdoor lighting or for indoor lighting (each "alternative lighting" has "alternative lighting characteristics", e.g. 1502). Of course, different capture lighting and different alternative lighting can be used and all configurable parameters can be adjusted to accommodate (parameter variations, e.g. as to the nature of the differences between the capture lighting and the various alternative lighting). If there is a more suitable shade for the lighting / makeup for which the user intends to use the skin care product, then the goal is to change mindE (such more suitable shade is a "contextually suggested product color").

[0536] The method 1300a starts at 1302 where the minimum dE (minimum color difference) and the resulting color (mindE color) is found as described herein (e.g. candidate shade matching or total color option).

[0537] At 1304, the dE values for the shades one and two shades darker than the mindE color are found, (i.e. mindElc and mindE2c, or one shade darker, two shades darker). Then at 1306 the dE values between the shades are found.

[0538] At 1308, the query is whether mindEl minus mindE is less than or equal to 1 (one can configure at 1308 but it is the dE between the shades in the color option selection). Of course, if there are many shades in the color option, then the next shade is more likely to be closer to mindE (meaning the query at 1308 is more likely to be "yes").

[0539] At 1310, if the answer is yes, then the recommended product for outdoor use (assuming such use has been specified) will be mindEl.

[0540] At 1312, a query is made whether mindE2 minus mindE1 is less than or equal to a configurable parameter (6 as an example for shades 1 to 6, using color shades from screenshot 1350) and mindE2 minus mindE1 is less than or equal to another configurable parameter (3 as an example for shades 7 to 10, using color shades from screenshot 1350). If so, then at 1316, mindE2C becomes the recommended product for indoor lighting (if the user requires such product recommendations to be provided, or it can be stored in application 18). If not, then mindE1C becomes the recommended product for indoor lighting.

[0541] Returning to 1308, if the query result is "No", then method 1300 continues to 1318, where mindE becomes the recommended product for outdoor lighting.

[0542] Then at 1320, a query is made whether mindE1 minus mindE is less than or equal to a configurable parameter (6 as an example for shades 1 to 6, using color shades from screenshot 1350) and mindE1 minus mindE is less than or equal to another configurable parameter (3 as an example for shades 7 to 10, using color shades from screenshot 1350). If so, then at 1322, mindEC becomes the recommended product for indoor lighting (if the user requires such product recommendations to be provided, or it can be stored in application 18). If not, then mindE1C becomes the recommended product for indoor lighting.

[0543] Turning to Figure 13b , screenshot 1350 can be an example screenshot when the "MyColor" tab 1356 is selected. Screenshot 1350 shows the measured skin color 1352 and one or more color options, which include total color options and their color values for specific color matches. Tab 1358 can take the user to a screenshot, which can be similar to screenshot 1200h, with recommended products, which can be color-matched skin care products, such as foundation, blush, etc.

[0544] At 1302b, the medium brown is determined to be mindE. At 1304b, the brown and dark brown are mindE1C and mindE2C, respectively. At 1306b and 1308b, the subtraction occurs and yields the values (0.925 and 3.197). Thus at 1318b, the brown (mindE1C) becomes the outdoor recommended product or color match, and at 1322, the brown becomes the indoor recommended product or color match.

[0545] Embodiments of the systems and methods described herein can be implemented in hardware or software, or a combination of both, Embodiments can be implemented in computer programs executing on programmable computers or electronic devices, each computer program including at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), and at least one communication interface. In some embodiments, the computer program is downloaded to the computer or electronic device from a remote source. In other embodiments, the computer program is downloaded to the computer or electronic device from a remote source.

[0546] Program code is applied to input data to perform the functions described herein and to generate output information. The output information is applied to one or more output devices, in known fashion. For example, this can be through an application programming interface, which is used to allow the interaction with the electronic device.

[0547] Each program can be implemented in a high level procedural or object oriented programming or scripting language, or both, to communicate with a computer system and / or to carry out a particular process. However, programs can be implemented in assembly or machine language, if desired, for specific applications or if a particular processor or device requires it. The language can be a compiled or interpreted language. Each such computer program can be stored in a storage media or device (e.g., ROM, magnetic disk, optical disc) that is readable by a general or special purpose programmable computer for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein. Embodiments of the system can also be considered to be implemented as a non-transitory computer-readable storage medium, configured with a computer program, where the storage media so configured causes a computer to operate in a specific and predefined manner to perform the described functions.

[0548] In addition, the systems and methods described herein can be distributed over computer program products that include physical, non-transitory computer readable media in various forms. The media can be provided in various forms including one or more diskettes, compact disks, tapes, chips, magnetic or electronic memory, etc. The non-transitory computer readable media can be digital and / or analog in some

[0549] It will be apparent to those skilled in the art that other configurations, hardware, etc. can be used in other embodiments of the present application products, methods and systems. It is to be understood that the specification is illustrative of the present application and that other embodiments within the spirit and scope of the application will be readily apparent to those skilled in the art.

[0550] The foregoing embodiments have been described by way of example only. The application should not be deemed to be limited to these embodiments, but is defined only by the claims which follow.

Claims

1. A system for performing one or more skin feature analyses, the system comprising: A skin feature application that runs on an electronic device with a camera capable of capturing images of the user, the skin feature application being configured to: Control the camera to capture a set of images of the user; A skincare treatment is performed on the set of images to obtain a set of skin analysis measurements; Communicating with skin analysis equipment: Send an activation signal to the moisture sensor to initiate moisture sensor readings; and Receive moisture sensor readings from the moisture sensor; A skin analysis device, connected to the electronic device, includes: The processor is configured as follows: Receive an activation signal from the skin feature application; In response to the activation signal, a moisture sensor reading is obtained from the moisture sensor; and Send the moisture sensor readings to the skin feature application; A moisture sensor is configured as follows: The moisture sensor readings are sent to the processor. The skincare treatment includes wrinkle treatment, and the wrinkle treatment further includes: Convert the image to L*ab colors; Hair removal; Apply a high-pass filter; Convert the image to a black and white image; Use morphological skeletons to identify possible textures and wrinkles; Applying the Hough transform; and The image is inverted so that the textures in the image are white.

2. A system for performing one or more skin feature analyses, the system comprising: A skin feature application that runs on an electronic device with a camera capable of capturing images of the user, the skin feature application being configured to: Control the camera to capture a set of images of the user; A skincare treatment is performed on the set of images to obtain a set of skin analysis measurements; Communicating with skin analysis equipment: Send an activation signal to the moisture sensor to initiate moisture sensor readings; and Receive moisture sensor readings from the moisture sensor; A skin analysis device, connected to the electronic device, includes: The processor is configured as follows: Receive an activation signal from the skin feature application; In response to the activation signal, a moisture sensor reading is obtained from the moisture sensor; and Send the moisture sensor readings to the skin feature application; A moisture sensor is configured as follows: The moisture sensor readings are sent to the processor. The skincare treatment includes pore treatment, and the pore treatment further includes: Convert the image into a grayscale image with a grayscale matrix; Calculate the average value of the grayscale matrix: Subtract each element of the grayscale matrix from the average value to obtain a new grayscale matrix; For each pixel with a pixel color in the new grayscale matrix: Consider the pixel window surrounding the pixel; Calculate the pixels within the specified range of pixel color values ​​in the pixel window; and If the pixel count in the pixel window exceeds a first quantity, the pixel color is set to white and the white pixel counter is incremented by 1; otherwise, the pixel color is set to black. Set the white pixel counter to the pore fraction.

3. A system for performing one or more skin feature analyses, the system comprising: A skin feature application that runs on an electronic device with a camera capable of capturing images of the user, the skin feature application being configured to: Control the camera to capture a set of images of the user; A skincare treatment is performed on the set of images to obtain a set of skin analysis measurements; Communicating with skin analysis equipment: Send an activation signal to the moisture sensor to initiate moisture sensor readings; and Receive moisture sensor readings from the moisture sensor; A skin analysis device, connected to the electronic device, includes: The processor is configured as follows: Receive an activation signal from the skin feature application; In response to the activation signal, a moisture sensor reading is obtained from the moisture sensor; and Send the moisture sensor readings to the skin feature application; A moisture sensor is configured as follows: The moisture sensor readings are sent to the processor. The skin feature application is also configured to activate an ultraviolet light source when the set of images is captured; The set of images includes images before applying sunscreen and one or more images after applying sunscreen; and The aforementioned skincare treatment includes a sun protection treatment, and the sun protection treatment further includes: Calculate the blue score of the set of images; Compare the blue score with the threshold blue score; and If the blue score exceeds the threshold blue score, it indicates that the sunscreen should be reapplied.

4. The system according to claim 3, wherein, The set of images also includes an image before sunscreen application, and the comparison further includes determining the difference between the blue score of the image before sunscreen application and the blue score of the image after sunscreen application, and the indication further includes indicating whether the difference between the blue score of the image before sunscreen application and the blue score of the image after sunscreen application exceeds a reduction threshold.

5. The system according to any one of claims 1 to 4, wherein, The skincare routine also includes sun protection.

6. The system according to any one of claims 1 to 4, wherein, The skincare routine also includes color matching.

7. The system according to any one of claims 1 to 4, wherein, The skin analysis device is detachably connected to the electronic device.

Images