Apparatus and method for visualizing cosmetic skin characteristics - Patents.com

Abstract

It allows consumers to understand their cosmetic skin characteristics. The method includes: a) acquiring a first digital image of a portion of a face in step 202, where a region of the input image is selected; b) defining a plurality of tiles across the acquired first digital image in step 204; c) analyzing the first digital image for each of the defined plurality of tiles for cosmetic skin characteristics in step 206; d) assigning a unique single-degree mark to each tile of the defined plurality of tiles based on the tile's analyzed cosmetic skin characteristics in step 208; and e) displaying some of the plurality of tiles, each having a uniquely assigned single-degree mark, to visualize the cosmetic skin characteristics in step 210. Some of the displayed plurality of tiles display cosmetic skin characteristics that are better than the cosmetic skin characteristic status of the non-displayed tiles based on the tile's analyzed cosmetic skin characteristics.

Description

[Technical field]

[0001] The present invention relates to a system and method for visualizing cosmetic skin characteristics. [Background technology]

[0002] Consumers seek to improve their physical appearance by using cosmetics and skin care treatments. As a result, there is a very wide selection of products available for consumers to choose from. Individual consumers often find it difficult to determine which products to apply and how to achieve the best results, given the individual's skin type and skin characteristics, as well as potentially different or changing skin characteristics. This problem is compounded because an individual's skin changes over time and / or beauty trends change over time. Beauty consultants at retail cosmetic counters are often tasked with identifying the skin characteristics of their clients that require treatment. Once the skin characteristic(s) is / are identified, the consultant communicates the type, amount, and location of the skin characteristic(s) to the client for explanation. Finally, a skin care product and application technique is recommended that is intended to improve the appearance of the identified skin characteristic and demonstrate the improvement to the consumer. Much of this approach relies on the consultant's subjective judgment and / or visual acuity.

[0003] A growing population of digitally savvy consumers, especially those in the younger demographics, tend to utilize data processing devices to interact with each other and make online purchasing decisions for skin and beauty care products instead of seeking consultation with a traditional beauty consultant. Some of these consumers believe that there is a higher level of objective fact that can be achieved with digital tools, and that such recommendations from digital sources have higher credibility for this consumer segment. Many attempts have been made to enable such consumers to perform personal skin analysis by analyzing digital images of themselves (e.g., "selfies") using computer modeling / algorithms. The results provided by the computer model can then be used to provide the consumer with the location and type of visually perceptible skin characteristics (e.g., visible spots, visible lines, and wrinkles) and subsequent product recommendations / treatment instructions.

[0004] One way to indicate / mark these visually perceptible skin characteristics to consumers is through a "heat map" to visualize their face and skin characteristic(s). However, there is typically a lot of information for consumers to understand. This may not be as problematic with visually perceptible skin characteristics, because consumers can use the visually perceptible characteristics as a reference point or target for skin care treatment. However, when the skin characteristics are not visually perceptible, the problem is considerably worse. This is especially true for younger demographics of consumers who may not have visually perceptible characteristics or may have a minimal amount of visually perceptible characteristics. Therefore, there is a need to quickly and easily identify visually perceptible characteristics and their location on the face, especially in younger demographics of consumers who typically lack many perceptible skin characteristics, especially when more than one such visually perceptible characteristic is present.

[0005] There is a growing body of research suggesting that some non-visually perceptible skin characteristics can become visually perceptible. To this end, if consumers identify these non-visually perceptible skin characteristics early in life and take preventative skin treatments against these characteristics, there is an increased chance of delaying and / or reducing the occurrence and / or delaying the onset of visually perceptible undesirable skin characteristics from later formation.

[0006] Although various skin evaluation digital tools have been developed to provide information on the above skin characteristics and meet the needs of consumers in the digital age, consumers are all overloaded with a large amount of information, and the process of understanding information in a short time is difficult for many consumers, especially in the context of cosmetic skin characteristic conditions, such as "existing skin condition and what can be done to improve it". However, consumers tend to view evaluation information on mobile devices such as mobile phones, or tend to have limited time to understand evaluation information. As a result, consumers can be quickly overwhelmed when a huge amount of information is displayed, resulting in "information overload". Information overload can lead to a situation where the consumer no longer understands the information or is no longer in a position to evaluate what needs to be done to improve their skin condition due to information overload. In such a situation, consumers may stop using the digital tool that creates information overload for them. Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, there is a need for a method and system that provides consumers with a way to identify one or more skin characteristics, display the location of those characteristics on the consumer's face in a simple manner to enable the consumer to understand their cosmetic skin characteristics, and obtain targeted and effective treatments for improving the condition of the cosmetic skin characteristics. [Means for solving the problem]

[0008] The present invention relates to a method for visualizing at least one cosmetic skin characteristic of a subject, the method comprising: a) obtaining a first digital image of at least a portion of a face of a subject, the first digital image being selected from at least one region of an input image of the face; b) defining a plurality of tiles across the acquired first digital image; c) analyzing the first digital image for each of the defined plurality of tiles for at least one cosmetic skin characteristic; d) assigning a unique degree mark to each tile of the defined plurality of tiles based on the analyzed at least one cosmetic skin characteristic of the tile; e) displaying at least some of the plurality of tiles, each having a uniquely assigned single degree indicia, to visualize at least one cosmetic skin characteristic; Here, at least some of the displayed plurality of tiles display cosmetic skin characteristics that are better compared to cosmetic skin characteristic states of non-displayed tiles of the plurality of tiles based on the analyzed at least one cosmetic skin characteristic of the tile. [Brief description of the drawings]

[0009] It should be understood that both the foregoing general description and the following detailed description are intended to provide an overview or framework for understanding the nature and character of the claimed subject matter, describing various embodiments. The accompanying drawings are included to provide a further understanding of the various embodiments, and are incorporated in and constitute a part of this specification. The drawings illustrate various embodiments described herein, and together with the description, serve to explain the principles and operation of the claimed subject matter. [Figure 1] FIG. 1 illustrates an exemplary system for visualizing at least one cosmetic skin characteristic over a network in accordance with the present invention. [Diagram 2]FIG. 2 illustrates an alternative exemplary system for visualizing cosmetic skin characteristics in accordance with the present invention. [Diagram 3] FIG. 1 is a block diagram illustrating components of an exemplary system for visualizing cosmetic skin characteristics in accordance with the present invention. [Figure 4A] 1 is a sequence of process flow diagrams illustrating a method for visualizing cosmetic skin characteristics in accordance with the present invention. [Figure 4B] 1 is a sequence of process flow diagrams illustrating a method for visualizing cosmetic skin characteristics in accordance with the present invention. [Figure 4C] 1 is a sequence of process flow diagrams illustrating a method for visualizing cosmetic skin characteristics in accordance with the present invention. [Diagram 5] 1 is a flow chart illustrating a method for visualizing cosmetic skin characteristics in accordance with the present invention. [Figure 6A] 1A-1D are a series of process flow diagrams detailing the steps of acquiring a first digital image in a method for visualizing cosmetic skin characteristics according to the present invention. [Figure 6B] 1A-1D are a series of process flow diagrams detailing the steps of acquiring a first digital image in a method for visualizing cosmetic skin characteristics according to the present invention. [Figure 6C] 1A-1D are a series of process flow diagrams detailing the steps of acquiring a first digital image in a method for visualizing cosmetic skin characteristics according to the present invention. [Figure 7] 1 is a flow chart showing the steps of acquiring a first digital image. [Figure 8] 3 is a photograph showing a step of defining a plurality of tiles in a method for visualizing cosmetic skin characteristics according to the present invention. [Figure 9] 13 is a flow chart illustrating a process for defining a number of tiles. [Figure 10] 5 is a flow chart illustrating a process 500 for analyzing image data for each of a plurality of defined tiles in accordance with the present invention. [Figure 11A] FIG. 2 is a process flow diagram illustrating a process for displaying multiple tiles in accordance with the present invention. [Figure 11B]FIG. 2 is a process flow diagram illustrating a process for displaying multiple tiles in accordance with the present invention. [Figure 11C] FIG. 2 is a process flow diagram illustrating a process for displaying multiple tiles in accordance with the present invention. [Figure 12] 4 is a flow chart illustrating a process for displaying multiple tiles in accordance with the present invention. [Figure 13A] FIG. 1 is a process flow diagram illustrating a method for visualizing at least one cosmetic skin characteristic in accordance with the present invention. [Figure 13B] FIG. 1 is a process flow diagram illustrating a method for visualizing at least one cosmetic skin characteristic in accordance with the present invention. [Figure 13C] FIG. 1 is a process flow diagram illustrating a method for visualizing at least one cosmetic skin characteristic in accordance with the present invention. [Figure 13D] FIG. 1 is a process flow diagram illustrating a method for visualizing at least one cosmetic skin characteristic in accordance with the present invention. [Figure 14] 1 is a flow chart illustrating a method for visualizing at least one cosmetic skin characteristic in accordance with the present invention. [Figure 15] 1 is a flow chart illustrating an alternative method for visualizing at least one cosmetic skin characteristic in accordance with the present invention. [Figure 16A] 1A-1C are screenshots illustrating exemplary user interfaces for visualizing cosmetic skin characteristics in accordance with the present invention. [Figure 16B] 1A-1C are screenshots illustrating exemplary user interfaces for visualizing cosmetic skin characteristics in accordance with the present invention. [Figure 16C] 1A-1C are screenshots illustrating exemplary user interfaces for visualizing cosmetic skin characteristics in accordance with the present invention. [Figure 16D] 1A-1C are screenshots illustrating exemplary user interfaces for visualizing cosmetic skin characteristics in accordance with the present invention. [Figure 16E]1A-1C are screenshots illustrating exemplary user interfaces for visualizing cosmetic skin characteristics in accordance with the present invention. [Figure 17A] 11A-11C are screenshots illustrating an exemplary user interface for providing additional input prompts for repeating a method for visualizing cosmetic skin characteristics according to the present invention. [Figure 17B] 11A-11C are screenshots illustrating an exemplary user interface for providing additional input prompts for repeating a method for visualizing cosmetic skin characteristics according to the present invention. [Figure 18A] 11A-11C are screenshots illustrating an exemplary user interface for displaying details of product recommendations for treating cosmetic skin characteristics in accordance with the present invention. [Figure 18B] 11A-11C are screenshots illustrating an exemplary user interface for displaying details of product recommendations for treating cosmetic skin characteristics in accordance with the present invention. [Figure 19] 16D is a variation of an exemplary user interface screenshot of FIG. 16D for displaying cosmetic skin characteristics in accordance with the present invention. [Figure 20] 1 is a screenshot showing an exemplary user interface for displaying details of a product recommendation for treating cosmetic skin characteristics in accordance with the present invention. [Figure 21] 1 is an exemplary user interface for visualizing cosmetic skin characteristics in accordance with the present invention. [Figure 22] 1 is an exemplary user interface for visualizing cosmetic skin characteristics in accordance with the present invention. [Figure 23] 1 is a screenshot showing an exemplary user interface including a heat map as an example of an image description for visualizing at least one cosmetic skin characteristic in accordance with the present invention. [Figure 24]24 is a screenshot showing an alternative variation of an image description for visualizing at least one cosmetic skin characteristic within the user interface of FIG. 23. [Diagram 25] 1 is a screenshot illustrating an exemplary user interface for visualizing at least one cosmetic skin characteristic in accordance with the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] It is known that when skin is exposed to stress (caused by UV, chronological aging, mental stress, environmental factors), skin can be damaged at various levels, including DNA level, cellular level, and tissue level.Such damage to skin can lead to skin defects.The existence of these skin defects significantly affects the visibility of skin, as described in the following examples. · When the stratum corneum is dry (winter / air conditions), light reflects more from the surface (surface reflection) which can promote micro-tissue formation on the skin that causes lines on the skin. When the dermis is damaged (by UV), the light is scattered less in the dermis (also known as dermal scattering) and the light can penetrate the skin (less subsurface reflection). As a result of less dermal scattering, the skin appears darker. When skin is chronically exposed to UV rays, it produces more melanin, which absorbs light and reduces subsurface reflection, so the skin appears darker.

[0011] The above skin defects appear to the consumer as visually imperceptible signs of poor skin quality. Consumers may consider these defects as subconscious impurities, but due to lack of conscious knowledge, they are unable to take measures to improve the defects. If the skin is subjected to chronic stress and / or untreated, these visually imperceptible impurities may eventually lead to visible and perceptible phenotypes (pigment spots, wrinkles, sagging).

[0012] The present invention relates to a method, apparatus, and system, and a graphical user interface, for visualizing at least one cosmetic skin characteristic of a subject, which may include a visually perceptible cosmetic skin characteristic or a visually non-perceptible cosmetic skin characteristic.

[0013] The method comprises: a) obtaining a first digital image of at least a portion of a face of a subject, the first digital image being selected from at least one region of an input image of the face; b) defining a plurality of tiles across the acquired first digital image; c) analyzing the first digital image for each of the defined plurality of tiles for at least one cosmetic skin characteristic; d) assigning a unique degree mark to each tile of the defined plurality of tiles based on the analyzed at least one cosmetic skin characteristic of the tile; e) displaying at least some of the plurality of tiles, each having a uniquely assigned single degree indicia, to visualize at least one cosmetic skin characteristic; Here, at least some of the displayed plurality of tiles display cosmetic skin characteristics that are better compared to cosmetic skin characteristic states of non-displayed tiles of the plurality of tiles based on the analyzed at least one cosmetic skin characteristic of the tile.

[0014] Surprisingly, it has been found that by displaying only some of the tiles that display a cosmetic skin characteristic condition that is better compared to the cosmetic skin characteristic condition of the non-displayed tiles of the plurality of tiles, a consumer can easily understand the cosmetic skin characteristic condition and therefore can make an informed decision to seek treatment to improve the cosmetic skin characteristic condition.

[0015] In particular, the technical effect of having non-display tiles is to minimize "information clutter" and "information overload" displayed on a single display, especially on a small form factor electronic portable device such as a mobile phone. A further technical effect is that the data processing resources required to generate a display in a short period of time can be reduced, thereby resulting in a better user experience when using the present invention (e.g., as a native application on the mobile phone or through a web application accessed through the mobile phone).

[0016] Prior to describing the present invention in detail, the following terms are defined, and terms not defined should be given their ordinary meanings as understood by those of ordinary skill in the art.

[0017] As used herein, “entropy” means the Shannon entropy (E) of a discrete random distribution (p(x)) and is defined by the following equation:

[0018]

number

[0019] E(p) represents the amount of information in a digital image or a color channel image in a color system after conversion of the digital image to a color channel image.

[0020] As used herein, "entropy statistics" refers to statistical methods that use entropy as a descriptive statistic to analyze a digital image or a color channel image. In a non-limiting example where the digital image is an RGB image, the entropy for each of the R (red), G (green), and B (blue) channels can be calculated separately. The entropy value of an image can be calculated at each pixel location (i,j) by calculating the entropy value of pixel values ​​in a two-dimensional region centered at (i,j). The two-dimensional region can be a portion of the color channel image. A programming software package such as Python can be used to calculate the entropy value.

[0021] As used herein, "cosmetic skin characteristics" include all skin characteristics that provide visual / aesthetic effects to an area of ​​the human body or that affect the appearance and / or feel of the skin. Some non-limiting examples of cosmetic skin characteristics can include skin topography, skin elasticity, skin tone, skin pigmentation, skin texture, skin pores, cosmetic skin inflammation, skin hydration, skin sebum level, acne, pigmented nevus, skin radiance, skin gloss, skin dullness, uneven tone, or skin barrier. The above cosmetic skin characteristics are standard terms, and corresponding definitions of cosmetic skin characteristics can be found in the following published references: "Handbook of cosmetic science and technology, 3rd Edition, pp. 217-219, 2002; ... rdThose skilled in the art will appreciate that the above-mentioned characteristics may be found in "Cosmetic Science and Technology-Theoretical Principles and Applications, editors Kazutami Sakamoto Robert Y. Lochhead, Howard I. Maibach, Yuji Yamashita, Elsavier, 2017" and "Cosmetic Dermatology: Products and Procedures, Editor(s): Zoe Diana Draelos, Blackwell Publishing Ltd, 2010". Cosmetic skin characteristics do not include skin characteristics associated with a medical condition or underlying medical condition.

[0022] As used herein, "perceptible cosmetic skin characteristics" refers to cosmetic skin characteristics that cannot be perceived or are not perceptible by a perceiver, i.e., a person, user, or subject. Cognition is derived from the word "perception," which refers to the organization, identification, and interpretation of sensory information to represent and understand the presented information or environment. All perception involves signals passing through the nervous system, which in turn result from physical or chemical stimulation of the sensory system. For example, vision involves light striking the retina of the eye, smell is mediated by odor molecules, and hearing involves pressure waves. Perception is shaped not only by the passive reception of these signals, but also by the learning, memory, expectations, and attention of the receiver. Perception can be divided into two processes: (1) related to processing sensory inputs, which transforms these low-level information into higher-level information (e.g., extracting shapes for object recognition), and (2) related to processing that is related to a person's concepts and expectations (or knowledge), repair mechanisms, and selective mechanisms (such as attention) that influence perception. For example, it may be considered that an object is visually imperceptible because the perceiver can see the object in process (1) but does not have the knowledge to perceive and know what the object represents / means in process (2).

[0023] As used herein, a "visually imperceptible cosmetic skin characteristic" includes any cosmetic skin characteristic that is not detectable by the naked eye or that is visually detectable by a consumer but is considered an imperceptible cosmetic skin characteristic because the consumer does not understand the cosmetic skin characteristic. Some non-limiting examples of visually imperceptible cosmetic skin characteristics that are not visually detectable by the naked eye include cosmetic skin irritation, skin sebum levels, or any underlying cosmetic skin characteristic.

[0024] As used herein, "Unaided" means without assistance from a diagnostic device.

[0025] As used herein, a "tile" includes a unit, such as a pixel, that forms part of a digital image, such that the "tile" forms the entirety of the digital image.

[0026] As used herein, "digital image data" includes image data acquired from an image acquisition device, including but not limited to a digital camera, a photo scanner, a computer readable storage medium capable of storing digital images, and any electronic device that includes a photographing capability. Digital image data may also include color channel images that are converted from RGB images to color channel images in a color system.

[0027] As used herein, "single degree indicia" includes all electronic visual representations, including but not limited to graphic symbols, numerical values, color codes, lighting techniques, and combinations thereof.

[0028] As used herein, "skin characteristic index" refers to a score that can be calculated based on a mathematical formula, or a model derived from statistical methods and data, or a look-up table (array of information). The skin characteristic index may be generated as a probability value indicating the state of at least a part of the cosmetic skin characteristic of the person's skin for a defined population of people, and preferably, the skin characteristic index is generated as a function of an entropy value defined by F (entropy value), the function being determined by a model established on a training data set, the training data set including (i) a plurality of color channel images of the defined population of people, each of the plurality of color channel images including facial skin of a person in the defined population of people, the facial skin including the cosmetic skin characteristic, and (ii) an associated class definition based on the cosmetic skin characteristic.

[0029] As used herein, "L * a * b *" refers to a commonly recognized color space defined by the International Commission on Illumination ("CIE"). The three coordinates are (i) the brightness of the color (i.e., L * = 0 produces black, L * = 100 indicates white diffusion), (ii) a color located between magenta and green (i.e., negative a * A positive a value indicates green. * values ​​indicate magenta), and (iii) the location of the color between yellow and blue (i.e., negative b * A positive b value indicates blue. * The value indicates yellow.

[0030] As used herein, "chromophore mapping" refers to the commonly recognized chromophore space for melanin mapping and hemoglobin mapping to determine melanin or hemoglobin concentrations that can be used as an indicator of overall skin tone. Mean melanin or mean hemoglobin may be calculated from the chromophore map data. In addition, skin tone uniformity can be measured from melanin uniformity or hemoglobin uniformity (e.g., standard deviation), which may also be calculated from the chromophore map data.

[0031] As used herein, "skin purity" refers to the appearance of at least a portion of a person's skin being free of skin imperfections, including but not limited to cosmetic skin characteristics that affect the irregular or non-uniform spectral characteristics composed of surface reflectance of skin topography morphology and / or subsurface reflectance of skin chromophores such as melanin, hemoglobin and / or keratinocytes, fibroblast-directed cellular metabolic products, as well as skin radiance, skin tone, and the like.

[0032] As used herein, "skin age" means apparent age, which refers to a person's skin age as visually estimated or perceived and compared to a standard age skin appearance based on appearance, preferably the person's face, preferably at least a portion of the person's face, more preferably at least a portion of a region of interest (ROI) on at least a portion of the person's face, even more preferably at least one ROI selected from the group consisting of the skin area around the eyes ("eye area"), the skin area around the cheeks ("cheek area"), the skin area around the mouth ("mouth area"), and combinations thereof.

[0033] As used herein, "skin tone" generally refers to the overall appearance of basic skin color or color uniformity. Skin tone is generally characterized over a large area of ​​skin. The area may be greater than 100 mm2, but larger areas are also contemplated, such as the entire facial skin or other body skin surfaces (e.g., arms, legs, back, hands, neck).

[0034] As used herein, "skin wrinkles" generally refers to folds, ridges or creases in the skin, including, but not limited to, fine lines, ultrafine lines, fine wrinkles, ultrafine lines, wrinkles, and lines. Skin wrinkles may be measured, for example, in terms of density and / or length.

[0035] As used herein, "skin radiance" generally refers to the amount of light that the skin reflects, and may also be referred to as skin gloss.

[0036] As used herein, "skin texture" generally refers to the topology or roughness of the skin surface.

[0037] As used herein, "skin tautness" generally refers to the firmness or elasticity of the skin.

[0038] As used herein, "skin sebum level" generally refers to the amount of sebum, an oily or waxy substance secreted by the sebaceous glands in the skin.

[0039] As used herein, "skin blemish" generally refers to a discoloration or uneven pigmentation of the skin (e.g., hyperpigmentation, mottled). Skin blemishes can be assessed, for example, with respect to density, size, and / or degree of discoloration.

[0040] As used herein, "skin care product" means a product that contains skin care actives and that regulates and / or improves the condition of the skin.

[0041] As used herein, "digital image" refers to a digital image formed by pixels in an imaging system, including but not limited to standard RGB, and the like, and under images acquired under different lighting conditions and / or modes. Non-limiting examples of digital images include color images (RGB), monochrome images, video, multispectral images, hyperspectral images, and the like. Non-limiting light conditions include light sources emitting white light, blue light, UV light, IR light, specific wavelengths of light, such as 100-1000 nm, 300-700 nm, 400-700 nm, or different combinations of the upper and lower limits of the above, or any integer combination of the ranges listed above, and the like. Digital images may be obtained from image acquisition devices, including but not limited to digital cameras, photo scanners, computer readable storage media capable of storing digital images, and any electronic device that includes a photographing capability.

[0042] In the following description, the system described is a system for visualizing cosmetic skin characteristics. Accordingly, the apparatus described is an apparatus for visualizing cosmetic skin characteristics, and the graphical user interface described is a graphical user interface for visualizing cosmetic skin characteristics. The graphical user interface may be further configured to provide product recommendations for treating at least one cosmetic skin characteristic.

[0043] In an exemplary embodiment, the system is a stand-alone imaging system (as shown in FIG. 2) located at a retail cosmetics counter for the purpose of analyzing and recommending cosmetics and skin care products. However, it is contemplated that the system and method may be configured for use anywhere, for example, via an electronic handheld device comprising an image acquisition unit and a display, as shown in FIG. 1, connected to an apparatus for generating a display on the display and a graphical user interface for visualizing cosmetic skin characteristics over a network.

[0044] system 1 is a schematic diagram illustrating a system 10 for visualizing cosmetic skin characteristics according to the present invention. The system 10 may include a network 100, which may be embodied as a wide area network (e.g., a cellular network, a public switched telephone network, a satellite network, the Internet), a local area network (e.g., Wireless Fidelity, Wi-Max, ZigBee™, Bluetooth™, etc.), and / or other forms of network functionality. Connected to the network 100 is a portable electronic device 12 and an apparatus 14 for generating a display on a display, a graphical user interface for visualizing cosmetic skin characteristics. An apparatus 104 is remotely located and connected to the portable electronic device via the network 100.

[0045] The portable electronic device 12 may be a mobile phone, tablet, laptop, personal digital assistant, and / or other data processing device configured to capture, store, and / or transfer digital images, such as digital photographs. Thus, the portable electronic device 12 may include an input device 12a for receiving user input, an image capture device 18, such as a digital camera, for capturing images, and an output device 12b for displaying the images. The portable electronic device 12 may also be configured for communication with other data processing devices via a network 100. The portable electronic device 12 may further include an image processing device (not shown) integrated with the image capture device 18 for analyzing the captured at least one color channel image using an entropy statistic to obtain an entropy value, and determining cosmetic skin characteristics of at least a portion of the person's skin based on the entropy value. The image processing device preferably comprises a processor having computer executable instructions. The portable electronic device 12 may further comprise a display generation unit (such as an electronic LED / LCD display, not shown) for generating a display for displaying content data describing the determined cosmetic skin characteristics. The device 14 may include a non-transitory computer-readable storage medium 14a (hereinafter, "storage medium") that stores image acquisition logic 144a, image analysis logic 144b, and graphical user interface (hereinafter, "GUI") logic 144c. The storage medium 14a may include random access memory (SRAM, DRAM, etc.), read-only memory (ROM), registers, and / or other forms of data processing storage hardware. The image acquisition logic 144a, image analysis logic 144b, and GUI logic 144c define computer-executable instructions. A processor 14b is coupled to the storage medium 14a, and the processor 14b is configured based on the computer-executable instructions to perform a method 200 for visualizing cosmetic skin characteristics of a subject according to the present invention, as described herein below with respect to the process flow diagrams of Figures 4A-4C and the flow chart of Figure 5.

[0046] method 4A and 5, when the processor 14b is initiated, the processor 14b generates a first digital image 51 of at least a portion of the face to be captured, for example, via the image capture logic 144a at step 202. The processor 14b defines a plurality of tiles 54 over the captured image data 20 (step 204). The plurality of tiles 54 may be adjacent to define a tile map 55 as shown in FIG. 4B. At step 206, the processor analyzes the image data for each of the defined plurality of tiles 54 for at least one cosmetic skin characteristic. At step 208, the processor assigns a unique degree mark 40 to each tile 54 of the defined plurality of tiles based on the analyzed at least one cosmetic skin characteristic. At least a portion of the plurality of tiles, each having a uniquely assigned degree mark, are displayed at step 210 to visualize the at least one cosmetic skin characteristic as shown in FIG. 4C. By analyzing image data of an input digital image provided by a user (consumer) and organizing and displaying the analyzed image data for each of a plurality of defined tiles in a single screenshot, the method 200 according to the present invention allows the user / consumer to easily identify cosmetic skin characteristics while avoiding the cumbersome task of navigating a user interface that displays information in separate windows under different lighting systems required to visualize the cosmetic skin characteristics. For example, certain conditions such as UV light may be required to visualize ultraviolet spots that are not perceptible to the naked eye.

[0047] In an exemplary embodiment, a second digital image with a uniquely assigned single degree mark for each tile may be inserted into the first digital image 51. It will be appreciated that the size of the tiles 54 may be defined by the number of pixels on the horizontal side (tile width, W) and the number of pixels on the vertical side (tile height, H). In an exemplary method according to the present invention, each tile may include a tile size of 100x100 pixels or less, between 1x1 pixels and 100x100 pixels, between 2x2 pixels and 100x100 pixels, between 5x5 pixels and 90x90 pixels, between 40x40 pixels and 70x70 pixels, or different combinations of the upper and lower limits of the above, or any integer combination within the ranges of the above. The technical effect of having tile sizes in the above ranges is to allow for shorter processing times for analysis of image data and therefore allowing the display to visualize at least one cosmetic skin characteristic in a shorter time.

[0048] Referring to FIG. 1, a network 100 may be used to obtain digital images from a portable electronic device 12 and transmit the digital images to a device 14 for use in a method 200 according to the present invention. An input device 12a may be coupled to or integral with the portable electronic device 12 for receiving user input to initiate a processor 14b. The portable electronic device 12 may include an output device 12b for displaying multiple tiles, each having a uniquely assigned single degree mark. The input device 12a may include, but is not limited to, a mouse, a touch screen display, and the like. The output device 12b may include, but is not limited to, a touch screen display, a non-touch screen display, a printer, a projector for projecting a facial image map 30 onto a display surface, such as a mirror as described below with respect to FIG. 2.

[0049] FIG. 2 is a perspective view of system 10 configured as a stand-alone imaging system located at a retail cosmetics counter for the purpose of visualizing at least one cosmetic skin characteristic and recommending cosmetic and skin care products based on the visualized at least one cosmetic skin characteristic. FIG. 3 is a block diagram of system 10 of FIG. 2. With reference to FIGS. 2 and 3, system 10 comprises a housing 11 for device 14 of FIG. 1 connected to image capture device 18 for acquiring digital images of a subject for visualizing at least one cosmetic skin characteristic. With reference to FIG. 2, system 10 may include a mirror 16, and image capture device 18 may be mounted behind mirror 16 in housing 11 such that image capture device 18 may be hidden from view. Image capture device 18 may be a digital camera, an analog camera connected to a digitizing circuit, a scanner, a video camera, or the like. System 10 may include lighting 30, such as LED lights, arranged around housing 11 to form an LED lighting system that assists in generating a digital image of the subject. System 10 has an input device 112a for receiving user input. System 10 may further include an output device 112b, such as a projector configured to receive and project face map 30 for display on mirror 16. The projector is not shown in FIG. 2 as it may be a peripheral component separate from housing 11 but coupled to device 14 to form system 10. System 10 may further include a second output device 112c, such as one or more speakers optionally coupled to an amplifier for generating audio guidance output, to complement and / or enhance the overall consumer experience.

[0050] To explain how the system 10 and method 200 function to visualize at least one cosmetic skin characteristic in accordance with the present invention, it is useful to understand how a digital image of a subject's face is acquired in step 202, how tiles are defined in step 204, how the image data is analyzed in step 206, how a single degree mark is uniquely assigned to each tile in step 208, and how the tiles are displayed in step 210. Accordingly, steps 202, 204, 206, 208, 210 of the method 200 in accordance with the present invention are described below in terms of individual processes for implementing each step. Each process may also be described as a subroutine, i.e., a sequence of program instructions that implement the corresponding step in accordance with the method 200 in accordance with the present invention.

[0051] Acquire digital images Step 202 of acquiring a digital image according to the method 200 of the present invention will now be described with reference to Figures 6A, 6B and 6C, which are a series of process flow diagrams illustrating how a first digital image is acquired, and Figure 7 is a flow chart of a process 300 of acquiring a digital image corresponding to step 202.

[0052] An input image 50a of a face 1 is shown in FIG. 6A. The input image 50a may be captured by a user using a camera 18 in step 302 of the process 300, for example, as shown in FIG. 7. FIG. 6B illustrates step 304 of cropping the input image 50a to obtain edited image data 50b including at least a portion of the face. The input image 50a may be cropped by identifying anchor features 1a of the face, including but not limited to facial features such as the eyes, nose, nostrils, corners of the mouth, etc., and cropped accordingly. As shown in FIG. 6B, the eyes are depicted as anchor features 1a, but it will be understood that this is merely an example and any salient or detectable facial feature or features may be the anchor features. The edited image data 50b may be the first digital image 51 obtained in step 308. Alternatively, as shown in FIG. 6C, the edited image data 50b may be further processed by cropping to remove one or more unnecessary portions of the input image 50a, thereby obtaining a first digital image 51 including at least a portion of the face 1 defined by a boundary line 52 in step 308. The obtained first digital image 51 may include at least one region of interest (ROI) 2 of at least a portion of the face 1 defined by the boundary line 52. The ROI 2 may be the entire face 1, preferably at least a portion of the face, more preferably one or more skin regions defining at least a portion of the face 1. Details of how the skin regions are defined are described below with reference to FIGS. 10A-10C and the flow chart of FIG. 11.

[0053] Optionally, the process 300 may include step 306, where the ROI2 may be selected from the group consisting of a skin area around the eye ("eye area 2a"), a skin area around the cheek ("cheek area 2b"), a skin area around the mouth ("mouth area 2c"), and combinations thereof, and preferably the ROI2 is a portion of at least a portion of the subject's face 1, and more preferably the acquired first digital image defines the left or right side of the face 1. The ROI2 may include an area of ​​at least 5%, 10% to 100%, 25% to 90% of the acquired first digital image.

[0054] Defining the tiles FIG. 8 is a photograph showing a plurality of tiles 54 on the first digital image data 51. FIG. 9 is a flow chart showing a process 400 for defining a plurality of tiles 54 on the first digital image data 51. Referring to FIG. 8, the first digital image data 51 includes at least a portion of a face 1 defined by a boundary line 52 as described above with reference to FIG. 6C. The process 400 includes defining a perimeter 53 surrounding the boundary line 52 surrounding the acquired first digital image (step 402). The acquired first digital image 51 is formed by a total number of pixels, for example, the acquired first digital image 51 may have a number of pixels determined in step 304 or step 306 depending on the image size after cropping of the input image 50a. Thus, an overall image size based on the acquired first digital image 51 may be defined in step 404. For example, if the tile size is set to 40x40 pixels to 70x70 pixels, then the number of tiles 54 forming multiple tiles 54 across the first digital image 51 acquired in step 406 is obtained accordingly by dividing the overall image size by the particular tile size.

[0055] Analyzing image data 10 is a flow chart illustrating a process 500 for analyzing image data for each of a defined plurality of tiles in accordance with the present invention. The process 500 may begin at step 502 by extracting at least one color channel from a captured first digital image to provide an extracted color channel image for analysis to obtain an entropy value for determining cosmetic skin characteristics. In the following description, the at least one color channel image may be selected from the group consisting of L color channel images, a-channel images, b-channel images, and c-channel images from a RGB color space, and combinations thereof. * a * b * an image of a color system, the entropy value being selected from the group consisting of an L-entropy value, an a-entropy value, a b-entropy value, a c-entropy value, and combinations thereof, and the function having the following formula: Skin characteristic index=A+B×(L-entropy value)+C×(a-entropy value)+D×(b-entropy value)+E×(c-entropy), In the formula, A, B, C, D, and E are constants, and at least one of B, C, D, and E is not 0.

[0056] However, it will be understood that at least one color channel may be a chromophore system, and at least one color channel may be a melanin channel or a hemoglobin channel. The color system may also be an HSL / HSV color system, and a CMYK color system.

[0057] The extracted color channels may be filtered at step 504 and the filtered color channels may be analyzed using entropy statistics to generate entropy values ​​for determining at least one cosmetic skin characteristic at step 506. It will be appreciated that the filtered color channels may also be analyzed using other descriptive statistics including, but not limited to, standard deviation, mean, etc. The technical effect of using entropy statistics is that it has a higher accuracy compared to other descriptive statistics and also allows for a faster processing time of the method according to the present invention.

[0058] In step 506, cosmetic skin characteristics of at least one portion of the person's skin are determined based on the entropy value.

[0059] At least one color channel image is * a * b * The image may be in a color system selected from the group consisting of a RGB color system, a HSL / HSV color system, and a CMYK color system.

[0060] Table 1 below sets out each entropy value having a corresponding color channel image and a corresponding cosmetic skin characteristic determined based on the entropy value. The color channel image in Table 1 is an L channel image selected from the group consisting of an L channel image, an a-channel image, a b-channel image, a c-channel image, and combinations thereof. * a * b * This is an image of a color system.

[0061] [Table 1]

[0062] Determining the cosmetic skin characteristic may include generating a skin characteristic index as a probability value indicative of the status of the cosmetic skin characteristic of at least a portion of the person's skin for a defined population of people. Specifically, in a visual perception test, a consumer may be asked to rank digital images (e.g., photographs) of the defined population of people for the cosmetic skin characteristic based on a predefined scale. The ranked digital images may be stored as a database to be analyzed according to method 500 to determine the entropy value that has the highest correlation with the cosmetic skin characteristic.

[0063] Alternatively, the skin characteristic index may be generated as a function of the entropy value defined by a function F(entropy value), the function being determined by a model established on a training data set. The training data set may include (i) a plurality of color channel images of a defined population of people, each of the plurality of color channel images including facial skin of a person in the defined population of people, the facial skin including cosmetic skin characteristics, and (ii) an associated class definition based on the cosmetic skin characteristics. Techniques for constructing the training data set are known to those skilled in the art in the field of image processing methods and will not be described further.

[0064] The model may be a regression model or a classification model, preferably a linear regression model, more preferably a machine learning linear regression model, and most preferably a machine learning support vector regression (SVR) model. The SVR model is a specific example of a Support Vector Machine (SVM) model. The SVM model may also be a support vector classification model.

[0065] The use of the SVR model allows for accuracy, repeatability, and speed advantages in the performance of the method when implemented as a native application on a portable electronic device. In particular, the weight of the SVR model allows native applications to have a smaller hardware footprint, so that the method according to the present invention can be easily deployed on portable electronic devices such as mobile phones having operating systems (OS), including but not limited to iOS for Apple™ phones or Android OS for Android phones.

[0066] The classification model may be used to classify consumers into multiple groups, preferably two groups, more preferably three groups, each group having a different degree of the same cosmetic skin characteristic condition, so as to define an associated class definition based on the numerical value of the skin characteristic index. For example, the method may display a heat map configured to classify skin areas into a high level cosmetic skin characteristic condition, or a low level cosmetic skin characteristic condition based on a threshold value assigned to each of the groups.

[0067] The at least one color channel image is selected from the group consisting of an L color channel image from a RGB color system, an a-channel image, a b-channel image, and a c-channel image, and combinations thereof. * a * b * an image of a color system, the entropy value being selected from the group consisting of an L-entropy value, an a-entropy value, a b-entropy value, a c-entropy value, and combinations thereof, and the function having the following formula: Skin characteristic index=A+B×(L-entropy value)+C×(a-entropy value)+D×(b-entropy value)+E×(c-entropy), In the formula, A, B, C, D, and E are constants, and at least one of B, C, D, and E is not 0.

[0068] It will be understood that the constants A, B, C, D, and E may vary based on the size and content of the training data set, and may be any numerical values ​​generated by the model based on the training data set.

[0069] In particular, each of the above entropy values ​​may be used alone or in combination with another one of the entropy values. For example, using a single entropy value can result in faster calculation speeds, which allows for smaller devices with very basic hardware to be used, thereby resulting in a more efficient and cost-effective product.

[0070] The at least one color channel image may be an L channel image, the entropy value is an L-entropy value, where C, D, E each have a value of 0, and the generated skin characteristic index indicates skin purity, skin tone, or skin radiance.

[0071] Surprisingly, the L-entropy value of the L color channel image was found to have the highest correlation with skin purity.

[0072] The technical effect of selecting an L-channel image as at least one color channel image for the analysis step to obtain an L-entropy value and determining skin purity based on the L-entropy value according to the method of the present invention is that the L-entropy value has the highest correlation (r=0.89) with skin purity compared to other entropy values ​​based on the analysis of the color channel image. The following is the data generated based on the correlation with the results from the visual perception test using statistical analysis with Pearson correlation coefficient (r). The correlation results are shown in Table 2 below.

[0073] [Table 2]

[0074] A higher Pearson correlation coefficient (r) means that the entropy value is a more contributing factor to the state of the cosmetic skin characteristic tested in the visual perception test. Specifically, the visual perception test is carried out based on a predetermined number of panelists=302 with panelist age=20-50. The panelists were asked to rank photos of people on skin purity (as an example of cosmetic skin characteristic) on a scale of 1 to 5. Based on the visual perception test results and the correlation results above, it was found that the L-channel entropy value of the filtered image (by frequency filter) has the highest correlation with the skin purity characteristic. Thus, the use of the L-entropy value of the L-channel to determine the skin purity of at least a part of the skin of a person in a digital image can convert the skin purity from a visually imperceptible cosmetic skin characteristic to a cosmetic skin characteristic that is explainable to the consumer in a manner appropriate for the consumer.

[0075] The at least one color channel image may be an a-channel image, the entropy value is an a-entropy value, where B, D, E each have a value of 0, and the generated skin characteristic index is indicative of skin inflammation.

[0076] The at least one color channel image may be a b-channel image, the entropy value is a b-entropy value, where B, C, E each have a value of 0, and the generated skin characteristic index is indicative of skin pigmentation or skin dullness.

[0077] The at least one color channel image may be a c-channel image, the entropy value being a c-entropy value, where B, C, and D each have a value of 0, and the generated skin characteristic index is indicative of a skin topography, preferably selected from the group consisting of pores, fine lines, wrinkles, sagging, skin elasticity, and combinations thereof.

[0078] Preferably, the age of the subject and the mean age of the defined population of people may each independently be between 18 and 60 years, preferably between 20 and 40 years, more preferably between 25 and 35 years, and even more preferably between 28 and 32 years.

[0079] Optionally, filtering the extracted color channel images may include using a frequency filter, preferably the frequency filter is selected from the group consisting of a Difference of Gaussian (DoG) filter, a Fast Fourier Transform (FFT) filter, a Wavelet transformation filter, and combinations thereof, more preferably the filter is a DoG filter. Optionally, method 200 may further include applying image correction factors to the filtered color channels prior to analyzing the filtered color channels.

[0080] 10, analyzing the image data may include analyzing at least two color channels, and more preferably three color channels. In particular, the red, yellow, and blue color channels can be described as follows: the red color channel is L; * a * b * If we are in color space, a-entropy is the entropy value of the filtered red color channel. If the yellow color channel is L * a * b * If in a color system, the b-entropy is the entropy value of the filtered yellow color channel. If the blue channel corresponds to the texture channel, the c-entropy is the entropy value of the blue color channel.

[0081] display The method according to the present invention may further comprise generating an image description corresponding to the above-mentioned generated skin characteristic index for visualizing the cosmetic skin condition. The image description may comprise a heat map (as shown in Fig. 16D, Fig. 25) and an aggregate score representation of the cosmetic skin characteristics, cosmetic skin characteristic terms such as skin age shown in the fourth area 194 of Fig. 16D, skin purity score shown in feature 934 of Fig. 25. The aggregate score may be calculated based on the above-mentioned generated skin characteristic index.

[0082] 11A-11C are process flow diagrams detailing the process of displaying multiple tiles according to the method of the present invention. FIG. 12 is a flow chart illustrating the process of displaying multiple tiles. FIG. 11A is a photograph showing a second digital image 60 inserted on a first digital image 51. The second digital image 60 includes at least a portion of a subject's face displaying multiple tiles 54, each having a uniquely assigned single degree mark 40. FIG. 11B shows three zones, a first zone 110, a second zone 120, and a third zone 130, displayed on the acquired first digital image based on the multiple tiles 54, each having a uniquely assigned single degree mark. Each zone 110, 120, 130 identifies a respective region of interest (ROI) 2 on the subject's face 1 as described above with reference to FIGS. 6A-6C and 7. 11C differs from FIG. 11B in that the border 52 and perimeter 53 are visible in the second digital image 60 of FIG. 11B but not in the second digital image 60 of FIG. 11C. The first zone 110 may include a first zone line having a first zone color 110a, the second zone 120 may include a second zone line having a second zone color 120a, and the third zone 130 may include a third zone line having a third zone color 130a. Based on the analyzed image data of the tiles 54 in each zone, the color of each zone line may be different to better visually distinguish tiles that visualize cosmetic skin characteristics that may be normal, beautiful, or vulnerable relative to other zones of the subject, as shown, for example, in the exemplary user interface of FIG. 16D.

[0083] 12 is a flow chart illustrating a process 600 for displaying a plurality of tiles according to the present invention. The process 600 may begin at step 602 where a processor reads the analyzed image data of each tile 54 and uniquely assigns a single degree mark to each tile 54 of the plurality of tiles based on the analyzed at least one visual cosmetic skin characteristic of the tile 54 (step 604). When the single degree mark is illuminated, the analyzed image data of each tile may be converted to reflect the luminance of the corresponding illumination in each tile at step 606. In a representative example, the zone 110 may have a lower illumination intensity in each of the tiles within the zone 110 relative to the illumination intensity in each of the tiles within the zone 120. Further, at step 608, the zones may be defined such that the first zone 110 may correspond to an eye zone, the second zone 120 may correspond to a cheek zone, and the third zone 130 may correspond to a mouth zone. An average index for the tiles of each zone may be calculated to assign product recommendations to the zones to treat the displayed cosmetic skin characteristics within the zones, and a diagnosis of the skin condition correlated with the displayed cosmetic skin characteristics according to the corresponding zone may be generated. Specifically, the method 200 may further include displaying at least one product recommendation to treat the displayed cosmetic skin characteristic.

[0084] 13A-13D are process flow diagrams illustrating a method for visualizing at least one cosmetic skin characteristic according to the present invention. FIG. 14 is a flow chart illustrating a method 700 for visualizing at least one cosmetic skin characteristic according to the present invention. FIG. 13A is a color image showing a first digital image of at least a portion of a subject's face displayed in step 702 of the method 700 of FIG. 14. FIG. 13B is a color image showing a second digital image of at least a portion of a subject's face and a plurality of tiles, each having a uniquely assigned single degree mark, which is inserted onto the first digital image in step 704. If desired, the first digital image may be converted to grayscale as shown in FIG. 13C to provide better contrast between the plurality of tiles, each having a uniquely assigned single degree mark, and the first digital image. In step 706, three zones are displayed on the second digital image based on the plurality of tiles, each having a uniquely assigned single degree mark.

[0085] Figure 15 is a flow chart illustrating a variation of the method 700 of visualizing at least one cosmetic skin characteristic as shown in Figure 14. Following step 706 of the method 700 of Figure 14, at least one product recommendation is displayed at step 708. At step 710, the user is prompted to select to end the method 700, which ends at step 712 if the user selects "yes." If the user selects "no," the steps of method 200 of Figure 5 are performed and method 700 returns to step 708.

[0086] Human Machine User Interface The present invention also relates to a human-machine user interface (hereinafter "user interface") for providing product recommendations for treating at least one cosmetic skin characteristic. The user interface may be a graphical user interface on a portable electronic device including a touchscreen display / display with an input device and an image acquisition device. The user interface may include a first region of the touchscreen display displaying a first digital image of at least a portion of the subject's face acquired from the image acquisition device, and a second digital image inset onto the first digital image, the second digital image having at least a portion of the subject's face, the displayed tiles each having a uniquely assigned single degree mark. The user interface may further include a second region of the touchscreen display different from the first region, the second region displaying a selectable icon for receiving user input, and an image of at least one product recommendation for treating the displayed cosmetic skin characteristic is displayed on the touchscreen display when the user activates the selectable icon.

[0087] 16A-16E are screenshots each showing an exemplary user interface for visualizing a cosmetic skin characteristic in accordance with the present invention, each of which cooperates with the other. Although FIGs. 16A-16E are described as a series of user interfaces provided in a sequential manner in response to a preceding user interface, it will be understood that the user interfaces of FIGs. 16A-16E can be programmed in a number of ways to define an overall user interface for visualizing at least one cosmetic skin characteristic in accordance with the method of the present invention, as described above. Preferably, all of the user interfaces of FIGs. 16A-16E define an exemplary user interface for visualizing a cosmetic skin characteristic in accordance with the present invention.

[0088] FIG. 16A shows a user interface 160 for receiving a first user input, preferably the first user input is the user's age. The user interface 160 may include a first area 162 for receiving the first user input. The first area 162 may include one or more user input mechanisms 164 for receiving the first user input. The user input mechanisms 164 may be, for example, selectable input icons corresponding to a predefined user characteristic, such as the user's age as shown in FIG. 16A. The user interface 160 may further include a second area 166 including corresponding instructions for the user to provide the first user input. The second area 166 may be located above the first area 162 to provide a more user-friendly interface. The user interface 160 may be part of a start option for starting the method 200 according to the present invention.

[0089] FIG. 16B illustrates a user interface 170 for receiving a second user input, preferably a cosmetic skin characteristic that concerns the user. The cosmetic skin characteristic can be described as a skin concern of the user. The user interface 170 may be provided in response to a selection of the first user input from the user input mechanism 164 of FIG. 16A. The user interface 170 may include a first region 172 for receiving the second user input. The first region 172 may include one or more user input mechanisms 174 for receiving the second user input. The user input mechanism 174 may be, for example, a selectable input icon corresponding to a given skin concern. The first region 172 may further include a description region 173 corresponding to the one or more input mechanisms 174, the description region 173 including a brief description of the cosmetic skin characteristic or skin concern. The user interface 170 may further include a second region 176 including corresponding instructions to the user for providing the user input. The second area 176 may be disposed above the first area 172 to provide a more user-friendly interface.

[0090] Figure 16C shows a user interface 180 for acquiring a user input image. The user interface 180 may include a first region 182 having instructions for aligning an anchor feature (such as an eye) to acquire a first digital image according to the process described in Figure 7. The user interface 180 may be provided in response to a selection of a second user input via one or more user input mechanisms 174 of Figure 16B.

[0091] FIG. 16D illustrates a user interface 190 for displaying at least one cosmetic skin characteristic. The user interface 190 may be provided after an input image of a user is acquired in the user interface 180 of FIG. 16C. The user interface 190 may include a first area 191 that displays a plurality of tiles, each having a uniquely assigned single degree mark, for visualizing at least one cosmetic skin characteristic according to the method of the present invention. The first area 191 may display similar features as shown in FIG. 13D, except that only the lines defining the plurality of tiles are turned off and / or set as invisible layers. The first area 191 may include a first zone 110 corresponding to an eye zone of at least a portion of the user's face, a second zone 120 corresponding to a cheek zone of at least a portion of the user's face, and a third zone 130 corresponding to a mouth zone of at least a portion of the user's face. As shown in FIG. 16D, the zone results may be displayed in a third area 193, whereby the zone results include an index that may be generated for each zone based on a relative comparison of the indices of the zones within at least a portion of the user's face. In an exemplary embodiment, depending on the zone results, the first zone 110 may be described as a normal / beautiful / weak zone, the second zone 120 may be described as a normal / beautiful / weak zone, and the third zone 130 may be described as a normal / beautiful / weak zone. Preferably, each zone may have a different description based on the relative difference of the zone results. The user interface 190 also includes a second area 192 for receiving a third user input. The second area 192 may include one or more user input mechanisms 1921 for receiving the third user input. The user input mechanism 1921 may be, for example, a selectable input icon for proceeding to a next step of the method according to the present invention, or the like. Optionally, the user interface 190 may include a fourth area 194 for displaying the user's skin age based on the analyzed at least one cosmetic skin characteristic of each tile of the plurality of tiles based on the acquired first digital image of at least a portion of the user's face.

[0092] FIG. 16E illustrates a user interface 200 including a first area 201 for displaying a product recommendation 210. The user interface 200 may be provided in response to selection of a user input feature 1921 from the user interface 190 of FIG. 16D. Optionally, the user interface 200 may include a second area 202 for providing details of the product recommendation 210. Preferably, the user interface 200 may include a third area 203 for receiving a fourth user input, such as, for example, a request for assistance from a product consultant for inquiring about and / or purchasing the product recommendation 210. The third area 203 may include one or more user input mechanisms 2031 for receiving the fourth user input. The user input mechanism 2031 may be, for example, a selectable input icon for proceeding to a next step of a method according to the present invention.

[0093] Figure 17A is a screenshot showing an exemplary user interface 300 for providing additional input prompts according to step 710 of method 700 for visualizing cosmetic skin characteristics in accordance with the present invention, as described with reference to Figure 15. User interface 300 may be provided in response to selection of user input mechanism 1921 of Figure 16D, or may be automatically generated by a processor after user interface 200 of Figure 16E.

[0094] 17A, the user interface 300 may include a first area 301 for receiving a fifth user input, for example for terminating the method according to step 712 of the method 700, and a second area 302 for receiving a sixth user input, for example for continuing with a next step according to the method of the present invention. The first area 301 and the second area 302 may include one or more user input mechanisms 3011, 3021 for receiving a fourth user input. The user input mechanisms 3011, 3021 may be, for example, selectable input icons for proceeding to a next step of the method of the present invention. Optionally, the user interface 300 may include a third area 303 including a timer mechanism for setting a period for the user to provide the fifth or sixth user input. FIG. 17B is a screenshot showing an exemplary user interface 400 for displaying step 712 of the method 700. The user interface 400 may include a first area 401 for showing content for confirming to the user the termination of the method 700.

[0095] 18A and 18B are screenshots illustrating example user interfaces 500, 600, respectively, for displaying details of product recommendations and how the product recommendations function to treat cosmetic skin characteristics in accordance with the present invention.

[0096] In an exemplary embodiment, the present invention relates to a user interface for providing product recommendations for treating at least one cosmetic skin characteristic, the user interface being on a portable electronic device including a display and an image capture device. The user interface may include a first region of the display displaying a first digital image of at least a portion of a subject's face captured from the image capture device and a second digital image inserted onto the first digital image, the second digital image having at least a portion of the subject's face, the displayed tiles each having a uniquely assigned single degree mark, a second region of the display distinct from the first region, the second region displaying a voice input component for receiving user input, the second region displaying an image of at least one product recommendation for treating the displayed cosmetic skin characteristic on the touch screen display when the user activates the voice input component, and a third region of the display distinct from the first region, displaying an image of at least one product recommendation for treating the displayed cosmetic skin characteristic.

[0097] FIG. 19 is a screenshot showing a variation of the exemplary user interface 700 of FIG. 16D for displaying cosmetic skin characteristics according to the present invention. Specifically, the user interface 700 of FIG. 19 has substantially the same features as the user interface 190 of FIG. 16D, and differs in the following additional features including a fifth area 195 for displaying at least one product recommendation 210, a sixth area 196 for receiving user input, such as to continue with a next step according to the method of the present invention, such as to add an additional product recommendation, and a seventh area 197 for receiving user input, such as to terminate the method of the present invention. The fifth area 195 and the sixth area 196 may include one or more user input features for receiving respective user input. The user input mechanism may be, for example, a selectable input icon for proceeding to a next step of the method of the present invention.

[0098] 20 is a screenshot illustrating an exemplary user interface 800 for displaying one or more product recommendations in accordance with the present invention. User interface 800 may be provided in response to selecting a user input function in sixth region 196. User interface 800 has substantially the same characteristics as user interface 700, differing in that user interface 800 displays multiple product recommendations 210.

[0099] The method according to the invention may comprise displaying at least one region of interest (ROI), preferably one to three ROIs, more preferably three ROIs, based on a plurality of tiles each having a uniquely assigned single degree marking. Preferably, the displayed one of the three ROIs displays a cosmetic skin characteristic condition that is more favorable compared to a cosmetic skin characteristic condition of another one of the three ROIs based on the analyzed at least one cosmetic skin characteristic of the tile.

[0100] 21 is a screenshot of an exemplary user interface 810 for displaying cosmetic skin characteristics. Specifically, the user interface 810 has substantially the same features as the user interface of FIG. 16D, except that multiple tiles are not displayed. Specifically, the user interface 810 includes three ROIs, each having a border defining a corresponding ROI. Specifically, the three ROIs include a first ROI 812 defining an eye region, a second ROI 814 defining a cheek region, and a third ROI 816 defining a mouth region. The cosmetic skin characteristics are dry skin / skin texture, and thus image descriptions 818, 820, 822 describing the condition of the cosmetic skin characteristics are placed next to each of the three ROIs 812, 814, 816. For example, the third ROI 816 may have an image description 822 that indicates that the third ROI 816 is a beauty zone on the face based on the analyzed cosmetic skin characteristics of the non-displayed tiles.

[0101] 22 is a screenshot of an exemplary user interface 830 for displaying cosmetic skin characteristics. Specifically, the user interface 830 has substantially the same features as the user interface 820 of FIG. 21, except that tiles 54 are displayed for each of the three ROIs 812, 814, 816. It will be appreciated that tiles 54 may be configured to be displayed for a single ROI, e.g., displaying only ROI 816, which indicates a beauty zone based on image description 822.

[0102] FIG. 23 shows a partial view of an exemplary user interface 900 including an image description 901 superimposed on a digital image 51 for visualizing at least one cosmetic skin characteristic according to the present invention. The image description 902 includes a heatmap generated based on the entropy values ​​output from the method 500 described above. The heatmap includes a first heatmap section 906 based on low entropy values ​​corresponding to a better cosmetic skin characteristic condition. The heatmap 904 further includes a second heatmap section 908 based on high entropy values ​​corresponding to a poorer cosmetic skin characteristic condition. The first heatmap section 906 is formed from a first plurality of tiles that are visually different from the second plurality of tiles in the second heatmap section 908. For example, the first plurality of tiles are transformed to display colors different from the colors of the second plurality of tiles. A non-displayed heatmap section 910 (hereinafter "hidden heatmap section 910") corresponds to entropy values ​​between the high entropy value and the low entropy value. The heat map segments may be configured as follows to display entropy information related to cosmetic skin characteristic conditions and skin characteristic indices, as outlined in Table 5 below.

[0103] [Table 3]

[0104] Figure 24 shows an alternative variation of an image description 902 for visualizing at least one cosmetic skin characteristic within the user interface 900 of Figure 23. The image description 920 differs from the image description 902 of Figure 19 in that the image description 920 includes a displayed region of interest (ROI) 922, which is transformed to display a color to indicate a poorer cosmetic skin characteristic condition relative to another non-displayed region of interest (ROI) 924 that corresponds to a better cosmetic skin characteristic condition. An advantage of only displaying a single heatmap segment (see Figure 20) or ROI is that a consumer viewing the user interface is not overloaded with too much visual information.

[0105] 25 is a screenshot illustrating an exemplary user interface 930 for visualizing at least one cosmetic skin characteristic according to the present invention, where the at least one cosmetic skin characteristic is skin purity. The user interface 930 differs from the user interface 902 of FIG. 19 in that the user interface 930 includes alternative text 932 describing the cosmetic skin characteristic and a cohesion score 934 based on the generated skin characteristic index. The user interface 930 may further include a gauge 936 and a gauge marker 938 along the gauge 936 that represents an aggregate score on a scale of 0 to 100. The gauge 936 is a different way of visualizing the cohesion score 934 and may be optional. The color of the gauge 936 may be configured to indicate a gradient of colors representing the first heatmap section 904 and the second heatmap section 906.

[0106] The method for determining cosmetic skin condition according to the present invention described herein may further include a step of tracking cosmetic skin characteristics over a predetermined period of time. For example, the user interface 930 as shown in FIG. 21 may include a first selectable icon 940 that, when selected, causes instructions to be received and steps to be executed by the processor to generate a calendar or schedule and create a cosmetic skin characteristic diary to track the improvement of cosmetic skin characteristics. For example, when the consumer uses on day 1, the date and face analysis are recorded and stored in memory. Subsequently, whenever the consumer uses the method according to the present invention in the future (after a predetermined period of time, one week, one month, six months), the consumer's facial skin is analyzed again and the consumer can compare how their facial skin looks after the predetermined period of time against day 1. The method according to the present invention may be configured to be a downloadable software application stored as a native application on a mobile electronic device or a web application that can be accessed via a consumer-specific login account, so that the consumer can perform a self-skin analysis based on the method according to the present invention and observe and / or monitor the improvement (reduction of ROI with poor cosmetic skin characteristic status) over a period of time.

[0107] The user interface 930 may further include a second selectable icon 942 that, when selected, enables a method for determining cosmetic skin characteristics in accordance with the present invention to be repeated, for example, method 500 described above.

[0108] Representative embodiments of the present disclosure described above can be listed and described in the following paragraphs. A. A method for visualizing at least one visually non-perceptible cosmetic skin characteristic of a subject, comprising: a) acquiring first digital image data of at least a portion of a face of a subject; and b) defining a plurality of tiles across the acquired image data. c) analyzing the image data for each of the defined plurality of tiles for at least one visually non-perceptible cosmetic skin characteristic; d) assigning a unique degree mark to each tile of the defined plurality of tiles based on the analyzed at least one visually imperceptible cosmetic skin characteristic of the tile; e) displaying a plurality of tiles, each having a uniquely assigned single degree indicia to visualize at least one visually imperceptible cosmetic skin characteristic.

[0109] B. The method of paragraph A, wherein the acquired first digital image data includes at least one region of interest (ROI) of at least a portion of the subject's face, the ROI being selected from the group consisting of the skin area around the eye ("eye area"), the skin area around the cheek ("cheek area"), the skin area around the mouth ("mouth area"), and combinations thereof, preferably the ROI is a portion of at least a portion of the subject's face, and more preferably the acquired first digital image data defines a left or right side of the face.

[0110] C. The method of paragraph A or B, wherein each tile comprises a tile size of 100x100 pixels or less, preferably between 1x1 pixels and 100x100 pixels, more preferably between 2x2 pixels and 100x100 pixels, more preferably between 5x5 pixels and 90x90 pixels, and even more preferably the tile size is between 40x40 pixels and 70x70 pixels.

[0111] D. The method of any one of paragraphs A, B, or C, wherein the analysis in step (c) includes analyzing at least 5%, preferably between 10% and 100%, and more preferably between 25% and 90% of the total surface area of ​​the plurality of tiles.

[0112] E. The method of any one of paragraphs A-D, further comprising displaying a comparison between a single degree mark for each tile of the defined plurality of tiles and a predetermined value associated with a population of people.

[0113] F. The method of any one of paragraphs A-E, further comprising interposing second digital image data of at least a portion of the subject's face with the displayed plurality of tiles, each having a uniquely assigned single degree marking.

[0114] G. The method of any one of paragraphs A-F, wherein the visually imperceptible cosmetic skin characteristic is selected from the group consisting of skin topography, skin elasticity, skin tone, skin pigmentation, skin texture, skin pores, cosmetic skin inflammation, skin hydration, skin sebum level, acne, pigmented nevus, skin radiance, skin gloss, skin dullness, uneven tone, or skin barrier.

[0115] H. The method of any one of paragraphs A-G, wherein the unitary degree indicia is selected from the group consisting of a graphic symbol, a numerical value, a color code, lighting, and combinations thereof.

[0116] I. The method of any one of paragraphs A-H, wherein the displaying in step (e) includes defining at least one region of interest (ROI), preferably one to three ROIs, more preferably three ROIs, on the acquired first digital image data based on a plurality of tiles, each having a uniquely assigned single degree marking.

[0117] J. The method of any one of paragraphs AI, further comprising displaying at least one product recommendation for treating the indicated non-visually perceptible cosmetic skin characteristic.

[0118] K. Analyzing image data i) extracting at least one color channel from the acquired digital image to provide an extracted color channel image; ii) filtering the extracted color channel images; and The method of any one of paragraphs A-J, comprising: iii) analyzing the filtered color channel images using entropy statistics to determine at least one visually imperceptible cosmetic skin characteristic.

[0119] L. The method of paragraph K, wherein filtering the extracted color channel image of step (ii) includes using a frequency filter, preferably the frequency filter is selected from the group consisting of a Difference of Gaussian (DoG) filter, a Fast Fourier Transform (FFT) filter, a wavelet transform filter, and combinations thereof, more preferably the filter is a DoG filter.

[0120] M. At least one color channel may be selected from a red color channel, a yellow color channel, and a blue color channel, preferably the method includes at least two color channels, more preferably three color channels, wherein: The red color channel is L * a * b * in color space, where a-entropy is the entropy value of the filtered red color channel, Yellow color channel is L * a * b * in the color system, where b-entropy is the entropy value of the filtered yellow color channel, The blue color channel corresponds to the texture channel, where c-entropy is the entropy value of the blue color channel.

[0121] N. A method further comprising comparing at least one visually imperceptible cosmetic characteristic to a predefined data set and assigning an index, assigning a unique degree mark to each tile based on the assigned index; The method of any one of paragraphs K, L, or M, displaying a plurality of tiles, each having a uniquely assigned index.

[0122] O. The method of paragraph N, wherein the index is calculated based on at least one of the a-entropy values, b-entropy values, and c-entropy values ​​of the visually imperceptible cosmetic skin characteristic, and the predefined data set includes predetermined a-entropy values, b-entropy values, and c-entropy values ​​of the same visually imperceptible cosmetic skin characteristic of a defined population.

[0123] P. The method according to paragraph O, wherein the age of the subject and the mean age of the defined population of people are each independently between 18 and 40 years, preferably between 20 and 40 years, more preferably between 25 and 35 years, and even more preferably between 28 and 32 years.

[0124] Q. At least one color channel image is red color channel, Preferably, the method of any one of paragraphs K, L, M, N, O or P, wherein the cosmetic skin characteristic that is not visually perceptible is skin irritation.

[0125] R. At least one color channel is a yellow color channel; Preferably, the method of any one of paragraphs K, L, M, N, O, P or Q, wherein the visually non-perceptible cosmetic skin characteristic is skin pigmentation / dullness.

[0126] S. At least one color channel is a texture channel, Preferably, the method of any one of paragraphs K, L, M, N, O, P, Q or R, wherein the visually imperceptible cosmetic skin characteristic is skin topography, more preferably pores, fine lines, fine wrinkles, sagging, skin elasticity.

[0127] T. The method of any one of paragraphs K, L, M, N, O, P, Q, R, or S, further comprising applying an image correction factor to the filtered color before analyzing the image data.

[0128] U. A graphical user interface for providing product recommendations for treating at least one visually non-perceptible cosmetic skin characteristic, the graphical user interface being on a portable electronic device including a touch screen display / display having an input device and an image capture device, the graphical user interface comprising: a first region of a touch screen display displaying first digital image data of at least a portion of the subject's face acquired from an image acquisition device and second digital image data interleaved onto the first digital image data, the second digital image data including at least a portion of the subject's face, the plurality of tiles displayed each having a uniquely assigned single degree indicia; and a second area of ​​the touchscreen display, distinct from the first area, that displays selectable icons for receiving user input, wherein an image of at least one product recommendation for treating the displayed visually imperceptible cosmetic skin characteristic is displayed on the touchscreen display when a user activates the selectable icon.

[0129] V. A human-machine interface for providing product recommendations for treating at least one visually non-perceptible cosmetic skin characteristic, the human user interface being on a handheld electronic device including a display and an image capture device, the graphical user interface comprising: a first region of a display displaying first digital image data of at least a portion of the subject's face acquired from an image acquisition device and second digital image data interleaved onto the first digital image data, the second digital image data comprising at least a portion of the subject's face, the displayed tiles each having a uniquely assigned single degree indicia; a second area of ​​the display distinct from the first area, the second area displaying a voice input component for receiving user input, wherein an image of the at least one product recommendation for treating the displayed non-visually perceptible cosmetic skin characteristic is displayed on the touch screen display when the user activates the voice input component; and a third area of ​​the touch screen display distinct from the first area, the third area displaying an image of at least one product recommendation for treating the displayed visually non-perceptible cosmetic skin characteristic.

[0130] W. An apparatus for visualizing at least one visually non-perceptible cosmetic skin characteristic of a subject, the apparatus comprising: a non-transitory computer-readable storage medium configured to store a digital image of a person and computer-executable instructions; a processor coupled to a storage medium, the processor comprising: a) acquiring first digital image data of at least a portion of a face of a subject; b) defining a plurality of tiles across the acquired image data; c) analyzing the image data for each of the defined plurality of tiles for at least one visually imperceptible cosmetic skin characteristic; d) assigning a unique degree mark to each tile of the defined plurality of tiles based on the analyzed at least one visually imperceptible cosmetic skin characteristic of the tile; e) a processor configured to display a plurality of tiles, each having a uniquely assigned single degree indicia to visualize at least one visually imperceptible cosmetic skin characteristic.

[0131] X. A system comprising a device according to paragraph W and a touch screen display operably connected to the device.

[0132] Y. A method for visualizing at least one cosmetic skin characteristic of a subject, comprising: a) acquiring first digital image data of at least a portion of a face of a subject, the first digital image data being selected from an input image region of at least the face; b) defining a plurality of tiles across the acquired first digital image data; c) analyzing the image data for each of the defined plurality of tiles for at least one cosmetic skin characteristic; d) assigning a unique degree mark to each tile of the defined plurality of tiles based on the analyzed at least one cosmetic skin characteristic of the tile; e) displaying at least some of the plurality of tiles, each having a uniquely assigned single degree mark to visualize at least one cosmetic skin characteristic; and f) at least some of the displayed plurality of tiles display cosmetic skin characteristics that are better compared to cosmetic skin characteristic statuses of non-displayed tiles of the plurality of tiles based on the analyzed at least one cosmetic skin characteristic of the tile.

[0133] Z. The method of claim Y, wherein the acquired image data includes at least one region of interest (ROI) of at least a portion of the subject's face, the ROI being selected from the group consisting of a skin area around the eye, a skin area around the cheek, a skin area around the mouth, and combinations thereof.

[0134] AA. A method for visualizing at least one cosmetic skin characteristic of a subject, comprising: a) displaying a first digital image of at least a portion of a subject's face; b) interposing a second digital image of at least a portion of the subject's face with a plurality of tiles, each having a uniquely assigned degree indicia; c) displaying at least one region of interest (ROI) of at least a portion of the subject's face based on a plurality of tiles on the second digital image data, each having a uniquely assigned single degree mark, wherein the ROI is selected from the group consisting of skin areas around the eyes, skin areas around the cheeks, skin areas around the mouth, and combinations thereof. A method comprising: displaying at least one region of interest (ROI) of at least a portion of the subject's face based on a plurality of tiles on the second digital image data, each having a uniquely assigned single degree mark.

[0135] BB. The method of paragraph AA, wherein at least one ROI is defined by a border having a color.

[0136] CC. The method of paragraph AA or BB, wherein the displaying step includes displaying two or three ROIs, one of the two or three ROIs having a border color different from each of the other ROIs of the two or three ROIs.

[0137] DD. An apparatus for visualizing at least one visually non-perceptible cosmetic skin characteristic of a subject, the apparatus comprising: a non-transitory computer-readable storage medium configured to store a digital image of a person and computer-executable instructions; a processor coupled to a storage medium, the processor comprising: a) displaying a first digital image of at least a portion of a face of a subject; b) interposing a second digital image of at least a portion of the subject's face with a plurality of tiles, each having a uniquely assigned degree indicia; and c) displaying, on the second digital image data, at least one region of interest (ROI) of at least a portion of the subject's face based on a plurality of tiles, each having a uniquely assigned single degree marking, wherein the ROI is selected from the group consisting of a skin area around the eye, a skin area around the cheek, a skin area around the mouth, and combinations thereof.

[0138] All documents cited herein, including any cross-referenced or related patents or patent applications, and any patent applications or patents to which this application claims priority or the benefit thereof, are incorporated herein by reference in their entirety, unless expressly stated to the contrary. The citation of any document shall not be deemed to be prior art to any invention disclosed or claimed herein, or to teach, suggest or disclose such invention, either alone or in combination with any other reference(s). Furthermore, to the extent that any meaning or definition of a term in this document conflicts with any meaning or definition of the same term in a document incorporated by reference, the meaning or definition assigned to that term in this document shall govern.

[0139] While particular embodiments of the present invention have been illustrated and described, it would be obvious to those skilled in the art that various other changes and modifications can be made without departing from the spirit and scope of the invention. It is therefore intended to cover in the appended claims all such changes and modifications that are within the scope of this invention.

Claims

1. 1. A graphical user interface for providing product recommendations for treating at least one cosmetic skin characteristic, said graphical user interface being on a portable electronic device including a touch screen display / display having an input device and an image capture device, said graphical user interface comprising: a first area of ​​a touch screen display displaying a first digital image of at least a portion of the subject's face acquired from the image acquisition device and a second digital image inserted onto the first digital image, the second digital image having at least some of the at least a portion of the subject's face and a plurality of tiles, each of the plurality of tiles having a single degree marking uniquely assigned based on at least one cosmetic skin characteristic analyzed for the plurality of tiles; a second area of ​​the touch screen display, different from the first area, the second area displaying selectable icons for receiving user input; at least some of the tiles of the plurality of tiles display cosmetic skin characteristics that are better based on the analyzed at least one cosmetic skin attribute of the tiles as compared to cosmetic skin characteristic states of tiles of the plurality of tiles that are assigned non-display as the single degree indicia; and wherein an image of at least one product recommendation for treating the displayed cosmetic skin characteristic is displayed on the touch screen display when a user activates the selectable icon.

2. 2. The graphical user interface of claim 1, wherein the at least one cosmetic skin characteristic is a visually imperceptible cosmetic skin characteristic, wherein the visually imperceptible cosmetic skin characteristic is a cosmetic skin characteristic that is not detectable by the naked eye or a cosmetic skin characteristic that is visually detectable by a consumer, but the consumer does not understand the cosmetic skin characteristic.

Images