Method for simulating the application of a cosmetic make-up product to a body surface

Abstract

The invention relates to a method for simulating the application of a make-up product to a body surface, comprising implementing a rendering engine configured to apply a make-up effect comprising at least one color transformation to at least a portion of input image data comprising a virtual made-up body surface, and configured to generate transformed image data simulating said application of the make-up product according to at least one characteristic color parameter of the virtually applied make-up product, characterized in that the method comprises a preliminary step of determining said characteristic color parameter comprising: - obtaining color data of said virtual made-up body surface; and - obtaining a value of the characteristic color parameter of the make-up product as a function of said color data of the virtual made-up body surface.

Description

[Technical field]

[0001] The present invention relates to a method for simulating the application of a cosmetic make-up product to a body surface. [Background technology]

[0002] "Cosmetic product" is understood to mean any product as defined in Regulation (EC) No. 1223 / 2009 of the European Parliament and of the Council of 30 November 2009 on Cosmetic Products. Cosmetic make-up products are more specifically intended to cover body surfaces with the aim of modifying the perceived color and / or texture of the body surface.

[0003] The present application is more specifically directed to simulating the application of a lip coloring product or a skin foundation, however, it is also applicable to simulating the application of a hair coloring product or nail polish.

[0004] The choice of products, especially shades, is so vast that it can be a challenge for users to choose a satisfactory makeup product.In particular, users are faced with the fact that they have to purchase and try multiple products, which can make them very disappointed with the products they have tried, especially when they consider the results of all or some of the products to be disappointing.Moreover, while it is still relatively easy to apply and remove lipstick or foundation, there are more problems with such a trial method in the case of hair coloring and nail enamel.

[0005] The widespread development of digital technologies in society has allowed the development of augmented reality software that allows users to virtually try on cosmetics. Thus, L'OREAL (https: / / www.loreal.com / en / articles / science-and-technology / l-oreal-modiface-brings-ai-powered-virtual-makeup-try-ons-to-amazon / ), MAYBELLINE (https: / / www.maybelline.com / virtual-try-on-makeup-tools) and NYX (https: / / www.nyxcosmetics.com / try-it-on.html) each propose, for example, that it is possible to virtually try on makeup products from their respective ranges. In general, such software allows the calculation and generation of corresponding images from a digital image of the user that show the makeup effect that would be obtained by applying the product under consideration. Such systems are known for virtually trying on foundations, lipsticks or hair colorings, as exists for clothing or glasses.

[0006] Such a system allows a user to easily, directly and immediately evaluate the makeup results of multiple products without having to purchase the products or apply and remove the makeup.

[0007] Examples of virtual try-out methods for make-up products are described, for example, in the literature, US Pat. No. 5,399,623 and US Pat. No. 5,499,633, as well as in the as yet unpublished applications in the applicant's name, US Pat. No. 5,499,623, US Pat. No. 5,499,633, and US Pat. No. 5,523,362.

[0008] In general, such methods implement one or more rendering engines configured to apply a set of transformations or effects to the input image as a function of one or more characteristic parameters corresponding to the virtually applied cosmetic product. These transformations may include one or more segmentation steps, particularly intended to locate and isolate the virtually treated body surface, before applying, among other things, shape, color and texture rendering modifications. These rendering engines perform these transformations using various computational methods, and may implement, in particular, machine learning techniques and neural networks trained for these purposes.

[0009] The virtual try-on system thus receives an image to be modified, typically a photograph of a user, in particular a facial photograph of the user, and a set of characteristic rendering parameters for the selected cosmetic product. These parameters include, in particular, color and / or shape and / or texture parameters. By applying the transformations of the rendering engine to the input image as a function of the representative parameters, the virtual try-on system generates a result image simulating the application of the cosmetic product to the relevant body area. This result image is presented to the user.

[0010] In the particular case of virtual trying of cosmetic make-up products, color parameters are obviously of particular importance: considering that a make-up product is applied in the form of a layer on a body surface that may be colored to a certain extent, the resulting color after application, and thus the color perceived, depends on several factors, including its covering capacity or opacity, and the color of the body surface to which it is applied.

[0011] However, for simplicity of implementation, many virtual try-on systems do not take this inherent characteristic into account and use fixed predefined parameters. User-specific variability in rendering of cosmetic application can be taken into account downstream by algorithmic processing by the rendering engine. However, this processing can be complex, requires intensive computational resources, and lacks precision in terms of color rendering.

[0012] Recently, new systems have been developed that no longer try to use fixed predefined parameters but try to extract these products from images of individuals wearing them. This makes it possible, in particular, to virtually try on, for example, cosmetics such as lipsticks worn by another individual, in particular a known celebrity. The problem is similar in that such methods take the resulting color for a reference individual and transfer it virtually to the user, without necessarily being able to make adjustments that may be required due to the user's own skin color. [Prior art documents] [Patent documents]

[0013] [Patent Document 1] US Patent Application Publication No. 2020 / 0160153A1 [Patent Document 2] U.S. Patent No. 9449412B1 [Patent Document 3] French Patent Application Publication No. 21 / 06030 [Patent Document 4] French Patent Application Publication No. 21 / 11155 [Patent Document 5] French Patent Application Publication No. 21 / 11164 Summary of the Invention [Problem to be solved by the invention]

[0014] Therefore, a requirement exists for a simple solution to improve the color rendering of make-up products in a virtual try-on system. [Means for solving the problem]

[0015] To this end, the present application provides a method for simulating the application of a makeup product to a body surface, the method comprising implementing a rendering engine configured to apply a makeup effect including at least one color transformation to at least a portion of input image data including a virtually made-up body surface, and configured to generate transformed image data simulating said application of the makeup product according to at least one characteristic color parameter of the virtually applied makeup product.

[0016] The method is: - acquiring color data of said body surface to be virtually made up; - obtaining values ​​of characteristic color parameters of a make-up product as a function of said color data of the body surface to be virtually made up, The method is characterized in that it includes a preliminary step of determining values ​​of said characteristic color parameters, including:

[0017] According to a first alternative embodiment, the color data of the body surface to be virtually made up is obtained from input image data. According to a second alternative embodiment, the color data of the body surface to be virtually made up may be obtained separately, the input image data being an image of a body area made up with a make-up product having a particular desired finish or texture.

[0018] Preferably, the colour data of the body surface to be virtually made up is obtained from a bare body area, i.e. a body area without make-up and preferably without any cosmetic product.

[0019] Therefore, by providing a rendering engine that is supplied with values ​​of color parameters that are determined from the color of the body area to be made up, which is no longer independent of the user, it is possible to easily and directly send the resulting color values ​​adapted to the user to the rendering engine without integrating additional complex transformations and calculations into the rendering engine.

[0020] In other words, while generally, selecting a cosmetic product having a certain shade results in color parameter values ​​that are determined and transmitted to the rendering engine that are independent of the color of the body area being considered, the present method provides for transmitting color parameter values ​​determined to the rendering engine not only related to the desired makeup product but also related to the color of the body area, so as to introduce into the rendering engine the characteristic color parameter values ​​that are most suitable for the user.

[0021] Thus, for a user who wishes to virtually try on a lipstick of Lancôme "Absolu Rouge" shade 397 "black berry matte", values ​​of the color parameters can be obtained as a function of the lip color. In reality, the same lipstick applied to light lips will not result in the same rendering as when applied to dark lips. For this reason, it is important to be able to provide a rendering engine that uses appropriate color parameter values ​​rather than the color parameter values ​​obtained for different users, independent of the user.

[0022] Preferably, the color values ​​and data are expressed as coordinates in a given color space, in particular a perceptible uniform color space such as the CIELAB color space (in particular with L*, a*, b* or polar coordinates L*, C*, h*). In particular, a perceptible uniform color space allows for easy determination of the delta E color difference, which represents the visual difference. Of course, alternatively or additionally, the colors can be expressed in other color spaces, in particular in RGB or even directly as hexadecimal computer code, in particular according to computer processing requirements.

[0023] The method that is the subject of the present application more particularly relates to the virtual make-up of the lip area with a lip make-up product, in particular a cosmetic make-up product, preferably a solid stick type lipstick or a lip varnish (gloss). Naturally, the method can also be carried out for the virtual make-up of other body areas, such as skin areas, in particular the skin of the face or eyelids, with a cosmetic make-up product, respectively a foundation or an eyeshadow.

[0024] According to an alternative preferred embodiment, the color parameter values ​​are obtained from a database that associates at least one cosmetic make-up product reference with a number of reference colors for the body area under consideration. Thus, for each cosmetic product reference (e.g., lipstick), the database may contain, for example, values ​​of color parameters corresponding to the rendering colors of said cosmetic product on light / medium / dark toned body areas (e.g., lips).

[0025] Advantageously, the value of the characteristic color parameter of the cosmetic product is selected from the database as the one assigned to the reference color of the considered body area that is closest to the color of the body area to be virtually made up, in particular obtained from the input image data. In particular, the proximity can be determined by calculating the color difference in the considered color space, for example by calculating the parameter ΔE in the CIELAB space.

[0026] The use of a database containing values ​​of color parameters for multiple reference colors of the body area under consideration allows a system to be maintained that is easily accessible, easy to poll, and allows a large number of reference colors of cosmetic products to be stored while maintaining sufficient accuracy and compatibility with the color of the user's body area.

[0027] Values ​​of color parameters for reference colors of various cosmetic products and various body areas can be obtained in various ways, for example, by measuring the color on the actual applied product or by simulation using a colorimetric rendering model (e.g., the Kubelka-Munk approximation) as described in the following paragraphs.

[0028] As an alternative to the use of a database containing various color parameter values, the values ​​of the characteristic color parameters of the cosmetic product can be obtained by applying a colorimetric rendering model to the intrinsic or specific color data of the cosmetic product under consideration and in particular the color values ​​of the body area to be made up obtained from the input image data. Thus, the color of the virtual body area to be made up is no longer matched to a similar reference color, but is directly used to determine the values ​​of said color parameters to be sent to the rendering engine of the virtual try-on system. The intrinsic color of the cosmetic product under consideration is understood to mean a color that is independent of the color of the skin surface on which it is intended to be applied. This color can be obtained by in vitro measurements under standardized conditions, for example after applying the product (for example by coating a layer of a defined thickness) on an opacity measurement support that includes a black-coated surface and a white-coated surface (an "opacity chart" such as the Leneta® chart), and preferably also under standardized lighting conditions (in particular D65). The color of the body surface is preferably obtained under similar or identical lighting conditions, for example using a device such as CHROMASPHERE®. Advantageously, the images that the user provides to the rendering engine can also be taken under the same lighting conditions.

[0029] According to the operation of the rendering engine of the virtual trial system, a single color parameter value may be obtained for the overall average color of the virtual make-up body surface. However, in order to improve the rendering accuracy, and given that the color of the virtual make-up body surface may not be uniform overall, the virtual make-up body surface may be subdivided into a number of smaller sub-regions, for each of which a color parameter value may be obtained. The color parameter value may also be obtained pixel by pixel, if necessary.

[0030] In an advantageous additional method, the rendering engine can receive a gloss / matt parameter.

[0031] Alternatively or additionally, as explained above, the color of the made up body surface may be obtained separately from input image data from a bare / unmade up body surface, where the input image data provided to the rendering engine may include the body surface made up with a product having the desired finish (matte / gloss) and / or texture, and the rendering engine only performs the color transformation.

[0032] The present invention will be better understood from the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0033] [Figure 1] 1 illustrates a schematic example of a process flow for a virtual try-on system implementing a rendering engine R configured to generate photo-realistic renderings of lip make-up product applied to an individual. [Diagram 2] 2 is a schematic representation of a method for determining values ​​of color parameters of the rendering engine of FIG. 1; [Diagram 3] 4 is a schematic representation of a method for determining the rendering color of a lip make-up product as a function of said lip color, for storage in the database of the determination methods of FIGS. 2 and 3, or for use directly as a color parameter of the rendering engine of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Although illustrated by way of example, by virtually making up the surface of the user's lips with lipstick, the method is applicable to other body surfaces (skin, especially face, hair) and other make-up products intended to be applied to said body surfaces. The present application relates in particular to cosmetic make-up products with a wide variety of shades, and to body surfaces with a wide variety of colors within the population.

[0035] 1 illustrates an example process flow for a virtual try-on system implementing a rendering engine R configured to generate a photo-realistic rendering of lip make-up product PM applied to an individual P3. As noted above, the general steps described may be applied mutatis mutandis to other types of cosmetic products for other body surfaces.

[0036] In step 20, the rendering engine R receives characterization parameters relating to the color COL of a lip make-up product PM, the reflectance REF of this product PM, and the texture TXT of this product PM.

[0037] In step 21, the rendering engine R also receives a source image X_S of the lips of person P3. In step 22, the rendering engine R may then calculate a model M3D of the lips of the source image X_S. In step 23, the rendering engine R may also estimate the light reflectance of the lips of the source image X_S.

[0038] As known per se, the source image X_S may be conventionally obtained by segmenting and clipping the region of interest (the lips) onto a larger image X_I of a person, including in particular the face.

[0039] Depending on the complexity of the system, the rendering engine R may then determine corrected color parameters in step 24 from the color parameters of the make-up product PM entered as input. These corrected color parameters take into account, for example, the lightness of the lips of the source image. Lightness parameters may be extracted from the source image X_I and its environment. In some cases, particularly in professional environments, the source images may be acquired under standardized conditions (see above). The standardized conditions are substantially identical to the conditions for determining the parameters characterizing the virtually tried-on cosmetic product PM. In such cases, no additional transformation may be necessary.

[0040] Then, in step 25, the rendering engine R generates a first intermediate image IM1, the rendering engine applying the corrected color parameters to the lips of the source image, taking into account the pre-computed lip model.

[0041] In step 26, the rendering engine R then optionally generates a second intermediate image IM2 in which the input reflectance parameters are applied to the lips of the first intermediate image IM1, also taking into account the light reflectance of the lips in the source image.

[0042] In step 27, the rendering engine R then optionally generates as output a transformed image IMT, where the texture parameters entered as input are applied to the lips of the second intermediate image.

[0043] The transformed image IMT (IMT=IM1 if the optional transformation step is not present) replaces the source image X_S on a larger image X_I to generate an image X_V containing a virtual make-up area of ​​the subject that can be presented to the user.

[0044] According to the present application, the value of a characterization parameter related to the color COL of the cosmetic make-up product PM is obtained through a preliminary step of determining said parameter, an embodiment of which is diagrammatically illustrated in FIG.

[0045] Determining the value of the color parameter COL of the cosmetic product PM includes obtaining color data LIP_P3 of the lips of a virtual made-up individual P3. This color data is obtained directly from an image I1 of the individual P3, preferably taken under standardized and calibrated lighting conditions (in particular D65). Optionally, the color data may be obtained separately, in particular using a colorimeter or a spectrophotometer. As indicated previously, the lip color data is obtained from the individual's unmade-up (bare) lips.

[0046] According to an alternative embodiment, image I1 of individual P3 taken with bare lips may serve as input image X_S of the rendering engine.

[0047] Alternatively, as explained above, the color data of the body surface to be virtually made up is obtained separately from the image I1 (image of lips without make-up) used to obtain the color of the lips. The input image X_S may then be an image I2 of a body area made up with a make-up product having a particular desired finish or texture. In this case, the rendering color of the selected lipstick is obtained from the color of the user's lips without make-up (image I1). Images (I2, I3) are also obtained from an individual P3 whose body surface is made up with a matte finish lipstick and a glossy finish lipstick, respectively. Depending on the desired finish of the cosmetic product to be virtually tried, any of these images may form the input image data X_S. The rendering engine R may then perform a simple color transformation, avoiding additional rendering transformation calculations.

[0048] From the lip color data LIP_P3 determined thereby and the reference of the virtually tried cosmetic PM LS C, a database DB containing a plurality of values ​​of color parameters recorded for various reference color values ​​of the lips for the reference cosmetic is polled.

[0049] More specifically, the database includes a table that associates one or more cosmetic references PM, in this case lipsticks LS A, LS B, LS C, LS D, with a number of reference lip colors LIPS1, LIPS2, LIPS3. For each cosmetic / lip color pair (LS i; LIPS j), the database may store a color value that represents the rendered color of the cosmetic PM (LS i) when applied to lips of color LIPj.

[0050] From the lip color LIP_P3 of the individual to be virtually made up, the closest color LIPj(P3) is determined, to which the color value corresponding to the cosmetic product PM(LSC) is input (in this case LIPS2). The color value of the cosmetic product PM (in this case LSC) when applied to the lips LIPS2 is extracted from the database and sent to the rendering engine as the color parameters COL of the cosmetic product PM(LSC) for virtually rendering it on the image of the individual P3.

[0051] Various reference lip colors can be determined by dividing the actual color measurement data from a population sample. Lip colors can advantageously be measured with a colorimeter or spectrophotometer under standardized and calibrated lighting conditions (D65, especially using a CHROMASPHERE® device). It is therefore possible to determine groups of between 3 and 10 lip colors, especially light, medium and dark lip colors, optionally with intermediate gradations. In particular, a "k-means clustering" method may be used, and lip colors may be selected from the center of each cluster.

[0052] The color values ​​of the different make-up products for the different reference lip colors may be obtained by measuring the color of the products actually applied to lips with the corresponding color and / or by calculation from a color rendering model (KM) implementing, for example, the Kubelka-Munk approximation or another physical rendering model as shown in FIG. 3. The colorimetric rendering model KM simulates the color rendered from the colorimetry of bare lips (ex vivo) and the colorimetry of the cosmetic product PM (in vitro) that corresponds to an intrinsic color value independent of the body surface color. Colorimetric corrections may be applied using colorimetric deviation parameters that may be determined by comparison of the results of the color simulation with the actual values ​​obtained on the products actually applied to the lips. All or part of such a simulation system may implement machine learning techniques, in particular for determining the colorimetric deviation parameters that allow to perform a color correction against a theoretical physical model. Such a system is described in particular in French patent application No. 3094201 for hair coloring.

[0053] The color rendering simulation system thus defined makes it possible to pre-calculate color rendering values ​​for various cosmetic products that can be virtually tried for each reference lip color. These values ​​are stored in a database.

[0054] In an alternative embodiment, the simulation system may directly generate the color parameter values ​​COL that are sent to the rendering engine R. In this case, the color of the user's lips is conventionally acquired and submitted to the simulation system along with known specific color data for the virtually tried-on cosmetic product PM. The simulation system then feeds back values ​​that correspond to an estimate or prediction of the color that will be rendered by applying the cosmetic product PM to the user's lips.

[0055] To improve the rendering of the virtual try-on system, it is also possible to send a matte / glossy parameter to the rendering engine, reflecting the matte or glossy finish of the virtually applied lipstick, which may in particular form part of the reflectance parameters that the rendering engine may be supplied with to optimize the light rendering.

[0056] Although not described, the selection of the cosmetic product to be virtually tried may be made, as is conventional, from an interface presenting all or part of a product catalog for which parameters are available and for which virtual trying is possible.

[0057] Moreover, all or part of the steps are intended to be computer-implemented, and to this end, the present application also relates to a computer program product comprising instructions, which when the program is executed by a computer, results in the implementation of at least the characteristic steps of the method. Preferably, the program also implements a virtual trial system and a rendering engine. The present application also relates to a system comprising a memory, in which the program product is stored. The system may also include any additional devices necessary for implementing the method, in particular any colorimetric data acquisition device.

Claims

1. 1. A method for simulating the application of a make-up product (PM) to a body surface, comprising: implementing a rendering engine (R) configured to apply a make-up effect comprising at least one color transformation to at least a portion of input image data (X_S) comprising said body surface to be virtually made up, and configured to generate transformed image data (X_V) simulating said application of a make-up product according to at least one characteristic color parameter (COL) of said virtually applied make-up product, The method further comprising: - obtaining color data (LIPS_P3) of said body surface to be virtually made up; - obtaining values ​​of said characteristic color parameters of said make-up product as a function of said color data of said body surface to be virtually made up; 4. The method of claim 3, further comprising a preliminary step of determining said characteristic color parameters, said preliminary step including:

2. 2. A method according to claim 1, characterised in that the body area to be virtually made up is the lip area and said cosmetic make-up product (PM) is a lip make-up product, in particular a lipstick (LSC).

3. 3. A method according to claim 1 or 2, characterized in that the colour data (LIPS_P3) of the body surface to be virtually made up is obtained from the input image data (X_S).

4. 3. The method according to claim 1 or 2, characterized in that the color data (LIPS_P3) of the body surface to be virtually made up is obtained separately from the input image data (X_S), in particular by color measurement using a colorimeter or spectrophotometer.

5. A method according to any one of claims 1 to 4, characterized in that the input image data (X_S) is an image (I2) of the body area made up with a make-up product having a particular desired finish or texture.

6. Method according to any one of claims 1 to 5, characterized in that the colour parameter values ​​(COL) are obtained from a database (DB) relating at least one cosmetic make-up product standard (PM LS C) to a number of standard colours (LIPS1, LIPS2, LIPS3) of the body area under consideration.

7. 7. The method according to claim 6, characterized in that the value of the characteristic color parameter (COL) of the cosmetic product (PM) is selected from the database as being assigned to the reference color of the considered body area which is closest to the color of the body area to be virtually made up obtained from the input image data.

8. 8. The method according to claim 1, wherein the values ​​of the characteristic color parameters (COL) of the cosmetic product (PM) are obtained by applying a colorimetric rendering model (KM) to the intrinsic color data of the cosmetic product under consideration and to the color data of the body area to be made up obtained from the input image data (X_S).

9. 9. The method according to claim 1, characterized in that values ​​of the color parameters (COL) are obtained for different partial regions of the body surface to be virtually made up, in particular for pixel-by-pixel color processing.

10. Method according to any one of the preceding claims, characterized in that the rendering engine (R) is capable of receiving a gloss / matt parameter.

Images