Method for simulating application of cosmetic makeup products to a body surface

Abstract

The invention relates to a method for simulating the application of a make-up product to a body surface, comprising implementing a rendering engine configured to apply a make-up effect comprising at least one color transformation to at least a portion of input image data comprising a virtual made-up body surface, and configured to generate transformed image data simulating said application of the make-up product according to at least one characteristic color parameter of the virtually applied make-up product, characterized in that the method comprises a preliminary step of determining said characteristic color parameter comprising: - obtaining color data of said virtual made-up body surface; and - obtaining a value of the characteristic color parameter of the make-up product as a function of said color data of the virtual made-up body surface.

Description

[Technical Field]

[0001] The present invention relates to a method for simulating the application of a cosmetic make-up product to a body surface. [Background technology]

[0002] "Cosmetic product" is understood to mean any product as defined in Regulation (EC) No. 1223 / 2009 of the European Parliament and of the Council of 30 November 2009 on cosmetic products. Cosmetic make-up products are more particularly intended to cover body surfaces with the aim of modifying the perceived color and / or texture of the body surface.

[0003] This application is more particularly directed to simulating the application of lip coloring products or skin foundation, however, it is also applicable to simulating the application of hair coloring products or nail polish.

[0004] Because the choice of products, especially shades, is so vast, selecting a satisfactory makeup product can be a challenge for users.In particular, users are faced with the fact that they have to purchase and try out multiple products, which can lead to great disappointment with the products, especially when the user finds that the results of all or some of the products they have tried are disappointing.Furthermore, while it is still relatively easy to apply and remove lipstick or foundation, there are more problems with such a trial method in the case of hair coloring and nail enamel.

[0005] The widespread development of digital technology in society has enabled the development of augmented reality software that allows users to virtually try on cosmetics. Thus, L'OREAL (https: / / www.loreal.com / en / articles / science-and-technology / l-oreal-modiface-brings-ai-powered-virtual-makeup-try-ons-to-amazon / ), MAYBELLINE (https: / / www.maybelline.com / virtual-try-on-makeup-tools), and NYX (https: / / www.nyxcosmetics.com / try-it-on.html) each offer, for example, the possibility of virtually trying on makeup products from their respective ranges. In general, such software allows a corresponding image to be calculated and generated from a digital image of the user, showing the makeup effect that would be obtained by applying the product under consideration. Such systems are known for virtually trying on foundation, lipstick, or hair coloring, just as they are for clothing or eyeglasses.

[0006] Such a system allows a user to easily, directly and immediately evaluate the makeup results of multiple products without having to purchase the products or apply and remove the makeup.

[0007] Examples of virtual try-on methods for make-up products are described, for example, in the documents U.S. Pat. No. 5,629,992 and U.S. Pat. No. 5,629,992, as well as in the as yet unpublished applications in the name of the applicant, U.S. Pat. No. 5,629,992 ...

[0008] In general, such methods implement one or more rendering engines configured to apply a set of transformations or effects to an input image as a function of one or more characteristic parameters corresponding to a virtually applied cosmetic product. These transformations may include one or more segmentation steps specifically intended to locate and isolate the virtually treated body surface before applying, among other things, shape, color, and texture rendering modifications. These rendering engines perform these transformations using various computational methods and may implement, in particular, machine learning techniques and neural networks trained for these purposes.

[0009] Thus, the virtual try-on system receives an image to be modified, typically a user's photograph, particularly a facial photograph of the user, and a set of characteristic rendering parameters for the selected cosmetic product. These parameters include, among others, color and / or shape and / or texture parameters. By applying the transformations of the rendering engine to the input image as a function of the representative parameters, the virtual try-on system generates a resultant image that simulates the application of the cosmetic product to the relevant body area. This resultant image is presented to the user.

[0010] In the particular case of virtual try-on of cosmetic make-up products, color parameters are obviously of particular importance: considering that a make-up product is applied in the form of a layer onto a body surface that may be colored to a certain extent, the resulting color after application, and thereby the color perceived, depends on several factors, including its covering capacity or opacity, and the color of the body surface onto which it is applied.

[0011] However, for simplicity of implementation, many virtual try-on systems do not take this inherent characteristic into account and use fixed, predefined parameters. User-specific variability in the rendering of cosmetic application can be taken into account downstream through algorithmic processing by the rendering engine. However, this processing can be complex, require intensive computational resources, and lack precision in terms of color rendering.

[0012] Recently, new systems have been developed that no longer seek to use fixed, predefined parameters but instead extract these products from images of individuals wearing them. This makes it possible, for example, to virtually try on cosmetics, such as lipstick, worn by another individual, especially a well-known celebrity. The problem is similar in that such methods take the resulting color for a reference individual and virtually transfer it to the user, without necessarily being able to make adjustments that may be required depending on the user's own skin color. [Prior art documents] [Patent documents]

[0013] [Patent Document 1] U.S. Patent Application Publication No. 2020 / 0160153A1 [Patent Document 2] U.S. Patent No. 9,449,412 B1 [Patent Document 3] French Patent Application Publication No. 21 / 06030 [Patent Document 4] French Patent Application Publication No. 21 / 11155 [Patent Document 5] French Patent Application Publication No. 21 / 11164 Summary of the Invention [Problem to be solved by the invention]

[0014] Therefore, there is a need for a simple solution to improve the color rendering of makeup products in a virtual try-on system. [Means for solving the problem]

[0015] To this end, the present application provides a method for simulating the application of a makeup product to a body surface, the method comprising implementing a rendering engine configured to apply a makeup effect including at least one color transformation to at least a portion of input image data including a virtually made-up body surface, and configured to generate transformed image data simulating said application of the makeup product according to at least one characteristic color parameter of the virtually applied makeup product.

[0016] The method is: - obtaining color data of the body surface to be virtually made up; - obtaining values ​​of characteristic color parameters of a make-up product as a function of said color data of the body surface to be virtually made up; The method is characterized in that it includes a preliminary step of determining values ​​of said characteristic color parameters, including:

[0017] According to a first alternative embodiment, the color data of the body surface to be virtually made up is obtained from input image data. According to a second alternative embodiment, the color data of the body surface to be virtually made up may be obtained separately, where the input image data may be an image of a body area made up with a makeup product having a particular desired finish or texture.

[0018] Preferably, the color data of the body surface to be virtually made up is obtained from a bare body area, i.e. a body area without make-up and preferably without any cosmetic product.

[0019] Therefore, by providing a rendering engine that is supplied with values ​​for color parameters that are determined from the color of the body area being made up, which is no longer independent of the user, it is possible to easily and directly send resulting color values ​​that are adapted to the user to the rendering engine without integrating additional complex transformations and calculations into the rendering engine.

[0020] In other words, while generally, selecting a cosmetic product having a certain shade results in color parameter values ​​that are determined and transmitted to the rendering engine that are independent of the color of the body area being considered, the present method provides for transmitting color parameter values ​​determined not only related to the desired makeup product but also related to the color of the body area to the rendering engine, so as to introduce into the rendering engine characteristic color parameter values ​​that are most suitable for the user.

[0021] Thus, for a user who wishes to virtually try on Lancôme "Absolu Rouge" lipstick in shade 397 "black berry matte," color parameter values ​​can be obtained as a function of their lip color. In reality, the same lipstick applied to lighter lips will not result in the same rendering as when applied to darker lips. For this reason, it is important to be able to provide a rendering engine that is user-independent or uses appropriate color parameter values ​​for different users, rather than the color parameter values ​​obtained.

[0022] Preferably, color values ​​and data are expressed as coordinates in a given color space, in particular a perceptually uniform color space such as the CIELAB color space (in particular using L*, a*, b* or polar coordinates L*, C*, h*). In particular, a perceptually uniform color space makes it possible to easily determine the Delta E color difference, which represents the visual difference. Of course, alternatively or additionally, colors can be expressed in other color spaces, in particular in RGB or even directly as hexadecimal computer code, in particular according to computer processing requirements.

[0023] The method that is the subject of the present application more particularly relates to the virtual make-up of the lip area with a lip make-up product, in particular a cosmetic make-up product that is preferably a solid stick type lipstick or a lip varnish (gloss). Naturally, the method can also be carried out for the virtual make-up of other body areas, such as skin areas, in particular the skin of the face or eyelids, with a cosmetic make-up product that is a foundation or an eyeshadow, respectively.

[0024] According to an alternative preferred embodiment, the color parameter values ​​are obtained from a database that associates at least one cosmetic makeup product standard with a plurality of standard colors for the body area under consideration. Thus, for each cosmetic product standard (e.g., lipstick), the database may contain, for example, values ​​of color parameters corresponding to the rendering color of said cosmetic product on a light / medium / dark-toned body area (e.g., lips).

[0025] Advantageously, the values ​​of the characteristic color parameters of the cosmetic product are selected from the database as those assigned to the reference color of the considered body area that is closest to the color of the body area to be virtually made up, in particular obtained from the input image data, and in particular the proximity can be determined by calculating the color difference in the considered color space, for example by calculating the parameter ΔE in the CIELAB space.

[0026] The use of a database containing color parameter values ​​for multiple reference colors of the body area under consideration allows for a system that is easily accessible and easy to poll, and allows for a large number of cosmetic reference colors to be stored while maintaining sufficient accuracy and compatibility with the color of the user's body area.

[0027] Color parameter values ​​for reference colors of various cosmetics and various body areas can be obtained in various ways, for example, by measuring the color on an actually applied product or by simulation using a colorimetric rendering model (e.g., the Kubelka-Munk approximation) as described in the following paragraphs.

[0028] As an alternative to using a database containing various color parameter values, the values ​​of the characteristic color parameters of a cosmetic product can be obtained by applying a colorimetric rendering model to the intrinsic or specific color data of the cosmetic product under consideration, and in particular the color values ​​of the body area to be made up obtained from the input image data. Thus, the color of the virtual body area to be made up no longer corresponds to a similar reference color but is directly used to determine the values ​​of the above color parameters to be sent to the rendering engine of the virtual try-on system. The intrinsic color of the cosmetic product under consideration should be understood to mean a color that is independent of the color of the skin surface to which it is intended to be applied. This color can be obtained by in vitro measurement under standardized conditions, for example, after applying the product (e.g., by coating a layer of a defined thickness) on an opacity measurement support including a black-coated surface and a white-coated surface (an "opacity chart" such as the Leneta® chart), and preferably also under standardized lighting conditions (e.g., D65). The color of the body surface is preferably obtained under similar or identical lighting conditions, for example, using a device such as CHROMASPHERE®. Advantageously, the images that the user provides to the rendering engine can also be taken under the same lighting conditions.

[0029] According to the operation of the rendering engine of the virtual trial system, a single color parameter value may be obtained for the overall average color of the virtual make-up body surface. However, in order to improve rendering accuracy and given that the color of the virtual make-up body surface may not be uniform overall, the virtual make-up body surface may be subdivided into multiple smaller sub-regions, and a color parameter value may be obtained for each of them. The color parameter value may also be obtained for each pixel as needed.

[0030] In an advantageous additional method, the rendering engine can receive a gloss / matt parameter.

[0031] Alternatively or additionally, as explained above, the color of the made-up body surface may be obtained separately from input image data from a bare / unmade-up body surface, where the input image data provided to the rendering engine may include the body surface made up with a product having a desired finish (matt / gloss) and / or texture, and the rendering engine simply performs the color transformation.

[0032] The present invention will be better understood from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0033] [Figure 1] 1 illustrates a schematic example of a process flow for a virtual try-on system implementing a rendering engine R configured to generate photorealistic renderings of lip makeup products applied to an individual. [Figure 2] 2 is a schematic representation of a method for determining values ​​of color parameters of the rendering engine of FIG. 1; [Figure 3] 4 is a schematic representation of a method for determining the rendering color of a lip makeup product as a function of said lip color, for storage in the database of the determination methods of FIGS. 2 and 3, or for use directly as a color parameter in the rendering engine of FIG. 1. DETAILED DESCRIPTION OF THE INVENTION

[0034] Although illustrated by way of example, by virtually making up the surface of a user's lips with lipstick, the method is applicable to other body surfaces (skin, especially face, hair) and other make-up products intended to be applied to said body surfaces. The present application particularly relates to cosmetic make-up products with a wide variety of shades, and to body surfaces with a wide variety of colors within a population.

[0035] 1 illustrates an example process flow for a virtual try-on system implementing a rendering engine R configured to generate a photorealistic rendering of lip makeup product PM applied to an individual P3. As noted above, the general steps described can be applied mutatis mutandis to other types of cosmetic products for other body surfaces.

[0036] In step 20, the rendering engine R receives characterization parameters relating to the color COL of the lip makeup product PM, the reflectance REF of this product PM, and the texture TXT of this product PM.

[0037] In step 21, the rendering engine R also receives a source image X_S of the lips of person P3. In step 22, the rendering engine R may then calculate a model M3D of the lips of source image X_S. In step 23, the rendering engine R may also estimate the light reflectance of the lips of source image X_S.

[0038] As known per se, the source image X_S may be conventionally obtained by segmenting and clipping the region of interest (lips) onto a larger image X_I of a person, including in particular the face.

[0039] Depending on the complexity of the system, the rendering engine R may then determine corrected color parameters in step 24 from the color parameters of the makeup product PM entered as input. These corrected color parameters may take into account, for example, the lightness of the lips in the source image. Lightness parameters may be extracted from the source image X_I and its environment. In some cases, particularly in professional environments, the source image may be acquired under standardized conditions (see above). The standardized conditions are substantially identical to the conditions for determining the parameters characterizing the virtually tried-on cosmetic product PM. In such cases, no additional transformation may be necessary.

[0040] Then, in step 25, the rendering engine R generates a first intermediate image IM1, the rendering engine applying the corrected color parameters to the lips of the source image, taking into account the pre-computed lip model.

[0041] In step 26, the rendering engine R then optionally generates a second intermediate image IM2 in which the input reflectance parameters are applied to the lips of the first intermediate image IM1, also taking into account the light reflectance of the lips in the source image.

[0042] In step 27, the rendering engine R then optionally generates a transformed image IMT as output, and the texture parameters entered as input are applied to the lips of the second intermediate image.

[0043] The transformed image IMT (IMT=IM1 if the optional transformation step is not present) replaces the source image X_S on the larger image X_I to generate an image X_V containing the virtual makeup area of ​​the subject that can be presented to the user.

[0044] According to the present application, the values ​​of the characterization parameters related to the color COL of the cosmetic make-up product PM are obtained through a preliminary step of determining said parameters, an embodiment of which is shown diagrammatically in FIG.

[0045] Determining the value of the color parameter COL of the cosmetic PM includes acquiring lip color data LIP_P3 of the virtual make-up-applied individual P3. This color data is acquired directly from an image I1 of the individual P3, preferably taken under standardized and calibrated lighting conditions (e.g., D65). Optionally, the color data may be acquired separately, in particular using a colorimeter or spectrophotometer. As previously indicated, the lip color data is acquired from the individual's bare lips without make-up.

[0046] According to an alternative embodiment, image I1 of individual P3 taken with bare lips may serve as input image X_S for the rendering engine.

[0047] Alternatively, as described above, color data of the body surface to be virtually made up is obtained separately from image I1 (image of lips without makeup) used to obtain the lip color. The input image X_S may then be image I2 of a body area made up with a makeup product having a particular desired finish or texture. In this case, the rendering color of the selected lipstick is obtained from the color of the user's lips without makeup (image I1). Images (I2, I3) are also obtained from individual P3, whose body surface is made up with a matte-finish lipstick and a glossy-finish lipstick, respectively. Depending on the desired finish of the virtually tried-on cosmetic product, either of these images may form the input image data X_S. The rendering engine R may then perform a simple color transformation, avoiding additional rendering transformation calculations.

[0048] From the lip color data LIP_P3 determined thereby and the reference of the virtually tried-on cosmetic PM LS C, a database DB containing multiple values ​​of color parameters recorded for various reference color values ​​of the lips for the reference cosmetic is polled.

[0049] More specifically, the database includes a table that associates one or more cosmetic reference PM, in this case lipsticks LS A, LS B, LS C, LS D, with multiple reference lip colors LIPS1, LIPS2, LIPS3. For each cosmetic / lip color pair (LS i; LIPS j), the database may store a color value that represents the rendered color of the cosmetic PM (LS i) when applied to lips of color LIP j.

[0050] From the lip color LIP_P3 of the individual to be virtually made up, the closest color LIPj(P3) is determined, and the color value corresponding to the cosmetic PM(LS C) is input to it (in this case LIPS2). The color value of the cosmetic PM (in this case LSC) when applied to the lip LIPS2 is extracted from the database and sent to the rendering engine as the color parameter COL of the cosmetic PM(LS C) for virtually rendering it on the image of the individual P3.

[0051] Various reference lip colors can be determined by dividing actual color measurement data from a population sample. Lip colors can be advantageously measured with a colorimeter or spectrophotometer under standardized and calibrated lighting conditions (D65, particularly using a CHROMASPHERE® device). Therefore, it is possible to determine groups of between 3 and 10 lip colors, particularly light, medium, and dark lip colors, optionally with intermediate gradations. In particular, a "k-means clustering" method can be used, and lip colors can be selected from the center of each cluster.

[0052] The color values ​​of various makeup products relative to various reference lip colors may be obtained by measuring the color of the products actually applied to lips of the corresponding color and / or by calculation from a color rendering model (KM) that implements, for example, the Kubelka-Munk approximation or another physical rendering model, such as that shown in FIG. 3. The colorimetric rendering model KM simulates the color rendered from the colorimetry of bare lips (ex vivo) and the colorimetry of cosmetic products PM (in vitro) that corresponds to intrinsic color values ​​independent of body surface color. Colorimetric corrections may be applied using colorimetric deviation parameters that can be determined by comparing the results of the color simulation with actual values ​​obtained on products actually applied to the lips. All or part of such a simulation system may implement machine learning techniques, in particular for determining colorimetric deviation parameters that allow color corrections to be made to theoretical physical models. Such a system is described, in particular, in French Patent Application No. 3094201 for hair coloring.

[0053] The color rendering simulation system thus defined allows for pre-calculation of color rendering values ​​for various cosmetic products that can be virtually tried on for each reference lip color. These values ​​are stored in a database.

[0054] In an alternative embodiment, the simulation system may directly generate color parameter values ​​COL that are sent to the rendering engine R. In this case, the color of the user's lips is conventionally obtained and submitted to the simulation system along with known specific color data for the virtually tried-on cosmetic product PM. The simulation system then feeds back values ​​corresponding to estimates or predictions of the color that will be rendered by applying the cosmetic product PM to the user's lips.

[0055] To improve the rendering of the virtual try-on system, it is also possible to send a matte / glossy parameter to the rendering engine, which reflects the matte or glossy finish of the virtually applied lipstick, which may in particular form part of the reflectance parameters that the rendering engine may be supplied with to optimize the light rendering.

[0056] Although not described, the selection of the cosmetic product to be virtually tried can be made conventionally from an interface that presents all or part of a product catalog for which parameters are available and for which virtual trying is possible.

[0057] Furthermore, all or part of the steps are intended to be implemented by a computer, and to this end, the present application also relates to a computer program product comprising instructions, which, when executed by a computer, results in the implementation of at least the characteristic steps of the method. Preferably, the program also implements a virtual trial system and a rendering engine. The present application also relates to a system comprising a memory, in which the program product is stored. The system may also include any additional devices necessary for implementing the method, in particular any colorimetry data acquisition device.

Claims

1. 1. A method for simulating the application of a makeup product (PM) to a body surface, comprising: operating a rendering engine (R) to apply a makeup effect, including at least one color transformation, to at least a portion of input image data (X_S) including the body surface to be virtually made up, to generate an image based on image data (X_V) that simulates said application of a makeup product according to at least one characteristic color parameter (COL) of the virtually applied makeup product; The method comprises: obtaining color data (LIPS_P3) of the body surface to be virtually made up; - obtaining values ​​of the characteristic color parameters of the make-up product as a function of the color data of the body surface to be virtually made up, the values ​​of the characteristic color parameters being obtained from a database (DB) that associates at least one cosmetic make-up product standard (PM LS C) with a number of standard colors (LIPS1, LIPS2, LIPS3) of the body area under consideration; a preliminary step of determining said characteristic color parameters, comprising:

2. 2. A method according to claim 1, characterized in that the body area to be virtually made up is the lip area and said make-up product (PM) is a lip make-up product, in particular a lipstick (LSC).

3. 3. A method according to claim 1 or 2, characterized in that the colour data (LIPS_P3) of the body surface to be virtually made up is obtained from the input image data (X_S).

4. 3. The method according to claim 1 or 2, characterized in that the color data (LIPS_P3) of the body surface to be virtually made up is obtained separately from the input image data (X_S), in particular by color measurement using a colorimeter or spectrophotometer.

5. 3. A method according to claim 1 or 2, characterized in that the input image data (X_S) is an image (I2) of the body area made up with a make-up product having a particular desired finish or texture.

6. 3. The method according to claim 1 or 2, characterized in that the value of the characteristic color parameter (COL) of the makeup product (PM) is selected from the database as being assigned to the reference color among the plurality of reference colors that is closest to the color of the body area to be virtually made up obtained from the input image data.

7. 3. The method according to claim 1 or 2, characterized in that the values ​​of the characteristic color parameters (COL) of the makeup product (PM) are obtained by applying a colorimetric rendering model (KM) to the intrinsic color data of the makeup product under consideration and to the color data of the body area to be made up obtained from the input image data (X_S), which simulates the color that should be obtained by applying the makeup product to the body area.

8. 3. A method according to claim 1 or 2, characterized in that values ​​of the color parameter (COL) are obtained for different partial regions of the body surface to be virtually made up, in particular for pixel-by-pixel color processing.

9. 3. Method according to claim 1 or 2, characterized in that the rendering engine (R) is able to receive a gloss / matt parameter.

Images