Makeup simulation system, makeup simulation method, makeup simulation program, and makeup simulation device

Abstract

To provide a makeup simulation system, device, method and program which enhance reproduction accuracy of makeup simulation.SOLUTION: In a system, a GPU of a server includes: an illumination environment estimation part for estimating illumination environment using a first image which is an image before makeup of a coating object or an image for illumination environment estimation; a first spectral reflectance calculation part for calculating first spectral reflectance in the coating object before the makeup from a first RGB value of the first image; a first color signal calculation part for calculating a first color signal in the coating object before the makeup using a first three-dimensional reflection characteristic model; a second color signal calculation part for calculating a second color signal of a cosmetic using a second three-dimensional reflection characteristic model; a third color signal calculation part for calculating a third color signal in the coating object in a makeup state; a second RGM value calculation part for calculating a second RGB value in the coating object in the makeup state; a second image generation part for generating a second image on the basis of the second RGB value; and a superimposed image generation part for generating a superimposed image where the first image and the second image are superimposed.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a makeup simulation system, a makeup simulation method, a makeup simulation program, and a makeup simulation device for reproducing a state in which a cosmetic is applied to an object to be coated.

Background Art

[0002] There is known a makeup simulation system that displays a simulation image of a face after applying a cosmetic using a user's face image. For example, Patent Document 1 describes a makeup simulation method for generating a simulation image of a face in a state where a cosmetic is applied by superimposing a pattern for each cosmetic on a face image of a user in an unmade-up state.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a cosmetic is applied to the face, the area where the cosmetic is applied has a color that combines the color of the skin and the color of the cosmetic. That is, when the color of the user's skin is different, even when the same cosmetic is applied, the area where the cosmetic is applied has a different color. However, in the method of simply superimposing the pattern of the cosmetic on the user's face image as described above, the color of the skin is not reflected in the simulation image, and the deviation between the simulation image and the appearance when the user actually applies makeup becomes large. Therefore, it is difficult to perform a highly accurate makeup simulation by the above method.

[0005] Therefore, the main object of the present invention is to provide a technique capable of enhancing the reproduction accuracy of makeup simulation. [Means for solving the problem]

[0006] The present invention A lighting environment estimation unit estimates the lighting environment in the first image or the lighting environment estimation image, using a first image which is an image of the object to be coated with the cosmetic before the cosmetic is applied, A first spectral reflectance calculation unit calculates a first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A first color signal calculation unit calculates a first color signal for the object to be coated before cosmetic application using the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions. A second color signal calculation unit calculates a second color signal of the cosmetic using a second three-dimensional reflection characteristic model expressed by the second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, the spectral distribution of the illumination light, and the geometric conditions. A third color signal calculation unit calculates a third color signal for the object to be coated in a cosmetic state after the cosmetic has been applied to the object, based on the first color signal and the second color signal. A second RGB value calculation unit calculates a second RGB value for the object to be coated in the cosmetic state from the third color signal, A second image generation unit that generates a second image based on the second RGB value, A superimposed image generation unit generates a superimposed image by superimposing the first image and the second image, A makeup simulation system equipped with To provide. The first image or the image used for estimating the lighting environment may be an RGB color image, a multiband image, or a multispectral image. The first image and the image used for estimating the lighting environment may be the same image. The aforementioned cosmetic simulation system may further include an illumination environment database that stores light source information for each illumination environment, which is a combination of the illumination spectral distribution, which is the spectral distribution of illumination light that can be irradiated onto the object to be coated, and the spatial distribution of the illumination light source that generates the illumination light. In cases where the first image or the image used for estimating the lighting environment is an RGB image or a multiband image, the cosmetic simulation system may further include a conversion matrix storage unit that stores a system conversion matrix for determining the spectral distribution of illumination light from RGB values, and the lighting environment estimation unit may estimate the lighting environment in the first image or the image used for estimating the lighting environment by estimating the spatial distribution of the illumination light source based on the illumination direction vector estimated from the first image or the image used for estimating the lighting environment and the spectral distribution of the illumination light determined from the RGB values ​​of the first image or the image used for estimating the lighting environment using the system conversion matrix for determining the spectral distribution of the illumination light. In cases where the first image or the image used for estimating the lighting environment is a multispectral image, the lighting environment estimation unit may estimate the lighting environment in the first image or the image used for estimating the lighting environment based on the spectral distribution of the illumination light obtained from the first image or the image used for estimating the lighting environment. The cosmetic simulation system may further include a cosmetic database that stores information relating the second reflection characteristic parameter to the physical properties of the cosmetic, and the second color signal calculation unit may read the second reflection characteristic parameter corresponding to the information relating to the physical properties of the cosmetic from the cosmetic database and calculate the second color signal of the cosmetic using a second three-dimensional reflection characteristic model expressed by the read second reflection characteristic parameter, the spectral distribution of the illumination light, and the geometric conditions. The second RGB value calculation unit may correct the third color signal based on the characteristics of human vision and the characteristics of the color image output device that outputs the superimposed image, and calculate the second RGB value of the object to be coated in the cosmetic state from the corrected third color signal. Furthermore, the present invention is A shape estimation unit that estimates the three-dimensional shape of the object to be coated from a first image, which is an image of the object to be coated before the cosmetic is applied, A lighting environment estimation unit that estimates the lighting environment in the first image or the lighting environment estimation image using the first image or the lighting environment estimation image, A first spectral reflectance calculation unit calculates a first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A first color signal calculation unit calculates a first color signal for the object to be coated before cosmetic application using the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions. A second color signal calculation unit calculates a second color signal of the cosmetic using a second three-dimensional reflection characteristic model expressed by the second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, the spectral distribution of the illumination light, and the geometric conditions. A third color signal calculation unit calculates a third color signal for the object to be coated in a cosmetic state after the cosmetic has been applied to the object, based on the first color signal and the second color signal. A second RGB value calculation unit calculates a second RGB value for the object to be coated in the cosmetic state from the third color signal, A second image generation unit that generates a second image based on the second RGB value, A superimposed image generation unit generates a superimposed image by superimposing the first image and the second image, A makeup simulation system equipped with We also offer it. Furthermore, the present invention is A lighting environment estimation unit estimates the lighting environment in the first image or the lighting environment estimation image, using a first image which is an image of the object to be coated with the cosmetic before the cosmetic is applied, A first spectral reflectance calculation unit calculates a first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A first color signal calculation unit calculates a first color signal for the object to be coated before cosmetic application using the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions. A feature area identification unit that identifies the feature area of ​​the object to be coated from the first image, A second color signal calculation unit calculates a second color signal of the cosmetic using a second three-dimensional reflection characteristic model expressed by the second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, the spectral distribution of the illumination light, and the geometric conditions. A third color signal calculation unit calculates a third color signal for the object to be coated in a cosmetic state after the cosmetic has been applied to the object, based on the first color signal and the second color signal. A second RGB value calculation unit calculates a second RGB value for the object to be coated in the cosmetic state from the third color signal, A second image generation unit that generates a second image based on the second RGB value, A superimposed image generation unit generates a superimposed image by superimposing the first image and the second image, A makeup simulation system equipped with We also offer it. Furthermore, the present invention is A skin texture identification unit identifies the surface irregularities of an object to be coated as a mathematical model from a first image, which is an image of the object to be coated before the cosmetic is applied. A lighting environment estimation unit that estimates the lighting environment in the first image or the lighting environment estimation image using the first image or the lighting environment estimation image, A first spectral reflectance calculation unit calculates a first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A first color signal calculation unit calculates a first color signal for the object to be coated before cosmetic application using the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions. A second color signal calculation unit calculates a second color signal of the cosmetic using a second three-dimensional reflection characteristic model expressed by the second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, the spectral distribution of the illumination light, and the geometric conditions. A third color signal calculation unit calculates a third color signal for the object to be coated in a cosmetic state after the cosmetic has been applied to the object, based on the first color signal and the second color signal. A second RGB value calculation unit calculates a second RGB value for the object to be coated in the cosmetic state from the third color signal, A second image generation unit that generates a second image based on the second RGB value, A superimposed image generation unit generates a superimposed image by superimposing the first image and the second image, A makeup simulation system equipped with the following features. The first color signal calculation unit or the second color signal calculation unit may calculate the change in shading or gloss due to the unevenness of the surface of the object to be coated, and further use the texture information obtained by the calculation, which includes information regarding the shading or gloss due to the unevenness of the surface of the object to be coated, to calculate the first color signal or the second color signal. Furthermore, the present invention is A step of estimating the lighting environment in the first image or the lighting environment estimation image, using a first image which is an image of the object to be coated with the cosmetic before the cosmetic is applied, A step of calculating the first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A step of calculating a first color signal in the object to be coated before cosmetic application using the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions, A step of calculating the second color signal of the cosmetic using a second three-dimensional reflection characteristic model expressed by the second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, the spectral distribution of the illumination light, and the geometric conditions; A step of calculating a third color signal in the object to be coated in a cosmetic state after the cosmetic has been applied to the object to be coated, from the first color signal and the second color signal, A step of calculating a second RGB value for the object to be coated in the cosmetic state from the third color signal, A step of generating a second image based on the second RGB value, A step of generating a superimposed image by superimposing the first image and the second image, Makeup simulation methods, including We also offer it. Furthermore, the present invention is On the computer, A process for estimating the lighting environment in the first image or the lighting environment estimation image, using a first image which is an image of the object to be coated with the cosmetic before the cosmetic is applied, A process for calculating the first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A process for calculating a first color signal in the object to be coated before cosmetic application, using the first spectral reflectance, a first reflection characteristic parameter defining the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions, A process for calculating the second color signal of the cosmetic using a second three-dimensional reflection characteristic model expressed by the second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, the spectral distribution of the illumination light, and the geometric conditions, A process to calculate a third color signal in the object to be coated in the cosmetic state after the cosmetic has been applied to the object, from the first color signal and the second color signal, A process for calculating the second RGB value of the object to be coated in the cosmetic state from the third color signal, A process for generating a second image based on the second RGB value, A process to generate a superimposed image by superimposing the first image and the second image, A makeup simulation program that runs We also offer it. Furthermore, the present invention is A lighting environment estimation unit estimates the lighting environment in the first image or lighting environment estimation image, using a first image which is an image of the object to be coated with the cosmetic before the cosmetic is applied, A first spectral reflectance calculation unit calculates a first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A first color signal calculation unit calculates a first color signal in the object to be coated before cosmetic application using the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions. A second color signal calculation unit calculates a second color signal of the cosmetic using a second three-dimensional reflection characteristic model expressed by the second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, the spectral distribution of the illumination light, and the geometric conditions. A third color signal calculation unit calculates a third color signal for the object to be coated in a cosmetic state after the cosmetic has been applied to the object, based on the first color signal and the second color signal. A second RGB value calculation unit calculates a second RGB value for the object to be coated in the cosmetic state from the third color signal, A second image generation unit that generates a second image based on the second RGB value, A superimposed image generation unit generates a superimposed image by superimposing the first image and the second image, A cosmetic simulation device equipped with We also offer it. [Effects of the Invention]

[0007] The present invention provides a technology that can improve the accuracy of cosmetic simulations. In particular, the present invention provides a technology that can synthesize a three-dimensional image of a cosmetic film rendered in 3D based on 3DCG technology with an image of the object to which the cosmetic film is applied. The effects of the present invention are not limited to those described herein, but may include any of the effects described herein. [Brief explanation of the drawing]

[0008] [Figure 1] A schematic diagram showing the overall configuration of the makeup simulation system. [Figure 2] A block diagram showing the configuration of a user terminal. [Figure 3] A block diagram showing the server configuration. [Figure 4] A functional block diagram of the GPU installed in the server. [Figure 5] A schematic diagram representing the app image shown in the first image. [Figure 6] A schematic diagram representing an app image showing the cosmetic product selection area. [Figure 7] A flowchart illustrating the processing steps in a makeup simulation system. [Figure 8] A schematic diagram illustrating the geometric model of reflection in a makeup simulation system. [Figure 9] A schematic diagram showing the reflected light distribution in the geometric model of reflection shown in Figure 8. [Figure 10] A schematic diagram illustrating interference light generated by a multilayer film of luminescent agents contained in cosmetics. [Figure 11] A schematic diagram representing an app image with superimposed images displayed. [Modes for carrying out the invention]

[0009] 1. First Embodiment The following describes a first embodiment of a makeup simulation system, a makeup simulation method, a makeup simulation program, and a makeup simulation device with reference to Figures 1 to 11. In this embodiment, the subject of the makeup simulation is the face, and the object to which the cosmetic is applied is the skin of the face or the lips. In the following description, "image" is a concept that includes both still images and videos. Furthermore, "video" is a concept that includes both real-time video footage and pre-recorded video footage.

[0010] [Overall structure] As shown in Figure 1, the cosmetic simulation system 1 comprises a user terminal 10 and a server 20. The user terminal 10 is an information processing device owned by user 2, and is, for example, a smartphone. The user terminal 10 may also be an information processing device such as a tablet, laptop computer, or desktop computer.

[0011] Server 20 is a server computer that performs various processes in the makeup simulation system 1, and is an example of a makeup simulation device for performing makeup simulations. Specifically, Server 20 generates a second image, which is an image of the skin and lips of the face in a made-up state, where cosmetics have been applied. Server 20 also superimposes the second image onto a first image, which is an image of the skin and lips of the face in an unmade-up state, before cosmetics have been applied. In this embodiment, the first image may include the object to be applied, and may also include objects other than the object to be applied. For example, if the object to be applied is the lips, the first image may include the lips and parts other than the lips, and specifically may be a face image.

[0012] The user terminal 10 and the server 20 are connected via a communication line, which is a network 3. Network 3 is, for example, a mobile communication system such as 4G or 5G for mobile phones, or a wireless LAN communication system such as Wi-Fi (registered trademark).

[0013] The user terminal 10 has an image display program installed as an application program. The image display program is a program that executes the process of displaying various images on the user terminal 10, such as the first image and superimposed images created by superimposing the first image and the second image. The image display program is downloaded from the application distribution device via the network and installed on the user terminal 10.

[0014] The user terminal 10 is equipped with a touch panel 13. Various images are displayed on the touch panel 13, such as the first image 30a, which is an image in its unpainted state, and the superimposed image 30b, which is a simulation result.

[0015] [User terminal] As shown in Figure 2, the user terminal 10 comprises a control unit 11, a memory 12, a touch panel 13, an imaging unit 14, and a communication unit 15. The control unit 11 is, for example, a CPU and controls the overall operation of the user terminal 10.

[0016] Memory 12 comprises main memory and a data storage unit. The main memory of memory 12 is, for example, RAM, and temporarily stores data and programs from the user terminal 10. The data storage unit of memory 12 is non-volatile memory, for example, flash memory. The data storage unit of memory 12 stores programs for executing various processes on the user terminal 10, such as image display programs.

[0017] The touch panel 13 is a display that enables touch operation input by combining a display device such as an LCD panel or an organic EL panel with a position input device. In other words, the touch panel 13 is a display unit on the user terminal 10 where the image 30 is displayed, and also an operation unit for inputting operations to the server 20.

[0018] The imaging unit 14 is, for example, an RGB camera equipped with an image sensor such as a CCD or CMOS. The imaging unit 14 detects light such as visible light and outputs image data composed of red (R), green (G), and blue (B) RGB values. In the following, the RGB values ​​of the image acquired by the imaging unit 14 will be referred to as the first RGB values.

[0019] The imaging unit 14 acquires a facial image of user 2 in an unmakeup state, consisting of first RGB values ​​(a first image including the skin and lips of the face in an unmakeup state). The imaging unit 14 may also acquire an image for estimating the lighting environment, which will be used in the lighting environment estimation unit described later.

[0020] The communication unit 15 communicates with the server 20 via the network 3. Specifically, the communication unit 15 transmits signals from the user terminal 10 to the server 20 and receives data such as images generated by the server 20. The communication unit 15 may also be able to communicate with information processing devices other than the server 20 (not shown) via the network 3. In this case, the communication unit 15 may receive, for example, the first image or the image for estimating the lighting environment from an information processing device other than the server 20.

[0021] 〔server〕 As shown in Figure 3, the server 20 comprises a control unit 21, a memory 24, and a communication unit 25. The control unit 21, as an example, comprises a CPU 22 and a GPU 23. The CPU 22 controls the overall operation of the server 20, excluding the image generation process. The GPU 23 executes the image generation process in the server 20.

[0022] Memory 24 comprises main memory and a data storage unit. The main memory of memory 24 is, for example, RAM, which temporarily stores data and programs of the server 20. The data storage unit of memory 24 is non-volatile memory, for example, flash memory such as an HDD or SSD.

[0023] The data storage unit of memory 24 stores programs for executing various processes on server 20, such as a makeup simulation program. The makeup simulation program is a program that causes server 20 to execute the process of generating a second image and then superimposing the first and second images to create a superimposed image. The processing in the makeup simulation program is controlled by the GPU 23 of the control unit 21.

[0024] The memory 24 includes a spectral conversion database 24a, a lighting environment database 24b, and a cosmetic database 24c, which serve as data storage units. The memory 24 may also further include a conversion matrix storage unit 24d that stores a system conversion matrix for determining the spectral distribution of illumination light from RGB values.

[0025] The spectral conversion database 24a stores a first conversion function M for converting the first RGB values ​​that constitute the object to be coated, which is the object to which the cosmetic is applied, in the first image to the first spectral reflectance S1(λ), ​​which is the spectral reflectance of the object in its uncoated state. In this embodiment, the spectral conversion database 24a stores a first conversion function M for converting the first RGB values ​​to the first spectral reflectance S1(λ) for each of the bare skin and lips on the face, which are examples of objects to be coated.

[0026] The first transformation function M is a transformation matrix obtained by measuring the RGB values ​​and spectral reflectance of coated objects for a large number of people and statistically analyzing the correspondence between the RGB values ​​and spectral reflectance of the coated objects for each color of the object measured by the person. As an example, the first transformation function M is a 61 × 3 transformation matrix that divides the visible wavelength range from 400 nm to 700 nm into 5 nm intervals.

[0027] As a specific example, the spectral conversion database 24a divides skin into three types—Type 1, Type 2, and Type 3—according to the amount of melanin pigment contained in the skin, which is an example of an object to be coated, and stores a first conversion function M corresponding to the skin color. Type 1 skin is skin within the first range, where the amount of melanin pigment is less than that of Types 2 and 3, and has a high relative reflectance of each wavelength when light is shone on the skin. Type 2 skin is skin within the second range, where the amount of melanin pigment is greater than that of the first range, and has a moderate relative reflectance of each wavelength when light is shone on the skin. Type 3 skin is skin within the third range, where the amount of melanin pigment is greater than that of the second range, and has a low relative reflectance of each wavelength when light is shone on the skin. In addition, the spectral conversion database 24a may, for example, divide the RGB values ​​of the skin into predetermined ranges and store a first conversion function M for each division of the RGB values ​​of the skin.

[0028] The lighting environment database 24b stores light source information for each lighting environment, which is a combination of the illumination spectral distribution E(λ), which is the spectral distribution of illumination light that may be irradiated onto the object to be coated, and the spatial distribution of the illumination light source that generates the illumination light. In this embodiment, illumination light sources include, for example, direct light sources that directly irradiate the object to be coated with illumination light, and reflected light sources from which objects present in the lighting environment reflect illumination light from the direct light source towards the object to be coated, but these are not particularly distinguished. Therefore, the lighting environment database 24b stores the spatial distribution of illumination light sources, including direct light sources and reflected light sources, and the illumination spectral distribution E(λ) of the illumination light irradiated from the illumination light sources, for all directions centered on the object to be coated. The spatial distribution of illumination light sources is used as information to define geometric conditions such as the angle of incidence of illumination light on the object to be coated.

[0029] Furthermore, the lighting environment referred to here is a combination of the illumination spectral distribution E(λ) and the spatial distribution of the illumination light source used to reproduce the lighting effect in a specific scene on the object to be coated. Specific examples of lighting environments include indoor environments where artificial direct light sources are applied, and outdoor environments where natural direct light sources are applied. Outdoor environments are further divided into time periods such as morning, noon, evening, and night. Examples of artificial direct light sources include incandescent bulbs, fluorescent lamps, and LED lights, while examples of natural direct light sources include the sun and moon.

[0030] The illumination spectral distribution E(λ) and the spatial distribution of illumination sources can be determined, for example, by acquiring a spectral image in all directions in an actual lighting environment and sampling the spectral distribution every 5 nm in the visible wavelength range (e.g., 400-700 nm) in all directions. In this case, the lighting environment database 24b stores the illumination spectral distribution E(λ) for each pixel in the spectral image. More specifically, the position of each pixel in the spectral image provides information that identifies the spatial distribution of illumination sources distributed in all directions around the object to be coated. The illumination spectral distribution E(λ) assigned to each pixel becomes the illumination spectral distribution E(λ) of the illumination light irradiated onto the object to be coated. In other words, by considering each pixel in the spectral image as an illumination source and assigning an illumination spectral distribution E(λ) to each pixel, the spatial distribution of illumination sources in all directions and the illumination spectral distribution E(λ) of the illumination light irradiated from each illumination source can be obtained. With this method, illumination sources for the entire scene can be processed together without distinguishing between each direct light source and each reflected light source in an actual lighting environment. Therefore, even when there are multiple light sources whose boundaries cannot be distinguished, it becomes possible to reproduce the lighting environment.

[0031] The cosmetic database 24c stores a second transformation function T(λ) for each cosmetic product. The second transformation function T(λ) is a transformation matrix for calculating the second spectral reflectance S2(λ), which is the spectral reflectance of the cosmetic, based on the first spectral reflectance S1(λ). Here, "cosmetics" refers to cosmetics applied to an object, such as foundation, blush, eye color, and lipstick. In other words, the cosmetic database 24c stores a second transformation function T(λ) for each type of cosmetic, such as foundation and blush, and for each variation of the cosmetic, such as color and use.

[0032] More specifically, the cosmetic database 24c stores a second transformation function T(λ) for each cosmetic product, corresponding to the color of the object to which it is applied. For example, if the cosmetic product is applied to the skin, the cosmetic database 24c stores a second transformation function T(λ) for each cosmetic product, corresponding to the first type of skin, the second type of skin, and the third type of skin.

[0033] The second transformation function T(λ) is, for example, used for a large number of people, and the spectral reflectance S of the object to be coated for each subject. a And the spectral reflectance S of the object to be coated by the subject with the cosmetic applied. b The spectral reflectance S was measured and measured. a and spectral reflectance S b The change in the measured value is statistically calculated for each color of the object being coated by the subject.

[0034] Furthermore, the cosmetic database 24c stores the reflective properties parameters of each cosmetic product. As will be explained in detail later, reflective properties parameters are parameters that depend on the surface properties of an object, such as the roughness of the object's surface, and define the reflective properties such as the intensity of reflected light and the spread of gloss with respect to the angle of incidence of illuminating light on the object's surface. The reflective properties parameters of a cosmetic can be determined, for example, from actual measured values ​​obtained by measuring instruments such as a gloss meter on the actual cosmetic product.

[0035] Additionally, the cosmetic database 24c may store the spectral reflectance and reflectance characteristic parameters of luminescent agents such as pearls and glitter contained in cosmetics such as eye shadows. Note that the spectral reflectance of luminescent agents is calculated using a reflectance component independent of the skin's spectral reflectance, as it is less affected by the color of the object to which the cosmetic is applied when applied to the skin. Therefore, the calculation process using the second transformation function T(λ) is not performed. Consequently, the spectral reflectance of luminescent agents is stored as the measured value obtained from a measuring instrument.

[0036] The conversion matrix memory unit 24d stores a system conversion matrix for determining the spectral distribution of illumination light from RGB values. This system conversion matrix can be created from a database that stores color signals obtained by multiplying the spectral distribution of known light sources by the spectral reflectance of many skins.

[0037] The communication unit 25 communicates with the user terminal 10 via the network 3. Specifically, the communication unit 25 receives signals from the user terminal 10 and transmits data such as images generated by the server 20 to the user terminal 10.

[0038] [GPU] As shown in Figure 4, the GPU 23 in the control unit 21 functions as a shape estimation unit 23a, a skin texture identification unit 23b, a lighting environment estimation unit 23c, a first spectral reflectance calculation unit 23d, a first color signal calculation unit 23e, a feature area identification unit 24f, a second spectral reflectance calculation unit 23g, a second color signal calculation unit 23h, a third color signal calculation unit 23i, a second RGB value calculation unit 23j, a second image generation unit 23k, and a superimposed image generation unit 23l.

[0039] The shape estimation unit 23a estimates the three-dimensional shape of the object to be coated from a first image, which is an image of the object in its uncoated state. It can calculate normal vectors corresponding to each pixel from the shape information of the object to reproduce the texture of cosmetic materials or skin. The normal vectors are used in the three-dimensional reflection characteristic model to calculate the representation of the texture of the cosmetic coating, such as shading and gloss. In particular, in AR (augmented reality) technology, in order to reproduce the texture of the target object or cosmetic coating that changes moment by moment, such as the orientation of a face or facial expression, it is preferable that the shape estimation unit 23a calculates normal vectors from the image in real time and continuously so that it can immediately respond to changes in the texture of the target object. It is preferable that the shape estimation of the object to be coated is performed after acquiring the first image and before calculating the first spectral reflectance. The estimation of the three-dimensional shape by the shape estimation unit 23a can be achieved, for example, by known estimation processing.

[0040] The skin texture identification unit 23b identifies the surface irregularities of the object to be applied to from the first image as a mathematical model in order to identify the skin texture, including the fine irregularities of the skin surface. If the object to be applied to is part of the face, the surface irregularities may be, for example, fine irregularities on the skin surface. Fine irregularities on the skin surface refer to the uneven shapes present on the skin surface, and specific examples include wrinkles, skin texture, blemishes, and scars.

[0041] Texture characteristics related to subtle skin wrinkles and texture, which affect skin feel, are not directly represented by shape information, but rather expressed as changes in shading and color using a mathematical model. In this case, differences in wrinkle texture can be expressed by changing the values ​​of the parameters given to the mathematical model, and texture can be estimated by estimating these parameters from the image.

[0042] The lighting environment estimation unit 23c estimates the lighting environment in the first image or the lighting environment estimation image using the first image or the lighting environment estimation image. As will be described in detail later, the lighting environment estimation unit 23c estimates the lighting environment in the first image or the lighting environment estimation image by estimating the spatial distribution of the illumination light based on the illumination direction vector estimated from the first image or the lighting environment estimation image and the spectral distribution of the illumination light obtained from the RGB values ​​of the first image or the lighting environment estimation image using a system transformation matrix for determining the spectral distribution of the illumination light. Specifically, the light source information estimated is estimated as the spatial distribution and spectral distribution of the illumination light in the scene.

[0043] The first spectral reflectance calculation unit 23d calculates the first spectral reflectance S1(λ) based on the first RGB value and the first conversion function M. The first color signal calculation unit 23e calculates the first color signal C1(λ), ​​which is the color signal when illumination light is reflected off the object to be coated before cosmetic application, based on the first three-dimensional reflection characteristic model, which is a three-dimensional reflection characteristic model of the object to be coated before cosmetic application.

[0044] The three-dimensional reflection characteristic model here is a mathematical model that describes the process of light reflection of an object. The spectral reflectance and reflection characteristic parameters of the object that constitute the three-dimensional reflection characteristic model define the color and texture of the object. By providing geometric conditions such as the spectral distribution of the illuminating light and the angle of incidence of the illuminating light to the three-dimensional reflection characteristic model, the color signal when the illuminating light is reflected by the object can be calculated. In other words, the first three-dimensional reflection characteristic model is a mathematical model that describes the process of light reflection in an uncoated object.

[0045] Furthermore, the color signal here refers to the spectral distribution (energy spectral distribution) of electromagnetic waves that are reflected by illuminating light off an object and incident on the visual system, such as a person's eye or a camera. The color signal is converted into color information such as RGB values ​​that are actually perceived by humans, using RGB color matching functions, etc. Note that the first color signal C1(λ) corresponds to the color signal when illuminating light is reflected off an uncoated object.

[0046] The feature identification unit 23f identifies feature parts of the object to be coated from the first image, which is an image of the object to be coated in an uncoated state. Identifying feature parts of the object to be coated may, for example, involve identifying the location and region of the object. For example, if the first image is a face image, the feature identification unit 23f identifies elements that make up the face, such as the eyes, nose, and mouth, and thereby identifies the location and region (outline or area enclosed by the outline) of the elements that make up the face. Identification of feature parts by the feature identification unit 23f can be achieved, for example, by known image recognition technology.

[0047] The second spectral reflectance calculation unit 23g calculates the second spectral reflectance S2(λ) based on the first spectral reflectance S1(λ) and the second transformation function T(λ). The second color signal calculation unit 23h calculates the second color signal C2(λ), which is the color signal when illumination light is reflected off the cosmetic, based on the second three-dimensional reflection characteristic model, which is a three-dimensional reflection characteristic model of the cosmetic. The second three-dimensional reflection characteristic model is a mathematical model that describes the process of light reflection in the cosmetic. In the following, the reflection characteristic parameters for the uncoated object are referred to as the first reflection characteristic parameters, and the reflection characteristic parameters for the cosmetic are referred to as the second reflection characteristic parameters.

[0048] The third color signal calculation unit 23i calculates the third color signal C3(λ), which is the color signal of the object to be coated in a cosmetic state, based on the first color signal C1(λ) and the second color signal C2(λ). The third color signal C3(λ) corresponds to the color signal when illumination light is reflected off the object to be coated in a cosmetic state, that is, off the object to be coated and the cosmetic applied to the object. The second RGB value calculation unit 23j calculates the second RGB value, which is the RGB value of the object to be coated in a cosmetic state, based on the third color signal C3(λ) calculated by the third color signal calculation unit 23i.

[0049] The second image generation unit 23k generates a second image, which is an image of the object to be coated in a cosmetic state, based on the second RGB value calculated by the second RGB value calculation unit 23j. The superimposed image generation unit 23l generates a superimposed image by superimposing the first image, which is an image of the object to be coated in an uncoated state, and the second image generated by the second image generation unit 23k. The superimposed image is an image that shows the simulation results.

[0050] [App image] As shown in Figure 5, the touch panel 13 of the user terminal 10 displays an application image 40, which is an application image displayed by an image display program. The application image 40 comprises an image display area 41 and an operation area 42. The image display area 41 is an area where images 30, such as a first image and a superimposed image obtained by superimposing the first image and the second image, are displayed.

[0051] The operation area 42 includes, for example, a switching object 42a, a cosmetic area selection object 42b, a cosmetic product selection object 42c, a lighting selection object 42d, and an operation object 42e. The switching object 42a is an object that switches the simulation result ON or OFF for each cosmetic product.

[0052] The makeup area selection object 42b is an object for manually specifying the makeup area 31, which is the area on the object to which the cosmetic product will be applied. One example of how to specify the makeup area 31 is to specify the area to which the cosmetic product will be applied by tapping on the first image 30a displayed in the image display area 41. For example, the makeup area 31 could be the entire face for foundation, the skin around the cheekbones for blush, the skin around the eyes for eye shadow, and the lips for lipstick. The makeup area 31 may also be set automatically using an image recognition function or the like.

[0053] The cosmetic selection object 42c is an object used to select the cosmetic product to be simulated. As shown in Figure 6, tapping the cosmetic selection object 42c displays the cosmetic selection area 43 in the image display area 41. In the cosmetic selection area 43, the cosmetics registered in the cosmetic database 24c are displayed as product objects 43a for each product. By tapping the product object 43a, the cosmetic product to be applied in the cosmetic simulation is selected.

[0054] For example, in the case of foundation, you select the product object 43a of the product you want to simulate from the product objects 43a displayed for each type of foundation, such as liquid foundation or powder foundation, and for each color.

[0055] As another example, the selection of the makeup area and the cosmetic product may be performed without using the makeup area selection object 42b and the cosmetic product selection object 42c. For example, by tapping the area to which the cosmetic product is to be applied in the first image displayed in the image display area 41, the cosmetic product corresponding to the tapped area may be automatically selected. Specifically, if the area near the cheekbone is tapped in the first image, blush will be automatically selected as the cosmetic product and applied using an application method suitable for that blush as specified by a makeup professional. This makes it possible to reproduce the image of how the cosmetic product would look if applied by a makeup professional with various makeup know-how. Multiple application methods may be selected, for example, according to age, atmosphere, or time, place, and occasion.

[0056] The lighting selection object 42d is an object used to select the lighting environment in the makeup simulation. By tapping the makeup selection object 42c, a lighting selection area (not shown) is displayed in the image display area 41. In the lighting selection area, various lighting environments registered in the lighting environment database 24b are displayed as lighting objects for each lighting environment. By tapping a lighting object, the lighting environment to be applied in the makeup simulation is selected. For example, if you want to perform the makeup simulation in a lighting environment different from the lighting environment of the location where the first image was taken, you can tap the lighting selection object to select any lighting environment.

[0057] The operation object 42e is an object used to move, rotate, enlarge, and reduce the image displayed in the image display area 41.

[0058] [Operation of the First Embodiment] Next, we will explain the operation of the makeup simulation system 1. As shown in Figure 7, in step S1, the control unit 11 of the user terminal 10 causes the imaging unit 14 to perform a process to acquire a facial image of user 2 in an unmakeup state (a first image including the skin and lips of the face in an unmakeup state). Then, the control unit 11 performs a process to send the first image, which is composed of first RGB values, to the server 20.

[0059] The first image acquired here may be, for example, a single still image, multiple still images taken from different angles, or a video.

[0060] Furthermore, when capturing the first image in step S1, it is affected by various factors such as lighting in the shooting environment. From the perspective of obtaining a first image that is more suitable for makeup simulation, for example, as a correction to reduce the influence of the shooting environment, it is possible to automatically adjust the brightness and saturation of the captured first image to an appropriate color according to the surrounding lighting environment using an image processing program stored in memory 12. Alternatively, by using the lighting environment database 24b, it may be possible to recreate images that anticipate how cosmetics will look (changes in color) in a scene different from the time of shooting.

[0061] In step S2, the shape estimation unit 23a of the server 20 estimates the three-dimensional shape of the object to be coated from the first image.

[0062] In step S3, the skin texture identification unit 23b identifies the surface irregularities of the object to be coated as a mathematical model from the first image. The skin texture identification unit 23b identifies the surface irregularities of the object to be coated using a mathematical model, for example, by improving the V-groove model of the Torrance-Sparrow model or the Oren-Nayer model.

[0063] In step S4, the lighting environment estimation unit 23c estimates the lighting environment in the first image using the first image transmitted from the user terminal 10. Alternatively, the lighting environment estimation unit 23c estimates the lighting environment in the lighting environment estimation image using the lighting environment estimation image. In one embodiment, the lighting environment estimation image is a different image from the first image, and may be, for example, an image that does not include the object to be coated as a subject, or an image that includes the object to be coated as a subject but is different from the first image. This makes it possible to estimate the lighting environment, which is not limited to the first image, for example, by estimating the lighting environment from ambient light or cast match images in an image in which the object to be coated is not included as a subject. Also, in one embodiment, the first image and the lighting environment estimation image may be the same image.

[0064] Specifically, the lighting environment estimation unit 23c estimates the illumination direction vector from the first image or the image used for lighting environment estimation. Furthermore, the lighting environment estimation unit 23c uses a system transformation matrix stored in the transformation matrix storage unit 14d to determine the spectral distribution of illumination light from the RGB values ​​of the first image or the image used for lighting environment estimation. Based on the obtained illumination direction vector and the spectral distribution of illumination light, the lighting environment estimation unit 23c estimates the lighting environment in the first image or the image used for lighting environment estimation by estimating the spatial distribution of the illumination light source. Specifically, the estimated light source information is estimated as the spatial and spectral distribution of illumination light within the scene. This estimated lighting environment information (scene information) is also recorded as independent lighting environment information, so it can be applied to images taken in other scenes. For example, if lighting environment information estimated in a certain office is available, it can be used to recreate a facial image taken in a different location.

[0065] As an example of the process for estimating the illumination direction vector in the illumination environment estimation unit 23c, we will explain the case where estimation is performed using a face image. First, based on the three-dimensional shape information of the face estimated by the shape estimation unit 23a, the normal vector corresponding to each pixel of the face image is calculated. If the face is a convex hull object, such as the cheek or forehead, then if the object is a non-glossy object (matte reflection), the illumination direction can be estimated by finding the brightest pixel compared to its surroundings, and the normal vector of that pixel matches the illumination direction. If the object is glossy, the illumination direction can be estimated by finding the brightest pixel compared to its surroundings, and the specular reflection direction vector of the line-of-sight direction vector of that pixel matches the illumination direction.

[0066] Next, User 2 taps an object in the operation area 42 displayed on the touch panel 13 to set the makeup area 31 to which the cosmetic product will be applied, and then selects the cosmetic product to be simulated. The control unit 11 of the user terminal 10 sends a signal to the server 20 that identifies the makeup area 31 set by User 2 and the cosmetic product selected by User 2. Note that multiple different types of cosmetic products may be selected, for example, blush and foundation.

[0067] In step S5, the first spectral reflectance calculation unit 23d performs a process to calculate the first spectral reflectance S1(λ) in the cosmetic area 31 based on the first RGB values ​​that constitute the first image. Specifically, the first spectral reflectance calculation unit 23d reads a first conversion function M corresponding to the color of the object to be coated in the cosmetic area 31 from the spectral conversion database 24a and performs a process to calculate the first spectral reflectance S1(λ) in the cosmetic area 31.

[0068] The first conversion function M read out here may be determined automatically, for example, according to the first RGB values ​​that constitute the cosmetic area 31 of the first image, or it may be determined by the user 2 selecting a color close to the object to be coated from a color sample such as a color code.

[0069] The first spectral reflectance S1(λ) can be approximated as a discrete 61-dimensional spectral reflectance vector s obtained by dividing the visible wavelength range from 400 nm to 700 nm into 5 nm intervals using the first transformation function M and the values ​​of each element R1, G1, and B1 in the first RGB values. The first spectral reflectance S1(λ) can be expressed by the following equations (1) and (2).

[0070]

number

[0071] By calculating the first spectral reflectance S1(λ) based on a first transformation function M corresponding to the color of the object to be coated, it is possible to calculate a first spectral reflectance S1(λ) that is closer to the actual spectral reflectance of the object to be coated. Furthermore, for example, when the object to be coated is the skin on the face, it is generally difficult to directly measure the spectral reflectance of the entire skin on the face because the measurement range of measuring instruments such as spectrophotometers is small. In contrast, as in this embodiment, by calculating the first spectral reflectance S1(λ) from the first RGB values ​​of an unmakeup-free face image, it is possible to suitably obtain the spectral reflectance of the object to be coated in an unmakeup-free state.

[0072] In this embodiment, for example, in areas where cosmetics are not applied, such as hair or accessories like earrings, i.e., in areas other than the cosmetic area 31, the process of calculating the first spectral reflectance S1(λ) from the first RGB values ​​is not performed. This simplifies the calculation process when calculating the first spectral reflectance S1(λ) from the first RGB values.

[0073] In step S6, the first color signal calculation unit 23e performs a process to calculate the first color signal C1(λ) based on the first spectral reflectance S1(λ), ​​the first reflection characteristic parameter, the illumination spectral distribution E(λ) of the illumination light, and the first three-dimensional reflection characteristic model expressed by the geometric conditions of the illumination light.

[0074] Specifically, the first color signal calculation unit 23e executes a process of calculating the first color signal C1(λ) using the illumination spectral distribution E(λ) and the spatial distribution of the illumination light source in the illumination environment estimated by the illumination environment estimation unit 23c and the first three-dimensional reflection characteristic model.

[0075] The first three-dimensional reflection characteristic model can be expressed by the following formula (3) using the first spectral reflectance S1(λ), the illumination spectral distribution E(λ), the first diffuse reflection function D a , and the first specular reflection function G a . Here, the first diffuse reflection function D a is a function that defines the diffuse reflection in the object to be coated in the unmade-up state. The first specular reflection function G a is a function that defines the specular reflection in the object to be coated in the unmade-up state. In the first three-dimensional reflection characteristic model, the first term on the right side represents the diffuse reflection component, and the second term on the right side represents the specular reflection component.

[0076]

Equation

[0077] Here, referring to FIGS. 8 and 9, the geometric model of reflection in the present embodiment will be described. As shown in FIG. 8, the virtual space 50 includes a virtual point 51, a virtual light source 52, and a virtual viewing point 53. The virtual point 51 is a point on the surface S of the object to be coated in the unmade-up state that receives light from the virtual light source 52. The virtual light source 52 irradiates illumination light on the virtual point 51. The virtual viewing point 53 is a viewing point for observing the virtual point 51.

[0078] Also, the illumination direction vector L is a vector from the virtual point 51 to the virtual light source 52. The normal vector N is a vector in the normal direction at the virtual point 51 on the surface S. The viewing direction vector V is a vector from the virtual point 51 to the virtual viewing point 53. The illumination direction vector L, the normal vector N, and the viewing direction vector V are parameters for defining the geometric conditions of the illumination light in the three-dimensional reflection characteristic model. And the angle formed by the illumination direction vector L and the normal vector N is the incident angle θ iLet the angle between the normal vector N and the line-of-sight vector V be the receiving angle θ. r Let's assume that.

[0079] Generally, when an object is exposed to illumination light, the color signal C0(λ) generated on the object's surface is expressed as C0(λ) = E0(λ)S0(λ), using the spectral reflectance S0(λ) of the object and the spectral distribution E0(λ) of the illumination light.

[0080] In this embodiment, in addition to simply calculating the object's color signal from the object's spectral reflectance and the spectral distribution of the illumination light, a three-dimensional reflection characteristic model that takes into account diffuse reflection and specular reflection on the object's surface is used to more accurately reproduce the gloss and texture of the object's surface.

[0081] Specifically, when a virtual point 51 receives light from a virtual light source 52, the diffuse reflected light generated in all directions at the virtual point 51 and the incident angle θ i and the light-receiving angle θ r When the two are approximated, a reflected light is produced consisting of specularly reflected light and reflected light.

[0082] As shown in Figure 9, the intensity distribution of reflected light at the virtual point 51 is represented by the reflected light distribution RD, which is the sum of the diffuse reflected light intensity α and the specular reflected light intensity I. The intensity of the reflected light at the virtual point 51 is determined by the receiving angle θ. r It shows different values ​​depending on the following: Specifically, the distance from the virtual point 51 to the reflected light distribution RD in Figure 9 is the receiving angle θ. r This represents the intensity of reflected light at that location.

[0083] The diffuse reflected light intensity α is a reflection characteristic parameter that defines the intensity of diffuse reflected light. The diffuse reflected light intensity α is determined by the angle of incidence θ. i Receiving angle θ r Regardless of the value, it remains approximately constant. The specular reflected light intensity I is a reflection characteristic parameter that defines the relative intensity of the specular reflected light to the diffuse reflected light intensity α. The specular reflected light intensity I is equal to the incident angle θ i Receiving angle θ r As the value approaches, it increases, and the angle of incidence θ iand the light-receiving angle θ r The maximum value is reached when these two conditions are met.

[0084] For example, the light-receiving angle θ r The angle of incidence is θ i The light-receiving angle θ is significantly different. r1 In this case, the intensity of the reflected light observed at the virtual viewpoint 53a will be the value of the diffuse reflected light intensity α. In contrast, the receiving angle θ r The angle of incidence is θ i The light receiving angle θ approximates this. r2 In this case, the intensity of the reflected light observed at the virtual viewpoint 53b is the sum of the diffuse reflected light intensity α and the specular reflected light intensity I.

[0085] First diffuse reflectance function D a This can be expressed by the following equation (4), using the first diffuse reflected light intensity α1 and the illumination direction vector L and normal vector N as geometric conditions for the illumination light. The first diffuse reflected light intensity α1 is the first reflection characteristic parameter that defines the intensity of diffuse reflected light on the uncoated object. Here, the first diffuse reflected light intensity α1 is treated as 1. The geometric conditions for the illumination light used in the following calculations are calculated from the spatial distribution of the illumination light source, the shape of the object to be coated, and the viewpoint position from which the object to be coated is observed.

[0086]

number

[0087] Furthermore, in the reflected light distribution RD, the range over which specular reflection occurs is expressed by the specular reflection parameter m. The specular reflection parameter m is a parameter that depends on the surface properties of the object, such as the roughness of the object's surface, and is a reflection characteristic parameter that defines the spread of gloss. The larger the specular reflection parameter m, the larger the range over which specular reflection occurs.

[0088] First specular reflection function G aThis can be expressed by the following equations (5) to (7), using the first specular reflection intensity I1, the first specular reflection parameter m1, and the illumination direction vector L, normal vector N, and line-of-sight direction vector V as geometric conditions for the illumination light. Note that equation (6) is the Beckmann function. The first specular reflection intensity I1 is a first reflection characteristic parameter that defines the intensity of specular reflection on an uncoated object. The first specular reflection parameter m1 is a first reflection characteristic parameter that defines the spread of gloss on an uncoated object. The first specular reflection intensity I1 and the first specular reflection parameter m1 are set to arbitrary values ​​in advance.

[0089]

number

[0090] Furthermore, the first specular reflection intensity I1 and the first specular reflection parameter m1 can also be determined, for example, from multiple images of user 2's face in an unmakeup state, with varying incident angles of illumination light. For example, in step S1, when acquiring an image of user 2's face in an unmakeup state, the first specular reflection intensity I1 and the first specular reflection parameter m1 can be determined by acquiring multiple images of user 2's face with varying incident angles of illumination light.

[0091] In this way, by calculating the first color signal C1(λ) using the first three-dimensional reflection characteristic model, the color of the object to be coated in its uncoated state, as well as the shading and gloss caused by the illumination light, can be reproduced as the first color signal C1(λ).

[0092] In step S6, the first color signal calculation unit 23e may calculate the change in shading or gloss due to the surface irregularities of the object to be coated, and further use the texture information, which includes the information regarding the shading or gloss of the surface of the object to be coated obtained by the calculation, to calculate the first color signal C1(λ). This makes it possible to calculate the shading and gloss (texture) corresponding to the irregularities of the object to be coated and to assign shading and gloss information to the color of the object to be coated.

[0093] In step S7, the feature identification unit 23f identifies the feature parts of the object to be coated from the first image. Specifically, the feature identification unit 23f identifies the positions and regions of elements that constitute a face, such as the eyes, nose, and mouth, from the first image.

[0094] In step S8, the second spectral reflectance calculation unit 23g performs a process to calculate the second spectral reflectance S2(λ) in the cosmetic region 31 based on the first spectral reflectance S1(λ) and the second transformation function T(λ). Specifically, the second spectral reflectance calculation unit 23g reads the second transformation function T(λ) corresponding to the color of the object to be coated in the cosmetic region 31 from the cosmetic database 24c and performs a process to calculate the second spectral reflectance S2(λ) in the cosmetic region 31.

[0095] The second transformation function T(λ) read out here may be determined, for example, in step S5 in combination with the first transformation function M, or it may be determined by user 2 selecting a color close to the object to be coated from a color sample such as a color code. The second spectral reflectance S2(λ) can be expressed by the following equation (8) using the first spectral reflectance S1(λ) and the second transformation function T(λ).

[0096]

number

[0097] By calculating the second spectral reflectance S2(λ) based on the first spectral reflectance S1(λ) and a second transformation function T(λ) corresponding to the color of the object to be coated, it is possible to calculate a second spectral reflectance S2(λ) that is closer to the spectral reflectance of the cosmetic when it is actually applied to the object and blended in.

[0098] In step S9, the second color signal calculation unit 23h performs a process to calculate the second color signal C2(λ) based on the second spectral reflectance S2(λ), the second reflection characteristic parameter, the illumination spectral distribution E(λ) of the illumination light, and the second three-dimensional reflection characteristic model expressed by the geometric conditions of the illumination light.

[0099] Specifically, the second color signal calculation unit 23h reads the second reflective characteristic parameter of the cosmetic selected by user 2 from the cosmetic database 24c. Furthermore, the second color signal calculation unit 23h performs a process to calculate the second color signal C2(λ) using the illumination spectral distribution E(λ) and spatial distribution of the illumination light source in the illumination environment estimated by the illumination environment estimation unit 23c, and the second three-dimensional reflective characteristic model.

[0100] The second three-dimensional reflection characteristic model uses the second spectral reflectance S2(λ), illumination spectral distribution E(λ), and second diffuse reflectance function D. b , and the second specular reflection function G b Using this, it can be expressed by the following equations (9) to (11). Note that the second diffuse reflectance function D b This is a function that defines diffuse reflectance in cosmetics. The second specular reflectance function G b This is a function that defines specular reflection in cosmetics. Also, B(N,V,L,m2) included in equation (11) is the Beckmann function. In equation (9), the first term on the right-hand side represents the diffuse reflection component, and the second term on the right-hand side represents the specular reflection component.

[0101]

number

[0102] In the second three-dimensional reflection characteristic model, the second diffuse reflected light intensity α2 is a parameter set for each cosmetic product and is a second reflection characteristic parameter that defines the intensity of the diffuse reflected light of the cosmetic. Here, the second diffuse reflected light intensity α2 is treated as 1. The second specular reflected light intensity I2 is a parameter set for each cosmetic product and is a second reflection characteristic parameter that defines the intensity of the specular reflected light of the cosmetic. The second specular reflection parameter m2 is a parameter set for each cosmetic product and is a second reflection characteristic parameter that defines the spread of gloss of the cosmetic.

[0103] Furthermore, if multiple cosmetic products are selected as the cosmetic products to be simulated, such as foundation and blush, in step S9, the second color signal calculation unit 23h performs the process of calculating the second color signal C2(λ) for each cosmetic product.

[0104] In this way, by calculating the second color signal C2(λ) using a second three-dimensional reflection characteristic model, the color of the cosmetic product, as well as the shading and gloss caused by the illumination light, can be reproduced as the second color signal C2(λ).

[0105] In step S9, the second color signal calculation unit 23h calculates the color signal C of the luminous agent, such as pearls and glitter contained in cosmetics such as eyeshadow, using a third three-dimensional reflection characteristic model, which is a three-dimensional reflection characteristic model of the luminous agent. G It is also possible to calculate (λ). In this case, the second color signal C2(λ) for the base portion of the cosmetic containing the luminescent agent, i.e., the portion of the cosmetic other than the luminescent agent, is calculated by the second three-dimensional reflection characteristic model, and the color signal C of the luminescent agent is calculated. G (λ) is calculated using the third three-dimensional reflection characteristic model.

[0106] As shown in Figure 10, the luminescent agent 60 contained in the cosmetic has a multilayer structure in which multiple thin films 61 are stacked. When light from a virtual light source 52 is shone on the luminescent agent 60, interference light dependent on the thickness d of each thin film 61 is observed at the virtual viewpoint 53. Therefore, the color signal C of the luminescent agent is determined for interference light from such a multilayer film. G By calculating (λ), the interference colors of the iridescent luminescent agent can be reproduced. Information about the multilayer structure of the luminescent agent 60, such as the number and thickness d of the thin films 61, is stored in the cosmetic database 24c.

[0107] The third three-dimensional reflection characteristic model is the spectral reflectance S of the luminescent agent. G The following equations (12) and (13) can be used to express the reflective properties of the luminescent agent, the illumination spectral distribution E(λ), and the geometric conditions of the illumination light. In the third three-dimensional reflective property model, the color signal C is only for the specular reflection component of the luminescent agent.G (λ) is calculated, and the diffuse reflectance component is ignored. Furthermore, the luminescent agent has a spectral reflectance S that depends on the geometric conditions of the illuminating light. G It has B(N,V,L,m) included in formula (13). G ) is the Beckmann function.

[0108]

number

[0109] Specular reflection function G of the luminescent agent G is a function that defines specular reflection in a luminescent material. Specular reflection intensity I G This is a specular reflection parameter of the luminescent agent, and it is a parameter that defines the intensity of specularly reflected light for each luminescent agent. G This parameter represents the reflective properties of the luminescent agent and defines the extent of gloss spread for each luminescent agent.

[0110] Also, the color signal C of the brightening agent G The process of calculating (λ) is performed in the region where the luminescent agent is scattered within the cosmetic region 31. Specifically, the region where the luminescent agent is scattered within the cosmetic region 31 can be estimated based on shape information that defines the size of the luminescent agent particles, and dispersion information that defines the dispersion state of the luminescent agent in the region where the cosmetic is actually applied. The region where the luminescent agent is scattered within the cosmetic region 31 is estimated, and the color signal C of the luminescent agent is calculated in that region. G By calculating (λ), the particle shape and dispersion state of the luminescent agent contained in the actual cosmetic can be reproduced. The shape information and dispersion information of the luminescent agent are stored in the cosmetic database 24c in advance.

[0111] Note that the color signal C of the brightening agent G When calculating (λ), the color signal C of the luminescent agent is used. G A correction may be made to increase the spectral intensity at a specific wavelength in (λ). Furthermore, the third three-dimensional reflection characteristic model may include parameters such as orientation parameters that define the orientation of the luminescent agent, and the thickness d of each thin film 61.

[0112] Furthermore, in step S9, the second color signal calculation unit 23h may calculate the change in shading or gloss due to the surface irregularities of the object to be coated, and further use the texture information, which includes the information regarding the shading or gloss of the surface of the object to be coated obtained by the calculation, to calculate the second color signal C2(λ). This makes it possible to calculate the shading and gloss (texture) corresponding to the irregularities of the object to be coated and to assign shading and gloss information to the cosmetic film.

[0113] In step S10, the third color signal calculation unit 23i calculates the third color signal C3(λ), which is the color signal of the second image including the object to be coated in the state of makeup, based on the first color signal C1(λ) and the second color signal C2(λ). In the region of the makeup area 31 where multiple cosmetic products are superimposed, the third color signal C3(λ) is calculated by adding up the second color signals C2(λ) of each cosmetic product.

[0114] As a specific example, the third color signal calculation unit 23i calculates the first color signal C1(λ) and the second color signal C in the region of the makeup area 31 where foundation and blush are superimposed. 2f (λ), C 2c The process of calculating the third color signal C3(λ) is performed based on the following equation (14) using (λ) and weight coefficients w1~w3. Second color signal C 2f (λ) is the color signal of foundation, which is an example of a cosmetic product. Second color signal C 2c (λ) is the color signal of blush, which is an example of a cosmetic product. The weighting coefficients w1 to w3 are coefficients used to weight each color signal, and are determined appropriately depending on the thickness of the cosmetic product in the simulation and the order in which the cosmetic products are applied in layers.

[0115]

number

[0116] In this way, by calculating the third color signal C3(λ) based on the first color signal C1(λ) and the second color signal C2(λ), it is possible to calculate the third color signal C3(λ) as the color signal of the object to be applied in the cosmetic state, in which the color of the object to be applied is reflected in the cosmetic.

[0117] Even when simulating a state where multiple cosmetic products are superimposed, a color signal can be calculated that reflects the color of the object to be applied and the color of the base cosmetic product on the outermost cosmetic product by calculating the third color signal C3(λ) based on the first color signal C1(λ) and the second color signal C2(λ) of each cosmetic product.

[0118] In step S10, the third color signal calculation unit 23i calculates the first color signal C1(λ), ​​the second color signal C2(λ), and the color signal C of the luminescent agent. G It is also possible to perform a process to calculate the third color signal C3(λ) based on (λ).

[0119] As a specific example, the third color signal calculation unit 23i performs the process of calculating the third color signal C3(λ) based on the following formula (15) in the region of the makeup area 31 where the foundation and the eye color containing pearl and glitter as luminescent agents are superimposed. Second color signal C 2e (λ) is the color signal of the eye color excluding the luminosity agent. Color signal C Gp (λ) is the color signal of pearl, an example of a luminescent agent contained in eye color. Color signal C Gl (λ) is the color signal of glitter, which is an example of a shimmering agent contained in eyeshadow.

[0120]

number

[0121] Thus, the first color signal C1(λ), ​​the second color signal C2(λ), and the color signal C of the luminescence agent G By calculating the color signal of the cosmetic state based on (λ), a third color signal C3(λ) can be calculated that reflects the color and gloss of the luminescent agents contained in the cosmetic.

[0122] In step S11, the second RGB value calculation unit 23j calculates the second RGB value, which is the RGB value of the second image, based on the third color signal C3(λ) calculated by the third color signal calculation unit 23i. Specifically, the second RGB value calculation unit 23j calculates the second RGB value based on the following equation (16) using the third color signal C3(λ) and the RGB color matching function. In equation (16), R2, G2, and B2 on the left side are the values ​​of each element in the second RGB value.

[0123]

number

[0124] In step S12, the second image generation unit 23k performs the process of generating a second image based on the second RGB values ​​calculated by the second RGB value calculation unit 23j.

[0125] In step S13, the superimposed image generation unit 23l generates a superimposed image by superimposing the first image and the second image. Furthermore, the superimposed image generation unit 23l transmits the generated superimposed image data to the user terminal 10 and performs the process of displaying the superimposed image, which is the simulation result, on the touch panel 13.

[0126] As shown in Figure 11, the touch panel 13 displays the superimposed image 30b in the image display area 41 of the application image 40. The makeup area 31 of the superimposed image 30b displays the skin color, the color of the base makeup product, and the skin's makeup state, reflecting the shadows caused by the lighting, which are examples of the object to be applied. In addition, the light receiving angle θ within the makeup area 31 r The angle of incidence is θ i In the glossy region 32, which is an approximate region, the gloss due to specular reflection is reproduced and displayed. With the above processing, the cosmetic simulation is completed.

[0127] Furthermore, while the superimposed image 30b is displayed, selecting a different cosmetic product allows the process in steps S5 to S11 to be executed again, generating and displaying a superimposed image 30b with the different cosmetic product applied. In this case, for example, the process in steps S5 and S6 can be omitted by storing the first color signal C1(λ) that has been calculated once in the main memory of memory 24. In this way, by repeatedly simulating various cosmetic products, it is possible to find a cosmetic product that suits one's preferences.

[0128] The first color signal C1(λ) and the second color signal C2(λ) are calculated based on the illumination spectral distribution E(λ) of the illumination environment and the spatial distribution of the illumination light source estimated by the illumination environment estimation unit 23c. This allows the illumination environment of the location where the first image was taken to be reflected in the superimposed image 30b as a simulation result. Therefore, a makeup simulation reflecting the illumination effect of the illumination environment in the first image can be performed. Furthermore, by using the illumination spectral distribution E(λ) calculated from measured values ​​of the spectral distribution in the actual illumination environment, various illumination environments can be easily reproduced. This makes it easy to perform, for example, a makeup simulation specifying an arbitrary illumination environment. It also makes it easy to compare the results of makeup simulations with different illumination environments.

[0129] By having the GPU 23 on the server 20 execute steps S2 to S13, the processing of steps S2 to S13 can be performed at high speed, enabling real-time makeup simulation. Therefore, for example, makeup simulation in augmented reality (AR) or mixed reality (MR) becomes possible.

[0130] Furthermore, for example, by using the cosmetic simulation system 1 to simulate various cosmetic products for user 2, it is possible to provide counseling to suggest the most suitable cosmetic product to meet user 2's needs.

[0131] The processing order of each step shown in Figure 7 may be changed as appropriate, as long as it does not hinder the processing. For example, steps S4 and S5 may be executed in this order, or their order may be reversed. That is, as described above, step S5 may be executed after step S4, or step S4 may be executed after step S5. Also, for example, step S3 may be executed after step S6 if the surface irregularities of the object to be coated are not considered in the first color signal calculation unit 23e.

[0132] [Effects of the First Embodiment] According to the first embodiment described above, the following effects can be obtained. (1-1) By calculating the first color signal C1(λ) using the first three-dimensional reflection characteristic model and the second color signal C2(λ) using the second three-dimensional reflection characteristic model, a third color signal C3(λ) can be calculated as the color signal of the object to be applied in the makeup state, in which the color of the object to be applied is reflected in the cosmetic. Then, by calculating the second RGB value from the third color signal C3(λ), a second image including the object to be applied in the makeup state can be generated. This makes it possible to obtain a second image of the makeup state in which the color of the object to be applied is reflected in the cosmetic. Therefore, it is possible to perform a makeup simulation with higher reproduction accuracy than simply superimposing the color of the cosmetic onto the area of ​​the object to be applied. In this specification, "reproduction accuracy" means that the three-dimensional texture of the cosmetic, such as gloss, color, and shading, which changes moment by moment in response to the position, orientation, or lighting environment of the object to be applied, can be reproduced in real time. Furthermore, since gloss due to illumination light can be reproduced, the texture of the object to be applied can be reproduced in the second image.

[0133] (1-2) By calculating the first spectral reflectance S1(λ) based on the first transformation function M corresponding to the color of the skin or lips as the object to be coated, it is possible to calculate the first spectral reflectance S1(λ) that is closer to the actual spectral reflectance of the skin or lips.

[0134] (1-3) By calculating the second spectral reflectance S2(λ) based on the first spectral reflectance S1(λ) and the second transformation function T(λ) corresponding to the color of the object to be coated, it is possible to calculate a second spectral reflectance S2(λ) that is closer to the spectral reflectance of the cosmetic when it is actually applied to the object and blends in with the object.

[0135] (1-4) By calculating the second color signal C2(λ) using the second spectral reflectance S2(λ) and second reflection characteristic parameter for each cosmetic product, a second image can be generated that reflects the color and reflection characteristics of each cosmetic product.

[0136] (1-5) By using the third three-dimensional reflection characteristic model, the color signal C of the luminescent agent contained in the cosmetic is G (λ) can be calculated. Then, the first color signal C1(λ), ​​the second color signal C2(λ), and the color signal C of the luminescence agent can be calculated. G By calculating the second RGB value from the third color signal C3(λ) calculated based on (λ), a second image can be generated that reproduces the gloss of the luminous agent in the cosmetic state. Furthermore, in the region where the luminous agent is scattered, estimated from the shape information and dispersion information, the color signal C of the luminous agent G By calculating (λ), it is possible to reproduce the particle shape and dispersion state of the luminescent agent contained in actual cosmetics.

[0137] (1-6) By calculating the first color signal C1(λ) and the second color signal C2(λ) based on the illumination spectral distribution E(λ) and the spatial distribution of the illumination light source of the illumination environment estimated as the illumination environment of the first image, the illumination environment is also reflected in the second image. Therefore, it is possible to perform a highly accurate makeup simulation that reflects the illumination environment in the first image.

[0138] (1-7) By using the GPU23 to perform the processing in steps S2 to S13 of the makeup simulation system 1, real-time image processing becomes possible, allowing for more suitable makeup simulation. For example, makeup simulation can be performed using augmented reality (AR) or mixed reality (MR).

[0139] (1-8) As parameters indicating the characteristics of cosmetics, the results measured by various measuring instruments such as spectrophotometers, gloss meters, and colorimeters can be stored in the cosmetic database 24c. This allows the measurement results from the measuring instruments to be directly used in cosmetic simulations.

[0140] (1-9) By performing cosmetic simulations using mathematical models (e.g., three-dimensional reflection property models), the reflection properties of cosmetic products can be reproduced based on physical simulations. This makes it possible to decompose the cosmetic product into individual mathematical models for each material component and describe each as an independent block structure.

[0141] The advantages of adopting the above structure are as follows: (Advantage 1) Because it can reproduce images that correspond to three-dimensional changes such as gloss, shadows, lighting, and color changes depending on the viewing angle, it is possible to reproduce the texture of real cosmetics and changes in texture that correspond to changes in observation conditions in a way that is true to the real thing. (Advantage 2) In cosmetic development, describing individual components using a block structure of mathematical models makes it possible to simulate the removal or addition of specific ingredients. In other words, it becomes possible to perform computer simulations (visual simulations) that respond to changes in the component composition of cosmetics without having to prepare actual cosmetic products. (Advantage 3) The functionality of the cosmetic simulation system 1 can be enhanced by increasing the accuracy of the mathematical model. On the other hand, enhancing the system's functionality may lead to a decrease in processing speed. Therefore, taking this into consideration, the cosmetic simulation system 1 can be sped up by lowering the accuracy of the mathematical model. (Advantage 4) For cosmetics with complex compositions, where modeling the entire product is difficult, lower-accuracy models, such as partially simplified models or models with assumptions, can be described independently as blocks. In this case, as measurement accuracy improves, the accuracy of the entire model can be gradually improved by replacing the lower-accuracy models (blocks) with higher-accuracy models.

[0142] The makeup simulation system 1 can be described as a highly scalable system, as explained by the four advantages above.

[0143] (1-10) By using the makeup simulation system 1 to simulate various cosmetic products for user 2, it is also possible to provide counseling to suggest the most suitable cosmetic product to meet user 2's needs.

[0144] (1-11) The cosmetic simulation system 1 allows for parameter adjustment to meet user requirements, and by using this function to simulate various cosmetic products, it is also possible to provide counseling to suggest the optimal cosmetic product to meet user 2's needs.

[0145] 2. Modified form of the first embodiment Modifications of the first embodiment described above will be described below. The modifications will be described focusing on the differences from the first embodiment. Configurations or elements not specifically described below are the same as those of the first embodiment.

[0146] 2-1. First variation The first modified example of the first embodiment differs from the first embodiment in that the cosmetic database 24c stores information regarding the physical properties of the cosmetic, and the second color signal calculation unit 23h calculates the second color signal C2(λ) using a second reflection characteristic parameter corresponding to the information regarding the physical properties of the cosmetic.

[0147] Specifically, the cosmetic database 24c stores information relating the second reflectivity parameter to the physical properties of the cosmetic. The cosmetic database 24c stores the physical properties of the cosmetic for each cosmetic product. Examples of the physical properties of a cosmetic include the formulation of the cosmetic and the properties of the raw materials contained in the cosmetic. More specifically, these include the spectral reflectance, type, and mixing ratio of the raw materials, as well as the shape and size of the luminescent agent. In addition, for complex materials composed of multiple materials or unknown materials, a mathematical model approximating the physical properties of the material in question and model parameters (glossiness, roughness, wavelength change of reflected light, etc.) can also be recorded in the cosmetic database 24c.

[0148] The second color signal calculation unit 23h reads a second reflective property parameter corresponding to information about the physical properties of the cosmetic from the cosmetic database 24c, and calculates the second color signal C2(λ) of the cosmetic using the read second reflective property parameter and the second three-dimensional reflective property model. The information about the physical properties of the cosmetic used by the second color signal calculation unit 23h to read the second reflective property parameter from the cosmetic database 24c may be, for example, information set by user 2. Specifically, the information about the physical properties of the cosmetic may be information input by user 2 via user terminal 10 and transmitted from user terminal 10 to server 20.

[0149] In the cosmetic simulation system 1 according to the first modified example, the second reflection characteristic parameters constituting the second three-dimensional reflection characteristic model can be set according to the physical characteristics of the cosmetic. That is, the physical characteristics of the cosmetic can be reflected in the model parameters of the second three-dimensional reflection characteristic model, which is a mathematical model. This makes it possible to calculate the second color signal C2(λ) based on the physical characteristics of the cosmetic.

[0150] By using the cosmetic simulation system 1 according to the first modified example, user 2 can perform a cosmetic simulation by setting the physical properties of the cosmetic. For example, consider a case where user 2 performs a cosmetic simulation of a cosmetic containing a luminescent agent. As one example, user 2 can set a numerical value for the size of the luminescent agent as information regarding the physical properties of the cosmetic. This allows for a simulation of, for example, how much the gloss appears to increase when the size of the luminescent agent is increased. As another example, user 2 can set a numerical value for the amount of oily components contained in the cosmetic as information regarding the physical properties of the cosmetic. This allows for a simulation of, for example, how much the gloss of the luminescent agent appears to decrease when the amount of oily components contained in the cosmetic is increased.

[0151] Furthermore, the cosmetic simulation system 1 relating to the first modification allows for parameter adjustment to suit the user's needs. By using this function to simulate various cosmetic products, it is also possible to provide counseling to suggest the optimal cosmetic product to suit the user's needs.

[0152] Furthermore, the cosmetic simulation system 1 according to the first modified example can be used, for example, by a cosmetic developer when designing a cosmetic. That is, the cosmetic simulation system 1 may be a system used for cosmetic design. The cosmetic developer can simulate how the cosmetic state of an object to which it is applied changes when the physical properties of the cosmetic are changed. The developer can then consider the types of raw materials and formulation composition contained in the cosmetic by referring to the simulation results. The cosmetic simulation system 1 can contribute to reducing the effort and time required to actually prototype multiple cosmetic products with different physical properties in the design of a cosmetic.

[0153] 2-2. Second variation The second modification of the first embodiment differs from the first embodiment in that the second RGB value calculation unit 23j corrects the third color signal C3(λ) based on the characteristics of human vision and the characteristics of the color image output device that outputs the superimposed image. Specifically, the second RGB value calculation unit 23j corrects the third color signal C3(λ) based on the characteristics of human vision and the characteristics of the color image output device, and calculates the second RGB value of the object to be coated in a cosmetic state from the corrected third color signal C3(λ). This makes it possible to correct the second RGB value to a value with little color difference from the actual object.

[0154] Specifically, the second RGB value calculation unit 23j converts the third color signal C3(λ) into a color to be displayed in the CIE L*a*b* space. Then, it converts the color from CIE L*a*b* to device RGB or device CMYK based on the color characteristics of the color image output device (monitor, projector, printer, etc.). The color image output device to be displayed has previously measured the color coordinates of each color, including the gamma characteristics and the white point of the RGB or CMYK values, and performs color conversion based on that information. In this case, each device is not limited to an RGB 3-color display, but may also be a multi-primary color display (for example, a display with more primary colors than RGB 3, such as Sharp's Quattron RGBY 4-color monitor) or a multi-color printer (a printer with more than 4 primary colors of ink (for example, a 6-color printer with more than 4 primary colors of ink, such as CMYK + light cyan + light magenta)).

[0155] 2-3. Third Variation The third modification of the first embodiment differs from the first embodiment in that the cosmetic database 24c further stores methods for applying cosmetics according to an impression or situation. As described above, the cosmetic database 24c stores a second transformation function T(λ) for each type of cosmetic and for each color of each cosmetic. The difference here is that, in addition to this second transformation function T(λ), it is possible to record the spatial distribution for application of each cosmetic (spatial shades, contours, etc.) and incorporate methods for applying cosmetics according to an impression or situation into the database.

[0156] The above impressions include, for example, cheerful, lively, quiet, and gentle. The above situations include, for example, working, going to an interview, going out with friends, and attending a wedding. The above methods of applying cosmetics include, for example, the application location, application area, and amount of cosmetics.

[0157] In the third modified example, the operation area 42 of the user terminal 10 shown in Figure 5 may display, for example, an impression selection object and a scene selection object (neither of which are shown). By tapping the impression selection object, user 2 may be able to select the impression they desire. Similarly, by tapping the scene selection object, user 2 may be able to select the scene they envision.

[0158] By using the third modified makeup simulation system 1, user 2 can simulate a makeup look suitable for a desired impression or situation.

[0159] Even with the same cosmetic product, changing the application method can alter the perceived appearance. However, it is difficult for someone unfamiliar with cosmetic application techniques to find the appropriate application method for the desired impression or situation. Therefore, information on cosmetic application methods tailored to different impressions or situations is pre-stored in the cosmetic database 24c, for example, based on the knowledge of cosmetic developers who are experts in cosmetic application techniques. This allows anyone to easily perform makeup simulations tailored to different impressions or situations.

[0160] 3. Second Embodiment In the first embodiment described above, an example was given where the first image is an RGB image. When the first image is a multiband image, the makeup simulation is performed by the same process as in the first embodiment. Next, as a second embodiment, an example is given where the first image is a multispectral image. Configurations or elements that are not specifically described below are the same as in the first embodiment.

[0161] In the second embodiment, the imaging unit 14 of the user terminal 10 is, for example, a multispectral camera. Here, the imaging unit 14 described in the first embodiment is an RGB camera having three wavelength bands, R, G, and B, but the RGB camera in the first embodiment may be a multiband camera that divides the visible wavelength band into about 4 to 20 bands. On the other hand, in the second embodiment, a multispectral camera (also called a hyperspectral camera) that finely divides the visible wavelength range to obtain detailed spectral waveforms is used as the imaging unit 14. The multiband camera outputs a multiband image. On the other hand, the multispectral camera outputs multispectral image data that records the spectrum. The multispectral camera may be, for example, a camera built into the user terminal 10, or a camera attached to or connected to the user terminal 10. The imaging unit 14 acquires a multispectral image, which is a face image of user 2 in an unmakeup state (a first image including the skin and lips of the face in an unmakeup state).

[0162] In the second embodiment, in step S4 shown in Figure 7, the illumination environment estimation unit 23c estimates the illumination environment in the first image or the illumination environment estimation image based on the spectral distribution of illumination light obtained from the first image or the illumination environment estimation image.

[0163] The illumination source estimation process includes obtaining the spectral distribution E(λ) of the illumination source. E(λ) requires a process to convert the camera output obtained from the first image into a spectral distribution. For example, the complex spectral distributions of LED and fluorescent lighting cannot be mathematically determined directly from RGB images; therefore, a process to determine the spectral distribution from the camera output (RGB image) is necessary. To achieve this, the spectral distribution characteristics of numerous light sources are measured in advance, and the information obtained from statistically analyzing this measurement data is used as a constraint to estimate the spectral distribution of the light source from the first image and the image used for estimating the lighting environment. In this case, as shown in equation (A) below, a system conversion matrix M for the illumination source is obtained from the RGB values. E The spectral distribution E(λ) of the light source is determined using this method. System conversion matrix M for illumination light source. EAs a method for determining this, known techniques can be used, for example (Tanaka, Mochizuki, "Omnidirectional Spectroscopic Image Measurement using RGB Camera and its Application to IBL," Journal of the Institute of Image Electronics Engineers of Japan, 2013). Note that the system transformation matrix M E This is a specific example of a system transformation matrix used to determine the spectral distribution of illumination light stored in the transformation matrix storage unit 14d, as described in the first embodiment.

[0164]

number

[0165] However, in the multispectral camera used in the second embodiment, the spectral distribution of the target can be directly captured, so the process of estimating the spectral distribution from the camera output described above can be omitted. In other words, if the first image or the image for estimating the lighting environment used in estimating the lighting environment is a multispectral image, the lighting environment estimation unit 23c can estimate the lighting environment without using the lighting environment database 24b and the system conversion matrix for determining the spectral distribution of the illumination light stored in the conversion matrix storage unit 24d.

[0166] In step S5 shown in Figure 7, the first spectral reflectance calculation unit 23d calculates RGB values ​​from the spectral information of the first image and sets the obtained RGB values ​​as the first RGB values ​​of the first image. Based on these first RGB values, the first spectral reflectance calculation unit 23d performs a process to calculate the first spectral reflectance S1(λ) in the cosmetic region 31 using the above equations (1) and (2).

[0167] Since a multiband camera cannot directly capture the spectral distribution of the target, if the first image or the image used for estimating the lighting environment is a multiband image, the same processing as described for the RGB image in the first embodiment is basically performed. In other words, the processing is performed according to the number of bands in the multiband image, rather than for the three bands R, G, and B.

[0168] According to the second embodiment described above, the following effects can be obtained. (2-1) Because the first image is a multispectral image, spectral information of the lighting environment can be directly obtained from the first image. Therefore, the lighting environment can be estimated with greater accuracy compared to when spectral information of the lighting environment is indirectly obtained from the first image, which is an RGB image.

[0169] 4. Third Embodiment In the first embodiment described above, an example was described in which a third color signal C3(λ) is calculated based on the first color signal C1(λ) and the second color signal C2(λ) calculated by spectral reflectance-based calculation processing, and then a second RGB value is calculated based on the third color signal C3(λ). Instead of the above processing, the second RGB value can also be obtained by performing RGB value-based calculation processing. As a third embodiment, the operation when performing RGB value-based calculation processing is described below.

[0170] Furthermore, the RGB value-based calculation process for determining the second RGB value is performed in the area of ​​the cosmetic region 31 where high reproduction accuracy is not required. That is, in the area where high reproduction accuracy is required, the second RGB value is calculated by the spectral reflectance-based calculation process described in the first embodiment. In the third embodiment, the RGB value-based calculation process for determining the second RGB value is performed by the second RGB value calculation unit 23j as a process from step S4 onwards, as shown in Figure 7.

[0171] First, the second RGB value calculation unit 23j calculates the first RGB value and the first diffuse reflectance function D. a , First specular reflection function G a Furthermore, a process is executed to calculate the third RGB value based on the RGB values ​​of the illumination light. The third RGB value is the RGB value of the object to be coated when illumination light is shone on the object, reflecting the illumination effects such as shadows and gloss. The third RGB values ​​R3, G3, B3 can be expressed by the following formula (17). Also, the RGB values ​​of the illumination light R L ,G L ,B LThis can be expressed by the following equation (18) using the illumination spectral distribution E(λ) and RGB color matching functions. The RGB values ​​of the illumination light are stored in the illumination environment database 24b for each illumination environment.

[0172]

number

[0173] Next, the second RGB value calculation unit 23j performs the process of calculating the fourth RGB value based on the first RGB value and the cosmetic conversion coefficient. The fourth RGB value is the RGB value of the cosmetic when it has been applied to the object to be applied and has blended into the object. In other words, the fourth RGB value does not reflect the lighting effects such as shadows and gloss caused by the illumination light in the lighting environment estimated in step S4. The cosmetic conversion coefficient is a coefficient calculated based on the second conversion function T(λ), and is a coefficient for converting the first RGB value to the fourth RGB value. The cosmetic conversion coefficient is stored in advance in the cosmetic database 24c for each cosmetic. The fourth RGB values ​​R4, G4, B4 can be expressed by the following equation (19). Also, the cosmetic conversion coefficient R T ,G T ,B T This can be expressed by the following equation (20) using the second transformation function T(λ) and the RGB color matching functions.

[0174]

number

[0175] Next, the second RGB value calculation unit 23j calculates the fourth RGB value and the second diffuse reflectance function D. b , second specular reflection function G b Furthermore, a process is performed to calculate the fifth RGB value based on the RGB values ​​of the illumination light. The fifth RGB value is the RGB value of the cosmetic product that reflects the illumination effects such as shading and gloss when the cosmetic product is irradiated with illumination light. The fifth RGB values ​​R5, G5, and B5 can be expressed by the following formula (21).

[0176]

number

[0177] Then, the second RGB value calculation unit 23j performs the process of calculating the second RGB value based on the third RGB value, the fifth RGB value, and the weight coefficient w. As a specific example, in the area of ​​the makeup area 31 where foundation and blush are superimposed, the second RGB value calculation unit 23j performs the process of calculating the second RGB value based on the following equation (22). R 5f ,G 5f ,B 5f This is the 5th RGB value of the foundation. 5c ,G 5c ,B 5c This is the 5th RGB value for cheek color.

[0178]

number

[0179] Furthermore, it is difficult to calculate the luminescence agents contained in cosmetics based on RGB values. Therefore, the 6th RGB value that reproduces the interference color of the luminescence agent is calculated using the above formula (12) based on spectral reflectance to obtain the luminescence agent's color signal C G After calculating (λ), the color signal C of the luminescent agent is calculated. G It is calculated from (λ) and the RGB color matching function. The process of calculating the sixth RGB value is performed by the second RGB value calculation unit 23j. The sixth RGB values ​​R6, G6, B6 can be expressed by the following equation (23).

[0180]

number

[0181] Then, the second RGB value calculation unit 23j can also execute a process of calculating the second RGB value in a state where the cosmetic containing the brightening agent is applied, based on the third RGB value, the fifth RGB value, the sixth RGB value, and the weight coefficient w. As a specific example, in the area of the makeup area 31 where the foundation and the eye color containing pearl and lame as brightening agents are superimposed, the second RGB value calculation unit 23j executes a process of calculating the second RGB values R2, G2, B2 based on the following formula (24). R 5e ,G 5e ,B 5e is the fifth RGB value of the part of the eye color excluding the brightening agent. R 6p ,G 6p ,B 6p is the sixth RGB value of pearl, which is an example of the brightening agent contained in the eye color. R 6l ,G 6l ,B 6l is the sixth RGB value of lame, which is an example of the brightening agent contained in the eye color.

[0182]

Equation

[0183] The second RGB value calculated by the RGB value-based calculation process as described above is used for the second image generation unit 23k to execute a process of generating a second image in step S12 shown in FIG. 7.

[0184] When calculating the second RGB value by the RGB value-based calculation process, a three-dimensional calculation process for the three colors of red, green, and blue is performed. On the other hand, when calculating the second RGB value based on the color signal calculated from the spectral reflectance as in the first embodiment, a 61-dimensional calculation process that divides the visible light region of 400 nm to 700 nm every 5 nm is performed.

[0185] In other words, by calculating the second RGB value using RGB value-based calculation processing, the calculation processing in cosmetic simulation can be simplified compared to the process of calculating the second RGB value based on the color signal calculated from spectral reflectance. Therefore, by performing RGB value-based calculation processing in areas of the cosmetic domain 31 where high reproduction accuracy is not required, the calculation processing can be simplified without significantly reducing the reproduction accuracy of the cosmetic simulation.

[0186] [Effects of the third embodiment] According to the third embodiment described above, the following effects can be obtained. (3-1) By calculating the second RGB value using RGB value-based calculation processing, the calculation processing in cosmetic simulation can be simplified compared to the process of calculating the second RGB value based on the color signal calculated from spectral reflectance. Therefore, by performing RGB value-based calculation processing in areas of the cosmetic domain 31 where high reproduction accuracy is not required, the calculation processing can be simplified without significantly reducing the reproduction accuracy of the cosmetic simulation.

[0187] Furthermore, the third embodiment described above can also be implemented with appropriate modifications as follows. A configuration is provided in which the second RGB value is calculated based on the third and fifth RGB values ​​calculated using RGB value-based calculation processing. However, the system is not limited to this, and either the third or fifth RGB value may be obtained by spectral-based calculation. Specifically, instead of the third RGB value, the value obtained by converting the first color signal C1(λ) to an RGB value using an RGB color matching function may be used. Also, instead of the fifth RGB value, the value obtained by converting the second color signal C2(λ) to an RGB value using an RGB color matching function may be used. With such a configuration, only arbitrary calculation processes that do not require high precision can be calculated on an RGB value basis, thus more preferably simplifying the calculation process for cosmetic simulation.

[0188] Furthermore, the first to third embodiments described above can be further modified as appropriate and implemented as follows. In the first to third embodiments described above, the subject of the makeup simulation was the face, and the object to which the cosmetic product is applied was the skin or lips on the face. However, the system is not limited to this, and the makeup simulation system 1 can be applied to any area to which a cosmetic product is applied, even if it is not the face. The object to which the cosmetic product is applied may be, for example, hair. In that case, the cosmetic product applied may be, for example, hair manicure. Alternatively, the object to which the cosmetic product is applied may be, for example, nails. In that case, the cosmetic product applied may be, for example, nail polish.

[0189] While the example illustrates a configuration in which steps S2 to S13 in the cosmetic simulation system 1 are executed using the GPU 23, the system is not limited to this configuration. For example, the server 20 may not have a GPU 23, and the CPU 22 may execute steps S2 to S13. In this case, although the processing speed of steps S2 to S13 on the server 20 will decrease, it is possible to omit the GPU 23. Also, for example, depending on the performance of the CPU 22, the CPU 22 may execute any of the steps S2 to S13.

[0190] A configuration is illustrated in which the first color signal C1(λ) and the second color signal C2(λ) are calculated based on the illumination spectral distribution E(λ) of the illumination environment and the spatial distribution of the illumination light source estimated by the illumination environment estimation unit 23c. However, the system is not limited to this, and for example, a configuration in which the estimated illumination environment can be changed by user 2 may also be used. In this case, for example, it is possible to simulate how the appearance of an object to which a cosmetic product has been applied changes depending on the difference in the illumination environment.

[0191] In step S8, an example was given of a configuration in which the second spectral reflectance S2(λ) is calculated based on the first spectral reflectance S1(λ) and the second transformation function T(λ) corresponding to the color of the object to be coated. However, the system is not limited to this, and the spectral reflectance of the cosmetic can be measured using a measuring instrument such as a spectrophotometer, and the measured value of the spectral reflectance of the cosmetic can be used as the second spectral reflectance S2(λ). In this case, although the reproduction accuracy of the second color signal C2 is slightly lower than when the second spectral reflectance S2(λ) is calculated based on the first spectral reflectance S1(λ) and the second transformation function T(λ), cosmetic simulations can be easily performed for various cosmetic products. For example, when a new cosmetic product is developed, the cosmetic simulation in this cosmetic simulation system 1 can be performed by measuring the second spectral reflectance and second reflection characteristic parameters of the new cosmetic product. In this case, the processing in step S8 is omitted.

[0192] A configuration is shown in which the first spectral reflectance S1(λ) is calculated based on a first transformation function M corresponding to the color of the object to be coated. However, the system is not limited to this, and for example, the first spectral reflectance S1(λ) may be calculated based on a general-purpose first transformation function M set for each object to be coated, such as skin or lips, regardless of the color of the object to be coated. In this case, although the difference between the superimposed image 30b as a simulation result and the actual appearance after makeup application will be slightly larger, the process of determining the first transformation function M according to the first RGB values ​​that constitute the first image 30a, or the procedure of user 2 selecting a color close to the object to be coated from a color sample, becomes unnecessary. Therefore, the process of calculating the first spectral reflectance S1(λ) from the first RGB values ​​can be simplified.

[0193] The example given shows a configuration in which a facial image of user 2 is acquired using the imaging unit 14 of the user terminal 10, but the system is not limited to this. For example, a facial image acquired by an external device such as a digital camera may be used as the first image. Also, the image for estimating the lighting environment may be, for example, an image acquired using the imaging unit 14 of the user terminal 10, or an image acquired by an external device other than the user terminal.

[0194] · In this embodiment, a configuration in which the GPU 23 included in the server 20 is used to execute the processes of steps S2 to S13 in the makeup simulation system 1 has been exemplified. However, the present invention is not limited to this. For example, a GPU may be provided in the control unit 11 of the user terminal 10, a makeup simulation program may be installed in the user terminal 10, and the processes of steps S2 to S13 may be executed by the GPU included in the user terminal 10.

[0195] · The makeup simulation system 1 may be configured to be capable of executing a process of purchasing makeup applied to the superimposed image as a simulation result, for example, using an image display program. In this case, a settlement unit for executing settlement processing is provided in the server 20. Thereby, for example, after searching for makeup that meets one's own desires in the makeup simulation system 1, the makeup can be purchased.

[0196] Note that the present invention is not limited to the above-described embodiments and modifications, and various modifications are possible. For example, the above-described embodiments and modifications may be appropriately combined.

Description of Reference Numerals

[0197] 1... Makeup simulation system 10... User terminal 11... Control unit 12... Memory 13... Touch panel 14... Imaging unit 20... Server 21... Control unit 23... GPU 23a... Shape estimation unit 23b... Skin quality identification unit 23c... Illumination environment estimation unit 23d... First spectral reflectance calculation unit 23e... First color signal calculation unit 23f... Feature part identification unit 23g... Second spectral reflectance calculation unit 23h... Second color signal calculation unit 23i... Third color signal calculation unit 23j...Second RGB value calculation unit 23k...Second image generation section 23l... Superimposed image generation unit 24...memory 24a...Spectroscopic Conversion Database 24b…Lighting Environment Database 24c... Cosmetics Database 24d...Conversion Matrix Memory Unit 30…Image 30a…First image 30b... Superimposed image 31…Makeup area 32…Glossy area 40… App image

Claims

1. A lighting environment estimation unit estimates the lighting environment in the first image or the lighting environment estimation image, using a first image which is an image of the object to be coated with the cosmetic before the cosmetic is applied, A first spectral reflectance calculation unit calculates a first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A first color signal calculation unit calculates a first color signal for the object to be coated before cosmetic application using the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions. A second color signal calculation unit calculates a second color signal of the cosmetic using a second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, and a second three-dimensional reflection characteristic model expressed by the spectral distribution of the illumination light and the geometric conditions. A third color signal calculation unit calculates a third color signal for the object to be coated in a cosmetic state after the cosmetic has been applied to the object, based on the first color signal and the second color signal. A second RGB value calculation unit calculates a second RGB value for the object to be coated in the cosmetic state from the third color signal, A second image generation unit generates a second image based on the second RGB value, A superimposed image generation unit generates a superimposed image by superimposing the first image and the second image, A makeup simulation system equipped with the following features.

2. The first image or the image for estimating the lighting environment is an RGB color image, a multiband image, or a multispectral image. The makeup simulation system according to claim 1.

3. The first image and the image used for estimating the lighting environment are the same image. The makeup simulation system according to claim 1 or 2.

4. The system further includes an illumination environment database that stores light source information for each illumination environment, which is a combination of the illumination spectral distribution, which is the spectral distribution of illumination light that can be irradiated onto the object to be coated, and the spatial distribution of the illumination light source that generates the illumination light. The makeup simulation system according to claim 1 or 2.

5. In the case where the first image or the image used for estimating the lighting environment is an RGB image or a multiband image, It further includes a conversion matrix storage unit that stores a system conversion matrix for determining the spectral distribution of illumination light from RGB values, The lighting environment estimation unit estimates the lighting environment in the first image or the lighting environment estimation image by estimating the spatial distribution of the illumination light source based on the illumination direction vector estimated from the first image or the lighting environment estimation image and the spectral distribution of the illumination light obtained from the RGB values ​​of the first image or the lighting environment estimation image using a system transformation matrix for determining the spectral distribution of the illumination light. The makeup simulation system according to claim 4.

6. In the case where the first image or the image used for estimating the lighting environment is a multispectral image, The lighting environment estimation unit estimates the lighting environment in the first image or the lighting environment estimation image based on the spectral distribution of the illumination light obtained from the first image or the lighting environment estimation image. The makeup simulation system according to claim 1 or 2.

7. The system further includes a cosmetic database that stores information relating the second reflection characteristic parameter to the physical properties of the cosmetic, The second color signal calculation unit reads the second reflection characteristic parameter corresponding to information regarding the physical properties of the cosmetic from the cosmetic database, and calculates the second color signal of the cosmetic using the read second reflection characteristic parameter, the spectral distribution of the illumination light, and the geometric conditions, which are expressed as a second three-dimensional reflection characteristic model. The makeup simulation system according to claim 1 or 2.

8. The second RGB value calculation unit corrects the third color signal based on the characteristics of human vision and the characteristics of the color image output device that outputs the superimposed image, and calculates the second RGB value of the object to be coated in the cosmetic state from the corrected third color signal. The makeup simulation system according to claim 1 or 2.

9. A shape estimation unit that estimates the three-dimensional shape of the object to be coated from a first image, which is an image of the object to be coated before the cosmetic is applied, A lighting environment estimation unit that estimates the lighting environment in the first image or the lighting environment estimation image using the first image or the lighting environment estimation image, A first spectral reflectance calculation unit calculates a first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A first color signal calculation unit calculates a first color signal for the object to be coated before cosmetic application using the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions. A second color signal calculation unit calculates a second color signal of the cosmetic using a second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, and a second three-dimensional reflection characteristic model expressed by the spectral distribution of the illumination light and the geometric conditions. A third color signal calculation unit calculates a third color signal for the object to be coated in a cosmetic state after the cosmetic has been applied to the object, based on the first color signal and the second color signal. A second RGB value calculation unit calculates a second RGB value for the object to be coated in the cosmetic state from the third color signal, A second image generation unit generates a second image based on the second RGB value, A superimposed image generation unit generates a superimposed image by superimposing the first image and the second image, A makeup simulation system equipped with the following features.

10. A lighting environment estimation unit estimates the lighting environment in the first image or the lighting environment estimation image, using a first image which is an image of the object to be coated with the cosmetic before the cosmetic is applied, A first spectral reflectance calculation unit calculates a first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A first color signal calculation unit calculates a first color signal for the object to be coated before cosmetic application using the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions. A feature area identification unit that identifies the feature area of ​​the object to be coated from the first image, A second color signal calculation unit calculates a second color signal of the cosmetic using a second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, and a second three-dimensional reflection characteristic model expressed by the spectral distribution of the illumination light and the geometric conditions. A third color signal calculation unit calculates a third color signal for the object to be coated in a cosmetic state after the cosmetic has been applied to the object, based on the first color signal and the second color signal. A second RGB value calculation unit calculates a second RGB value for the object to be coated in the cosmetic state from the third color signal, A second image generation unit generates a second image based on the second RGB value, A superimposed image generation unit generates a superimposed image by superimposing the first image and the second image, A makeup simulation system equipped with the following features.

11. A skin texture identification unit identifies the surface irregularities of an object to be coated as a mathematical model from a first image, which is an image of the object to be coated before the cosmetic is applied. A lighting environment estimation unit that estimates the lighting environment in the first image or the lighting environment estimation image using the first image or the lighting environment estimation image, A first spectral reflectance calculation unit calculates a first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A first color signal calculation unit calculates a first color signal for the object to be coated before cosmetic application using the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions. A second color signal calculation unit calculates a second color signal of the cosmetic using a second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, and a second three-dimensional reflection characteristic model expressed by the spectral distribution of the illumination light and the geometric conditions. A third color signal calculation unit calculates a third color signal for the object to be coated in a cosmetic state after the cosmetic has been applied to the object, based on the first color signal and the second color signal. A second RGB value calculation unit calculates a second RGB value for the object to be coated in the cosmetic state from the third color signal, A second image generation unit generates a second image based on the second RGB value, A superimposed image generation unit generates a superimposed image by superimposing the first image and the second image, A makeup simulation system equipped with the following features.

12. The first color signal calculation unit or the second color signal calculation unit The first color signal or the second color signal is calculated using the texture information obtained from the calculation, which includes information regarding the shading or gloss due to the surface irregularities of the object to be coated, and the first color signal or the second color signal is calculated using the texture information obtained from the calculation, which includes information regarding the shading or gloss due to the surface irregularities of the object to be coated. The makeup simulation system according to claim 11.

13. A step of estimating the lighting environment in the first image or the lighting environment estimation image, using a first image which is an image of the object to be coated with the cosmetic before the cosmetic is applied, A step of calculating the first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A step of calculating a first color signal in the object to be coated before cosmetic application using the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions, A step of calculating the second color signal of the cosmetic using a second three-dimensional reflection characteristic model expressed by the second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, the spectral distribution of the illumination light, and the geometric conditions; A step of calculating a third color signal in the object to be coated in a cosmetic state after the cosmetic has been applied to the object, from the first color signal and the second color signal, A step of calculating a second RGB value in the object to be coated in the cosmetic state from the third color signal, A step of generating a second image based on the second RGB value, A step of generating a superimposed image by superimposing the first image and the second image, A makeup simulation method, including...

14. On the computer, A process for estimating the lighting environment in the first image or the lighting environment estimation image, using a first image which is an image of the object to be coated with the cosmetic before the cosmetic is applied, A process for calculating the first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A process for calculating a first color signal in the object to be coated before cosmetic application, using the first spectral reflectance, a first reflection characteristic parameter defining the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions, A process for calculating the second color signal of the cosmetic using a second three-dimensional reflection characteristic model expressed by the second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, the spectral distribution of the illumination light, and the geometric conditions, A process to calculate a third color signal in the object to be coated in the cosmetic state after the cosmetic has been applied to the object, from the first color signal and the second color signal, A process for calculating the second RGB value of the object to be coated in the cosmetic state from the third color signal, A process for generating a second image based on the second RGB value, A process for generating a superimposed image by superimposing the first image and the second image, A makeup simulation program that performs the following actions.

15. A lighting environment estimation unit estimates the lighting environment in the first image or lighting environment estimation image, using a first image which is an image of the object to be coated with the cosmetic before the cosmetic is applied, A first spectral reflectance calculation unit calculates a first spectral reflectance of the object to be coated before cosmetic application from the first RGB values ​​of the first image, A first color signal calculation unit calculates a first color signal for the object to be coated before cosmetic application using the first spectral reflectance, a first reflection characteristic parameter that defines the reflection characteristics of the object to be coated before cosmetic application, and a first three-dimensional reflection characteristic model expressed by the spectral distribution of illumination light and geometric conditions. A second color signal calculation unit calculates a second color signal of the cosmetic using a second spectral reflectance of the cosmetic, a second reflection characteristic parameter defining the reflection characteristics of the cosmetic, and a second three-dimensional reflection characteristic model expressed by the spectral distribution of the illumination light and the geometric conditions. A third color signal calculation unit calculates a third color signal for the object to be coated in a cosmetic state after the cosmetic has been applied to the object, based on the first color signal and the second color signal. A second RGB value calculation unit calculates a second RGB value for the object to be coated in the cosmetic state from the third color signal, A second image generation unit generates a second image based on the second RGB value, A superimposed image generation unit generates a superimposed image by superimposing the first image and the second image, A makeup simulation device equipped with the following features.

Images