Application effect visualization device, application effect visualization method, and application effect visualization program

Abstract

To provide an application effect visualization device, a method, and a program, for motivating users to apply cosmetics.SOLUTION: An application effect visualization device comprises: a video acquisition unit that acquires a live view video of a part of a user's body including at least a skin photographed using a photographing unit; a reception unit which accepts input of prescribed information pertaining to the user; a superimposition image selection unit that selects a superimposition image representing chronological changes of the skin of the user based on the information; a reflectance deriving unit that detects incoming light incoming into the skin and reflecting light reflected from the skin before and after the user applies the cosmetics to the skin, and derives reflectance of the incoming light; a determination unit which determines an application range and application amount of the cosmetics based on changes of the reflectance; and a change unit which changes a superimposition range and superimposing rate of a superimposition image to be superimposed onto the live view video based on the determined application range and application amount of the cosmetics. A display control unit displays the live view video by superimposing the superimposition image thereon based on the changed superimposition range and superimposing rate.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an application effect visualization device, an application effect visualization method, and an application effect visualization program. [Background technology]

[0002] In the above technical field, Patent Document 1 describes that an overlay is displayed in an area where cosmetics are to be applied, and when the application of cosmetics in the area where the overlay is displayed satisfies the application conditions, the overlay is deleted (paragraphs

[0058] to

[0060] , etc.). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 131852 [Patent Document 2] Patent No. 6742108 Summary of the Invention [Problem to be solved by the invention]

[0004] However, while the technology described in Patent Document 1 above can determine whether or not a cosmetic has been applied, it does not allow the user to visually confirm the effects of applying the cosmetic, and therefore does not reliably motivate the user to apply the cosmetic. [Means for solving the problem]

[0005] In order to solve the above problems, the application effect visualization device according to the present invention comprises: an image acquisition unit that acquires a live view image of a part of the user's body including at least the skin, the live view image being captured using the imaging unit; a receiving unit that receives input of predetermined information related to the user; a superimposition image selection unit that selects a superimposition image that represents a change in the user's skin over time and is to be superimposed on the live view video based on the received predetermined information; a display control unit that displays the selected superimposed image superimposed on the live view image; a reflectance deriving unit that detects incident light that is incident on the skin and reflected light that is reflected from the skin before and after the user applies the cosmetic to the skin, and derives the reflectance of the incident light; a determination unit that determines the application area and application amount of the cosmetic based on the change in reflectance; a changing unit that changes the overlapping range and overlapping rate of the superimposed image that is superimposed on the live view video based on the determined application range and application amount of the cosmetic; Equipped with The display control unit displays the superimposed image on the live view video based on the changed superimposition range and superimposition rate.

[0006] In order to solve the above problems, the method for visualizing application effects according to the present invention comprises: an image capturing step of capturing a live view image of a part of the user's body including at least the skin, captured using the imaging unit; a receiving step of receiving input of predetermined information related to the user; a superimposition image selection step of selecting a superimposition image that represents a change in the skin of the user over time and is to be superimposed on the live view video based on the received predetermined information; a display control step of superimposing the selected superimposed image on the live view video and displaying it; a reflectance deriving step of detecting incident light incident on the skin and reflected light reflected from the skin before and after the user applies the cosmetic to the skin, and deriving the reflectance of the incident light; a determining step of determining an application area and an application amount of the cosmetic based on the change in reflectance; a changing step of changing an overlapping range and an overlapping rate of the superimposed image to be superimposed on the live view video based on the determined applied range and applied amount of the cosmetic; Including, In the display control step, the superimposed image is displayed superimposed on the live view video based on the changed superimposition range and superimposition ratio.

[0007] Furthermore, in order to solve the above problems, the application effect visualization program according to the present invention comprises: an image capturing step of capturing a live view image of a part of the user's body including at least the skin, captured using the imaging unit; a receiving step of receiving input of predetermined information related to the user; a superimposition image selection step of selecting a superimposition image that represents a change in the skin of the user over time and is to be superimposed on the live view video based on the received predetermined information; a display control step of superimposing the selected superimposed image on the live view video and displaying it; a reflectance deriving step of detecting incident light incident on the skin and reflected light reflected from the skin before and after the user applies the cosmetic to the skin, and deriving the reflectance of the incident light; a determining step of determining an application area and an application amount of the cosmetic based on the change in reflectance; a changing step of changing an overlapping range and an overlapping rate of the superimposed image to be superimposed on the live view video based on the determined applied range and applied amount of the cosmetic; on the computer, In the display control step, the superimposed image is displayed superimposed on the live view video based on the changed superimposition range and superimposition ratio. [Effects of the Invention]

[0008] According to the present invention, the user can visually confirm the effect obtained by applying the cosmetic material, which reliably motivates the user to apply the cosmetic material. [Brief explanation of the drawings]

[0009] [Figure 1A] 1 is a diagram for explaining an outline of an application effect visualization device according to a first embodiment of the present invention. FIG. [Figure 1B] FIG. 3 is another diagram for explaining the outline of the application effect visualization device according to the first embodiment of the present invention. [Figure 2] 1 is a block diagram for explaining the configuration of an application effect visualization device according to a first embodiment of the present invention. [Figure 3A] FIG. 2 is a diagram showing an example of a superimposition image selection table included in the application effect visualization device according to the first embodiment of the present invention. [Figure 3B] FIG. 3 is a diagram showing an example of an overlapping ratio table included in the application effect visualization device according to the first embodiment of the present invention. [Figure 4] 1 is a diagram illustrating a hardware configuration of an application effect visualization device according to a first embodiment of the present invention. [Figure 5] 3 is a flowchart illustrating a processing procedure of the application effect visualization device according to the first embodiment of the present invention. [Figure 6] FIG. 4 is a block diagram illustrating the configuration of an application effect visualization device according to a second embodiment of the present invention. [Figure 7] FIG. 10 is a diagram showing an example of an overlapping ratio table included in the application effect visualization device according to the second embodiment of the present invention. [Figure 8] FIG. 4 is a diagram illustrating the hardware configuration of an application effect visualization device according to a second embodiment of the present invention. [Figure 9] 10 is a flowchart illustrating a processing procedure of an application effect visualization device according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail by way of example with reference to the drawings. However, the configurations, numerical values, processing flows, functional elements, etc. described in the following embodiments are merely examples, and are open to modification and alteration, and are not intended to limit the technical scope of the present invention to the following description.

[0011] [First embodiment] An application effect visualization device 100 according to a first embodiment of the present invention will be described with reference to FIGS. 1A to 5. First, it is said that 80% of skin aging is caused by photoaging, with the remaining aging caused by erythema and darkening. Therefore, protecting the skin from ultraviolet light (ultraviolet rays) is an important factor in preventing skin aging. However, because ultraviolet light cannot be seen with the naked eye, the ultraviolet light protection effect of cosmetics intended to protect the skin from ultraviolet light cannot be seen with the naked eye. Furthermore, even if a user 110 applies cosmetics, it takes time for the effects to appear, so the effects cannot be immediately felt. Some users 110 find applying cosmetics somewhat bothersome.

[0012] Here, UV (Ultra Violet) cameras are known as a technology for visualizing ultraviolet light, but a common method of presenting the ultraviolet light protection effect using a UV camera is to make the area where cosmetics have been applied appear blacked out. This presentation method only lets the user know whether or not cosmetics have been applied, and is not an attractive presentation method for the user 110, and does not provide an experience that motivates the user to actively protect their skin from ultraviolet light by applying cosmetics.

[0013] Therefore, as shown in FIG. 1A, in the application effect visualization device 100, when a user 110 captures an image of his or her own face using the imaging camera of the mobile terminal 120, an AR (Augmented Reality) image 130 is displayed in which a predetermined superimposed image is superimposed on the current image of the face captured by the user 110 (live view image).

[0014] The superimposed image superimposed on the live view video is an image showing changes over time in the skin of the face of the user 110, such as an image of sunburned and flushed skin, or an image of aged skin with noticeable spots and wrinkles. The application effect visualization device 100 then displays the AR video 130, in which such a superimposed image is superimposed on the live view video of the user 110, on the display of the mobile terminal 120 or the like.

[0015] Here, the application effect visualization device 100 is connected via a network to a mobile terminal 120, such as a smartphone or tablet terminal, carried by a user 110. Note that, although the application effect visualization device 100 (server) connected to the mobile terminal 120 via a network will be described as an example, the mobile terminal 120 may also perform various processes that are performed in the application effect visualization device 100, for example.

[0016] When the user 110 captures an image of his or her own face or the like using the imaging camera of the mobile terminal 120, the captured image of the face (live view image 140) is transmitted to the application effect visualization device 100. The application effect visualization device 100 acquires the transmitted image of the face and generates AR image 130 by superimposing an image representing changes over time in the skin of the user 110's face on the current image of the face being captured by the user 110. When the application effect visualization device 100 transmits the generated AR image 130 to the mobile terminal 120, the AR image is displayed on the display of the mobile terminal 120. The image representing changes over time is, for example, an image representing the condition of the facial skin of the user 110 after several days, several months, or several years have passed.

[0017] 1A, when user 110 is outdoors on a sunny day, and the skin on his or her face is directly exposed to ultraviolet light (ultraviolet rays) contained in sunlight, the effects of the ultraviolet light make the skin more susceptible to future blemishes and wrinkles. Therefore, application effect visualization device 100 presents user 110, as AR image 130, with the future state of the skin of user 110 when, for example, the user properly applies cosmetics with ultraviolet light protection, when the user does not properly apply the cosmetics, or when the user does not apply the cosmetics at all.

[0018] Then, while viewing the AR video 130 (a live view video in which blemishes, wrinkles, etc. are superimposed), the user 110 applies, for example, a cosmetic product with ultraviolet light protection to skin, such as the face. The application effect visualization device 100 then generates the AR video 130 by changing the superimposition range and superimposition rate of the superimposed image for the area where the cosmetic product has been applied, depending on the amount and area of ​​application of the cosmetic product, and displays the AR video 130 on the display of the mobile terminal 120. That is, the superimposition range and superimposition rate (transmittance) of the superimposed image change according to the state of application of the cosmetic product. For example, the superimposed image of blemishes, wrinkles, etc. gradually fades depending on the amount of application of the cosmetic product, and when the amount of application of the cosmetic product reaches an appropriate level, the superimposed image becomes invisible. Therefore, in areas where the superimposed image is no longer superimposed, a live view video of the skin with the cosmetic product applied is displayed. Conversely, in areas where the cosmetic product is not applied, the AR video 130 with the superimposed image still superimposed is displayed. For the portion where the cosmetic has been applied, a second superimposed image (another superimposed image different from the superimposed image) may be superimposed on the portion where the superimposed image has been removed, depending on the amount of cosmetic applied. Here, the second superimposed image may be, for example, an image of skin that has been made beautiful by the application of the cosmetic, or an image of skin that has undergone natural aging with the effects of photoaging removed, but is not limited to these.

[0019] This allows the user 110 to protect their skin from ultraviolet light using cosmetics, thereby enabling them to change their skin in a desirable direction. Furthermore, the user 110 can experience the image of their skin with blemishes, wrinkles, etc. changing into an image of skin without blemishes, wrinkles, etc., and therefore feel that applying cosmetics is a desirable action for them. In this way, the application effect visualization device 100 can strongly motivate the user 110 to actively protect their skin from ultraviolet light. Furthermore, for example, the application effect visualization device 100 can also superimpose a superimposed image on areas where protection from ultraviolet light is particularly important, allowing the user 110 to efficiently apply cosmetics with ultraviolet light protection effects.

[0020] 1B, an overview of generating AR video 130 will be described. Note that, while an example in which AR video 130 is generated by superimposing a superimposed image on live view video 140 of the face of user 110 will be described here, the body part on which the superimposed image is superimposed is not limited to the face, and may be any part of the skin.

[0021] User 110 captures live view video 140 using the imaging camera of mobile terminal 120. User 110 then inputs predetermined information, such as user 110's skin color, age, race, and gender, using mobile terminal 120. User 110 also inputs the number of years for information about his or her future appearance, such as the condition of his or her facial skin due to skin aging in a few years, and selects photoaging as the cause of skin aging. Note that, in this example, the description is given assuming that user 110 has selected photoaging as the cause of skin aging, but examples of skin aging caused by ultraviolet light are not limited to photoaging and include, for example, darkening.

[0022] The application effect visualization device 100 selects an image to be superimposed (superimposed image 150) based on information input by the user 110. After the superimposed image 150 is selected, the application effect visualization device 100 determines the superimposition rate and the superimposition range of the superimposed image 150.

[0023] For example, as shown in the superimposition rate image 160, if the superimposition rate is 100%, the superimposition image 150 is superimposed as is on the entire face of the user 110. Note that the superimposition image 150 is not superimposed on the eyes and mouth (portions other than the skin) of the face of the user 110, so that holes appear. Here, the superimposition rate is expressed as 0 to 100%, but it may also be expressed as, for example, 0 (0%) to 1 (100%).

[0024] Then, as the superimposition rate of the superimposed image 150 decreases, the application effect visualization device 100 displays the superimposed image 150 superimposed on the live view video 140 so that, for example, the superimposed image 150 gradually fades. For example, the superimposed image 150 displays blemishes and wrinkles displayed in the AR video 130 so that they gradually fade. Also, for example, in the superimposed image 131, since no cosmetic product has been applied, the superimposed image 150, which shows the state of photoaging after a predetermined number of years has passed, is superimposed as it is on the entire live view video 140 and displayed as the superimposed image 131 on the display of the mobile terminal 120, for example.

[0025] Furthermore, when user 110 applies cosmetics, an overlay image is superimposed according to the amount and area of ​​application. For example, consider a case where user 110 applies cosmetics to the right half of their face and does not apply cosmetics to the left half of their face. If this state is observed using the UV camera described in Patent Document 2, for example, it will appear as shown in application image 170. That is, in application image 170, the hatched areas are the areas where cosmetics are applied, and the non-hatched areas are the areas where cosmetics are not applied. In an image captured by an actual UV camera, the hatched areas appear black in the areas where cosmetics are applied because the cosmetics absorb ultraviolet light, and the areas where cosmetics are not applied appear white because the ultraviolet light is reflected by user 110's skin.

[0026] The application effect visualization device 100 then generates a superimposed image 132 based on the application image 170. That is, in the superimposed image 132, since the cosmetic is applied to the right half of the face, the superimposed image 150 that was superimposed on the right half of the face (hatched portion) disappears, the superimposition rate becomes 0%, and the original live view video 140 is displayed. Conversely, the superimposed image 150 is displayed superimposed at a predetermined superimposition rate on the left half of the face. In this way, the application effect visualization device 100 determines the superimposition range and superimposition rate of the superimposed image 150 using an image from a UV camera, for example. To determine the superimposition range and superimposition rate of the superimposed image 150 using a UV camera, for example, the technology described in Patent Document 2 can be used.

[0027] 1B, the contours of the faces of the people appearing in the live view video 140 and the superimposed image 150 are different. However, in the application effect visualization device 100, when the superimposed image 150 is actually superimposed on the live view video 140, alignment is performed as follows. That is, feature points are extracted from the live view video 140 and the superimposed image 150, and the superimposed image 150 is superimposed on the live view video 140 so that the extracted feature points match. Note that the contours of the face of the user 110 and the contours of the face of the person appearing in the superimposed image 150 are different. In this case, the application effect visualization device 100, for example, appropriately trims the portion that extends beyond the contours of the face of the user 110, adjusting the contours so that they match the contours of the face of the user 110. Furthermore, for any missing parts of the facial contour of the user 110, the application effect visualization device 100 adjusts the contour by appropriately expanding it, etc., so that it matches the facial contour of the user 110.

[0028] In this way, in the application effect visualization device 100, the overlapping ratio and overlapping range of the superimposed image (superimposed image 150) are changed according to the amount and range of application of the cosmetic, thereby enabling the user 110 to apply the cosmetic while checking the application effect of the cosmetic.

[0029] As described above, the application effect visualization device 100 allows the user 110 to visually confirm the condition of the user's 110's skin after a predetermined number of years have passed and the effects of applying cosmetics to prevent skin aging. Therefore, for example, if the user 110 is unsure how much cosmetics to use on a day spent outdoors, the user 110 can use the application effect visualization device 100 to perform appropriate skin care that suits how they spend that day. That is, by specifying the day's planned activities, the user 110 can use the application effect visualization device 100 to check the effects of the cosmetics and perform reliable skin care, even if the skin becomes darker red on a day spent outdoors and the red color does not disappear unless a larger amount of cosmetics is applied.

[0030] Furthermore, if the user 110 has sensitive skin and is unsure whether it is okay to use the same amount of cosmetics as other people, the application effect visualization device 100 can be used to display darker spots, deeper wrinkles, and redder skin for people who are more susceptible to sunburn. In this way, the application effect visualization device 100 can change the superimposed image 150 in various ways depending on the skin type of the user 110, allowing the user 110 to take appropriate skin care measures that suit their own skin type.

[0031] Furthermore, the user 110 can use the application effect visualization device 100 to check whether there are any problems with the application area or amount of the cosmetics several hours after applying the cosmetics in the morning. For example, when the user 110 is out and about, the user 110 can use the application effect visualization device 100 instead of a mirror, and reapply the cosmetics frequently while looking at the AR image 130, which gives the user 110 a sense of security. As described above, the application effect visualization device 100 allows the user 110 to check the application state of the cosmetics in various situations.

[0032] Next, the configuration of the application effect visualization device 100 will be described with reference to Fig. 2. The application effect visualization device 100 includes an image acquisition unit 201, a reception unit 202, a superimposed image selection unit 203, a display control unit 204, a reflectance derivation unit 205, a determination unit 206, and a change unit 207. The display control unit 204 further includes a feature point extraction unit 241, and the reflectance derivation unit 205 further includes an ultraviolet light detection unit 251.

[0033] The image acquisition unit 201 acquires a live view image 140 of a part of the body of the user 110, including at least the skin, captured using an imaging unit. In other words, the image acquisition unit 201 acquires an image of the user 110's own face, arms, etc., captured by the imaging camera of the mobile terminal 120 (for example, the in-camera of the mobile terminal 120). The imaging camera may be included in the application effect visualization device 100, or may be a dedicated camera device separate from the mobile terminal 120 or the application effect visualization device 100.

[0034] The receiving unit 202 receives input of predetermined information related to the user 110. Here, the predetermined information includes information related to the skin of the user 110, information related to the external environment in which the user 110 is placed, and information related to the elapsed time of the skin condition that the user 110 wants to know. The information related to the skin of the user 110 (skin information) includes at least one of the skin color, age, race, and sex of the user 110. In addition, the information may include information such as the place of residence, hobbies, and lifestyle habits. Furthermore, the information may include information other than the information shown here, as long as it is considered to affect changes in the skin of the user 110 over time, and may include, for example, information such as a medical history of skin diseases or other diseases.

[0035] Furthermore, the information about the external environment in which the user 110 is placed is, for example, information about whether the user 110 is outdoors or indoors, etc. Note that the information may also include information about whether it is daytime or nighttime, whether in the sun or in the shade, or information about the weather, such as whether it is sunny or cloudy.

[0036] Furthermore, the information that user 110 wants to know about the elapsed time of the skin condition is, for example, "erythema after 1 day" if user 110 wants to know the degree (degree of progression) of erythema after 1 day, and "photoaging after 10 years" if user 110 wants to know the degree (degree of progression) of photoaging after 10 years.

[0037] For example, the user 110 may input the predetermined information using an input device of the mobile terminal 120, and the reception unit 202 may receive the input information from the mobile terminal 120. Alternatively, the reception unit 202 may receive information input by the user 110 using an input device provided in the application effect visualization device 100.

[0038] Based on the received predetermined information, the superimposed image selection unit 203 selects a superimposed image 150 that represents changes in the user 110's skin over time and is to be superimposed on the live view video 140. For example, if the user 110 is outdoors during the day and wants to know the state of photoaging at a certain point in the future, the superimposed image selection unit 203 selects an image in which photoaging has progressed for a predetermined number of years as the superimposed image 150 to be superimposed, taking into consideration information such as the user 110's age. The superimposed image 150 is stored in advance in, for example, a predetermined internal storage or external storage, and the superimposed image selection unit 203 selects a superimposed image 150 that meets the user 110's request from such storage. Note that the selected superimposed image 150 is not limited to a photoaging image and may be, for example, an erythema image or a melanoma image. The superimposed image selection unit 203 may select one or more of these images. That is, when multiple superimposed images 150 are selected, the multiple superimposed images 150 are superimposed on the live view video 140.

[0039] The display control unit 204 displays the selected superimposed image 150 superimposed on the live view video 140. The display control unit 204 displays an image (AR video 130) in which the superimposed image 150 is superimposed on the live view video 140, for example, on the display of the mobile terminal 120 carried by the user 110. The display control unit 204 displays, for example, a photoaging image superimposed on the live view video 140 as the superimposed image 150.

[0040] The display control unit 204 further includes a feature point extraction unit 241. The feature point extraction unit 241 extracts feature points from each of the live view video 140 and the superimposed image 150. For example, in the case of a face, the feature points include the eyebrows, eyes, nose, and mouth. The display control unit 204 then superimposes the superimposed image 150 on the live view video 140 so that the extracted feature points of the live view video 140 and the superimposed image 150 match, thereby generating and displaying the AR video 130.

[0041] Furthermore, when superimposing the superimposed image 150 on the live view video 140, even if the feature points of the two are matched and superimposed, for example, the outlines of the two may not match, and one of the two may protrude beyond the other or may be insufficient. In this case, the display control unit 204, for example, trims the protruding portion or expands the insufficient portion, superimposing the outline of the superimposed image 150 and the outline of the live view video 140 so that they match, and displays the AR video 130.

[0042] The reflectance derivation unit 205 detects incident light incident on the skin and reflected light reflected from the skin before and after the user 110 applies cosmetics to the skin, and derives the reflectance of the incident light. In areas where cosmetics are applied, the incident light is absorbed by the cosmetics, and in areas where cosmetics are not applied, the incident light is reflected by the skin. Therefore, the reflectance derivation unit 205 derives the reflectance of the incident light incident on the skin area included in the live view video 140 before and after the application of cosmetics. In other words, because the amount of reflected light changes due to the application of cosmetics, the amount and area of ​​application of cosmetics can be determined by deriving the reflectance of the incident light.

[0043] The reflectance derivation unit 205 further includes an ultraviolet light detection unit 251. The ultraviolet light detection unit 251 detects ultraviolet light from incident light. The incident light contains various light components. Among these, ultraviolet light has a large effect on, for example, photoaging. Therefore, by detecting the ultraviolet light component from the incident light in the ultraviolet light detection unit 251, the user 110 can view, for example, an AR video 130 about photoaging.

[0044] The reflectance deriving unit 205 then derives the reflectance of incident light from the detected ultraviolet light using, for example, the technology described in Patent Document 2. That is, the reflectance is mathematically derived from internally scattered light by placing polarizing plates at two locations, one on the incident light side and one on the reflected light side of the subject, to remove specularly reflected light (gloss). In this way, using the technology described in Patent Document 2 makes it possible to quantitatively evaluate the application effect of cosmetics, but the technology for deriving reflectance is not limited to this, and other technologies may also be used.

[0045] The determination unit 206 determines the application area and application amount of the cosmetic based on the change in reflectance. That is, the determination unit 206 derives the change in reflectance before and after application of the cosmetic from the reflectance of the incident light derived by the reflectance derivation unit 205, and determines the application area and application amount of the cosmetic from the derived change in reflectance. The incident light is absorbed in the area where the cosmetic is applied, but the reflectance changes depending on the amount of the cosmetic applied, etc., so the determination unit 206 can use this to determine the application area and application amount of the cosmetic. The determination unit 206 determines the application area and application amount of the cosmetic based on, for example, the change in reflectance of ultraviolet light detected by the ultraviolet light detection unit 251 from the incident light. The ultraviolet light transmittance of the cosmetic can also be determined using the technology described in Patent Document 2.

[0046] Based on the determined application range, application amount, and ultraviolet light transmittance of the cosmetic, the change unit 207 changes the overlap range and overlap rate of the superimposed image 150 to be superimposed on the live view video 140. In other words, the area where the cosmetic is applied is no longer affected by the incident light (ultraviolet light), so there is no need to already superimpose the superimposed image 150 on that area, or the effect of the incident light on the skin is reduced. Therefore, the change unit 207 changes the overlap range and overlap rate of the superimposed image 150 depending on the degree of application of the cosmetic.

[0047] Therefore, since the areas where the cosmetic is sufficiently applied are not affected by ultraviolet light, the display control unit 204 superimposes the superimposed image 150 at a superimposition rate of 0% in those areas. In other words, the superimposed image 150 is not superimposed in those areas, and the live view video 140 itself is displayed.

[0048] On the other hand, for example, in an area where cosmetics have been applied but the amount of application is insufficient, the display control unit 204 superimposes the superimposed image 150 on the live view video 140 at the superimposition ratio changed by the changing unit 207. Therefore, the display control unit 204 displays the superimposed image 150 superimposed on the live view video 140 at a superimposition ratio according to the reflectance of incident light in the area where cosmetics has been applied.

[0049] For example, if the amount of cosmetic applied is half the amount required for the cosmetic to exert its ultraviolet light protection effect, the change unit 207 changes the superimposition rate of the superimposed image 150 to 50%. Then, the display control unit 204 displays the superimposed image 150 superimposed on the live view video 140 at the changed superimposition rate. In this case, the display control unit 204 displays the superimposed image 150 superimposed on the live view video 140 at, for example, about half the transparency compared to when the superimposed image 150 is superimposed at 100%. In other words, the display control unit 204 displays the superimposed image 150 superimposed on the live view video 140 so that the superimposed image 150 gradually fades as the superimposition rate decreases. In addition to changing the transparency, for example, blemishes and wrinkles in the superimposed image 150 may gradually fade as the superimposition rate changes.

[0050] In the above description, photoaging has been used as an example of skin aging caused by ultraviolet light, but examples of skin aging caused by ultraviolet light are not limited to this and include, for example, darkening. Therefore, for example, the display control unit 204 may display a darkening image as the superimposed image 150 superimposed on the live view video 140.

[0051] An example of a superimposed image selection table 301 included in the application effect visualization device 100 will be described with reference to FIG. 3A . The superimposed image selection table 301 stores a superimposed image 312 in association with predetermined information 311. The predetermined information 311 is information about the user 110. The predetermined information 311 includes skin information, external environment information, and skin condition information. The skin information includes the skin color, age, race, and gender of the user 110. The external environment information is information about the environment in which the user 110 is placed, such as information about whether the user 110 is outdoors or indoors, and, if outdoors, information about whether it is daytime or nighttime.

[0052] The skin condition information is information about the user's future appearance (skin condition) that the user 110 wants to know. If the user 110 wants to know about wrinkles or blemishes on the skin, the user 110 can select "photoaging" and "elapsed time" such as years. The application effect visualization device 100 then refers to the superimposed image selection table 301 to select the superimposed image 150 to be superimposed on the live view video 140.

[0053] 3B , an example of the superimposition ratio table 302 held by the application effect visualization device 100 will be described. The superimposition ratio table 302 stores a superimposition ratio 322 in association with an ultraviolet light reflectance 321. The ultraviolet light reflectance 321 ([%]) is the reflectance of ultraviolet light derived by the reflectance derivation unit 205. The superimposition ratio 322 is a value indicating the ratio of superimposition when the superimposed image 150 is superimposed on the live view video 140. The application effect visualization device 100 then refers to the superimposition ratio table 302 and superimposes the superimposed image 150 on the live view video 140 at a superimposition ratio corresponding to the ultraviolet light reflectance.

[0054] The hardware configuration of the application effect visualization device 100 will be described with reference to FIG. 4. The CPU (Central Processing Unit) 410 is a processor for arithmetic and control, and executes programs to realize the various functional components of the application effect visualization device 100 shown in FIG. 2. The CPU 410 may have multiple processors and execute different programs, modules, tasks, threads, etc. in parallel. The ROM (Read Only Memory) 420 stores fixed data such as initial data and programs, as well as other programs. The network interface 430 communicates with other devices via a network. The CPU 410 is not limited to a single CPU, and may include multiple CPUs or a GPU (Graphics Processing Unit) for image processing. The network interface 430 preferably has a CPU independent of the CPU 410 and writes and reads transmitted and received data to and from the RAM (Random Access Memory) 440. It is also preferable to provide a DMAC (Direct Memory Access Controller) (not shown) for transferring data between the RAM 440 and the storage 450. Furthermore, the CPU 410 processes the data after recognizing that the data has been received or transferred to the RAM 440. The CPU 410 also prepares the processing results in the RAM 440, and leaves the subsequent transmission or transfer to the network interface 430 or DMAC.

[0055] The RAM 440 is a random access memory used by the CPU 410 as a work area for temporary storage. A storage area for storing data necessary for implementing this embodiment is reserved in the RAM 440. The live view video data 441 is video data captured by the user 110 using the mobile terminal 120. The selected superimposed image 442 is image data of the superimposed image 150 selected to be superimposed on the live view video 140. The ultraviolet light reflectance 443 is the ultraviolet light reflectance derived by the reflectance derivation unit 205. The superimposition range data 444 is data regarding the portion on which the superimposed image 150 is to be superimposed, such as coordinate data indicating the superimposition range. The superimposition rate data 445 is data regarding the rate at which the superimposed image 150 is to be superimposed.

[0056] The transmitted / received data 446 is data transmitted and received via the network interface 430. The RAM 440 also has an application execution area 447 for executing various application modules.

[0057] The storage 450 stores a database, various parameters, or the following data or programs required to implement this embodiment. The storage 450 stores a superimposed image selection table 301 and a superimposed ratio table 302. The superimposed image selection table 301 is a table that manages the relationship between predetermined information 311 and superimposed images 312. The superimposed ratio table 302 is a table that manages the relationship between ultraviolet light reflectance 321 and superimposed ratio 322.

[0058] The storage 450 further stores an image acquisition module 451, a reception module 452, a superimposed image selection module 453, a display control module 454, a reflectance derivation module 455, a determination module 456, and a change module 457. The display control module 454 stores a feature point extraction module 4541, and the reflectance derivation module 455 stores an ultraviolet light detection module 4551.

[0059] The image acquisition module 451 is a module that acquires a live view image 140 of a part of the body of the user 110, including at least the skin, captured using an imaging unit. The reception module 452 is a module that receives input of predetermined information related to the user 110. The superimposed image selection module 453 is a module that selects a superimposed image 150 that represents changes over time in the skin of the user 110 and is to be superimposed on the live view image 140, based on the received predetermined information. The display control module 454 is a module that superimposes and displays the selected superimposed image 150 on the live view image 140. The feature point extraction module 4541 is a module that extracts feature points of the live view image 140 and the superimposed image 150.

[0060] The reflectance derivation module 455 detects incident light incident on the skin and reflected light reflected from the skin before and after the user 110 applies a cosmetic to the skin, and derives the reflectance of the incident light. The ultraviolet light detection module 4551 detects ultraviolet light from the incident light. The determination module 456 determines the application area and amount of cosmetic based on changes in reflectance. The change module 457 changes the overlap area and overlap rate of the superimposed image 150 superimposed on the live view video 140 based on the determined application area and amount of cosmetic. These modules 451 to 457 are loaded into the application execution area 447 of the RAM 440 by the CPU 410 and executed. The control program 458 is a program for controlling the entire application effect visualization device 100.

[0061] The input / output interface 460 interfaces input / output data with input / output devices. A display unit 461 and an operation unit 462 are connected to the input / output interface 460. A storage medium 464 may also be connected to the input / output interface 460. A speaker 463 serving as an audio output unit, a microphone (not shown) serving as an audio input unit, or a GPS position determination unit may also be connected. Note that the RAM 440 and storage 450 shown in FIG. 4 do not include programs or data related to the general-purpose functions of the application effect visualization device 100 or other feasible functions.

[0062] Next, the processing procedure of the application effect visualization device 100 will be described with reference to the flowchart shown in Fig. 5. This flowchart is executed by the CPU 410 in Fig. 4 using the RAM 440, and realizes each functional configuration of the application effect visualization device 100 in Fig. 2.

[0063] In step S501, the image acquisition unit 201 acquires live view image 140 of a part of the body of the user 110, including at least the skin. In step S503, the reception unit 202 receives input of predetermined information related to the user 110. In step S505, the superimposed image selection unit 203 selects a superimposed image 150, which represents changes over time in the skin of the user 110, to be superimposed on the live view image 140, based on the received predetermined information.

[0064] In step S507, the display control unit 204 displays the selected superimposed image 150 superimposed on the live view video 140. In step S509, the ultraviolet light detection unit 251 detects ultraviolet light contained in the incident light. In step S511, the reflectance derivation unit 205 detects ultraviolet light contained in the incident light that enters the skin and reflected light that reflects from the skin before and after the user 110 applies the cosmetic to the skin, and derives the reflectance of the ultraviolet light contained in the incident light.

[0065] In step S513, the determination unit 206 determines the application area, application amount, and ultraviolet light transmittance of the cosmetic based on the change in reflectance. In step S515, the change unit 207 changes the overlapping area and overlapping rate of the overlapping image 150 to be overlapped on the live view video 140 based on the determined application area, application amount, and ultraviolet light transmittance of the cosmetic. In step S517, the display control unit 204 displays the overlapping image 150 superimposed on the live view video 140 based on the changed overlapping area and overlapping rate. In step S519, the application effect visualization device 100 determines whether there is a change in the reflectance of the incident light. If it is determined that there is a change in the reflectance of the incident light (YES in step S519), the application effect visualization device 100 returns to step S513. If it is determined that there is no change in the reflectance of the incident light (NO in step S519), the application effect visualization device 100 ends the processing.

[0066] According to this embodiment, the user can visually confirm the effects of applying cosmetics, which reliably motivates the user to apply cosmetics. Furthermore, the user's future appearance can be shown, allowing the user to easily imagine what their future appearance will be, which more reliably motivates the user to apply cosmetics. Furthermore, because ultraviolet light contained in the incident light is detected, the user can more specifically visually confirm the future occurrence of wrinkles and age spots on the skin due to photoaging, which more strongly motivates the user to apply cosmetics.

[0067] [Second embodiment] Next, an application effect visualization device 600 according to a second embodiment of the present invention will be described with reference to FIGS. 6 to 9. FIG. 6 is a block diagram illustrating the configuration of the application effect visualization device 600 according to this embodiment. The application effect visualization device 600 according to this embodiment differs from the first embodiment in that it includes an infrared light detection unit and an absorptance derivation unit. Other configurations and operations are similar to those of the first embodiment, and therefore the same configurations and operations are denoted by the same reference numerals and detailed description thereof will be omitted.

[0068] First, it is known that infrared light contained in incident light can cause erythema as an effect of incident light on the skin of user 110. Therefore, the application effect visualization device 600 is configured to detect the infrared light contained in the incident light and visually present to user 110 the effect of the infrared light on the skin of user 110. In other words, if user 110 wants to know the possibility of developing erythema in the future, by using application effect visualization device 600, user 110 can visually recognize the state of erythema.

[0069] The application effect visualization device 600 includes an infrared light detection unit 601 and an absorptance deriving unit 602. The infrared light detection unit 601 detects infrared light from incident light. The absorptance deriving unit 602 derives the absorptance of infrared light in the skin when the user 110 applies a cosmetic to the skin. In other words, when the user 110 applies a cosmetic to the skin, infrared light contained in the incident light that enters the skin of the user 110 is absorbed. Therefore, by deriving the absorptance of infrared light, it becomes possible to quantitatively evaluate the effect of applying the cosmetic.

[0070] The determination unit 206 then determines the application area and amount of the cosmetic based on the change in the infrared light absorptance. As described above, infrared light is absorbed in areas where the cosmetic is applied, and therefore the application area and amount of the cosmetic can be determined according to the change in the infrared light absorptance, and the superimposition area and superimposition rate of the superimposed image 150 can be changed. The display control unit 204 displays the superimposed image 150 (erythema image) superimposed on the live view video 140 based on the changed superimposition area and superimposition rate.

[0071] 7, an example of the superimposition ratio table 701 held by the application effect visualization device 600 will be described. The superimposition ratio table 701 stores a superimposition ratio 712 in association with an infrared light absorptance 711. The infrared light absorptance 711 ([%]) is the absorptance of infrared light derived by the absorptance derivation unit 602. The superimposition ratio 712 is a value indicating the ratio of superimposition when the superimposed image 150 is superimposed on the live view video 140. The application effect visualization device 600 then refers to the superimposition ratio table 701 and superimposes the superimposed image 150 on the live view video 140 at a superimposition ratio corresponding to the infrared light absorptance.

[0072] The hardware configuration of the application effect visualization device 600 will be described with reference to Fig. 8. The RAM 840 is a random access memory used by the CPU 410 as a work area for temporary storage. The RAM 840 has a storage area reserved for storing data necessary for realizing this embodiment. The infrared light absorptance 841 is the absorptance of infrared light derived by the absorptance derivation unit 602.

[0073] The storage 850 stores a database, various parameters, or the following data or programs required to implement this embodiment. The storage 850 stores the superimposition ratio table 701. The superimposition ratio table 701 is a table that manages the relationship between the infrared light absorptance 711 and the superimposition ratio 712.

[0074] The storage 850 further stores an infrared light detection module 851 and an absorptance derivation module 852. The infrared light detection module 851 is a module that detects infrared light contained in incident light. The absorptance derivation module 852 is a module that detects infrared light (incident light) incident on the skin and infrared light reflected from the skin before and after the user 110 applies cosmetic makeup to the skin, and derives the absorptance of infrared light. These modules 851 to 852 are read into the application execution area 447 of the RAM 840 by the CPU 410 and executed.

[0075] Next, the processing procedure of the application effect visualization device 600 will be described with reference to the flowchart shown in Fig. 9. This flowchart is executed by the CPU 410 in Fig. 8 using the RAM 840, and realizes each functional configuration of the application effect visualization device 600 in Fig. 6.

[0076] In step S901, the infrared light detection unit 601 detects infrared light. In step S903, the absorptance derivation unit 602 derives the absorptance of infrared light in the skin when the user 110 applies a cosmetic to the skin. Then, the display control unit 204 displays, for example, a superimposed image 150 of erythema on the live view video 140. In step S905, the application effect visualization device 600 determines whether there has been a change in either the reflectance or the absorptance. If it is determined that there has been a change (YES in step S905), the application effect visualization device 600 returns to step S513. If it is determined that there has been no change (NO in step S905), the application effect visualization device 600 ends the process.

[0077] According to this embodiment, infrared light contained in the incident light is detected, allowing the user to visually confirm the future occurrence of erythema, which is skin aging caused by infrared light, thereby providing the user with even stronger motivation to apply cosmetics.

[0078] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified as appropriate. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. Furthermore, systems or devices that combine separate features included in each embodiment in any manner are also included in the scope of the present invention.

[0079] The present invention may also be applied to a system consisting of multiple devices or to a single device. Furthermore, the present invention may also be applied when an information processing program that realizes the functions of the embodiments is supplied to a system or device and executed by a built-in processor. Therefore, the technical scope of the present invention also includes a program installed on a computer to realize the functions of the present invention, a medium storing the program, a WWW (World Wide Web) server from which the program is downloaded, and a processor that executes the program. In particular, the technical scope of the present invention also includes a non-transitory computer-readable medium storing a program that causes a computer to execute at least the processing steps included in the above-described embodiments.

Claims

1. an image acquisition unit that acquires a live view image of a part of the user's body including at least the skin, the live view image being captured using the imaging unit; a receiving unit that receives input of predetermined information about the user, including information about the user's skin, information about the external environment in which the user is placed, and information about the elapsed time of the skin condition that the user wants to know; a superimposition image selection unit that selects a superimposition image that represents a change in the skin of the user over time and is to be superimposed on the live view video based on the received predetermined information; a display control unit that displays the selected superimposed image superimposed on the live view image; a reflectance deriving unit that detects incident light that is incident on the skin and reflected light that is reflected from the skin before and after the user applies the cosmetic to the skin, and derives the reflectance of the incident light; a determination unit that determines the application area and application amount of the cosmetic based on the change in reflectance; a changing unit that changes an overlapping range of the superimposed image to be superimposed on the live view video and an overlapping ratio that indicates a ratio at which the superimposed image is to be superimposed on the live view video, based on the determined application range and application amount of the cosmetic; Equipped with The display control unit superimposes and displays the superimposed image on the live view video based on the changed superimposition range and superimposition rate, thereby allowing the user to visually confirm the effect of applying the cosmetic product in terms of information regarding the elapsed time of the skin condition that the user wants to know.

2. the reflectance deriving unit further includes an ultraviolet light detecting unit that detects ultraviolet light from the incident light, The application effect visualization device according to claim 1 , wherein the determination unit determines the application area, application amount, and ultraviolet light transmittance of the cosmetic based on the change in the ultraviolet light reflectance.

3. an infrared light detection unit that detects infrared light from the incident light; an absorptance deriving unit that derives the absorptance of the infrared light in the skin when the user applies a cosmetic to the skin; Furthermore, The application effect visualization device according to claim 1 or 2, wherein the determination unit determines the application area and application amount of the cosmetic based on the change in the absorption rate.

4. the display control unit further includes a feature point extraction unit that extracts feature points of the live view video and the selected superimposed image, 4. The application effect visualization device according to claim 1, wherein the selected overlay image is superimposed on the live view video so as to match the extracted feature points.

5. The application effect visualization device according to claim 1 , wherein the information about the skin includes at least one of the skin color, age, race, and sex of the user.

6. 6. The coating effect visualization device according to claim 1, wherein the superimposed image includes at least one of a photoaging image, an erythema image, and a darkening image.

7. an image capturing step of capturing a live view image of a part of the user's body including at least the skin, captured using the imaging unit; a receiving step of receiving input of predetermined information about the user, including information about the user's skin, information about the external environment in which the user is placed, and information about the elapsed time of the skin condition that the user wants to know; a superimposition image selection step of selecting a superimposition image that represents a change in the skin of the user over time and is to be superimposed on the live view video based on the received predetermined information; a display control step of superimposing the selected superimposed image on the live view video and displaying it; a reflectance deriving step of detecting incident light incident on the skin and reflected light reflected from the skin before and after the user applies the cosmetic to the skin, and deriving the reflectance of the incident light; a determining step of determining an application area and an application amount of the cosmetic based on the change in reflectance; a changing step of changing an overlapping range of the superimposed image to be superimposed on the live view video and an overlapping ratio indicating a ratio at which the superimposed image is to be superimposed on the live view video, based on the determined application range and application amount of the cosmetic; Including, In the display control step, the superimposed image is superimposed on the live view video based on the changed superimposition range and superimposition rate, thereby allowing the user to visually confirm the effect of applying the cosmetic product in terms of information regarding the elapsed time of the skin condition that the user wants to know.

8. an image capturing step of capturing a live view image of a part of the user's body including at least the skin, captured using the imaging unit; a receiving step of receiving input of predetermined information about the user, including information about the user's skin, information about the external environment in which the user is placed, and information about the elapsed time of the skin condition that the user wants to know; a superimposition image selection step of selecting a superimposition image that represents a change in the skin of the user over time and is to be superimposed on the live view video based on the received predetermined information; a display control step of superimposing the selected superimposed image on the live view video and displaying it; a reflectance deriving step of detecting incident light incident on the skin and reflected light reflected from the skin before and after the user applies the cosmetic to the skin, and deriving the reflectance of the incident light; a determining step of determining an application area and an application amount of the cosmetic based on the change in reflectance; a changing step of changing an overlapping range of the superimposed image to be superimposed on the live view video and an overlapping ratio indicating a ratio at which the superimposed image is to be superimposed on the live view video, based on the determined application range and application amount of the cosmetic; on the computer, In the display control step, the application effect visualization program superimposes and displays the superimposed image on the live view video based on the changed superimposition range and superimposition rate, thereby allowing the user to visually confirm the effect of applying the cosmetic product in terms of information regarding the elapsed time of the skin condition that the user wants to know.

Images