Information processing system, information processing method, and program

Abstract

An information processing system disclosed herein includes a control unit. The control unit receives application information indicating the application state of a topical skin preparation to the body of a user, transfers an image indicating the application state on a two-dimensional or three-dimensional model of at least a part of the body of a person on the basis of the application information, and generates presentation information including the model to which the image indicating the application state was transferred, and transmits the presentation information to a user terminal of the user.

Description

Information processing system, information processing method and program

[0001] The present invention relates to an information processing system, an information processing method, and a program for providing information on the state of application of an external skin preparation to a user's body.

[0002] BACKGROUND ART Conventionally, there has been a technology for visualizing the application state of cosmetics such as ultraviolet protection preparations (sunscreens) using an ultraviolet camera (UV camera) to prevent uneven application or missed application.

[0003] For example, Patent Document 1 below discloses a system that shows areas on an image of a user's face captured by a UV camera where sunscreen is not applied sufficiently, and displays a message recommending that sunscreen be applied to the areas where sunscreen is not applied sufficiently.

[0004] Special Publication No. 2024-515589

[0005] However, the technology of Patent Document 1 has a problem in that the facial image of the user captured by the UV camera is used as is for display, making it difficult to easily grasp the state of application of the cosmetic.

[0006] In view of the above circumstances, an object of the present invention is to provide an information processing system, an information processing method, and a program that enable a user to easily grasp the application state of an external skin preparation.

[0007] An information processing system according to one aspect of the present invention includes a control unit that receives application information indicating an application state of a topical skin preparation on a user's body, transfers an image indicating the application state onto a two-dimensional or three-dimensional model that resembles at least a portion of a human body based on the application information, generates presentation information including the model onto which the image indicating the application state has been transferred, and transmits the presentation information to a user terminal of the user.

[0008] An information processing method according to another aspect of the present invention includes receiving application information indicating the application state of a topical skin preparation on a user's body; transferring an image indicating the application state onto a two-dimensional or three-dimensional model that resembles at least a part of a human body based on the application information; and generating presentation information including the model onto which the image indicating the application state has been transferred, and transmitting the presentation information to a user terminal of the user.

[0009] A program according to yet another aspect of the present invention causes an information processing device to perform the following steps: receiving application information indicating the application state of a topical skin preparation on a user's body; transferring an image indicating the application state onto a two-dimensional or three-dimensional model that resembles at least a part of a human body based on the application information; and generating presentation information including the model onto which the image indicating the application state has been transferred, and transmitting the presentation information to a user terminal of the user.

[0010] According to an information processing system according to one embodiment of the present invention, it is possible for a user to easily grasp the application state of an external skin preparation, but this effect is not intended to limit the present invention.

[0011] 1 is a diagram showing the configuration of an anti-UV formulation application support system according to one embodiment of the present invention. FIG. 2 is a diagram showing the hardware configuration of an anti-UV formulation application support server according to one embodiment of the present invention. FIG. 3 is a diagram showing the configuration of a database held by an anti-UV formulation application support server according to one embodiment of the present invention. FIG. 4 is a flowchart showing the flow of anti-UV formulation application support information provision processing by an anti-UV formulation application support server according to one embodiment of the present invention. FIG. 5 is a diagram showing an example of display of anti-UV formulation application support information provided by an anti-UV formulation application support server according to one embodiment of the present invention. FIG. 6 is a diagram showing an example of display of anti-UV formulation application support information provided by an anti-UV formulation application support server according to one embodiment of the present invention. FIG. 7 is a flowchart showing the flow of anti-UV formulation application support information provision processing by an anti-UV formulation application support server according to another embodiment of the present invention. FIG. 8 is a table showing another example of presentation of application support information based on the results of a medical interview by an anti-UV formulation application support server according to another embodiment of the present invention. FIG. 9 is a diagram showing an example of display of anti-UV formulation application support information provided by an anti-UV formulation application support server according to another embodiment of the present invention. FIG. 10 is a table showing a score and determination of the likelihood of uneven application used by an anti-UV formulation application support server according to another embodiment of the present invention.

[0023] FIG. 1 is a diagram showing an example of a plurality of UV camera images taken from different angles of the same user's face, used by a UV protection preparation application support server according to another embodiment of the present invention. FIG. 2 is a flowchart showing the operation flow in which a UV protection preparation application support server according to another embodiment of the present invention superimposes UV protection ability information on an avatar using a UV camera image synthesized from a plurality of UV camera images.

[0024] FIG. 3 is a diagram conceptually showing the operation flow in which a UV protection preparation application support server according to another embodiment of the present invention superimposes UV protection ability information on a three-dimensional model using a UV camera image synthesized from a plurality of UV camera images.

[0025] FIG. 4 is a diagram conceptually showing the operation flow in which a UV protection preparation application support server according to another embodiment of the present invention superimposes UV protection ability information on a user's face image using a UV camera image synthesized based on a plurality of UV camera images and from which the influence of reflected light has been removed.1 is a diagram conceptually illustrating the flow of another example of the operation of an UV protection preparation application support server according to another embodiment of the present invention, in which UV protection function information is superimposed on a user's face image using a UV camera image synthesized based on a plurality of UV camera images from which the influence of reflected light has been removed. FIG. 2 is a diagram conceptually illustrating the manner in which an UV protection preparation application support server according to another embodiment of the present invention creates a UV camera image by synthesizing a plurality of UV camera images of a user's hands.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, a UV protection preparation is taken as an example of an external preparation for skin.

[0013] [System Configuration] As shown in FIG. 1, this system includes a UV protection preparation application support server 100 on the Internet 50 and a plurality of user terminals 200.

[0014] The UV protection preparation application support server 100 (hereinafter also simply referred to as the server 100) is a server (information processing device) that can provide information to support the user in applying the ultraviolet (UV) protection preparation (such as advice information on the application state and application method) to the user terminal 200. The server 100 is connected to a plurality of user terminals 200 via a network such as the Internet 50.

[0015] The user terminals 200 (200A, 200B, 200C, etc.) are terminals used by users, such as smartphones, mobile phones, tablet PCs (personal computers), notebook PCs, desktop PCs, etc. When a user applies a UV protection formulation, the user terminal 200 accesses the server 100 via an application (application assistance app) installed on the user terminal 200, transmits an application assistance request together with application information (UV camera image) showing the application status of the UV protection formulation by the user, and receives and displays the application assistance information on the screen.

[0016] [Hardware Configuration of UV Protection Preparation Application Support Server] As shown in FIG. 2 , the server 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an input / output interface 15, and a bus 14 connecting these to each other.

[0017] The CPU 11 is a control unit that accesses the RAM 13 and the like as needed, and performs various arithmetic processing while controlling all of the blocks of the server 100. The ROM 12 is a non-volatile memory that permanently stores firmware such as the OS, programs, and various parameters to be executed by the CPU 11. The RAM 13 is used as a working area for the CPU 11, and temporarily stores the OS, various applications currently being executed, and various data currently being processed.

[0018] The input / output interface 15 is connected to a display unit 16, an operation reception unit 17, a storage unit 18, a communication unit 19, and the like.

[0019] The display unit 16 is a display device that uses, for example, an LCD (Liquid Crystal Display), an OELD (Organic ElectroLuminescence Display), a CRT (Cathode Ray Tube), or the like.

[0020] The operation reception unit 17 is, for example, a pointing device such as a mouse, a keyboard, a touch panel, or other input device. When the operation reception unit 17 is a touch panel, the touch panel can be integrated with the display unit 16.

[0021] The storage unit 18 is a non-volatile memory such as a hard disk drive (HDD), a flash memory (SSD; solid state drive), or other solid-state memory. The storage unit 18 stores the OS, various applications, and various data.

[0022] As will be described later, particularly in this embodiment, the memory unit 18 has a user information database, a coating information database, a 3D model information database, and a coating support information database, in addition to programs such as applications necessary for the coating support information provision processing described later.

[0023] The communication unit 19 is, for example, a NIC (Network Interface Card) for Ethernet or various modules for wireless communication such as a wireless LAN, and is responsible for communication processing with the user terminal 200 .

[0024] Although not shown, the basic hardware configuration of the user terminal 200 is substantially the same as that of the server 100. An application support app corresponding to the UV protection formulation application support service provided by the server 100 is installed in the storage unit of the user terminal 200, and a UV camera can be connected, and a corresponding application is also installed. The application allows the user to take a color image of themselves while viewing it, just like taking a normal selfie, and a UV camera image is acquired internally.

[0025] 3, the UV protection formulation application support server 100 has, in the storage unit 18, a user information database 31, an application information database 32, a 3D model information database 33, and an application support information database 34. Note that each of these databases may be stored in a storage device or server externally connected to the server 100, rather than in the storage unit 18.

[0026] The user information database 31 stores attribute information for each user who is a member of the support service provided by the UV protection preparation application support server 100. The user attribute information includes general information such as the user's name, a user ID for identifying the user, age (generation), occupation, address (area of ​​residence), gender, and email address, as well as information about the user's skin, such as the user's skin type, cosmetic preferences, and skin problems the user has.

[0027] The application information database 32 stores, for each user ID, application information indicating the state of application of a UV protection preparation to the user's face, received from the user terminal 200. In this embodiment, the application information is a UV camera image of the user's applied face taken with a UV camera (ultraviolet camera).

[0028] The 3D model information database 33 stores information about 3D models to which the UV protection function information converted from the application information is to be transferred. The 3D models are, for example, human-like avatars (whole body or upper body), and multiple types are stored so that the user can select one. The avatars may be generated based on user information (face image, profile information, etc.) when the application support information is requested from the user terminal 200 or in advance.

[0029] The application support information database 34 stores text data such as advice information corresponding to the UV protection capability information, image data used for display, etc. The advice information includes information about areas where the makeup has been missed or unevenly applied and the reasons for this, and methods for preventing missed applications and applying makeup evenly (such as precautions to take when applying makeup, the appropriate timing for reapplication, and the appropriate formulation). Examples of advice information include, but are not limited to, "XX (area) looks like it's about to get sunburned," "XX (area) is unevenly applied," "Has the mask rubbed it off?" and "XX (area) is safe." The application support information database 34 also stores, for each user ID, transmission history information of the application support information provided to the user terminal 200, including the 3D model after transferring the UV protection capability information and the advice information.

[0030] These databases are used by mutual reference as necessary in the UV protection preparation application support process by the UV protection preparation application support server 100 described later.

[0031] [Operation of UV Protection Preparation Application Support Server] Next, the operation of the server 100 configured as described above will be described. The operation is executed by the cooperation of hardware such as the CPU 11 and communication unit 19 of the server 100 and software stored in the storage unit 18. For convenience, in the following description, the CPU 11 is the subject of the operation.

[0032] FIG. 4 is a flowchart showing the flow of a process for providing UV protection preparation application support information by the UV protection preparation application support server.

[0033] As shown in the figure, the CPU 11 of the server 100 first determines whether or not it has received a UV camera image and an application support request from the user terminal 200 (step 41). Note that with a UV camera connected to the user terminal 200, the user can acquire a UV camera image by taking a photo using an interface (color image display) similar to a normal selfie, and then transmits the image together with the application support request via the application support app. At this time, the CPU 11 may also receive information about a 3D model selected by the user from a list of 3D models displayed on the application support app.

[0034] When it is determined that an application assistance request has been received from the user terminal 200 (Yes in step 41), the CPU 11 recognizes each part of the user's face from the UV camera image (step 42). Information on each recognized part is stored in the application information database 32.

[0035] Specifically, the CPU 11 recognizes facial feature points using, for example, facemesh from mediapipe, an open source program provided by Google LLC. However, other face detectors, such as libraries like OpenCV or Dlib, or models trained using deep learning techniques, may also be used to recognize facial feature points. The CPU 11 may also perform image preprocessing to facilitate the acquisition of facial feature points. Examples of preprocessing include brightness adjustment, contrast adjustment, gamma correction, brightness inversion, and edge processing.

[0036] Next, the CPU 11 acquires UV protection preparation application information for each recognized facial region (step 43). Specifically, the CPU 11 identifies the UV reflectance of each recognized facial region, performs coordinate transformation on the coordinates of each acquired feature point to deform the image (e.g., perform affine transformation), and creates a mesh-like UV camera-exploded image with uniform feature point coordinates. Facial regions include, for example, the forehead, temples, between the eyebrows, inner corners of the eyes, around the eyes, nose, under the nose, cheekbones, cheeks, chin, and face line. The higher the UV reflectance of a region, the less UV protection preparation is applied, and the lower the UV protection ability (the more susceptible to sunburn).

[0037] Next, the CPU 11 creates a new texture by superimposing the UV camera image having uniform feature point coordinates on the facial portion of the texture that serves as the base of the 3D model extracted from the 3D model database 33. The CPU 11 then inverts the brightness of the texture to convert the UV protection preparation application information (reflectance) for each part of the face into a texture that serves as UV protection capability information for each part of the face (display information indicating the susceptibility to sunburn) (step 44). That is, the CPU 11 generates UV protection capability information estimated from the UV reflectance identified from the UV protection preparation application information.

[0038] As a method for calculating the reflectance from a UV camera image, the method described above treats the brightness values ​​in the image as relative brightness values, but other known methods can also be used, such as dividing the brightness value of the area where the UV protection preparation has been applied by the brightness value of the area where it has not been applied, or calculating the reflectance by comparing it with the reflectance of a standard area whose reflectance is already known by simultaneously capturing an image of the area.

[0039] Next, the CPU 11 transfers the texture of the UV protection function information for each facial part onto the texture of the 3D model, converting it into protection function presentation information (step 45). The texture of the 3D model may be adjusted in advance so that the uniform feature point coordinates obtained by the coordinate transformation match the facial position of the 3D model, or the feature point coordinates of the UV camera-extracted image may not match the facial position of the 3D model texture in advance, and the UV camera-extracted image may be overlaid by adjusting its position and size so that it appropriately overlaps the facial part of the 3D model texture.

[0040] The CPU 11 then generates a 3D model using the protection function presentation information, adds advice information corresponding to the UV protection function information, and displays the model on the support application of the user terminal 200 as application support information (step 46).

[0041] The CPU 11 may also directly transfer the UV protection function information converted from the UV protection preparation application information based on the coordinates of the recognized feature points onto the texture of the 3D model without creating the UV camera expanded image and new texture.

[0042] By transferring UV protection information to a 3D model, it is possible to display areas that are lost in a 2D image, such as the sides of the nose. Furthermore, the 3D model allows transfer regardless of the user's orientation at the time of capture, and allows for changes in facial expression and posture.

[0043] FIG. 5 is a diagram showing an example of the display of the application support information.

[0044] As shown in the figure, the application support information includes the 3D model 51 as an avatar (face portion). The facial surface of the 3D model includes part identification information 52 that identifies parts (areas) of the face with low UV protection capability (easily sunburned), and protection capability identification information 53 that identifies the level of UV protection capability (sunburn risk) of the parts identified by the part identification information 52. The protection capability identification information 53 is displayed in three levels, for example, "low," "medium," and "high," and the part identification information 52 is filled in with a color corresponding to the level.

[0045] Also, for example, below the 3D model 51, advice information 54 is displayed that indicates areas where paint has been left out or where paint has been unevenly applied, the reasons for this, and the application method that should be adopted.

[0046] In the example shown in the figure, the area under the eyes and the bridge of the nose are identified as having a low risk of sunburn, the cheeks as having a medium risk of sunburn, and the tip of the nose as having a high risk of sunburn. Advice information is displayed suggesting that there is a risk of "W-shaped" sunburn, and that the tip of the nose, which is at particularly high risk of sunburn, may be rubbing against the mask.

[0047] The 3D model 51 may be configured so that the user can change the viewpoint of the application state by operating it with a finger or by automatically rotating it or by selecting a motion. This allows the user to see the application state of the UV protection preparation and the risk of sunburn in three dimensions in detail, and to become aware of their own application habits. The 3D model 51 may also be played as an animated video, with a background such as strong sunlight or the sea.

[0048] FIG. 6 is a diagram showing another example of display of the application support information.

[0049] In the example of Figure 5, as shown in the figure, areas with a high risk of sunburn (low UV protection ability) are displayed on a single 3D model 51, distinguished according to the risk level. However, as shown in the figure, the CPU 11 (application support app) may generate area identification information 52 and protection ability identification information 53 on the 3D model 51 for each area with a different application state, and switch between them in response to, for example, the user's scrolling operation.

[0050] In the example shown in the figure, it is possible to switch between displaying an area where the amount of application is small ((A) in the figure), an area where there is a lot of uneven application (the difference between the maximum and minimum amount of application) ((B) in the figure), an area where the amount of application is sufficient ((C) in the figure), and an area where there is a significant difference in the amount of application between the left and right sides of the face ((D) in the figure). Note that the contents corresponding to each screen of (A) to (D) in the figure may be displayed on a single 3D model 51 in a different display manner, for example, in different colors, in a manner similar to that shown in Figure 5.

[0051] After checking these screens, the user applies additional UV protection preparation to areas where the application is uneven or the amount of application is insufficient. After reapplication, the user may send an application assistance request to the server 100 for confirmation. Furthermore, the user may send an application assistance request to the server 100, for example, several hours after application, to confirm the sustained effect of the UV protection function of the UV protection preparation, and may reapply the UV protection preparation as appropriate according to the application assistance information displayed in response.

[0052] As described above, according to this embodiment, the UV protection preparation application support server 100 converts the reflectance of the UV camera image into sunburn susceptibility and transfers it to a 3D model for display, thereby allowing the user to easily understand the application status of the UV protection preparation.

[0053] [Modifications] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the gist of the present invention.

[0054] The UV protection function information in the above-described embodiment is not limited to information indicating the amount of topical skin preparation (UV protection agent) applied, but may also indicate UV protection effect, estimated UV transmittance, estimated SPF value, sunburn risk, etc.

[0055] In the above-described embodiment, the server 100 receives, as application information for the UV protection formulation, UV camera images from the user terminal 200. In addition to the UV camera images, the server 100 may also receive response information to a medical interview (questionnaire) for the user from the user terminal 200, and use the response information to generate application assistance information.

[0056] The medical interview information may include, but is not limited to, information about dominant hand (right, left, or ambidextrous), hairstyle (select from multiple hairstyle images), awareness of sunburn (select areas of the face where sunscreen is easily forgotten, areas that are known to come off easily, areas that are easily sunburned, areas where sunscreen has been applied but where sunburn has been experienced), application time of sunscreen, amount applied, application action, behavior after application (how to spend time), and application formulation.

[0057] In this case, the server 100 associates the answer information for the above questionnaire items with the uneven application pattern and stores it in the application information database 32, and as in the above embodiment, generates a 3D model 51 based on UV protection function information converted from UV protection preparation application information for each part based on UV camera images, and identifies the uneven application pattern corresponding to the answer information for the questionnaire items received from the user terminal 200, and generates and displays advice information 54 based on that.

[0058] The server 100 indicates areas on the 3D model 51 where uneven application has occurred using area identification information 52, and if, for example, it is determined that a gradation has occurred as UV protection function information, and the application action information in response to the medical interview includes information that the UV protection preparation is usually applied starting from the forehead, then it displays advice information 54 such as, "Are you applying the preparation starting from the forehead? The amount may have been less in the area where it was applied last."

[0059] Furthermore, if the UV protection function information identifies that the overall protection rate is low and the amount applied as a response to the medical interview is determined to be lower than the ideal value, advice information 54 such as "You may want to try applying a little more sunscreen" is displayed.

[0060] Furthermore, if the UV protection information identifies uneven application in multiple areas, and the response information to the medical interview includes information about the application action, such as that the UV protection preparation is usually spread from a specific area, advice information 54 such as "It would be a good idea to first put a little on the face (5 points) and then spread it over each area" or "When applying to one area, it would be a good idea to try to spread it over several times" is displayed.

[0061] Furthermore, if UV protection information obtained from UV camera images taken several hours after application identifies that the amount of sunscreen applied to the nose and cheeks is small, and if the response information to the medical interview includes information that the subject will wear a mask after application, advice information 54 such as "If you spend time wearing a mask, the sunscreen on your nose and cheeks will come off, so it would be a good idea to be careful and reapply it" will be displayed.

[0062] Furthermore, if UV protection function information obtained from UV camera images taken several hours after application identifies that the amount of sunscreen applied to the inner corners of the eyes is insufficient, and if the patient's response to the medical interview includes information about rubbing their eyes after application, advice information 54 such as "Did you rub your eyes because they were itchy? Not only your makeup but also your sunscreen may come off, so you may need to reapply." is displayed.

[0063] Furthermore, if UV protection information obtained from UV camera images taken several hours after application identifies that the amount of sunscreen applied to the sides of the face is small, and if the answer to the medical interview information indicates that the hairstyle is with the bangs down, advice information 54 such as "When you wear your hair down, you should be careful as the sunscreen may rub off." is displayed.

[0064] In addition, even if the UV camera image confirms that the amount of UV protection agent applied has not decreased in an area, if the answer information to the medical interview indicates that there is a possibility that the UV protection agent may subsequently come off, the server 100 may present the possibility of this as advice information 54.

[0065] 7 is a table showing another example of the presentation of application support information through a medical interview. When the answers shown in the center column of the figure are obtained in response to questions about how to apply sunscreen and how to manage bangs, as shown in the left column of the figure, the server 100 presents advice information 54 for preventing a decrease in the effectiveness of the sunscreen, as shown in the right column of the figure.

[0066] In this way, by utilizing the answers to the medical questionnaire, users can understand their application actions, habits that need to be corrected, and things they should be careful of in their daily lives after application, and can take appropriate measures.

[0067] In the above-described embodiment, the UV camera image is used as the application information, but it is also possible to generate the application support information without using the UV camera image.

[0068] For example, the server 100 may store correspondence information between the response information for each item in the above questionnaire information and UV protection function information (areas where uneven application or missed applications are likely to occur), and may estimate the corresponding areas where uneven application or missed applications are likely to occur from the response information received from the user terminal 200, and generate application support information based on the results.

[0069] In this case, the response information of the subject user to the medical questionnaire information and the application status of the subject user (results of missed applications or uneven application) are learned, and a learning model is generated that uses the response information as an explanatory variable to estimate the status of missed applications or uneven application, and the application assistance information is generated using the response information received from the user terminal 200 and the learning model.

[0070] The server 100 may also store correspondence between facial features (bones, skin color, partial redness of the skin, skin surface condition (scaling, scales, wrinkles, texture, pores, etc.)) and UV protection function information (areas prone to uneven application or missed applications), estimate the corresponding areas prone to uneven application or missed applications from the face image (color image) received from the user terminal 200, and generate application support information based on the results. Information for estimating the facial bone structure may include information such as the facial contour line and shading of the face image, the coordinates of feature points in the mesh-like expanded image, and their relative positional relationships.

[0071] Here, the skeleton includes macroscopic skin shapes such as the three-dimensional characteristics of the brow bone, information on the contours around the eyeballs, nasolabial folds, mid-cheek line, contours of the philtrum, cheekbones, etc. Furthermore, instead of or in addition to areas prone to uneven application or missed spots, the server 100 may identify and display areas less prone to uneven application or missed spots.

[0072] For example, the server 100 may classify the facial features into a plurality of face types in advance, store correspondence information between the face types and UV protection capability information, and generate application support information including UV protection capability information corresponding to the face type determined from the face image received from the user. The server 100 may also classify the face types based on the results of visual labeling by a dedicated person or feature scoring.

[0073] FIG. 8 is a flowchart showing the flow of the UV protection preparation application support information providing process by the server 100 in this case.

[0074] As shown in the figure, the CPU 11 of the server 100 first determines whether or not an application assistance request has been received together with a face image from the user terminal 200 (step 71).

[0075] When it is determined that an application assistance request has been received from the user terminal 200 (Yes in step 71), the CPU 11 extracts the facial features of the user from the face image (step 72).

[0076] Next, the CPU 11 identifies the user's face type based on the extracted facial features (step 73).

[0077] Next, the CPU 11 obtains estimated UV protection capability information for each part of the user's face based on the identified face type from the correspondence information (step 74).

[0078] Next, the CPU 11 transfers the texture of the UV protection function information for each part of the face obtained above onto the texture that serves as the base of the 3D model extracted from the 3D model database 33, thereby converting it into protection function presentation information (step 75).

[0079] The CPU 11 then generates a 3D model 51 using the above-mentioned protection function presentation information, adds advice information 54 corresponding to the above-mentioned UV protection function information, and displays it on the support application of the user terminal 200 as application support information (step 76).

[0080] FIG. 9 is a diagram showing an example of display of application support information obtained by the process of FIG.

[0081] As shown in Figures (A) and (B), areas of the user's face type that are likely to be left unpainted or painted unevenly are identified as area identification information 52 on the screen for each defensive ability identification information 53, and can be displayed by switching screens using the user's scrolling operation.

[0082] Areas prone to uneven application can be identified by a dedicated expert through visual inspection of the image test results. For example, these areas can be identified by focusing on the following points: - Information on the contours of the philtrum: Faces with large contours are more likely to have missed areas or uneven application. - Information on the wrinkles from under the eyes to the cheeks (mid-cheek line): Faces with pronounced wrinkles are more likely to have missed areas around this line. - Ratio of the length connecting the inner corners of both eyes to the overall width of the face: The smaller this ratio (the closer the eyes are), the more likely the area between the eyes is to be missed. - Height between the inner corners of the eyes and the forehead: Faces with a large difference will have more uneven application in this area. - Information on the contours between the eyelids and forehead: Faces with small contours often have the makeup applied to the area under the eyebrows.

[0083] FIG. 11 shows a table in which the likelihood of uneven coating is scored and determined based on the above criteria.

[0084] The scores from 1 to 5 in the figure indicate the results of visually observing each facial feature (whether there is an uneven philtrum, whether there is a mid-cheek line, whether the inner corners of the eyes are narrow, whether the brow bones are three-dimensional, whether there are unevenness around the eyeballs) based on the following criteria when observing the color face image again: 1 point Does not apply to that item 2 points Does not apply to that item at all 3 points Can't say either way 4 points Applies to that item roughly 5 points Applies to that item

[0085] For example, the server 100 may generate a learning model that learns the facial image of the subject user and the application state of the subject user (results of missed applications or uneven application), and directly estimates the UV protection ability (state of missed applications or uneven application) using the facial image as an explanatory variable, and generate the application assistance information using the facial image received from the user terminal 200 and the learning model.

[0086] In addition, the server 100 may estimate UV protection capability using information on responses to the above-mentioned medical interview information and information on UV protection preparations used by the user in addition to features extracted from the facial image, and generate application support information.

[0087] In such a case, if it is determined based on the characteristics of the user's facial image and the user's response information that there is a low possibility that the user will forget to apply the product or apply it unevenly, information indicating this may be displayed as application support information, as shown in the same figure (C).

[0088] Furthermore, UV protection capability varies depending on skin color, with darker skin generally providing higher UV protection. Therefore, the application support information may be displayed by identifying the user's skin color from a facial image and correcting the UV protection capability information acquired from the UV camera image accordingly (a user with darker skin may be displayed with a higher UV protection capability for the same UV reflectance (application state) than a user with lighter skin). Furthermore, for a user with reddish skin in certain areas, the UV protection capability for the reddish areas may be displayed as lower for the same UV reflectance (application state).

[0089] Furthermore, the server 100 may estimate UV protection capability using three pieces of application information: a UV camera image, information on responses to the medical questionnaire, and a normal face (skin) image; alternatively, it may estimate UV protection capability using two pieces of information: a UV camera image and a normal face (skin) image; alternatively, it may estimate UV protection capability using two pieces of information: a response to the medical questionnaire and a normal face (skin) image; or alternatively, it may estimate UV protection capability using only a normal face (skin) image.

[0090] In the above-described embodiment, the brightness indicating the UV reflectance of each part in the UV camera image is inverted and transferred to the 3D model for display, but the transferred 3D model and the UV camera image itself before transfer may be switched and displayed by user operation.

[0091] In the above-described embodiment, a UV protection formulation is described as the cosmetic product, but the cosmetic product is not limited to this, and may be any cosmetic product, such as skin care cosmetics (lotion, emulsion), makeup base, foundation, etc., that is applied to the entire face (or the entire face or other parts of the body) and that is desired to be applied evenly but that cannot actually be applied evenly (the application state is difficult to see with the user's naked eye). In this case, the server 100 may estimate the skin care effect, etc. of the cosmetic product, instead of or in addition to the UV protection ability described above, based on application information of the cosmetic product (at least one of a UV camera image, information on responses to medical questionnaire information, and a normal face (skin) image).

[0092] Furthermore, topical skin preparations other than cosmetics may also be targeted, such as pharmaceuticals such as white petrolatum, heparinoid preparations, and urea-containing preparations, or quasi-drugs such as medicated cosmetics, skin disinfectants, and insect repellents. In this case, the server 100 may estimate the effects of the pharmaceuticals or quasi-drugs, such as treatment and prevention of skin diseases, moisturizing, prevention of rough and chapped skin, prevention of acne, sterilization of the skin, and repelling of pests, based on the application information of the pharmaceuticals or quasi-drugs.

[0093] In the above-described embodiment, an infrared camera image of the user's face taken by an infrared camera may be used instead of the UV camera image. The infrared camera image can obtain the moisture distribution on the user's skin surface, and based on the infrared camera image, it is possible to identify areas where the amount of lotion applied is small and provide the user with application assistance information.

[0094] In the above-described embodiment, application assistance information is generated regarding the state of application of cosmetics to the user's face, but the target is not limited to the face, and application assistance information may be generated regarding the state of application to other parts of the body, such as the head (hair), neck, arms, legs, back, etc. In this case, the 3D model used includes not only the face but also other parts of the body.

[0095] In the above embodiment, an example was shown in which the state of application of cosmetics was transferred to a 3D model, but the state of application may also be transferred to a 2D model and presented. Furthermore, the 3D model and 2D model may not only directly represent a person, but also include anthropomorphized animal characters.

[0096] In the above-described embodiment, an example was shown in which the cosmetic application state obtained from one UV camera image was transferred to a 3D or 2D model, but it is also possible to obtain the cosmetic application state from multiple UV camera images obtained for the same imaging subject.

[0097] That is, when only one UV camera image is used, the image may become unclear due to the reflection of reflected light, which may hinder quantification of the application state. Therefore, the server 100 receives multiple ultraviolet camera images of the user's face to which cosmetics have been applied, each captured with an ultraviolet camera from different angles or each captured with a different light irradiation angle, generates application information indicating the application state of the cosmetics from which the influence of reflected light in the ultraviolet camera images has been removed, based on the multiple ultraviolet camera images, generates presentation information to be presented to the user based on the generated application information, and transmits the presentation information to the user terminal 200.

[0098] 12 shows an example of multiple UV camera images of the same user's face taken at different UV irradiation angles, where the multiple UV camera images may be individually captured still images or extracted from one or more videos.

[0099] As shown in the figure, it can be seen that reflected light R is reflected on the forehead, nose, cheeks, etc. in each UV camera image.

[0100] Therefore, in order to eliminate the influence of the reflected light, the server 100 identifies areas from the plurality of UV camera images that have brightness values ​​above a predetermined threshold, and generates UV protection function information by synthesizing the areas from the plurality of UV camera images excluding the identified areas.

[0101] Fig. 13 is a flowchart showing the process by which the server 100 superimposes UV protection function information onto a three-dimensional model using UV camera images that have been created based on multiple UV camera images and from which the effects of reflected light have been removed, and Fig. 14 is a diagram conceptually illustrating the process.

[0102] As shown in both figures, the CPU 11 of the server 100 first acquires multiple UV camera images of the user's face taken at different UV irradiation angles (step 81). Although five UV camera images of the same user are shown in Figures 12 and 13, the number of acquired UV camera images is not limited to this.

[0103] Next, the CPU 11 develops each UV camera image into a mesh (step 82).

[0104] Specifically, as described above, the CPU 11 acquires feature points from each UV camera image, identifies identical objects (parts) that differ only in UV irradiation conditions based on the feature points, and performs coordinate transformation on the coordinates of each feature point to deform the image (for example, perform affine transformation) to create a mesh-like UV camera expanded image with uniform feature point coordinates (aligned). Based on the feature points in each UV camera image, the CPU 11 identifies identical objects that differ only in UV irradiation conditions. There are rule-based and machine learning techniques for feature point extraction.

[0105] Rule-based techniques include edge detection (Sobel, Canny), corner detection (Harris, FAST), feature descriptors (SIFT, SURF), texture features (Gabor, GLCM), and shape features (Hu moments).Machine learning techniques typically include convolutional neural networks (CNNs), but also include models such as AlexNet and ResNet, and combinations of these with transfer learning and segmentation methods (U-Net, Mask R-CNN).

[0106] Next, the CPU 11 identifies, from each of the meshed UV camera images, portions having a brightness value equal to or greater than a predetermined threshold as reflected light, and removes the reflected light portions (step 83).

[0107] Next, the CPU 11 synthesizes the plurality of mesh-like exfoliated images from which the reflected light has been removed to create one exfoliated image (step 84).

[0108] That is, the CPU 11 aligns the multiple UV camera images from which reflected light has been removed to corresponding coordinates on the same mesh, and then synthesizes the images using, for example, statistical processing to generate a single image.

[0109] For the statistical processing, for example, the average value of the brightness of a plurality of UV camera images is used, but the minimum value of the brightness of the plurality of images or the median value of the brightness of the plurality of images may also be used.

[0110] Alternatively, instead of the statistical processing, a method based on machine learning may be used, in which the CPU 11 uses a model that has previously trained a large number of UV camera images to decompose the images into information on reflected light, shadow, and shape, and then selectively removes the reflected light information from the target UV camera images.

[0111] The CPU 11 then superimposes the synthesized UV camera image onto the texture referenced by the three-dimensional model (avatar), generates a three-dimensional model with the synthesized UV camera image superimposed, and displays it as ultraviolet protection function information (step 85).

[0112] Note that if the same location can be identified from multiple UV camera images and it is known which part of the 3D model (avatar) it corresponds to, it is not necessary to develop the image into a mesh.

[0113] The CPU 11 may also display the mesh-like image as it is as application support information without superimposing it on the texture of the three-dimensional model.

[0114] Furthermore, the CPU 11 may remove reflected light from the multiple UV camera images, and then, instead of generating a single composite image, obtain information on the coating condition (uneven coating, coating distribution) from each image, and based on that information, superimpose that information on the texture of the three-dimensional model.

[0115] After displaying the ultraviolet protection function information based on the multiple UV camera images, the CPU 11 may control the three-dimensional model so that the viewpoint of the application state of the ultraviolet protection function information can be changed in accordance with user operations on the three-dimensional model.

[0116] That is, the CPU 11 may prepare an image on which the UV protection function information is superimposed for each of the above irradiation angles, and may display the three-dimensional model by rotating it, for example, around a vertical axis (spine) according to the irradiation angle specified by the user's operation on the three-dimensional model.

[0117] Furthermore, as shown in FIG. 15, the CPU 11 may superimpose a UV camera image synthesized based on multiple UV camera images, from which the influence of reflected light has been removed, on a photograph of the user's face instead of a three-dimensional model (avatar).

[0118] That is, the CPU 11 may superimpose a mesh expanded image, which has been synthesized from a plurality of UV camera images of the user's face and from which reflected light has been removed, by deforming the mesh expanded image to fit the shape of the face in the user's face image.

[0119] In this case, as shown in FIG. 16, the CPU 11 may acquire feature points from each of the multiple UV camera images, and then synthesize them into a single UV camera image using the above-mentioned statistical processing or the like, without expanding them into a mesh, and align them using coordinate conversion based on the coordinates of the feature points in the synthesized UV camera image (top of the figure) and the coordinates of the feature points in a photograph of the user's face (corresponding to one of the imaging angles of the multiple UV camera images) (bottom of the figure), and then superimpose the synthesized UV camera image on the photograph of the face.

[0120] It should be noted that the object to be imaged in the process using the multiple UV camera images is not limited to the user's face, but may be, for example, the arms, hands, body, back of the body, or a set of face and neck.

[0121] 17 is a conceptual diagram showing how the server 100 creates a UV camera image by combining multiple UV camera images of a user's hand. Fig. 17(a) shows an image irradiated with UV light from the right, Fig. 17(b) shows an image irradiated with UV light from above, Fig. 17(c) shows an image irradiated with UV light from the left, and Fig. 17(d) shows an image obtained by combining these images by averaging their brightness.

[0122] As shown in the figure, the area where the topical skin preparation has been applied can be seen more clearly in the synthesized image (d) compared to any of the original images (a) to (c).

[0123] As described above, the server 100 can easily remove the influence of reflected light in a UV camera image capturing the application state of a topical skin preparation by combining multiple UV camera images.

[0124] Although only one server 100 is shown in the above embodiment, the processes executed by the server 100 may be distributed and executed by a plurality of servers. For example, the process of acquiring the application state of a UV protection agent from a UV camera image, the process of extracting features from a face image, and the process of transferring UV protection function information to a 3D model and displaying it may be executed by different servers.

[0125] Among the inventions described in the claims of this application, the invention described as an "information processing method" is one in which each step is automatically performed by at least one device such as a computer through information processing by software, and is not performed by a human using a device such as a computer. In other words, the "information processing method" is an information processing method using computer software, and is not a method in which a human operates a calculation tool called a computer.

[0126] 11: CPU 18: Storage unit 19: Communication unit 31: User body information database 32: Application information database 33: 3D model information database 34: Application support information database 51: 3D model 52: Body part identification information 53: Defense function identification information 54: Advice information 100: UV protection preparation application support server 200: User terminal R: Reflected light

Claims

1. An information processing system comprising a control unit that receives application information indicating the application status of a topical skin preparation on a user's body, transfers an image indicating the application status onto a two-dimensional or three-dimensional model that resembles at least a part of a human body based on the application information, and generates presentation information including the model onto which the image indicating the application status has been transferred, and transmits the presentation information to the user terminal of the user.

2. The information processing system according to claim 1, wherein the model is a two-dimensional or three-dimensional model that resembles at least a human face.

3. The information processing system according to claim 1, wherein the topical skin preparation is a cosmetic.

4. The information processing system according to claim 3, wherein the cosmetic is an ultraviolet protection preparation, and the control unit estimates the ultraviolet protection ability of the ultraviolet protection preparation for each part of the user's body based on the application state, and generates ultraviolet protection ability information indicating the estimated ultraviolet protection ability as the presented information.

5. The information processing system according to claim 4, wherein the control unit generates the ultraviolet protection function information as image information or video information that alerts the user to at least the areas where the ultraviolet protection function is low.

6. The information processing system according to claim 4, wherein the control unit generates the ultraviolet protection function information so that the viewpoint of the application state can be changed in response to the user's operation on the model.

7. The information processing system of claim 4, wherein the application information is ultraviolet image information of the user's body captured by an ultraviolet camera, and the control unit, based on the ultraviolet image information, identifies areas of the body with higher ultraviolet reflectance as areas with less application of the ultraviolet protection preparation, and generates information indicating that the ultraviolet protection ability of areas with less application is low as the ultraviolet protection ability information.

8. The information processing system according to claim 7, wherein the control unit identifies areas where the amount of application is large based on the ultraviolet image information, and generates information indicating that the ultraviolet protection ability of the areas where the amount of application is large is high as the ultraviolet protection ability information.

9. The information processing system according to claim 7 or 8, wherein the control unit identifies areas where the unevenness in the amount of application is large or areas where the difference in the amount of application is large based on the ultraviolet image information, and generates information indicating that the ultraviolet protection ability of the areas where the unevenness or difference is large is insufficient as the ultraviolet protection ability information.

10. The information processing system of claim 4, wherein the application information is a facial image of the user received from the user terminal, and the control unit extracts characteristic information relating to bone structure or skin color from the facial image, classifies the user into multiple facial types according to the characteristic information, estimates areas where uneven application or forgetting to apply the UV protection preparation is likely to occur based on the facial types, and generates information indicating that the UV protection ability of the areas is low as the UV protection ability information.

11. The information processing system of claim 4, wherein the control unit receives response information to medical questionnaire information regarding the application of the UV protection preparation from the user terminal, and generates, based on the response information, UV protection function information indicating that the UV protection function of the area in question is low.

12. The information processing system according to claim 4, wherein the control unit stores advice information indicating the application method or application formulation to be adopted depending on the application state, and transmits the advice information depending on the received application information together with the presentation information.

13. The information processing system of claim 1, wherein the control unit receives, as the application information, a plurality of ultraviolet images of specific parts of the user's body taken by an ultraviolet camera, and generates an image showing the application state based on the plurality of ultraviolet images.

14. The information processing system described in claim 13, wherein the plurality of ultraviolet images are each taken from a different angle or taken by changing the angle of light irradiation, and the control unit synthesizes the plurality of ultraviolet images to generate an image showing the coating state with the effects of reflected light in the ultraviolet images removed.

15. The information processing system described in claim 14, wherein the control unit identifies areas from the plurality of ultraviolet images each having a brightness value equal to or greater than a predetermined threshold, and synthesizes the areas of the plurality of ultraviolet images excluding the identified areas.

16. An information processing method comprising: receiving application information indicating the application status of a topical skin preparation on a user's body; transferring an image indicating the application status onto a two-dimensional or three-dimensional model that resembles at least a part of a human body based on the application information; generating presentation information including the model onto which the image indicating the application status has been transferred; and transmitting the presentation information to the user terminal of the user.

17. A program that causes an information processing device to execute the steps of: receiving application information indicating the application status of a topical skin preparation on a user's body; transferring an image indicating the application status onto a two-dimensional or three-dimensional model that resembles at least a part of a human body based on the application information; and generating presentation information including the model onto which the image indicating the application status has been transferred, and transmitting the information to the user terminal of the user.

Images