Information processing system, information processing method, and program

Abstract

To allow a user to easily grasp the application state of an external skin preparation.SOLUTION: An information processing system includes a control unit. The control unit receives application information indicating an application state of an external skin preparation on a user's body, transfers an image indicating the application state onto a two-dimensional or three-dimensional model that imitates at least a part of a human body on the basis of the application information, generates presentation information including the model onto which the image indicating the application state has been transferred, and transmits the presentation information to the user's user terminal.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program for providing information on the state of application of an external skin preparation to a user's body. [Background technology]

[0002] Conventionally, there has been technology that uses an ultraviolet camera (UV camera) to visualize the application state of cosmetics such as ultraviolet protection preparations (sunscreens) and prevent uneven application or missed applications.

[0003] For example, Patent Document 1 below discloses that an image of a user's face captured by a UV camera shows areas where sunscreen is not applied sufficiently, and displays a message recommending that sunscreen be applied to the areas where sunscreen is not applied sufficiently. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2024-515589 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology of Patent Document 1 has the problem that it is difficult to easily grasp the state of application of cosmetics because the image of the user's face captured by the UV camera is used for display as is.

[0006] In view of the above circumstances, an object of the present invention is to provide an information processing system, an information processing method, and a program that enable a user to easily grasp the application state of an external skin preparation. [Means for solving the problem]

[0007] An information processing system according to one aspect of the present invention includes a control unit that receives application information indicating an application state of a topical skin preparation on a user's body, transfers an image indicating the application state onto a two-dimensional or three-dimensional model that resembles at least a part of a human body based on the application information, generates presentation information including the model onto which the image indicating the application state has been transferred, and transmits the presentation information to a user terminal of the user.

[0008] An information processing method according to another aspect of the present invention includes: receiving application information indicating the application status of the topical skin preparation on the user's body; Based on the application information, an image showing the application state is transferred onto a two-dimensional or three-dimensional model that resembles at least a part of a human body; and generating presentation information including the model onto which the image showing the application state has been transferred, and transmitting the information to the user terminal of the user.

[0009] According to yet another aspect of the present invention, there is provided a program for executing the program on an information processing device, receiving application information indicating the application status of the topical skin preparation to the user's body; a step of transferring an image showing the application state onto a two-dimensional or three-dimensional model that resembles at least a part of a human body based on the application information; generating presentation information including the model onto which the image showing the application state has been transferred, and transmitting the information to the user terminal of the user; [Effects of the Invention]

[0010] According to an information processing system according to one embodiment of the present invention, it is possible for a user to easily grasp the application state of an external skin preparation, but this effect is not intended to limit the present invention. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram showing the configuration of a UV protection preparation application support system according to one embodiment of the present invention. [Figure 2]1 is a diagram showing the hardware configuration of a UV protection preparation application support server according to one embodiment of the present invention. [Figure 3] 1 is a diagram showing the configuration of a database possessed by a UV protection preparation application support server according to one embodiment of the present invention. [Figure 4] 10 is a flowchart showing the flow of a UV protection preparation application support information providing process performed by a UV protection preparation application support server according to one embodiment of the present invention. [Figure 5] 10 is a diagram showing an example of display of UV protection preparation application support information provided by a UV protection preparation application support server according to one embodiment of the present invention. FIG. [Figure 6] 10 is a diagram showing an example of display of UV protection preparation application support information provided by a UV protection preparation application support server according to one embodiment of the present invention. FIG. [Figure 7] 10 is a flowchart showing the flow of a UV protection preparation application support information providing process performed by a UV protection preparation application support server according to another embodiment of the present invention. [Figure 8] 10 is a table showing another example of application support information presented by a UV protection preparation application support server according to another embodiment of the present invention based on the results of a medical interview. [Figure 9] 10 is a diagram showing an example of display of UV protection preparation application support information provided by a UV protection preparation application support server according to another embodiment of the present invention. FIG. [Figure 10] 10 shows a table used by a UV protection agent application support server according to another embodiment of the present invention, which scores and determines the likelihood of uneven application. [Figure 11] 10A and 10B are diagrams showing examples of multiple UV camera images taken from different angles of the same user's face, used by a UV protection preparation application support server according to another embodiment of the present invention. [Figure 12] 10 is a flowchart showing the operation flow of a UV protection preparation application support server according to another embodiment of the present invention, in which UV protection function information is superimposed on an avatar using a UV camera image synthesized from multiple UV camera images. [Figure 13]FIG. 10 is a diagram conceptually illustrating the flow of operations performed by a UV protection preparation application support server according to another embodiment of the present invention, in which UV protection function information is superimposed on a three-dimensional model using a UV camera image synthesized from multiple UV camera images. [Figure 14] FIG. 10 is a conceptual diagram illustrating the flow of operations performed by a UV protection preparation application support server according to another embodiment of the present invention, in which UV protection function information is superimposed on a user's face image using a UV camera image synthesized based on multiple UV camera images and from which the influence of reflected light has been removed. [Figure 15] FIG. 10 is a conceptual diagram illustrating the flow of another example of the operation of a UV protection preparation application support server according to another embodiment of the present invention, in which UV protection function information is superimposed on a user's face image using a UV camera image that has been synthesized based on multiple UV camera images and from which the influence of reflected light has been removed. [Figure 16] A conceptual diagram showing how a UV protection preparation application support server according to another embodiment of the present invention creates a UV camera image by combining multiple UV camera images of a user's hands. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, a UV protection preparation is taken as an example of an external preparation for skin.

[0013] [System Configuration] As shown in FIG. 1, this system includes a UV protection preparation application support server 100 on the Internet 50 and a plurality of user terminals 200.

[0014] The UV protection preparation application support server 100 (hereinafter also simply referred to as the server 100) is a server (information processing device) that can provide information to support the user in applying the ultraviolet (UV) protection preparation (such as advice information on the application state and application method) to the user terminal 200. The server 100 is connected to a plurality of user terminals 200 via a network such as the Internet 50.

[0015] The user terminals 200 (200A, 200B, 200C, etc.) are terminals used by users, such as smartphones, mobile phones, tablet PCs (Personal Computers), notebook PCs, desktop PCs, etc. When a user applies a UV protection formulation, the user terminal 200 accesses the server 100 via an application (application assistance app) installed on the user terminal 200, transmits an application assistance request together with application information (UV camera image) showing the application status of the UV protection formulation by the user, and receives and displays the application assistance information on the screen.

[0016] [Hardware configuration of UV protection formulation application support server] As shown in FIG. 2, the server 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an input / output interface 15, and a bus 14 connecting these components to one another.

[0017] The CPU 11 is a control unit that accesses the RAM 13 and other memory as needed, and performs various arithmetic processing while controlling all of the blocks in the server 100. The ROM 12 is a non-volatile memory that permanently stores firmware such as the OS, programs, and various parameters to be executed by the CPU 11. The RAM 13 is used as a working area for the CPU 11, and temporarily stores the OS, various applications currently being executed, and various data currently being processed.

[0018] The input / output interface 15 is connected to a display unit 16, an operation reception unit 17, a storage unit 18, a communication unit 19, and the like.

[0019] The display unit 16 is a display device that uses, for example, an LCD (Liquid Crystal Display), an OLED (Organic ElectroLuminescence Display), a CRT (Cathode Ray Tube), or the like.

[0020] The operation reception unit 17 is, for example, a pointing device such as a mouse, a keyboard, a touch panel, or other input device. When the operation reception unit 17 is a touch panel, the touch panel can be integrated with the display unit 16.

[0021] The storage unit 18 is a non-volatile memory such as a hard disk drive (HDD), a flash memory (solid state drive (SSD)), or other solid-state memory. The storage unit 18 stores the OS, various applications, and various data.

[0022] As will be described later, particularly in this embodiment, the memory unit 18 has a user information database, a coating information database, a 3D model information database, and a coating support information database, in addition to programs such as applications necessary for the coating support information provision processing described later.

[0023] The communication unit 19 is, for example, a NIC (Network Interface Card) for Ethernet or various modules for wireless communication such as wireless LAN, and is responsible for communication processing with the user terminal 200.

[0024] Although not shown, the basic hardware configuration of the user terminal 200 is substantially the same as that of the server 100. An application support app corresponding to the UV protection formulation application support service provided by the server 100 is installed in the storage unit of the user terminal 200, and a UV camera can be connected, and a corresponding application is also installed. The application allows the user to take a color image of themselves while viewing it, just like taking a normal selfie, and a UV camera image is acquired internally.

[0025] [Database configuration of UV protection formulation application support server] 3, the UV protection formulation application support server 100 has, in a storage unit 18, a user information database 31, an application information database 32, a 3D model information database 33, and an application support information database 34. Note that each of these databases may be stored in a storage device or server externally connected to the server 100, rather than in the storage unit 18.

[0026] The user information database 31 stores attribute information for each user who is a member of the support service provided by the UV protection preparation application support server 100. The user attribute information includes general information such as the user's name, a user ID for identifying the user, age (generation), occupation, address (area of ​​residence), gender, and email address, as well as information about the user's skin, such as the user's skin type, cosmetic preferences, and skin problems the user has.

[0027] The application information database 32 stores, for each user ID, application information indicating the state of application of a UV protection preparation to the user's face, received from the user terminal 200. In this embodiment, the application information is a UV camera image of the user's applied face taken with a UV camera (ultraviolet camera).

[0028] The 3D model information database 33 stores information about 3D models to which the UV protection function information converted from the application information is to be transferred. The 3D models are, for example, human-like avatars (whole body or upper body), and multiple types are stored so that the user can select one. The avatars may be generated based on user information (face image, profile information, etc.) when the application support information is requested from the user terminal 200 or in advance.

[0029] The application support information database 34 stores text data such as advice information corresponding to the UV protection performance information, image data used for display, etc. The advice information includes information about areas where the makeup has been omitted or unevenly applied, the reasons for this, and methods for eliminating omissions and applying makeup evenly (such as precautions to take when applying makeup, the appropriate timing for reapplication, and the appropriate formulation). Examples of advice information include, but are not limited to, "XX (area) looks like it's about to get sunburned," "XX (area) is unevenly applied," "Has the mask rubbed it off?", and "XX (area) is safe." The application support information database 34 also stores, for each user ID, transmission history information for application support information provided to the user terminal 200, including the 3D model after transferring the UV protection performance information and advice information.

[0030] These databases are used by mutual reference as necessary in the UV protection preparation application support process by the UV protection preparation application support server 100, which will be described later.

[0031] [Operation of UV protection agent application support server] Next, a description will be given of the operation of the server 100 configured as described above. The operation is executed by the cooperation of hardware such as the CPU 11 and communication unit 19 of the server 100 and software stored in the storage unit 18. For convenience, in the following description, the CPU 11 is the subject of the operation.

[0032] FIG. 4 is a flowchart showing the flow of a process for providing UV protection preparation application support information by the UV protection preparation application support server.

[0033] As shown in the figure, the CPU 11 of the server 100 first determines whether or not an application support request has been received together with a UV camera image from the user terminal 200 (step 41). Note that the user can acquire a UV camera image by taking a picture using an interface (color image display) similar to a normal selfie with a UV camera connected to the user terminal 200, and transmits the image together with the application support request via the application support app. At this time, the CPU 11 may also receive information about a 3D model selected by the user from a list of 3D models displayed on the application support app.

[0034] When it is determined that an application assistance request has been received from the user terminal 200 (Yes in step 41), the CPU 11 recognizes each part of the user's face from the UV camera image (step 42). Information on each recognized part is stored in the application information database 32.

[0035] Specifically, the CPU 11 recognizes facial feature points using, for example, facemesh from mediapipe, an open source program provided by Google LLC. However, other face detectors, such as libraries like OpenCV or Dlib, or models trained using deep learning techniques, may also be used to recognize facial feature points. The CPU 11 may also perform image preprocessing to facilitate the acquisition of facial feature points. Examples of preprocessing include brightness adjustment, contrast adjustment, gamma correction, brightness inversion, and edge processing.

[0036] Next, CPU 11 acquires UV protection agent application information for each recognized facial region (step 43). Specifically, CPU 11 identifies the UV reflectance for each recognized facial region, performs coordinate transformation on the coordinates of each acquired feature point to deform the image (e.g., perform affine transformation), and creates a mesh-like UV camera-deployed image with uniform feature point coordinates. Facial regions include, for example, the forehead, temples, between the eyebrows, inner corners of the eyes, around the eyes, nose, under the nose, cheekbones, cheeks, chin, and face line. The higher the UV reflectance of a region, the less UV protection agent is applied, and the lower the UV protection ability (the more susceptible to sunburn).

[0037] Next, the CPU 11 creates a new texture by superimposing the UV camera image with uniform feature point coordinates on the facial portion of the texture that serves as the base of the 3D model extracted from the 3D model database 33. The CPU 11 then inverts the brightness of the texture to convert the UV protection preparation application information (reflectance) for each part of the face into a texture that serves as UV protection ability information for each part of the face (display information indicating susceptibility to sunburn) (step 44). In other words, the CPU 11 generates UV protection ability information estimated from the UV reflectance identified from the UV protection preparation application information.

[0038] The above method for calculating reflectance from a UV camera image shows a method in which the brightness values ​​in the image are treated as relative brightness values, but this is not limited to this. Other known methods can also be used, such as dividing the brightness value of the area where the UV protection agent has been applied by the brightness value of the area where it has not been applied, or taking an image of a standard area whose reflectance is already known and calculating the reflectance by comparing it with the reflectance of the standard area.

[0039] Next, CPU 11 transfers the texture of the UV protection function information for each facial part onto the texture of the 3D model to convert it into protection function presentation information (step 45). The texture of the 3D model may be adjusted in advance so that the uniform feature point coordinates obtained by the coordinate transformation match the facial position of the 3D model, or the feature point coordinates of the UV camera-extracted image and the facial position of the 3D model texture may not be matched in advance, and the UV camera-extracted image may be superimposed by adjusting its position and size so that it appropriately overlaps the facial part of the 3D model texture.

[0040] The CPU 11 then generates a 3D model using the above-mentioned protection function presentation information, adds advice information corresponding to the above-mentioned UV protection function information, and displays the model on the support application of the user terminal 200 as application support information (step 46).

[0041] The CPU 11 may directly transfer the UV protection function information converted from the UV protection preparation application information based on the coordinates of the recognized feature points onto the texture of the 3D model without creating the UV camera-extracted image and new texture.

[0042] By transferring UV protection information to a 3D model, it is possible to display areas that are lost in 2D images, such as the sides of the nose. Furthermore, the 3D model allows transfer regardless of the user's orientation at the time of capture, and allows for changes to facial expression and posture.

[0043] FIG. 5 is a diagram showing a display example of the application support information.

[0044] As shown in the figure, the application support information has the above-mentioned 3D model 51 as an avatar (face portion). The facial surface of the 3D model has part identification information 52 that identifies parts (areas) of the face that have low UV protection capability (areas that are prone to sunburn), and protection capability identification information 53 that identifies the level of UV protection capability (risk of sunburn) of the parts identified by the part identification information 52. The protection capability identification information 53 is shown in three levels, for example, "low," "medium," and "high," and the part identification information 52 is filled in with a color corresponding to the level.

[0045] In addition, for example, at the bottom of the 3D model 51, advice information 54 is displayed showing areas where paint has been left out or where the paint has been unevenly applied, the reasons for this, and the application method that should be adopted.

[0046] In the example shown in the figure, the area under the eyes and the bridge of the nose are identified as having a low risk of sunburn, the cheeks as having a medium risk of sunburn, and the tip of the nose as having a high risk of sunburn.Advice information is displayed suggesting that there is a risk of "W-shaped" sunburn, and that the tip of the nose, which is particularly at risk of sunburn, may be rubbing against the mask.

[0047] The 3D model 51 may be configured so that the user can change the viewpoint of the application state by operating it with a finger or by automatically rotating it or by selecting a motion. This allows the user to see the application state of the UV protection preparation and the risk of sunburn in three dimensions in detail, and to understand their own application habits. The 3D model 51 may also be played as an animated video, with a background such as strong sunlight or the sea.

[0048] FIG. 6 is a diagram showing another example of display of the application support information.

[0049] In the example of Figure 5, as shown in the figure, areas with a high risk of sunburn (low UV protection ability) are displayed on a single 3D model 51, distinguished according to the level of risk. However, as shown in the figure, the CPU 11 (application support app) may generate area identification information 52 and protection ability identification information 53 on the 3D model 51 for each area with a different application state, and switch between them for display in response to, for example, the user's scrolling operation.

[0050] In the example shown in the figure, it is possible to switch between displaying an area where the amount of application is small ((A) of the figure), an area where there is a lot of uneven application (the difference between the maximum and minimum amount of application) ((B) of the figure), an area where the amount of application is sufficient ((C) of the figure), and an area where there is a significant difference in the amount of application between the left and right sides of the face ((D) of the figure). Note that the content corresponding to each screen of (A) to (D) of the figure may be displayed on a single 3D model 51 in a different display manner, for example, in different colors, in a manner similar to that shown in Figure 5.

[0051] After checking these screens, the user applies additional UV protection preparation to areas where the application is uneven or insufficient. After reapplication, the user may send an application assistance request to server 100 for confirmation. Furthermore, the user may send an application assistance request to server 100, for example, several hours after application, to confirm the sustained effect of the UV protection function of the UV protection preparation, and may reapply the UV protection preparation as appropriate according to the application assistance information displayed in response.

[0052] As described above, according to this embodiment, the UV protection preparation application support server 100 converts the reflectance of the UV camera image into sunburn susceptibility and transfers it to a 3D model for display, thereby allowing the user to easily understand the application status of the UV protection preparation.

[0053] [Variations] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and it goes without saying that various modifications can be made within the scope of the gist of the present invention.

[0054] The UV protection function information in the above-described embodiment is not limited to information indicating the amount of topical skin preparation (UV protection agent) applied, but may also indicate UV protection effect, estimated UV transmittance, estimated SPF value, sunburn risk, etc.

[0055] In the above-described embodiment, the server 100 receives UV camera images as application information for the UV protection formulation from the user terminal 200. In addition to the UV camera images, the server 100 may also receive response information to a medical interview (questionnaire) for the user from the user terminal 200, and use the response information to generate application assistance information.

[0056] The medical interview information may include, but is not limited to, information about dominant hand (right-handed, left-handed, ambidextrous), hairstyle (select from multiple hairstyle images), awareness of sunburn (select areas of the face where sunscreen is easily forgotten, areas that are known to come off easily, areas that are easily sunburned, areas where sunscreen has been applied but where sunburn has occurred), application time of sunscreen, amount applied, application action, behavior after application (how to spend time), and application formulation.

[0057] In this case, the server 100 associates the answer information of the above questionnaire items with the uneven application pattern and stores them in the application information database 32, and as in the above embodiment, generates a 3D model 51 based on UV protection function information converted from UV protection preparation application information for each part based on UV camera images, and identifies the uneven application pattern corresponding to the answer information of the questionnaire items received from the user terminal 200, and generates and displays advice information 54 based on that.

[0058] The server 100 indicates areas on the 3D model 51 where uneven application has occurred using area identification information 52, and if, for example, it identifies that a gradation has occurred as UV protection function information, and if the application action information in response to the medical interview includes information that the UV protection preparation is usually applied starting from the forehead, it displays advice information 54 such as, "Are you applying the preparation starting from the forehead? The amount may have been less in the area where it was applied last."

[0059] Furthermore, if the UV protection function information identifies that the overall protection rate is low and the amount applied as a response to the medical interview is determined to be lower than the ideal value, advice information 54 such as "You may want to try applying a little more sunscreen" is displayed.

[0060] Furthermore, if uneven application of the UV protection agent to multiple areas is identified as information on UV protection performance, and if the application action information in response to the medical interview indicates that the UV protection agent is usually spread from a specific area, advice information 54 such as "It would be a good idea to first apply a small amount to the face (5 points) and then spread it over each area" or "When applying to one area, it would be a good idea to try to spread it over several times" is displayed.

[0061] Furthermore, if UV protection information obtained from UV camera images taken several hours after application identifies that the amount of sunscreen applied to the nose and cheeks is small, and if the response information to the medical interview includes information that the person will wear a mask after application, advice information 54 such as "If you spend time wearing a mask, the sunscreen on your nose and cheeks will come off, so it's a good idea to be careful and reapply it" will be displayed.

[0062] Furthermore, if UV protection performance information obtained from a UV camera image taken several hours after application identifies that the amount of sunscreen applied to the inner corner of the eye is insufficient, and if the patient's response to the medical interview indicates that the patient rubbed their eyes after application, advice information 54 such as "Did you rub your eyes because they were itchy? Not only your makeup but also your sunscreen may come off, so you may need to reapply the sunscreen" is displayed.

[0063] Furthermore, if UV protection performance information obtained from UV camera images taken several hours after application identifies that the amount of sunscreen applied to the sides of the face is small, and if the answer to the medical interview information indicates that the person has their bangs down, advice information 54 such as "When you wear your hair down, you should be careful as the sunscreen may rub off." is displayed.

[0064] In addition, even if the UV camera image confirms that the amount of UV protection agent applied has not decreased in an area, if there is an item that is judged to have the possibility of the UV protection agent coming off in the future based on the response information to the medical interview, the server 100 may present the possibility of this as advice information 54.

[0065] 7 is a table showing another example of the presentation of application support information through a medical interview. When the answers shown in the center column of the figure are obtained in response to questions about how to apply sunscreen and how to manage bangs, as shown in the left column of the figure, the server 100 presents advice information 54 for preventing a decrease in the effectiveness of the sunscreen, as shown in the right column of the figure.

[0066] In this way, by utilizing the answers to the medical questionnaire, users can understand their application actions, habits that need to be corrected, and things they should be careful of in their daily lives after application, and can take appropriate measures.

[0067] In the above-described embodiment, the UV camera image is used as the application information, but it is also possible to generate the application support information without using the UV camera image.

[0068] For example, the server 100 may store correspondence information between the response information for each item in the above medical questionnaire and UV protection ability information (areas where uneven application or missed applications are likely to occur), and may estimate the corresponding areas where uneven application or missed applications are likely to occur from the response information received from the user terminal 200, and generate application support information based on the results.

[0069] In this case, the response information of the subject user to the medical questionnaire information and the application status of the subject user (results of missed applications or uneven application) are learned, and a learning model is generated that uses the response information as an explanatory variable to estimate the status of missed applications or uneven application, and the application support information is generated using the response information received from the user terminal 200 and the learning model.

[0070] The server 100 may also store correspondence between facial features (skeleton, skin color, partial redness of the skin, skin surface condition (scaling, scales, wrinkles, texture, pores, etc.)) and UV protection function information (areas prone to uneven application or missed applications), estimate the corresponding areas prone to uneven application or missed applications from the face image (color image) received from the user terminal 200, and generate application support information based on the results. Information for estimating facial skeleton may include information such as the facial contour line and shading of the face image, the coordinates of feature points in the mesh-like expanded image, and their relative positional relationships.

[0071] Here, the skeleton includes macro skin shapes such as the three-dimensional characteristics of the brow bone, information on the unevenness around the eyeballs, nasolabial folds, mid-cheek line, unevenness of the philtrum, cheekbones, etc. Furthermore, instead of or in addition to areas prone to uneven application or missed applications, the server 100 may identify and display areas less prone to uneven application or missed applications.

[0072] For example, the server 100 may classify the facial features into multiple face types in advance, store correspondence information between the face types and UV protection ability information, and generate application support information including UV protection ability information corresponding to the face type determined from the face image received from the user. The server 100 may also classify the face types based on the results of visual labeling by a dedicated person or feature scoring.

[0073] FIG. 8 is a flowchart showing the flow of the UV protection preparation application support information providing process by the server 100 in this case.

[0074] As shown in the figure, the CPU 11 of the server 100 first determines whether or not an application assistance request has been received together with a face image from the user terminal 200 (step 71).

[0075] If it is determined that an application assistance request has been received from the user terminal 200 (Yes in step 71), the CPU 11 extracts the facial features of the user from the face image (step 72).

[0076] Next, the CPU 11 identifies the user's face type based on the extracted facial features (step 73).

[0077] Next, the CPU 11 obtains estimated UV protection capability information for each part of the user's face based on the identified face type from the correspondence information (step 74).

[0078] Next, the CPU 11 transfers the texture of the UV protection function information for each part of the face obtained above onto a texture that serves as the base of the 3D model extracted from the 3D model database 33, thereby converting it into protection function presentation information (step 75).

[0079] The CPU 11 then generates a 3D model 51 using the above-mentioned protection function presentation information, adds advice information 54 corresponding to the above-mentioned UV protection function information, and displays it on the support application of the user terminal 200 as application support information (step 76).

[0080] FIG. 9 is a diagram showing an example of display of application support information obtained by the process of FIG.

[0081] As shown in Figures (A) and (B), areas of the user's face that are prone to missing or uneven application are identified as area identification information 52 on the screen for each defensive ability identification information 53, and can be displayed by switching screens using the user's scroll operation.

[0082] Areas prone to uneven coating can be identified by a dedicated inspector visually based on the image test results. For example, areas prone to uneven coating can be identified by focusing on the following points: - Information on the contours of the philtrum: Faces with large contours are prone to missing areas or uneven application. - Information on wrinkles from under the eyes to the cheeks (mid-cheek line): If your face has pronounced wrinkles, it's easy to miss application of the product in that area. The ratio of the length between the inner corners of both eyes to the overall width of the face: The smaller this ratio (the closer the eyes are to each other), the more likely it is that the area between the eyes will be left unpainted. · Between the inner corners of the eyes and the height of the forehead: For faces with a large difference, the application around these areas will be uneven. - Information on the contours between the eyelids and forehead: For faces with small contours, the product can often be applied to the area under the eyebrows.

[0083] FIG. 11 shows a table in which the likelihood of uneven coating is scored and determined based on the above criteria.

[0084] The scores of 1 to 5 in the figure indicate the results of visually scoring each item of each facial feature (whether there is an uneven philtrum, whether there is a mid-cheek line, whether the inner corners of the eyes are narrow, whether the brow bones are three-dimensional, whether there are unevenness around the eyeballs) when observing the color face image again, based on the following criteria. 1 point: Does not apply to that item 2 points: Hardly applicable to the item 3 points Neither 4 points: Roughly corresponds to the item 5 points Applicable to that item

[0085] Note that the classification process into the face types is not essential. For example, the server 100 may generate a learning model that learns the face image of the subject user and the state of application of the subject user (results of missed applications or uneven application), and directly estimates the UV protection ability (state of missed applications or uneven application) using the face image as an explanatory variable, and generate the application assistance information using the face image received from the user terminal 200 and the learning model.

[0086] Furthermore, the server 100 may estimate UV protection capability using information on responses to the above-mentioned medical interview information and information on UV protection preparations used by the user in addition to the features extracted from the face image, and generate application support information.

[0087] In such a case, if it is determined based on the characteristics of the user's facial image and the user's response information that there is a low possibility that the user will forget to apply the product or apply it unevenly, information indicating this may be displayed as application support information, as shown in the same figure (C).

[0088] Furthermore, UV protection capability varies depending on skin color, with darker skin generally providing higher UV protection capability. Therefore, the skin color may be identified from the user's facial image, and the UV protection capability information obtained from the UV camera image may be corrected accordingly to display application support information (a higher UV protection capability may be displayed for a user with darker skin than for a user with lighter skin, given the same UV reflectance (application state)). Furthermore, for users with reddish skin in certain areas, the UV protection capability for those reddish areas may be displayed as lower given the same UV reflectance (application state).

[0089] Furthermore, the server 100 may estimate UV protection capability using three pieces of application information: a UV camera image, information on responses to medical questionnaire information, and a normal face (skin) image; alternatively, it may estimate UV protection capability using two pieces of information: a UV camera image and a normal face (skin) image; alternatively, it may estimate UV protection capability using two pieces of information: information on responses to medical questionnaire information and a normal face (skin) image; or alternatively, it may estimate UV protection capability using only a normal face (skin) image.

[0090] In the above-described embodiment, the brightness indicating the UV reflectance of each part in the UV camera image was inverted and transferred to the 3D model for display, but the transferred 3D model and the UV camera image itself before transfer may be switched and displayed by user operation.

[0091] In the above-described embodiment, a UV protection formulation is described as the cosmetic, but the cosmetic is not limited to this, and may be any cosmetic that is applied to the entire face (or the entire other area), such as skin care cosmetics (lotion, emulsion), makeup base, foundation, etc., and that is desired to be applied evenly but that cannot actually be applied evenly (the application state is difficult to see with the user's naked eye). In this case, the server 100 may estimate the skin care effect, etc. of the cosmetic, instead of or in addition to the above-described UV protection ability, based on application information of the cosmetic (at least one of a UV camera image, information on responses to medical questionnaire information, and a normal face (skin) image).

[0092] Furthermore, topical skin preparations other than cosmetics may also be targeted, such as pharmaceuticals such as white petrolatum, heparinoid preparations, and urea-containing preparations, or quasi-drugs such as medicated cosmetics, skin disinfectants, and insect repellents. In this case, the server 100 may estimate the effects of the pharmaceuticals or quasi-drugs, such as treatment and prevention of skin diseases, moisturizing, prevention of rough and chapped skin, prevention of acne, sterilization of the skin, and repelling pests, based on the application information of the pharmaceuticals or quasi-drugs.

[0093] In the above-described embodiment, an infrared camera image of the user's face taken by an infrared camera may be used instead of a UV camera image. The infrared camera image can obtain the moisture distribution on the user's skin surface, and based on the infrared camera image, it is possible to identify areas where the amount of lotion applied is small and provide the user with application assistance information.

[0094] In the above-described embodiment, application support information is generated regarding the state of application of cosmetics to the user's face, but the target is not limited to the face, and application support information may be generated regarding the state of application to other parts of the body, such as the head (hair), neck, arms, legs, back, etc. In this case, the 3D model used is one that depicts not only the face but also other parts.

[0095] In the above embodiment, an example was shown in which the state of application of cosmetics was transferred to a 3D model, but the state of application may also be transferred to a 2D model and presented. Furthermore, the 3D model and 2D model may not only directly represent a person, but also include anthropomorphized animal characters.

[0096] In the above-described embodiment, an example was shown in which the cosmetic application state obtained from one UV camera image was transferred to a 3D or 2D model, but it is also possible to obtain the cosmetic application state from multiple UV camera images obtained of the same imaging subject.

[0097] That is, when only one UV camera image is used, the reflected light may interfere with the image, making it unclear and hindering quantification of the application state. Therefore, server 100 receives multiple ultraviolet camera images of the user's face to which cosmetics have been applied, each captured with an ultraviolet camera from different angles or each captured with different light irradiation angles, generates application information indicating the application state of the cosmetics from which the effects of reflected light in the ultraviolet camera images have been removed, based on the multiple ultraviolet camera images, generates presentation information for the user based on the generated application information, and transmits the presentation information to user terminal 200.

[0098] 12 shows an example of multiple UV camera images of the same user's face taken at different UV irradiation angles. The multiple UV camera images may be separately captured still images or may be extracted from one or more videos.

[0099] As shown in the figure, reflected light R can be seen on the forehead, nose, cheeks, etc. in each UV camera image.

[0100] Therefore, in order to eliminate the influence of the reflected light, the server 100 identifies areas from the plurality of UV camera images that have brightness values ​​above a predetermined threshold, and generates UV protection function information by synthesizing the areas from the plurality of UV camera images excluding the identified areas.

[0101] Fig. 13 is a flowchart showing the process by which server 100 superimposes UV protection function information onto a 3D model using UV camera images that have been created based on multiple UV camera images and from which the effects of reflected light have been removed, and Fig. 14 is a diagram conceptually illustrating the process.

[0102] As shown in both figures, the CPU 11 of the server 100 first acquires multiple UV camera images of the user's face taken at different UV irradiation angles (step 81). Although five UV camera images of the same user are shown in Figures 12 and 13, the number of acquired UV camera images is not limited to this.

[0103] Next, the CPU 11 develops each UV camera image into a mesh (step 82).

[0104] Specifically, as described above, the CPU 11 acquires feature points from each UV camera image, and based on the feature points, identifies identical objects (parts) that differ only in UV irradiation conditions. It then transforms the image (for example, performs an affine transformation) by performing coordinate transformation on the coordinates of each feature point, and creates a mesh-like UV camera unfolded image with uniform (aligned) feature point coordinates. Based on the feature points in each UV camera image, it identifies identical objects that differ only in UV irradiation conditions. There are rule-based and machine learning techniques for feature point extraction.

[0105] Rule-based techniques include edge detection (Sobel, Canny), corner detection (Harris, FAST), feature descriptors (SIFT, SURF), texture features (Gabor, GLCM), and shape features (Hu moments).Machine learning techniques typically include convolutional neural networks (CNNs), but also include models such as AlexNet and ResNet, and combinations of these with transfer learning and segmentation methods (U-Net, Mask R-CNN).

[0106] Next, the CPU 11 identifies, from each of the meshed UV camera images, portions having a brightness value equal to or greater than a predetermined threshold, as reflected light, and removes the reflected light portions (step 83).

[0107] Next, the CPU 11 synthesizes the plurality of mesh-like exfoliated images from which the reflected light has been removed to create one exfoliated image (step 84).

[0108] That is, the CPU 11 aligns the multiple UV camera images from which reflected light has been removed to corresponding coordinates on the same mesh, and then synthesizes the images by, for example, statistical processing to generate a single image.

[0109] For the statistical processing, for example, the average value of the brightness of a plurality of UV camera images is used, but the minimum value of the brightness of the plurality of images or the median value of the brightness of the plurality of images may also be used.

[0110] Alternatively, instead of the statistical processing, a method based on machine learning may be used, in which the CPU 11 uses a model that has previously trained a large number of UV camera images to decompose the images into information on reflected light, shadow, and shape, and then selectively removes the reflected light information from the target UV camera images.

[0111] Then, the CPU 11 superimposes the synthesized UV camera image on the texture referenced by the three-dimensional model (avatar), generates a three-dimensional model with the synthesized UV camera image superimposed, and displays it as ultraviolet protection function information (step 85).

[0112] Furthermore, if the same location can be identified from multiple UV camera images and it is known which part of the 3D model (avatar) it corresponds to, it is not necessary to expand the image into a mesh.

[0113] Furthermore, the CPU 11 may display the image developed into the mesh form as it is as application support information without superimposing it on the texture of the three-dimensional model.

[0114] Furthermore, the CPU 11 may remove reflected light from the multiple UV camera images, and then, instead of generating a single composite image, obtain information on the coating condition (uneven coating, coating distribution) from each image, and based on that information, superimpose that information on the texture of the three-dimensional model.

[0115] After displaying the ultraviolet protection function information based on the plurality of UV camera images, the CPU 11 may control the three-dimensional model so that the viewpoint of the ultraviolet protection function information can be changed in response to a user's operation on the three-dimensional model.

[0116] That is, the CPU 11 may prepare an image on which the UV protection function information is superimposed for each of the above irradiation angles, and may display the three-dimensional model by rotating it, for example, around a vertical axis (spine) according to the irradiation angle specified by the user's operation on the three-dimensional model.

[0117] Furthermore, as shown in FIG. 15, the CPU 11 may superimpose a UV camera image synthesized based on a plurality of UV camera images, from which the influence of reflected light has been removed, on a photograph of the user's face, instead of on a three-dimensional model (avatar).

[0118] That is, the CPU 11 may deform and superimpose the mesh expanded image, which is synthesized from the plurality of UV camera images of the user's face and from which reflected light has been removed, in accordance with the shape of the face in the user's face image.

[0119] In this case, as shown in FIG. 16, the CPU 11 may acquire feature points from each of the multiple UV camera images, and then synthesize them into a single UV camera image using the above-mentioned statistical processing or the like, without expanding them into a mesh, and align them using coordinate transformation based on the coordinates of the feature points in the synthesized UV camera image (top of the figure) and the coordinates of the feature points in a photograph of the user's face (corresponding to one of the imaging angles of the multiple UV camera images) (bottom of the figure), and then superimpose the synthesized UV camera image on the photograph of the face.

[0120] It should be noted that the object to be imaged in the process using the plurality of UV camera images is not limited to the user's face, but may be, for example, the arms, hands, body, back of the body, or a set of face and neck.

[0121] 17 is a conceptual diagram showing how the server 100 creates a UV camera image by combining multiple UV camera images of a user's hand. Figure 17(a) shows an image irradiated with UV light from the right, Figure 17(b) shows an image irradiated with UV light from above, Figure 17(c) shows an image irradiated with UV light from the left, and Figure 17(d) shows an image obtained by combining these images by averaging their brightness.

[0122] As shown in the figure, the area where the topical skin preparation has been applied can be seen more clearly in the synthesized image (d) compared to any of the original images (a) to (c).

[0123] As described above, the server 100 can easily remove the influence of reflected light in a UV camera image capturing the application state of an external skin preparation by combining a plurality of UV camera images.

[0124] Although only one server 100 is shown in the above embodiment, the processes executed by the server 100 may be distributed and executed by multiple servers. For example, the process of acquiring the application status of a UV protection agent from a UV camera image and the process of extracting features from a facial image may be executed by separate servers from the process of transferring UV protection function information to a 3D model and displaying it.

[0125] Among the inventions described in the claims of this application, the invention described as an "information processing method" is one in which each step is automatically performed by at least one device such as a computer through software-based information processing, and is not performed by a human using a device such as a computer. In other words, the "information processing method" is an information processing method using computer software, and is not a method in which a human operates a computing tool called a computer. [Explanation of symbols]

[0126] 11...CPU 18...Storage section 19…Communications Department 31...User physical information database 32...Application information database 33...3D model information database 34...Application support information database 51...3D model 52...Part identification information 53…Defense capability identification information 54...Advice information 100…UV protection agent application support server 200...User terminal R…Reflected light

Claims

1. receiving application information indicating the application status of the topical skin preparation on the user's body; Based on the application information, an image showing the application state is transferred onto a two-dimensional or three-dimensional model that resembles at least a part of a human body; generating presentation information including the model onto which the image showing the application state has been transferred, and transmitting the information to a user terminal of the user; Control Unit An information processing system comprising:

2. The model is a two-dimensional or three-dimensional model that resembles at least a human face. The information processing system according to claim 1 .

3. The topical skin preparation is a cosmetic. The information processing system according to claim 1 .

4. The cosmetic is an ultraviolet protection preparation, The control unit estimates the ultraviolet protection ability of the ultraviolet protection preparation for each part of the user's body based on the application state, and generates ultraviolet protection ability information indicating the estimated ultraviolet protection ability as the presented information. The information processing system according to claim 3 .

5. The control unit generates the ultraviolet protection capability information as image information or video information that warns the user of at least the areas with low ultraviolet protection capability. The information processing system according to claim 4 .

6. The control unit generates the ultraviolet protection function information so that a viewpoint of the application state can be changed in response to an operation of the user on the model. The information processing system according to claim 4 .

7. The application information is ultraviolet image information of the user's body captured by an ultraviolet camera, The control unit identifies, based on the ultraviolet image information, a part of the body with a higher ultraviolet reflectance as a part where the amount of the ultraviolet protection preparation applied is smaller, and generates, as the ultraviolet protection ability information, information indicating that the ultraviolet protection ability of the part where the amount of the application is smaller is lower. The information processing system according to claim 4 .

8. The control unit identifies the area where the amount of application is large based on the ultraviolet image information, and generates information indicating that the ultraviolet protection ability of the area where the amount of application is large is high as the ultraviolet protection ability information. The information processing system according to claim 7 .

9. The control unit identifies areas where the unevenness in the amount of application is large or areas where the difference in the amount of application is large based on the ultraviolet image information, and generates information indicating that the ultraviolet protection ability of the areas where the unevenness or difference is large is insufficient as the ultraviolet protection ability information.

9. The information processing system according to claim 7 or 8.

10. the application information is a facial image of the user received from the user terminal, The control unit extracts feature information related to bone structure or skin color from the face image, classifies the user into a plurality of face types according to the feature information, estimates areas where uneven application or missed application of the UV protection preparation is likely to occur based on the face types, and generates information indicating that the UV protection ability of the areas is low as the UV protection ability information. The information processing system according to claim 4 .

11. The control unit receives, from the user terminal, response information to medical interview information regarding the application of the ultraviolet protection preparation, and generates, based on the response information, information indicating that the ultraviolet protection ability of the area is low as the ultraviolet protection ability information. The information processing system according to claim 4 .

12. The control unit stores advice information indicating an application method or application formulation to be adopted depending on the application state, and transmits the advice information depending on the received application information together with the presented information. The information processing system according to claim 4 .

13. The control unit receiving, as the application information, a plurality of ultraviolet images of a specific part of the user's body taken by an ultraviolet camera; An image showing the coating state is generated based on the plurality of ultraviolet images. The information processing system according to claim 1 .

14. The plurality of ultraviolet images are captured from different angles or captured by changing the light irradiation angle, The control unit combines the plurality of ultraviolet images to generate an image showing the coating state from which the influence of reflected light in the ultraviolet images has been removed. The information processing system according to claim 13.

15. The control unit identifies regions from the plurality of ultraviolet images each having a brightness value equal to or greater than a predetermined threshold, and synthesizes regions from the plurality of ultraviolet images excluding the identified regions. The information processing system according to claim 14.

16. receiving application information indicating the application status of the topical skin preparation on the user's body; Based on the application information, an image showing the application state is transferred onto a two-dimensional or three-dimensional model that resembles at least a part of a human body; generating presentation information including the model onto which the image showing the application state has been transferred, and transmitting the information to the user terminal of the user; Information processing methods.

17. In the information processing device, receiving application information indicating the application status of the topical skin preparation to the user's body; a step of transferring an image showing the application state onto a two-dimensional or three-dimensional model that resembles at least a part of a human body based on the application information; generating presentation information including the model onto which the image showing the application state has been transferred, and transmitting the information to a user terminal of the user; A program that executes the following.

Images