Information processing device, information processing method, and information processing program

Abstract

An information processing device according to the present invention comprises an acquisition unit, a conversion unit, an inference unit, and a generation unit. The acquisition unit acquires an object image that is an image of a first object and also acquires, as a reference image, an image of a second object that is in a target state for the first object. On the basis of one of the object image and the reference image, the conversion unit converts the other of the object image and the reference image to an image of the first object that reflects the target state. On the basis of the object image and the converted image obtained by the conversion, the inference unit infers a processing procedure for converting the first object from a prescribed state to the target state. On the basis of the object image, the generation unit generates, as an output image to be outputted to a user, an image of the first object in which the state of the first object has been changed in accordance with the processing procedure.

Description

Information processing device, information processing method, and information processing program

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program.

[0002] BACKGROUND ART Techniques are known for presenting information about facial surface texture (for example, the state of adhesion of a coating film on skin to which cosmetics have been applied) to a user.

[0003] Japanese Patent Application Laid-Open No. 2022-24565

[0004] However, the above-mentioned conventional techniques do not necessarily provide adequate support to a user in changing a specific state to a target state. For example, the above-mentioned conventional techniques merely present changes in facial surface texture or facial shape to make it easier to understand, and do not take into consideration the ability to appropriately guide a user in a specific state to a target state that the user is aiming for.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and an information processing program that can appropriately support a user in changing a specific state to a target state.

[0006] According to the present disclosure, an information processing device includes an acquisition unit that acquires an object image that is an image of a first object and a reference image related to a second object different from the first object; a conversion unit that generates a transformed image of the first object based on the object image and the reference image; an estimation unit that estimates a processing procedure for changing the appearance of the first object to one based on the second object based on the transformed image and the object image; and a generation unit that generates an image of the first object whose appearance has been changed in accordance with the processing procedure as an output image based on the object image.

[0007] As described above, according to the present disclosure, it is possible to appropriately support a user in changing a specific state to a target state. Note that the effects described herein are not necessarily limited to those described herein, and may be any of the effects described in the present disclosure.

[0008] 1 is a diagram illustrating an overall picture of information processing according to an embodiment. FIG. 2 is a diagram illustrating an example of a system according to an embodiment. FIG. 3 is a diagram illustrating an example of the configuration of a server device according to an embodiment. FIG. 4 is a diagram illustrating an example of the configuration of a learning device according to an embodiment. FIG. 5 is an explanatory diagram illustrating a preprocessing method in a learning phase. FIG. 6 is an explanatory diagram illustrating a learning method in a learning phase. FIG. 7 is a diagram (1) illustrating an example of an estimation method realized in information processing according to an embodiment. FIG. 8 is a diagram (2) illustrating an example of an estimation method realized in information processing according to an embodiment. FIG. 9 is a diagram illustrating an example of the configuration of a server device according to a first modification. FIG. 10 is a flowchart illustrating an estimation processing procedure according to the first modification. FIG. 11 is a flowchart illustrating a malfunction detection processing procedure according to the first modification. FIG. 12 is a block diagram illustrating an example of the hardware configuration of a computer corresponding to an information processing device according to an embodiment.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] One or more embodiments (including examples, modifications, and application examples) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from one another. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects.

[0011] The present disclosure will be described in the following order: 1. Introduction 2. Overview 3. Overview of the Proposed Technology 4. System Configuration 5. Configuration of the Server Device 6. Configuration of the Learning Device 7. Method for Estimating Work Procedures 7-1. Information Processing Method (1) 7-2. Information Processing Method (2) 8. Modification 1 8-1. Configuration of the Server Device 8-2. Processing Procedure (1) 8-3. Processing Procedure (2) 9. Modification 2 10. Other 11. Hardware Configuration 12. Summary

[0012] [1. Introduction] The proposed technology of the present disclosure may be suitably applied to assisting with tasks that require multiple processing steps. The proposed technology of the present disclosure assists a user in achieving a desired appearance. For example, the proposed technology of the present disclosure may be suitably applied to situations such as assisting with facial makeup tasks to achieve a desired makeup look, assisting with hair and makeup tasks to achieve a desired hairstyle, and assisting with painting tasks to achieve a desired paint look. In the following embodiments, information processing when the proposed technology of the present disclosure is applied to assisting with facial makeup tasks will be described, but similar information processing also applies to other situations.

[0013] [2. Overview] For example, even if you have a photo of your face with the makeup you want to achieve, it can be difficult to accurately grasp the steps you should take to achieve that look. Therefore, there is a need for a system that shows you multiple steps to achieve the desired makeup look.

[0014] For example, when applying makeup, even if there is a facial photograph to use as reference for the makeup, the shape of the face and parts, i.e., the facial shape, the base color of the face, etc. may differ from that of the makeup model in the facial photograph, and the finished product may differ in many cases. Therefore, there is a need for makeup instructions that take into account one's own facial shape, base color of the face, and the lighting environment during makeup application.

[0015] Therefore, in order to solve the above-mentioned problems, the present disclosure proposes a system that uses a 3D Morphable Model (3DMM) that can express facial shape, facial base color, and even various lighting environments by adjusting lighting using parameters, thereby supporting a user in approaching a target makeup state even if there are differences in facial shape, facial base color, or lighting environment between the user and a makeup model that represents the user's desired makeup state.

[0016] Specifically, the proposed technology of the present disclosure realizes the following information processing: (i) A facial image of a makeup model in a target makeup state (target appearance) is converted to estimate the appearance of the makeup model in a live photo. (ii) The appearance is converted so that the facial shape and other aspects of the makeup model are closer to that of the user, while preserving the texture information of the makeup model. (iii) Using the user's face before makeup and the appearance of the makeup model, the makeup application procedure between the two is inferred by a machine learning model obtained through learning.

[0017] Here, the makeup workflow refers to multiple steps leading up to the completion of the makeup, with the target makeup state defined as the completed makeup stage. Instructions for the makeup content are automatically generated for each step. A short video showing how to apply the makeup and tips is also generated for each step. An image is also generated for each step, reflecting the user's appearance at that step in a 3D (three-dimensional) facial model.

[0018] In addition, with the proposed technology of the present disclosure, it is not necessary to automatically generate instructions for makeup content, generate a short video showing how to apply makeup and tips, and generate a 3D face model that reflects the appearance; it is sufficient to perform at least one of these.

[0019] [3. Overview of Proposed Technology] First, information processing (information according to an embodiment) related to the proposed technology of the present disclosure will be described using FIG. 1 . FIG. 1 is a diagram illustrating an overview of information processing according to an embodiment. FIG. 1 conceptually illustrates the above-described contents (i) to (iii). FIG. 1 also illustrates a scene in which a person P1 (user) in a natural state, i.e., in an unmade-up state, requests presentation of a work procedure for achieving a target makeup state by using a facial image of a person Px (makeup model) in a target makeup state as a reference image. In response to this request, a server device 100, which is an example of an information processing device, estimates and presents a work procedure using information processing according to an embodiment.

[0020] 1 , a person P1 (user) uses the user device 10 to input a face image IM1 (an example of an object image) of his or her own face (an example of a first object) before makeup (appearance before makeup) to the server device 100 (step S1). In this manner, the face image IM1 shows the appearance of the face of the person P1 before makeup. Note that the face image IM1 may be, for example, a single still image captured by the imaging function of the user device 10, or may be a moving image.

[0021] The person P1 also inputs a facial image IM2 of a person (an example of a second object) with the makeup state (an example of a target appearance) that the person P1 is aiming for, i.e., the face of the makeup model Px, as a reference image (reference image) to the server device 100 (step S2). In this way, the facial image IM2 shows the facial appearance of the makeup model Px with the makeup state that the person P1 is aiming for. The facial image IM2 may be, for example, an image obtained through a web search or a facial image of another person photographed by the person P1. The makeup model Px may be a celebrity or actress favored by the person P1. The makeup model Px may also be a close relative of the person P1 (e.g., family, friends, etc.), or the person P1 himself or herself when he or she has created makeup that he or she likes. The facial image IM2 may be a single still image or a moving image.

[0022] When the server device 100 receives the facial images IM1 and IM2, it uses a makeup procedure estimation model M (machine learning model) to estimate an operation procedure (processing procedure) for changing the pre-makeup state to a target makeup state (step S3). For example, the server device 100 inputs a 3D face model, which is a three-dimensional image generated by applying the facial image IM1 to the 3DMM, and a 3D face model, which is a three-dimensional image generated by applying the facial image IM2 to the 3DMM, into the makeup procedure estimation model M, and estimates the operation procedure based on the output result.

[0023] Each 3D face model is adjusted so that the facial surface texture other than the makeup texture is consistent between the person P1 and the makeup model Px. For example, the server device 100 adjusts other conditions other than the makeup texture to match between the person P1 and the makeup model Px so that only the texture difference before and after makeup can be accurately compared. Here, the other conditions other than the makeup texture may include the lighting environment in the space (e.g., the shooting space) where the face image IM1 was obtained.

[0024] According to the example shown in FIG. 1 , the server device 100 also performs a process of converting one of the two 3D face models based on the other. For example, the server device 100 converts the facial shape of the 3D face model generated based on face image IM2 to match the facial shape of the 3D face model generated based on face image IM1. As a result of the conversion process, the server device 100 can obtain a 3D face model in an unmade-up state having the skin base color of person P1 and a 3D face model in a target makeup state having the facial shape of person P1. Note that the 3D face models obtained here have consistent conditions other than makeup as a result of the adjustment process described above. Therefore, the server device 100 inputs the 3D models after the adjustment and conversion processes into the makeup procedure estimation model M.

[0025] 1 shows an example in which the server device 100 estimates a 10-step work procedure (some steps omitted) based on information output from the makeup procedure estimation model M. Specifically, the example shows the server device 100 estimating the following: a first step "putting on colored contact lenses," a second step "applying makeup base," a third step "applying foundation and eye shadow," an eighth step "drawing eyebrows," a ninth step "putting on a wig," and a tenth step "applying lips."

[0026] 1, the server device 100 generates output information to be output (presented) to the person P1 based on the estimation result (step S4). Specifically, the server device 100 generates an instruction statement SM to be presented to the person P1 and a 3D face model FM that reflects, as the work result, how the appearance of the person P1 will change if the work procedure is actually performed.

[0027] 1 , the server device 100 generates an instruction sentence SM1 instructing the first step based on the first step, "Put on colored contact lenses." The server device 100 also generates a 3D face model FM1 that reflects the changes in appearance that will occur in the facial appearance of person P1 when performing the first step, based on the 3D face model generated based on the facial image IM1. In other words, the server device 100 processes the 3D face model before makeup, changing the appearance of person P1 to one in which colored contact lenses are worn.

[0028] The server device 100 generates an instruction sentence SM2 instructing the second step based on the second step, "apply makeup base." The server device 100 also generates a 3D face model FM2 that further reflects the changes in appearance that will occur in the face of the person P1 when the work in the second step is performed, from the 3D face model FM1 that reflects the appearance changes up to the first step. Specifically, the server device 100 further processes the 3D face model FM1 that reflects the colored contact lens state, thereby changing the appearance of the person P1 to one that shows that the person P1 is wearing makeup base.

[0029] The server device 100 generates an instruction statement SM3 instructing the third procedure based on the third procedure "foundation + eye shadow." The server device 100 also generates a 3D face model FM3 that further reflects the changes in appearance that will occur in the facial appearance of the person P1 when the work in the third procedure is performed, from the 3D face model FM2 that reflects the appearance changes up to the second procedure. Specifically, the server device 100 further processes the 3D face model FM2 in a state where a makeup base has been applied, thereby changing the appearance of the person P1 to a state where they are wearing foundation and eye shadow.

[0030] The server device 100 generates an instruction sentence SM8 instructing the eighth step based on the eighth step, "Draw eyebrows." The server device 100 also generates a 3D face model FM8 that further reflects the changes in appearance that will occur in the facial appearance of the person P1 when the work in the eighth step is performed, from the 3D face model FM7 (not shown) that reflects the appearance changes up to the seventh step. Specifically, the server device 100 further processes the 3D face model FM7 to change the appearance of the person P1 to one in which eyebrows are drawn.

[0031] The server device 100 generates an instruction sentence SM9 instructing the ninth step based on the ninth step, "Put on a wig." The server device 100 also generates a 3D face model FM9 that further reflects the changes in appearance that will occur in the facial appearance of the person P1 when the work in the ninth step is performed, from the 3D face model FM8 that reflects the appearance changes up to the eighth step. Specifically, the server device 100 further processes the 3D face model FM8 with drawn eyebrows, thereby changing the appearance of the person P1 to one wearing a wig.

[0032] The server device 100 generates an instruction statement SM10 instructing the tenth step based on the tenth step, "lips." The server device 100 also generates a 3D face model FM10 that further reflects the changes in appearance that will occur in the facial appearance of the person P1 when the tenth step is performed, on the 3D face model FM9 that reflects the appearance changes up to the ninth step. Specifically, the server device 100 further processes the 3D face model FM9 in a state where the person P1 is wearing a wig, thereby changing the appearance of the person P1 to a state where the person P1 is wearing lips.

[0033] Next, the server device 100 controls output so that the output information generated in step S4 is output to the user device 10 of the person P1 (step S5). For example, the server device 100 may output an instruction statement and a 3D face model FM associated with each other for each work procedure. For example, the server device 100 outputs an instruction statement SM1 and a 3D face model FM1 associated with each other. Furthermore, the server device 100 outputs an instruction statement SM2 and a 3D face model FM2 associated with each other. Furthermore, the server device 100 outputs an instruction statement SM3 and a 3D face model FM3 associated with each other. The same applies to other combinations of instruction statements SM and 3D face models FM. The person P1 can view the 3D face model FM from various viewpoints using the user device 10. For example, the person P1 can freely rotate the 3D face model FM using the user device 10.

[0034] 2 is a diagram showing an example of a system according to an embodiment. Fig. 1 shows a system 1 as an example of a system according to an embodiment. Information processing according to an embodiment is realized in the system 1.

[0035] 2, system 1 includes user device 10, learning device 30, and server device 100. User device 10, learning device 30, and server device 100 are communicably connected via network N via wired or wireless communication. The number of user devices 10, learning devices 30, and server devices 100 included in system 1 is not limited.

[0036] The user device 10 is an information processing terminal used by a person who wishes to receive guidance for changing the appearance of a first object to an appearance based on a second object different from the first object, i.e., a target appearance. For example, the user device 10 may be a smartphone, a wearable device, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), etc. An application for transmitting and receiving information to and from the server device 100 may be installed in the user device 10.

[0037] The learning device 30 learns an estimation model for estimating a processing procedure for changing the appearance of the first object to a target appearance. The estimation model may be learned using various known machine learning techniques, as appropriate. For example, the estimation model may be learned using a supervised machine learning technique such as a support vector machine (SVM). The estimation model may also be learned using an unsupervised machine learning technique. The estimation model may also be learned using a deep learning technique. The estimation model may also be learned using various deep learning techniques, as appropriate, such as a deep neural network (DNN), a recurrent neural network (RNN), or a convolutional neural network (CNN).

[0038] The server device 100 is a cloud computer that plays a central role in performing information processing according to the embodiment. For example, the server device 100 inputs image information (e.g., an original image and a reference image) acquired via the user device 10 into a machine learning model generated by the learning device 30, and estimates a processing procedure for changing the state represented by the original image to a target state represented by the reference image based on the output information of the model.

[0039] 5. Configuration of the Server Device The server device 100 according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example configuration of the server device 100 according to the embodiment. As shown in Fig. 3, the server device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0040] (Communication Unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC), etc. For example, the communication unit 110 is connected to the network N via a wired or wireless connection, and transmits and receives information between the user device 10 and the learning device 30.

[0041] (Storage Unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 has a model data storage unit 121, an image data storage unit 122, and an estimation procedure data storage unit 123.

[0042] The model data storage unit 121 stores data of an estimation model for estimating a processing procedure for changing a predetermined state of a first object to a target state.

[0043] The image data storage unit 122 stores various image data used in the information processing according to the embodiment.

[0044] The estimation procedure data storage unit 123 stores a processing procedure for changing a predetermined state of the first object to a target state, and data generated based on the processing procedure.

[0045] (Control Unit 130) The control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like using RAM as a work area to execute various programs (e.g., the information processing program according to the embodiment) stored in a storage device inside the server device 100. The control unit 130 is also realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0046] As shown in Fig. 3, the control unit 130 has an image acquisition unit 131, an adjustment unit 132, a conversion unit 133, an estimation unit 134, a generation unit 135, and an output control unit 136, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Fig. 3, and may have other configurations as long as they perform the information processing described below. Furthermore, the connection relationships between the processing units of the control unit 130 are not limited to the connection relationships shown in Fig. 3, and may be other connection relationships.

[0047] (Image Acquisition Unit 131) The image acquisition unit 131 acquires an image input by a user as an image to be used for estimating a processing procedure. For example, the image acquisition unit 131 acquires an object image, which is an image of a first object in a specific appearance state. The image acquisition unit 131 also acquires an image of a second object having an appearance that is targeted by the first object as a reference image. The image acquisition unit 131 also stores the object image and the reference image in the image data storage unit 122.

[0048] (Adjustment unit 132) The adjustment unit 132 adjusts the object image and the reference image to match other conditions other than the appearance of the first object indicated by the object image and the appearance of the target indicated by the reference image. For example, the adjustment unit 132 removes information about the lighting environment estimated based on the object image from the object image and removes information about the lighting environment estimated based on the reference image from the reference image, thereby removing the lighting environment conditions between the object image and the reference image. Furthermore, the adjustment unit 132 may unify the lighting environment conditions between the object image and the reference image by correcting the reference image using the lighting environment information estimated based on the object image in a state in which the lighting environment information estimated based on the reference image has been removed from the reference image.

[0049] (Conversion unit 133) Based on either the object image or the reference image, the conversion unit 133 generates a converted image in which the other of the object image and the reference image is converted into an image of a first object whose appearance reflects the target. For example, the conversion unit 133 generates the converted image by converting the reference image so that feature information of a second object extracted from the reference image matches feature information of the first object extracted from the object image. Taking the example of supporting facial makeup work, the conversion unit 133 converts the reference image so that the facial shape of a makeup model extracted from the reference image matches the facial shape of a user extracted from the object image.

[0050] (Estimation unit 134) The estimation unit 134 estimates a processing procedure for changing the appearance of a first object to one based on a second object, based on the transformed image generated by the transformation unit 133 and the object image. Specifically, the estimation unit 134 estimates a processing procedure for changing the appearance of the first object to the appearance of a target targeted by the first object, based on output information from an estimation model that receives as input a pair of the transformed image and the object image. The target appearance is possessed by a second object different from the first object.

[0051] (Generation unit 135) Based on the object image, the generation unit 135 generates, as an output image, an image of the first object whose appearance has changed in accordance with the processing procedure estimated by the estimation unit 134. For example, the generation unit 135 generates, as the output image, an image in which the appearance of the first object according to the processing procedure is reflected as a work result.

[0052] The generation unit 135 may also generate, as output information to be output together with the output image, an instruction statement instructing the user to perform a task according to a processing procedure. The generation unit 135 may also generate, as output information, a detailed statement that explains the content of the instruction statement in more detail, based on a predetermined language model and the instruction statement.

[0053] (Output control unit 136) The output control unit 136 presents the output information generated by the generation unit 135 to the user. Specifically, the output control unit 136 controls the output so that the output information generated by the generation unit 135 is output to the user device 10. Note that the instruction sentence and the detailed sentence may be output in text format or in audio format.

[0054] 6. Configuration of the Learning Device The learning device 30 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example configuration of the learning device 30 according to the embodiment. As shown in Fig. 4, the learning device 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

[0055] (Communication Unit 31) The communication unit 31 is realized by, for example, a network interface card (NIC), etc. For example, the communication unit 31 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the server device 100.

[0056] (Storage Unit 32) The storage unit 32 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 has a model data storage unit 32a. The model data storage unit 32a stores data of an estimation model for estimating a processing procedure for changing the appearance of a first object to a target appearance.

[0057] (Control unit 33) The control unit 22 is realized by a CPU, an MPU, or the like using RAM as a work area to execute various programs (e.g., the information processing program according to the embodiment) stored in a storage device inside the learning device 30. The control unit 33 is realized by an integrated circuit such as an ASIC or an FPGA.

[0058] As shown in Fig. 3, the control unit 33 has an acquisition unit 33a, a generation unit 33b, and a learning unit 33c, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the control unit 33 is not limited to the configuration shown in Fig. 4, and may be other configurations as long as they perform the information processing described below. Furthermore, the connection relationships between the processing units of the control unit 33 are not limited to the connection relationships shown in Fig. 4, and may be other connection relationships.

[0059] (Acquisition unit 33 a) The acquisition unit 33 a acquires information constituting the learning data. For example, the acquisition unit 33 a acquires a video consisting of images of an action that changes the appearance of a predetermined object related to the first object into the appearance of a completed state, and speech information within the video that explains the content of the action.

[0060] (Generation unit 33b) The generation unit 33b generates learning data by combining a video consisting of images of actions that change the appearance of a specified object related to the first object to the appearance of a completed state with speech information within the video that explains the content of the actions.

[0061] (Learning Unit 33c) The learning unit 33c uses the learning data to train a model to learn the relationship between images before and after a change in appearance of a predetermined object and the actions resulting from the change in appearance. For example, when a pair of a transformed image and an object image is input, the learning unit 33c trains the model to learn the above relationship so that the model outputs information on a processing procedure for changing the appearance of a first object to a target appearance, thereby generating an estimation model. The target appearance is defined by the transformed image.

[0062] Here, a preprocessing technique in the learning phase will be described with reference to FIG. 5. FIG. 5 is an explanatory diagram illustrating the preprocessing technique in the learning phase. FIG. 5 shows a scene in which learning data is generated using a video VD consisting of a group of images of makeup actions that change an arbitrary person Py from an unmade-up state to a made-up state, and speech information within the video VD that explains the content of the makeup actions. In other words, FIG. 5 shows an example in which learning data is generated based on a makeup video with audio.

[0063] The video VD is acquired by the acquisition unit 33 a and processed into learning data by the generation unit 33 b. Specifically, the generation unit 33 b generates the learning data using the video VD, which is composed of a group of images of makeup actions that change an arbitrary person Py from an unmade-up state to a made-up state, and speech information in the video VD that explains the content of the makeup actions.

[0064] According to the example of FIG. 5 , the generation unit 33b extracts, from among the actions performed in the video VD, actions necessary for facial makeup (actions for achieving a completed makeup state) and timestamps at which the actions necessary for facial makeup were performed, in association with each other. The generation unit 33b may extract pairs of actions necessary for facial makeup and timestamps at which the actions necessary for facial makeup were performed by analyzing speech information (audio data) included in the video VD. Furthermore, the video VD may contain subtitle information or timestamps transcribed from the speech information (audio data) included in the video VD, inserted in advance as metadata. In such cases, the generation unit 33b can extract pairs of actions necessary for facial makeup and timestamps at which the actions necessary for facial makeup were performed, based on the metadata.

[0065] 5 shows an example in which the generation unit 33b has extracted a pair of a playback time "3:50" and a necessary action "skin care" as a pair of a timestamp and an action necessary for facial makeup. Also shown is an example in which the generation unit 33b has extracted a pair of a playback time "4:55" and a necessary action "primer" as a pair of a timestamp and an action necessary for facial makeup. Also shown is an example in which the generation unit 33b has extracted a pair of a playback time "7:20" and a necessary action "foundation" as a pair of a timestamp and an action necessary for facial makeup. Also shown is an example in which the generation unit 33b has extracted a pair of a playback time "7:38" and a necessary action "concealer" as a pair of a timestamp and an action necessary for facial makeup. Descriptions of the other combinations shown in the figure will be omitted.

[0066] Next, the generation unit 33b searches for a range corresponding to the extracted required action from within the video VD for each required action. In the above example, the generation unit 33b searches for a range in the video VD where the required action "skin care" was actually performed, based on the pair of the playback time "3:50" and the required action "skin care." Figure 5 shows an example in which the generation unit 33b searches for a video range RA1 as the range in the video VD where the required action "skin care" was actually performed.

[0067] Furthermore, the generation unit 33b searches for a range in the moving image VD where the necessary action "groundwork" was actually performed, based on the pair of the playback time "4:55" and the necessary action "groundwork." Figure 5 shows an example in which the generation unit 33b searches for a moving image range RA2 as a range in the moving image VD where the necessary action "groundwork" was actually performed.

[0068] Furthermore, the generation unit 33b searches for a range in the moving image VD where the necessary action "applying foundation" was actually performed, based on the pair of the playback time "7:20" and the necessary action "applying foundation." Figure 5 shows an example in which the generation unit 33b searches for a moving image range RA as a range in the moving image VD where the necessary action "applying foundation" was actually performed.

[0069] Furthermore, the generation unit 33b searches for a range in the video VD where the required action "concealer" was actually performed, based on the pair of the playback time "7:38" and the required action "concealer." Figure 5 shows an example in which the generation unit 33b searches for a video range RA4 as a range in the video VD where the required action "concealer" was actually performed.

[0070] Next, a learning method in the learning phase will be described with reference to Fig. 6. Fig. 6 is an explanatory diagram illustrating the learning method in the learning phase. Fig. 6 continues to use the example of Fig. 5. Fig. 6 shows a scene in which learning data is generated and learning processing is performed based on the pair of playback time "17:30" and required action "lip."

[0071] For example, the generation unit 33b acquires a representative frame image FL12 of facial features from a moving image range RA12 in which the required action "lip" was actually performed in the moving image VD. The generation unit 33b also acquires a representative frame image FL11 of facial features from a moving image range RA11 in which the required action "eyebrows," which is immediately before the required action "lips," was actually performed. In this way, the generation unit 33b acquires representative frames of facial features before and after each action required for facial makeup.

[0072] Then, the generation unit 33b generates a combination of frame image FL11, frame image FL12, and the action resulting from the state change from the appearance of person Py shown in frame image FL11 (appearance before applying lipstick) to the appearance of person Py shown in frame image FL12 (appearance after applying lipstick), i.e., the required action ``lip,'' as one piece of learning data.

[0073] Furthermore, in this example, the learning unit 33c causes a model to learn the relationship between the frame image FL11, the frame image FL12, and the required action "lips." For example, the learning unit 33c learns that the task of "applying lipstick" is required to change the appearance shown in frame image FL11 (the appearance before applying lipstick) to the appearance shown in frame image FL12 (the appearance after applying lipstick). Note that the learning unit 33c may also use frame images of a completed makeup state as learning data to learn the difference from the completed makeup state.

[0074] 6 focuses on the required action "lips," the generation unit 33b learns the changes in appearance before and after each extracted required action. For example, the generation unit 33b acquires a representative frame image FL2 of facial features from a moving image range RA2 in which the required action "foundation" was actually performed in the moving image VD. The generation unit 33b also acquires a representative frame image FL1 of facial features from a moving image range RA1 in which the required action "skin care," which is one action before the required action "foundation," was actually performed.

[0075] Then, the generation unit 33b generates a combination of frame image FL1, frame image FL2, and the action resulting from the change in appearance of person Py shown in frame image FL1 (appearance before the base is applied) to the appearance of person Py shown in frame image FL2 (appearance after the base is applied), i.e., the required action ``base'', as one piece of learning data.

[0076] Furthermore, the learning unit 33c trains the model to understand the relationship between the frame image FL1, the frame image FL2, and the necessary action “base coat.” For example, the learning unit 33c learns that the task of “applying base coat” is required to change the appearance shown in frame image FL1 (the appearance before the base coat is applied) to the appearance shown in frame image FL2 (the appearance after the base coat is applied).

[0077] It is preferable that the learning unit 33c learns the appearance changes before and after the required action from many videos VD rather than from one video VD, which allows the learning unit 33c to generate an estimation model with higher accuracy.

[0078] 7. Work Procedure Estimation Method] A work procedure estimation method realized by information processing according to an embodiment will be described with reference to Figures 7 and 8. Continuing with the example shown in Figure 1, Figure 7 describes an adjustment processing method for unifying conditions between face image IM1 (object image) and face image IM2 (reference image), and a conversion processing method for matching human features between face image IM1 and face image IM2. Figure 8 describes a work procedure estimation method and a method for outputting information based on the estimation result.

[0079] [7-1. Information Processing Method (1)] Fig. 7 is a diagram (1) showing an example of an estimation method realized by information processing according to an embodiment. Fig. 7 shows an example in which a person P1 (user) inputs a face image IM1 of his or her own face in a pre-makeup state (appearance before makeup) to the server device 100 using the user device 10. Fig. 7 also shows an example in which the person P1 inputs a face image IM2 of a person in a makeup state (target appearance) that the person P1 is aiming for, i.e., a makeup model Px, to the server device 100 as a reference image. Here, it is assumed that the face images IM1 and IM2 are input at the same time.

[0080] The image acquisition unit 131 acquires a facial image IM1 in response to an image input by the person P1 (step S101). The generation unit 135 applies the facial image IM1 to a three-dimensional predictive model (e.g., 3D MMM) to generate a 3D facial model FMx of the person P1 (step S102). The facial surface texture of the 3D facial model FMx includes the appearance features of the person P1 before makeup and the person P1's base skin color (skin color). The facial surface texture of the 3D facial model FMx is affected by the light source (e.g., shadow information) used in the space where the facial image IM1 was captured.

[0081] Here, the adjustment unit 132 estimates information about the lighting environment in the space where the face image IM1 was captured, based on the 3D face model FMx (step S103). For example, the adjustment unit 132 may estimate, as the lighting environment information, the intensity of light from a light source used in the capture space of the face image IM1, the angle at which light is irradiated from the light source to the person P1, etc.

[0082] Then, the adjustment unit 132 removes the information about the lighting environment estimated in step S103 from the 3D face model FMx (step S104). For example, the adjustment unit 132 removes the influence of the light source used in the shooting space of the face image IM1 on the appearance of the face image IM1 based on the information about the lighting environment. As a result, by removing the influence of the light source, the adjustment unit 132 can obtain a 3D face model FMxx that reflects, as the face surface texture, the appearance features of the person P1 before makeup application and the base skin color of the person P1 in a state and color that are close to the real thing.

[0083] The image acquisition unit 131 acquires a facial image IM2 in response to an image input by the person P1 (step S201). The generation unit 135 then applies the facial image IM2 to a three-dimensional predictive model (e.g., 3DMM) to generate a 3D facial model FMy of the makeup model Px (step S202). The facial surface texture of the 3D facial model FMy includes features of the makeup-applied state. The facial surface texture of the 3D facial model FMy is also affected by the light source (e.g., shadow information) used in the space where the facial image IM2 was captured.

[0084] Therefore, the adjustment unit 132 estimates information about the lighting environment in the space where the face image IM2 was captured, based on the 3D face model FMy (step S203). For example, the adjustment unit 132 may estimate, as the lighting environment information, the intensity of light from a light source used in the capture space of the face image IM2, the angle at which light is irradiated from the light source to the makeup model Px, etc.

[0085] Then, the adjustment unit 132 removes the information about the lighting environment estimated in step S203 from the 3D face model FMy (step S204). For example, the adjustment unit 132 removes the influence of the light source used in the shooting space of the face image IM2 on the appearance of the face image IM2 based on the information about the lighting environment. As a result, by removing the influence of the light source, the adjustment unit 132 can obtain a 3D face model FMxx that reflects the appearance features of the made-up state in a state and color that are close to the real thing as the face surface texture.

[0086] Here, the processes of steps S104 and S204, which remove the lighting environment information from both 3D face models, are adjustment processes for unifying conditions (lighting conditions) other than the pre-makeup state and the post-makeup state between the person P1 and the makeup model Px. However, in the adjustment process for unifying conditions other than the pre-makeup state and the post-makeup state, the adjustment unit 132 may perform adjustment process for matching the lighting conditions on the makeup model Px side to the lighting conditions on the person P1 side, rather than simply removing the lighting environment information.

[0087] For example, the adjustment unit 132 may correct the 3D face model FMy using the information on the lighting environment estimated in step S103 (the lighting conditions on the person P1 side) in a state in which the information on the lighting environment estimated in step S203 (the lighting conditions on the makeup model Px side) has been removed from the 3D face model FMy. More specifically, the adjustment unit 132 applies the information on the lighting environment estimated in step S103 to the 3D face model FMy from which the information on the lighting environment estimated in step S203 has been removed, thereby correcting the facial surface texture of the 3D face model FMy so that the facial surface texture of the 3D face model FMy corresponds to the lighting conditions on the person P1 side.

[0088] Furthermore, the processes of steps S101 to S105 and the processes of steps S201 to S205 may be performed in parallel. Then, after these processes are completed, the conversion unit 133 extracts facial feature information (step S305). Specifically, the conversion unit 133 extracts facial feature information of the person P1 from the 3D face model FMxx, and extracts facial feature information of the makeup model Px from the 3D face model FMyy. For example, the conversion unit 133 may extract facial shape information of the person P1 from the 3D face model FMxx, and extract facial shape information of the makeup model Px from the 3D face model FMyy. The facial shape information may include not only information indicating the facial contour, but also information indicating the contours of the face (e.g., nose shape, nose height, lip shape, lip thickness, etc.).

[0089] Next, the conversion unit 133 converts the 3D face model FMyy so that the facial feature information of the makeup model Px matches the facial feature information of the person P1 based on the feature information extracted in step S305 (step S306). For example, the conversion unit 133 converts the shape of the 3D face model FMyy so that the facial shape of the makeup model Px matches the facial shape of the person P1. As a result, the conversion unit 133 can obtain a 3D face model FMyyx in which the facial feature information of the makeup model Px matches the facial feature information of the person P1.

[0090] The conversion unit 133 may also perform UV mapping (step S307). Specifically, the conversion unit 133 performs UV mapping on the 3D face model FMxx to obtain a 2D face image UVG1 as a two-dimensional UV map. The conversion unit 133 also performs UV mapping on the 3D face model FMyyx to obtain a 2D face image UVG2 as a two-dimensional UV map.

[0091] According to the processing described above, the facial surface textures other than the makeup texture are unified between the 2D facial image UVG1 (person P1) and the 2D facial image UVG2 (makeup model Px). Specifically, in the initial stage before adjustment or conversion processing has been performed, differences in facial shape, lighting environment, etc. exist between the target image and the reference image, making it difficult to accurately compare the effects of makeup alone, i.e., the differences in facial surface texture before and after makeup. However, the facial shape and lighting conditions are unified between the 2D facial images UVG1 and UVG2 obtained through the processing up to this point, leaving only the differences between the pre-makeup and post-makeup states. Therefore, the server device 100 can accurately extract only the differences in facial surface texture before and after makeup. By comparing these differences and using the skin base color as a hint, the server device 100 can accurately estimate the work procedure. The estimation processing method performed from step S307 onward is described with reference to FIG. 8.

[0092] Note that in step S306, the conversion unit 133 may perform the reverse process instead of converting the 3D face model FMyy so as to match the facial feature information of the makeup model Px with the facial feature information of the person P1. Specifically, the conversion unit 133 may convert the 3D face model FMxx so as to match the facial feature information of the makeup model Px with the facial feature information of the person P1.

[0093] [7-2. Information Processing Method (2)] FIG. 8 is a diagram (2) illustrating an example of an estimation method implemented by the information processing according to the embodiment. First, the estimation unit 134 inputs a UV map or a 3D face model to the makeup procedure estimation model M (step S0401). For example, the estimation unit 134 inputs the 2D face images UVG1 and UVG2 generated in step S307 of FIG. 7 to the makeup procedure estimation model M. As another example, the estimation unit 134 may also input a 3D face model to the makeup procedure estimation model M. Specifically, the estimation unit 134 may input a set of the 2D face image UVG1 and the 3D face model FMxx, and a set of the 2D face image UVG2 and the 3D face model FMyyx to the makeup procedure estimation model M. As yet another example, the estimation unit 134 may employ a method in which a 3D face model is input but a UV map is not input. Specifically, the estimation unit 134 may simply input the 3D face model FMxx and the 3D face model FMyyx into the makeup procedure estimation model M. The learning method of the makeup procedure estimation model M is as described with reference to FIGS. 5 and 6.

[0094] The estimation unit 134 estimates makeup-related work procedures for changing the pre-makeup state shown in the facial image IM1 to the target makeup state shown in the facial image IM2 based on the information output from the makeup procedure estimation model M (step S402). Fig. 8 shows an example in which the estimation unit 134 has estimated the following procedures, similar to the example in Fig. 1: a first procedure "putting in colored contact lenses," a second procedure "applying makeup base," a third procedure "foundation and eye shadow," an eighth procedure "drawing eyebrows," a ninth procedure "putting on a wig," and a tenth procedure "lips."

[0095] Note that the estimation unit 134 does not necessarily estimate a 10-step work procedure. For example, when a face image IM1 with partially applied makeup is input instead of a face image IM1 without makeup, the estimation unit 134 may estimate a work procedure consisting of fewer steps. Furthermore, depending on the base skin color of the person P1, the estimation unit 134 may estimate a work procedure consisting of more steps. The reason why the number of steps and the content of the work procedure can be changed according to the situation is that the information processing according to the embodiment is not simply a rule-based estimation but aims to bring the atmosphere closer to a target state by using a machine learning model.

[0096] Furthermore, the generation unit 135 generates an instruction statement SM instructing the user to perform the work according to the work procedure estimated in step S402 (step S403). Fig. 8 shows an example in which the generation unit 135 generates an instruction statement SM1 instructing the user to perform the first step, for example, "put on colored contact lenses," similar to the example in Fig. 1 .

[0097] In addition, for each work procedure estimated in step S402, the generation unit 135 generates a 3D face model FM that reflects, as the work result, how the appearance of the person P1 will change if the work procedure is actually performed according to the content of the work procedure (step S404).

[0098] Specifically, the generation unit 135 generates a 3D face model FM1 that reflects the change in appearance that occurs in the facial appearance of the person P1 when the work is performed on the 3D face model FMxx in step 1. In other words, the generation unit 135 processes the 3D face model FMxx in an unmakeup state to change the appearance of the person P1 to one in which color contact lenses are worn.

[0099] Furthermore, the generation unit 135 generates a 3D face model FM2 that further reflects the appearance changes that will occur in the appearance of the person P1's face when work is performed in the second step, from the 3D face model FM1 that reflects the appearance changes up to the first step. Specifically, the generation unit 135 further processes the 3D face model FM1 that reflects the colored contact lens state, thereby changing the appearance of the person P1 to one that resembles a state in which a makeup base is applied.

[0100] Furthermore, the generation unit 135 generates a 3D face model FM3 that further reflects the changes in appearance that will occur in the appearance of the person P1's face when work is performed in step 3 on the 3D face model FM2 that reflects the changes in appearance up to step 2. Specifically, the generation unit 135 further processes the 3D face model FM2 in a state where a makeup base has been applied, thereby changing the appearance of the person P1 to a state where they have applied foundation and eye shadow.

[0101] The generation unit 135 generates a 3D face model FM8 that further reflects the appearance changes that will occur in the appearance of the person P1's face when the work in the eighth step is performed on the 3D face model FM7 that reflects the appearance changes up to the seventh step. Specifically, the server device 100 further processes the 3D face model FM7 to change the appearance of the person P1 to one with drawn eyebrows.

[0102] Furthermore, the generation unit 135 generates a 3D face model FM9 that further reflects the changes in appearance that will occur in the appearance of the face of the person P1 when work is performed in a 9th step on the 3D face model FM8 that reflects the changes in appearance up to the 8th step. Specifically, the generation unit 135 further processes the 3D face model FM8 with drawn eyebrows, thereby changing the appearance of the person P1 to one in which they are wearing a wig.

[0103] Furthermore, the generation unit 135 generates a 3D face model FM10 that further reflects the appearance changes that will occur in the appearance of the face of the person P1 when work is performed in a tenth step on the 3D face model FM9 that reflects the appearance changes up to the ninth step. Specifically, the generation unit 135 further processes the 3D face model FM9 in a state where the person P1 is wearing a wig, thereby changing the appearance to one in which the person P1 is wearing lipstick.

[0104] The generation unit 135 may input the instruction SM generated in step S403 into the large-scale language model LLM (step S405), and based on the output information, may further generate a detailed instruction that explains the content of the instruction SM in more detail (step S406). For example, based on the instruction SM1 and the large-scale language model, the generation unit 135 can generate a detailed instruction following the sequence: "Place the lens on the tip of the index finger of your dominant hand" → "Pull down your lower eyelid with the middle finger of your dominant hand" → "Once the lens is correctly placed on the pupil of your eye, slowly release the finger that was holding down the eyelid."

[0105] Next, the output control unit 136 controls output so that output information associating an instruction statement SM with a 3D face model FM for each work procedure is output to the user device 10 of the person P1 (step S407). Note that the method for outputting the detailed instruction statement is not limited. For example, when an instruction statement SM is selected by the person P1, the output control unit 136 may cause the user device 10 to output a detailed instruction statement corresponding to the selected instruction statement SM. On the other hand, when display space is available, the output control unit 136 may also cause the detailed instruction statement to be displayed together with the instruction statement SM.

[0106] [8. Modification 1] From here, Modification 1 of the present disclosure will be described. For example, the server device 100 may be implemented in various different aspects other than the above embodiment. Therefore, hereinafter, the server device 100 according to Modification 1 of the present disclosure will be referred to as "server device 100A."

[0107] Taking facial makeup as an example, even when a work procedure is presented, the user may not be able to follow the work procedure or may perform actions that differ from those indicated in the work procedure. In other words, the user may perform incorrect actions that differ from the initial work procedure estimated by the server device 100. Therefore, the server device 100A has a function that, when detecting an incorrect user action, re-estimates a work procedure for changing the current makeup state to a target state. This function may be newly incorporated into the server device 100 according to the embodiment described in FIG. 3 , and a detailed functional configuration will be described below.

[0108] [8-1. Configuration of the Server Device] The server device 100A according to Modification 1 will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example configuration of the server device 100A according to Modification 1. As shown in Fig. 3, the server device 100A has a communication unit 110, a storage unit 120, and a control unit 130A. The communication unit 110 and the storage unit 120 are the same as those in Fig. 3, and therefore description thereof will be omitted.

[0109] (Control unit 130A) The control unit 130A is realized by a CPU, an MPU, or the like executing various programs (e.g., the information processing program according to Modification 1) stored in a storage device inside the server device 100A using RAM as a work area. The control unit 130A is also realized by an integrated circuit such as an ASIC or FPGA.

[0110] As shown in FIG. 9 , the control unit 130A has an image acquisition unit 131, an adjustment unit 132, a conversion unit 133, an estimation unit 134, a generation unit 135, an output control unit 136, and a detection unit 137, and realizes or executes the information processing functions and actions described below. As such, the control unit 130A newly includes a detection unit 137 compared to the server device 100. The internal configuration of the control unit 130A is not limited to the configuration shown in FIG. 9 , and may be other configurations as long as they perform the information processing described below. Furthermore, the connection relationships between the processing units included in the control unit 130A are not limited to the connection relationships shown in FIG. 9 , and may be other connection relationships.

[0111] In the above embodiment, the image acquisition unit 131 acquires, as input information for the estimation process, a single object image showing an original state before the appearance is changed toward a target state. For example, the image acquisition unit 131 acquires, as input information, a face image IM1 showing a state before makeup is applied, as a single still image captured by the imaging function of the user device 10.

[0112] However, the image acquisition unit 131 may sequentially acquire object images in which actions that change the appearance of the first object are captured in real time. For example, a scene in which a user uses the user device 10 to capture a video in real time of the user gradually applying makeup from an unapplied state. Another scene in which a user uses the user device 10 to capture still images or video of the user's makeup application process. In such a case, the image acquisition unit 131 sequentially acquires object images captured in real time.

[0113] (Detection Unit 137) The detection unit 137 detects an erroneous action of the user based on the object images sequentially acquired by the image acquisition unit 131. For example, the detection unit 137 may detect, as an erroneous action, a discrepancy between the action procedure actually performed to change the appearance of the first object and the processing procedure estimated so far.

[0114] (Estimation unit 134) When an erroneous operation is detected by the detection unit 137, the estimation unit 134 re-estimates the processing procedure for changing the current appearance of the first object into the target appearance, using the latest object image among the object images acquired sequentially.

[0115] [8-2. Processing Procedure (1)] Before describing the procedure for the re-estimation process when an erroneous operation is detected, the procedure for the estimation process corresponding to the sequential acquisition of object images will be described. Fig. 10 is a flowchart showing the estimation process procedure according to Modification 1. Fig. 10 illustrates a scenario in which a facial image IM1 corresponding to a state before makeup is applied and a facial image IM2 corresponding to a makeup state that a person P1 is aiming for are checked, and a facial image IM1 of a current makeup state during makeup application is further input.

[0116] First, the image acquisition unit 131 determines whether a new facial image IM1 has been acquired (step S1001). While the image acquisition unit 131 has not been able to acquire a new facial image IM1 (step S1001; No), the image acquisition unit 131 waits until a new facial image IM1 can be acquired. On the other hand, when a facial image IM1 captured in real time of the person P1 applying makeup is input to the server device 100 by the user device 10 (during this time, the person P1 is applying makeup while following the work procedure presented by the server device 100), the image acquisition unit 131 can determine that a new facial image IM1 has been acquired. The new facial image IM1 may be a single still image or a moving image.

[0117] When a new facial image IM1 is acquired (step S1001; Yes), the estimation unit 134 estimates the work procedure for changing the current makeup state indicated by the new facial image IM1 to a target makeup state based on the acquired new facial image IM1 and the previously input facial image IM2 (step S1002).

[0118] Note that the estimation unit 134 actually estimates the work procedure by inputting a 2D face image UVG1 (or a 3D face model FMxx) generated based on the new face image IM1 and a 2D face image UVG2 (or a 3D face model FMyyx) generated based on the face image IM2 into the makeup procedure estimation model M. As described in FIG. 7 , before the UV map (2D face image UVG1, 2D face image UVG2) is obtained, the adjustment unit 132 performs an adjustment process and the conversion unit 133 performs a conversion process.

[0119] Then, the generation unit 135 generates output information to be presented to the person P1 based on the work procedures estimated in step S1002 (step S1003). Specifically, the generation unit 135 generates, for each work procedure, an instruction statement SM and a 3D face model that reflects the results of the work procedure.

[0120] 10, the server device 100 repeats the estimation process in response to input of a face image IM1 in the middle of applying makeup. If the server device 100 detects an incorrect action based on a comparison between the makeup actions actually performed by the person P1 and the work procedures estimated up to that point, the server device 100 presents the output information generated in step S1003 to the person P1 as the re-estimation result. This point will be described in more detail in FIG. 11.

[0121] 11 is a flowchart showing the malfunction detection process procedure according to Modification 1. The image acquisition unit 131 determines whether or not a new facial image IM1 has been acquired (step S1101). If the image acquisition unit 131 has not been able to acquire a new facial image IM1 (step S1101; No), the image acquisition unit 131 waits until a new facial image IM1 can be acquired.

[0122] On the other hand, if a new facial image IM1 is acquired (step S1101; Yes), the detection unit 137 performs image analysis on the newly acquired facial image IM1 and identifies the makeup operation that is currently being performed (step S1102).

[0123] Then, the detection unit 137 compares the work procedures estimated by the estimation unit 134 so far with the actual makeup operations identified in step S1102, and detects whether there is a discrepancy between the estimated work procedures and the actual makeup operations (step S1103). For example, the detection unit 137 may compare the earliest estimation result (i.e., the work procedures estimated based on the face image IM1 corresponding to the pre-makeup state and the face image IM2 corresponding to the target makeup state of the person P1) with the actual makeup operations.

[0124] If the detection unit 137 cannot detect any deviation between the estimated work procedure and the actual makeup operation (step S1103; No), the process returns to step S1101.

[0125] On the other hand, if a discrepancy is detected between the estimated work procedure and the actual makeup application operation (step S1103; Yes), the output control unit 136 acquires output information corresponding to the currently acquired face image IM1 as information on the re-estimated work procedure (step S1104). Specifically, the output control unit 136 acquires output information generated by the procedure of FIG. 10 using the new face image IM1 acquired in step S1101.

[0126] Then, the output control unit 136 controls the output so that the acquired output information is output to the user device 10 of the person P1 (step S1105).

[0127] 9. Modification 2 Next, a description will be given of Modification 2 of the present disclosure. The process according to Modification 1 of the present disclosure may also be performed by the server device 100A.

[0128] For example, if the detection unit 137 detects a discrepancy between the estimated work procedure and the actual makeup application, it may dynamically determine a new target appearance based on the makeup application that is currently being performed. For example, the detection unit 137 may compare the estimated work procedure with the actual makeup application and detect a discrepancy in which the step of "applying colored contact lenses" was skipped and the step of "applying makeup base" was performed. In such a case, the detection unit 137 may determine, for example, a "natural makeup" state that does not look unnatural even without the "colored contact lenses" as the new target appearance. The detection unit 137 may determine the new target appearance according to a rule base, or may estimate a makeup state that suits without the "colored contact lenses" using a machine learning model.

[0129] Furthermore, the estimation unit 134 may estimate a procedure for changing the current makeup state indicated by the new facial image IM1 to the new target appearance based on the new facial image IM1 and the facial image IMx of the new target appearance. The output control unit 136 may then recommend the estimated procedure to the person P1. For example, the output control unit 136 may present the procedure for changing the appearance to the new target appearance along with a comment such as, "It looks like you're not wearing colored contact lenses. In this state, why not try applying makeup using the following procedure?"

[0130] [10. Other] In the above embodiment, an example was shown in which server device 100 estimated a processing procedure for changing the appearance of a first object to a target appearance targeted by the first object as a processing procedure for changing the appearance of a first object to a target appearance based on a second object. However, server device 100 does not necessarily need to estimate a processing procedure for changing the appearance of the first object to a target appearance. For example, server device 100 may estimate a processing procedure for changing the appearance of the first object to an arbitrary appearance desired by the user, or may estimate a processing procedure for changing the appearance of the first object to an appearance preferred by the user.

[0131] [11. Hardware Configuration] An example hardware configuration of a computer corresponding to the information processing device (e.g., server device 100 and server device 100A) according to each of the above-described embodiments will be described using Fig. 12. Fig. 12 is a block diagram showing an example hardware configuration of a computer corresponding to the information processing device according to an embodiment of the present disclosure. Note that Fig. 12 shows an example of the hardware configuration of a computer corresponding to the information processing device according to each embodiment, and is not necessarily limited to the configuration shown in Fig. 12.

[0132] 12 , the computer 1000 includes a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0133] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0134] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0135] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100, data used by such programs, etc. Specifically, HDD 1400 records program data 1450. Program data 1450 is an example of an information processing program for realizing an information processing method according to an embodiment of the present disclosure, and data used by such information processing program.

[0136] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0137] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display device, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0138] For example, when the computer 1000 functions as the server device 100, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200, thereby realizing various processing functions executed by the processes shown in Fig. 3 etc. In other words, the CPU 1100, the RAM 1200, etc., work together with software (an information processing program loaded onto the RAM 1200) to realize an information processing method by an information processing device according to an embodiment of the present disclosure.

[0139] [12. Summary] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of the embodiments and modifications may be combined as appropriate.

[0140] Furthermore, the effects of each embodiment described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0141] The present disclosure may also be configured as follows: (1) An information processing device including: an acquisition unit that acquires an object image that is an image of a first object and a reference image of a second object different from the first object; a conversion unit that generates a transformed image in which the first object is transformed based on the object image and the reference image; an estimation unit that estimates a processing procedure for changing the appearance of the first object to one based on the transformed image and the object image, based on the transformed image and the object image; and a generation unit that generates, as an output image, an image of the first object whose appearance has been changed in accordance with the processing procedure, based on the object image. (2) The information processing device according to (1), wherein the acquisition unit acquires, as the reference image, an image of the second object having an appearance of a target targeted by the first object as an appearance based on the second object, the conversion unit generates the converted image in which the other of the object image and the reference image is converted into an image of the first object reflecting the appearance of the target, based on one of the object image or the reference image, and the estimation unit estimates a processing procedure for changing the appearance of the first object to the appearance of the target. (3) The information processing device according to (2), further comprising an adjustment unit that adjusts between the object image and the reference image to match conditions other than the appearance of the first object and the appearance of the target, and the conversion unit converts the other image using one of the object image or the reference image adjusted by the adjustment unit. (4) The information processing device according to (3), wherein the adjustment unit removes information about a lighting environment estimated based on the object image from the object image and removes information about a lighting environment estimated based on the reference image from the reference image, thereby eliminating lighting environment conditions between the object image and the reference image. (5) The information processing device according to (3) or (4), wherein the adjustment unit corrects the reference image using information about the lighting environment estimated based on the object image in a state in which the information about the lighting environment estimated based on the reference image has been removed from the reference image, thereby aligning lighting environment conditions between the object image and the reference image.(6) The information processing device according to any one of (2) to (5), wherein the conversion unit converts the reference image so as to match feature information of the second object extracted from the reference image with feature information of the first object extracted from the object image. (7) The information processing device according to any one of (2) to (6), further comprising a learning unit that generates a model using a pair of the converted image and the object image as input, wherein the estimation unit estimates a processing procedure for changing the appearance of the first object to the target appearance based on output information of the model. (8) The information processing device according to (7), wherein the learning unit uses a combination of a video consisting of images of actions that change the appearance of a predetermined object related to the first object to a completed state and speech information within the video explaining the content of the actions as learning data to have a model learn a relationship between images before and after showing a change in appearance of the predetermined object and the actions caused by the change in appearance. (9) The information processing device according to (8), wherein the learning unit causes the model to learn the relationship so as to output information on a processing procedure for changing the appearance of the first object to the target appearance when a pair of the converted image and the object image is input. (10) The information processing device according to any one of (1) to (9), wherein the generation unit generates, as the output image, an image of the first object in which an appearance according to the processing procedure is reflected in the appearance of the first object as a work result. (11) The information processing device according to any one of (1) to (10), wherein the generation unit generates, as output information to be output together with the output image, an instruction statement instructing to perform work according to the processing procedure. (12) The information processing device according to (11), wherein the generation unit further generates, as the output information, a detailed statement that explains the content of the instruction statement in more detail based on a predetermined language model and the instruction statement.(13) The information processing device according to any one of (2) to (12), wherein the acquisition unit sequentially acquires, as the object images, object images in which actions that change the appearance of the first object are captured in real time, and further includes a detection unit that detects an erroneous action based on the sequentially acquired object images, and the estimation unit, when an erroneous action is detected by the detection unit, re-estimates a processing procedure for changing the current appearance of the first object to the target appearance using the latest object image among the sequentially acquired object images. (14) The information processing device according to (13), wherein the detection unit detects, as the erroneous action, a discrepancy between the processing procedure that is actually being performed to change the appearance of the first object and the processing procedure that has been estimated so far. (15) The information processing device according to (13) or (14), wherein, when the erroneous action is detected, the detection unit identifies an appearance of a new target related to an action procedure actually being performed to change the appearance of the first object, and the estimation unit re-estimates a processing procedure for changing the current appearance of the first object to an appearance of the new target by using the latest object image among the object images sequentially acquired. (16) An information processing method executed by an information processing device, the information processing method including: an acquisition step of acquiring an object image that is an image of the first object and a reference image of a second object different from the first object, a transformation step of generating a transformed image in which the first object has been transformed, based on the object image and the reference image, an estimation step of estimating a processing procedure for changing the appearance of the first object to one based on the transformed image and the object image, and a generation step of generating, as an output image, an image of the first object whose appearance has been changed in accordance with the processing procedure, based on the object image.(17) An information processing program for causing a computer to execute the following steps: an acquisition step for acquiring an object image that is an image of a first object and a reference image of a second object different from the first object; a transformation step for generating a transformed image of the first object based on the object image and the reference image; an estimation step for estimating a processing procedure for changing the appearance of the first object to one based on the second object based on the transformed image and the object image; and a generation step for generating, as an output image, an image of the first object whose appearance has been changed in accordance with the processing procedure based on the object image.

[0142] REFERENCE SIGNS LIST 1 System 10 User device 30 Learning device 100 Server device 120 Storage unit 121 Model data storage unit 122 Image data storage unit 123 Estimation procedure data storage unit 130 Control unit 131 Image acquisition unit 132 Adjustment unit 133 Conversion unit 134 Estimation unit 135 Generation unit 136 Output control unit

Claims

1. An information processing device comprising: an acquisition unit that acquires an object image, which is an image of a first object, and a reference image of a second object different from the first object; a conversion unit that generates a transformed image in which the first object has been transformed based on the object image and the reference image; an estimation unit that estimates a processing procedure for changing the appearance of the first object to one based on the second object, based on the transformed image and the object image; and a generation unit that generates an image of the first object whose appearance has been changed in accordance with the processing procedure as an output image, based on the object image.

2. The information processing device according to claim 1, wherein the acquisition unit acquires an image of the second object having the appearance of a target targeted by the first object as the reference image as an appearance based on the second object, the conversion unit generates the converted image in which the other of the object image and the reference image is converted into an image of the first object reflecting the appearance of the target based on either one of the object image or the reference image, and the estimation unit estimates a processing procedure for changing the appearance of the first object to the appearance of the target.

3. An information processing device as described in claim 2, further comprising an adjustment unit that performs adjustments between the object image and the reference image to match the appearance of the first object and other conditions other than the appearance of the target, and the conversion unit converts the other image using either the object image or the reference image after adjustment by the adjustment unit.

4. The information processing device of claim 3, wherein the adjustment unit removes information about the lighting environment estimated based on the object image from the object image and removes information about the lighting environment estimated based on the reference image from the reference image, thereby removing lighting environment conditions between the object image and the reference image.

5. The information processing device of claim 3, wherein the adjustment unit adjusts lighting environment conditions between the object image and the reference image by removing information about the lighting environment estimated based on the reference image from the reference image and correcting the reference image using information about the lighting environment estimated based on the object image.

6. The information processing device according to claim 2, wherein the conversion unit converts the reference image so as to match the feature information of the second object extracted from the reference image with the feature information of the first object extracted from the object image.

7. The information processing device according to claim 2, further comprising a learning unit that generates a model using a pair of the transformed image and the object image as input, wherein the estimation unit estimates a processing procedure for changing the appearance of the first object to the target appearance based on output information of the model.

8. The information processing device according to claim 7, wherein the learning unit uses a combination of a video consisting of images of actions that change the appearance of a specified object related to the first object to a completed state and speech information within the video explaining the content of the actions as learning data, and trains a model to learn the relationship between images before and after showing the change in appearance of the specified object and the actions resulting from the change in appearance.

9. The information processing device according to claim 8, wherein the learning unit causes the model to learn the relationship so as to output information on a processing procedure for changing the appearance of the first object to the target appearance when a pair of the transformed image and the object image is input.

10. The information processing device according to claim 1, wherein the generation unit generates, as the output image, an image of the first object in which an appearance of the first object corresponding to the processing procedure is reflected as a work result.

11. The information processing device according to claim 1, wherein the generation unit generates an instruction statement for instructing a task to be performed in accordance with the processing procedure as output information to be output together with the output image.

12. The information processing device according to claim 11, wherein the generation unit further generates, as the output information, a detailed sentence that explains the content of the instruction sentence in more detail based on a predetermined language model and the instruction sentence.

13. The information processing device according to claim 2, wherein the acquisition unit sequentially acquires, as the object images, object images in which an action that changes the appearance of the first object is captured in real time, and further comprises a detection unit that detects an erroneous action based on the sequentially acquired object images, and when an erroneous action is detected by the detection unit, the estimation unit re-estimates a processing procedure for changing the current appearance of the first object to the target appearance using the latest object image among the sequentially acquired object images.

14. The information processing device according to claim 13, wherein the detection unit detects, as the erroneous operation, a deviation between the operation procedure actually being performed to change the appearance of the first object and the processing procedure estimated so far.

15. The information processing device of claim 13, wherein when the erroneous action is detected, the detection unit identifies the appearance of a new target associated with the action procedure actually being performed to change the appearance of the first object, and the estimation unit re-estimates the processing procedure for changing the current appearance of the first object to the appearance of the new target using the latest object image among the object images successively acquired.

16. An information processing method executed by an information processing device, comprising: an acquisition step of acquiring an object image, which is an image of a first object, and a reference image of a second object different from the first object; a conversion step of generating a converted image in which the first object has been converted based on the object image and the reference image; an estimation step of estimating a processing procedure for changing the appearance of the first object to one based on the second object, based on the converted image and the object image; and a generation step of generating, as an output image, an image of the first object whose appearance has been changed in accordance with the processing procedure, based on the object image.

17. An information processing program for causing a computer to execute the following steps: an acquisition step for acquiring an object image, which is an image of a first object, and a reference image of a second object different from the first object; a transformation step for generating a transformed image in which the first object has been transformed based on the object image and the reference image; an estimation step for estimating a processing procedure for changing the appearance of the first object to one based on the second object, based on the transformed image and the object image; and a generation step for generating, as an output image, an image of the first object whose appearance has been changed in accordance with the processing procedure, based on the object image.

Images