Information processing device operation method, information processing device, and program

Abstract

To make it possible to convert images so as not to damage attributes of original face images while expressing various makeup methods.SOLUTION: An operation method of an information processing device includes: a first step of converting a first face image to a second face image by applying makeup information to the first face image by a prescribed procedure; a second step of converting the first face image to a third face image with the second face image as a reference image by an image conversion model generated by machine learning processing for converting a conversion object face image by applying makeup information included in a reference face image; and a third step of adjusting the image conversion model by reducing a loss of the first face image in the third face image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an operation method of an information processing device, an information processing device, and a program. [Background technology]

[0002] A technique for converting the style of a target image using an image conversion model that has been machine-learned to apply the style of a reference image to another image is known (see, for example, Patent Document 1). Such a technique is applied in fields such as beauty. For example, a makeup conversion method has been proposed that uses an image conversion model that adds makeup information from a reference image to an image of a face without makeup, thereby converting a user's face image into an image of a face with makeup, and virtually trying out the makeup. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-135822 Summary of the Invention [Problem to be solved by the invention]

[0004] Image conversion models trained by machine learning using techniques such as GAN (Generative Adversarial Network) to learn various styles of reference images have the problem that the learning is unstable, and the attributes of the target image are lost when the style of the target image is converted. In the case of makeup conversion, there is a risk that the texture of the original facial image will be lost. Furthermore, makeup techniques depend on the personal skills of the makeup artist and change daily, so variations are likely to occur. Therefore, generating an image conversion model that can express any makeup style using machine learning would require endless training, which is not practical.

[0005] In view of the above, the following describes an operation method of an information processing device that enables image conversion that expresses various makeup techniques without damaging the attributes of the original facial image. [Means for solving the problem]

[0006] In order to solve the above problem, the operating method of an information processing device in the present disclosure includes a first step of adding makeup information to a first facial image in accordance with a predetermined procedure to convert it into a second facial image; a second step of converting the first facial image into a third facial image using the second facial image as a reference image using an image conversion model generated by machine learning a process of converting a facial image to be converted by adding makeup information contained in a reference facial image; and a third step of adjusting the image conversion model by reducing loss of the first facial image in the third facial image.

[0007] The information processing device of the present disclosure also includes a storage unit that stores an image conversion model, the image conversion model being generated by machine learning a process of converting a facial image to be converted by adding makeup information included in a reference facial image, and being adjusted so as to reduce loss of the first facial image in the second facial image when the first facial image is converted into the second facial image using a facial image to which makeup information has been added in a predetermined procedure as a reference image; and a control unit that executes the predetermined procedure to convert an input facial image into an output facial image using the image conversion model.

[0008] Furthermore, the program in the present disclosure is a program executed by an information processing device, wherein the information processing device is capable of using an image transformation model that is generated by machine learning a process of transforming a facial image to be transformed by adding makeup information contained in a reference facial image, and that is adjusted to reduce loss of a first facial image in a second facial image when the first facial image is transformed into a second facial image using a facial image to which makeup information has been added in a predetermined procedure as a reference image, and includes a first step of executing the predetermined procedure on an input facial image, and a second step of converting the facial image on which the first step has been executed into an output facial image using the image transformation model. [Effects of the Invention]

[0009] According to the operation method of the information processing device and the like in the present disclosure, it is possible to perform image conversion that expresses various makeup techniques without damaging the attributes of the original facial image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system. [Figure 2] FIG. 10 is a flowchart illustrating an example of an operation procedure of the server device. [Figure 3] FIG. 10 is a flowchart illustrating an example of an operation procedure of the server device. [Figure 4] 10A and 10B are diagrams illustrating a face image used in image conversion. [Figure 5] FIG. 10 is a sequence diagram illustrating an example of an operation procedure of the server device and the terminal device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described.

[0012] [System Configuration] FIG. 1 is a diagram illustrating an example of a configuration of an embodiment of the present invention. The information processing system 1 includes a server device 10 and a terminal device 12 that are connected to each other via a network 11 so that they can communicate with each other. In the information processing system 1, the server device 10 performs machine learning using various information sent from the terminal device 12. The terminal device 12 is, for example, one or more personal computers. The personal computer may include a tablet terminal device, a smartphone, etc. The server device 10 is, for example, one or more server computers. When the server device 10 is a single server computer, the server device 10 may be multiple server computers that cooperate to execute the operations of this embodiment and provide a cloud service. The network 11 is, for example, a local area network (LAN), the Internet, an ad hoc network, a metropolitan area network (MAN), a mobile communication network, or other networks, or any combination thereof.

[0013] The server device 10 acquires a facial image obtained by capturing a person's face from the terminal device 12, performs machine learning using the facial image, and generates an image transformation model 108. The terminal device 12 is, for example, a device owned by a user, or a device installed in a physical store and used by a salesperson. The image transformation model 108 is an image transformation model that adds makeup information of a facial image with makeup applied, i.e., a reference image, to a facial image without makeup (an image including the entire face of a person viewed from the front). The server device 10 also adjusts the image transformation model 108 using an image in which makeup information is added to a facial image without makeup (hereinafter referred to as an original facial image) in a predetermined procedure, so that the attributes of the original facial image are not lost when the original facial image is transformed using the image transformation model 108.

[0014] Specifically, the server device 10 executes a first step (hereinafter referred to as an algorithm processing step) of adding makeup information to an original face image according to a predetermined procedure to convert the original face image into a primary face image. The server device 10 also executes a second step (hereinafter referred to as a makeup conversion step) of converting the original face image into a secondary face image using the primary face image as a reference face image by using an image conversion model 108 generated by machine learning a process of converting a face image to be converted by adding makeup information contained in a reference face image. The server device 10 then executes a third step (hereinafter referred to as an adjustment step) of adjusting the image conversion model 108 by reducing loss of the original face image in the secondary face image. Here, the server device 10 corresponds to an "information processing device."

[0015] According to this embodiment, the server device 10 executes an algorithm processing step using a procedure for reproducing a makeup technique used by a makeup artist, thereby enabling the simulation of the makeup artist's technique without machine learning. Furthermore, by separating the procedure for adding makeup information as an algorithm processing step, the amount of information processing required for machine learning can be reduced, improving the stability of machine learning. Furthermore, the server device 10 converts an original facial image into a secondary facial image using the image transformation model 108 and adjusts the image transformation model 108 by reducing loss of the original facial image in the secondary facial image. This makes it possible to prevent the loss of attributes of the original facial image when the original facial image is converted into a secondary facial image through the algorithm processing step and the makeup conversion step. In other words, image transformation is possible that reproduces various makeup techniques while preserving the attributes of the original facial image, particularly so-called textures such as gloss, matte, sheer, and foggy.

[0016] Next, the configurations of the server device 10 and the terminal device 12 will be described.

[0017]

[0018] The server device 10 includes a communication unit 101, a storage unit 102, a control unit 103, an input unit 105, and an output unit 106. When the server device 10 is configured with two or more server computers, these components are appropriately arranged in the two or more server computers.

[0019] The communication unit 101 includes one or more communication interfaces. The communication interface is, for example, a LAN interface. The communication unit 101 receives information used in the operation of the server device 10 and transmits information obtained by the operation of the server device 10. The server device 10 is connected to a network 11 by the communication unit 101 and communicates information with a terminal device 12 via the network 11.

[0020] The storage unit 102 includes, for example, one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of them, that function as a main storage device, an auxiliary storage device, or a cache memory. The semiconductor memory is, for example, a random access memory (RAM) or a read-only memory (ROM). The RAM is, for example, a static RAM (SRAM) or a dynamic RAM (DRAM). The ROM is, for example, an electrically erasable programmable read-only memory (EEPROM). The storage unit 102 stores information used in the operation of the control unit 103 and information obtained by the operation of the control unit 103. The storage unit 102 stores an image transformation model 108 generated by the control unit 103 based on information sent from the terminal device 12.

[0021] The control unit 103 includes one or more processors, one or more dedicated circuits, or a combination thereof. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit), or a dedicated processor such as a GPU (Graphics Processing Unit) specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc. The control unit 103 executes information processing related to the operation of the server device 10 while controlling each unit of the server device 10.

[0022] The functions of the server device 10 are realized by a processor included in the control unit 103 executing a control program. The control program is a program for causing the processor to function as the control unit 103. Alternatively, some or all of the functions of the server device 10 may be realized by a dedicated circuit included in the control unit 103. Alternatively, the control program may be stored in a non-transitory recording / storage medium readable by the control unit 103, and read by the control unit 103 from the medium.

[0023] The input unit 105 includes one or more input interfaces. The input interfaces are, for example, physical keys, capacitance keys, a pointing device, a touch screen integrated with a display, or a microphone that accepts voice input. The input unit 105 accepts an operation to input information used in the operation of the server device 10 and sends the input information to the control unit 103.

[0024] The output unit 106 includes one or more output interfaces. The output interface is, for example, a display or a speaker. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. The output unit 106 outputs information obtained by the operation of the server device 10.

[0025] The terminal device 12 includes a communication unit 121 , a storage unit 122 , a control unit 123 , an input unit 125 , and an output unit 126 .

[0026] The communication unit 121 includes a communication module compatible with wired or wireless LAN standards, a module compatible with mobile communication standards such as LTE, 4G, 5G, etc. The terminal device 12 is connected to the network 11 by the communication unit 121 via a nearby router device or a mobile communication base station, and performs information communication with the server device 10, etc. via the network 11.

[0027] The storage unit 122 includes one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these. The semiconductor memories are, for example, RAM or ROM. The RAM is, for example, SRAM or DRAM. The ROM is, for example, EEPROM. The storage unit 122 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 122 stores information used in the operation of the control unit 123 and information obtained by the operation of the control unit 123.

[0028] The control unit 123 has, for example, one or more general-purpose processors such as a CPU, an MPU (Micro Processing Unit), etc., or one or more dedicated processors such as a GPU specialized for a specific process. Alternatively, the control unit 123 may have one or more dedicated circuits such as an FPGA, an ASIC, etc. The control unit 123 performs overall control of the operation of the terminal device 12 by operating according to a control / processing program or operating according to an operating procedure implemented as a circuit. The control unit 123 then transmits and receives various information to and from the server device 10, etc. via the communication unit 121, and performs the operation according to this embodiment.

[0029] The functions of the terminal device 12 are realized by a processor included in the control unit 123 executing a control program. The control program is a program for causing the processor to function as the control unit 123. Alternatively, some or all of the functions of the terminal device 12 may be realized by a dedicated circuit included in the control unit 123. Alternatively, the control program may be stored in a non-transitory recording / storage medium readable by the control unit 123, and read by the control unit 123 from the medium.

[0030] The input unit 125 includes one or more input interfaces. The input interfaces include, for example, physical keys, capacitive keys, a pointing device, and a touch screen integrated with a display. The input interfaces also include a microphone for receiving voice input and a camera for capturing captured images. The input interfaces may also include a scanner or camera for scanning image codes, and an IC card reader. The input unit 125 receives an operation for inputting information used in the operation of the control unit 123 and sends the input information to the control unit 123. The input unit 125 also sends images captured by the camera to the control unit 123.

[0031] The output unit 126 includes one or more output interfaces. The output interfaces include, for example, a display and a speaker. The display is, for example, an LCD or an organic EL display. The output unit 126 outputs information obtained by the operation of the control unit 123.

[0032] [Generating an image transformation model] 2 is a flowchart illustrating an example of the operation of the server device 10 related to the generation of the image transformation model 108. Each step is executed by the control unit 103.

[0033] In step S20, the control unit 103 acquires face images necessary for machine learning to generate an image transformation model. The face images include an original face image to be transformed without makeup and a reference face image with makeup. The original face image is generated by capturing an image of a person's face without makeup. The reference face image is generated by capturing an image of a person's face with makeup. The image of the person's face is captured by, for example, the terminal device 12. For example, the control unit 103 receives multiple original face images and multiple reference face images sent from the terminal device 12 via the communication unit 101 and stores them in the storage unit 102. The control unit 103 may acquire multiple original face images and multiple reference face images from open data.

[0034] In step S22, the control unit 103 performs machine learning. The control unit 103 performs deep learning using, for example, GAN. The control unit 103 has a module corresponding to a generator that adds makeup information of a reference face image to an original face image, and a module corresponding to a classifier that classifies the face image generated by the generator from the original face image. The control unit 103 generates an image transformation model 108 by training the generator and the classifier in an adversarial manner. The control unit 103 stores the generated image transformation model 108 in the storage unit 102.

[0035] [Adjusting the image transformation model] Fig. 3 is a flowchart for explaining an example of the operation of the server device 10 relating to the adjustment of the image transformation model 108. Each step is executed by the control unit 103. Fig. 4 is a diagram for explaining a face image used for adjusting the image transformation model 108. The procedure of Fig. 3 will be explained with reference to Fig. 4.

[0036] In step S30, the control unit 103 acquires an original face image Isrc for executing the algorithm processing step. The original face image Isrc is a face image without makeup, and is generated by, for example, the terminal device 12 capturing an image of a person's face without makeup. For example, the control unit 103 receives the original face image Isrc sent from the terminal device 12 via the communication unit 101 and stores it in the storage unit 102. The control unit 103 may acquire multiple original face images Isrc from open data.

[0037] In step S32, the control unit 103 executes an algorithm processing step. The control unit 103 assigns predetermined makeup information MU to the original face image Isrc and converts the original face image Isrc into a primary face image Isyn. The makeup information MU includes one or more of hue, brightness, and saturation to be assigned to areas of the original face image Isrc, including the eyes, nose, cheeks, and lips (hereinafter referred to as makeup areas). The makeup information MU and the procedure for assigning the makeup information are image processing procedures that are arbitrarily set in advance. The hue, brightness, and saturation assigned to the makeup areas may be arbitrarily determined quantitatively, or color information already assigned to a reference face image may be extracted and applied. The reference face image can be arbitrarily selected by the user, for example, to match their own image. The reference image includes, for example, face images with various types of makeup, such as elegant, cool, and trendy, as well as face images of real people. The makeup information MU may be input by an operator at the terminal device 12 and sent from the terminal device 12 to the server device 10. The control unit 103 executes image processing to apply eyeliner, nose shadow, cheek color, lip color, or the like of a color arbitrarily set in the makeup information MU to the eyes, bridge of the nose, cheeks, or lips, respectively, in the facial image Isrc.

[0038] In step S34, the control unit 103 executes a makeup conversion step. The control unit 103 inputs the original face image Isrc as a face image to be converted and the primary face image Isyn as a reference face image to the image conversion model 108. The image conversion model 108 extracts feature information such as facial shape, three-dimensional effect, and surface condition from the original face image Isrc and makeup information from the primary face image Isyn using an encoder, adds the makeup information of the primary face image Isyn to the original face image Isrc, and converts it into a secondary face image Ig using a decoder.

[0039] In step S36, the control unit 103 executes an adjustment step. When the secondary face image Ig is used as a parameter of a loss function L, the control unit 103 adjusts the parameters of the image transformation model 108 so as to minimize the value of the loss function L. The loss function L includes, for example, one or more of adversarial loss, makeup loss, perceptual loss, MGE (Mean Gradient Error) loss, and color loss.

[0040] Adversarial loss is a loss function for training a generator to deceive a classifier in a GAN. The control unit 103 causes a classifier to classify whether the secondary face image Ig is the result of a makeup transformation process or the original face image Isrc, and uses the result to train the generator to deceive the classifier. In this way, parameters of the image transformation model 108 are adjusted so that the loss of the original face image Isrc in the secondary face image Ig is reduced. The classifier may include a global classifier and a local classifier. The global classifier uses the entire face image to classify whether the secondary face image Ig is the result of a makeup transformation process or the original face image Isrc. The local classifier uses makeup-applied areas to classify whether the secondary face image Ig is the result of a makeup transformation process or the original face image Isrc. Using both a global classifier and a local classifier makes it possible to improve the learning accuracy of the generator.

[0041] Makeup loss is a loss function for training the generator to deceive the classifier regarding color distribution. The control unit 103 generates a pseudo-face image having the same color distribution as the primary face image Isyn by histogram matching based on the primary face image Isyn. The histogram matching may be performed on the color distribution of the entire face image or on the color distribution of the makeup-applied areas. The control unit 103 then causes the classifier to identify whether the pseudo-face image or secondary face image Ig is the result of the histogram matching or makeup conversion process or the original face image Isrc, and uses the result to train the generator to deceive the classifier. In this way, the parameters of the image conversion model 108 are adjusted so that the loss of the original face image Isrc in the secondary face image Ig is reduced.

[0042] Perceptual loss is a loss function for training the generator to deceive the classifier regarding the contours of a facial image. The control unit 103 acquires a facial image during conversion from an intermediate layer when the generator converts a primary facial image Isyn into a secondary facial image Ig, and generates an edge image by extracting the edges of each part of the image, such as the eyes, nose, and mouth. The control unit 103 then causes the classifier to distinguish whether the edges of the edge image are edges of an edge image resulting from a makeup conversion process or edges of the original facial image Isrc, and uses the result to train the generator to deceive the classifier. In deep learning, edge information of the converted image is extracted as a feature in the intermediate layer. By performing training using the edge image in the intermediate layer, the parameters of the image conversion model 108 are adjusted to reduce loss related to the edges of the original facial image Isrc in the secondary facial image Ig. In other words, the image conversion model 108 is adjusted so that the facial contours in the secondary facial image Ig match the facial contours in the original facial image Isrc.

[0043] The MGE loss is a loss function related to the contour of a facial image. The control unit 103 applies a differential filter to each of the original facial image Isrc and the secondary facial image Ig to increase the resolution of the edges. The control unit 103 then adjusts the parameters of the image transformation model 108 so that the edges in the high-resolution secondary facial image Ig match the edges in the high-resolution original facial image Isrc. In this way, the image transformation model 108 is adjusted so that the loss related to the edges of the original facial image Isrc in the secondary facial image Ig is reduced.

[0044] Color loss is a loss function for reinforcing makeup loss with respect to color distribution. The control unit 103 derives the differences between the average and variance of colors defined by makeup information MU assigned to the original face image Isrc and the average and variance of colors in the primary face image Isyn of the makeup-applied area. The control unit 103 then adjusts the parameters of the image transformation model 108 to reduce these differences. In this way, the parameters of the image transformation model 108 are adjusted so that the loss of the original face image Isrc in the secondary face image Ig is reduced.

[0045] Through the above procedure, the image transformation model 108 is adjusted.

[0046] The adjusted image transformation model 108 is used to transform any original face image. Specifically, the control unit 103 performs algorithm processing on any original face image to generate a primary face image, and then transforms the primary face image into a secondary face image using the image transformation model 108. Here, the results of verifying the operation of the image transformation model 108 are shown.

[0047] [Verification 1] The server device 10 performed algorithmic processing to apply the lip color contained in a reference facial image to an original facial image by histogram matching. 150 arbitrary facial images from open data were used as the original facial images. Each original facial image was converted into a primary facial image by algorithmic processing, and the primary facial image was converted into a secondary facial image by the image conversion model 108. U-net architecture was employed for the image conversion model 108. Furthermore, makeup information was added to each original facial image using related technologies. Related technologies employed included BeautyGAN, PSGAN (Pose and Expression Robust Spatial-Aware GAN), and CPM (Color-Pattern Makeup Transfer). Then, scores indicating impressions regarding the following four items were obtained from 12 subjects (women aged 18 to 23) for the secondary facial images and the facial images to which makeup information was added using each related technology. <Item 1> Is the lip color of the reference face image reflected? <Item 2> Is the skin texture of the original facial image maintained? <Item 3> Is the texture of the lips in the original facial image maintained? <Item 4> <1> ~ <3> Overall rating of

[0048] Table 1 below shows the results of tallying the scores for each item for the secondary face image of this embodiment and face images from each related technology. A higher score indicates a more favorable impression. As shown in Table 1, in <Item 2> to <Item 4>, the secondary face image of this embodiment garnered a more favorable impression than the face images from the related technology. [Table 1]

[0049] [Verification 2] The server device 10 performed algorithmic processing to apply the lip color contained in the reference facial image to the original facial image through histogram matching. An arbitrary facial image from open data was used as the original facial image. Four lip colors were used: crimson, reddish brown, vermilion, and crimson red. During the algorithmic processing, texture information from the original facial image was deleted to remove texture. After the original facial image was converted into a primary facial image through the algorithmic processing, the server device 10 converted the primary facial image into a secondary facial image using four image conversion models 108 adjusted with different combinations of loss functions. Here, a U-net architecture was employed for the image conversion model 108. The four combinations of loss functions used to adjust the image conversion model 108 are as follows: <Pattern 1> Adversarial loss, Makeup loss, Perceptual loss, and Color loss <Pattern 2> Adversarial loss, Makeup loss, MGE loss, and Color loss <Pattern 3> Adversarial loss, Makeup loss, Perceptual loss, and MGE loss <Pattern 4> Adversarial loss, Makeup loss, Perceptual loss, MGE loss, and Color loss

[0050] An arbitrary original face image was converted into a total of 16 secondary face images, with four patterns for each of the four lip colors. Then, a qualitative evaluation of the secondary face images for each pattern was performed for each lip color. The qualitative evaluation for each pattern was as follows: <Pattern 1> The texture of the original face image cannot be reproduced <Pattern 2> The contours are prominent <Pattern 3> Learning becomes unstable <Pattern 4> All of the above are resolved

[0051] [Verification 3] The server device 10 performed algorithmic processing to apply the lip color contained in the reference facial image to the original facial image using histogram matching. An arbitrary facial image from open data was used as the original facial image. After the algorithmic processing, the lip color was clustered in color space to identify glossy areas, and a gloss reduction process (filling the glossy areas with the color of the non-glossy areas) was performed. The algorithmic processing converted the original facial image into a primary facial image, which was then converted into a secondary facial image by the image conversion model 108. Here, the image conversion model 108 employed a U-net architecture. As a result, a qualitative evaluation was obtained that the three-dimensional effect of the original facial image was reproduced in the gloss-reduced areas of the secondary facial image.

[0052] [Example] Fig. 5 is a sequence diagram for explaining an example of the operation of the information processing system 1 in the embodiment. The procedure in Fig. 5 relates to the cooperative operation between the server device 10 having the image transformation model 108 adjusted by the procedure of this embodiment and the terminal device 12. The terminal device 12 is used by a user who tries on makeup using, for example, a captured image of their own face.

[0053] In step S50, the terminal device 12 captures an image of the user. In response to an operation input by the user to the input unit 125, the control unit 123 of the terminal device 12 captures an image using the camera included in the input unit 125. As a result, the terminal device 12 acquires an original face image.

[0054] In step S51, the terminal device 12 accepts input for selecting an algorithmic process. The control unit 123 of the terminal device 12 executes, for example, an application program that provides virtual makeup. The control unit 123 displays a selection menu on, for example, a display included in the output unit 126. Then, the control unit 123 selects the type of algorithmic process and the color to be applied to the facial image by the algorithmic process in response to a user's operation input to the input unit 125. The selected algorithmic process is the application of eyeliner, nose shadow, cheek color, lip color, or the like. The control unit 123 may display a reference facial image including makeup information as a sample, and the user may select the algorithmic process by selecting a sample.

[0055] In step S52, the terminal device 12 sends the original facial image and an image conversion request to the server device 10. The image conversion request includes information specifying the selected algorithm processing. The control unit 123 sends the original facial image and the image conversion request via the communication unit 121. In the server device 10, the control unit 103 receives the information sent from the terminal device 12 via the communication unit 101.

[0056] In step S53, the server device 10 executes algorithm processing on the original face image. The control unit 103 executes the designated algorithm processing on the original face image. As a result, the original face image is converted into a primary face image.

[0057] In step S55, the control unit 103 of the server device 10 converts the primary face image into a secondary face image using the image conversion model 108.

[0058] In step S56, the server device 10 sends the secondary facial image for output to the terminal device 12. The control unit 103 sends the secondary facial image via the communication unit 101. In the terminal device 12, the control unit 123 receives the information sent from the server device 10 via the communication unit 121.

[0059] In step S57, the terminal device 12 displays the secondary facial image. The control unit 123 causes the secondary facial image to be displayed on a display included in the output unit 126, for example.

[0060] According to the above-described procedure, it is possible to output a natural-looking face image that reproduces the texture of the original face image even after makeup is applied using the adjusted image transformation model 108. The user can check the face image of their own face with the specified makeup applied.

[0061] As described above, according to this embodiment, it is possible to perform image conversion that expresses various makeup techniques without damaging the attributes of the original facial image.

[0062] In the above description, the server device 10 corresponds to the "information processing device." However, the server device 10 and the terminal device 12 may cooperate to configure the "information processing device," or the terminal device 12 may correspond to the "information processing device."

[0063] In the above-described embodiment, the processing / control program that defines the operation of the terminal device 12 may be stored in the memory unit 102 of the server device 10 or in the memory unit of another server device, and may be downloaded to the terminal device 12 via the network 11, or may be stored in a computer-readable non-transitory recording / storage medium and read by the terminal device 12 from the medium.

[0064] Although the embodiments have been described above based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each means, step, etc. can be rearranged so as not to be logically inconsistent, and multiple means, steps, etc. can be combined or divided into one. [Explanation of symbols]

[0065] 10: Server device 11: Network 12: Terminal device 101, 121: Communications Department 102, 122: Storage section 103, 123: control unit 105, 125: Input section 106, 126: Output section 108: Image transformation model Isrc: Original face image Isyn: Primary facial image Ig: Secondary face image L: Loss function

Claims

1. A method for operating an information processing device, comprising: a first step of adding makeup information to a first facial image in accordance with a predetermined procedure to convert the first facial image into a second facial image; a second step of converting the first facial image into a third facial image using the second facial image as a reference image by using an image conversion model generated by machine learning a process of converting a facial image to be converted by adding makeup information included in a reference facial image; a third step of adjusting the image transformation model by reducing the loss of the first facial image in the third facial image; A method of operation including:

2. In claim 1, the predetermined procedure is a procedure of applying makeup information of one or more of hue, brightness, and saturation to a portion of the first face image including the eyes, the bridge of the nose, the cheeks, and the lips; How it works.

3. In claim 2, the predetermined procedure does not include machine learning; How it works.

4. In claim 2, the loss is a loss of one or more of hue, brightness, and saturation of the part in the first face image, or a loss of an edge of the part; How it works.

5. In claim 1, the image transformation model includes an encoder that extracts makeup information from the reference face image and a decoder that transforms the face image to be transformed; How it works.

6. a storage unit for storing an image transformation model, the image transformation model being generated by machine learning a process of transforming a target facial image by adding makeup information included in a reference facial image, and adjusted so as to reduce loss of the first facial image in the second facial image when the first facial image is transformed into the second facial image using a facial image to which makeup information has been added in a predetermined procedure as a reference image; a control unit that executes the predetermined procedure and then converts the input facial image into an output facial image using the image conversion model; An information processing device having the above.

7. A program executed by an information processing device, The information processing device can use an image conversion model that is generated by machine learning a process of converting a facial image to be converted by adding makeup information included in a reference facial image, and that is adjusted so as to reduce loss of the first facial image in the second facial image when the first facial image is converted into the second facial image using a facial image obtained by adding makeup information to a first facial image in a predetermined procedure as a reference image; a first step of executing the predetermined procedure on an input face image; a second step of converting the facial image subjected to the first step into an output facial image using the image conversion model; Including, the program.

Images