Method for synthesizing an image using artificial intelligence and method for matching a hair stylist based on the synthesized image

Abstract

The present specification relates to an image transformation technique, in a method of synthesizing an image in which a hair model of a GAN (Generative Adversarial Networks) structure is learned using a plurality of learning data on a hairstyle, and an image including a user and a hair image of a new hairstyle are received, a synthesized image synthesized based on the user's image and the hair image masked using a mask for a hair region is generated using the learned hair model.

Description

TECHNICAL FIELD

[0001] The present specification relates to an image transformation technique, and more particularly, to an image synthesis method of obtaining a new photo based on a user's photo using machine learning, and a method of generating an image changing the user's hairstyle and matching the user with a suitable hair stylist based on the same. BACKGROUND

[0002] At present, there are various image transformation and synthesis techniques for generating a new image using an original image. Depending on what kind of data the original and new images are or the purpose of transformation or the degree of transformation, the type of technique selected also differs. In recent years, with the development of artificial intelligence technology, such artificial intelligence technology is also applied in image transformation and synthesis, and GAN (Generative Adversarial Networks) or an autoencoder disclosed in the following prior art document has become a representative means.

[0003] "Generative Adversarial Networks", Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, 2014.

[0004] In the image generation and transformation technique applying GAN, an artificial neural network receives various noise inputs to generate a new image that did not exist in the past or to convert into an image or a video having other forms or information. The conventional deep learning technique generally learns one multi-layered artificial neural network with respect to learning data, and GAN applies one generative neural network that finally generates a virtual image that is difficult to distinguish whether it is a real image through the mutual action of two artificial neural networks.

[0005] On the other hand, most of the image synthesis services sold in the current market are only capable of synthesizing a pre-selected fixed type of hairstyle, and thus the number of images derived in a similar style to the actual is limited. Although there is a research example in which a relatively more natural and excellent result can be derived in the case of synthesizing an image using a GAN model, it fails to reach the level of a practical service that takes a real person as an object. SUMMARY

[0006] TECHNICAL PROBLEM

[0007] An embodiment of the present specification is to solve the technical problem of solving the weakness of the conventional image synthesis technology of the fixed manner in which the synthesis type is limited, and solving the problem of obtaining an image of a desired form from an original photo of a practical user or losing the main features or information of the original photo in the case of applying a deep learning technology such as a GAN, and further overcoming the fact that the conventional technology of matching a user and a hair stylist is mostly focused on the condition of schedule or fee.

[0008] Solution to the technical problem

[0009] To solve the above technical problem, an image synthesis method of an image synthesis device including at least one processor of one embodiment of the present specification includes the steps of: the image synthesis device learning a hair model of a GAN (Generative Adversarial Networks) structure using a plurality of learning data about a hairstyle; the image synthesis device receiving an image of a user and a hair image of a new hairstyle; the image synthesis device masking the image of the user using a mask for a hair region; and the image synthesis device generating a synthesized image based on the masked image of the user and the hair image using the learned hair model.

[0010] In the image synthesis method of one embodiment, the step of learning the hair model includes the steps of: a generator receiving a latent vector in a latent space to generate a fake image; and a discriminator receiving the fake image and a real image to calculate a loss with respect to a difference therebetween, the generator learning in a manner to generate a fake image similar to the real image based on the loss, and the discriminator learning in a manner to discriminate whether the loss is within a threshold value based on the loss.

[0011] In the image synthesis method of one embodiment, the step of learning the hair model further includes the step of: generating a latent space in which similar hairstyle distributions are arranged in adjacent spaces using an encoder to invert semantic features of a hairstyle from actual images including a plurality of hairstyles.

[0012] In the image synthesis method of one embodiment, the discriminator described above includes a first discriminator that determines whether the virtual image and the real image are the same face, and a second discriminator that determines whether the virtual image and the real image are the same hairstyle, and losses calculated by the first discriminator and the second discriminator are provided to the generator to simultaneously guide learning of the face and the hairstyle. In addition, the first discriminator is learned on the basis of a plurality of face photos of the same person, and the second discriminator is learned on the basis of a plurality of hairstyle photos of the same hairstyle.

[0013] To address the above technical problems, a method for matching a hairstyle designer based on image synthesis, of another embodiment of the present specification, includes a matching system including at least one processor, the method including the steps of: the matching system receiving an image of a user; the matching system setting a desired hairstyle input from the user, generating a synthesized image synthesized according to the hairstyle from the image of the user using an image synthesis algorithm; and recommending a hairstyle designer corresponding to the hairstyle of the synthesized image generated by the matching system, the image synthesis algorithm learning a hair model of a GAN (Generative Adversarial Networks) structure using a plurality of learning data on hairstyles, receiving a hair image including an image of a user and a new hairstyle, masking the image of the user using a mask for a hair region, and generating a synthesized image synthesized based on the masked image of the user and the hair image using the learned hair model.

[0014] In the hairstyle designer matching method of another embodiment, the step of recommending the hairstyle designer includes the step of displaying at least one or more hairstyle designer candidates in consideration of at least one of the fields of practice and the history of a plurality of hairstyle designers.

[0015] In the hairstyle designer matching method of another embodiment, the step of recommending the hairstyle designer further includes the step of displaying at least one of the practice fees, the practice areas, and the possible practice dates of the displayed hairstyle designer candidates together, thereby guiding a practice appointment between the user and the hairstyle designer candidates.

[0016] In another embodiment of the hairstylist matching method, in the above-described image synthesis algorithm, a latent space in which similar hairstyles are distributed in adjacent spaces is generated by inverting semantic features of a hairstyle from actual images including a plurality of hairstyles according to an encoder, a generator receives a latent vector in the latent space to generate a fake image, a discriminator receives the fake image and a real image to calculate a loss with respect to a difference therebetween, the generator learns in a manner that the fake image similar to the real image is generated based on the loss, and the discriminator learns in a manner that whether the loss is within a threshold value is determined based on the loss, thereby learning the hair model.

[0017] In another embodiment of the hairstylist matching method, the discriminator includes a first discriminator that determines whether the fake image and the real image are the same face, and a second discriminator that determines whether the fake image and the real image are the same hairstyle, and the losses calculated by the first discriminator and the second discriminator are provided to the generator to simultaneously guide learning of the face and the hairstyle, the first discriminator is learned based on a plurality of face photos of the same person, and the second discriminator is learned based on a plurality of hairstyle photos of the same hairstyle.

[0018] On the other hand, a recording medium readable by a computer, which records a program for causing a computer to execute the above-described image synthesis method and the hairstylist matching method, is provided.

[0019] Effects of Invention

[0020] The embodiments of the present specification can generate a synthetic image in which a hairstyle is changed to a desired hairstyle from an actual photo of a user using a deep learning technique, and in particular, a hair model that masks a hair region and learns a face and a hairstyle separately can change only a hairstyle while preserving inherent appearance characteristics of the user, and an image synthesis technique is introduced in a platform that connects a user and a hairstylist, thereby enabling matching of a hairstylist based on a changed hairstyle of the user. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which are included to provide a further understanding of the present specification and are incorporated in and constitute a part of the detailed description, illustrate embodiments of the present specification and together with the detailed description, explain technical features of the present specification.

[0022] Figure 1is a diagram showing the basic idea of the image synthesis method disclosed by the embodiment of the present specification.

[0023] Figure 2 is a diagram showing the basic structure of a GAN (Generative Adversarial Networks).

[0024] Figure 3 is a diagram showing the process of image synthesis disclosed by the embodiment of the present specification in outline.

[0025] Figure 4 is a flowchart showing the method of synthesizing an image according to one embodiment of the present specification.

[0026] Figure 5 is a diagram showing the structure of the generator and discriminator for performing image synthesis according to one embodiment of the present specification.

[0027] Figure 6 is a diagram for explaining the hair model learning process according to one embodiment of the present specification.

[0028] Figure 7 is a flowchart showing the method of matching a hair stylist based on image synthesis according to another embodiment of the present specification.

[0029] Figures 8a to 12 is a diagram showing the process flow of an application embodying the hair stylist matching method according to another embodiment of the present specification.

[0030] Figure 13 is a block diagram showing a hair stylist matching system according to another embodiment of the present specification.

[0031] <Explanation of Symbols>

[0032] 10: Hair stylist (hair stylist terminal)

[0033] 20: User (user terminal)

[0034] 30: Matching system

[0035] 31: Communication unit

[0036] 32: Processor

[0037] 33: Memory DETAILED DESCRIPTION

[0038] Hereinafter, specific description will be given of the embodiments of the present specification with reference to the accompanying drawings. However, in the following description and the accompanying drawings, detailed description of well-known functions or structures that make the gist of the embodiments unclear is omitted. Meanwhile, throughout the specification, unless specifically noted otherwise, 'comprising' a certain constituent element does not mean excluding other constituent elements, but means further including other constituent elements.

[0039] In addition, the terms of first, second, and the like are used to describe various constituent elements, but the above terms do not have a limiting effect on the above constituent elements. The above terms are used to distinguish one constituent element from other constituent elements. For example, the first constituent element can be named as the second constituent element, and similarly the second constituent element can be named as the first constituent element, without departing from the scope of the claims of the present invention.

[0040] The terms used in the present specification are used only to describe specific embodiments, and do not have a limiting effect on the present specification. In the specification, unless explicitly defined otherwise, the singular expression includes the meaning of the plural. In the present application, the terms of 'including' or 'having' or the like are used to specify the existence of the described features, numbers, steps, actions, constituent elements, components, or combinations thereof, and do not exclude the existence or additional possibility of one or more other features or numbers, steps, actions, constituent elements, components, or combinations thereof.

[0041] Unless specifically defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as understood by a person skilled in the art in general. The meaning of the terms defined in advance in general is consistent with the meaning possessed in the related art, and in the present specification, it cannot be interpreted as an exceptional or overly formal meaning unless explicitly defined.

[0042] Figure 1 As a figure showing the basic idea of the image synthesis method disclosed in the embodiments of the present specification, a synthetic image (C) is generated by referring to a target image (B) from an original image (A). At this time, the original image (A) can be an actual photo of a user, and the target image (B) can be a photo different from the hairstyle of the original image (A). The finally generated synthetic image (C) can be a photo that reflects only the features of the hairstyle included in the target image (B) in the person (user) of the original image (A). For this, an operation of replacing information about the hairstyle in the original image (A) with information about the hairstyle in the target image (B) is required, and although such a change of the hairstyle is made, it is necessary to maintain the personal features of the user in the original image (A) in order to achieve a natural image synthesis.

[0043] Figure 2is a diagram showing a basic structure of a GAN (Generative Adversarial Networks). The GAN is a generative model in which two neural networks, a generator (Generator) 210 that learns a probability distribution and a discriminator (Discriminator) 230 that discriminates a set different from each other, learn from each other. The generator 210 is trained to create a virtual example that most deceives the discriminator, and the discriminator 230 is trained to most accurately discriminate the virtual example suggested by the generator 210 from an actual example. In this way, the generator 210 is trained in opposition to deceive the discriminator 230, and thus the GAN generates a similar product (a virtual product like an actual product) that is very similar to the actual example through a process in which the two neural networks develop in opposition to each other. By such a feature, the GAN is evaluated as being suitable for image generation or synthesis.

[0044] However, in the image synthesis technology using the GAN, several problems are found.

[0045] First, as exemplified in Figure 2 , when random noise is given as an input value to the generator 210, in the case of synthesizing an image using an actual photo of a user, it is difficult to generate an image of a desired form from the original photo. Therefore, it is necessary to constitute an input value before the generator 210, not simply from image information generated at random, but to extract a certain feature from an original photo and project the feature.

[0046] Second, a disadvantage is found in which the entire characteristics included in an image are all taken over. For example, considering a target as shown in Figure 1 , it is intended to deform only a hairstyle and a makeup, but a skin color, a makeup, other features of a face, and the like are also reflected in a synthesized image, and a synthesized image that somewhat differs from an original face is generated. As a result, the user feels that an unwanted feature that is not his or her own face is mixed in, and it is necessary to further supplement a weak point of a deformed image.

[0047] Third, a problem occurs in which, in image synthesis, a feature (for example, a hairstyle) of a context image cannot be maintained according to a scale of a target image. That is, it is necessary to appropriately regulate a feature portion included in a synthesized image according to a purpose.

[0048] Embodiments of the present specification developed in recognition of the above problems provide a technical means in which, as an input value, an actual photo of a user is used, and in saving a feature of a target image, only a feature to be changed is focused on among features of each region included in an image to synthesize.

[0049] Figure 3 is a diagram schematically illustrating a process of image synthesis disclosed in an embodiment of the present specification.

[0050] When an original image 310 of an actual photo of a user who takes a photo is input, a pre-processing process (cropping 320 to aligning process) is performed in consideration of the synthesis performance. For example, the original image 310 is a full-body photo or a photo in which a part of a region is biased within a photo region or mixed with a plurality of other objects, and in the view of the hairstyle synthesis targeted in the present embodiment, it is preferable to crop or even align so as to place only the face and hair region in the center.

[0051] Then, a mask 330 is received in such a manner that only a specific hair region within an image is designed in advance, and a masking process is performed on the image. At the same time, a hair image 340 to be transformed is received. At this time, the hair image 340 is different in hairstyle from the original image 310, and a target hairstyle to be changed by a user can be input.

[0052] Thus, a synthesized image 360 based on a hair model 350 learned in advance is generated from the image subjected to the masking process and the hair image 340 to be changed, and is output. Here, how to learn a deep learning model and apply it to image synthesis will be described later through Figures 5 to 6 and will be specifically described.

[0053] Figure 4 is a flowchart illustrating a method of synthesizing an image according to an embodiment of the present specification. From the viewpoint of embodiment, an image synthesis device including at least one processor can perform a process defined by each step of Figure 4 , and drive software including commands regarding each step through the above processor.

[0054] In the S410 step, the image synthesis device learns a hair model of a GAN (Generative Adversarial Networks) structure using a plurality of learning data regarding a hairstyle.

[0055] First, hairstyle images of Koreans, i.e., existing data, were collected as learning data for learning a hair model. In the embodiment, 500,000 photos were used, which were composed of 440,000 photos of just after a beauty treatment, 10,000 photos of hair tied up / hairstyles, and 50,000 photos of everyday styles. The photos were labeled and distinguished according to the same criteria and used as a data set or can be used according to the purpose of the service. All data sets were augmented by 100% through augmentation. The importance of a data set of data collected in a manner having relevance for the performance of a specific job is self-evident. In particular, the individual data types constituting the data set, the data form of the type, and the quality of the data greatly affect the artificial intelligence learning and even the prediction performance, and thus the data set disclosed in the embodiment will be specifically disclosed below.

[0056] [Table 1]

[0057]

[0058]

[0059] The data types of Table 1 are sequentially and respectively explained as follows.

[0060] 1) The beauty salon uniform data set can provide neat hair styles after a treatment in the form of an image (extension name png), an excel file (extension name csv), and a JSON (Java Script Object Notation) in the form of a JSON. JSON is a text-based standard format for representing data structured in a Javascript object syntax, and is used when data is transmitted in a web application. Exif data (Exchangeable Image File Format data) can be different in pattern due to exif tag version differences, and thus it is preferable not to use csv but to provide it in the form of a JSON.

[0061] 2) The beauty salon long tail data set can provide hair styles with low treatment possibility after a treatment in the form of an image (extension name png), an excel file (extension name csv), and a JSON. The long tail data is required when training an AI model, but it can also be inconvenient to use the data. The word 'long tail' from statistics refers to a phenomenon in which a plurality of events with low occurrence probability are distributed longer on one side of a statistical distribution, and the long tail greatly affects the design and operation of an AI system. Existing AI systems are particularly weak with respect to long tail data because they are not included in AI learning data requiring a large amount of data due to the low occurrence probability.

[0062] 3) Daily hairstyle dataset can be provided in the form of an image (extension png), an excel file (extension csv), JSON, a dataset that cannot be immediately distinguished from the style after returning from the beauty salon for more than 2 weeks, and the background and lighting of the photo are various and have much noise.

[0063] 4) Special hairstyle dataset can be provided in the form of an image (extension png), an excel file (extension csv), JSON, a hairstyle (pinned hair, baldness, very long hair, etc.) that is not performed in a beauty salon but is maintained by most people.

[0064] When designing the dataset, the most important thing to consider is data balance. It is necessary to be designed in such a way that the data is evenly distributed according to the appropriate classification criteria, thereby minimizing the data bias that can be expected during learning. In the present embodiment, the dataset is constructed in such a way that both trend and equal distribution are achieved at the same time with data that has a long tail effect in the part where the hairstyle with a lot of actual orders is present.

[0065] In addition, the hairstyle image collected in the present embodiment, i.e., the new collection data, is the technical application area, i.e., the hair salon and the hairstyle designer, which takes the customer's pre / post-processed photos, and maintains the same pattern information as the existing data (Korean hairstyle image). The file structure of the new collection data is exemplified as follows.

[0066] The "Annotation.csv" file can have a structure as in Table 2 below.

[0067] [Table 2]

[0068]

[0069]

[0070] Annotation refers to the work of adding respective metadata such as the object or image category used to explain the original data in the form of 'tags' to the dataset. That is, it corresponds to the work of displaying annotations on the original data, so that artificial intelligence can understand the content of the data. The explanation information data can display various forms and explanation information according to the functional purpose, and here the hairstyle name, hairstyle type, hair length, hair color, bangs, baldness degree, side style, age, front representative 2D shot, left and right angle, up and down angle, color, hair parting type, gender, special hairstyle distinction, Segment rgb average, etc. are provided in the form of csv.

[0071] The "Meta-Annotation.csv" file can have a structure as in Table 3 below.

[0072] [Table 3]

[0073] Name Type Explanation id categorical Temporary ID path string Picture file path source categorical Shooting set id collect-type categorical Purchase category author string Shooter collect-date timestamp Shooting date polygon1 json list Hair segment coordinates polygon2 json list Face segment coordinates before-after categorical Pre- / post-operation height integer Resolution vertical width integer Resolution horizontal device string Shooting equipment

[0074] Metadata, which is structured data about data, i.e., data for explaining other data, is data given to content according to a prescribed rule in order to effectively find and utilize sought information among a large amount of information. Metadata refers to information attached to data in order to analyze, classify, and add additional information to certain data, i.e., structured information. From the data side, it corresponds to labeling in order to explain data. Labeling refers to adding information about an object, i.e., metadata, when recognizing the object in an image, and can provide a path to a picture file of a hairstyle, a shooting set, a photographer, a shooting date, a hair-face segment coordinate, a resolution, a shooting device, etc. in csv form.

[0075] The "optional-Annotation.csv" file can have a structure as in Table 4 below.

[0076] [Table 4]

[0077] Name Type Explanation source categorical Shooting set id hair-width categorical Hair thickness water-repellency categorical Water-repellent hair natural-curl categorical Natural curl damage categorical Damage degree

[0078] Optional annotation, which is data providing additional explanation about hair, can provide a shooting set, hair thickness, hydrophobic hair, natural curl or not, damage degree, etc. in csv form.

[0079] The "exifData.csv" file can have a structure as in Table 5 below.

[0080] [Table 5]

[0081] Name Type Explanation path string File path

[0082] Table 5 can provide a path to stored data in csv form.

[0083] As described above, in order to solve the problem that details cannot be adjusted at the time of image synthesis, the embodiment of the present specification introduces an inversion process of generating noise that reflects features input to an image to be generated from an actual photo, while saving features of a target image, that is, semantic knowledge. That is, an encoder that converts an image (actual photo) into noise for image-to-image conversion is disclosed. The encoder can perform various functions of generating a latent vector that reflects features of an image and transforming poses, expressions, and the like of an image or interpolating two images to generate an averaged image. The present embodiment aims to derive a latent vector that focuses on features regarding a hairstyle.

[0084] Figure 5 is a diagram showing the structure of a generator and discriminators for performing image synthesis of one embodiment of the present specification, and more specifically shows a process of learning a hair model of Figure 4 (S410).

[0085] The generator 510 receives a latent vector in a latent space to generate a fake image. In addition, the discriminators 531, 533 receive the above-described fake image and a real image to calculate a loss with respect to the difference therebetween. The generator 510 learns in such a manner that the fake image similar to the real image is generated on the basis of the above-described loss, and the discriminators 531, 533 learn in such a manner that whether the above-described loss is within a threshold value is discriminated on the basis of the above-described loss.

[0086] However, unlike the GAN technology that generally has one discriminator, the embodiment of the present specification includes at least two discriminators 531, 533. When a hairstyle is changed using a conventional GAN under a target of changing a hairstyle, there is a problem that the face appearance of the user is also changed, and the discriminators of the present embodiment do not simply judge whether a synthesized fake photo is similar to a real photo, but also judge by distinguishing the face and the hairstyle in the photo, respectively. For this reason, the discriminators are divided into two, one discriminator is used to discriminate whether the face (person) of the predicted photo is the same as the face of the current user, and the other discriminator is used to discriminate whether the hairstyle of the predicted photo is the same as the target hairstyle.

[0087] Referring to Figure 5, Fig. 2 illustrates two discriminators, the first discriminator 531 performs the role of determining whether the virtual image and the real image are the same face, and the second discriminator 533 performs the role of determining whether the virtual image and the real image are the same hairstyle. Then, the losses calculated by the above-mentioned first discriminator 531 and the above-mentioned second discriminator 533 are provided to the generator 510 while learning the face and the hairstyle is simultaneously guided.

[0088] On the other hand, the above-mentioned first discriminator 531 needs to learn based on multiple face photos of the same person, so as a learning data set, multiple photos of the same person (e.g., person 1_photo 1, person 1_photo 2, person 2_photo 1, person 2_photo 2,...) can be received. In addition, the above-mentioned second discriminator 533 needs to learn based on multiple hairstyle photos of the same hairstyle, so as a learning data set, multiple photos of the same hairstyle (e.g., target hairstyle 1_photo 1, target hairstyle 1_photo 2, same hairstyle 2_photo 1, same hairstyle 2_photo 2,...) can be received.

[0089] From the above-mentioned two types of discriminators 531, 533, it can be seen that learning of respectively determining the identity of the face and the identity of the hairstyle needs to be performed. Therefore, in the above-mentioned Figure 4 In the S410 step of the image synthesis device, not only images related to the hairstyle, but also images related to the face are needed in the learning data needed to learn the hair model. For this reason, the learning data can include image data containing a face region for face learning, image data containing a hair region for hairstyle learning, and data for masking the hair region.

[0090] Figure 6 As a diagram for explaining the process of learning the hair model of one embodiment of the present specification, Fig. 3 illustrates learning using an encoder (Encoder) 610 and a decoder (Decoder) 630.

[0091] First, in order to extract features from actual photos, when one photo is input to the encoder 610, an encoded feature is output. Of course, at this time, the photo input is a photo related to the hairstyle, and only by being preprocessed and input, the hair model related to the target hairstyle can be learned. Then, the decoder 630 performs the action of reproducing the original photo by receiving the feature. When performing this series of processes multiple times for each photo, the encoded features of photos similar in hairstyle are displayed as adjacent points in the latent space and are learned, and in the case of photos different in hairstyle, they will appear as distant results in the latent space.

[0092] In summary, in the process of learning the hair model, the semantic features of the hair style are inverted from the actual images including a plurality of hair styles using the encoder 610, so that a latent space in which similar hair styles are distributed in adjacent spaces can be generated. Through such a process, the problem that the random noise cannot project the features of the actual photo in the conventional GAN technology can be solved.

[0093] In this way, after the learning of the encoder 610 is completed, even if any photo is input, the encoded features include information about the hair style, and thus according to the convenience of implementation, the features can be provided to the generator of the GAN.

[0094] When the learning of the hair model is completed according to the above-described process, it is returned to Figure 4 The remaining structure of the present embodiment will be described.

[0095] In step S430, the image synthesis device receives an image of a user and a hair image of a new hair style. At this time, the image of the user can be an actual photo in which various features about the appearance of the user are desired to be preserved as they are.

[0096] In step S450, the image synthesis device masks the image of the user using a mask for the hair region. In this process, the domain of the deformation is controlled in such a manner that various features about the appearance are preserved as they are in the actual photo of the user, and only the hair style is changed.

[0097] In step S470, the image synthesis device generates a synthesized image synthesized based on the masked image of the user and the hair image using the learned hair model. The hair model learned in the foregoing includes one generator and two discriminators, and in particular, the learning of the generator is simultaneously guided by the first discriminator that discriminates whether the face is the same or not and the second discriminator that discriminates whether the hair style is the same or not. Accordingly, the synthesized image generated by the hair model disclosed in the present embodiment can obtain a result of naturally reflecting only the hair style while preserving the features in the original image (the actual photo of the user) for the features other than the target hair style (for example, skin color or makeup).

[0098] Hereinafter, a platform application technology applying the image synthesis method for the hair style will be described.

[0099] Figure 7 is a flowchart illustrating a method of matching a hair style designer based on image synthesis according to another embodiment of the present specification. From the viewpoint of implementation, a matching system including at least one processor can performFigure 7 the processing procedure defined by each of the steps, and drives software including commands regarding each of the steps through the above-described processor. Regarding the processing procedure of image synthesis, the above is explained in detail, so here only a brief overview is given to omit repeated explanations. Figures 4 to 6

[0100] In the S710 step, the matching system receives an image of a user. For example, a user who wishes to change a hairstyle takes an actual photo of himself or herself and provides an image of the user to the matching system.

[0101] In the S730 step, the above-described matching system pre-learns a hair model applied in an image synthesis algorithm. Or the above-described matching system can receive only a result (hair model) learned through a physically separately separated other device.

[0102] In the S750 step, the above-described matching system sets a desired hairstyle input by the user and generates a synthesized image according to the above-described hairstyle from an image of the above-described user using an image synthesis algorithm. Here, in the above-described image synthesis algorithm, a hair model of a GAN (Generative Adversarial Networks) structure is learned using a plurality of learning data regarding hairstyles, and a hair image including an image of a user and a new hairstyle is received, the image of the user is masked using a mask for a hair region, and a synthesized image synthesized based on the masked image of the user and the hair image is generated using the learned hair model.

[0103] In addition, in the image synthesis algorithm, a latent space in which similar hairstyle distributions are distributed in adjacent spaces is generated using an encoder to invert semantic features of hairstyles from actual images including a plurality of hairstyles, a generator receives a latent vector within the latent space to generate a fake image, a discriminator receives the fake image and a real image and calculates a loss regarding a difference therebetween, the generator is learned in a manner to generate a fake image similar to a real image based on the loss, and the discriminator is learned in a manner to determine whether the loss is within a critical value based on the loss, so that the hair model can be learned.

[0104] ​Further, preferably, the above-described discriminator includes a first discriminator that discriminates whether the above-described virtual image and the above-described real image are the same face, and a second discriminator that discriminates whether the above-described virtual image and the above-described real image are the same hairstyle, and a loss calculated by the above-described first discriminator and the above-described second discriminator is provided to the above-described generator to simultaneously guide learning of the face and the hairstyle, the above-described first discriminator is learned based on a plurality of face photos of the same person, and the above-described second discriminator is learned based on a plurality of hairstyle photos of the same hairstyle.

[0105] In the S770 step, the matching system recommends a hair stylist corresponding to the hairstyle of the synthesized image generated in the S750 step. To this end, the matching system can be a cooperative platform that connects a hair salon or a hair stylist working at a hair salon with a user. That is, a plurality of hair stylists can be registered in the matching system, and a hair stylist who meets the user's requirements can be recommended by a project and various treatment conditions that can be performed by the hair stylist. In the case of selecting a hair stylist recommended by the user, a convenient function that can handle a treatment reservation and a settlement in one platform can be provided.

[0106] As described above, in the S770 step of recommending a hair stylist, at least one or more hair stylist candidates are displayed considering at least one of the treatment fields and the history of a plurality of hair stylists. Further, at least one of the treatment fees, the treatment area, and the possible treatment dates of the displayed hair stylist candidates are displayed together, so that a treatment reservation between the user and the hair stylist candidate can be guided.

[0107] Figures 8a to 12 FIG. 1 is a diagram illustrating a process flow of an application embodying a hair stylist matching method according to another embodiment of the present specification.

[0108] Figure 8a and Figure 8b A user interface of the matching application is illustrated. First, in the S710 step, the user takes a real photo of himself / herself and displays it on the screen to select various items to be changed in the current hairstyle. For example, the hair length, the wave, the hair shape, and even the hair color can be suggested as selection items. Second, in the S720 step, the user finds a photo of himself / herself from a terminal (for example, a smartphone) storage device and finds a photo of another person (for example, an entertainer) as a target and suggests it as a target photo. Then, the selection result is predicted and the synthesized image generated from the photo of himself / herself and the target photo is confirmed. This can be an image that predicts the result when a hair treatment is performed. Figure 8a Figure 8b

[0109] Figure 9 Figure 8a or​​​Figure 8b the selected item to display the composite image on the screen. With Figure 8a or Figure 8b comparing the original photo (user image) of Figure 9 the composite image of the same person's features while showing a very natural image of the change in hair length and wave volume. In this way, the user can select the designer search button in the screen of Figure 9 the matching service of the matching system (platform).

[0110] Figure 10 show the hair designers that can be matched according to the user's conditions. These hair designers are the hair designers that can perform the procedure on the composite image generated above, and according to the need, the results of the search condition added with the additionality can be displayed. For example, the history range or the degree of popularity desired by the user, etc. are added as the additional condition to display only the results after the screening. At this time, the user can select one hair designer to enter the detailed screen.

[0111] Figure 11 show the services that can be performed by the selected hair designer. The user selects at least one service provided by the hair designer to enter the reservation screen of Figure 12 .

[0112] Figure 12 show the available time of the selected hair designer. According to the need, as exemplified, each of the hair designers searched above is displayed on one screen, thereby guiding the user's other selection. In this way, the user specifies the available time to complete the reservation. According to the need, the screen about the settlement is entered to provide various options about the advance settlement.

[0113] Figure 13 As a block diagram of the hair designer matching system showing another embodiment of the present specification, the matching method of Figure 7 is reorganized from the viewpoint of the hardware structure.

[0114] The hair designer 10 can be a terminal held by the hair designer or a reservation terminal of a hair salon, and can be connected to the matching system 30 through the network.

[0115] The user 20 is connected to the matching system 30 through the network using a terminal or a PC held by the user, etc.

[0116] The matching system 30 has a communication unit 31 for connecting with the stylist 10 and the user 20 through a network, which mediates the user's salon matching and reservation. The matching system 30 loads or stores matching software including a command defining a series of processes for processing a matching request received from the user 20 into a memory 33, and has a processor 32 for executing the matching software loaded or stored into the memory 33.

[0117] The matching software receives an image of the user and sets a desired hairstyle received from the user 20, and generates a composite image regarding the above-described hairstyle from the above-described image of the user using an image synthesis algorithm, and recommends a command of the stylist 10 corresponding to the generated above-described composite image of the hairstyle. Here, the image synthesis algorithm learns a hair model of a GAN (Generative Adversarial Networks) structure using a plurality of learning data regarding a hairstyle, and receives a hair image including an image of the user and a new hairstyle, masks the above-described image of the user using a mask for a hair region, and generates a composite image synthesized based on the masked above-described image of the user and the above-described hair image using the learned above-described hair model.

[0118] The matching system prompted by Figure 13 stores personalized data using photo data obtained from a client, and stores a large amount of learning data regarding other hairstyles using a treatment result photo input by a stylist. In this case, the designer actively provides the result of his or her own treatment to the matching system for marketing purposes, so as to achieve the goal of showing the client, and from the viewpoint of the matching system, it can become an opportunity to obtain high-quality learning data.

[0119] On the other hand, the embodiments of the present specification can be embodied as computer-readable codes readable by a computer in a recording medium. The computer-readable recording medium includes all kinds of recording devices storing data readable by a computer system.

[0120] As examples of the computer-readable recording medium, ROM, RAM, CD-ROM, magnetic tapes, floppy disks, optical data storage devices, etc. are included. In addition, the computer-readable recording medium is distributed in a computer system connected by a network, and is stored and executed in a distributed manner. And the functional program, code, and code segment for embodying the embodiments can be easily derived by programmers in the technical field to which the present specification belongs.

[0121] The present specification has been described above with various embodiments thereof. It should be understood by those skilled in the art that various modifications can be made within the scope of the essential characteristics of the present specification, and each of the embodiments can be implemented in a modified form. Therefore, the disclosed embodiments should not be considered in a limiting sense, but rather in an illustrative sense. The scope of the present specification should not be defined according to the above description, but should be defined according to the claims, and all differences within the scope equivalent thereto are included in the present specification.

[0122] Industrial applicability

[0123] According to the above-described embodiments of the present specification, a synthetic image in which a user's actual photo is changed into a desired hairstyle can be generated using a deep learning technique, and in particular, a hair model that masks a hair region, learns a face and a hairstyle separately, changes only a hairstyle while maintaining one's own inherent appearance characteristics, introduces an image synthesis technique into a platform that connects a user and a hairstyle designer, and thus guides a hairstyle designer to match based on a user's changed hairstyle.

Claims

1. An image synthesis method of synthesizing an image by an image synthesis device including at least one processor, comprising the steps of: The image synthesizing apparatus learns a hair model of a GAN, i.e., a generative adversarial network structure, using a plurality of learning data including: image data of a face region for learning a face, image data of a hair region for learning a hairstyle, and data of a hair region subjected to a masking process; the image synthesis device receiving an image of a user and a hair image of a new hairstyle; the image synthesis device masking the image of the user using a mask of the hair region; and the image synthesis device generating a synthesized image based on the masked image of the user and the hair image using the learned hair model, in the step of learning the hair model, a generator generates a virtual image, a discriminator receives the virtual image and a real image to distinguish a difference therebetween, and both compete with each other to learn, the discriminator includes a first discriminator and a second discriminator to separately distinguish a face and a hairstyle within a photo to make a determination, wherein the first discriminator receives a plurality of face photos of the same person as a learning data set to learn whether the virtual image and the real image are the same face, and the second discriminator receives a plurality of hairstyle photos of the same hairstyle as a learning data set to learn whether the virtual image and the real image are the same hairstyle. 2.The image synthesis method of claim 1, wherein the step of learning the hair model includes the steps of: a generator receives a latent vector in a latent space to generate a virtual image; and a discriminator receives the virtual image and a real image to calculate a loss with respect to a difference therebetween, the generator learns in a manner that generates a virtual image similar to a real image based on the loss, the discriminator learns in a manner that discriminates whether the loss is within a threshold value based on the loss. 3.The image synthesis method of claim 2, wherein the step of learning the hair model further includes the step of: using an encoder, generating a latent space in which similar hairstyle distributions are adjacent to each other by inverting semantic features of a hairstyle from actual images including a plurality of hairstyles. 4.The image synthesis method of claim 2, wherein the discriminator provides losses calculated by the first discriminator and the second discriminator, respectively, to the generator to simultaneously guide learning of a face and a hairstyle. 5.The image synthesis method of claim 1, wherein the learning data includes: a first data set providing a hairstyle image immediately after a procedure; a second data set providing a hairstyle image of a low possibility of a procedure as long tail data; a third data set providing a daily hairstyle image that cannot immediately distinguish a hairstyle; and a fourth data set providing a hairstyle image maintained by most people without a procedure, the data sets are configured to include data of a part in which a hairstyle having a long tail effect has many actual orders. 6.A hair stylist matching method of matching a hair stylist based on image synthesis by a matching system including at least one processor, comprising the steps of: the matching system receiving an image of a user; the matching system setting a desired hairstyle input by the user, generating a synthesized image based on the hairstyle from the image of the user using an image synthesis algorithm; and recommending a hair stylist corresponding to the hairstyle of the synthesized image generated by the matching system, The above image synthesis algorithm learns a hair model using a GAN, i.e., a generative adversarial network structure, by masking a plurality of learning data, wherein the plurality of learning data including image data including a face region for face learning, image data including a hair region for hairstyle learning, and data on which a mask processing is performed on the hair region, and receiving an image of a user and a hair image of a new hairstyle, masking the image of the user using a mask on the hair region, generating a synthesized image based on the masked image of the user and the hair image using the learned hair model, a generator generating a virtual image, and a discriminator receiving the virtual image and a real image to distinguish a difference therebetween, both of which compete with each other to learn a hair model, the discriminator including a first discriminator and a second discriminator to separately distinguish a face and a hairstyle within a photo to determine, wherein the first discriminator receives a plurality of face photos of the same person as a learning data set to learn whether the virtual image and the real image are the same face; and the second discriminator receives a plurality of hairstyle photos of the same hairstyle as a learning data set to learn whether the virtual image and the real image are the same hairstyle. 7.The hair stylist matching method of claim 6, wherein the step of recommending the hair stylist includes the step of: displaying at least one or more hair stylist candidates considering at least one of a service field and a history of a plurality of hair stylists. 8.The hair stylist matching method of claim 7, wherein the step of recommending the hair stylist further includes the step of: displaying at least one of a service fee, a service area, and a service possible date of the displayed hair stylist candidate together to guide a service appointment between the user and the hair stylist candidate. 9.The hair stylist matching method of claim 6, wherein in the image synthesis algorithm, a semantic feature of a hairstyle is inverted from an actual image including a plurality of hairstyles using an encoder to generate a latent space in which similar hairstyles are distributed in adjacent spaces, a generator receives a latent vector in the latent space to generate a virtual image, a discriminator receives the virtual image and a real image to calculate a loss with respect to a difference therebetween, the generator learns to generate a virtual image similar to the real image based on the loss, the discriminator learns to determine whether the loss is within a threshold value based on the loss, and the losses calculated by the first discriminator and the second discriminator are provided to the generator to simultaneously guide learning of a face and a hairstyle to learn the hair model.

10. The method of claim 6, wherein, The learning data includes: a first data set providing hairstyle images immediately after a treatment; a second data set providing hairstyle images with low possibility of a treatment immediately after a treatment, which is long tail data; a third data set providing daily hairstyle images that are not immediately distinguishable from the hairstyle; a fourth data set providing hairstyle images that are maintained by most people without a treatment, and The data sets are configured in such a way that the data of the part of the hairstyle with a long tail effect and a large number of actual orders are included together.

Images