Virtual makeup solution providing system using generative AI

Abstract

Provided is a virtual makeup solution provision system using generative AI (artificial intelligence), comprising: a user terminal that captures a face and outputs a makeup image of virtual makeup to the face; and a makeup service providing server including: a receiving unit that receives the face image from the user terminal; a feature extraction unit that analyzes and classifies the facial image using a pre-constructed feature extraction model and outputs facial feature information; a color extraction unit that inputs the facial feature information to a generative artificial intelligence (AI) (Generative Intelligence) based on an LLM (Large Language Model), which has been constructed in advance, and extracts a makeup color corresponding to the facial feature information; and a virtual makeup unit that outputs a makeup image of a virtual makeup by using a diffusion-based generative formula AI in which the extracted makeup color is implanted in the facial image.

Description

Technical Field

[0001] This invention relates to a virtual makeup solution providing system utilizing generative AI (artificial intelligence), providing a system capable of extracting the optimal makeup color based on facial feature information and using diffusion-based generative AI to synthesize the makeup color onto a facial image, thereby providing a makeup image. Background Technology

[0002] Modern society is an era of advanced technological convergence, with various attempts to attract customers through innovative and intelligent services that integrate IT technologies. Technological innovation in the beauty industry is also accelerating, with the industry launching services and products utilizing artificial intelligence. In particular, with the rapid rise of a generation that values ​​personal taste as the main consumer group in the beauty industry, AI technology is being used to provide more precise and personalized beauty solutions. Unlike in the past when experience was merely a promotional tool in the beauty industry, it is evolving into a method of recommending customized products through facial image analysis.

[0003] Recently, methods for virtual makeup using artificial intelligence have been proposed. For example, Korean Patent Publication No. 2019-0116052 (published on October 14, 2019), which is prior art, discloses a technique that uses a pre-built deep learning model to extract facial feature points, diagnoses individual color based on these feature points, and then performs virtual makeup. Furthermore, Korean Patent No. 10-2515436 (announced on March 29, 2023) discloses a technique that, after acquiring a facial image, segments the face according to body parts, generates information for each body part, extracts makeup products for each body part, and then applies the product to each body part using a GAN (Generative Adversarial Network)-based synthesis method.

[0004] The former discloses a technology that diagnoses individual color based solely on facial features, rather than considering skin tone, eye color, or lip color to recommend the best makeup color. The latter discloses a technology that uses GANs to synthesize colors and faces; therefore, during the synthesis process, the face may differ from the original face, potentially resulting in a sense of incongruity or discomfort. Summary of the Invention

[0005] Technical issues

[0006] One embodiment of the present invention is proposed to solve the problems described above, and aims to provide a virtual makeup solution system that utilizes generative AI, taking into account not only facial shape but also facial color, thereby extracting the optimal makeup color and enabling the extracted makeup color to be naturally and seamlessly integrated with the original face.

[0007] Furthermore, this invention aims to provide a virtual makeup solution system that utilizes generative AI. When a user terminal captures a face, it extracts facial feature information from the facial image using a pre-built feature extraction model, inputs the facial feature information into a generative AI based on LLM (Large Language Model) to extract the makeup color corresponding to the facial feature information, and then inputs the extracted makeup color into a diffusion-based generative AI to generate a seamless makeup image.

[0008] However, the technical problem to be solved in this embodiment is not limited to the technical problem described above, and other technical problems may also exist.

[0009] Technical solution

[0010] According to an embodiment of the present invention, a virtual makeup solution providing system utilizing generative AI can be provided, comprising: a user terminal that captures a face and outputs a virtual makeup image to the face; and a makeup service providing server comprising: a receiving unit that receives the facial image from the user terminal; a feature extraction unit that analyzes and classifies the facial image using a pre-built feature extraction model and outputs facial feature information; a color extraction unit that inputs the facial feature information into a pre-built generative AI (generative artificial intelligence) based on LLM (Large Language Model) to extract makeup colors corresponding to the facial feature information; and a virtual makeup unit that uses diffusion-based generative AI to implant the extracted makeup colors into the facial image and outputs a virtual makeup image.

[0011] In addition, a virtual makeup solution provision system utilizing generative AI can be provided, wherein the pre-built feature extraction model uses BiseNet (Bilateral Segmentation Network for Real-time Semantic Segmentation) to identify facial contours and facial components, select facial feature points, analyze the length, proportion and angle of each part, and extract skin color, lip color and pupil color.

[0012] In addition, a virtual makeup solution provision system utilizing generative AI can be provided, characterized in that the pre-built feature extraction model uses ResNet (Residual Neural Network), which is a classification model that classifies the face based on the facial contour lines, facial constituent elements, facial feature points, the length, proportion and angle of each part, facial skin color, lip color and pupil color, and preset classification items.

[0013] In addition, a virtual makeup solution providing system utilizing generative AI can be provided, characterized in that the pre-built LLM-based generative AI is trained using a dataset of facial feature information and makeup colors through instruction tuning, and when the facial feature information is input into the LLM-based generative AI, it outputs keywords of the makeup colors.

[0014] In addition, a virtual makeup solution provision system utilizing generative AI can be provided, characterized in that the diffusion-based generative AI is stable diffusion as a latent diffusion model.

[0015] In addition, a virtual makeup solution provision system utilizing generative AI can be provided, characterized in that the diffusion-based generative AI converts the keywords of makeup colors output by the LLM-based generative AI into prompts to generate the makeup image.

[0016] In addition, a virtual makeup solution provision system utilizing generative AI can be provided, characterized in that the diffusion-based generative AI generates unit makeup images by segmenting the facial image into preset makeup regions, converting keywords of the makeup colors into prompts in the segmented makeup regions, and then re-inserting the unit makeup images into the facial image as the original image to replace them, thereby generating the makeup image.

[0017] In addition, a virtual makeup solution providing system utilizing generative AI can be provided, characterized in that the makeup service providing server further includes: a selection and reflection unit, which, after extracting the makeup color corresponding to the facial feature information, transmits the lip color from the makeup color to the user terminal; when the user terminal selects the desired lip color, it extracts and presents the eyeshadow color and blush color corresponding to the selected lip color based on a pre-built database.

[0018] Furthermore, a method for providing a virtual makeup solution utilizing generative AI can be provided. This method, running on a makeup service provider server, includes: receiving a facial image from a user terminal; analyzing and classifying the facial image using a pre-built feature extraction model to output facial feature information; inputting the facial feature information into a pre-built LLM (Large Language Model)-based generative AI to extract makeup colors corresponding to the facial feature information; and using diffusion-based generative AI to implant the extracted makeup colors into the facial image to output a virtual makeup image.

[0019] In addition, a method for providing a virtual makeup solution utilizing generative AI can be provided, characterized in that the LLM-based generative AI is a generative AI trained by instruction optimization using a dataset of facial feature information and makeup colors.

[0020] Invention Effects

[0021] According to one embodiment of the present invention, makeup colors that not only match the user's facial features but also the color of their face can be extracted and recommended. Moreover, even if the makeup product is not actually used, a seamless makeup image as if the user is actually wearing makeup can be generated. Thus, it can not only recommend the best makeup colors and products suitable for the individual but also guide accurate and rapid purchasing decisions. Attached Figure Description

[0022] Figure 1 This is a diagram illustrating a system for providing a virtual makeup solution utilizing generative AI according to an embodiment of the present invention.

[0023] Figure 2 It is used for explanation Figure 1 The system includes a framework diagram of a makeup service provider server.

[0024] Figure 3 This is a diagram illustrating the overall flow of a method for providing virtual makeup services according to an embodiment of the present invention.

[0025] Figures 4 to 22 This is a diagram illustrating an embodiment of a virtual makeup service according to an embodiment of the present invention.

[0026] Figure 23 This is an operation flowchart illustrating a method for providing virtual makeup services according to an embodiment of the present invention. Detailed Implementation

[0027] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement the invention. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. Furthermore, for the sake of accurate illustration, parts irrelevant to the description have been omitted from the drawings, and similar reference numerals have been used for similar parts throughout the specification.

[0028] Throughout this specification, when referring to a part as being "connected" to another part, this includes not only "direct connection" but also "electrical connection" with other components in between. Furthermore, when referring to a part as "including" a certain constituent element, unless specifically stated otherwise, it does not exclude other constituent elements, but rather means that other constituent elements are included. This should be understood as not precluding the existence or additional possibility of one or more other features, figures, steps, operations, constituent elements, devices, or combinations thereof.

[0029] Throughout this specification, the degree terms "about," "substantially," etc., when referring to inherent manufacturing or material tolerances, are used to mean approximately or equal to that value, and are intended to prevent unethical infringers from unfairly using disclosures that mention precise or absolute values ​​to aid in understanding the invention. The degree terms "(performed) ~ step" or "~ step" used throughout this specification do not mean "for the ~ step."

[0030] In this specification, "unit" includes a unit implemented in hardware, a unit implemented in software, and a unit implemented using both. Furthermore, one unit can be implemented using two or more hardware components, and two or more units can be implemented using one hardware component. Moreover, "unit" is not limited to software or hardware; it can also be configured to reside in addressable storage media or to power one or more processors. Thus, as an example, "unit" includes components such as software components, individual software components, class components, and task components; and processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided in components and "units" can be combined with or further broken down into additional components and "units" by a smaller number of components and "units". Furthermore, components and "units" can also be implemented as one or more CPUs within a boot device or secure multimedia card.

[0031] In this specification, a portion of an operation or function described as being performed by a terminal, device, or equipment may also be performed by a server connected to that terminal, device, or equipment. Similarly, a portion of an operation or function described as being performed by a server may also be performed by a terminal, device, or equipment connected to that server.

[0032] In this specification, what is described as part of an operation or function that maps or matches with a terminal can be understood as mapping or matching the terminal's inherent number or personal identification information as the terminal's identifying data.

[0033] The present invention will now be described in detail with reference to the accompanying drawings.

[0034] Figure 1 This diagram illustrates a system for providing a virtual makeup solution utilizing generative AI according to an embodiment of the present invention. (Reference) Figure 1 The virtual makeup solution provider system 1 utilizing generative AI may include at least one user terminal 100, a makeup service provider server 300, and at least one information provider server 400. However, this... Figure 1 The virtual makeup solution system 1 utilizing generative AI described in this paper is merely one embodiment of the present invention and therefore does not constitute a complete invention. Figure 1 This invention is interpreted in a limited way.

[0035] At this time, generally Figure 1 The various components are connected via a network 200. For example, such as... Figure 1 As shown, at least one user terminal 100 can connect to the cosmetic service providing server 300 via network 200. Additionally, the cosmetic service providing server 300 can connect to at least one user terminal 100 and at least one information providing server 400 via network 200. Furthermore, at least one information providing server 400 can connect to the cosmetic service providing server 300 via network 200.

[0036] Here, a network refers to a connection structure that allows information exchange between various nodes, such as multiple terminals and servers. Examples of such networks include Local Area Networks (LANs), Wide Area Networks (WANs), World Wide Web (WWWs), wired and wireless data communication networks, telephone networks, and wired and wireless television communication networks. Examples of wireless data communication networks include 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), 5GPP (5th Generation Partnership Project), 5G NR (New Radio), 6G (6th Generation of Cellular Networks), LTE (Long Term Evolution), WIMAX (Worldwide Interoperability for Microwave Access), Wi-Fi, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth networks, NFC (Near-Field Communication) networks, satellite broadcasting networks, analog broadcasting networks, DMB (Digital Multimedia Broadcasting) networks, etc., but are not limited to these.

[0037] In the following text, the term "at least one" is defined as a term that includes both singular and plural forms, and it will be apparent that even without the term "at least one," the constituent elements may exist singly or in multiples, and may refer to either singular or plural forms. Furthermore, the constituent elements may be singular or in multiples, which may vary depending on the embodiment.

[0038] User terminal 100 can be a user terminal that uses a webpage, application page, program, or application related to virtual makeup services to take a picture of the face and output a makeup image of the face as virtual makeup.

[0039] Here, the user terminal 100 can be implemented by a computer capable of connecting to a remote server or terminal via a network. This computer may include, for example, a laptop, desktop, or laptop computer equipped with a navigation device and a web browser. In this case, the user terminal 100 can be implemented by a terminal capable of connecting to a remote server or terminal via a network. User terminal 100, for example, as a wireless communication device that ensures portability and mobility, may include all types of handheld wireless communication devices such as navigators, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminals, smartphones, smartpads, tablet PCs, etc.

[0040] The makeup service provider server 300 can be a server that provides virtual makeup service web pages, application pages, programs, or applications. Alternatively, the makeup service provider server 300 can receive facial images from the user terminal 100, extract facial feature information using a pre-built feature extraction model, and then extract makeup colors corresponding to the facial feature information using a pre-built LLM-based generative AI. Furthermore, the makeup service provider server 300 can synthesize the extracted makeup colors into the facial image using diffusion-based generative AI, thereby minimizing any incongruity with the original image. Finally, the makeup service provider server 300 can be a server that provides the resulting makeup image to the user terminal 100.

[0041] Here, the makeup service provider server 300 can be implemented by a computer that can connect to a remote server or terminal via a network. This computer can include, for example, a laptop, desktop computer, or laptop equipped with a navigation device and a web browser.

[0042] Information providing server 400 may be a server that provides feature extraction models, LLM-based generative AI, and diffusion-based generative AI prototypes, or information or datasets for training such prototypes, to makeup service providing server 300, with or without utilizing virtual makeup service-related web pages, application pages, programs, or applications. Here, a prototype can be understood as AI or a tool trained using a dataset to achieve a solution according to an embodiment of the present invention. Information providing server 400 may be implemented by a computer that can connect to a remote server or terminal via a network. The computer may include, for example, a laptop, desktop, or laptop equipped with a navigator and a web browser. At least one information providing server 400 may be implemented by a terminal that can connect to a remote server or terminal via a network.

[0043] Figure 2 It is used for explanation Figure 1 The system includes a framework diagram of a server providing cosmetic services. Figure 3 This is a diagram illustrating the overall flow of a virtual makeup service provision method according to an embodiment of the present invention. Figures 4 to 23 This is a diagram illustrating an embodiment of a virtual makeup service according to an embodiment of the present invention.

[0044] refer to Figure 2 The makeup service provider server 300 may include a receiving unit 310, a feature extraction unit 320, a color extraction unit 330, a virtual makeup unit 340, and a selection and response unit 350.

[0045] When a cosmetic service providing server 300 or other servers (not shown) operating in conjunction with it transmit virtual cosmetic service applications, programs, application pages, web pages, etc., to a user terminal 100 and an information providing server 400 according to an embodiment of the present invention, the user terminal 100 and the information providing server 400 can install or open the virtual cosmetic service applications, programs, application pages, web pages, etc. Furthermore, scripts running in a web browser can be used to drive service programs in the user terminal 100 and the information providing server 400. Here, a web browser, as a program that supports the use of web (WWW: World Wide Web) services, refers to a program that receives and displays hypertext described in HTML (Hypertext Markup Language), such as Chrome, Edge (Microsoft Edge), Safari, Firefox, Whale, UC Browser, etc. Furthermore, an application refers to an application on a terminal, such as an application (App) running on a mobile terminal (smartphone).

[0046] Before describing in detail the various constituent elements according to an embodiment of the present invention, the general flow can be roughly described as follows:

[0047] Virtual Makeup Solutions Utilizing Generative AI

[0048] - Input the facial image into the feature extraction model to extract facial feature information.

[0049] - Input facial feature information into an LLM-based generative AI to extract makeup colors.

[0050] - Input makeup colors into a diffusion-based generative AI to generate makeup images.

[0051] refer to Figure 2 The receiving unit 310 can receive facial images from the user terminal 100. At this time, it is acceptable for the data transmitted to the receiving unit 310 to be video containing a face rather than a facial image. The user terminal 100 can capture a face and upload the captured facial image.

[0052] The feature extraction unit 320 can analyze and classify facial images using a pre-built feature extraction model, outputting facial feature information. According to an embodiment of the present invention, the feature extraction model may include: a morphological extraction model, which extracts various contour lines, facial constituent elements, and feature points to determine facial morphology; a color extraction model, which extracts skin color, lip color, and pupil color; and a classification model, used to classify the user's face based on the extracted facial feature information. Here, feature points may correspond to... Figure 5 The features shown are landmarks, but not limited to these. Various well-known models can be used as foundation models for each model. Firstly, the morphological extraction model can use ResNet to extract contours, facial components, and landmarks. BiseNet (Bilateral Segmentation Network for Real-time Semantic Segmentation) can also be used for face segmentation, but the models used for segmentation are not limited to this. Furthermore, for example, such as... Figure 8 As shown, the color extraction model can be a color extraction and classification model based on LAB color classification, such as light skin, red skin, natural skin, and bronze skin. At this time, methods for extracting skin tone, etc., include various well-known technologies, including the applicant's Korean Patent Publication No. 2023-0122242 (published on August 22, 2023), therefore detailed descriptions are omitted.

[0053] Classification models can categorize faces based on predefined classification criteria, including facial contours, facial features, facial landmarks, the length, proportions, and angles of different parts, skin tone, lip color, and pupil color. For example... Figure 7 As shown, facial shapes can be categorized based on their contours, such as oval, round, angular, oblong, or inverted triangle. Similarly, cheekbones can be classified as normal or lateral, based on their prominence and direction. Chins can be categorized as angular, normal, or pointed, and chin length as long, medium, or short. Eyes can be categorized as droopy or upturned, eyebrow spacing as wide or narrow, nose length as long or short, eyelids as double or single, and eye size as large or small. In this case, the standard values ​​used to distinguish between length and shortness, width and narrowness, etc., can be compared with... Figure 6 The test set (generated with 60 objects) is the same, but not limited to, this data. Of course, various other classification models can exist, so it is not limited to this.

[0054] The color extraction unit 330 can input facial feature information into a pre-built LLM-based generative AI (Generative Artificial Intelligence) to extract makeup colors corresponding to the facial feature information. At this point, the pre-built LLM-based generative AI, after being trained using a dataset of facial feature information and makeup colors through instruction tuning, will output keywords for the makeup colors when facial feature information is input into the LLM-based generative AI.

[0055] In one embodiment of the present invention, makeup colors can be extracted in two steps. A method can be used where a color corresponding to a preset area is recommended in the first step, and when that color is provided to the user and feedback is received, colors corresponding to other areas are recommended accordingly.

[0056] In one embodiment, after recommending lip color using facial feature information (first step), when the user terminal 100 selects a lip color, it recommends eyeshadow and blush colors corresponding to the selected lip color (second step). This is specifically shown in Table 1 below.

[0057] Table 1

[0058]

[0059] <LLM-based Generative AI>

[0060] LLM-based generative AI can be implemented using, but is not limited to, the open-source Dolly. In this case, Dolly is a commercially licensed Large Scale Language Model (LLM) from Databricks. LLM-based generative AI can also be instruction-tuned using a dataset of [facial feature information - makeup color]. Here, building the dataset may require preparation and preprocessing of data for each type of makeup solution, makeup guide, and makeup product. Referring to the two steps mentioned above, as follows... Figures 9 to 11 As shown, this could be a generative AI trained and optimized using datasets of [facial feature information - lip color] and [lip color - eyeshadow color - blush color]. At this point, as... Figure 12 As shown, facial feature information can be information expressing facial morphology, such as a pointed chin, small eyes, and a long nose, or information expressing facial color, such as natural skin tone, but is not limited to these.

[0061] The virtual makeup unit 340 can output a virtual makeup image by using diffusion-based generative AI to implant extracted makeup colors into a facial image. The term "implantation" is used here to highlight the synthesis achieved through the diffusion-based generative AI of this invention, in contrast to existing virtual makeup that overlays makeup colors onto a facial image. It is explicitly stated that the term "synthesis" may be used if its meaning is unclear.

[0062] User terminal 100 can output a makeup image for virtual facial makeup. In this case, diffusion-based generative AI can be considered as a stable diffusion model, similar to a latent diffusion model. Stable diffusion can be understood as a text-to-image AI model distributed under an open-source license.

[0063] This model is roughly composed of three artificial neural networks: CLIP, UNet, and VAE (Variational Auto Encoder). It works as follows: when a user inputs text, the text encoder (CLIP) converts the text into tokens, a language that UNet can understand. UNet then uses these tokens to de-noise randomly generated noise. Repeated de-noising generates a suitable image, and the VAE's role is to convert this image into pixels. Existing diffusion probability image generation models consume resources exponentially with increasing resolution. Unlike these existing models, this invention introduces an autoencoder at both the front and back ends, inserting or removing noise in a much smaller latent space rather than the entire image. Therefore, even when generating relatively high-resolution images, resource usage is significantly reduced, making it usable even with ordinary consumer graphics cards.

[0064] <Diffusion-based Generative AI>

[0065] According to an embodiment of the present invention, diffusion-based generative AI can convert keywords of makeup colors output by LLM-based generative AI into prompts to generate makeup images. Continuing with the above example, if suitable makeup colors for [pointed chin - small eyes - long nose - natural skin tone] are selected as [lip color - bright orange], [eyeshadow color - light orange], and [blush - light red], these can be converted into prompts to generate makeup images. At this time, a prompt in the form of "makeup with bright orange lip color, light orange eyeshadow, and light red blush" can be generated. Since the input items are the same or similar, a separate text-to-text generative AI can be prepared to generate prompts, allowing the prompt format to be customized, changing only the colors or products contained in the prompt to generate the prompt.

[0066] <Customization>

[0067] refer to Figure 13 In addition to using the makeup colors mentioned above to apply makeup (color) to facial images, you can also customize the makeup based on "reference" or "text" as needed, such as applying the same makeup as the reference image or applying makeup in the style of K-pop idols.

[0068] At this point, early stopping DDIM (Denoising Diffusion Implicit Models) inversion can be used to preserve facial structure and identity features while enabling extensive customization through various conditional inputs such as reference images, specific RGB colors, or text descriptions. This allows for the use of keyword suggestions for makeup colors from LLM-based generative AI outputs in image generation, as well as in techniques like makeup transfer. A model according to an embodiment of the present invention demonstrated significantly high-quality results when evaluated for makeup completion and realism by 15 makeup experts, thus validating its technical capabilities. For more details on DDIM, please refer to the paper (Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising diffusion implicit models." arXiv preprint arXiv:2010.02502 (2020)).

[0069] <Splitting-Makeup-Merging>

[0070] like Figure 14As shown, according to an embodiment of the present invention, diffusion-based generative AI can generate a makeup image by segmenting a facial image into preset makeup regions, converting keywords of makeup colors into prompts within the segmented makeup regions, generating unit makeup images, and then re-inserting the unit makeup images into the original facial image and replacing them. (Reference) Figure 14 It can be confirmed that all areas of the face except for the eyes, eyebrows, and lips, as well as the eye and lip areas, are separated. The eyeshadow application area should be larger than the eye area. To achieve this, as shown in the lower left corner, a workflow of [eye area - eye area dilation - mask translation - subtraction - gradient smoothing] can be used to provide a brush-like eyeshadow effect. The same applies to lip color or blush if the dilation or subtraction operation is removed. After the makeup for each part is completed, it is composited onto the original image, i.e., the facial image. This maintains the makeup appearance without significant changes compared to the original image. Here, skin represents the skin, lip represents the lips, and eyes represents the eyes. src is an abbreviation for the source of the original image, tgt is an abbreviation for the target image, i.e., the target of the makeup image, and α can refer to the weight of each part.

[0071] After the selection unit 350 extracts the makeup color corresponding to the facial feature information, it transmits the lip color from the makeup color to the user terminal 100. When the user terminal 100 selects the desired lip color, it extracts and presents the eyeshadow and blush colors corresponding to the selected lip color based on a pre-built database. Table 1 has already explained this, so it will not be repeated here.

[0072] The overall process according to an embodiment of the present invention can be summarized as shown in Figures 3(a) to 3(b). Figure 3 (d). The illustrations for each step are shown in Table 2 below.

[0073] Table 2

[0074]

[0075] Figures 15 to 19 This is an example of makeup application based on RGB. Figure 20 This is an example of makeup application based on a reference image. In this case, DMC (Diffusion Model Customization) is the makeup result according to an embodiment of the present invention. Further, Figure 21 Showing the results of changing hairstyles or lenses, Figure 22 The diagram reflects the level of makeup (light makeup). The result of heavy makeup.

[0076] As described above Figures 2 to 22 Matters not mentioned in the methods of providing virtual makeup services are consistent with those mentioned above. Figure 1 The description of the virtual makeup service delivery method is the same as or can be easily inferred from the description, therefore the following description is omitted.

[0077] Figure 23 This illustrates the use of an embodiment of the invention. Figure 1 The generative AI virtual makeup solution provides a diagram of the data transmission and reception process between the various components within the system. The following will illustrate this process. Figure 23 This describes one example of the process of sending and receiving data between various components; however, this document is not limited to the embodiments described above, and those skilled in the art should understand that... Figure 23 The process of sending and receiving data shown can be modified according to the various embodiments described above.

[0078] refer to Figure 23 The makeup service server receives facial images from the user terminal (S5100), analyzes and classifies the facial images using a pre-built feature extraction model, and outputs facial feature information (S5200).

[0079] In addition, the makeup service provider server inputs facial feature information into a pre-built generative AI (generative artificial intelligence) based on LLM (Large Language Model), extracts makeup colors corresponding to the facial feature information (S5300), and uses diffusion-based generative AI to implant the extracted makeup colors into the facial image to output a virtual makeup image (S5400).

[0080] The order of the steps (S5100 to S5400) described above is merely an example and is not limited thereto. That is, the order of the steps (S5100 to S5400) can be varied, and some steps can be run simultaneously or deleted.

[0081] As described above Figure 23 Matters not mentioned in the methods of providing virtual makeup services are consistent with those mentioned above. Figures 1 to 22 The description of the virtual makeup service delivery method is the same as or can be easily inferred from the description, therefore the following description is omitted.

[0082] pass Figure 23The virtual makeup service provision method described according to one embodiment can also be implemented in the form of a recording medium containing computer-executable instructions, such as a computer-running application or program module. A computer-readable medium can be any computer-accessible medium, including all volatile and non-volatile, removable and non-removable media. Furthermore, a computer-readable medium can include all computer storage media. Computer storage media includes all volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing computer-readable instructions, data structures, program modules, or other data-like information.

[0083] The aforementioned method for providing virtual makeup services according to an embodiment of the present invention can be run by an application that is installed by default on the terminal (which may include programs contained in the platform or operating system installed by default on the terminal), or it can be run by an application (i.e., a program) that the user directly installs on the main terminal through an application provider server such as an application store server, an application, or a web server related to the service. In this sense, the aforementioned method for providing virtual makeup services according to an embodiment of the present invention can be implemented by an application (i.e., a program) that is installed by default on the terminal or directly installed by the user, and recorded on a computer-readable recording medium such as a terminal.

[0084] The following is an example of an embodiment of the present invention.

[0085] Project 1 provides a system for a virtual makeup solution utilizing generative AI, comprising: a user terminal that captures a face and outputs a virtual makeup image to the face; and a makeup service server comprising: a receiving unit that receives the facial image from the user terminal; a feature extraction unit that analyzes and classifies the facial image using a pre-built feature extraction model and outputs facial feature information; a color extraction unit that inputs the facial feature information into a pre-built generative AI (generative artificial intelligence) based on LLM (Large Language Model) to extract makeup colors corresponding to the facial feature information; and a virtual makeup unit that uses diffusion-based generative AI to implant the extracted makeup colors into the facial image and outputs a virtual makeup image.

[0086] Project 2 provides a system for Project 1’s virtual makeup solution using generative AI. The pre-built feature extraction model uses BiseNet (Bilateral Segmentation Network for Real-time Semantic Segmentation) to identify facial contours and facial components, select facial feature points, analyze the length, proportion and angle of each part, and extract skin color, lip color and pupil color.

[0087] Project 3 is a virtual makeup solution system utilizing generative AI, which is based on Projects 1 and 2. Its characteristic is that the pre-built feature extraction model uses ResNet (Residual Neural Network). ResNet is a classification model that classifies the face based on the facial contour lines, facial components, facial feature points, the length, proportion and angle of each part, facial skin color, lip color and pupil color, and preset classification items.

[0088] Project 4 is a virtual makeup solution system that utilizes generative AI, as described in Projects 1 to 3. Its characteristic is that the pre-built LLM-based generative AI is trained using a dataset of facial feature information and makeup colors through instruction tuning. When the facial feature information is input into the LLM-based generative AI, it outputs keywords of the makeup colors.

[0089] Project 5 is a virtual makeup solution system that utilizes generative AI, as described in Projects 1 to 4. The system is characterized in that the diffusion-based generative AI is stable diffusion, which is a latent diffusion model.

[0090] Project 6 is a virtual makeup solution system that utilizes generative AI, as described in Projects 1 to 5. The system is characterized in that the diffusion-based generative AI converts the keywords of makeup colors output by the LLM-based generative AI into prompts to generate the makeup image.

[0091] Project 7 is a virtual makeup solution system utilizing generative AI, as described in Projects 1 to 6. The system is characterized by using diffusion-based generative AI to segment the facial image into preset makeup regions, convert keywords of the makeup colors into prompts within the segmented makeup regions, generate unit makeup images, and then reinsert these unit makeup images into the original facial image to replace the original image, thereby generating the makeup image.

[0092] Project 8 is a virtual makeup solution providing system utilizing generative AI, which is based on Projects 1 to 7. The system is characterized in that the makeup service providing server further includes a selection and reflection unit, which, after extracting the makeup color corresponding to the facial feature information, transmits the lip color from the makeup color to the user terminal. When the user terminal selects the desired lip color, it extracts and presents the eyeshadow color and blush color corresponding to the selected lip color based on a pre-built database.

[0093] Project 9 is a method for providing a virtual makeup solution using generative AI. As a makeup solution provision method running on a makeup service provision server, the method includes: receiving a facial image from a user terminal; analyzing and classifying the facial image using a pre-built feature extraction model to output facial feature information; inputting the facial feature information into a pre-built LLM (Large Language Model)-based generative AI to extract makeup colors corresponding to the facial feature information; and using diffusion-based generative AI to implant the extracted makeup colors into the facial image to output a virtual makeup image.

[0094] Project 10 is a method for providing a virtual makeup solution using generative AI, as described in Project 9. The characteristic of the method is that the LLM-based generative AI is a generative AI trained by instructions using a dataset of facial feature information and makeup colors.

[0095] The foregoing description of the present invention is merely illustrative. Those skilled in the art will understand that it can be easily modified into other specific forms without altering the technical concept or essential characteristics of the invention. Therefore, the embodiments described above should be understood as exemplary in all respects, not as limiting. For example, the constituent elements described as a single form can also be implemented separately; similarly, the constituent elements described separately can also be implemented in a combined form.

[0096] It should be interpreted that the scope of the invention is defined by the patent claims rather than the detailed description described herein, and all modifications or variations derived from the scope and meaning of the patent claims and their equivalents are included within the scope of the invention.

Claims

1. A system for providing virtual makeup solutions utilizing generative AI, comprising: The user terminal captures a face and outputs a virtual makeup image to the face; and A makeup service provider server includes: a receiving unit that receives facial images from the user terminal; The system includes a feature extraction unit that analyzes and classifies the facial image using a pre-built feature extraction model and outputs facial feature information; a color extraction unit that inputs the facial feature information into a pre-built LLM-based generative AI and extracts the makeup color corresponding to the facial feature information; and a virtual makeup unit that uses a diffusion-based generative AI that embeds the extracted makeup color into the facial image to output a virtual makeup image.

2. The virtual makeup solution providing system utilizing generative AI according to claim 1, wherein, The pre-built feature extraction model utilizes BiseNet to identify facial contours and facial components, select facial feature points, analyze the length, proportion, and angle of each part, and extract skin color, lip color, and pupil color.

3. The virtual makeup solution providing system utilizing generative AI according to claim 2, characterized in that, The pre-built feature extraction model uses ResNet, which is a classification model that classifies the face based on the facial contour, facial components, facial feature points, length, proportion and angle of each part, skin color, lip color and pupil color, and preset classification items.

4. The virtual makeup solution providing system utilizing generative AI according to claim 1, characterized in that, The pre-built LLM-based generative AI is trained and optimized using a dataset of facial feature information and makeup colors. When the facial feature information is input into the LLM-based generative AI, it outputs keywords related to the makeup colors.

5. The virtual makeup solution providing system utilizing generative AI according to claim 1, characterized in that, The diffusion-based generative AI is a stable diffusion that serves as a potential diffusion model.

6. The virtual makeup solution providing system utilizing generative AI according to claim 5, characterized in that, The diffusion-based generative AI converts the keywords of makeup colors output by the LLM-based generative AI into prompts to generate the makeup image.

7. The virtual makeup solution providing system utilizing generative AI according to claim 5, characterized in that, The diffusion-based generative AI generates unit makeup images by segmenting the facial image into preset makeup regions, converting keywords of the makeup colors into prompts in the segmented makeup regions, and then re-inserting the unit makeup images into the original facial image to replace them.

8. The virtual makeup solution providing system utilizing generative AI according to claim 1, characterized in that, The cosmetic service provider server also includes: The selection unit extracts the makeup color corresponding to the facial feature information and then transmits the lip color from the makeup color to the user terminal. When the user terminal selects the desired lip color, it extracts and presents the eyeshadow color and blush color corresponding to the selected lip color based on a pre-built database.

9. A method for providing a virtual makeup solution utilizing generative AI, which serves as a makeup solution providing method running on a makeup service providing server, the method comprising: The steps for receiving facial images from a user terminal; The steps of analyzing and classifying the facial images using a pre-built feature extraction model and outputting facial feature information; The steps involve inputting the facial feature information into a pre-built LLM-based generative AI to extract the makeup color corresponding to the facial feature information; and The step involves using diffusion-based generative AI to embed the extracted makeup colors into the facial image and outputting a virtual makeup image.

10. The method for providing a virtual makeup solution utilizing generative AI according to claim 9, characterized in that, The LLM-based generative AI is a generative AI trained by adjusting instructions using a dataset of facial feature information and makeup colors.

Images