Digital makeup artist

Abstract

To provide a digital makeup artist, and a method for makeup advice and makeup instructions in an interactive manner.SOLUTION: A digital makeup artist system includes a mobile device. The mobile device includes a user interface 310 for interacting with a digital makeup artist 305. The digital makeup artist performs an interactive dialog with a user in order to capture needs of the user. An arithmetic circuit analyzes a face image of the user to identify face parts, analyzes the face image to determine facial characteristics, and generates image frames to be displayed in synchronization with the interaction with the digital makeup artist, based on the analyzed face image, the needs of the user, stored cosmetic routine information, common makeup looks, cosmetic products for skin types and ethnicity, and look preferences of the user.SELECTED DRAWING: Figure 13

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of priority to U.S. Non - Provisional Application No. 17 / 138,078, filed on December 30, 2020, U.S. Non - Provisional Application No. 17 / 138,143, filed on December 30, 2020, French Application Serial No. 2107909, filed on July 22, 2021, and French Application Serial No. 2107906, filed on July 22, 2021, the entire contents of which are hereby incorporated by reference.

[0002] The present disclosure is directed to digital makeup artists and methods for interactive makeup advice and makeup guidance.

Background Art

[0003] Mobile applications, or apps, have been developed in recent years to provide assistance in the search and selection for cosmetic purchases. The app can provide tools for searching for a specific type of makeup, or tools for searching for products that might be a user's favorite, or tools for simply purchasing a previously used product. Some apps assist in the color selection of lipsticks or eyeshadows by displaying color palettes. Some apps provide a color - matching function that helps search for colors that harmonize with the colors from clothing, accessories, or images. There are also apps that allow videos on how to apply a specific type of makeup to be available.

[0004] Some apps utilize the cameras of smartphones, tablets, and notebook computers to provide product trial applications. Some of these applications are implemented as web applications or as a single app. Some of these applications require the use of the camera to take a self-portrait with a smartphone camera, upload the photo to a web application, and apply virtual cosmetics to the uploaded image. These applications can provide various options such as smoothing the skin, raising the cheekbones, and adjusting the eye color. These applications can provide users with the function of adding any type and color of makeup or changing the intensity of the color.

[0005] However, the trial applications provided so far are for creating looks using photo editing tools. Some of the previous trial applications have a one-step function of overlaying makeup types and colors from the uploaded photos and then editing the makeup-applied photos. In addition, photo editing tools for mobile devices with cameras have become popular for use on social media. In VSCO, which is one of the photo sharing apps, photos can be edited or filtered before sharing. Many VSCO filters are available to obtain specific effects. VSCO filters may be composed of values applicable to photos, including exposure, temperature, contrast, fade, saturation, color tone, and skin tone. Such photo editing tools provide a function to edit photos, but photo editing does not provide a personal makeup experience. Also, the previous trial application tools do not provide the creation of custom looks. For example, a user may desire a date-night look. The previous trial applications may provide a date-night look, but do not provide advice on which cosmetics to use that the user may consider optimal and how to apply various types of cosmetics to create the look. Instead, in an attempt to obtain a custom look, the user may be able to perform some editing on the face image. Also, the user may desire a date-night look based on the user's mood or the mood the user wants to depict.

[0006] These prior trial web applications or apps lack complete personalization as they perform processes that are not customized makeup routines. The typical processes of prior virtual trial applications rely on templates and looks created for others. When a user desires a specific look in mind or wants to try a new look, the user may face the situation of having to edit a look created for someone else. There is a need to provide a custom trial experience for a specific user that enables interaction in a way comparable to the user's experience with a personal makeup artist. A makeup experience is required where a personal makeup artist teaches the user the steps to achieve the desired look on the user's face.

[0007] Social media apps have been developed to assist users in creating their anime-style avatars. These anime-style avatars can be customized in terms of hairstyle, hair color, face shape and color, makeup, eyebrows, nose shape, etc. However, users may want to post actual face makeup images on social media.

[0008] The foregoing description of the "Background Art" is intended to present the context of the present disclosure generally. The research of the inventors, now named as such, in the scope described in this "Background Art" section, and aspects of the description that may not be regarded as prior art in a different manner at the time of filing, are not admitted as prior art to the present invention, either explicitly or implicitly.

Summary of the Invention

Means for Solving the Problems

[0009] One aspect is a digital makeup artist system that includes a mobile device having a display device, an arithmetic circuit, and a memory, a makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and a database system for storing the user's look preferences, and a machine learning system for analyzing facial images, and the mobile device includes a user interface for interacting with a digital makeup artist, wherein the digital makeup artist performs a two-way interaction with the user to capture the user's needs including one or more types of makeup looks, indoor or outdoor looks, skin condition, facial problem areas, and favorite facial features. The arithmetic circuit is configured to input a user's facial image, analyze the user's facial image through a machine learning system that analyzes the facial image to identify facial parts, determine facial characteristics including one or more of skin tone, eye color, hair color, lip color, and skin texture, and generate an image frame to be displayed on the display device in synchronization with the interaction with the digital makeup artist. The image frame is generated based on the analyzed user's facial image, the user's needs obtained through the interaction with the user, one or more stored makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's look preferences.

[0010] One aspect is a digital makeup artist system that includes a mobile device having a display device, an arithmetic circuit, and a memory, a makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and a database system for storing the user's look preferences, and a machine learning system for analyzing a face image, and the mobile device includes a user interface for interacting with a digital makeup artist, wherein the digital makeup artist obtains initial information including one or more of a type of makeup look, an indoor or outdoor look, a skin condition, a problem area of the face, and a favorite facial feature of the user, and performs a two-way interaction with the user to provide advice including a request for makeup consultation. The arithmetic circuit is configured to input a face image of the user, analyze the face image of the user through a machine learning system that analyzes the face image to identify face parts, determine face characteristics including one or more of skin tone, eye color, lip color, hair color, and skin texture, and generate an image frame to be displayed on the display device in synchronization with the interaction with the digital makeup artist to provide advice. The image frame is generated in synchronization with the interaction based on the analyzed face image of the user, the initial information obtained through the interaction with the user, one or more stored makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's look preferences.

[0011] The foregoing summary of the exemplary embodiments and the following detailed description are merely exemplary aspects of the teachings of the present disclosure and are not limiting.

[0012] A more complete understanding of the present disclosure and many of its attendant advantages will be readily obtained by reference to the following detailed description when considered in connection with the accompanying drawings, as the present disclosure and its attendant advantages become better understood.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21A

Figure 21B

Figure 21C

Figure 21D

Figure 21E

Figure 21F

Figure 22A

Figure 22B

Figure 22C

Figure 22D

Figure 22E

Figure 23

Figure 24A

Figure 24B

Figure 24C

Figure 24D

Best Mode for Carrying Out the Invention

[0014] In the following detailed description, reference is made to the accompanying drawings which form a part hereof. In the drawings, like reference numerals generally refer to like components unless the context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the present disclosure.

[0015] Aspects of the present disclosure are directed to a digital makeup artist that can be consulted for makeup advice and guidance in a manner comparable to the experience a user can have with a personal makeup artist. The disclosed digital makeup artist provides a makeup experience that teaches the user the steps to achieve a desired look on their face.

[0016] Figure 1 is a schematic diagram of a system according to an exemplary aspect of the present disclosure. Embodiments include a software application or a mobile application (app). For the purposes of the present disclosure, hereinafter in this specification, the term mobile application (app) is used interchangeably with software application, and a makeup application will be used with reference to a process of applying makeup either virtually or physically. The software application can be executed on a desktop computer or a laptop computer 103. The mobile application can be executed on a tablet computer or other mobile device 101. For the purposes of the present disclosure, the software application and the mobile application are described from the perspective of the mobile application 111. In each case, the mobile application 111 can be downloaded and installed on each device 101, 103. In some embodiments, the desktop computer or laptop computer 103 can be configured to include a microphone 103a as an audio input device. The microphone 103a can be a device that connects to the desktop computer or laptop computer 103 via a USB port or an audio input port, or wirelessly via a Bluetooth wireless protocol. The mobile device 101 may include a built-in microphone. In some embodiments, the software application or the mobile application may include a communication function to operate in cooperation with the cloud service 105. The cloud service 105 may include a database management service 107 and a machine learning service 109. The database management service 107 can be of any type of database management system provided by the cloud service 105. For example, the database management service 107 may include a database accessed using Structured Query Language (SQL), or an unstructured database accessed by keys, generally referred to as No_SQL.Machine learning service 109 can execute machine learning to enable the scaling up and high-performance computing that may be required for machine learning. Also, a software application or a mobile application can be downloaded from cloud service 105. Although Figure 1 shows a single cloud service, a laptop computer, and a mobile device, it should be understood that any number of mobile devices, laptop computers, and desktop and tablet computers can be connected to one or more cloud services.

[0017] Figure 2 is a block diagram of a mobile computer device. In one implementation, the functions and processes of mobile device 101 can be implemented by one or more respective processing / arithmetic circuits 226. The same or similar processing / arithmetic circuits 226 can be applied to a tablet computer or a laptop computer. The processing circuit includes a programmed processor such that the processor includes the circuit. The processing circuit can also include devices such as application-specific integrated circuits (ASICs) and conventional circuit components arranged to perform the recited functions. Note that a circuit refers to a circuit or a system of circuits. In this specification, a circuit may be within one computer system or distributed throughout an entire network of computer systems.

[0018] Next, the hardware of processing / arithmetic circuit 226 according to an exemplary embodiment will be described with reference to Figure 2. In Figure 2, processing / arithmetic circuit 226 includes a mobile processing unit (MPU) 200 that executes the processes described in this specification. Processing data and instructions can be stored in memory 202. Also, these processes and instructions may be stored on a portable storage medium or stored remotely. Processing / arithmetic circuit 226 can have a removable subscriber identity module (SIM) 201 that includes information specific to the network service of mobile device 101.

[0019] Furthermore, the inventiveness of the present application is not limited by the form of the computer-readable medium in which the instructions of the present invention process are stored. For example, the instructions can be stored in a FLASH memory, a synchronous random access memory (SDRAM), a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a solid-state hard disk, or any other information processing device that the processing / arithmetic circuit 226 communicates with, such as a server or a computer.

[0020] Furthermore, the inventiveness of the present application can be provided as a utility application, a background daemon, or a component of the operating system, or a combination thereof, that runs in cooperation with the MPU 200 and the operating system. The operating system may be for a laptop computer or a desktop computer, such as the Mac OS, Windows 10, or a Unix operating system. In the case of a mobile device such as a smartphone or a tablet computer, a mobile operating system such as Android, Microsoft® Windows® 10 Mobile, Apple iOS®, or other mobile operating systems known to those skilled in the art may be used.

[0021] To implement the processing / arithmetic circuit 226, the hardware elements can be realized by various circuit elements known to those skilled in the art. For example, the MPU 200 may be a Qualcomm mobile processor, an Nvidia mobile processor, an Atom (registered trademark) processor of Intel Corporation in the United States, a Samsung mobile processor, or an Apple A7 mobile processor, or other processor types recognizable by those skilled in the art. Alternatively, the MPU 200 may be implemented using a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or discrete logic circuits such as those recognizable by those skilled in the art. Further, the MPU 200 may be implemented as a plurality of processors that perform parallel cooperative processing to execute the processing instructions of the present invention described above.

[0022] The processing / arithmetic circuit 226 of FIG. 2 also includes a network controller 206, such as an Intel Ethernet PRO network interface card of Intel Corporation in the United States, for interfacing with the network 224. As can be understood, the network 224 can be a public network such as the Internet, or a private network such as a LAN or WAN network, or any combination thereof, and can also include a PSTN or ISDN subnetwork. The network 224 can also be wired, such as an Ethernet network. The processing circuit can include various types of communication processors for wireless communication, including 3G, 4G, and 5G wireless modems, WiFi (registered trademark), Bluetooth (registered trademark), GPS, or other known wireless communication forms.

[0023] The processing / arithmetic circuit 226 includes a universal serial bus (USB) controller 225 that can be managed by the MPU 200.

[0024] The processing / arithmetic circuit 226 further includes a display screen controller 208, such as an NVIDIA (registered trademark) GeForce (registered trademark) GTX or Quadro (registered trademark) graphics adapter from NVIDIA Corporation in the United States, for interfacing with the display screen 210. The I / O interface 212 interfaces with buttons 214, such as for volume control. In addition to the I / O interface 212 and the display screen 210, the processing / arithmetic circuit 226 may further include a microphone 241 and one or more cameras 231. The microphone 241 may have associated circuitry 240 for processing sound into a digital signal. Similarly, the camera 231 may include a camera controller 230 for controlling the image capture operation of the camera 231. In an exemplary embodiment, the camera 231 may include a charge-coupled device (CCD). The processing / arithmetic circuit 226 may include an audio circuit 242 for generating an audio output signal and may include any audio output ports.

[0025] The power management and touch screen controller 220 manages the power used by the processing / arithmetic circuit 226 and touch control. The communication bus 222 may be an Industry Standard Architecture (ISA), Extended Industry Standard Architecture (EISA), Video Electronics Standards Association (VESA), Peripheral Component Interconnect (PCI), or the like, for interconnecting all components of the processing / arithmetic circuit 226. Descriptions of the general features and functionality of the display screen 210, the buttons 214, and the display screen controller 208, the power management controller 220, the network controller 206, and the I / O interface 212 are omitted in this specification for brevity since these features are well known.

[0026] FIG. 3 shows a user interface screen having an avatar according to an exemplary aspect of the present disclosure. For the purposes of the present disclosure, the disclosed avatar is a graphic representation of a digital makeup artist, but may be an image of makeup artist 305 and may represent a person who can communicate with the user using speech or text. The speech input can be activated by selecting an icon 311 representing a microphone. A sub-screen 301 including an avatar or image 305 may provide an area for interaction. The avatar or image 305 can output speech to the computer's audio output 242 or an external speaker connected to the computer 101. The avatar or image 305 can output text 309. The sub-screen may include an input box 307 in which the user can type text. In some embodiments, when the user speaks towards the microphone 241, the speech is converted to text and the text is displayed in the input box 307. The sub-screen 301 may be included in a user interface window 310. The window 310 may be a graphical object controlled by the operating system of the computer system 103. The window 310 may display a browser window or window of a software application or mobile application. The window 310 may include a menu icon 303, and selecting the menu icon 303 may display a menu of items that can be selected and perform a specified function.

[0027] Avatar 305 can be implemented as a software object that executes animations. In some embodiments, as further described below, Avatar 305 can be synchronized with the video played by Video Component 320. The control of Video Component 320 can be executed by voice commands using Microphone 241 when Microphone Icon 311 is activated. Avatar 305 can respond to speech input by performing natural language processing on the input and outputting a speech response through Speech Output 242. To synchronize the avatar with the video, Avatar 305 can transfer messages to Video Component 320, and Video Component 320 can share information such as timing information with Avatar 305. The video played by Video Component 320 can be divided into chapters. Chapters can be identified by chapter names and times. The time can be in seconds from the start of the video or as a percentage of the total length of the video. In some embodiments, the video is a series of makeup application steps in which the frames of the video are generated based on images of the user's face and the makeup applied. For the purposes of this disclosure, a makeup application step can include applying a specific type of makeup to a specific part of an image of the user's face. Specific makeup types can include specific cosmetics and specific characteristics of the cosmetics.

[0028] In some embodiments, Avatar 305 is implemented as an interactive agent that can receive questions and provide solutions to those questions. The interactive agent can also respond to specific types of questions asked by the user. The interactive agent can respond to statements made by the user that are responses to questions asked by the interactive agent. The interactive agent stores state information in Cloud Service 105. In some implementations, the interactive agent may be pre-created using software tools such as the Bot Framework SDK available in the Microsoft Azure cloud service. The operation of the interactive agent will become clearer in the following description.

[0029] Figure 4 shows a user interface screen for a mobile device according to an exemplary aspect of the present disclosure. In some embodiments, the user interface is provided for a smaller screen, such as the screen of smartphone 101. By providing a smaller screen, interaction with avatar 305 can occur through voice and / or text. Speech input can be initiated by selecting icon 311 representing a microphone. Controls and text related to guiding the interaction may be displayed on a portion of display screen 210. Avatar 305 can output speech to computer voice output 242 or an external speaker connected to computer 101. Avatar 305 can output text 309 to display screen 210. A portion of display screen 210 may include input box 307 in which the user can type text. In some embodiments, when the user speaks toward microphone 241, the speech is converted to text and the text is displayed in input box 307. Display screen 210 may display an image or video 401 of the user's face.

[0030] Figure 5 is a schematic sequence diagram for interaction between a user and a digital makeup artist for custom guidance, according to an exemplary aspect of the present disclosure. The digital makeup artist 520 can take the form of an avatar 305 or can be an interactive agent that is interacted with through speech. The digital makeup artist 520 can be executed by the mobile application 111 and can represent a fictional character or an actual makeup artist. The system 510 can be a desktop computer, a laptop computer 103, a tablet computer, a mobile device 101, or other computer system having a display function. The system 510 can be a combination of any of a desktop computer, a laptop computer, a tablet computer, or a mobile device and a cloud service 105. The system can have a voice function. The user input 530 can be in the form of a text input area and / or a microphone input.

[0031] Conventional makeup application guidance videos include basic video functions such as play, pause, and stop, and may include a time slider for moving the start time of video playback while the video is paused. In the disclosed embodiments, the digital makeup artist 520 is used to provide interaction in custom guidance.

[0032] The user can use the mobile application 111 for purposes such as use cases where the user has a look in mind and wants to show a digital makeup artist how that look can be created. The mobile application 111 can perform custom guidance for creating a look. At 501, the digital makeup artist 520 of the mobile application 111 can conduct an initial interaction with the user 530 to obtain some information about the user and in particular the user's needs. The digital makeup artist 520 can start the interaction by displaying and / or speaking an initial question such as "What type of look do you want to create?".

[0033] At 503, the user can state the desired type of look. The user's utterance can be input as text in the input block 307. Alternatively, the user can choose to interact with the digital makeup artist 520 via voice by selecting the microphone icon 311. The system 510 can extract keywords from the user's utterance for use in selecting a look pre-stored in the database 107. FIG. 6 is a user interface for the user to input an explanation of the type of look.

[0034] The user interface can provide a list of various types of looks. Examples of types of looks include, by way of a few examples, seasonal looks (spring, summer, autumn), event looks (date night, dinner with girlfriend, special date, out with mother-in-law, holiday, party, New Year's Eve, bridal, prom), looks based on time to complete (quick makeup, average makeup, time-consuming makeup), mood looks (cheerful, happy, notice me), styles (natural, evening, night out, gothic, work, beach), aesthetic looks (VSCO, eGirl, soft girl), etc.

[0035] The digital makeup artist 520 can further display and / or speak questions such as "What is your experience level in applying makeup?" to obtain information about the user's experience level. The user may be provided with a list of experience levels and select from it, or alternatively, input the experience level verbally. In one or more embodiments, information about the user 530 including the experience level can be found from the stored user profile.

[0036] The levels of experience can include beginner / novice level, experienced level, expert level, and professional. The beginner / novice level may be a user with little to no experience in applying makeup. The experienced level may be a user who has previously applied makeup and thus has some degree of experience. The expert level may be a user who has not only applied makeup for a certain period such as one year or more, but has also followed procedures to learn the proper way to apply makeup. The professional level may be a user who applies makeup to others.

[0037] The digital makeup artist 520 can further display and / or speak questions such as "Are your makeup looks mainly for indoor wear or outdoor wear?" to improve the types of makeup that can be recommended.

[0038] The digital makeup artist 520 can further display and / or speak questions such as "Is your skin dry?" to obtain information about the user's skin condition that may be difficult to obtain by image processing technology. Other questions may be related to obtaining information about the problem areas of the face and the user's favorite facial areas that the user would like to consider.

[0039] The digital makeup artist 520 can perform a custom guidance to teach steps for creating a user's desired look. The digital makeup artist 520 can teach the steps using the user's own face, and the steps can apply makeup in a way that creates a look that would be visible when applied to the user's face. At 505, the camera 231 can be used to capture an image or video of the user's face. The user can view the captured image or video on the display screen 210.

[0040] At 507, the system 510 can analyze the image or video to obtain information about the user's face. For example, the image can be analyzed to identify the face included in the image, determine the shape of the face, determine the facial parts, and determine other characteristics that can be used to provide advice during the guidance. Conventional image processing algorithms have been used to identify features of the face in the image, such as image segmentation, brightness level, average color, etc. However, in recent years, artificial neural networks in face recognition have been developed. Some of the artificial neural networks for face recognition are based on the structure of the human visual cortex. The architecture of the artificial neural network for computer vision continues to be improved. For the purposes of the present disclosure, two non-limiting artificial neural networks that can be used to analyze the captured image or video are described.

[0041] In some embodiments, provided that there are computer resources sufficient to execute the processing of the artificial neural network, one or both of the artificial neural networks may be executed by the mobile device 101 or the laptop computer 103. In some embodiments, one or both of the artificial neural networks may be executed by the machine learning service 109 within the cloud service 105. In some embodiments, the artificial neural network may be trained in the machine learning service 109, and the trained artificial neural network may be executed on the mobile device 101 or the laptop computer 103.

[0042] To perform face recognition, the mobile application 111 may execute image processing operations to improve the facial features of the image, such as improving the lighting. For example, the user may take a selfie without noticing when bright light or sunlight is coming from the direction behind the user. The mobile application 111 may brighten the user's face image. Other image processing operations may be executed to improve the image quality.

[0043] FIG. 7 is a flowchart showing the face analysis steps in more detail.

[0044] In 701, in order to determine the shape of the face, analysis can be performed on the captured image. A machine learning model can be used to detect the shape of the captured user's face. The machine learning model can be trained to classify face shapes using face images having known face shapes. In recent years, image classification has been performed using an artificial neural network inspired by the way the visual field of the human brain works when recognizing an object. This image classification artificial neural network is a group of models known as a convolutional neural network (CNN). Other approaches for image classification have also been proposed and are continuously improved. Examples of other approaches that may be used for such image classification include linear regression, decision trees, and support vector machines. As described above, the machine learning model can be trained using the machine learning service 109 of the cloud service 105.

[0045] In the disclosed embodiment, the architecture of the machine learning model that can be used to classify face shapes is a CNN. FIG. 8 is a block diagram of a CNN for classifying face shapes. The dimensions and activation functions of the CNN can be varied depending on the available processing power and the desired accuracy. The dimensions include the number of channels, the size of each layer, and the number of layers. Examples of activation functions include logistic and rectified linear units.

[0046] A convolutional neural network (CNN) may be composed of several types of layers. The convolutional component 803 may be composed of a convolutional layer 803a, a pooling layer 803c, and a rectified linear unit layer 803b. The convolutional layer 803a is for developing a two-dimensional activation map that detects specific locations of features at all given spatial positions. The pooling layer 803c functions as a form of downsampling. The rectified linear unit layer 803b applies an activation function to increase the decision function and the non-linear characteristics of the entire network without affecting the receptive field of the convolutional layer itself. The fully connected layer 805 includes neurons that have connections to all activations of the previous layer. The loss layer specifies how network training fills the deviation between the predicted layer and the true layer. The loss layer 807 detects classes in a set of mutually exclusive classes.

[0047] In some embodiments, the loss layer 807 may be a softmax function. The softmax function provides probabilities for each class. In an exemplary embodiment, the class 809 may include face shapes of square, rectangle, circle, ellipse, oval, rhombus, triangle, and heart shape.

[0048] In 703, the mobile application may analyze face features and landmarks. Similar to the face shape, the captured user's face features and landmarks may be detected using a machine learning model. The machine learning model may be trained to detect face landmarks. Similar to the classification of face shapes, other approaches to classification may be used. Also, a CNN architecture similar to FIG. 8 may be similarly used for face landmark detection.

[0049] Figure 9 is a schematic diagram of a deep learning neural network for face landmark detection. Similar to the architecture of FIG. 8, the deep learning neural network is a convolutional neural network. To improve the training of the deep learning neural network, residual connections may be included. In one or more embodiments, an inverted residual structure may be included in which a residual connection is made to a previous layer in the network. Regarding the architecture of FIG. 9, the network is provided as two stages, 903 and 905. The first stage 903 is a convolutional stage for performing feature extraction. In the second stage 905, predictions are made in the region of interest.

[0050] The architecture of the first stage 903 includes a convolutional section 903a that performs convolutional and max pooling operations when given an input face image 901. The convolutional section 903a is connected to an inverted residual structure 903b. A mask layer 903c is connected to the inverted residual structure 903b. The size of the mask layer 903c is a size corresponding to the number of landmarks (e.g., 2×L, where L is the number of landmarks). The mask layer 903c encodes the spatial layout of the input object.

[0051] The architecture of the second stage 905 includes an inverted residual structure 905b that is connected to the inverted residual structure 903b of the first stage 903. Also, the mask layer 903c of the first stage 903 is applied to the result of the inverted residual structure 905b and provided as an input for performing cropping of the ROI (region of interest) in the ROI and concatenate block 911. The ROI and concatenate block 911 is based on the number of channels and the number of landmarks of the inverted residual structure 905b. The prediction block 913 predicts the landmarks and approximate positions within the mask layer 905c. The prediction for the region of interest in the second stage 903 is combined with the landmarks estimated by the mask layer 903c for the entire image, and the output landmarks are obtained in the output layer 907.

[0052] In one or more embodiments, landmarks for a face include the eyes, nose, lips, cheekbones, the area around the eyes including the eyebrows and eyelids, and the hair. In some embodiments, the landmarks can include possible facial abnormalities.

[0053] The specific dimensions of each layer and the number of layers can depend on parameters including the desired accuracy, the hardware for running the machine learning model, and the length of time for training the machine learning model. The machine learning model can be trained using the machine learning service 109 of the cloud service 105.

[0054] The analysis 507 of facial features can further include detection of the lip shape 705, the eyelid shape 707, and the hairstyle 709. The detected landmarks can be used to calculate the contours of the lips, eyes, and hairstyle. In addition, other facial features such as skin color / tone, eye color, lip color, hair color 711, and skin texture 713 can also be determined from the face image. Skin color / tone and skin texture can be determined using image processing techniques. The skin color / tone can be analyzed to determine RGB coordinates and assign a name to the skin color / tone. Analysis of the skin color can reveal differences in skin tone. The types of skin tone can include fair, light, medium, or deep. Similarly, eye color, lip color, and hair color can also be determined using image processing techniques. Skin texture can be analyzed using analysis of differences in brightness.

[0055] A further feature of the face image can be illumination (brightness of the image). At 715, the image illumination (brightness) can also be determined using image processing techniques. Brightness can be defined as a measure of the total perceived light in the image. In some embodiments, the brightness of the image can be increased or decreased from the initially captured brightness level as a starting point.

[0056] At 717, past looks preferences can be retrieved from the database 107. Past looks preferences can include cosmetic characteristics such as the range of colors, color tones, and finishes used in past looks. Past user preferences can include cosmetic characteristics for specific parts of the face and can include options for cosmetics applied for a specific look, particularly for specific parts of the face.

[0057] At 509, one or more makeup filters can be selected / retrieved from the database 107 based on the facial features (parts of the face) determined by face analysis (507 and FIG. 7) and past looks preferences. Some of the stored makeup face filters can be filters previously created through custom guidance with the digital makeup artist 520 or generation of a custom look during a makeup consultation.

[0058] A makeup filter is a feature mask that can be blended with a base frame, i.e., an image having a front view of the user's face. The feature mask is blended with the base frame to generate an image having a resulting face image with modified features. The app can adjust the feature mask to align with the facial features in the base frame using the positions of the boundaries of the facial features determined using face analysis 507. The feature mask consists of RGB pixel values.

[0059] One or more makeup filters can be retrieved from the database 107 using a recommender system. FIG. 10 is a schematic diagram regarding the recommender system. The recommender system 1000 can be used to retrieve makeup filters used when showing a method of applying virtual makeup (515 in FIG. 5). The recommender system 1000 operates based on the image data that can be stored in the database 107 and the indexed database 1005 of makeup filters.

[0060] The indexed database 1005 may also include answers to common questions regarding makeup, common makeup looks, cosmetics according to a specific skin type (skin tone, skin dryness, skin texture) and / or a person's specific ethnicity. Makeup information extracted from external databases or websites, such as product review information including evaluations and public comments, may be added to the indexed database 1005.

[0061] The indexed database 1005 may further include categories of cosmetics, including categories of makeup that provide combinations of functions such as appearance and skin care. Cosmetics in this category may include foundations with anti-aging quality or sun protection, and cosmetics blended with drugs or other skin treatment products. Another category may be skin care products for a skin care routine to be performed before applying makeup.

[0062] The recommender system 1000 includes a recommendation engine 1007 that searches for and ranks recommended makeup filters. When applying a specific virtual makeup, the recommended makeup filters may be for the looks and virtual makeup that the user entered in step 501. In some embodiments, the recommended makeup filters may be searched based on the user's preferences or favorites. The personal user preferences may be makeup characteristics entered by the user when the app was first set up. The personal user preferences may be one or more makeup filters previously constructed during a custom guidance or consultation with the digital makeup artist 520. Favorites may be makeup characteristics flagged by the user as favorites. The personal user preferences and favorites may be for a specific part of the face or for the entire face.

[0063] In one or more embodiments, the recommendation engine 1007 may use a looks feature matrix. FIG. 11 shows a non-limiting matrix of exemplary looks features according to an exemplary aspect of the present disclosure. The matrix of looks features shown in FIG. 11 is a partial matrix showing two types of virtual makeup for the sake of brevity. Other types of virtual makeup that may be included in the matrix of looks features include, but are not limited to, foundation, mascara, concealer, blush, and eyebrow pencil. The matrix of looks features may be stored in an app within the mobile device so as to be compared with a vector of desired features. The desired features may be the current preference vector of the user and may take into account the user's current experience level and desired look (501). The recommendation engine 1007 may rank the recommendations using one or more similarity metrics and scoring algorithms. In one embodiment, the recommendation engine 1007 may generate a series of features to enhance the recommendations in order to promote creativity by changing specific characteristics for the virtual makeup from those recommended. For example, if the recommendation engine 1007 ranks a recommendation at the top among the recommendations searched, one or more characteristics may be changed to increase the similarity score. Alternatively, the recommendation engine 1007 may change one or more characteristics in the searched recommendations, such as the color tone or finish, up or down by one level (e.g., change the color tone up or down by one level from the stored color tone). In one or more embodiments, the recommendation engine 1007 may adjust the application gestures to be approximately accurate based on the user's experience level.

[0064] In one or more embodiments, the recommendation engine 1007 can be implemented by or complemented with a machine learning model. The machine learning model can be trained to select foundation shades according to a specific skin undertone, skin type, and / or ethnicity. The machine learning model can be trained to select lipstick shades based on the color of the user's lips. The machine learning model can be trained to select eyeshadow shades based on the color of the user's eyes and skin undertone.

[0065] The machine learning model for the recommendation engine 1007 can be trained using the data of the indexed database 1005 as well as data from external databases, particularly external databases having images and videos that the user can publicly disclose their custom looks. Exemplary external databases include, by way of example, databases in publicly available social media platforms such as Facebook, Instagram, Snapchat, TikTok, and video conferencing platforms such as Zoom, Microsoft Teams, Google Hangouts, or Google Meet.

[0066] The machine learning model for the recommendation engine 1007 can be based on algorithms for relatively small datasets, including decision trees, random forests, or single-layer neural network - perceptrons. By providing a large dataset consisting of thousands of training data examples, it becomes possible to use a deep learning neural network as the machine learning model. The architecture of the deep learning neural network may be a variant of a convolutional neural network for recognizing features of color images.

[0067] The recommendation engine 1007 can output one or more recommendations to a recommended user interface (511). The recommended user interface can also display a series of video frames showing the application of the selected recommendation.

[0068] At 511, one or more looks options may be displayed on the display screen 210.

[0069] FIG. 12 is an exemplary user interface in a mobile application. A digital makeup artist 520 may display custom recommendations 1201 of various looks. At 513, the user may select a recommended look. The look may be selected by a pointing device that moves a pointer on the display screen 210, a pointing device on the touch screen 221, or by entering a voice command such as "select look 1". There may be more custom recommendations that can be displayed on the display screen 210. The user may be able to display other recommended looks, for example, by selecting a function to display the next screen of looks. In one embodiment, the user interface may provide a scroll bar 1203 that may enable scrolling to display additional looks.

[0070] When the user selects a look, at 515, the system 510 may begin executing steps to create the selected look. FIG. 13 shows a user interface that may be displayed to initiate guidance in a mobile application according to an exemplary aspect of the present disclosure. During guidance, at 517, information may be fed back to the system 510 to improve future interactions. Also, the guidance is interactive. The guidance may be controlled using commands including "SLOW DOWN", "PAUSE", "RE-DO A STEP", "SKIP A STEP". A message 1301 may be displayed to inform the user that commands can be used to control the guidance. The commands are selectable on the user interface using a pointing device or, if the microphone 241 is active (e.g., via the microphone icon 311), can be input as verbal commands. The guidance being played can be modified to obtain a custom guidance experience. For example, the command "RE-DO A STEP" may include options for modifying the guidance. The system 510 can save the changes made to the guidance as look preference data that can be used to improve future interactions in the database 107. As described above, the version of the mobile application 111 may be for a device having a smaller screen than a laptop computer or a desktop computer. FIG. 14 is an exemplary mobile application according to an exemplary aspect of the present disclosure. With a smaller screen provided, the interaction with the digital makeup artist 520 may be via voice and / or text. In that case, the commands can be input verbally. During the guidance, a face image of the user 1401 may be displayed, and the guidance can be added to apply makeup 1403a of color 1403b selected from the makeup palette 1403 to the part 1405 of the face image 1401.

[0071] Figures 15, 16, 17, and 18 are flowcharts for video control commands. As described above, guidance can be controlled using commands including "deceleration", "temporary stop", "redo step", and "skip step". Figure 15 is a flowchart for the "deceleration" video control command. The "deceleration" command can execute function S1501 to decrease the playback speed by a certain amount (X%). Additionally, the invocation of the "deceleration" command may indicate that the user believes the portion of the video being played is complex or requires careful viewing. The execution of the function to decrease the playback speed may include storing information indicating that the step is complex at S1503.

[0072] Figure 16 is a flowchart for the "temporary stop" video control command. The "temporary stop" command can execute function S1601 to stop the playback of the video at a certain time point (position in seconds or time code).

[0073] Figure 17 is a flowchart for the "redo step" video control command. The "redo step" command can execute a function that utilizes chapters. In some embodiments, the start of each chapter may include a start image frame and a time point. Alternatively, the start of each chapter may be marked by a chapter frame. The "redo step" command can execute a function that can return to the beginning of the current chapter and start playback from the start image of that chapter. At S1701, that function may include reading the current chapter name. At S1703, that function may store information indicating that the steps associated with the current chapter are complex. At S1705, that function can read an identifier indicating the start position of the chapter, which may be a time point or a chapter frame.

[0074] As described below, in some embodiments, video frames can be dynamically generated, for example, before the playback of a chapter or as the video frames are being played. By dynamically generating video frames, changes can be incorporated into the video, such as changing to use different color tones and / or finished cosmetics. The start of a chapter can include a start image that can be a combination of a face image and a mask filter. The makeup application performed during the playback of a chapter can be saved as the next state of the face image combined with the mask filter.

[0075] In S1707, the video can be played from the beginning of the chapter.

[0076] Figure 18 is a flowchart of a "step skip" video control command. The "step skip" command can execute a function that can skip the video to the start of the next chapter. This function can start in S1801 by reading the name of the current chapter. Skipping a step may indicate that the step is simple and that there is no need to show the user how to execute the step. In S1803, this function can store information indicating that the step is simple. In S1805, this function can read the name of the next chapter and, in S1807, can read the time point associated with the next chapter. In S1809, this function can reset the video to start from the time point of the next chapter.

[0077] FIG. 19 is a block diagram of a video playback component according to an exemplary aspect of the present disclosure. In some embodiments, the video playback component 320 may generate a video for guidance before playing a video. In particular, the video playback component may perform operations on one or more frames in the guidance video before rendering the frames for display. Also, the guidance video may be divided into chapters. The chapters may be used to divide the makeup guidance video into individual steps, and each step of the chapter may not only be played but also paused or stopped. Each chapter may be designed by a start frame. The chapters of the video may be for steps of applying makeup to a specific part of the user's face or for applying a specific type of makeup. The video may start from an image of the user's face captured by a camera. The video may utilize information obtained through analysis of the face image, such as the position and label of a portion of the user's face image, color, texture, lighting, etc. Also, the video may utilize looks selected by the user. In an initial setting, the video may be for guidance on how to create a look selected using the user's face image and information obtained from face analysis.

[0078] As described above, a chapter of a video may be for a step of applying makeup to a specific part of a user's face and may be for applying a specific type of makeup. For example, a chapter may be for a step of applying concealer to the eyelids of a user's face image. The concealer may be a cosmetic provided for a selected look. In some embodiments, makeup looks may have a set of chapters, each chapter may include one or more cosmetics, and the cosmetics may have a set of characteristics such as color, range, tone, and finish. The video playback component 320 may generate a selected look for a specific user's face and cosmetics. A chapter may start with an image of the user's face made up with previous chapters so that the chapter represents the cumulative result of the makeup application. In some embodiments, the cumulative result is one or more mask filters separate from the original face image.

[0079] With respect to FIG. 19, a face image, or a mask filter of face image 1903, may be provided at the start of a chapter as a starting image for further digital makeup to be applied for guidance. The start of a chapter may include position information 1907 of the part of the face (face feature) where the cosmetics are to be applied, and information regarding the cosmetics, their characteristics, and the type of strokes that may be applied by a specific type of makeup applicator. A chapter may include an accompanying audio component 1901. A chapter may include a series of mask filters 1905 of a desired facial expression that may be used to create video frames for guidance. Each video frame may be generated by blending a previous frame 1911 with a desired facial expression 1913 to obtain a resulting facial expression 1915. One or more feature masks representing applying a specific makeup using a makeup applicator for a specific facial feature may be used.

[0080] Figure 20 shows a blending process that can be used to create video frames based on a desired face pose and an original face pose. Blending of facial features is achieved as follows. 1. The desired face pose 2001 is recolored (2003) to match the color(s) of the original face pose, obtaining a recolored face pose 2005. 2. A feature mask filter 2007 is applied to the recolored face pose 2005. 3. The inverse value 2011 of the feature mask filter (i.e., the value obtained by subtracting each mask value in the range from 0 to 1 from 1) is applied to the original face pose 2009. 4. The resulting images from 2 and 3 are added pixel by pixel (2013) to create the final blended face pose image 2015.

[0081] When the guidance is completed, or when the user has completed as much guidance as they want to achieve, the user interface may provide options for saving the completed look. If the user selects to save the completed look, at 519 "Yes", at 521, the look can be saved as the completed look, or the steps taken to apply makeup during the guidance can be saved as a custom filter, or a series of custom filters. Additionally, at 523, the user interface may provide options (525) for transferring the custom look or applying / transferring the custom filter as an image to another platform.

[0082] Exemplary operation of the system is provided in FIGS. 21A-21F. FIGS. 21A-21F are sequence schematic diagrams for an exemplary use case of an exemplary dialogue using digital makeup artist 520 for makeup guidance, in accordance with an exemplary aspect of the present disclosure. The sequence schematic diagrams of FIGS. 21A-21F include operations and communications by system 510, digital makeup artist 520, and user 530. As described above, system 510 can be a computer device such as, by way of example, a desktop computer, laptop computer 103, tablet computer, or mobile device 101. Also, system 510 can take the form of a combination of a computer device and cloud service 105 (see FIG. 1). In either case, digital makeup artist 520 can be executed as part of mobile application 111, and user 530 can interact with digital makeup artist 520 through any of speech, text, or a combination of speech and text.

[0083] User 530 may first select a mobile application executed in the user interface of a computer device. When the mobile application is launched, User 530 may be given the option to use Digital Makeup Artist 520 to receive instruction or to receive advice in a makeup consultation session (described below). If the user selects to use Digital Makeup Artist 520 for instruction, at 2101, Digital Makeup Artist 520 may ask, "What kind of look do you want to create?" At 2103, User 530 may state the name of the look. The look may be an existing type of look or the name of a new look that User 530 has in mind. Alternatively, System 510 may provide a list of existing looks for User 530 to select from. The existing looks may be stored in Database 107 or may be provided locally in App 111.

[0084] In some embodiments, at 2105, Digital Makeup Artist 520 may ask User 530 to provide an experience level in applying makeup. The experience level may be categories such as beginner, experienced, expert, and professional. System 510 may present a list of experience levels for User 530 to select from. At 2107, User 530 indicates the level of experience.

[0085] At 2109, Digital Makeup Artist 520 may ask User 530 which part of the face the user wants to apply makeup to in the instruction. The instruction may be provided for a part of the face or the entire face. At 2111, the user may choose to have the instruction presented for a part of the face such as the eyes.

[0086] In some embodiments, at 2113, the digital makeup artist 520 may ask the user 530 how much time they have for the guidance. The guidance can be a shortened version if the user doesn't have much time, or a full version if the user has sufficient time for full guidance. The user 530 may provide a qualitative response or a response in terms of the amount of time in minutes. At 2115, the user 530 can provide a qualitative response such as "I don't have much time." The system 510 may interpret this response as indicating that the shortened version of the guidance should be executed. However, the user 530 can further indicate that they intend to save the guidance in an intermediate state for later completion. In the latter case, the system 510 can save the guidance at a point and make it possible to start and play back from the intermediate point.

[0087] In some embodiments, at 2117, the digital makeup artist 520 may ask the user 530 whether the user 530 wishes to use makeup and makeup applicators that are already available, such as makeup the user 530 has purchased, or whether the user 530 prefers that the system 510 provide the makeup and makeup applicators that may be required for the guidance. At 2121, the user 530, who has some makeup, can provide a qualitative response indicating acceptance of the system 510 selecting the makeup and makeup applicator.

[0088] At 2123, the digital makeup artist 520 may ask the user 530 to take a photo of their face. The photo may be taken with the camera 231 built into the mobile device 101 or with an external camera.

[0089] At 2125, the system 510 may perform analysis of the facial image. As described above, the facial analysis may be performed to obtain the positions of the facial parts, and the facial features including skin color, skin texture, lighting, as well as the past looks preferences, and these are information that may be used in generating the video for guidance.

[0090] At 2127, the system 510 may select the makeup to be applied in the guidance.

[0091] At 2129, the digital makeup artist 520 may ask the user 530 whether the user likes the makeup selection. At 2131, the user 530 may respond by not agreeing to the selection, but rather desiring that the guidance be performed using the makeup provided by the user.

[0092] At 2133, the digital makeup artist 520 may indicate that the guidance is to start. In some embodiments, at 2135, the digital makeup artist 520 may provide a list of products that may be used during the guidance if the user 530 wants to obtain the products to apply makeup during the guidance.

[0093] At 2141, the system 510 may start generating and playing the guidance video. In an exemplary guidance, the digital makeup artist 520 may start the guidance by stating the initial makeup that may be applied. At 2143, the digital makeup artist 520 may speak or provide in text an instruction indicating that concealer is applied first and then primer is applied.

[0094] The instructional video can be played by the system 510 in conjunction with instructions from the digital makeup artist 520 through either speech output, text output, or both speech output and text output. At 2145, the digital makeup artist 520 can provide instructions for selecting an eyeshadow. At 2147, the digital makeup artist 520 can provide instructions for selecting a blending brush. At 2149, the digital makeup artist 520 can provide instructions for applying the eyeshadow.

[0095] At a time point following the instructions, the user 530 can input video control commands. As an example, at 2151, the user 530 can input a "redo step" command. In some embodiments, the command can include a request to modify the step, such as "redo with the next color up". The digital makeup artist 520 can output a request to clarify what the user means by "the next color up". At 2153, the system 510 can generate new frames for the video starting from the eyeshadow selection at 2145 using the new color.

[0096] At 2155, the digital makeup artist 520 can provide instructions for selecting a blending brush. At 2157, the digital makeup artist 520 can provide instructions for applying the eyeshadow.

[0097] At 2159, the user 530 can input a command such as "pause". The system 510 can execute the pause function until the user 530 inputs a "play" command.

[0098] At 2161, the digital makeup artist 520 may provide instructions for performing blending with eyeshadow of a different shade. At 2163, the digital makeup artist 520 may provide instructions for selecting the next shade of eyeshadow. At 2165, the digital makeup artist 520 may provide instructions for selecting a blending brush. At 2167, the digital makeup artist 520 may provide instructions for performing blending.

[0099] At 2169, the user 530 can input a command such as "redo step" to redo the blending. At 2171, the system 510 may restart the video from the beginning of the steps for performing the blending.

[0100] At 2173, the digital makeup artist 520 may provide instructions for making the eyebrows stand out. At 2175, the digital makeup artist 520 may provide instructions for performing the step of blurring the wrinkles. At 2177, the digital makeup artist 520 may provide instructions for performing the step of selecting a pearl finish concealer.

[0101] At 2179, the user 530 can input a statement that the concealer is too bright and ask whether the shade of the concealer can be lowered. At 2181, the digital makeup artist 520 can respond to try another concealer, which can be executed by issuing a "redo step" that involves changing to another concealer.

[0102] In 2183, depending on where the chapter starts, system 510 can restart the video from the start of the wrinkle smoothing step. Next, in 2185, system 510 can, for example, select a mate finish concealer and play the video with the selected concealer. In 2187, user 530 can enter a "pause" command to take time to closely examine the results of the new concealer. In 2189, user 530 can enter a "play" command to start playing the video.

[0103] In 2191, digital makeup artist 520 can provide instructions for performing the step of shaping around the eyes. In 2193, digital makeup artist 520 can provide instructions for performing the step of brightening the eyes.

[0104] In 2195, system 510 can select a pencil suitable for the user's skin tone. In 2197, system 510 can draw on the face image using the selected pencil.

[0105] In 2199, digital makeup artist 520 can provide instructions for performing the step of applying eyeliner. At this point, in 2201, user 530 can choose to stop the video so that it can be restarted at a later time.

[0106] At a future time point when the video is completed, in 2203, the digital makeup artist 520 can output a question asking whether the user wants to save the completed looks. In 2205, the user can select to save the look and instruct the system 510 to remember the look. In 2207, the system 510 saves the final look to the database 107. In some embodiments, the final look can be saved to the database as a preferred look in a certain type of look. For example, a look created using the guidance can be saved as a preferred look in the type of look first described (see 503 in FIG. 5). The final look can also be transferred to other platforms such as a social media platform or a video conferencing platform. Some current social media platforms where the user can post photos or videos include, by way of example, Facebook, LinkedIn, Instagram, TikTok, and Snapchat. Current video conferencing platforms include, by way of example, Microsoft Teams, Facetime (registered trademark), Google Hangouts, or Google Meet, Zoom, GoToMeeting, Skype. In addition, changes made during the guidance, such as color selection and color finishing, can also be saved as look preference data.

[0107] In some embodiments, in 2207, the system 510 can save one or more mask filters related to the creation of the final look to the database 107. The one or more mask filters can be transferred to other platforms and can be used to create custom looks for other platforms.

[0108] Social media and video conferencing have not only created a need for on-the-spot makeup but have also influenced social subcultures such as viscogirls and aesthetic looks such as e-girls and soft girls. That need is virtual makeup looks for social media and video conferencing, i.e., augmented reality.

[0109] Exemplary operation of the system is provided in FIGS. 22A-22E to create a virtual trial for social media or video conferencing. FIGS. 22A-22E are sequence schematic diagrams for an exemplary use case of an exemplary dialogue using digital makeup artist 520 for makeup guidance, according to an exemplary aspect of the present disclosure.

[0110] If the user selects to use digital makeup artist 520 for guidance in creating a virtual trial for a video conference, at 2221, digital makeup artist 520 may ask, "What look do you want to create?" At 2223, user 530 may state that they want to apply an e-girl look. The e-girl look may be a look that the user wants to try when posting to social media or having a video conference with friends or colleagues. An existing e-girl look may be stored in database 107 or provided locally in app 111.

[0111] In some embodiments, at 2225, digital makeup artist 520 may ask user 530 to provide a level of experience in applying makeup. The level of experience may be categories such as beginner, experienced, expert, and professional. System 510 may present a list of experience levels for user 530 to select. At 2227, user 530 indicates an experience level, being an experienced level but applying an e-girl look for the first time.

[0112] At 2229, the digital makeup artist 520 may ask the user 530 which part of the face they want to apply makeup to. The guidance may be provided for a part of the face or the entire face. At 2231, the user can choose to have guidance presented for the entire face.

[0113] In some embodiments, at 2233, the digital makeup artist 520 may ask the user 530 how much time they have for the guidance. The guidance can be a shortened version if the user has little time, or a full version if the user has sufficient time for full guidance. The user 530 may provide a qualitative response or a response in terms of the amount of time in minutes. At 2235, the user 530 can provide a qualitative response such as "I don't have much time." The system 510 may interpret this response as indicating that the shortened version of the guidance should be executed. In some embodiments, at 2237, the digital makeup artist 520 may ask the user 530 whether the user 530 wishes to use digital makeup and makeup applicators that are already available, such as digital cosmetics the user 530 has purchased, or whether the user 530 wishes for the system 510 to provide digital makeup and makeup applicators that may be required for the guidance. At 2241, the user 530 can provide a qualitative response indicating that they have some digital cosmetics.

[0114] At 2243, the digital makeup artist 520 may ask the user 530 to take a photo of their face. The photo may be taken with the camera 231 built into the mobile device 101 or with an external camera.

[0115] At 2245, the system 510 may perform analysis of the facial image. As described above, the facial analysis may be performed to obtain the positions of facial parts, and facial features including skin color, skin texture, lighting, as well as past looks preferences, and these are information that may be used when generating a guidance video.

[0116] At 2247, the system 510 may select digital makeup to apply in the guidance.

[0117] At 2249, the digital makeup artist 520 may ask the user 530 whether the user likes the selected digital makeup. At 2251, the user 530 may respond by not agreeing to the selection and rather desiring to perform the guidance using the makeup provided by the user.

[0118] At 2253, the digital makeup artist 520 may indicate that the guidance is to start. In some embodiments, at 2255, the digital makeup artist 520 may provide a list of digital cosmetics that may be used during the guidance if the user 530 wants to obtain products to apply makeup during the guidance.

[0119] At 2261, the system 510 may start generating and playing the guidance video. In an exemplary guidance, the digital makeup artist 520 may start the guidance by stating the initial makeup that may be applied. At 2263, the digital makeup artist 520 may speak or provide in text an instruction indicating that the primer is to be smoothly applied for pre-treatment of the foundation.

[0120] The instructional video can be played by the system 510 in conjunction with instructions from the digital makeup artist 520 through either speech output, text output, or both speech output and text output. At 2265, the digital makeup artist 520 can provide instructions for selecting a foundation with a light range and a natural finish. At 2267, the digital makeup artist 520 can provide instructions for sweeping blush across the nasal bridge and slightly extending it onto the cheeks. At 2269, the digital makeup artist 520 can provide instructions for applying highlighter powder to the tip of the nose using a tapered brush.

[0121] At a time point following the instructions, the user 530 can input video control commands. As an example, at 2271, the user 530 can input a "redo step" command. In some embodiments, the command can include a request to modify the step, such as redoing with the next blush color. The digital makeup artist 520 can output a request to clarify what the user means by "the next blush color". At 2273, the system 510 can generate a new frame for the video starting with sweeping the blush at 2267 using a new blush color.

[0122] At 2275, the digital makeup artist 520 can provide instructions for applying a mechanical pencil to the eyebrows with a light stroke. At 2277, the digital makeup artist 520 can provide instructions for combing the eyebrows.

[0123] At 2279, the user 530 can input a command such as "pause". The system 510 can execute the pause function until the user 530 inputs a "play" command.

[0124] At 2281, the digital makeup artist 520 may provide instructions for selecting an eyeshadow with a pink tone. At 2283, the digital makeup artist 520 may provide instructions for selecting a foot applicator for applying the eyeshadow. At 2285, the digital makeup artist 520 may provide instructions for applying the eyeshadow to the eyelids. At 2287, the digital makeup artist 520 may provide instructions for performing blending with a fluffy blending brush for the creases.

[0125] At 2289, the user 530 can enter a command such as "redo step" to redo the blending. At 2291, the system 510 may restart the video from the beginning of the step of performing the blending.

[0126] At 2293, the digital makeup artist 520 may provide instructions for creating wings across each eyelid using a wing stencil. At 2295, the digital makeup artist 520 may provide instructions for performing the step of applying mascara to the eyelashes with several swipes. At 2297, the digital makeup artist 520 may provide instructions for applying just a little lip gloss.

[0127] At 2299, the user 530 can enter a statement that the lip gloss is too light and ask whether the lip gloss can be made stronger.

[0128] At a future time point when the video is completed, in 2301, the digital makeup artist 520 may output a question asking whether the user wants to save the completed looks. In 2303, the user 530 can select to save the look and instruct the system 510 to save the look. In 2305, the system 510 saves the final look to the database 107. In some embodiments, the final look may be saved in the database as a preferred look in a specific type of look, such as a personal E-girl look. For example, an E-girl look created with guidance may be saved as a preferred look of the E-girl look described first (see 503 in FIG. 5). The final look may also be transferred to other platforms such as a social media platform or a video conferencing platform. Some current social media platforms where users can post photos or videos include, by way of example, Facebook, LinkedIn, Instagram, TikTok, and Snapchat. Current video conferencing platforms include, by way of example, Microsoft Teams, FaceTime (registered trademark), Google Hangouts, or Google Meet, Zoom, GoToMeeting, Skype, etc. In addition, changes made during the guidance, such as color selection and color finishing, can also be stored as data on look preferences.

[0129] In some embodiments, in 2305, the system 510 may save one or more mask filters related to the creation of the final look to the database 107. The one or more mask filters can be transferred to other platforms and used to create custom looks for other platforms.

[0130] In addition to providing a guidance service for creating custom looks, digital makeup artist 520 can provide personalized makeup consultations. Digital makeup artist 520 can provide advice on ways to enhance the user's makeup look, or advice on improvements that may be added to the user's makeup application technique. Digital makeup artist 520 can provide advice on makeup application techniques that can better address problem areas or bring out special facial features. Digital makeup artist 520 can provide advice on makeup applications suitable for the time of day (morning, noon, night), the location where the user is expected to go (mainly indoors with artificial lighting, mainly outdoors with sunlight and lighting), or the current skin condition (dry skin). Digital makeup artist 520 can provide advice on ways to bring out the user's personality.

[0131] FIG. 23 is a sequence schematic diagram for an interaction between a digital makeup artist and a user for makeup consultation according to an exemplary aspect of the present disclosure.

[0132] As described above, the user 530 may be given the option to use the digital makeup artist 520 to receive teachings during guidance or to receive advice during a makeup consultation session. At 2351, the user 530 can request makeup advice via the user interface window 310 (see FIG. 3). At 2353, the system 510 may request the user 530 to take a photo or video of their face. In a manner similar to the above regarding guidance, the system 510 can analyze the user's face image to identify facial parts, skin tone, lip color, hair color, and skin texture. At 2355, the system 510 can conduct a two-way conversation with the user 530 to obtain further information related to the user's needs, including concerns about skin condition, indoor / outdoor looks, favorite facial features, and any facial feature concerns. At 2357, the user 530 can input information about skin condition, such as dry skin, indoor / outdoor looks, favorite features, and concerns about facial features.

[0133] When user input information and preference information that may be pre-stored in the database 107 and mask filters for various types of looks stored in the database 107 are provided, at 2359, the system 510 creates one or more custom recommendations for a makeup routine. As described above, custom recommendations can be obtained using a recommendation system. The recommendation system 1000 can be used to search for makeup filters used when creating custom recommendations for a makeup routine and can also be used to search for makeup routines such as skincare. At 2361, the system 510 can display one or more recommended makeup routines that are searched. The makeup routine can be generated using the blending process shown in FIG. 20.

[0134] Recommendation system 1000 includes a recommendation engine 1007 that searches for and ranks recommended makeup filters. The recommendation engine 1007 can also draw from an external repository of additional information, including one or more of frequently asked questions and answers, common makeup looks, cosmetics for specific skin types and conditions, and ethnicity. The recommendation engine 1007 can further draw from an external repository of makeup categories, including skincare and makeup with skincare qualities. Makeup with skincare qualities can include SPF-formulated foundations for protection from sunlight, makeup with anti-aging qualities. When applying a specific virtual makeup, the recommended makeup filter can be for features the user entered as favorites in step 2357 and concerns about facial features.

[0135] In one or more embodiments, as described above, the recommendation engine 1007 can be supplemented with a machine learning model for recommending makeup color tones based on skin undertone.

[0136] In some embodiments, at 2363, the user interface window 310 can receive input from the user 530 and improve or adjust the recommended makeup routine. Further user input can be in the form of improvements or adjustments to specific features shown in the recommended routine. For example, the user can input that the color of the lipstick is too gaudy. Further user input can be in the form of improvements or adjustments to the overall makeup routine. For example, the user can input that the makeup routine is too bold, or that the user prefers long wear for makeup. At 2365, the system 510 can adjust the recommended makeup routine according to makeup look data that harmonizes with the type of user input and adjustments. For example, the system 510 can make adjustments by searching for a mask filter for long wear makeup stored in the database 107. At 2367, the system 510 can generate and display a modified makeup routine for the user's face image using the searched mask filter. At 2369, the system 510 can provide recommendations regarding cosmetics that can be used to create a face image completed using the makeup routine.

[0137] In addition, at 2371, the system 510 can store in the database 107 as a preference for makeup looks the completed face image and the adjustments to the makeup routine used to create the completed face image. For example, the mask filter for long wear makeup can be stored with a label indicating that it is a preference for makeup looks (makeup filter) for the makeup routine (look) in a feature matrix such as that of FIG. 11.

[0138] In one or more embodiments, at 2373, the user 530 can choose to transfer / publish the created finished face image to a platform that provides a live video or a still image where the user wants to create a presence. Examples of platforms that provide live videos include social media platforms and video conferencing platforms such as Facebook, LinkedIn, Google Hangouts, or Google Meet, Facetime (registered trademark), Microsoft Teams, TikTok, Zoom.

[0139] To describe the digital makeup artist 520, exemplary operations are provided in FIGS. 24A - 24D. FIGS. 24A - 24D are sequence schematic diagrams for an exemplary dialogue using a digital makeup artist for makeup consultation according to an exemplary aspect of the present disclosure. The sequence schematic diagrams of FIGS. 24A - 24D include operations and communications by the system 510, the digital makeup artist 520, and the user 530.

[0140] At 2401, the user 530 can select a mobile application for the digital makeup artist. When the user interface window 310 is provided, at 2403, the user 530 can ask to obtain makeup advice. At 2405, the digital makeup artist 520 can ask whether the user has a preferred look for which advice is desired. At 2407, the user 530 can respond with an answer that further specifies the desired look. The digital makeup artist 520 can further narrow down the type of advice to be given by asking at 2409 whether the user has a preference for makeup looks. The user 530 can respond at 2411 that they have no preference, but rather desire the system 510 to select a look.

[0141] At 2413, the system 510 may request that the user take a photo or video of their face and will perform an analysis on the user's face image. The results of the analysis may include the positions of the parts of the user's face, as well as characteristics such as skin color, skin texture, lighting, and preferences for previous looks.

[0142] At 2415, the digital makeup artist 520 may ask the user if they have a favorite facial expression. The user 530 may respond with one or more favorite facial expressions, for example, lips as in 2417. At 2421, the system 510 may create a custom recommendation for one or more makeup routines. At 2423, the system 510 may display the makeup routine and cosmetic characteristics. At 2425, the digital makeup artist 520 may ask the user if they want to make adjustments to the recommended makeup routine. At 2427, the user 530 may respond that the makeup looks are too prominent.

[0143] At 2429, the digital makeup artist 520 may respond by asking if it is a particular part or parts of the face that are prominent, or if the whole face is too prominent. At 2431, the user 530 may respond that the whole face is too prominent. At 2433, the digital makeup artist 520 may notify the user 530 that adjustments will be made to the makeup look.

[0144] At 2435, the system 510 may perform an adjustment of the makeup look that may include a search for mask filters from the database 107, and the selection of the mask filter may take into account preferences for past looks. At 2437, the system 510 may display the adjusted makeup look. At 2439, the user 530 may view the adjusted makeup look and provide further feedback such as that the eyes look a bit too dark and request that the eyes be brightened.

[0145] At 2441, the system 510 can adjust the makeup look, and at 2443, display the further adjusted look. At 2445, the digital makeup artist 520 can notify the user that the system has increased the color tone of the eyeshadow and convey to the user whether the adjustment has sufficiently improved the eye brightness. At 2447, the user 530 can reply that the adjustment is in the right direction but may be better with a little more adjustment.

[0146] At 2449, the system 510 can further adjust the color tone and at 2451, display the further adjusted appearance. At 2453, the digital makeup artist 520 can again inform the user that the system has increased the color tone and again ask the user 530 whether the adjustment is sufficient. At 2455, the user 530 can reply that the adjustment looks good.

[0147] When the makeup look is completed, at 2461, the system 510 can display the final makeup look. Additionally, at 2463, the system 510 can display the cosmetics that can be used to create the final makeup look, and at 2465, save the makeup routine, the final makeup look, and the adjustments made as the user's look preference to the database 107.

[0148] Through interaction with the digital makeup artist 520, the user 530 can improve makeup looks and enhance makeup application techniques. The digital makeup artist 520 provides makeup advice tailored to the user. The digital makeup artist 520 continues to improve its recommendation degree through the user's look preferences and the accumulation of custom looks. The digital makeup artist 520 can try out makeup for the user and teach the user how to apply makeup in creating custom looks before the user applies makeup to their face. In addition, the digital makeup artist 520 can create custom makeup looks based on the stored look preferences and information about the user's facial features.

[0149] In embodiments, words such as "a", "an", etc. generally have the meaning of "one or more" unless otherwise specified.

[0150] Numerous modifications and variations of the present invention are possible in light of the above teachings. For example, data collected from the skin tones and textures of various consumers will enable the scaling of artificial neural networks for more consumers than a single consumer. The artificial neural network will be able to predict the rendering of new cosmetic formulations for each product color tone. As a result, it should be understood that within the scope of the appended claims, the present invention may be practiced in ways other than specifically described herein.

[0151] As a result, it should be understood that within the scope of the appended claims, the present invention may be practiced in ways other than specifically described herein.

[0152] The above disclosure also encompasses the embodiments listed below.

[0153] (1) Digital Makeup Artist System. The digital makeup artist system includes a mobile device having a display device, an arithmetic circuit, and a memory, a database system for storing makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's preference for looks, and a machine learning system for analyzing a face image. The mobile device includes a user interface for interacting with the digital makeup artist. The digital makeup artist performs a two-way interaction with the user to capture the user's needs including one or more of the type of makeup looks, indoor or outdoor looks, skin condition, problem areas of the face, and favorite facial features. The arithmetic circuit inputs the user's face image, analyzes the user's face image through the machine learning system to identify facial parts, determines facial features including one or more of skin tone, eye color, hair color, lip color, and skin texture, and is configured to generate an image frame to be displayed on the display device in synchronization with the interaction with the digital makeup artist. The image frame is generated based on one or more of the user's analyzed face image, the user's needs obtained through the interaction with the user, the stored makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's preference for looks.

[0154] (2) The arithmetic circuit is configured to play a video by displaying the generated image frames at a predetermined playback speed. The video includes chapters, and each chapter is a step of a makeup routine. The input from the user includes video control commands for controlling the playback of the video. The digital makeup artist system according to feature (1).

[0155] (3) The digital makeup artist system according to feature (2), wherein the video control command includes any one of reducing the playback speed of the video, temporarily stopping the video playback, restarting from the beginning of the video step, and skipping to the next step of the video.

[0156] (4) When the input from the user is a video control command for reducing the playback speed, the arithmetic circuit reduces the playback speed at a ratio with respect to a predetermined playback speed, and stores in the database a display indicating that the current chapter being played at the time of the command input is complex, the digital makeup artist system according to feature (2) or (3).

[0157] (5) The digital makeup artist system according to feature (2) or (3), wherein the input from the user is a video control command for restarting from the beginning of the video step, and the arithmetic circuit is configured to read a time point related to the beginning of the current chapter being played at the time of the command input, and play the video from the frame at the time point.

[0158] (6) The input from the user is a video control command for restarting from the beginning of the video step, and further includes a command for adjusting the generation of the image frame. The arithmetic circuit is configured to read a time point related to the beginning of the current chapter being played, and generate an adjusted image frame for playing the video from the frame at the time point, the digital makeup artist system according to feature (2) or (3).

[0159] (7) The digital makeup artist system according to feature (6), wherein the adjustment of the generation of the image frame includes a change to the color characteristics of the makeup for the facial parts in the face image, and the characteristics of the color are one or more of range, color tone, and finish.

[0160] (8) The digital makeup artist system according to feature (6), wherein the user input includes a type of looks, and generation adjustment of the image frame takes into account the user's preference for looks related to the type of looks.

[0161] (9) The digital makeup artist system according to feature (6), wherein makeup looks created based on generation adjustment of the image frame are stored in the database as preferred makeup looks.

[0162] (10) The digital makeup artist system according to any one of features (1) to (9), wherein the arithmetic circuit is further configured to analyze the user's face image via the machine learning system to determine one or more of a face shape, a lip shape, an eyelid shape, and a hairstyle, and to analyze the user's face image to determine one or more of a skin tone, an eye color, a hair color, and a skin texture, and the image frame is generated based on a makeup routine stored in relation to the face shape, the lip shape, the eyelid shape, the hairstyle, the skin tone, the eye color, the hair color, and the skin texture.

[0163] (11) The digital makeup artist system according to any one of features (1) to (10), wherein the two-way interaction with the user performed by the digital makeup artist includes speech input, the mobile device performs natural language processing on the speech input, and the digital makeup artist outputs a speech response.

[0164] (12) The digital makeup artist system according to feature (11), wherein the two-way interaction with the user includes the digital makeup artist uttering a question to obtain information regarding the state of the user's skin.

[0165] (13) The digital makeup artist system according to feature (11), wherein the two-way interaction with the user includes the digital makeup artist uttering a question to obtain information about problem areas on the user's face.

[0166] (14) The digital makeup artist system according to feature (12), wherein the two-way interaction with the user includes the digital makeup artist receiving speech input indicating that the user's skin is dry, and the image frame is generated based on a makeup routine stored in relation to the indication that the user's skin is dry.

[0167] (15) The digital makeup artist system according to any one of features (1) to (14), wherein the makeup routine information and cosmetics stored in the database system are for skin care, and the image frame is generated based on the makeup routine information and cosmetics for skin care.

[0168] (16) The digital makeup artist system according to any one of features (1) to (15), wherein the makeup routine information and cosmetics stored in the database system are for skin care type makeup, and the image frame is generated based on the makeup routine information and cosmetics for skin care type makeup.

[0169] (17) The digital makeup artist system according to feature (16), wherein the skin care type makeup includes cosmetics having anti-aging quality, and the image frame is generated based on a makeup routine stored in relation to the application of the cosmetics having anti-aging quality.

[0170] (18) The digital makeup artist system according to any one of features (1) to (17), wherein the image frame is generated based on the analyzed face image of the user including the skin tone, in order to provide a makeup routine using a color tone optimal for the skin tone.

[0171] (19) The digital makeup artist system according to any one of features (1) to (18), further comprising a skin undertone machine learning model for selecting a color tone of cosmetics suitable for a specific skin undertone of the face image.

[0172] (20) The digital makeup artist system according to feature (19), wherein the cosmetics are foundation, and the skin undertone machine learning model is for selecting the foundation suitable for the specific skin undertone of the face image.

[0173] (21) Digital Makeup Artist System. The digital makeup artist system includes a mobile device having a display device, an arithmetic circuit, and a memory, a database system that stores makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's look preferences, and a machine learning system for analyzing a face image. The mobile device includes a user interface for interacting with the digital makeup artist. The digital makeup artist obtains initial information including one or more of the type of makeup looks, indoor or outdoor looks, skin condition, problem areas of the face, and favorite facial features, and conducts a two-way interaction with the user to provide advice including requesting a makeup consultation. The arithmetic circuit inputs the user's face image, analyzes the user's face image through the machine learning system to identify facial parts, analyzes the face image to determine facial features including one or more of skin tone, eye color, lip color, hair color, and skin texture, and is configured to generate an image frame to be displayed on the display device in synchronization with the interaction with the digital makeup artist to provide the advice. The image frame is generated in synchronization with the interaction based on one or more of the user's analyzed face image, the initial information obtained through the interaction with the user, the stored makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's look preferences.

[0174] (22) The digital makeup artist system according to feature (21), wherein the arithmetic circuit is configured to conduct the makeup consultation through the two-way interaction between the user and the digital makeup artist, and the arithmetic circuit executes the interaction to create a custom recommendation for a makeup routine according to the favorite facial features.

[0175] (23) The arithmetic circuit is configured to perform the makeup consultation through the two-way interaction between the user and the digital makeup artist, and the interaction includes prompting the digital makeup artist to input at least one problem face area that the user is concerned about. The arithmetic circuit performs the interaction to create a custom recommendation for a makeup routine tailored to the problem face area of concern. The digital makeup artist system according to feature (21) or (22).

[0176] (24) The arithmetic circuit conducts the two-way interaction to create a custom recommendation for a makeup routine, receives further input including a request for adjustment of the recommended makeup routine, and is configured to perform the adjustment based on the request for adjustment of the recommended makeup routine, the stored makeup looks, and the user's look preferences to create a refined makeup routine. The digital makeup artist system according to any one of features (21) to (23).

[0177] (25) The arithmetic circuit is configured to store the adjustment in the database together with the refined makeup routine as the user's look preference included in the user's look preferences. The digital makeup artist system according to feature (24).

[0178] (26) The arithmetic circuit is configured to output recommended cosmetics and skin care products for the refined makeup routine. The digital makeup artist system according to feature (24).

[0179] (27) The request for adjustment of the recommended makeup routine includes a request to change the makeup characteristics of an image of the entire face. The digital makeup artist system according to feature (24).

[0180] (28) The digital makeup artist system according to feature (27), wherein the change request for the makeup characteristics includes one or more changes among the range, color tone, and finish of each makeup color in the face image.

[0181] (29) The adjustment request for the recommended makeup routine includes a change request for the makeup characteristics of the face part in the face image, and the change request for the makeup characteristics includes one or more changes among the range, color tone, and finish of the makeup color of the face part in the face image. The digital makeup artist system according to feature (24).

[0182] (30) The two-way interaction with the user includes the digital makeup artist receiving speech input indicating that the user's skin is dry, and the image frame is generated based on the makeup routine stored in relation to the indication that the user's skin is dry. The digital makeup artist system according to feature (22).

[0183] (31) The makeup routine information and cosmetics stored in the database system are for skin care, and the image frame is generated based on the makeup routine information and cosmetics for skin care. The digital makeup artist system according to any one of features (21) to (30).

[0184] (32) The makeup routine information and cosmetics stored in the database system are for skin care type makeup, and the image frame is generated based on the makeup information and cosmetics for skin care type makeup. The digital makeup artist system according to any one of features (21) to (31).

[0185] (33) The makeup of the skin care type includes cosmetics having anti-aging quality, and the image frame is generated based on a makeup routine stored in relation to the application of the cosmetics having anti-aging quality. The digital makeup artist system according to feature (32).

[0186] (34) The arithmetic circuit conducts the dialogue to create a custom recommendation for a makeup routine adapted to the skin tone of the user, and the image frame of the makeup routine is generated based on the analyzed face image of the user to create the custom recommendation using the color tone of the cosmetics optimal for the skin tone. The digital makeup artist system according to any one of features (21) to (33).

[0187] (35) The digital makeup artist system according to feature (34), further including a skin undertone machine learning model for selecting the color tone of the cosmetics suitable for a specific skin undertone of the face image.

[0188] (36) The digital makeup artist system according to feature (35), wherein the cosmetics are foundation, and the skin undertone machine learning model is for selecting the color tone of the foundation suitable for the specific skin undertone of the face image.

[0189] (37) The arithmetic circuit conducts the dialogue to create a custom recommendation for a makeup routine adapted to the shape and color of the user's lips, and the image frame of the makeup routine is generated based on the analyzed face image of the user to create the custom recommendation using the color tone and / or finish of the cosmetics for the user's face optimal for the shape and color of the user's lips. The digital makeup artist system according to any one of features (21) to (37).

[0190] (38) The digital makeup artist system according to any one of features (21) to (37), wherein the arithmetic circuit performs the dialogue to create a custom recommendation for a makeup routine including obtaining the user's skin type and ethnicity, and the image frame of the makeup routine is generated based on the user's skin type, ethnicity, and the user's preference for looks to create the custom recommendation using cosmetics stored in a database specified for the skin type and the ethnicity.

[0191] (39) The digital makeup artist system according to feature (38), wherein the makeup routine information and cosmetics stored in the database system are for makeup of a skin care type according to the skin type and ethnicity, and the image frame is generated based on the makeup routine information and cosmetics for makeup of a skin care type according to the skin type and ethnicity.

[0192] (40) The digital makeup artist system according to feature (38), further including an ethnicity machine learning model for selecting a color tone of cosmetics suitable for a specific ethnicity of the user's face image.

Explanation of Reference Numerals

[0193] 101 Smartphone 103 Laptop Computer 103a Microphone 105 Cloud Service 107 Database 109 Machine Learning Service 111 Mobile Application 200 Mobile Processing Unit (MPU) 201 Subscriber Identification Module (SIM) 202 Memory 206 Network Controller 208 Display Screen Controller 210 Display Screen 212 I / O Interface 214 buttons 220 Power management and touch screen controller 221 Touch screen 222 Communication bus 224 Network 225 Controller 226 Arithmetic circuit 230 Camera controller 231 Camera 240 Microphone circuit 241 Microphone 242 Audio circuit 301 Sub-screen 303 Menu icon 305 Makeup artist 307 Input box 309 Text 310 User interface window 311 Microphone icon 320 Video component 401 Video 501 First interaction 510 System 511 Recommended user interface 520 Digital makeup artist 530 User input 705 Lip shape 707 Eyelid shape 709 Hair style 711 Color 713 Skin texture 715 Lighting 717 Looks preference 803 Component 803a Layer 803b Rectified linear unit layer 803c Pooling layer 805 Fully connected layer 807 Loss layer 809 Class 901 Input face image 903 First stage 905 Second Stage 907 Output Layer 911 Connection Block 913 Prediction Block 1000 Recommendation System 1005 Database 1007 Recommendation Engine 1201 Custom Recommendation 1203 Scroll Bar 1301 Message 1401 Facial Image 1403 Makeup Palette 1403a Makeup 1403b Color 1405 Parts 1901 Audio Component 1903 Facial Image 1905 Mask Filter 1907 Location Information 1911 Previous Frame 2007 Feature Mask Filter 2011 Inverse Value 2015 Image 2357 Step

Claims

1. A mobile device having a display device, an arithmetic circuit, and a memory, and A database system that stores makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's look preferences, and A machine learning system for analyzing facial images, including The mobile device includes a user interface for interacting with a digital makeup artist, and the digital makeup artist performs a two-way interaction with the user to capture the user's needs including one or more of the type of makeup look, indoor or outdoor look, skin condition, facial problem areas, and favorite facial features, The arithmetic circuit Inputs the facial image of the user, Analyzes the facial image of the user via the machine learning system to identify facial parts, Analyzes the facial image to determine facial features including one or more of skin tone, eye color, hair color, lip color, and skin texture, Generates an image frame to be displayed on the display device in synchronization with the interaction with the digital makeup artist, Is configured as The image frame is generated based on the analyzed facial image of the user, the user's needs obtained through the interaction with the user, the stored makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and one or more of the user's look preferences, Digital makeup artist system.

2. The arithmetic circuit is configured to play a video by displaying the generated image frame at a predetermined playback speed, The video includes chapters, and each chapter is a step of a makeup routine, The digital makeup artist system according to claim 1, wherein the input from the user includes a video control command for controlling the playback of the video.

3. The digital makeup artist system according to claim 2, wherein the video control command includes one of reducing the playback speed of the video, pausing the video playback, restarting from the beginning of the video step, and skipping to the next step of the video.

4. When the input from the user is a video control command to decrease the playback speed, the arithmetic circuit decreases the playback speed at a rate with respect to a predetermined playback speed, and stores in the database a display indicating that the current chapter being played at the time of the command input is complex. The digital makeup artist system according to claim 3.

5. The input from the user is a video control command to restart from the beginning of the step of the video, and the arithmetic circuit reads a time point related to the beginning of the current chapter being played at the time of the command input, and is configured to play the video from the frame at the time point. The digital makeup artist system according to claim 3.

6. The input from the user is a video control command to restart from the beginning of the step of the video, and further includes a command to adjust the generation of image frames. The arithmetic circuit reads a time point related to the beginning of the current chapter being played, and is configured to generate adjusted image frames for playing the video from the frame at the time point. The digital makeup artist system according to claim 3.

7. The adjustment of the generation of the image frames includes a change to the color characteristics of the makeup for the face portion in the face image. The characteristics of the color are one or more of range, color tone, and finish. The digital makeup artist system according to claim 6.

8. The user input includes the type of look. The adjustment of the generation of the image frames takes into account the user's look preference related to the type of look. The digital makeup artist system according to claim 6.

9. The makeup looks created based on the adjustment of the generation of the image frames are stored in the database as preferred makeup looks. The digital makeup artist system according to claim 6.

10. The arithmetic circuit is configured to analyze the face image of the user via the machine learning system to determine one or more of face shape, lip shape, eyelid shape, and hairstyle, and to analyze the face image of the user to determine one or more of skin tone, eye color, hair color, and skin texture. The image frame is generated based on a makeup routine stored in relation to the face shape, the lip shape, the eyelid shape, the hairstyle, the skin tone, the eye color, the hair color, and the skin texture. The digital makeup artist system according to claim 1.

11. The two-way interaction with the user performed by the digital makeup artist includes speech input. The digital makeup artist system according to claim 1, wherein the mobile device performs natural language processing on the speech input and the digital makeup artist outputs a speech response.

12. The digital makeup artist system according to claim 11, wherein the two-way interaction with the user includes the digital makeup artist uttering a question to obtain information regarding the state of the user's skin.

13. The digital makeup artist system according to claim 11, wherein the two-way interaction with the user includes the digital makeup artist uttering a question to obtain information regarding the problem area of the user's face.

14. The two-way interaction with the user includes the digital makeup artist receiving speech input indicating that the user's skin is dry. The digital makeup artist system according to claim 12, wherein the image frame is generated based on a makeup routine stored in relation to the indication that the user's skin is dry.

15. The makeup routine information and cosmetics stored in the database system are for skin care. The digital makeup artist system according to claim 1, wherein the image frame is generated based on the makeup routine information and cosmetics for skin care.

16. The makeup routine information and cosmetics stored in the database system are for skin care type makeup. The digital makeup artist system according to claim 1, wherein the image frame is generated based on the makeup routine information and cosmetics for skin care type makeup.

17. The skin care type makeup includes cosmetics having anti-aging quality. The digital makeup artist system according to claim 16, wherein the image frame is generated based on a makeup routine stored in relation to the application of the cosmetic having the anti-aging quality.

18. The digital makeup artist system according to claim 1, wherein the image frame is generated based on the analyzed face image of the user including the skin tone in order to provide a makeup routine using a tone optimal for the skin tone.

19. The digital makeup artist system according to claim 1, further comprising a skin undertone machine learning model for selecting a tone of a cosmetic suitable for a specific skin undertone of the face image.

20. The cosmetic is a foundation, The digital makeup artist system according to claim 19, wherein the skin undertone machine learning model is for selecting the foundation suitable for the specific skin undertone of the face image.

21. A mobile device having a display device, an arithmetic circuit, and a memory, A database system that stores makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's look preferences, A machine learning system for analyzing a face image, The mobile device includes a user interface for interacting with a digital makeup artist, and the digital makeup artist obtains initial information including one or more of the type of makeup look, indoor or outdoor look, skin condition, problem areas of the face, and favorite facial features, and provides advice including requesting a makeup consultation by performing a two-way interaction with the user, The arithmetic circuit, Inputs the face image of the user, Analyzes the face image of the user through the machine learning system to identify facial parts, Analyzes the face image to determine facial features including one or more of skin tone, eye color, lip color, hair color, and skin texture, Generates an image frame to be displayed on the display device in synchronization with the interaction with the digital makeup artist for providing the advice, Is configured as The image frame is generated in synchronization with the conversation based on one or more of the analyzed face image of the user, the initial information obtained through the conversation with the user, the stored makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's look preferences. Digital makeup artist system.

22. The arithmetic circuit is configured to conduct makeup consultation through the two-way conversation between the user and the digital makeup artist. The digital makeup artist system according to claim 21, wherein the arithmetic circuit executes the conversation to create a custom recommendation for a makeup routine tailored to the user's favorite facial features.

23. The arithmetic circuit is configured to conduct makeup consultation through the two-way conversation between the user and the digital makeup artist. The conversation includes prompting the digital makeup artist to input at least one problematic facial area that the user is concerned about. The digital makeup artist system according to claim 21, wherein the arithmetic circuit conducts the conversation to create a custom recommendation for a makeup routine tailored to the problematic facial area that the user is concerned about.

24. The arithmetic circuit conducts the two-way conversation to create a custom recommendation for a makeup routine, receives further input including a request for adjustment of the recommended makeup routine, and makes the adjustment based on the request for adjustment of the recommended makeup routine, the stored makeup looks, and the user's look preferences to create a refined makeup routine. The digital makeup artist system according to claim 21, configured as such.

25. The digital makeup artist system according to claim 24, wherein the arithmetic circuit is configured to store the adjustment together with the refined makeup routine in the database as the user's look preferences included in the user's look preferences.

26. The digital makeup artist system according to claim 24, wherein the arithmetic circuit is configured to output recommended cosmetics and skin care products for the refined makeup routine.

27. The digital makeup artist system according to claim 24, wherein the adjustment requirement of the recommended makeup routine includes a requirement to change makeup characteristics of an image of the entire face.

28. The digital makeup artist system according to claim 27, wherein the change requirement of the makeup characteristics includes one or more changes among a range, a color tone, and a finish of each makeup color in the face image.

29. The adjustment requirement of the recommended makeup routine includes a requirement to change makeup characteristics of a face part in the face image, The digital makeup artist system according to claim 24, wherein the change requirement of the makeup characteristics includes one or more changes among a range, a color tone, and a finish of the makeup color of the face part in the face image.

30. The two-way interaction with the user includes the digital makeup artist receiving a speech input indicating that the user's skin is dry, The digital makeup artist system according to claim 22, wherein the image frame is generated based on a makeup routine stored in relation to the indication that the user's skin is dry.

31. The makeup routine information and cosmetics stored in the database system are for skin care, The digital makeup artist system according to claim 21, wherein the image frame is generated based on the makeup routine information and cosmetics for skin care.

32. The makeup routine information and cosmetics stored in the database system are for skin care type makeup, The digital makeup artist system according to claim 21, wherein the image frame is generated based on the makeup information and cosmetics for skin care type makeup.

33. The skin care type makeup includes cosmetics having anti-aging quality, The digital makeup artist system according to claim 32, wherein the image frame is generated based on a makeup routine stored in relation to the application of the cosmetics having anti-aging quality.

34. The arithmetic circuit performs the interaction to create a custom recommendation for a makeup routine adapted to the skin tone of the user, The digital makeup artist system according to claim 21, wherein the image frame of the makeup routine is generated based on the analyzed face image of the user to create the custom recommendation for using the color tone of the cosmetics optimal for the skin tone.

35. The digital makeup artist system according to claim 34, further comprising a skin undertone machine learning model for selecting the color tone of the cosmetics suitable for a specific skin undertone of the face image.

36. wherein the cosmetics are foundation, The digital makeup artist system according to claim 35, wherein the skin undertone machine learning model is for selecting the color tone of the foundation suitable for the specific skin undertone of the face image.

37. The arithmetic circuit conducts the dialogue to create a custom recommendation for a makeup routine adapted to the shape and color of the user's lips, The digital makeup artist system according to claim 21, wherein the image frame of the makeup routine is generated based on the analyzed face image of the user to create the custom recommendation for using the color tone and / or finish of the cosmetics for the user's face that are optimal for the shape and color of the user's lips.

38. The arithmetic circuit conducts the dialogue to create a custom recommendation for a makeup routine including obtaining the skin type and ethnicity of the user, The digital makeup artist system according to claim 21, wherein the image frame of the makeup routine is generated based on the skin type and ethnicity of the user and the user's preference for looks to create the custom recommendation for using the cosmetics stored in the database designated for the skin type and ethnicity.

39. The makeup routine information and cosmetics stored in the database system are for skin care type makeup according to skin type and ethnicity, The digital makeup artist system according to claim 38, wherein the image frame is generated based on the makeup routine information and cosmetics for skin care type makeup according to skin type and ethnicity.

40. The digital makeup artist system according to claim 38, further comprising an ethnicity machine learning model for selecting a color tone of the cosmetic suitable for a specific ethnicity of the face image of the user.

Images