Digital Makeup Artist

Abstract

The digital makeup artist system includes a mobile device and a database system that stores makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and look preferences of a user. The mobile device includes a user interface for interacting with the digital makeup artist. The digital makeup artist performs two-way interaction with a user to capture the user's needs, including type of makeup look, indoor or outdoor looks, skin condition, facial problem areas, and favorite facial features. The computing circuitry analyzes the user's facial image to identify facial features, analyzes the facial image to determine facial features, and generates image frames based on the analyzed facial image, the user's needs, the stored makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's look preferences, which are displayed in synchronization with the interaction with the digital makeup artist.

Description

Technical Field

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to Non-Provisional Application No. 17 / 138,078 filed on December 30, 2020, Non-Provisional Application No. 17 / 138,143 filed on December 30, 2020, French Application Serial No. 2107909 filed on July 22, 2021, and French Application Serial No. 2107906 filed on July 22, 2021, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to digital makeup artists, and methods for interactive makeup advice and makeup instruction. Background Art

[0003] In recent years, mobile applications, or apps, have been developed to provide assistance in searching and selecting cosmetics for purchase. Apps may provide tools for searching for a specific type of makeup, for searching products that a user may like, or simply for purchasing products that have been used previously. Some apps assist in selecting lipstick or eyeshadow colors by displaying a color palette. Some apps provide a color matching function that assists in searching for colors that harmonize with colors from clothes, accessories, or images. Some also make videos of methods for applying a specific type of makeup available.

[0004] Some apps utilize the cameras on smartphones, tablets, and laptops to offer product trial applications. These applications may be implemented as web applications or as standalone apps. Some require users to take a self-portrait with their smartphone camera, upload the photo to the web application, and then use the camera to apply virtual cosmetics to the uploaded image. These applications may offer various options such as smoothing skin, lifting cheekbones, and adjusting eye color. They may also allow users to add any type and color of makeup, or change the intensity of the colors.

[0005] However, the trial applications offered so far have been photo editing tools that create looks. Some of the earlier trial applications have a one-step function that allows users to overlay makeup types and colors onto uploaded photos and then edit the resulting photo. In addition, photo editing tools for camera-equipped mobile devices are becoming popular for use on social media. VSCO, one of the photo-sharing apps, allows users to edit and filter photos before sharing them. Many VSCO filters are available to achieve specific effects. VSCO filters may consist of values ​​that can be applied to photos, including exposure, temperature, contrast, fade, saturation, hue, and skin tone. While such photo editing tools provide the functionality to edit photos, photo editing does not provide a personalized makeup experience. Furthermore, the earlier trial application tools do not offer the creation of custom looks. For example, a user might want a date night look. While the earlier trial application may offer a date night look, it does not provide advice on which cosmetics to use that would be best suited to the user, or how to apply different types of cosmetics to create the look. Alternatively, users might be able to make some edits to their face images in an attempt to obtain a custom look. They might also want a date-night look based on their mood, or the mood they want to portray.

[0006] These preceding trial web applications or apps lack complete personalization because they perform processes that are not customized makeup routines. Typical preceding virtual trial applications rely on templates and looks created for other people. When a user wants a specific look they have in mind, or wants to try a new look, they may face the challenge of having to edit a look created for someone else. There is a need to provide a customized trial experience for a specific user that allows for interaction in a way that is comparable to an experience with a personal makeup artist. A makeup experience is needed where a personal makeup artist guides the user through the steps to achieve the desired look on their face.

[0007] Social media apps are being developed to help users create their own anime-style avatars. These anime-style avatars can be customized with features such as hairstyle, hair color, face shape and color, makeup, eyebrows, and nose shape. However, users may prefer to post images of their actual faces with makeup on social media.

[0008] The preceding “Background Art” description is intended to provide a general context for this disclosure. The research of the inventors currently named in this “Background Art” section, and any aspects of the description that may not have been otherwise considered prior art at the time of filing, are not considered prior art to the present invention, either expressly or implicitly. [Overview of the Initiative] [Means for solving the problem]

[0009] One embodiment is a digital makeup artist system comprising a mobile device having a display device, an arithmetic circuit, and memory; a database system for storing makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's look preferences; a machine learning system for analyzing facial images, wherein the mobile device includes a user interface for interacting with a digital makeup artist, the digital makeup artist engaging in two-way interaction with the user to capture the user's needs, including one or more types of makeup looks, indoor or outdoor looks, skin condition, facial problem areas, and favorite facial features. The arithmetic circuit is configured to take a facial image of the user as input, analyze the facial image via a machine learning system that analyzes the facial image to identify facial parts, determine facial characteristics including one or more of skin tone, eye color, hair color, lip color, and skin texture, and generate an image frame to be displayed on the display device in synchronization with the interaction with the digital makeup artist. The aforementioned image frame is generated based on the analyzed user's facial image, the user's needs obtained through interaction with the user, one or more stored makeup routines, general makeup looks, cosmetics tailored to skin type and ethnicity, and the user's preferences for appearance.

[0010] One embodiment is a digital makeup artist system comprising a mobile device having a display device, an arithmetic circuit, and memory; a database system for storing makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's look preferences; a machine learning system for analyzing facial images; and the mobile device comprising a user interface for interacting with a digital makeup artist, wherein the digital makeup artist acquires initial information including one or more of the following: type of makeup looks, indoor or outdoor looks, skin condition, facial problem areas, and favorite facial features, and performs a two-way interaction with the user to provide advice, including requests for makeup consultation. The arithmetic circuit is configured to take the user's facial image as input, analyze the facial image via a machine learning system that analyzes the user's facial image to identify facial parts, determine facial characteristics including one or more of the following: skin tone, eye color, lip color, hair color, and skin texture, and generate image frames to be displayed on the display device in synchronization with the interaction with the digital makeup artist in order to provide advice. The aforementioned image frame is generated in sync with the conversation, based on the analyzed user's facial image, initial information obtained through interaction with the user, one or more stored makeup routines, a general makeup look, cosmetics appropriate for skin type and ethnicity, and the user's preferences for appearance.

[0011] The above summary of exemplary embodiments and the following detailed description are merely illustrative and not limiting to the teachings of this disclosure.

[0012] A more complete understanding of the many associated benefits of this disclosure will be readily available as the disclosure and its associated benefits are better understood by referring to the following detailed description, which is considered in conjunction with the accompanying drawings. [Brief explanation of the drawing]

[0013] [Figure 1] This is a schematic diagram of a system according to an exemplary aspect of this disclosure. [Figure 2] This is a block diagram of a computer system for a mobile device. [Figure 3] A user interface screen having an avatar according to an exemplary aspect of this disclosure is shown. [Figure 4] This document shows a user interface screen for a mobile device according to an exemplary aspect of the present disclosure. [Figure 5] This is a schematic sequence diagram for an interaction between a digital makeup artist and a user for custom instruction, according to an exemplary aspect of the present disclosure. [Figure 6] This is a user interface for inputting a type of look according to an exemplary aspect of the present disclosure. [Figure 7] This is a flowchart of face analysis according to an exemplary aspect of the present disclosure. [Figure 8] This is an illustrative schematic diagram of a convolutional neural network architecture for facial shape classification. [Figure 9] This is a schematic diagram of an exemplary deep learning neural network for facial landmark detection. [Figure 10] This is a schematic diagram for a recommender system according to an exemplary aspect of the present disclosure. [Figure 11] A matrix of non-limiting looks features according to an exemplary aspect of this disclosure is shown. [Figure 12] This is a user interface in a mobile application according to an exemplary aspect of the present disclosure. [Figure 13] This is a user interface in a mobile application according to an exemplary aspect of the present disclosure. [Figure 14] This is an exemplary mobile application in accordance with exemplary embodiments of the present disclosure. [Figure 15] This is a flowchart for a “Slow Down” video control command according to an exemplary aspect of the present disclosure. [Figure 16]It is a flowchart for a "PAUSE" video control command in accordance with an exemplary aspect of the present disclosure. [Figure 17] It is a flowchart for a "RE-DO LAST STEP" video control command in accordance with an exemplary aspect of the present disclosure. [Figure 18] It is a flowchart for a "SKIP CURRENT STEP" video control command in accordance with an exemplary aspect of the present disclosure. [Figure 19] It is a block diagram of a video playback component in accordance with an exemplary aspect of the present disclosure. [Figure 20] It shows a blending process that can be used to generate video frames based on desired features and original features. [Figure 21A] It is a schematic sequence diagram for an exemplary interaction using a digital makeup artist for makeup instruction in accordance with an exemplary aspect of the present disclosure. [Figure 21B] It is a schematic sequence diagram for an exemplary interaction using a digital makeup artist for makeup instruction in accordance with an exemplary aspect of the present disclosure. [Figure 21C] It is a schematic sequence diagram for an exemplary interaction using a digital makeup artist for makeup instruction in accordance with an exemplary aspect of the present disclosure. [Figure 21D] It is a schematic sequence diagram for an exemplary interaction using a digital makeup artist for makeup instruction in accordance with an exemplary aspect of the present disclosure. [Figure 21E] It is a schematic sequence diagram for an exemplary interaction using a digital makeup artist for makeup instruction in accordance with an exemplary aspect of the present disclosure. [Figure 21F] It is a schematic sequence diagram for an exemplary interaction using a digital makeup artist for makeup instruction in accordance with an exemplary aspect of the present disclosure. [Figure 22A]This is a schematic sequence diagram for an exemplary dialogue use case using a digital makeup artist 520 for makeup instruction, according to an exemplary aspect of the present disclosure. [Figure 22B] This is a schematic sequence diagram for an exemplary dialogue use case using a digital makeup artist 520 for makeup instruction, according to an exemplary aspect of the present disclosure. [Figure 22C] This is a schematic sequence diagram for an exemplary dialogue use case using a digital makeup artist 520 for makeup instruction, according to an exemplary aspect of the present disclosure. [Figure 22D] This is a schematic sequence diagram for an exemplary dialogue use case using a digital makeup artist 520 for makeup instruction, according to an exemplary aspect of the present disclosure. [Figure 22E] This is a schematic sequence diagram for an exemplary dialogue use case using a digital makeup artist 520 for makeup instruction, according to an exemplary aspect of the present disclosure. [Figure 23] Figure 23 is a schematic sequence diagram of an interaction between a digital makeup artist and a user for a makeup consultation, according to an exemplary embodiment of the present disclosure. [Figure 24A] This is a schematic sequence diagram for an exemplary interaction using a digital makeup artist for a makeup consultation, according to an exemplary aspect of the present disclosure. [Figure 24B] This is a schematic sequence diagram for an exemplary interaction using a digital makeup artist for a makeup consultation, according to an exemplary aspect of the present disclosure. [Figure 24C] This is a schematic sequence diagram for an exemplary interaction using a digital makeup artist for a makeup consultation, according to an exemplary aspect of the present disclosure. [Figure 24D] This is a schematic sequence diagram for an exemplary interaction using a digital makeup artist for a makeup consultation, according to an exemplary aspect of the present disclosure. [Modes for carrying out the invention]

[0014] In the following detailed description, the accompanying drawings, which constitute part of this specification, are referenced. In the drawings, similar symbols are treated as ordinary components unless otherwise indicated in the context. The exemplary embodiments described in the detailed description, drawings, and claims are not intended to limit. Other embodiments may be used and other modifications may be made without departing from the spirit or scope of this disclosure.

[0015] The nature of this disclosure relates to digital makeup artists that users can consult for makeup advice and guidance in a manner comparable to the experience they might have with a personal makeup artist. The disclosed digital makeup artists provide a makeup experience that teaches users the steps to achieve a desired look on their face.

[0016] Figure 1 is a schematic diagram of a system according to an exemplary embodiment of the present disclosure. Embodiments include a software application or a mobile application (app). For the purposes of this disclosure, the term mobile application (app) will be used interchangeably with software application, and makeup application will be used to refer to the process of applying makeup virtually or physically. A software application may run on a desktop computer or laptop computer 103. A mobile application may run on a tablet computer or other mobile device 101. For the purposes of this disclosure, software applications and mobile applications will be described in terms of a mobile application 111. In each case, the mobile application 111 may be downloaded and installed on the respective devices 101, 103. In some embodiments, the desktop computer or laptop computer 103 may be configured to include a microphone 103a as an audio input device. The microphone 103a may be a device that connects to the desktop computer or laptop computer 103 via a USB port or an audio input port, or wirelessly via the Bluetooth radio protocol. The mobile device 101 may have a built-in microphone. In some embodiments, the software application or mobile application may include communication functions to operate in conjunction with the cloud service 105. The cloud service 105 may include a database management service 107 and a machine learning service 109. The database management service 107 may be any type of database management system provided by the cloud service 105, for example, the database management service 107 may include a database accessed using a structured query language (SQL) or an unstructured database accessed by keys, commonly known as No_SQL.The machine learning service 109 can perform machine learning to enable the scaling and high-performance computing that may be required for machine learning. Additionally, software or mobile applications can be downloaded from the cloud service 105. While Figure 1 shows a single cloud service, laptop computer, and mobile device, it should be understood that any number of mobile devices, laptop computers, and desktop and tablet computers can connect to one or more cloud services.

[0017] Figure 2 is a block diagram of a mobile computer device. In one implementation, the functions and processing of the mobile device 101 may be implemented by one or more processing / arithmetic circuits 226. The same or similar processing / arithmetic circuits 226 may be applied to a tablet computer or a laptop computer. A processing circuit includes a programmed processor, such that a processor includes a circuit. A processing circuit may also include devices such as application-specific integrated circuits (ASICs) and conventional circuit components arranged to perform the functions mentioned. Note that "circuit" refers to a circuit or a system of circuits. In this specification, a circuit may be located within a single computer system or distributed across a network of computer systems.

[0018] Next, the hardware of the processing / arithmetic circuit 226 according to an exemplary embodiment will be described with reference to Figure 2. In Figure 2, the processing / arithmetic circuit 226 includes a mobile processing unit (MPU) 200 that performs the processing described herein. Processing data and instructions may be stored in memory 202. These processing and instructions may also be stored in a portable storage medium or remotely. The processing / arithmetic circuit 226 may have an interchangeable subscriber identification module (SIM) 201 that contains information specific to the network services of the mobile device 101.

[0019] Furthermore, the inventive step of the present invention is not limited by the form of the computer-readable medium in which the instructions for the processing of the present invention are stored. For example, the instructions may be stored in any other information processing device with which the processing / arithmetic circuit 226 communicates, such as flash memory, synchronous random access memory (SDRAM), random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), solid-state hard disk, or a server or computer.

[0020] Furthermore, the inventive step of the present invention may be provided as a utility application, background daemon, or operating system component, or a combination thereof, that runs in conjunction with the MPU200 and the operating system. The operating system may be for a laptop or desktop computer, such as Mac OS, Windows 10, or a Unix operating system. For mobile devices such as smartphones or tablet computers, mobile operating systems such as Android, Microsoft® Windows® 10 Mobile, Apple iOS®, and other mobile operating systems known to those skilled in the art may be used.

[0021] To realize the processing / arithmetic circuit 226, the hardware elements can be implemented by various circuit elements known to those skilled in the art. For example, the MPU 200 may be a Qualcomm mobile processor, an Nvidia mobile processor, an Intel Atom® processor, a Samsung mobile processor, or an Apple A7 mobile processor, or any other processor type recognizable to those skilled in the art. Alternatively, the MPU 200 may be implemented using a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or discrete logic circuits recognizable to those skilled in the art. Furthermore, the MPU 200 may be implemented as multiple processors that work in parallel to execute the processing instructions of the present invention as described above.

[0022] The processing / arithmetic circuit 226 in Figure 2 also includes a network controller 206, such as an Intel Ethernet PRO network interface card from Intel, Inc., for interfacing with the network 224. As can be understood, the network 224 can be a public network such as the Internet, or a private network such as a LAN or WAN network, or any combination thereof, and may also include a PSTN or ISDN subnetwork. The network 224 may also be wired, such as an Ethernet network. The processing circuit may include various types of communication processors for wireless communication, including 3G, 4G and 5G wireless modems, WiFi®, Bluetooth®, GPS, or other known forms of wireless communication.

[0023] The processing / arithmetic circuit 226 includes a Universal Serial Bus (USB) controller 225 which can be managed by the MPU 200.

[0024] The processing / arithmetic circuit 226 further includes a display screen controller 208, such as an NVIDIA® GeForce® GTX or Quadro® graphics adapter from NVIDIA, Inc., for interfacing with the display screen 210. The I / O interface 212 interfaces with buttons 214, such as for volume control. In addition to the I / O interface 212 and the display screen 210, the processing / arithmetic circuit 226 may further include a microphone 241 and one or more cameras 231. The microphone 241 may have associated circuitry 240 for processing sound into a digital signal. Similarly, the camera 231 may include a camera controller 230 for controlling the image acquisition operation of the camera 231. In an exemplary embodiment, the camera 231 may include a charge-coupled device (CCD). The processing / arithmetic circuit 226 may include audio circuitry 242 for generating sound output signals and may include an optional sound output port.

[0025] The power management and touchscreen controller 220 manages the power used by the processing / arithmetic circuit 226 and touch control. The communication bus 222 may be an Industry Standard Architecture (ISA), Extended Industry Standard Architecture (EISA), Video Electronics Standards Association (VESA), Peripheral Interface (PCI), or similar, for interconnecting all components of the processing / arithmetic circuit 226. A general description of the features and functionality of the display screen 210, buttons 214, and the display screen controller 208, power management controller 220, network controller 206, and I / O interface 212 is omitted herein for brevity, as these features are publicly known.

[0026] Figure 3 shows a user interface screen having an avatar according to an exemplary embodiment of the present disclosure. For the purposes of the present disclosure, the disclosed avatar is a graphic representation of a digital makeup artist, but may also be an image of the makeup artist 305 and may represent a person who can communicate with the user using speech or text. Speech input can be activated by selecting an icon 311 representing a microphone. A subscreen 301 containing the avatar or image 305 may provide an area for interaction. The avatar or image 305 may output speech to the computer's audio output 242 or to an external speaker connected to the computer 101. The avatar or image 305 may output text 309. The subscreen may include an input box 307 in which the user can type text. In some embodiments, when the user speaks into the microphone 241, the speech is converted to text and the text is displayed in the input box 307. The subscreen 301 may be contained within a user interface window 310. The window 310 may be a graphical object controlled by the operating system of the computer system 103. Window 310 may display a browser window or window of a software application or mobile application. Window 310 may include a menu icon 303, and selecting the menu icon 303 may display a menu of items that can be selected and perform the specified function.

[0027] Avatar 305 may be implemented as a software object that performs animation. In some embodiments, as further described below, avatar 305 may be synchronized with a video played by video component 320. Control of video component 320 may be performed by voice commands made using microphone 241 when microphone icon 311 is activated. Avatar 305 may respond to speech input by performing natural language processing on the input and outputting a speech response through voice output 242. To synchronize the avatar with the video, avatar 305 may transfer messages to video component 320, and video component 320 may share information such as timing information with avatar 305. The video played by video component 320 may be divided into chapters. Chapters may be identified by chapter name and time. Time may be the number of seconds from the beginning of the video, or a percentage of the total length of the video. In some embodiments, the video is a series of makeup application steps in which video frames are generated based on images of the user's face and the makeup being applied. For the purposes of this disclosure, the makeup application step may include applying a specific type of makeup to a specific part of an image of the user's face. The specific type of makeup may include a specific cosmetic product and the specific properties of the cosmetic product.

[0028] In some embodiments, the avatar 305 is implemented as an interactive agent that can receive questions and provide solutions to those questions. The interactive agent can also respond to certain types of questions asked by the user. The interactive agent can respond to user statements, which are responses to questions asked by the interactive agent. The interactive agent stores state information in the cloud service 105. In some implementations, the interactive agent may be pre-built using software tools such as the Bot Framework SDK available on the Microsoft Azure cloud service. The operation of the interactive agent will become clearer in the following description.

[0029] Figure 4 shows a user interface screen for a mobile device according to an exemplary embodiment of the present disclosure. In some embodiments, the user interface is provided for smaller screens, such as the screen of a smartphone 101. By providing a smaller screen, interaction with the avatar 305 can be done via voice and / or text. Speech input can be activated by selecting an icon 311 representing a microphone. Part of the display screen 210 may display controls and text related to guiding the interaction. The avatar 305 can output speech to the computer's audio output 242 or to an external speaker connected to the computer 101. The avatar 305 can output text 309 to the display screen 210. Part of the display screen 210 may include an input box 307 in which the user can type text. In some embodiments, when the user speaks into the microphone 241, the speech is converted to text and the text is displayed in the input box 307. The display screen 210 may display an image or video 401 of the user's face.

[0030] Figure 5 is a schematic sequence diagram for a user-to-digital makeup artist interaction for custom instruction, according to an exemplary aspect of this disclosure. The digital makeup artist 520 may take the form of an avatar 305 or may be an interactive agent that interacts through speech. The digital makeup artist 520 may be run by a mobile application 111 and may represent a fictional person or an actual makeup artist. The system 510 may be a desktop computer, a laptop computer 103, a tablet computer, a mobile device 101, or another computer system with display capabilities. The system 510 may be a combination of a desktop computer, a laptop computer, a tablet computer, or a mobile device with a cloud service 105. The system may have voice capabilities. User input 530 may take the form of a text input area and / or microphone input.

[0031] Conventional makeup application instructional videos include basic video functions such as play, pause, and stop, and may include a time slider for moving the start time of video playback while the video is paused. In the disclosed embodiment, a digital makeup artist 520 is used to provide interaction in custom instruction.

[0032] A user can use the mobile application 111 for purposes such as a use case where the user has a look in mind and wants a digital makeup artist to show them how that look can be created. The mobile application 111 may provide customized guidance for creating the look. In 501, the digital makeup artist 520 of the mobile application 111 may conduct an initial conversation with the user 530 to obtain some information about the user and, in particular, the user's needs. The digital makeup artist 520 may initiate the conversation by displaying and / or speaking an initial question such as, "What type of look would you like to create?"

[0033] In 503, the user can describe the desired type of look. The user's statement can be entered as text in input block 307. Alternatively, the user can choose to interact with the digital makeup artist 520 by voice by selecting the microphone icon 311. System 510 may extract keywords from the user's statement to be used in selecting a pre-stored look from database 107. Figure 6 shows the user interface for the user to enter a description of the type of look.

[0034] The user interface can offer a list of various look types. Examples of look types include seasonal looks (spring, summer, fall), event looks (date night, dinner with girlfriend, special date, outing with mother-in-law, holiday, party, New Year's Eve, bridal, prom), looks based on time to completion (quick makeup, average makeup, time-consuming makeup), mood looks (cheerful, happy, notice me), style (natural, evening, night out, gothic, work, beach), and aesthetic looks (VSCO, eGirl, soft girl).

[0035] The digital makeup artist 520 may also display and / or speak questions such as, "What is your experience level in applying makeup?" to obtain information about the user's experience level. The user may be provided with a list of experience levels from which to select, or may voice-enter their experience level. In one or more embodiments, information about the user 530, including their experience level, may be found in a stored user profile.

[0036] Experience levels can include beginner, experienced, expert, and professional. A beginner may be a user with little to no experience applying makeup. An experienced user may have applied makeup before and therefore possess some experience. An expert may have applied makeup for a certain period, such as a year or more, and have also followed the steps to learn how to apply makeup properly. A professional may be a user who applies makeup to others.

[0037] Digital makeup artist 520 may also display and / or discuss questions such as, "Is your makeup look primarily for indoor wear or outdoor wear?" to further enhance the type of makeup that can be recommended.

[0038] The digital makeup artist 520 may also display and / or speak questions such as, "Is your skin dry?" to obtain information about the user's skin condition that may be difficult to obtain through image processing techniques. Other questions may concern problem areas of the face and favorite areas of the face that the user would like to be considered.

[0039] The digital makeup artist 520 can perform customized instruction to teach the user the steps to create their desired look. The digital makeup artist 520 can use the user's own face to teach the steps, and the steps can apply makeup in a way that creates the look that will appear when applied to the user's face. In 505, the camera 231 can be used to capture an image or video of the user's face. The user can view the captured image or video on the display screen 210.

[0040] In 507, the system 510 may analyze an image or video to obtain information about the user's face. For example, an image may be analyzed to identify faces contained in the image, to determine the shape of the face, to determine facial features, and to determine other characteristics that may be used to provide advice during instruction. Conventional image processing algorithms have been used to identify facial features in an image, such as image segmentation, brightness levels, and average color. However, in recent years, artificial neural networks for face recognition have been developed. Some artificial neural networks for face recognition are based on the structure of the human visual cortex. The architecture of artificial neural networks for computer vision continues to improve. For the purposes of this disclosure, we describe two non-restrictive artificial neural networks that may be used to analyze captured images or videos.

[0041] In some embodiments, one or both of the artificial neural networks may be run on a mobile device 101 or a laptop computer 103, provided that there are sufficient computing resources to perform the processing of the artificial neural networks. In some embodiments, one or both of the artificial neural networks may be run on a machine learning service 109 within a cloud service 105. In some embodiments, the artificial neural networks may be trained within the machine learning service 109, and the trained artificial neural networks may be run on the mobile device 101 or a laptop computer 103.

[0042] To perform face recognition, the mobile application 111 may perform image processing operations to improve the facial features of the image, such as improving lighting. For example, a user may unknowingly take a selfie when bright light or sunlight is coming from behind them. The mobile application 111 may brighten the user's facial image. Other image processing operations may be performed to improve image quality.

[0043] Figure 7 is a flowchart illustrating the face analysis steps in more detail.

[0044] In 701, analysis may be performed on the captured image to determine the shape of the face. A machine learning model may be used to detect the shape of the captured user's face. The machine learning model may be trained to classify face shapes using face images with known face shapes. In recent years, image classification has been performed using artificial neural networks that are inspired by how the visual cortex of the human brain works when recognizing objects. These image classification artificial neural networks are a group of models known as convolutional neural networks (CNNs). Other approaches for image classification have been proposed and are being continuously improved. Some other approaches that may be used for such image classification include linear regression, decision trees, and support vector machines. As mentioned above, the machine learning model may be trained using the machine learning service 109 of the cloud service 105.

[0045] In the disclosed embodiments, the architecture of a machine learning model that may be used to classify face shapes is a CNN. Figure 8 is a block diagram of a CNN for classifying face shapes. The dimensions and activation function of the CNN may vary depending on the available processing power and the desired accuracy. Dimensions include the number of channels, the size of each layer, and the number of layers. Activation functions include logistic and rectified linear units.

[0046] A convolutional neural network (CNN) may consist of several types of layers. The convolutional component 803 may consist of a convolutional layer 803a, a pooling layer 803c, and a rectified linear unit layer 803b. The convolutional layer 803a is for unfolding a two-dimensional activation map that detects specific locations of features at all given spatial locations. The pooling layer 803c functions as a form of downsampling. The rectified linear unit layer 803b applies an activation function to increase the decision function and the nonlinear properties of the entire network without affecting the receptive field of the convolutional layer itself. The fully connected layer 805 contains neurons that have connections to all activations of the previous layer. The loss layer specifies how network training fills the deviation between the predicted layer and the true layer. The loss layer 807 detects a class in a set of mutually exclusive classes.

[0047] In some embodiments, the loss layer 807 may be a softmax function, which provides probabilities for each class. In exemplary embodiments, class 809 may include square, rectangular, circular, elliptical, oblong, rhombus, triangular, and heart-shaped face shapes.

[0048] In 703, the mobile application can analyze facial features and landmarks. Similar to facial shape, the captured user's facial features and landmarks can be detected using a machine learning model. The machine learning model can be trained to detect facial landmarks. Similar to facial shape classification, other approaches to classification can be used. Also, a CNN architecture similar to that in Figure 8 can be used for facial landmark detection.

[0049] Figure 9 is a schematic diagram of a deep learning neural network for facial landmark detection. Similar to the architecture in Figure 8, the deep learning neural network is a convolutional neural network. Residual connections may be included to improve the training of the deep learning neural network. In one or more embodiments, an inverted residual structure may be included in which residual connections are made to previous layers in the network. With respect to the architecture in Figure 9, the network is provided as two stages, 903 and 905. The first stage 903 is a convolutional stage for performing feature extraction. The second stage 905 performs prediction in the region of interest.

[0050] The architecture of the first stage 903 includes a convolution section 903a that performs convolution and maximum pooling operations when given an input face image 901. The convolution section 903a is connected to an inverted residual structure 903b. A mask layer 903c is connected to the inverted residual structure 903b. The size of the mask layer 903c is a size corresponding to the number of landmarks (e.g., 2 × L, which is the number of landmarks). The mask layer 903c encodes the spatial layout of the input object.

[0051] The architecture of the second stage 905 includes an inverted residual structure 905b connected to the inverted residual structure 903b of the first stage 903. The mask layer 903c of the first stage 903 is applied to the result of the inverted residual structure 905b and provided as input for performing ROI (Region of Interest) cropping in the concatenate block 911. The ROI and concatenate block 911 are based on the number of channels and landmarks in the inverted residual structure 905b. The prediction block 913 predicts the landmarks and their approximate locations within the mask layer 905c. The predictions for the region of interest in the second stage 903 are combined with the landmarks estimated by the mask layer 903c for the entire image, resulting in the output landmark in the output layer 907.

[0052] In one or more embodiments, landmarks on the face include the eyes, nose, lips, cheekbones, the area around the eyes including eyebrows and eyelids, and hair. In some embodiments, landmarks may include possible facial abnormalities.

[0053] The specific dimensions and number of layers for each layer may depend on parameters including the desired accuracy, the hardware on which the machine learning model is run, and the length of time required to train the machine learning model. The machine learning model can be trained using the machine learning service 109 of the cloud service 105.

[0054] The facial feature analysis 507 may further include the detection of lip shape 705, eyelid shape 707, and hairstyle 709. The detected landmarks can be used to calculate the contours of the lips, eyes, and hairstyle. In addition, other facial features such as skin color / tone, eye color, lip color, hair color 711, and skin texture 713 can also be determined from the facial image. Skin color / tone and skin texture can be determined using image processing techniques. Skin color / tone can be analyzed to determine RGB coordinates and assign names to the skin color / tone. Skin color analysis can reveal differences in skin tone. Skin tone types may include fair, light, medium, or deep. Similarly, eye color, lip color, and hair color can also be determined using image processing techniques. Skin texture can be analyzed using brightness difference analysis.

[0055] Further features of a facial image may include illumination (image brightness). In 715, image illumination (brightness) may also be determined using image processing techniques. Brightness can be defined as a measure of the total amount of perceived light in an image. In some embodiments, the brightness of an image may be increased or decreased from the initial brightness level, starting from the initially captured brightness level.

[0056] In 717, past looks preferences can be retrieved from database 107. Past looks preferences may include cosmetic characteristics, including the range of colors, tones, and finishes used in past looks. Past user preferences may include cosmetic characteristics for specific parts of the face, and may include cosmetic choices applied to specific looks, particularly specific parts of the face.

[0057] In 509, one or more makeup filters may be selected / retrieved from the database 107 based on facial features (facial parts) determined by facial analysis (507 and Figure 7) and past looks preferences. Some of the stored makeup face filters may be filters previously created through custom guidance or the generation of custom looks during a makeup consultation with a digital makeup artist 520.

[0058] The makeup filter is a feature mask that can be blended with a base frame, i.e., an image containing a front view of the user's face. The feature mask is blended with the base frame to produce an image with a resulting face image that has modified features. The app can adjust the feature mask to align with the face features in the base frame using the boundary positions of the face features determined using face analysis 507. The feature mask consists of RGB pixel values.

[0059] One or more makeup filters can be retrieved from database 107 using a recommender system. Figure 10 is a schematic diagram of the recommender system. The recommender system 1000 can be used to retrieve makeup filters used when demonstrating how to apply virtual makeup (515 in Figure 5). The recommender system 1000 operates based on image data and an indexed database 1005 of makeup filters that can be stored in database 107.

[0060] Indexed database 1005 may include answers to common makeup questions, general makeup looks, and cosmetics tailored to specific skin types (skin tone, dryness, texture) and / or specific ethnicities of individuals. Indexed database 1005 may also include makeup information extracted from external databases or websites, such as product review information including ratings and public comments.

[0061] The indexed database 1005 may further include a category of cosmetics, which includes a category of makeup that offers a combination of functions such as appearance and skincare. Cosmetics in this category may include foundations with anti-aging properties or sun protection, and cosmetics blended with medicines or other skin treatment products. Another category may be skincare products for skincare routines performed before applying makeup.

[0062] The recommender system 1000 includes a recommendation engine 1007 that searches for and ranks recommended makeup filters. When applying a specific virtual makeup look, the recommended makeup filters may be for the look and virtual makeup entered by the user in step 501. In some embodiments, the recommended makeup filters may be searched based on the user's preferences or favorites. Personal user preferences may be makeup characteristics entered by the user when the app was first set up. Personal user preferences may be one or more makeup filters previously built during custom instruction or consultation with a digital makeup artist 520. Favorites may be makeup characteristics that the user has flagged as favorites. Personal user preferences and favorites may be for specific parts of the face or for the entire face.

[0063] In one or more embodiments, the recommendation engine 1007 may use a looks feature matrix. Figure 11 shows a non-limiting looks feature matrix according to an exemplary aspect of the present disclosure. For brevity, the looks feature matrix shown in Figure 11 is a partial matrix showing two types of virtual makeup. Other types of virtual makeup that may be included in the looks feature matrix include, but are not limited to, foundation, mascara, concealer, blush, and eyebrow pencil. The looks feature matrix may be stored in an app on a mobile device so as to be compared against a vector of desired features. The desired features may be a vector of the user's current preferences and may take into account the user's current experience level and desired looks (501). The recommendation engine 1007 may rank recommendations using one or more similarity metrics and scoring algorithms. In one embodiment, the recommendation engine 1007 may generate a set of features that enhance recommendations in order to encourage creativity by modifying specific properties for virtual makeup from those that have been recommended. For example, if the recommendation engine 1007 ranks a recommendation higher among the searched recommendations, it may modify one or more characteristics to increase the similarity score. Alternatively, the recommendation engine 1007 may change one or more characteristics in the searched recommendation, such as color tone or finish, up or down by one level (for example, changing the color tone one level up or down from the stored color tone). In one or more embodiments, the recommendation engine 1007 may adjust the application gestures to be approximately accurate based on the user's experience level.

[0064] In one or more embodiments, the recommendation engine 1007 may be implemented by or complemented by a machine learning model. The machine learning model may be trained to select foundation shades according to a specific skin undertone, skin type, and / or ethnicity. The machine learning model may be trained to select lipstick shades based on the user's lip color. The machine learning model may be trained to select eyeshadow shades based on the user's eye color and skin undertone.

[0065] Machine learning models for the recommended engine 1007 can be trained using data from the indexed database 1005, as well as data from external databases, particularly those containing images and videos where users can publish their custom looks. Examples of external databases include, to name a few, publicly available databases from social media platforms such as Facebook, Instagram, Snapchat, and TikTok, and video conferencing platforms such as Zoom, Microsoft Teams, Google Hangouts, or Google Meet.

[0066] Machine learning models for the recommended engine 1007 can be based on algorithms for relatively small datasets, including decision trees, random forests, or single-layer neural networks—perceptrons. Providing a large dataset consisting of thousands of training data examples makes it possible to use deep learning neural networks as machine learning models. The architecture of the deep learning neural network may be a variation of a convolutional neural network for recognizing features in color images.

[0067] The recommendation engine 1007 may output one or more recommendations to the recommendation user interface (511). The recommendation user interface may also display a series of video frames illustrating the application of the selected recommendations.

[0068] In 511, one or more appearance options may be displayed on the display screen 210.

[0069] Figure 12 shows an exemplary user interface in a mobile application. The digital makeup artist 520 may display custom recommendations 1201 of various looks. In 513, the user may select a recommended look. Looks can be selected by moving a pointer on a pointing device on the display screen 210, by using a pointing device on the touchscreen 221, or by entering a voice command such as "Select look 1". There may be more custom recommendations that can be displayed on the display screen 210. The user may view other recommended looks, for example, by selecting a function to view the next screen of looks. In one embodiment, the user interface may provide a scroll bar 1203 that may allow scrolling to display additional looks.

[0070] When the user selects a look, in 515, the system 510 may begin executing steps to create the selected look. Figure 13 shows a user interface that may be displayed to initiate instruction in a mobile application according to an exemplary aspect of this disclosure. During instruction, in 517, information may be fed back to the system 510 to improve future interactions. Instruction is also interactive. Instruction can be controlled using commands including “Slow Down,” “Pause,” “Redo a Step,” and “Skip a Step.” Message 1301 may be displayed to inform the user that commands are available to control the instruction. Commands can be selected on the user interface using a pointing device, or entered as oral commands if the microphone 241 is active (e.g., via the microphone icon 311). The instruction being played can be modified to obtain a custom instruction experience. For example, the “Redo a Step” command may include options for making changes to the instruction. System 510 can store the changes made to the instruction in database 107 as look preference data that can be used to improve future interactions. As described above, a version of the mobile application 111 may be for a device with a smaller screen than a laptop or desktop computer. Figure 14 is an exemplary mobile application according to an exemplary aspect of the present disclosure. With a smaller screen provided, interaction with the digital makeup artist 520 may be via voice and / or text. In that case, commands can be entered verbally. During the instruction, a facial image of user 1401 may be displayed, and the instruction may add makeup 1403a of a color 1403b selected from the makeup palette 1403 to a part 1405 of the facial image 1401.

[0071] Figures 15, 16, 17, and 18 are flowcharts for video control commands. As mentioned above, instruction can be controlled using commands including "slow down," "pause," "redo step," and "skip step." Figure 15 is a flowchart for the "slow down" video control command. The "slow down" command can perform the function S1501, which reduces the playback speed by a certain amount (X%). In addition, calling the "slow down" command may indicate that the user believes that a part of the video being played is complex or requires careful viewing. Executing the function to reduce the playback speed may include storing information indicating that the step is complex in S1503.

[0072] Figure 16 is a flowchart of the "Pause" video control command. The "Pause" command can execute function S1601, which stops video playback at a certain time point (position in seconds, or time code).

[0073] Figure 17 is a flowchart of the “Redo Step” video control command. The “Redo Step” command can perform functions that utilize chapters. In some embodiments, the beginning of each chapter may include a starting image frame and a time point. Alternatively, the beginning of each chapter may be marked by a chapter frame. The “Redo Step” command can perform functions that return to the beginning of the current chapter and start playback from the starting image of that chapter. In S1701, this function may include reading the name of the current chapter. In S1703, this function may store information indicating that the steps associated with the current chapter are complex. In S1705, this function may read an identifier indicating the starting position of the chapter, which may be a time point or a chapter frame.

[0074] As will be described later, in some embodiments, video frames may be dynamically generated, for example, before chapter playback or as video frames are played. Dynamically generating video frames allows changes to be incorporated into the video, for example, by changing to use different color tones and / or finishes of cosmetics. The beginning of a chapter includes a start image, which may be a combination of a face image and a mask filter. Makeup applications performed during chapter playback may be saved as the next state of the face image combined with the mask filter.

[0075] In S1707, the video can be played from the beginning of the chapter.

[0076] Figure 18 is a flowchart of the “Skip Step” video control command. The “Skip Step” command can perform a function that skips the video to the beginning of the next chapter. This function can be started in S1801 by reading the name of the current chapter. Skipping a step may indicate that the step is simple and that the user does not need to be shown how to perform it. In S1803, this function may store information indicating that the step is simple. In S1805, this function may read the name of the next chapter, and in S1807, it may read the time point associated with the next chapter. In S1809, this function may reset the video to start from the time point of the next chapter.

[0077] Figure 19 is a block diagram of a video playback component according to an exemplary embodiment of the present disclosure. In some embodiments, the video playback component 320 may generate an instructional video before playing the video. In particular, the video playback component may perform operations on one or more frames in the instructional video before rendering the frames for display. The instructional video may also be divided into chapters. Chapters may be used to divide a makeup instructional video into individual steps, and each step in a chapter may not only be played but also paused or stopped. Each chapter may be designed by a starting frame. Chapters in the video may be for steps of applying makeup to a specific part of the user's face, or for applying a specific type of makeup. The video may begin with an image of the user's face captured by a camera. The video may utilize information obtained through analysis of the face image, such as the position and label of parts of the user's face image, skin tone, texture, and lighting. The video may also utilize a look chosen by the user. By default, the video may be for instruction on how to create a selected look using the user's facial image and information obtained from facial analysis.

[0078] As described above, video chapters may be for steps of applying makeup to specific parts of the user's face, or for applying a specific type of makeup. For example, one chapter may be for a step of applying concealer to the eyelids of the user's face image. The concealer may be a cosmetic product provided for a selected look. In some embodiments, the makeup look may have a set of chapters, each chapter may contain one or more cosmetics, the cosmetics having a set of characteristics such as color, range, tone, and finish. The video playback component 320 may generate a look selected for a specific user's face and cosmetics. A chapter may begin with an image of the user's face made up in a previous chapter, so that the chapter represents the cumulative result of the makeup application. In some embodiments, the cumulative result is one or more mask filters separate from the original face image.

[0079] With respect to Figure 19, a face image, or a mask filter of face image 1903, may be provided at the beginning of a chapter as a starting image from which further digital makeup is applied for instruction. The beginning of a chapter may include positional information 1907 of the part of the face (facial feature) to which the cosmetics are applied, and information about the cosmetics, their properties, and the type of strokes that may be applied by a particular type of makeup applicator. The chapter may include an accompanying audio component 1901. The chapter may include a series of mask filters 1905 of the desired facial features that can be used to create video frames for instruction. Each video frame may be generated by blending the preceding frame 1911 with the desired facial features 1913 to obtain the resulting facial features 1915. One or more feature masks may be used to represent applying specific makeup using a makeup applicator for a particular facial feature.

[0080] Figure 20 illustrates a blending process that can be used to create video frames based on desired and original facial features. Blending of facial features is achieved as follows: 1. The desired facial features 2001 are recolored (2003) to harmonize with the original facial features' color(s), resulting in the recolored facial features 2005. 2. The recolored facial features (2005) are then subjected to a feature mask filter (2007). 3. The original facial features (2009) are multiplied by the inverse of the feature mask filter (2011), which is the value obtained by subtracting each mask value in the range of 0 to 1 from 1. 4. The obtained images 2 and 3 are added together pixel by pixel (2013) to create the final blended facial features image 2015.

[0081] When instruction is complete, or when the user has completed as much instruction as they wish, the user interface may offer an option to save the completed look. If the user chooses to save the completed look, in 519 "Yes", in 521 the look may be saved as a completed look, or the steps taken to apply makeup during instruction may be saved as a custom filter, or a set of custom filters. In addition, in 523 the user interface may offer an option to transfer the custom look, or to apply / transfer the custom filters as an image to another platform (525).

[0082] An exemplary operation of the system is provided in Figures 21A–21F. Figures 21A–21F are sequence schematics for an exemplary interaction use case using the digital makeup artist 520 for makeup instruction, according to an exemplary aspect of the present disclosure. The sequence schematics in Figures 21A–21F include operations and communication by the system 510, the digital makeup artist 520, and the user 530. As described above, the system 510 may be a computer device such as a desktop computer, laptop computer 103, tablet computer, or mobile device 101, to name a few common types. The system 510 may also take the form of a combination of a computer device and a cloud service 105 (see Figure 1). In any case, the digital makeup artist 520 may be run as part of a mobile application 111, and the user 530 may interact with the digital makeup artist 520 through speech, text, or a combination of speech and text.

[0083] User 530 may first select a mobile application to run on the user interface of a computer device. Once the mobile application is started, User 530 may be given the choice of using the digital makeup artist 520 to receive instruction in a tutorial or to receive advice in a makeup consultation session (described later). If the user chooses to use the digital makeup artist 520 for tutorial, in 2101, the digital makeup artist 520 may ask, "What kind of look would you like to create?" In 2103, User 530 may state a name for the look. The look may be an existing type of look, or it may be the name of a new look that User 530 has in mind. Alternatively, System 510 may provide User 530 with a list of existing looks to choose from. Existing looks may be stored in Database 107, or they may be provided locally in App 111.

[0084] In some embodiments, in 2105, the digital makeup artist 520 may ask the user 530 to provide their level of experience in applying makeup. The level of experience may be categorized as beginner, experienced, expert, and professional. The system 510 may present a list of experience levels for the user 530 to select. In 2107, the user 530 indicates their level of experience.

[0085] In 2109, the digital makeup artist 520 may ask the user 530 which part of the face they would like the makeup applied to in the instruction. The instruction may be provided for a part of the face or for the entire face. In 2111, the user may choose to have the instruction presented for a part of the face, such as the eyes.

[0086] In some embodiments, in 2113, the digital makeup artist 520 may ask the user 530 how much time they have for the instruction. The instruction may be a shortened version if the user has little time, or a full version if the user has enough time for the full instruction. The user 530 may provide a qualitative response, or a response that is a quantity of time in minutes. In 2115, the user 530 may provide a qualitative response such as, "I don't have much time." The system 510 may interpret this response as meaning that a shortened version of the instruction should be performed. However, the user 530 may also indicate that they are willing to save the instruction in an intermediate state for later completion. In the latter case, the system 510 may allow the instruction to be saved at a point and played back starting from that intermediate point.

[0087] In some embodiments, in 2117, the digital makeup artist 520 may ask the user 530 to decide whether the user 530 wants to use any necessary makeup and makeup applicators that are already available, such as cosmetics purchased by the user 530, or whether the user 530 prefers that the system 510 be able to provide any makeup and makeup applicators that may be needed for instruction. In 2121, the user 530 may have some cosmetics but provide a qualitative response indicating acceptance of the system 510 selecting the makeup and makeup applicators.

[0088] In 2123, the digital makeup artist 520 may ask the user 530 to take a picture of their face. The picture may be taken with the camera 231 built into the mobile device 101, or with an external camera.

[0089] In 2125, system 510 may perform facial image analysis. As described above, facial analysis may be performed to obtain the position of facial features, as well as facial characteristics including skin color, skin texture, lighting, and past looks preferences, which are information that can be used when generating instructional videos.

[0090] In 2127, system 510 may select makeup to be applied during instruction.

[0091] In 2129, the digital makeup artist 520 may ask the user 530 if they like the makeup selection. In 2131, the user 530 may respond that they do not agree with the selection and would rather the instruction be performed using the makeup provided by the user.

[0092] In 2133, the digital makeup artist 520 may indicate that instruction is about to begin. In some embodiments, in 2135, the digital makeup artist 520 may provide a list of products that may be used during instruction if the user 530 wishes to obtain products to apply makeup during the instruction.

[0093] In 2141, system 510 may begin generating and playing an instructional video. In an example instruction, digital makeup artist 520 may begin instruction by describing the initial makeup that can be applied. In 2143, digital makeup artist 520 may provide spoken or texted instructions indicating that concealer is applied first, followed by primer.

[0094] The instructional video can be played back by the system 510 in conjunction with instructions from the digital makeup artist 520, either through speech output, text output, or both speech output and text output. In 2145, the digital makeup artist 520 may provide instructions for selecting eyeshadow. In 2147, the digital makeup artist 520 may provide instructions for selecting a blending brush. In 2149, the digital makeup artist 520 may provide instructions for applying eyeshadow.

[0095] At a time point aligned with the instructions, user 530 can input video control commands. For example, at 2151, user 530 can input the “redo step” command. In some embodiments, the command may include a request to make a change to the step, such as “redo with the next color.” The digital makeup artist 520 may output a request clarifying what the user means by “the next color.” At 2153, system 510 may generate a new frame for the video, starting with the eyeshadow selection at 2145 using the new color.

[0096] In 2155, the digital makeup artist 520 may provide instructions for selecting a blending brush. In 2157, the digital makeup artist 520 may provide instructions for applying eyeshadow.

[0097] In 2159, user 530 can enter a command such as "pause". System 510 may execute the pause function until user 530 enters the "play" command.

[0098] In 2161, the digital makeup artist 520 may provide instructions for performing blending with an eyeshadow of a different shade. In 2163, the digital makeup artist 520 may provide instructions for selecting an eyeshadow of the next shade. In 2165, the digital makeup artist 520 may provide instructions for selecting a blending brush. In 2167, the digital makeup artist 520 may provide instructions for performing blending.

[0099] In step 2169, user 530 can enter a command such as "redo step" to redo the blending. In step 2171, system 510 can restart the video from the beginning of the blending step.

[0100] In 2173, the digital makeup artist 520 may provide instructions for performing a step to accentuate eyebrows. In 2175, the digital makeup artist 520 may provide instructions for performing a step to blur wrinkles. In 2177, the digital makeup artist 520 may provide instructions for performing a step to select a pearlescent concealer.

[0101] In 2179, user 530 may input that the concealer is too light and ask if the shade of the concealer can be toned down. In 2181, digital makeup artist 520 may respond by saying that they will try a different concealer, which can be done by issuing a “redo step” command that involves changing to a different concealer.

[0102] At 2183, system 510 may restart the video from the beginning of the wrinkle blurring step, depending on where the chapter starts. Next, at 2185, system 510 may select a mate finish concealer as an example and play the video with the selected concealer. At 2187, user 530 may enter the “pause” command to take time to carefully examine the results of the new concealer. At 2189, user 530 may enter the “play” command to start playing the video.

[0103] In 2191, the digital makeup artist 520 may provide instructions for performing steps to shape the area around the eye. In 2193, the digital makeup artist 520 may provide instructions for performing steps to brighten the eye.

[0104] In step 2195, system 510 may select a pencil suitable for the user's skin tone. In step 2197, system 510 may draw on the facial image using the selected pencil.

[0105] At 2199, the digital makeup artist 520 may provide instructions for performing the step of applying eyeliner. At this point, at 2201, the user 530 may choose to stop the video so that it can be restarted at a later time.

[0106] At some point in the future after the video is completed, in 2203, the digital makeup artist 520 may output a question asking whether the user wants to save the finished look. In 2205, the user may choose to save the look and instruct the system 510 to remember the look. In 2207, the system 510 saves the final look to the database 107. In some embodiments, the final look may be saved in the database as a preferred look in a particular type of look. For example, a look created using guidance may be saved as a preferred look in the type of look described first (see 503 in Figure 5). The final look may also be transferred to other platforms such as social media platforms or video conferencing platforms. Some current social media platforms to which users can post photos or videos include Facebook, LinkedIn, Instagram, TikTok, and Snapchat. Some current video conferencing platforms include Microsoft Teams, Facetime®, Google Hangouts, or Google Meet, Zoom, GoToMeeting, and Skype. In addition, changes made during instruction, such as color selection and finish, can also be saved as data on the user's visual preferences.

[0107] In some embodiments, in 2207, the system 510 may store one or more mask filters related to the creation of the final look in the database 107. One or more mask filters may be transferred to other platforms and may be used to create custom looks for other platforms.

[0108] Social media and video conferencing have not only created a need for on-the-spot makeup, but have also influenced social subcultures like Visco Girls, and aesthetic looks like E-Girls and Soft Girls. This need has led to virtual makeup looks for social media and video conferencing—in other words, augmented reality.

[0109] An exemplary operation of the system is provided in Figures 22A–22E to create a virtual trial for social media or video conferencing. Figures 22A–22E are sequence schematic diagrams for an exemplary interaction use case using a digital makeup artist 520 for makeup instruction, according to an exemplary embodiment of this disclosure.

[0110] If a user chooses to use the digital makeup artist 520 for guidance in creating a virtual trial for a video conference, in 2221, the digital makeup artist 520 may ask, "What look would you like to create?" In 2223, the user 530 may state that they would like to have an E-girl look. The E-girl look may be a look that the user wants to try out for posting on social media or when video conferencing with friends or colleagues. Existing E-girl looks may be stored in the database 107 or provided locally in the app 111.

[0111] In some embodiments, in 2225, the digital makeup artist 520 may ask the user 530 to provide a level of experience in applying makeup. The level of experience may be categorized as beginner, experienced, expert, and professional. The system 510 may present a list of experience levels for the user 530 to select. In 2227, the user 530 indicates an experience level, stating that they are experienced but it is their first time applying an E-girl look.

[0112] In 2229, the digital makeup artist 520 may ask the user 530 which part of the face they would like the makeup applied to in the instruction. The instruction may be provided for a part of the face or for the entire face. In 2231, the user may choose to be presented with instructions for the entire face.

[0113] In some embodiments, in 2233, the digital makeup artist 520 may ask the user 530 how much time they have for instruction. The instruction may be a shortened version if the user has little time, or a full version if the user has enough time for full instruction. The user 530 may provide a qualitative response, or a response that is a quantity of time in minutes. In 2235, the user 530 may provide a qualitative response such as, "I don't have much time." The system 510 may interpret this response as meaning that a shortened version of the instruction should be performed. In some embodiments, in 2237, the digital makeup artist 520 may ask the user 530 to determine whether the user 530 wants to use any necessary digital makeup and makeup applicators that are already available, such as digital cosmetics purchased by the user 530, or whether the user 530 wants the system 510 to be able to provide any digital makeup and makeup applicators that may be needed for the instruction. In 2241, user 530 can provide a qualitative response indicating that they have some digital cosmetics.

[0114] In 2243, the digital makeup artist 520 may ask the user 530 to take a picture of their face. The picture may be taken with the camera 231 built into the mobile device 101, or with an external camera.

[0115] In 2245, system 510 may perform facial image analysis. As described above, facial analysis may be performed to obtain the position of facial features, as well as facial characteristics including skin color, skin texture, lighting, and past looks preferences, which are information that can be used when generating instructional videos.

[0116] In 2247, system 510 may select digital makeup to be applied in instruction.

[0117] In 2249, the digital makeup artist 520 may ask the user 530 if they like the digital makeup selection. In 2251, the user 530 may respond that they do not agree with the selection and would rather perform the instruction using the makeup provided by the user.

[0118] In 2253, the digital makeup artist 520 may indicate that instruction is about to begin. In some embodiments, in 2255, the digital makeup artist 520 may provide a list of digital cosmetics that may be used during instruction if the user 530 wishes to obtain products to apply makeup during instruction.

[0119] In 2261, system 510 may start generating and playing an instructional video. In an exemplary instruction, digital makeup artist 520 may start the instruction by describing the initial makeup that can be applied. In 2263, digital makeup artist 520 may speak or text instructions indicating that a primer should be applied smoothly as a pre-treatment for foundation.

[0120] The instructional video can be played back by the system 510 in conjunction with instructions from the digital makeup artist 520, either through speech output, text output, or both speech output and text output. In 2265, the digital makeup artist 520 may provide instructions for selecting a foundation with a light coverage and natural finish. In 2267, the digital makeup artist 520 may provide instructions for sweeping blush across the bridge of the nose and extending slightly down the cheeks. In 2269, the digital makeup artist 520 may provide instructions for applying highlighter powder to the tip of the nose using a tapered brush.

[0121] At a time point aligned with the instructions, user 530 can input video control commands. For example, at 2271, user 530 can input a “redo step” command. In some embodiments, the command may include a request to make a change to the step, such as redoing with a higher blush. The digital makeup artist 520 may output a request to clarify what the user means by “higher blush.” At 2273, system 510 may generate a new frame for the video, beginning with sweeping the blush at 2267 using the new blush.

[0122] In 2275, the digital makeup artist 520 may provide instructions for applying a mechanical pencil to the eyebrows with light strokes. In 2277, the digital makeup artist 520 may provide instructions for combing the eyebrows.

[0123] In 2279, user 530 can enter a command such as "pause". System 510 may execute the pause function until user 530 enters the "play" command.

[0124] In 2281, the digital makeup artist 520 may provide instructions for selecting a pink-toned eyeshadow. In 2283, the digital makeup artist 520 may provide instructions for selecting a foot applicator for applying the eyeshadow. In 2285, the digital makeup artist 520 may provide instructions for applying the eyeshadow to the eyelid. In 2287, the digital makeup artist 520 may provide instructions for blending into wrinkles with a fluffy blending brush.

[0125] In step 2289, user 530 can enter a command such as "redo step" to redo the blending. In step 2291, system 510 can restart the video from the beginning of the blending step.

[0126] In 2293, the digital makeup artist 520 may provide instructions for creating wings across each eyelid using a wing stencil. In 2295, the digital makeup artist 520 may provide instructions for performing the step of applying mascara to the eyelashes with a few swipes. In 2297, the digital makeup artist 520 may provide instructions for applying a very small amount of lip gloss.

[0127] In 2299, user 530 can input a statement that the lip gloss is too sheer and ask if it can be made stronger.

[0128] At some point in the future after the video has been completed, in 2301, the digital makeup artist 520 may output a question asking whether the user wants to save the finished look. In 2303, the user 530 may choose to save the look and instruct the system 510 to save the look. In 2305, the system 510 saves the final look to the database 107. In some embodiments, the final look may be saved in the database as a preferred look in a particular type of look, such as a personal E-girl look. For example, an E-girl look created using the guidance may be saved as a preferred look in the E-girl look described earlier (see 503 in Figure 5). The final look may also be transferred to other platforms, such as social media platforms or video conferencing platforms. Some current social media platforms to which users can post photos or videos include, to name a few, Facebook, LinkedIn, Instagram, TikTok, and Snapchat. Current video conferencing platforms include, to name a few, Microsoft Teams, FaceTime®, Google Hangouts, or Google Meet, Zoom, GoToMeeting, and Skype. In addition, changes made during instruction, such as color selection and finish, can also be stored as data on appearance preferences.

[0129] In some embodiments, in 2305, the system 510 may store one or more mask filters related to the creation of the final look in the database 107. One or more mask filters may be transferred to other platforms and may be used to create custom looks for other platforms.

[0130] In addition to providing instructional services for creating custom looks, Digital Makeup Artist 520 can offer personalized makeup consultations. They can provide advice on how to enhance a user's makeup look or suggest improvements that may be added to their makeup application techniques. They can also offer advice on makeup application techniques that may better address problem areas or highlight specific facial features. Furthermore, they can advise on makeup applications suitable for different times of day (morning, noon, night), locations the user is likely to be in (primarily indoors with artificial lighting, primarily outdoors with sunlight and artificial lighting), or current skin condition (dry skin). Finally, they can offer advice on how to enhance the user's individuality.

[0131] Figure 23 is a schematic sequence diagram of an interaction between a digital makeup artist and a user for a cosmetics consultation, according to an exemplary embodiment of the present disclosure.

[0132] As described above, user 530 may be given the choice of using the digital makeup artist 520 to receive instruction during a lesson or to receive advice during a cosmetics consultation session. In 2351, user 530 can request cosmetics advice via the user interface window 310 (see Figure 3). In 2353, system 510 may request user 530 to take a photograph or video of their face. In a similar manner to that described above for the lesson, system 510 may analyze the user's facial image to identify facial features, skin tone, lip color, hair color, and skin texture. In 2355, system 510 may engage in a two-way conversation with user 530 to obtain further information related to the user's needs, including skin condition, indoor / outdoor looks, favorite facial features, and concerns about any facial features. In 2357, user 530 can input information about skin condition, e.g., dry skin, indoor / outdoor looks, favorite features, and concerns about facial features.

[0133] Given user input information and preference information that may be pre-stored in database 107, as well as mask filters for various types of looks stored in database 107, in 2359, system 510 creates one or more custom recommendations for a makeup routine. As described above, custom recommendations can be obtained using a recommender system. The recommender system 1000 may be used to search for makeup filters used when creating custom recommendations for makeup routines, and may also be used to search for makeup routines such as skincare routines. In 2361, system 510 may display one or more recommended makeup routines that have been found. Makeup routines may be generated using the blending process shown in Figure 20.

[0134] The recommender system 1000 includes a recommendation engine 1007 that searches for and ranks recommended makeup filters. The recommendation engine 1007 may also draw from external repositories for additional information, including one or more of the following: frequently asked questions and answers, general makeup looks, cosmetics for specific skin types and conditions, and ethnicity. The recommendation engine 1007 may further draw from external repositories for cosmetic categories, including skincare and makeup with skincare qualities. Makeup with skincare qualities may include SPF-containing foundations for sun protection and makeup with anti-aging qualities. When applying specific virtual makeup, the recommended makeup filters may be based on the user's favorite features and facial feature concerns entered in step 2357.

[0135] In one or more embodiments, as described above, the recommendation engine 1007 may be complemented by a machine learning model for recommending makeup shades based on skin undertone.

[0136] In some embodiments, in 2363, the user interface window 310 can improve or adjust the recommended makeup routine based on input from the user 530. Further user input may be improvements or adjustments to specific features of the recommended routine. For example, the user may input that the lipstick color is too bright. Further user input may be improvements or adjustments to the overall makeup routine. For example, the user may input that the makeup routine is too bold, or that the user prefers longwear makeup. In 2365, the system 510 may adjust the recommended makeup routine according to makeup look data that harmonizes with the user input and type of adjustment. For example, the system 510 may make adjustments by searching for mask filters for longwear makeup stored in the database 107. In 2367, the system 510 may use the retrieved mask filters to generate and display a modified makeup routine for the user's face image. In 2369, system 510 may provide recommendations regarding cosmetics that may be used to create a finished facial image using a makeup routine.

[0137] In addition, in 2371, the system 510 may store the completed facial image and adjustments to the makeup routine used to create the completed facial image in the database 107 as preferences for makeup looks. For example, a mask filter for long-wear makeup may be stored in a feature matrix as shown in Figure 11, along with a label indicating that it is a preference for makeup looks (makeup filter) for a makeup routine (look).

[0138] In one or more embodiments, 2373 allows the user 530 to choose to transfer / publish the created final face image to a platform that provides live video or still images in which the user wishes to make an impression. Platforms that provide live video include, to name a few, social media platforms and video conferencing platforms, including Facebook, LinkedIn, Google Hangouts, or Google Meet, Facetime®, Microsoft Teams, TikTok, and Zoom.

[0139] To illustrate the digital makeup artist 520, exemplary operation is provided in Figures 24A–24D. Figures 24A–24D are schematic sequence diagrams for exemplary interactions using the digital makeup artist for makeup consultation, according to exemplary embodiments of the present disclosure. The schematic sequence diagrams in Figures 24A–24D include operations and communications by the system 510, the digital makeup artist 520, and the user 530.

[0140] In 2401, user 530 can select a mobile application for a digital makeup artist. Once the user interface window 310 is provided, in 2403, user 530 can ask to get makeup advice. In 2405, the digital makeup artist 520 may ask if the user has a preferred look for which they would like advice. In 2407, user 530 can respond in a way that further clarifies the desired look. In 2409, the digital makeup artist 520 may further narrow down the type of advice to give by asking if the user has any preferences for makeup looks. User 530 may respond in 2411 that they have no preferences but would rather the system 510 choose a look for them.

[0141] In 2413, system 510 may request the user to take a photograph or video of their face and perform an analysis on the image of the user's face. The results of the analysis may include the position of the user's facial features, as well as characteristics such as skin color, skin texture, lighting, and previous looks preferences.

[0142] In 2415, the digital makeup artist 520 may ask the user if they have a favorite facial feature. The user 530 may answer with one or more favorite facial features, for example, lips, as in 2417. In 2421, the system 510 may create custom recommendations for one or more makeup routines. In 2423, the system 510 may display the makeup routine and cosmetic characteristics. In 2425, the digital makeup artist 520 may ask the user if they would like to make adjustments to the recommended makeup routine. In 2427, the user 530 may respond that the makeup looks are too prominent.

[0143] At 2429, the digital makeup artist 520 may respond by asking whether a specific part or multiple parts of the face are prominent, or whether the entire face is too prominent. At 2431, the user 530 may respond that the entire face is too prominent. At 2433, the digital makeup artist 520 may notify the user 530 that adjustments will be made to the makeup look.

[0144] In 2435, system 510 performs a make-up look adjustment, which may include searching for mask filters from database 107, and the selection of mask filters may take into account past look preferences. In 2437, system 510 may display the adjusted make-up look. In 2439, user 530 can review the adjusted make-up look and provide further feedback, such as noticing that the eyes appear a little too dark and requesting that the eyes be made brighter.

[0145] At 2441, the system 510 may make adjustments to the makeup look, and at 2443, it may display the further adjusted look. At 2445, the digital makeup artist 520 may notify the user that the system has increased the eyeshadow shade and tell the user whether the adjustments have sufficiently improved the brightness of the eyes. At 2447, the user 530 may respond that the adjustments are in the right direction, but that they might be even better with a little more tweaking.

[0146] In 2449, the system 510 may further adjust the color tone, and in 2451, it may display the further adjusted appearance. In 2453, the digital makeup artist 520 may again inform the user that the system has increased the color tone and again ask the user 530 whether the adjustment is sufficient. In 2455, the user 530 may respond that the adjustment looks good.

[0147] Once the makeup look is complete, in 2461, the system 510 may display the final makeup look. In addition, in 2463, the system 510 may display the cosmetics that may be used to create the final makeup look, and in 2465, the makeup routine, the final makeup look, and any adjustments made may be stored in the database 107 as the user's look preference.

[0148] Through interaction with the digital makeup artist 520, the user 530 can improve their makeup look and enhance their makeup application techniques. The digital makeup artist 520 provides personalized makeup advice. The digital makeup artist 520 continuously improves its recommendation level through the accumulation of the user's look preferences and custom looks. The digital makeup artist 520 can allow the user to try out makeup for themselves before applying it to their face and teach them how to apply makeup to create custom looks. In addition, the digital makeup artist 520 can create custom makeup looks based on stored look preferences and information about the user's facial features.

[0149] In the embodiments, unless otherwise specified, words such as "a" and "an" generally mean "one or more."

[0150] Numerous modifications and variations of the present invention are possible in light of the above teachings. For example, data collected from the skin tones and textures of various consumers would allow for scaling of the artificial neural network to more consumers than to a single consumer. The artificial neural network could predict the rendering of a new cosmetic formulation for each product shade. As a result, it should be understood that, within the scope of the appended claims, the present invention may be carried out in ways other than those specifically described herein.

[0151] As a result, it should be understood that the present invention may be implemented in ways other than those specifically described herein, within the scope of the attached claims.

[0152] The above disclosure also includes the embodiments listed below.

[0153] (1) Digital Makeup Artist System. The digital makeup artist system includes a mobile device having a display device, a processing circuit, and memory; a database system that stores makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's look preferences; and a machine learning system for analyzing facial images, wherein the mobile device includes a user interface for interacting with the digital makeup artist, and the digital makeup artist engages in two-way interaction with the user to capture the user's needs, including one or more of the following: type of makeup look, indoor or outdoor look, skin condition, facial problem areas, and favorite facial features. The arithmetic circuit is configured to take the user's face image as input, analyze the user's face image via the machine learning system to identify facial features, analyze the face image to determine facial features including one or more of skin tone, eye color, hair color, lip color, and skin texture, and generate an image frame to be displayed on the display device in synchronization with the interaction with the digital makeup artist, the image frame being generated based on one or more of the user's analyzed face image, the user's needs obtained through the interaction with the user, the stored makeup routine information, general makeup looks, cosmetics appropriate for skin type and ethnicity, and the user's preferences for looks.

[0154] (2) The digital makeup artist system according to feature (1), wherein the calculation circuit is configured to play a video by displaying the generated image frames at a predetermined playback speed, the video includes chapters, each chapter being a step in a makeup routine, and user input includes video control commands that control the playback of the video.

[0155] (3) The digital makeup artist system according to feature (2), wherein the video control command includes any of the following: slowing down the playback speed of the video, pausing video playback, restarting from the beginning of a video step, or skipping to the next step of the video.

[0156] (4) The digital makeup artist system according to feature (2) or (3), wherein, if the input from the user is a video control command to reduce the playback speed, the calculation circuit reduces the playback speed by a ratio to a predetermined playback speed, and stores in the database a display indicating that the current chapter being played at the time of the command input is complex.

[0157] (5) The digital makeup artist system according to feature (2) or (3), wherein the input from the user is a video control command to restart from the beginning of the video step, and the calculation circuit is configured to read a time point associated with the beginning of the current chapter being played at the time of the command input, and to play the video from the frame of the time point.

[0158] (6) The digital makeup artist system according to feature (2) or (3), wherein the input from the user is a video control command to restart from the beginning of the video step, and further includes a command to adjust the generation of image frames, and the calculation circuit is configured to read a time point associated with the beginning of the current chapter being played, and to generate adjusted image frames for playing the video from the frames at the time point.

[0159] (7) The digital makeup artist system according to feature (6), wherein the generation adjustment of the image frame includes changing the color characteristics of the makeup for the facial features in the face image, wherein the color characteristics are one or more of range, hue, and finish.

[0160] (8) The digital makeup artist system according to feature (6), wherein the user input includes a look type, and the image frame generation adjustment takes into account the user's look preferences related to the look type.

[0161] (9) The digital makeup artist system according to feature (6), wherein a makeup look created based on the generation and adjustment of the image frame is stored in the database as a preferred makeup look.

[0162] (10) The digital makeup artist system according to any one of features (1) to (9), wherein the calculation circuit is further configured to analyze the user's facial image via the machine learning system to determine one or more of the face shape, lip shape, eyelid shape, and hairstyle, and to analyze the user's facial image to determine one or more of the skin tone, eye color, hair color, and skin texture, and the image frame is generated based on a makeup routine stored in relation to the face shape, lip shape, eyelid shape, hairstyle, skin tone, eye color, hair color, and skin texture.

[0163] (11) The digital makeup artist system according to any one of features (1) to (10), wherein the two-way interaction between the digital makeup artist and the user includes speech input, the mobile device performs natural language processing on the speech input, and the digital makeup artist outputs a speech response.

[0164] (12) The digital makeup artist system according to feature (11), wherein the two-way interaction with the user includes the digital makeup artist asking questions in order to obtain information about the condition of the user's skin.

[0165] (13) The digital makeup artist system according to feature (11), wherein the two-way interaction with the user includes the digital makeup artist asking questions to obtain information about problem areas on the user's face.

[0166] (14) The digital makeup artist system according to feature (12), wherein the two-way interaction with the user involves the digital makeup artist receiving speech input indicating that the user's skin is dry, and the image frame is generated based on a makeup routine stored in relation to the instruction that the user's skin is dry.

[0167] (15) A digital makeup artist system according to any one of features (1) to (14), wherein the makeup routine information and cosmetics stored in the database system are for skincare, and the image frame is generated based on the makeup routine information and cosmetics for skincare.

[0168] (16) A digital makeup artist system according to any one of features (1) to (15), wherein the makeup routine information and cosmetics stored in the database system are for skincare-type makeup, and the image frame is generated based on the makeup routine information and cosmetics for skincare-type makeup.

[0169] (17) The digital makeup artist system according to feature (16), wherein the skincare type makeup includes cosmetics having anti-aging properties, and the image frame is generated based on a makeup routine stored in relation to the application of the cosmetics having anti-aging properties.

[0170] (18) A digital makeup artist system according to any one of features (1) to (17), wherein the image frame is generated based on the analyzed facial image of the user, including the skin tone, in order to provide a makeup routine that uses a tone best suited to the skin tone.

[0171] (19) A digital makeup artist system according to any of features (1) to (18), further comprising a skin undertone machine learning model for selecting a cosmetic shade suitable for a specific skin undertone of the facial image.

[0172] (20) The digital makeup artist system according to feature (19), wherein the cosmetic is a foundation, and the skin undertone machine learning model is for selecting the foundation suitable for the specific skin undertone of the face image.

[0173] (21) Digital Makeup Artist System. The digital makeup artist system includes a mobile device having a display device, a processing circuit, and memory; a database system that stores makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's look preferences; and a machine learning system for analyzing facial images, wherein the mobile device includes a user interface for interacting with the digital makeup artist, and the digital makeup artist engages in two-way interaction with the user to provide advice, including obtaining initial information such as the type of makeup look, indoor or outdoor looks, skin condition, facial problem areas, and favorite facial features, and requesting a makeup consultation. The calculation circuit is configured to input the user's facial image, analyze the user's facial image via the machine learning system to identify facial features, analyze the facial image to determine facial features including one or more of skin tone, eye color, lip color, hair color, and skin texture, and generate an image frame to be displayed on the display device in synchronization with the interaction with the digital makeup artist to provide the advice, wherein the image frame is generated in synchronization with the interaction based on one or more of the user's analyzed facial image, the initial information obtained through the interaction with the user, the stored makeup routine information, general makeup looks, cosmetics according to skin type and ethnicity, and the user's preferences for looks.

[0174] (22) The digital makeup artist system according to feature (21), wherein the calculation circuit is configured to conduct the makeup consultation through the bidirectional dialogue between the user and the digital makeup artist, and the calculation circuit performs the dialogue to create custom recommendations for a makeup routine tailored to the user's favorite facial features.

[0175] (23) The digital makeup artist system according to feature (21) or (22), wherein the calculation circuit is configured to conduct the makeup consultation through the bidirectional dialogue between the user and the digital makeup artist, the dialogue includes prompting the digital makeup artist to input at least one problematic facial area of ​​concern to the user, and the calculation circuit conducts the dialogue to create custom recommendations for a makeup routine tailored to the problematic facial area of ​​concern.

[0176] (24) The digital makeup artist system according to any one of features (21) to (23), wherein the calculation circuit is configured to perform the bidirectional dialogue to create a custom recommendation for a makeup routine, receive further input including a request for adjustment of the recommended makeup routine, and perform the adjustment based on the request for adjustment of the recommended makeup routine, the stored makeup looks and the user's look preferences to create a refined makeup routine.

[0177] (25) The digital makeup artist system according to feature (24), wherein the calculation circuit is configured to store the adjustments in the database as user looks preferences included in the user's looks preferences, together with the refined makeup routine.

[0178] (26) The digital makeup artist system according to feature (24), wherein the calculation circuit is configured to output recommended cosmetics and skincare products for the sophisticated makeup routine.

[0179] (27) The digital makeup artist system according to feature (24), wherein the adjustment request for the recommended makeup routine includes a request to change the makeup characteristics of the image of the entire face.

[0180] (28) The digital makeup artist system according to feature (27), wherein the request for modification of the makeup characteristics includes a modification of one or more of the range, tone, and finish of each makeup color in the face image.

[0181] (29) The digital makeup artist system according to feature (24), wherein the adjustment request for the recommended makeup routine includes a request to change the makeup characteristics of a face portion in the face image, and the request to change the makeup characteristics includes changing one or more of the range of makeup colors, tone, and finish of the face portion in the face image.

[0182] (30) The digital makeup artist system according to feature (22), wherein the two-way interaction with the user includes the digital makeup artist receiving speech input indicating that the user's skin is dry, and the image frame is generated based on a makeup routine stored in relation to the instruction that the user's skin is dry.

[0183] (31) A digital makeup artist system according to any of features (21) to (30), wherein the makeup routine information and cosmetics stored in the database system are for skincare, and the image frame is generated based on the makeup routine information and cosmetics for skincare.

[0184] (32) A digital makeup artist system according to any one of features (21) to (31), wherein the makeup routine information and cosmetics stored in the database system are for skincare-type makeup, and the image frame is generated based on the makeup information and cosmetics for skincare-type makeup.

[0185] (33) The digital makeup artist system according to feature (32), wherein the skincare type makeup includes cosmetics having anti-aging properties, and the image frame is generated based on a makeup routine stored in relation to the application of the cosmetics having anti-aging properties.

[0186] (34) A digital makeup artist system according to any of features (21) to (33), wherein the calculation circuit performs the interaction to create custom recommendations for a makeup routine tailored to the user's skin tone, and the image frames of the makeup routine are generated based on the user's analyzed facial image to create the custom recommendations that use the optimal shades of cosmetics for the skin tone.

[0187] (35) The digital makeup artist system according to feature (34), further comprising a skin undertone machine learning model for selecting a shade of the cosmetic suitable for a specific skin undertone of the facial image.

[0188] (36) The digital makeup artist system according to feature (35), wherein the cosmetic is a foundation, and the skin undertone machine learning model is for selecting the shade of the foundation that is suitable for the specific skin undertone of the face image.

[0189] (37) A digital makeup artist system according to any of features (21) to (37), wherein the calculation circuit performs the interaction to create custom recommendations for a makeup routine tailored to the shape and color of the user's lips, and the image frames of the makeup routine are generated based on the analyzed facial image of the user to create the custom recommendations to use the best shades and / or finishes of cosmetics for the user's face that are best suited to the shape and color of the user's lips.

[0190] (38) A digital makeup artist system according to any of features (21) to (37), wherein the calculation circuit performs the interaction which includes obtaining the user's skin type and ethnicity to create custom recommendations for a makeup routine, and the image frames of the makeup routine are generated based on the user's skin type and ethnicity and the user's preferences for looks in order to create the custom recommendations which include using cosmetics stored in a database specified for the skin type and ethnicity.

[0191] (39) The digital makeup artist system according to feature (38), wherein the makeup routine information and cosmetics stored in the database system are for skincare-type makeup according to skin type and ethnicity, and the image frame is generated based on the makeup routine information and cosmetics for skincare-type makeup according to skin type and ethnicity.

[0192] (40) The digital makeup artist system according to feature (38), further comprising an ethnic machine learning model for selecting a shade of the cosmetic suitable for a specific ethnicity of the user's facial image. [Explanation of Symbols]

[0193] 101 Smartphones 103 Laptop Computer 103a Microphone 105 Cloud Services 107 Databases 109 Machine Learning Services 111 Mobile Applications 200 Mobile Processing Units (MPUs) 201 Subscriber Identification Module (SIM) 202 memory 206 Network Controller 208 Display screen controller 210 Display screen 212 I / O interfaces 214 buttons 220 Power Management and Touchscreen Controllers 221 Touchscreen 222 Communications Bus 224 Network 225 Controller 226 Arithmetic circuit 230 Camera Controller 231 Camera 240 Microphone Circuit 241 Microphone 242 Audio Circuits 301 Sub-screen 303 Menu Icon 305 Makeup Artist 307 Input Box 309 Text 310 User Interface Window 311 Microphone icon 320 video components 401 Videos 501 The First Dialogue 510 System 511 Recommended User Interface 520 Digital Makeup Artists 530 User Input 705 Lip shape 707 Eyelid shape 709 Hairstyle 711 colors 713 Skin texture 715 Lighting 717 Looks preferences 803 Components 803a layer 803b Rectifier Linear Unit Layer 803c pooling layer 805 Fully Connected Layer 807 Loss layer 809 Class 901 Input facial image 903 Stage 1 905 Stage 2 907 Output Layer 911 Connecting Block 913 Prediction Block 1000 Recommender System 1005 Database 1007 Recommended Engine 1201 Custom Recommended 1203 Scroll bar 1301 Message 1401 Face Images 1403 Makeup Palette 1403a Makeup 1403b color 1405 Parts 1901 Audio Components 1903 Face image 1905 Mask Filter 1907 Location information 1911 Pre-frame 2007 Feature Mask Filter 2011 Reverse Price 2015 Image 2357 steps

Claims

1. A mobile device having a display device, an arithmetic circuit, and memory, A database system that stores makeup routine information, general makeup looks, cosmetics tailored to skin type and ethnicity, and the user's preferences for appearance. A machine learning system for analyzing facial images, the machine learning system comprising a convolutional neural network architecture configured to classify the shape of the face and detect facial landmarks of the face, The mobile device includes a user interface for interacting with a digital video makeup artist, and the digital video makeup artist engages in two-way interaction with the user to understand the user's needs, including one or more of the following: type of makeup look, indoor or outdoor look, skin condition, facial problem areas, and favorite facial features. The aforementioned arithmetic circuit, Enter the user's face image, The machine learning system analyzes the user's facial image to identify facial features, the shape of the face, and facial landmarks of the face. The aforementioned facial image is analyzed to determine facial features including one or more of the following: skin tone, eye color, hair color, lip color, and skin texture. In sync with the aforementioned two-way interaction, image frames for the video to be displayed on the display device are generated. It is configured in such a way, The image frame is generated based on the analyzed user's facial image, the user's needs obtained through the two-way dialogue, and one or more of the stored makeup routine information, general makeup looks, cosmetics appropriate for skin type and ethnicity, and the user's preferences for appearance. The user input includes commands to adjust the generation of image frames in order to adjust the makeup routine, and the arithmetic circuit is configured to generate the adjusted image frames and play a video containing the adjusted image frames. Digital makeup artist system.

2. The calculation circuit is configured to play a video by displaying the generated image frames at a predetermined playback speed. The aforementioned video includes chapters, and each chapter is a step in a makeup routine. The digital makeup artist system according to claim 1, wherein the user input includes video control commands for controlling the playback of the video.

3. The digital makeup artist system according to claim 2, wherein the video control command includes one of the following: slowing down the playback speed of the video, pausing video playback, restarting from the beginning of a video step, or skipping to the next step of the video.

4. The digital makeup artist system according to claim 3, wherein, if the input from the user is a video control command to reduce the playback speed, the calculation circuit reduces the playback speed by a ratio to a predetermined playback speed, and stores in the database a display indicating that the current chapter being played at the time of the command input is complex.

5. The digital makeup artist system according to claim 3, wherein the input from the user is a video control command to restart from the beginning of the video step, and the calculation circuit is configured to read a time point associated with the beginning of the current chapter being played at the time of the command input, and to play the video from the frame of the time point.

6. The digital makeup artist system according to claim 3, wherein the input from the user is a video control command to restart from the beginning of the video step, and further includes a command to adjust the generation of image frames, and the calculation circuit is configured to read a time point associated with the beginning of the current chapter being played, and to generate adjusted image frames for playing the video from the frames at the time point.

7. The generation adjustment of the image frame includes changing the color characteristics of the makeup for the face portion in the face image, The digital makeup artist system according to claim 6, wherein the characteristic of the color is one or more of range, hue, and finish.

8. The user input includes the type of looks, The digital makeup artist system according to claim 6, wherein the generation and adjustment of the image frame takes into account the user's preference for the look related to the type of look.

9. The digital makeup artist system according to claim 6, wherein the makeup look created based on the generation and adjustment of the image frame is stored in the database as a preferred makeup look.

10. The calculation circuit is further configured to analyze the user's facial image via the machine learning system to determine one or more of the lip shape, eyelid shape, and hairstyle, and to analyze the user's facial image to determine one or more of the skin tone, eye color, hair color, and skin texture. The image frame is generated based on a makeup routine stored in relation to the face shape, lip shape, eyelid shape, hairstyle, skin tone, eye color, hair color, and skin texture. The digital makeup artist system according to claim 1.

11. The two-way interaction between the user and the digital video makeup artist includes speech input. The digital makeup artist system according to claim 1, wherein the mobile device performs natural language processing on the speech input and the digital video makeup artist outputs a speech response.

12. The digital makeup artist system according to claim 11, wherein the two-way interaction with the user includes the digital video makeup artist uttering questions in order to obtain information regarding the condition of the user's skin.

13. The digital makeup artist system according to claim 11, wherein the two-way interaction with the user includes the digital video makeup artist asking questions to obtain information about the problem area on the user's face.

14. The two-way interaction with the user occurs when the digital video makeup artist receives speech input indicating that the user's skin is dry. The digital makeup artist system according to claim 12, wherein the image frame is generated based on a makeup routine stored in relation to an instruction that the user's skin is dry.

15. The makeup routine information and cosmetics stored in the database system are for skincare, The digital makeup artist system according to claim 1, wherein the image frame is generated based on the makeup routine information and cosmetics for skincare.

16. The makeup routine information and cosmetics stored in the database system are for skincare-type makeup. The digital makeup artist system according to claim 1, wherein the image frame is generated based on the makeup routine information and cosmetics for skincare-type makeup.

17. The aforementioned skincare-type makeup includes cosmetics having anti-aging properties. The digital makeup artist system according to claim 16, wherein the image frame is generated based on a makeup routine stored in relation to the application of the cosmetic having anti-aging properties.

18. The digital makeup artist system according to claim 1, wherein the image frame is generated based on the analyzed facial image of the user, including the skin tone, in order to provide a makeup routine that uses the optimal tone for the skin tone.

19. The digital makeup artist system according to claim 1, further comprising a skin undertone machine learning model for selecting a cosmetic shade suitable for a specific skin undertone of the facial image.

20. The aforementioned cosmetic is a foundation. The digital makeup artist system according to claim 19, wherein the skin undertone machine learning model is for selecting the foundation suitable for the specific skin undertone of the facial image.

21. A mobile device having a display device, an arithmetic circuit, and memory, A database system that stores makeup routine information, general makeup looks, cosmetics tailored to skin type and ethnicity, and the user's preferences for appearance. A machine learning system for analyzing facial images, the machine learning system comprising a convolutional neural network architecture configured to classify the shape of the face and detect facial landmarks of the face, The mobile device includes a user interface for interacting with a digital video makeup artist, and the digital video makeup artist engages in two-way interaction with the user to provide advice, including requesting a makeup consultation, by obtaining initial information including one or more of the following: type of makeup look, indoor or outdoor look, skin condition, facial problem areas, and favorite facial features. The aforementioned arithmetic circuit, Enter the user's face image, The machine learning system analyzes the user's facial image to identify facial features, the shape of the face, and facial landmarks of the face. The aforementioned facial image is analyzed to determine facial features including one or more of the following: skin tone, eye color, lip color, hair color, and skin texture. To provide the aforementioned advice, the system generates image frames for a video to be displayed on the display device in synchronization with the two-way dialogue. It is configured in such a way, The image frame is generated in sync with the two-way dialogue based on one or more of the user's analyzed facial image, the initial information obtained through the two-way dialogue, the stored makeup routine information, general makeup looks, cosmetics appropriate for skin type and ethnicity, and the user's preferences for appearance. The user input includes commands to adjust the generation of image frames in order to adjust the makeup routine, and the arithmetic circuit is configured to generate the adjusted image frames and play a video containing the adjusted image frames. Digital makeup artist system.

22. The calculation circuit is configured to provide makeup consultation through the two-way dialogue between the user and the digital video makeup artist. The digital makeup artist system according to claim 21, wherein the calculation circuit performs the bidirectional interaction that creates custom recommendations for a makeup routine tailored to the favorite facial features.

23. The calculation circuit is configured to conduct the makeup consultation through the two-way dialogue between the user and the digital video makeup artist. The aforementioned two-way interaction includes prompting the digital video makeup artist to input at least one problematic facial area of ​​concern to the user, The digital makeup artist system according to claim 21, wherein the calculation circuit performs the bidirectional dialogue that creates custom recommendations for a makeup routine tailored to the problem area of ​​the face of concern.

24. The aforementioned arithmetic circuit, The aforementioned interactive dialogue is performed to create custom recommendations for a makeup routine. We have received further input, including requests for adjustments to the recommended makeup routine. Based on the adjustment requests for the recommended makeup routine, the stored makeup looks, and the user's preferred look, the adjustments are made to create a refined makeup routine. A digital makeup artist system according to claim 21, configured as described above.

25. The digital makeup artist system according to claim 24, wherein the calculation circuit is configured to store the adjustments in the database as user looks preferences included in the user's looks preferences, along with the refined makeup routine.

26. The digital makeup artist system according to claim 24, wherein the calculation circuit is configured to output recommended cosmetics and skincare products for the sophisticated makeup routine.

27. The digital makeup artist system according to claim 24, wherein the adjustment request for the recommended makeup routine includes a request to change the makeup characteristics of the image of the entire face.

28. The digital makeup artist system according to claim 27, wherein the request to change the makeup characteristics includes changing one or more of the color range, hue, and finish of each makeup in the face image.

29. The adjustment request for the recommended makeup routine includes a request to change the makeup characteristics of the face portion in the face image. The digital makeup artist system according to claim 24, wherein the request for modification of the makeup characteristics includes modifying one or more of the range of colors, tone, and finish of the makeup on the face portion in the face image.

30. The two-way interaction with the user includes the digital video makeup artist receiving speech input indicating that the user's skin is dry. The digital makeup artist system according to claim 22, wherein the image frame is generated based on a makeup routine stored in relation to an instruction that the user's skin is dry.

31. The makeup routine information and cosmetics stored in the database system are for skincare, The digital makeup artist system according to claim 21, wherein the image frame is generated based on the makeup routine information and cosmetics for skincare.

32. The makeup routine information and cosmetics stored in the database system are for skincare-type makeup. The digital makeup artist system according to claim 21, wherein the image frame is generated based on the makeup routine information and cosmetics for skincare-type makeup.

33. The aforementioned skincare-type makeup includes cosmetics having anti-aging properties. The digital makeup artist system according to claim 32, wherein the image frame is generated based on a makeup routine stored in relation to the application of the cosmetic having anti-aging properties.

34. The calculation circuit performs the bidirectional dialogue to create custom recommendations for a makeup routine tailored to the user's skin tone. The digital makeup artist system according to claim 21, wherein the image frame of the makeup routine is generated based on the analyzed facial image of the user to create the custom recommendation that uses the optimal shade of cosmetics for the skin tone.

35. The digital makeup artist system according to claim 34, further comprising a skin undertone machine learning model for selecting a shade of the cosmetic suitable for a specific skin undertone of the facial image.

36. The aforementioned cosmetic is a foundation. The digital makeup artist system according to claim 35, wherein the skin undertone machine learning model is for selecting the shade of the foundation that is suitable for the specific skin undertone of the face image.

37. The calculation circuit performs the bidirectional interaction that creates custom recommendations for a makeup routine tailored to the user's lip shape and lip color. The digital makeup artist system according to claim 21, wherein the image frames of the makeup routine are generated based on the analyzed facial image of the user to create the custom recommendation to use the optimal shade and / or finish of cosmetics for the user's face that is best suited to the shape and color of the user's lips.

38. The calculation circuit performs the two-way interaction that creates custom recommendations for a makeup routine, including obtaining the user's skin type and ethnicity. The digital makeup artist system according to claim 21, wherein the image frames of the makeup routine are generated based on the user's skin type and ethnicity and the user's preferences for looks in order to create the custom recommendations using cosmetics stored in a database specified for the skin type and ethnicity.

39. The makeup routine information and cosmetics stored in the database system are for skincare-type makeup, tailored to skin type and ethnicity. The digital makeup artist system according to claim 38, wherein the image frame is generated based on the makeup routine information and cosmetics for skincare-type makeup, according to skin type and ethnicity.

40. The digital makeup artist system according to claim 38, further comprising an ethnicity machine learning model for selecting a shade of the cosmetic suitable for a specific ethnicity of the user's facial image.

Images