Digital Makeup Palette

Abstract

The augmented reality system for makeup includes a makeup goal unit including a computational circuit operably coupled to a graphical user interface configured to generate one or more instances of user-selectable makeup goals and receive user-selected makeup goal information; a makeup palette unit operably coupled to the makeup goal unit, the makeup palette unit including a computational circuit configured to generate at least one digital makeup palette for digital makeup products according to the user-selected makeup goal information; and a makeup goal visualization unit including a computational circuit configured to generate one or more instances of virtual try-on according to the user-selected makeup goal information.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority to U.S. Non-Provisional Application No. 17 / 137,970, filed December 30, 2020, U.S. Non-Provisional Application No. 17 / 138,031, filed December 30, 2020, French Application No. 2,107,923, filed July 22, 2021, and French Application No. 2,107,904, filed July 22, 2021, and incorporates the entire contents of all such applications by reference. [Technical Field]

[0002] The present disclosure is directed to a digital makeup palette and a method for a personalized augmented reality experience using the digital makeup palette. [Background technology]

[0003] Smartphones with front-facing cameras provide the ability to take photos and videos of a person holding the camera in a manner that allows the person holding the camera to view the images being captured. A variety of mobile applications, also referred to as apps, have been developed that utilize front-facing cameras. Common apps are those that allow users to take self-portraits, also referred to as selfies, insert the selfies into social media contexts, and forward the selfies for sharing with others via email or text message.

[0004] Some cosmetic companies have begun developing apps to help users select cosmetics. The apps can provide tools to search for specific types of makeup, search for favorite products, or purchase previously used products. Some apps provide tutorials on how to apply specific types of makeup. Some apps help users select lipstick or eyeshadow colors by displaying color palettes. Some apps offer color matching features to help users find colors that match clothing and accessories or colors from photos.

[0005] Some cosmetic companies have begun to take advantage of smartphone, tablet, and laptop cameras by offering product try-on applications. Some of these applications are implemented as web applications, or apps. These try-on applications work by allowing users to take a self-portrait with their smartphone camera, upload the photo to the web application, and apply virtual makeup products to the uploaded image. These try-on applications can offer a variety of options, such as smoothing skin, lifting the sternum, and adjusting eye color. These try-on applications can provide users with the ability to add any type and color of makeup product, as well as change the color intensity.

[0006] However, such try-on applications tend to create a look using photo editing tools. Some previous try-on applications offer one-step functionality that starts with an uploaded photo, applies makeup types and colors, and then allows the user to edit the made-up photo. Such tools do not capture the personal makeup experience. Additionally, previous try-on application tools do not create custom looks. For example, a user may want a Friday date night look. Previous try-on applications may provide a Friday date night look, but that look may not be what the user had in mind. Further editing may be performed using the provided tools to attempt to achieve what the user believes to be their Friday date night look. However, such an approach is limited by the functionality of the editing tools. A user may want a Friday date night look based on the user's mood or the mood the user wants to express, which may require extensive editing.

[0007] These previous try-on web applications or apps lack sufficient personalization because previous virtual try-on applications were limited to physical makeup products from specific brands. Users may not be able to create the specific look they have in mind or want to try. It is necessary to provide a custom try-on experience for each specific user, where users can enjoy trying out makeup without being disappointed by unrealistic looks.

[0008] Additionally, a user may have problem areas that they would like to address with makeup, such as blemishes, scars, age spots, hyperpigmentation, etc. A user may also want to accentuate certain facial features, such as cheekbones, eyes, lips, etc. There is a need to provide a custom try-on experience that can help address a particular problem area or optimal facial feature for a particular user.

[0009] The alternative of distributing dedicated formula personalization machines is time consuming and expensive as it may require adding a sampling system to the machine to dispense small amounts of product, allow consumers to test the product, and then modify the product recipe according to the trends measured on the consumer's skin.

[0010] Furthermore, personalized try-on services for e-commerce cannot be extended to end consumers with smartphones because there is too much variability in each consumer's skin problem areas and facial features.

[0011] The foregoing "Background" discussion is intended to generally present the context for the present disclosure. To the extent described in this "Background" section, the work of the currently named inventors, and aspects of the description that may not otherwise be considered prior art at the time of filing, are not admitted expressly or impliedly to be prior art to the present invention. Summary of the Invention

[0012] The augmented reality system for makeup includes: a makeup goal unit including a computational circuit operably coupled to a graphical user interface configured to generate one or more instances of user-selectable makeup goals and receive user-selected makeup goal information; a makeup palette unit operably coupled to the makeup goal unit, the makeup palette unit including a computational circuit configured to generate at least one digital makeup palette for digital makeup products according to the user-selected makeup goal information; and a makeup goal visualization unit including a computational circuit configured to generate one or more instances of virtual try-on according to the user-selected makeup goal information.

[0013] The augmented reality system for makeup includes: a makeup goal unit including a computational circuit operably coupled to a graphical user interface configured to generate one or more instances of user-selectable makeup goals and receive user-selected makeup goal information; a makeup palette unit operably coupled to the makeup goal unit, the makeup palette unit including a computational circuit configured to generate at least one digital makeup palette for digital makeup products; and a makeup goal visualization unit including a computational circuit configured to analyze the user's face to determine one or more of facial shape, facial landmarks, skin tone, hair color, eye color, lip shape, eyelid shape, hairstyle, and lighting, and to automatically create one or more instances of a custom virtual try-on for the user according to the user-selected makeup goal information and the at least one digital makeup palette generated based on the analysis of the user's face.

[0014] The foregoing general description and the following detailed description of the exemplary embodiments are merely exemplary aspects of the teachings of the present disclosure and are not limiting. [Brief explanation of the drawings]

[0015] A more complete understanding of the present disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, in which:

[0016] [Figure 1] FIG. 1 is a diagram of a system according to an exemplary aspect of the present disclosure. [Figure 2] FIG. 1 is a block diagram of a computer system for a mobile device. [Figure 3A-3B]3A and 3B are flowcharts of methods for creating a custom look according to an exemplary embodiment of the present disclosure, where FIG. 3A illustrates a method in which a user creates their own custom look, and FIG. 3B illustrates a method in which a custom look is created by a mobile application. [Figure 4] 10 is an exemplary user interface for selecting whether a look is user-created or app-created, according to an exemplary aspect of the present disclosure. [Figure 5] 1 is a flowchart of a method for obtaining a digital makeup palette according to an exemplary aspect of the present disclosure. [Figure 6] 1 illustrates an exemplary digital makeup palette according to an exemplary aspect of the present disclosure. [Figure 7] 1 illustrates an exemplary digital makeup according to an exemplary aspect of the present disclosure. [Figure 8] 10 is a flowchart illustrating the face analysis step in more detail, according to an exemplary aspect of the present disclosure. [Figure 9] FIG. 1 is a block diagram of a CNN for facial shape classification. [Figure 10] FIG. 1 is a diagram of a deep learning neural network for facial landmark detection. [Figure 11] 10 is an exemplary user interface for selecting a virtual product to apply. [Figure 12] 1 is an exemplary user interface for a user to select whether to apply makeup or receive recommendations on how to apply makeup. [Figure 13] 1 is an exemplary mobile application according to an exemplary aspect of the present disclosure. [Figure 14] FIG. 1 is a diagram of a recommender system. [Figure 15] 13 illustrates an exemplary look-makeup matrix for the recommender system of FIG. 12. [Figure 16]1 illustrates a blending process that can be used to create a facial image based on desired and original features. [Figure 17] 1 is a flowchart of steps for applying virtual makeup according to an exemplary embodiment of the present disclosure. [Figure 18] 10 is a flow chart of steps for recording areas and swipes while applying makeup. [Figure 19] 1 is a flowchart of steps for analyzing a user's makeup application steps to estimate problem areas or best feature areas. [Figure 20] 1 is an exemplary user interface for storing makeup looks according to an exemplary aspect of the present disclosure. [Figure 21] 1 is a flowchart of a method for custom application of a digital palette according to an exemplary aspect of the present disclosure. [Figure 22] 1 is an exemplary user interface showing the status of a custom makeup application. [Figure 23] 1 is a flowchart of a method for selecting a makeup filter according to an exemplary aspect of the present disclosure. [Figure 24] 1 is an exemplary user interface for saving a makeup look. [Figure 25] FIG. 1 is a block diagram of a reinforcement learning architecture. [Figure 26] FIG. 1 is a flow diagram of a machine learning model according to an exemplary aspect of the present disclosure.

[0017] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, like symbols generally identify like elements unless context dictates otherwise. The exemplary embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented. DETAILED DESCRIPTION OF THE INVENTION

[0018] Aspects of the present disclosure are directed to a digital makeup palette in an augmented reality setup. The digital makeup palette is a color assortment for digital makeup for a single part of the face or the entire face. The augmented reality setup can capture the steps a user takes as they apply makeup and can create custom makeup filters for the applied makeup. The augmented reality setup can analyze the steps to identify what the user considers to be problem areas and best features. The results of the analysis can be used to improve custom recommendations. The augmented reality setup can perform the analysis using a machine learning model. The machine learning model can include an artificial neural network that estimates the problem areas and best features.

[0019] FIG. 1 is a diagram of a system according to an exemplary aspect of the present disclosure. Embodiments include a software application or a mobile application (app). For purposes of this disclosure, the terms app and makeup application will be used interchangeably hereinafter, and makeup application will be used in connection with the process of applying digital makeup, either virtually or physically. The software application may run on a desktop computer or laptop 103. The mobile application may run on a tablet computer or other mobile device 101. For purposes of this disclosure, the software application and mobile application will be described with reference to a mobile application 111. In each case, the mobile application 111 may be downloaded and installed on the respective device 101, 103. In some embodiments, the desktop computer or laptop 103 may be configured with a microphone 103a as an audio input device. The microphone 103a may be a device that connects to the desktop computer or laptop 103 via a USB port, an audio input port, or wirelessly via a Bluetooth wireless protocol. The mobile device 101 may be a mobile phone or smartphone with a built-in microphone. In some embodiments, the software or mobile application 111 may include communication capabilities that operate in conjunction with cloud services 105. Cloud services 105 may include database management services 107 and machine learning services 109. Database management services 107 may be any type of database management system offered by cloud services 105. For example, database management services 107 may include databases accessed using Structured Query Language (SQL) and unstructured databases accessed by key, commonly referred to as No SQL.A machine learning service 109 may perform machine learning to enable the scale-up and high-performance computing that may be required for machine learning. Additionally, software or mobile applications 111 may be downloaded from a cloud service 105. While Figure 1 shows a single cloud service 105, laptop 103, and mobile device 101, it should be understood that numerous mobile devices, laptops, and desktop and tablet computers may be connected to one or more cloud services.

[0020] The software or mobile application 111 may be implemented as an augmented reality system including a makeup goal unit operably coupled to a graphical user interface, a makeup palette unit coupled to the makeup goal unit, and a makeup goal visualization unit. The makeup goal unit may be configured to generate one or more instances of user-selectable makeup goals and receive user-selected makeup goal information. The makeup palette unit may be configured to generate at least one digital makeup palette for digital makeup products according to the user-selected makeup goal information. The makeup goal visualization unit may be configured to generate one or more instances of virtual try-on according to the user-selected makeup goal information. Each of the makeup goal unit, makeup palette unit, and makeup goal visualization unit may include computational circuitry of a computer system ranging from a mobile computing device 101, 103 to a desktop computing device. A minimum requirement is that the computing device include an interactive display device.

[0021] FIG. 2 is a block diagram of a mobile computing device. In one implementation, the functions and processes of the mobile device 101 may be implemented by one or more respective processing / computing circuits 226. The same or similar processing / computing circuits 226 may be included in a tablet computer or a laptop. Desktop computers may be similarly configured, but in some cases may not include an integrated touchscreen 221, microphone 241, or camera 231. A processing circuit includes a programmed processor, just as a processor includes computing circuits. A processing circuit may also include devices such as application-specific integrated circuits (ASICs) and conventional circuit components configured to perform the described functions. Note that circuitry refers to a circuit or a system of circuits. Here, computing circuits may be located within a single computer system or distributed across a network of computer systems.

[0022] A hardware description of the processing / computing circuitry 226 according to an exemplary embodiment will now be described with reference to Figure 2. In Figure 2, the processing / computing circuitry 226 includes a mobile processing unit (MPU) 200 that executes the processes described herein. Process data and instructions may be stored in memory 202. These processes and instructions may be stored on a portable storage medium or may be stored remotely. The processing / computing circuitry 226 may have a replaceable subscriber identity module (SIM) 201 that contains information specific to the network services of the mobile device 101.

[0023] Furthermore, the advancement is not limited to the form of computer-readable medium on which the instructions of the inventive process are stored. For example, the instructions may be stored in flash memory, secure digital random access memory (SDRAM), random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), solid-state hard disk, or any other information processing device with which the processing / computing circuitry 226 communicates, such as, for example, a server or computer.

[0024] Additionally, the advancements may be provided as utility applications, background daemons, and / or operating system components that run in combination with MPU 200 and a mobile operating system such as Android, Microsoft® Windows® 10 Mobile, Apple iOS®, and other systems known to those skilled in the art.

[0025] To realize the processing / computing circuitry 226, the hardware elements may be realized by various circuit elements known to those skilled in the art. For example, the MPU 200 may be a Qualcomm mobile processor, an Nvidia mobile processor, an Intel Atom processor, a Samsung mobile processor, or an Apple A7 mobile processor, or other processor types recognizable by those skilled in the art. Alternatively, as will be understood by those skilled in the art, the MPU 200 may be implemented on a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or using discrete logic circuits. Furthermore, the MPU 200 may be implemented as multiple processors cooperating in parallel to execute instructions of the inventive process described above.

[0026] The processing / computing circuitry 226 of FIG. 2 also includes a network controller 206, such as an “Intel Ethernet Pro” network interface card from Intel Corporation, USA, for interfacing with a network 224. As can be appreciated, the network 224 can be a public network, such as the Internet, or a private network, such as a LAN or WAN network, or any combination thereof, and can also include a PSTN or ISDN subnetwork. The network 224 can be wired, such as an Ethernet network. The processing circuitry may include various types of communications processors for wireless communications, including 3G, 4G, and 5G wireless modems, WiFi, Bluetooth, GPS, or other known forms of wireless communications.

[0027] The processing / computing circuitry 226 includes a universal serial bus (USB) controller 225 that may be managed by the MPU 200 .

[0028] The processing / computation circuitry 226 further includes a display controller 208, such as an NVIDIA® GeForce® GTX or Quadro® graphics adapter from NVIDIA Corporation, USA, for interfacing with the display 210. The I / O interface 212 interfaces with buttons 214 for volume control, etc. In addition to the I / O interface 212 and the display 210, the processing / computation circuitry 226 may further include a microphone 241 and one or more cameras 231. The microphone 241 may have associated circuitry 240 for processing audio into a digital signal. Similarly, the camera 231 may include a camera controller 230 for controlling the image capture operation of the camera 231. In an exemplary embodiment, the camera 231 may include a charge-coupled device (CCD). The processing / computation circuitry 226 may include an audio circuitry 242 for generating an audio output signal and may also include an optional audio output port.

[0029] The power management and touchscreen controller 220 manages the power used by the processing / computation circuitry 226 and the touch controls. The communication bus 222, which may be an Industry Standard Architecture (ISA), Extended Industry Standard Architecture (EISA), Video Electronics Standards Association (VESA), Peripheral Component Interface (PCI), or the like, is for interconnecting all components of the processing / computation circuitry 226. A description of the display 210, buttons 214, and the general features and functionality of the display controller 208, power management controller 220, network controller 206, and I / O interface 212 is omitted herein for the sake of brevity, as these features are well known.

[0030] 3A and 3B are flowcharts of a method for creating a custom look and special treatment of facial problem areas and best facial features. Fig. 3A is a flowchart of a method for creating a custom look by a user applying virtual makeup products with a digital palette, according to an exemplary embodiment of the present disclosure.

[0031] The disclosed embodiments include a digital makeup palette. The digital makeup palette is a virtual palette for digital makeup. In this disclosure, the terms virtual makeup and digital makeup may be used interchangeably. Digital makeup has an assortment of colors to choose from. A particular digital makeup may have an associated makeup application gesture and one or more facial features that are typically applied, and the digital makeup palette includes characteristics such as coverage, shade, and finish. Unlike physical makeup products, digital makeup is not limited to colors derived from chemical compositions and may include a wider range of colors. Digital makeup may also utilize coverage, shade, and finish generated using display device characteristics, such as the application of various filters for color temperature, exposure, contrast, and saturation, as well as control of RGB and HCl values.

[0032] Coverage is typically the actual coverage of the digital makeup based on the percentage of pigment contained in the digital makeup. Coverage typically pertains to foundation makeup, but can also refer to corrective makeup or primer. Light coverage makeup may contain less than about 18% pigment. Medium coverage products may contain about 18% to 23% pigment. Full coverage makeup may contain up to about 35% pigment. Some makeup products may contain a higher amount of pigment. In some embodiments, the coverage of the digital makeup is implemented as an opacity filter that represents a single brush stroke of the virtual makeup.

[0033] Digital makeup shades can range from light to dark, and in some cases from very light to deep or even very dark. A shade may represent a single color, such as skin color. In one or more embodiments, digital makeup shades are implemented as a range of displayed colors, for example, shades of red displayed according to RGB values.

[0034] Digital makeup finishes may include common finishes such as matte (dull), cream (glossy or shiny), frost (reflective), and glitter (glitter particles). Finishes may be defined by the amount of light reflected. Matte reflects very little light. Cream retains a pearlescent sheen. Frost and glitter reflect the most light. In one or more embodiments, digital makeup finishes are realized as a color luminance (brightness). Matte may have a low luminance value and hide imperfections. Frost may emit more luminance.

[0035] Digital makeup may include various filters such as blur, color temperature, and saturation. Blur may be applied to defective areas to make the defects less noticeable.

[0036] Prior to obtaining the makeup palette, in S301, the user may launch the app 111 on a mobile device, tablet, laptop, or desktop computer. The app 111 may ask the user, via a makeup objective unit, what type of look the user wants to create. To assist the user in answering this question, the app 111 may generate a list of predefined makeup looks, and the user may select a predefined makeup look. The predefined makeup looks may include seasonal looks (spring, summer, fall), event looks (Friday date night, girls' night out, special date, out with the in-laws, vacation, party, New Year's Eve, bridal, prom), looks based on time to completion (quick makeup, average makeup, slow makeup), mood looks (cheerful, happy, notice me), style (natural, evening, glamour, gothic, office), and aesthetic looks (VSCO, eGirl, soft girl), to name a few.

[0037] Additionally, app 111 may ask the user, via the makeup objective unit, to define their experience level in using makeup. User experience levels may include beginner / novice level, experienced level, expert level, and professional. The beginner / novice level may be a user with little or no experience applying makeup. The experienced level may be a user who has applied makeup before and therefore has some experience. The expert level may be a user who has been applying makeup for a period of time, such as for more than a year, and has taken steps to learn how to properly apply makeup. The professional level may be a user who applies makeup to other users. In some embodiments, app 111 may provide an interface that the user may use to create a user profile, which may include, among other things, inputting the user's experience level.

[0038] The app 111 may use the selected look and the user's experience level as a starting point. For example, a user who is new to applying makeup may want to experiment and learn about applying makeup. An experienced user who has previously applied makeup and wants to expand their knowledge and creativity may want to try a new makeup product or makeup look. An expert user may have extensive experience applying makeup but would like to expand their creativity and obtain a quality look similar to that created by a professional makeup artist. The app 111 may then use the selected look and the user's experience level when providing recommendations at a later stage.

[0039] In S303, the app 111 may provide the user with the option of having the app 111 provide a custom look or the user may apply virtual makeup to their facial image. In some embodiments, in S305, the makeup palette unit may generate at least one digital makeup palette. In particular, the user may obtain a digital makeup palette for a particular virtual makeup look, for example, by downloading digital makeup from the app 111 store or from a website that provides digital makeup. In one or more embodiments, the user may modify the digital makeup palette for variations in makeup looks. For example, the user may modify the digital makeup palette for a makeup look, such as a VSCO girl look, to be more or less dramatic. A less dramatic look may require obtaining a different digital makeup palette for the makeup look or may require obtaining different digital makeup for facial features, such as the lips, eyelids, and nose.

[0040] FIG. 4 is an example of a graphical user interface for app 111 that includes the ability to select how to apply makeup. In some embodiments, the user may obtain a digital makeup palette (S305) before deciding whether to have app 111 perform a custom look or whether the user will apply digital makeup (S303). Referring back to FIG. 4, user interface 400 may display products 401 obtained by the user, such as foundation 401a, eye shadow 401b, and concealer 401c. User interface 400 may provide the user with a selection of functions, such as creating a custom look (403) or creating a look by manually applying one or more virtual makeups (405) (see S303 in FIG. 3A).

[0041] In some embodiments, the app 111 may provide the user with a list of predefined looks, from which the user may select a predefined look as a starting point. Upon selecting a predefined look, the app 111 may provide the user with a set of recommended digital makeup and / or digital makeup palettes for the selected look. Based on the set of recommendations, the user may obtain the digital makeup and digital makeup palettes from the database 107 or from a makeup provider, for example, from the makeup provider's website.

[0042] 5 is a flowchart of a method for obtaining a digital makeup palette. Referring to FIG. 5, in S501, a user inputs a desired look via a makeup objective unit, and in S503, inputs a level of makeup experience. In S505, the user obtains a digital makeup palette via a makeup palette unit. The desired look (also referred to herein as a virtual try-on) may be selected from a list of predefined looks or may be input as a look name that reflects a predefined look. In some cases, the user may input a new look that does not have a predefined look or a look that is a modification of a predefined look.

[0043] The digital makeup palette may be a palette for creating a specific type of makeup look. The digital makeup palette may be purchased from a makeup company, similar to physical makeup products, or obtained from a website specializing in digital makeup products. FIG. 6 illustrates a user interface with a digital makeup palette according to an exemplary embodiment of the present disclosure. The user interface may include a digital makeup palette 601 for a specific makeup look 603 and a specific user experience level 605. The digital makeup palette 601 may include buttons for selecting specific digital makeup of a specific color, coverage, shade, and finish. The user experience level 605 may be controlled by a slider bar, ranging from general application to precision application. The user interface may include a button for selecting a makeup applicator tool 607.

[0044] A digital makeup palette contains one or more specific digital makeup looks, and like physical makeup, they have specific colors, coverages, shades, and finishes. Unlike physical makeup, coverage may be implemented as an opacity filter, shades as a range of RGB values, and finishes may be color intensity or color brightness.

[0045] The digital makeup palette may also be a general-purpose makeup palette. Furthermore, the digital makeup palette may be for a specific virtual makeup application on a portion of the face. FIG. 7 illustrates a user interface having one or more virtual makeup containers and one or more makeup applicator tool containers. The virtual makeup may be products 701 or products obtained from one or more makeup provider websites. The virtual makeup products may be stored in a makeup bag of a user 703. A user experience level 705 may be controlled by a slider bar, ranging from general application to precision application. The makeup applicator tools may be stored in a container 707. Various makeup applicator tools may be used to apply a specific virtual makeup product. Applicator tool types may include brushes, sponge makeup applicators, and makeup applicator puffs. Brushes may be of various widths and may have angled, flat, or pointed tips. Specialized brushes, such as mascara brushes, have bristles. A typical sponge applicator is a sponge swab with one or two tips. Some sponges are flat oval-shaped. Some sponges may be wedge-shaped. Puffs may come in a variety of sizes and materials. Some makeup products are in the form of makeup pencils, such as eyebrow pencils, eyeliner pencils, and lip liner pencils. Concealer and highlighter products may have a built-in pen-like dispenser.

[0046] Virtual makeup may include a stylus, a mouse, a physical applicator tool with built-in motion sensors, or even an applicator tool that may be configured to operate according to actual physical gestures using a user's finger. Physical gestures may be performed to cause a virtual brush to apply brush strokes according to the movement and force of the stylus. In some embodiments, the stylus may be used on a 3D touch screen of a mobile device where the amount of force on the touch screen generates lines of a commensurate thickness. In some embodiments, the stylus may take the form of a makeup applicator and include both motion and force sensors to detect the movement and force of brush strokes as a user virtually applies makeup to a facial image using the stylus.

[0047] If the user chooses to create their own custom look (does it themselves in S303), then in S307 the mobile application 111 running on the mobile device 101 or laptop 103 can use its built-in camera capabilities to capture an image of the user's face. In some embodiments, the camera 231 is used to capture video of the user. In some embodiments, the camera 231 is used to capture several images of the user's face from slightly different directions and / or in different lighting conditions. Alternatively, a previously captured image, images, or video may be uploaded to the mobile application 111. Additionally, the previously captured image, images, or video may be taken using an external camera device or retrieved from the mobile device's or laptop's internal or external storage device.

[0048] In some embodiments, once an image is captured, in S309, the mobile application 111 may perform facial recognition to identify features and their locations within the facial image, including lips, eyes, nose, ears, and hair. To perform facial recognition, the mobile application 111 may perform image processing operations to enhance image characteristics, such as improving lighting. For example, a user may inadvertently take a photo of themselves with bright light or sunlight shining from behind them. The mobile application 111 may brighten the user's facial image. Other image processing operations may be performed to improve image quality.

[0049] FIG. 8 is a more detailed flow chart of the face analysis step.

[0050] In S801, the captured image may be analyzed to determine a facial shape. In one or more embodiments, the facial shape of the captured user's face may be detected using a machine learning model. The machine learning model may be trained to classify facial shapes using facial images with known facial shapes. Recently, image classification has been performed using a type of neural network inspired by how the visual cortex of the human brain works to recognize objects. The neural network is a family of networks known as convolutional neural networks (CNNs). Other approaches to image classification have been proposed and continue to be improved. Other approaches that may be used for image classification include linear regression, decision trees, random forests, and support vector machines. In some embodiments, the machine learning model may be trained remotely using the machine learning service 109 of the cloud service 105.

[0051] In one or more embodiments, the architecture of a machine learning model that can be used to classify face shapes is a CNN. Figure 9 is a block diagram of a CNN for classifying face shapes. The dimensions and activation functions of the CNN can vary depending on the available processing power and the required accuracy. The dimensions include the number of channels, the number of neurons in each layer, and the number of layers. Possible activation functions include a logistic function, a rectified linear unit, etc.

[0052] A convolutional neural network (CNN) may be composed of several types of layers. The convolutional component 903 may consist of a convolutional layer 903a, a pooling layer 903c, and a normalized linear unit layer 903b. The convolutional layer 903a develops a two-dimensional activation map that detects the specific location of features at every given spatial location. The pooling layer 903c acts as a form of downsampling. The normalized linear unit layer 903b applies an activation function to increase the nonlinearity of the decision function and the overall network without affecting the receptive field of the convolutional layer itself. The fully connected layer 905 contains neurons with connections to all activations between previous layers. The loss layer specifies how the network training penalizes deviations between predicted and true values. The loss layer 907 detects classes within a set of mutually exclusive classes. One type of loss layer is a softmax function, which provides an output value for each of multiple classes.

[0053] In some embodiments, the loss layer 907 may be a softmax function, which provides a probability value for each class. In an exemplary embodiment, the classes 909 may include square, rectangle, circle, oval, ellipse, diamond, triangle, and heart-shaped shapes.

[0054] In S803, the mobile application 111 may further analyze facial features and landmarks. The captured user's facial features and landmarks, as well as the face shape, may be detected using a machine learning model. The machine learning model may be trained to detect facial landmarks. Using face shape classification, a CNN architecture similar to that of FIG. 9 may be used for facial landmark detection. Other approaches to classification may also be used.

[0055] FIG. 10 is a diagram of a deep learning neural network for facial landmark detection. Similar to the architecture of FIG. 9, the deep learning neural network is a convolutional neural network. Residual connections may be included to improve training of the deep learning neural network. In one or more embodiments, an inverted residual structure may be included, where the residual connections are made to earlier layers in the network. With respect to the architecture of FIG. 10, the network is provided as two stages, 1003 and 1005. The first stage 1003 is a convolutional stage for feature extraction. The second stage 1005 performs prediction within the region of interest.

[0056] The architecture of the first stage 1003 includes a convolution section 1003a that performs convolution and max-pooling operations when provided with an input face image 1001. The convolution section 1003a is connected to an inverted residual structure 1003b. A mask layer 1003c is connected to the inverted residual structure 1003b. The size of the mask layer 1003c is based on the number of landmarks (e.g., 2×L, the number of landmarks). The mask layer 1003c encodes the spatial layout of the input object.

[0057] The architecture of the second stage 1005 includes an inverted residual structure 1005b connected to the inverted residual structure 1003b of the first stage 1003. A mask layer 1003c of the first stage 1003 is also applied to the result of the inverted residual structure 1005b, providing it as input to perform region of interest cropping in an ROI and concatenation block 1011. The ROI and concatenation block 1011 is based on the number of channels and the number of landmarks in the inverted residual structure 1005b. A prediction block 1013 predicts landmarks and their approximate locations in the mask layer 1005c. The region of interest prediction of the second stage 1003 is combined with the landmarks estimated by the mask 1003c for the entire image to obtain output landmarks in the output layer 1007.

[0058] In one or more embodiments, the facial landmarks include the eyes, nose, lips, cheekbones, the area around the eyes including eyebrows, eyelids, and hair. In some embodiments, the landmarks may include possible facial abnormalities.

[0059] The specific dimensions of each layer and the number of layers may depend on parameters such as the desired accuracy, the hardware on which the machine learning model runs, the training time of the machine learning model, etc. The machine learning model may be trained using the machine learning service 109 of the cloud service 105.

[0060] The facial feature analysis S803 may further include lip shape detection S805, eyelid shape detection S807, and hairstyle detection S809. The detected landmarks can be used to calculate lip, eye, and hairstyle contours. Furthermore, other facial features such as skin color S811 and skin texture S813 may also be determined from the face image. Skin color and skin texture may be determined using image processing techniques. Skin tone types may include, but are not limited to, fair, light, medium, dark, etc. Skin texture types include, but are not limited to, soft, smooth, rough, and leathery.

[0061] An additional feature of the facial image may be lighting (image brightness). At S815, image processing techniques may be used to determine the image brightness (lightness). Brightness may be defined as a measure of the total amount of light perceived in an image. In some embodiments, the image brightness may increase or decrease from an initial brightness level when captured.

[0062] In S817, preferences for past looks may be retrieved from database 107. The preferences for past looks may include digital makeup characteristics such as color, coverage, shade, finish, and application gestures used for the past looks. Past user preferences may include digital makeup characteristics for specific parts of the face and may also include digital makeup selections applied to specific looks.

[0063] In S311, the user interface may include a function for selecting virtual makeup. Figure 11 is an exemplary user interface for selecting virtual makeup to apply. User interface screen 1100 may include message 1101 including instructions for selecting virtual makeup using pointer 1103.

[0064] In S313, once the virtual makeup is selected, the mobile application 111 can perform a function to activate the selected virtual makeup. The virtual makeup may be activated by obtaining characteristics of the virtual makeup, including a swipe gesture of the applicator and a representative area of the face to which the virtual makeup will be applied. In some embodiments, the data associated with the virtual makeup may include coverage, color, and finish.

[0065] In S315, the mobile application 111 may display a message asking the user whether they would like recommendations on how to apply virtual makeup. An example of a user interface for displaying a request for a recommendation message is shown in FIG.

[0066] 12 is an exemplary user interface for a user to select whether to apply makeup or have a mobile application recommend how to apply the makeup. User interface 1200 can display button 1203 for selecting a recommendation on how to apply virtual makeup 1205. User interface 1200 may alternatively display button 1201 instructing the user to swipe a stylus or mouse to apply virtual makeup to a facial image 1207.

[0067] If the user chooses not to obtain recommendations (NO in S315), in S317, the user may specify a location on the facial image where virtual makeup should be applied. In S321, the user may apply virtual makeup by swiping at a location on the face using a stylus or mouse via the makeup objective visualization unit. FIG. 13 is an exemplary user interface on the mobile device 101. When button 1201 is selected, the user interface may display a facial image 1301 and a digital makeup palette 1303. The user may select a color 1303b from the digital makeup palette 1303 and apply virtual makeup 1303a to a specific location 1305 using a swipe gesture with the stylus 1310. The screen on the mobile device 101 may be a touchscreen that includes a zoom function that can be used to zoom in or out on the facial image 1301 to adjust the display of facial features. In some embodiments, the touchscreen mode may be switched to allow virtual makeup to be applied to the facial image using a stylus without moving the image.

[0068] In another decision, if the user selects to get recommendations (YES in S315), in S319 the mobile application 111 indicates the location on the face image where the virtual makeup should be applied.

[0069] FIG. 14 is a diagram of a recommender system. The recommender system 1400 may be used to illustrate a method for applying virtual makeup (S319 in FIG. 3A). The recommender system 1400 operates based on an indexed database 1405 of image data and makeup filters. The recommender system 1400 includes a recommendation engine 1407 that searches and ranks recommendations. In the case of applying a particular virtual makeup, the recommendations may relate to the look and virtual makeup input by the user in step S301. In some embodiments, the recommendations may be obtained based on user preferences or favorites. Personal user preferences may be makeup characteristics input by the user when the app 111 is initially set up. Favorites may be makeup characteristics that the user flags as favorites. Personal preferences and favorites may relate to specific facial features or the entire face.

[0070] In some embodiments, the recommendation engine 1407 may use a look-feature matrix. FIG. 15 illustrates a non-limiting look-feature matrix according to an exemplary aspect of the present disclosure. The look-feature matrix in FIG. 15 is a partial matrix illustrating two types of virtual makeup for brevity. Other types of virtual makeup may be included in the matrix, including, but not limited to, foundation, mascara, concealer, blush powder, and eyebrow pencil, to name a few. The look-feature matrix may be stored in the mobile device app 111 and compared to a vector of desired features. The desired features may be current user preferences and may take into account the user's current experience level and desired look. The recommendation engine 1407 may rank the recommendations using one or more similarity metrics and scoring algorithms. In one embodiment, the recommendation engine 1407 may generate a set of features that enhance the recommendations to encourage creativity by changing certain characteristics of the virtual makeup from the recommended one. For example, if the recommendation engine 1407 ranks one recommendation higher from among multiple retrieved recommendations, it may modify one or more characteristics to increase the similarity score. Alternatively, the recommendation engine 1407 may modify one or more characteristics in the retrieved recommendation, such as color or finish, by one level up or down (e.g., modifying the color one level up or down from the stored color). In one or more embodiments, the recommendation engine 1407 can adjust application gestures to be more or less accurate based on the user's experience level.

[0071] The recommendation engine 1407 can output one or more recommendations to a recommendation user interface (S319), which can display a series of video frames illustrating the application of the selected recommendation.

[0072] In one or more embodiments, the recommendation video frames may be generated using a facial image of the user and one or more makeup filters stored in database 1405. Indexed database 1405 may provide the one or more makeup filters used to create the sequence of video frames. Figure 16 illustrates a blending process that may be used to create a facial image based on desired features and original features in the facial image.

[0073] The blending of facial features is done as follows: 1. The desired feature 1601 is recolored 1603 to match the color of the original feature, resulting in a recolored feature 1605. 2. The recolored feature portion 1605 is multiplied by the feature mask 1607. 3. The original feature portion 1609 is multiplied by the inverse 1611 of the feature mask (i.e., 1 minus each mask value ranging from 0 to 1). 4. The resulting images 2 and 3 are added pixel by pixel 1613 to create the final blended feature image 1615.

[0074] The boundaries of the original features may have been determined during the facial analysis step S309. A sequence of video frames may be generated as an animation showing how to apply virtual makeup to specific facial features.

[0075] In S321, the user may apply makeup mimicking the application shown in the virtual makeup by making one or more swipes at the facial location of the facial image using a stylus or mouse configured to draw as a particular type of applicator.

[0076] FIG. 17 is a flowchart of steps for applying virtual makeup according to an exemplary embodiment of the present disclosure. In S1701, a user may interact with a user interface to select or touch a starting point for applying virtual makeup. In S1703, a user may perform a gesture to apply virtual makeup. The gesture may be a swipe, a line-drawing, or a tapping motion. A swipe may be performed, for example, to apply mascara to eyelashes. A thicker applicator may be used with a swipe to apply broader strokes, such as eyeshadow. A line-drawing motion may be used, for example, to apply eyeliner. A line-drawing motion may be used with thicker lines to apply lipstick. A tapping motion may be used to apply face powder.

[0077] In one or more embodiments, in S1705, the gesture may be analyzed based on the user's experience level to determine whether the gesture was applied incorrectly, i.e., incorrectly. A novice user may be more tolerant of errors than an experienced user. In such a case, a gesture that exceeds a tolerance amount for an experienced user may be determined to be an error, while the tolerance amount may be greater for a novice user. If a gesture by a novice user exceeds a higher tolerance amount, the gesture may be determined to be an error. In S1707, the app 111 determines whether the gesture was applied incorrectly, i.e., incorrectly. If it is determined that the gesture was performed beyond the tolerance amount (YES in S1707), a notification message may be displayed in S1709 to inform the user that the gesture may have been applied incorrectly and / or to ask the user to confirm that the gesture was applied properly. The app may provide the user with an option to redo the application of the virtual makeup in S1711. If there is no mistake (NO in S1707), or if the user selects not to redo the virtual makeup (NO in S1711), the application 111 proceeds to the next step S323.

[0078] In some embodiments, the region and swipe motions may be constrained or controlled to remain within facial features. For example, when a stylus touches the screen, the mobile application 111 may detect its location as being within a facial feature. As the stylus moves, swipes may be drawn on the screen, but not outside the boundaries of the facial features, for example, as determined in the facial analysis step S309. The drawing on the screen may be consistent with characteristics such as the coverage, shade, and finish of the makeup product. The drawing on the screen may be performed according to common application gestures and facial regions.

[0079] In S323, the mobile application 111 can record the area and swipe movements as the user applies the virtual makeup in the memory 202 of the mobile device 101, 103. Figure 18 is a flowchart of the steps for recording the area and swipes while applying makeup. Referring to Figure 18, in S1801, the mobile application 111 can track and record each step and related data in memory, including the location on the face image where the virtual makeup is applied and the number of swipes.

[0080] In S325, the mobile application 111 analyzes the recorded positions and swipes of the virtual makeup and the features of the virtual makeup to estimate problem areas or best features of the user's face. The positions may be mapped to facial features. FIG. 19 is a flowchart of steps for analyzing a user's steps of applying makeup to estimate problem areas or best features. Referring to FIG. 19, in S1901, the mobile application 111 may analyze the makeup swipes to identify potential problem areas. Potential problem areas may include age spots, scars, age spots, hyperpigmentation, and the like. Potential problem areas may be areas of the face that the user considers problematic or undesirable features. In other words, potential problem areas may be areas that the user desires to conceal or change their appearance.

[0081] The mobile application 111 can identify potential problem areas by detecting anomalous swipe gestures at specific facial feature locations. An anomalous swipe gesture may include a sudden change in direction or a sudden change in force that is not made accidentally. The mobile application 111 can identify potential problem areas by detecting the user applying different virtual makeup or alternate colors from a digital makeup palette to specific facial areas (i.e., virtual makeup with different coverage characteristics and / or different shades).

[0082] In S1903, the mobile application 111 may analyze the makeup swipe to identify a best facial feature. The best facial feature may include cheekbones, eye color, eyelashes, lip shape, or any feature the user desires to emphasize. The mobile application 111 may detect a best facial feature by detecting a change in makeup application to a facial feature that differs by a threshold amount from the average makeup application to the same facial feature. For example, the mobile application 111 may detect a best facial feature by detecting the application of a color in a shade and / or finish that differs from the typical shade and / or finish of the color applied to the facial area. In the case of eye color, the mobile application 111 may detect that eye color is a best facial feature by detecting the application of a particular eyeshadow color.

[0083] In S1905, the mobile application 111 can compare the identified problem areas and best facial features to previously stored recommendations. The mobile application 111 can determine that some new problem areas may exist or that some problem areas are no longer potential problem areas. The mobile application 111 can increase the importance of problem areas that were previously considered potential problem areas.

[0084] In S1907, the results of the comparison may be used to adjust the recommendations so that the recommendation engine 1407 assigns a higher score to recommendations with verified problem areas. The recommendations may be adjusted if new problem areas and best facial feature portions, or problem areas and best facial feature portions that are no longer potential, may be used to support changing the recommendations to new or no longer potential ones.

[0085] A user may apply virtual makeup from a digital makeup palette in a manner that corrects problem areas or accentuates their best features. In one or more embodiments, problem areas may be corrected by applying a filter to blur imperfections in the problem areas. For example, blurring an area in a facial image that contains blemishes may make the blemishes less noticeable.

[0086] As mentioned above, potential problem areas can be areas of the face that the user considers problematic or undesirable features. Best facial features include cheekbones, eye color, eyelashes, lip shape, or any feature that the user wants to emphasize.

[0087] At S327, the mobile application 111 can store the verified problem areas, the verified best facial features, and the user makeup application in the database 1405 as future custom recommendations.

[0088] 3B, in S331, the user may select to repeat the steps of applying virtual makeup to another virtual makeup. After all desired virtual makeup has been applied, the user may select to save the created look in database 107 in S335 (YES in S333). In S337, the user may select to move / publish the created look in S339 to a social media platform or other platform with live video (YES in S337). In one or more embodiments, the look may be stored as a makeup filter that can be applied to another facial image.

[0089] FIG. 20 is an exemplary user interface for storing a makeup look. The user interface 2000 may display a completed facial image 2001 and provide a button 2003 for the ability to save the completed facial image. The completed facial image may be stored as an underlying facial image and one or more filters that can be applied to the underlying facial image to recreate the completed facial image. In some embodiments, the completed facial image may be stored as the underlying facial image and recorded swipes of makeup products. The user interface 2000 may further provide a button 2005 for the ability to move the completed facial image to a platform that provides live video or still images, such as a social media platform or a video conferencing platform. Social media platforms include Facebook, LinkedIn, Instagram, YouTube, Snapchat, and TikTok, to name a few. Video conferencing platforms include Microsoft Teams, FaceTime, Google Hangouts or Google Meet, and Zoom, to name a few.

[0090] In some embodiments, one or more makeup filters may be provided on the social media platform or video conferencing platform to recreate the finished facial image, and the one or more filters may be applied to another base image to obtain a new finished facial image.

[0091] In some embodiments, in S315, the user may transfer the digital makeup palette and captured facial image to another user. In such a case, S321 through S327 may occur while the other user is performing the makeup application. The other user may have a higher level of experience in applying makeup, or may be someone the original user believes can create the type of makeup look the original user would prefer.

[0092] FIG. 21 is a flowchart of a method for customizing a digital palette application. If a user selects to create a custom makeup look (S303, Custom), in S2101, the user may be prompted to capture an image, multiple images, or video of the user's face. The camera 231 of the mobile device 101 or an external camera may be used to capture an image or video of the user's face. In S2103, the mobile application 111 may analyze the captured user's face. FIG. 22 is an exemplary user interface for illustrating the creation of a custom makeup application. FIG. 8 is a more detailed flowchart of the face analysis steps, as described above. FIG. 9 is a block diagram of a CNN for classifying face shapes, as described above. The dimensions and activation functions of the CNN can vary depending on the available processing power and desired accuracy. Dimensions include channels, the size of each layer, and the number of layers. Activation functions include logistic functions, rectified linear units, etc. FIG. 10 is a diagram of a deep learning neural network for facial landmark detection, as described above.

[0093] In S2105, one or more makeup filters may be selected / retrieved from database 107 based on facial features identified by the facial analysis (S2103 and FIG. 8) and past look preferences. Some of the stored makeup face filters may be filters previously created by the user (when selecting "Do It Myself" in S303). Some of the makeup filters may be common looks.

[0094] FIG. 23 is a flowchart of a method for selecting makeup filters. In S2301, a face shape is obtained from the analysis result of S2103. In S2303, landmarks are obtained from the analysis result of S2103. In S2305, skin, hair, eye, facial color, and brightness characteristics are obtained from the analysis of S2103. In S2307, past look preferences for a digital makeup palette may be obtained. In S2309, possible face filters for the landmarks, face shape, skin color, hairstyle, eyelid shape, and past preferences are obtained from database 107. In S2311, a subset of the obtained face filters may be selected. Selection criteria may include random selection from among the possible face filters, selection of a face filter that best matches the past look preference, or selection of at least one face filter that differs from the past look preference but can provide the user with different creative look options to provide a custom look to the user.

[0095] In S2107, the obtained makeup filters may be overlaid on the facial image to obtain one or more custom looks. The overlay process may include aligning the makeup filters based on facial shape and facial landmarks. The blending process of FIG. 16 may be used to perform the overlay process by creating a facial image based on desired features and original features in the facial image.

[0096] In S2111, the user may select to save the look created by the mobile application 111 to the database 107 (YES in S2109). In S2113, the user may select (YES in S2113) and in S2115, select to move / publish the created makeup look to a social media platform or a video conferencing platform.

[0097] FIG. 24 is an exemplary user interface for storing makeup looks. User interface 2400 may display completed facial images 2401 and provide a button 2403 for saving each completed facial image. The completed facial images may be stored as an underlying facial image and one or more makeup filters that can be applied to the underlying facial image to recreate the completed facial image. In some embodiments, the completed facial images may be stored as recorded swipes of the underlying facial image and makeup products. User interface 2400 may further provide buttons (not shown) for transferring the completed facial image to a social media platform or a video conferencing platform. Social media platforms include Facebook, LinkedIn, Instagram, Snapchat, YouTube, and TikTok, to name a few. Examples of video conferencing platforms include Microsoft Teams, FaceTime, Google Hangouts or Google Meet, and Zoom.

[0098] Numerous modifications and variations of the present invention are possible in light of the above teachings. For example, data collected from various consumer skin tones and textures would allow scaling of the artificial neural network to one or more consumers. The artificial neural network would be able to predict the rendering of a new cosmetic formula for each product shade.

[0099] In one or more embodiments, a form of machine learning, such as reinforcement learning, may be used to learn what the user considers to be problem areas and which areas the user wants to emphasize as the best facial features.

[0100] 25 is a block diagram of types of reinforcement learning architectures. Of note, a variety of architectures and algorithms have been developed for reinforcement learning, such as deep reinforcement learning, Q-learning, and deep Q-networks, to name a few. It should be understood that this disclosure provides a general description of reinforcement learning and applies to a variety of approaches to reinforcement learning.

[0101] Unlike supervised learning, reinforcement learning is a form of machine learning that does not require prior knowledge of the output. Instead, an action output by an actor results in a reward indicating whether the action was appropriate. A reinforcement learning system may involve an actor directing movement actions within an environment, and choosing an action may result in a reward in the form of a score of a specific value. The movement action results in a new state of the environment. The score is fed back to the actor, and the machine learning component is adjusted. An example of a movement action may be an action in which an actor in the environment moves to a new location to perform a task, which increases the actor's score value. The increase in score confirms that the movement action was beneficial. The next movement action may be an action in which the actor in the environment moves in a way that does not reach the new location, resulting in a negative score, or at least an action that does not increase the score value. The decrease in score is fed back as a negative impact, and the machine learning component may be adjusted to learn that the movement action directed by the actor was not an appropriate choice given the state of the environment. Thus, reinforcement learning can continue to be employed as the actor continues to direct movement actions.

[0102] With reference to Figure 25, an agent 2510 interacts with its environment 2520 in discrete time steps via an artificial neural network 2513. At each time t, the agent 2510 typically receives an observation with an associated reward. The agent then selects an action from a set of available actions, which is then sent to the environment 2520. The environment 2520 transitions to a new state, and the reward associated with that transition is determined. The goal of the reinforcement learning agent 2510 is to collect as many rewards as possible. The agent 2510 can choose any action (possibly randomly) depending on the history of previous actions.

[0103] In one embodiment, a reinforcement learning system may be configured to learn what a user considers problem areas and which areas the user wants to emphasize as their best facial features, which may be provided as two reinforcement learning processes. FIG. 26 is a flow diagram of a machine learning system according to an exemplary aspect of the present disclosure. As described above, reinforcement learning generally learns through reward 2520a feedback. In some embodiments, feedback may be provided in the form of voice interaction with the mobile application 111 as the user applies makeup products to the facial image. Voice feedback may be provided using the microphones 103a, 241, and the voice feedback may be provided in response to questions and utterances output via the audio circuitry 242.

[0104] In one embodiment, the reinforcement learning system 2600 may take the form of multiple reinforcement learning models. One reinforcement learning model 2603 may detect a problem area based on one or a series of swipes 2601 of makeup product on the facial image. The reinforcement learning system 2600 may verify the detection of the problem area (i.e., provide a reward feedback) by asking a question such as, "Are you applying makeup to the problem area?" Another reinforcement learning model 2605 may detect a best facial feature based on one or a series of swipes 2601 of makeup product on the facial image. The reinforcement learning system 2600 may verify the detection of the best facial feature (i.e., provide a reward feedback) by asking a question such as, "Are you applying makeup to the best facial feature?" The reinforcement learning system may use the location of the problem area and the best facial feature information to provide more specific questions such as, "Are you applying makeup to blemishes?" or "Are you applying makeup to enhance your eye color?"

[0105] While reinforcement learning system 2600 in FIG. 26 is a series of machine learning models, an alternative approach may be to include a machine learning component that first classifies one or a series of swipes as the problem region, the best facial feature area, or neither, and provides the results of the initial classification to either reinforcement learning model 2603, reinforcement learning model 2605, or neither model.

[0106] The user's response may be used to apply a reward to the reinforcement learning system, which may be a positive or negative score depending on the user's response, which may be used to adjust the parameters of the respective machine learning model 2603 or 2605.

[0107] Another approach to detecting problem areas or best facial features using continuous learning similar to reinforcement learning is regression analysis. The advantage of regression analysis is that it is fast to calculate. However, nonlinear regression analysis models are more suitable for predictive data prediction. Makeup swipe data can be difficult to predict clearly because it may occur for reasons other than the problem area or best facial feature.

[0108] In one embodiment, the terms "a," "an," etc. generally have the meaning "one or more," unless otherwise indicated.

[0109] In one embodiment, the terms "about," "approximate," "approximately," and similar terms generally refer to a range that includes the specified value within a margin of 20%, 10%, or preferably 5%, and any value therebetween.

[0110] Consequently, it is to be understood that within the scope of the appended claims, the invention may be practiced other than as specifically described herein.

[0111] The above disclosure also encompasses the embodiments listed below.

[0112] (1) An augmented reality system for makeup. The augmented reality system includes: a makeup goal unit including a computational circuit operably coupled to a graphical user interface configured to generate one or more instances of user-selectable makeup goals and receive user-selected makeup goal information; a makeup palette unit operably coupled to the makeup goal unit, the makeup palette unit including a computational circuit configured to generate at least one digital makeup palette for digital makeup products according to the user-selected makeup goal information; and a makeup goal visualization unit including a computational circuit configured to generate one or more instances of virtual try-on according to the user-selected makeup goal information.

[0113] (2) The augmented reality system for makeup described in feature (1), wherein the computational circuitry of the makeup objective visualization unit is further configured to receive one or more digital images of a user including at least a portion of the user's face, analyze the user's facial images to identify facial features, track and record in memory at least one gesture by the user applying a digital makeup product to the user's facial image, analyze the at least one gesture to estimate problem areas on the user's face or estimate enhancement of certain facial features, and store the estimated problem areas or estimated enhanced facial features together with the applied coverage, color, and finish in memory.

[0114] (3) The augmented reality system for makeup described in feature (2), wherein the computational circuitry of the makeup goal visualization unit is further configured to generate a virtual representation of a portion of the user's face including at least one modification consistent with the makeup goal information selected by the user.

[0115] (4) The augmented reality system for makeup described in feature (2) or (3), further comprising a touchscreen, wherein the at least one gesture by the user includes one or more swipes on the touchscreen, and the computational circuitry of the makeup objective visualization unit is further configured to detect the one or more swipes and apply the selected color to a location within the image of the user's face.

[0116] (5) The augmented reality system for makeup described in feature (4), wherein the computational circuitry of the makeup purpose visualization unit is further configured to detect one or more swipes on the touchscreen and apply a selected color to an area of the image bounded by a boundary of a facial feature located at a position within the user's facial image.

[0117] (6) The augmented reality system for makeup described in feature (2) or (3), wherein the computational circuitry of the makeup objective visualization unit is further configured to receive a user's experience level in applying makeup, detect one or more swipes on the touchscreen, and apply a selected color to an area of the user's facial image at a location of a facial feature indicated by the swipe, the facial feature having a boundary, and analyze the applied color to determine whether the one or more swipes are beyond a tolerance amount from the boundary, the tolerance amount being based on the user's experience level in applying makeup.

[0118] (7) The augmented reality system for makeup described in feature (4) or (5), wherein the touchscreen is a three-dimensional touchscreen that senses the amount of pressure applied to the screen, and wherein at least one gesture by the user includes swiping on the three-dimensional touchscreen with a consistent pressure on the screen, and the computational circuitry is further configured to detect one or more swipes and the pressure of the swipes and apply the selected color to a location within the user's facial image with a thickness that depends on the pressure.

[0119] (8) The augmented reality system for makeup described in feature (2) or (3), wherein the computational circuitry of the makeup objective visualization unit is further configured to analyze the gesture and estimate the problem domain using a problem domain reinforcement learning model.

[0120] (9) The augmented reality system for makeup described in feature (2) or (3), wherein the computational circuitry of the makeup objective visualization unit is further configured to analyze the gesture and estimate facial feature enhancements using a best feature reinforcement learning model.

[0121] (10) The augmented reality system for makeup described in feature (2) or (3), wherein the computational circuitry of the makeup objective visualization unit is further configured to distinguish between gestures for problem areas and gestures for highlighted facial features using a gesture identification machine learning model.

[0122] (11) The augmented reality system for makeup described in feature (2) or (3), wherein the computational circuitry of the makeup objective visualization unit is further configured to use a voice output function of the mobile device to ask the user whether they would like recommendations on how to apply the digital makeup product to the user's facial image.

[0123] (12) The augmented reality system for makeup described in any of features (1) to (11), wherein the computational circuitry of the makeup palette unit is configured to obtain the subculture palette from a plurality of preset subculture palettes for a plurality of digital makeup products.

[0124] (13) The augmented reality system for makeup described in feature (12), wherein the computational circuitry of the makeup objective visualization unit is further configured to generate a virtual representation of the user's face using a subculture palette, the virtual representation including at least one modification to make the user's face more or less dramatic.

[0125] (14) The augmented reality system for makeup described in feature (12) or (13), wherein the computational circuitry of the makeup objective visualization unit is further configured to generate a virtual representation of the user's face using a subculture palette, including at least one modification to remove blemishes by blurring a portion of the user's face.

[0126] (15) The augmented reality system of any of features (1) to (14), wherein the computational circuitry of the makeup purpose visualization unit is further configured to transfer the virtual representation of the user's face to a platform for live video.

[0127] (16) The augmented reality system for makeup described in any of features (1) to (15), wherein the computational circuitry of the makeup goal visualization unit is further configured to transmit one or more digital images of the user including at least a portion of the user's face to a second makeup goal visualization unit for a second user, and the computational circuitry of the second makeup goal visualization unit is configured to analyze the user's facial image to identify facial features, track and record in memory at least one gesture by the second user applying a digital makeup product to the user's facial image, analyze the at least one gesture to estimate problem areas on the user's face or estimate enhancement of certain facial features, and store in its memory the estimated problem areas or estimated enhanced facial features along with the applied coverage, shade, and finish.

[0128] (17) The augmented reality system for makeup described in feature (16), wherein the computational circuitry of the second makeup objective visualization unit is further configured to receive a second user experience level in applying makeup, detect one or more swipes on the touchscreen, apply a selected color to an area of the user's facial image at a location of a facial feature indicated by the swipe, the facial feature having a boundary, and analyze the applied color to determine whether the one or more swipes are beyond a tolerance amount from the boundary, the tolerance amount being based on the second user's experience level in applying makeup.

[0129] (18) The augmented reality system for makeup described in feature (16) or (17), wherein the computational circuitry of the second makeup objective visualization unit is further configured to analyze the gesture and estimate the problem domain using a problem domain reinforcement learning model.

[0130] (19) The augmented reality system for makeup described in any of features (16) to (18), wherein the computational circuitry of the second makeup objective visualization unit is further configured to analyze gestures and estimate facial feature enhancements using a best feature reinforcement learning model.

[0131] (20) The augmented reality system for makeup described in any of features (16) to (19), wherein the computing circuitry of the second makeup objective visualization unit is further configured to distinguish between gestures for problem areas and gestures for highlighted facial features using a gesture identification machine learning model.

[0132] (21) An augmented reality system for makeup, comprising: a makeup goal unit including a computational circuit operably coupled to a graphical user interface configured to generate one or more instances of user-selectable makeup goals and receive user-selected makeup goal information; a makeup palette unit operably coupled to the makeup goal unit, the makeup palette unit including a computational circuit configured to generate at least one digital makeup palette for digital makeup products; and a makeup goal visualization unit including a computational circuit configured to analyze a user's face to identify one or more of face shape, facial landmarks, skin tone, hair color, eye color, lip shape, eyelid shape, hairstyle, and lighting, and automatically create one or more instances of a custom virtual try-on for the user according to the user-selected makeup goal information and the at least one digital makeup palette generated based on the analysis of the user's face.

[0133] (22) The augmented reality system for makeup according to feature (21), wherein the at least one digital makeup palette includes a plurality of colors having different coverages, shades, and finishes.

[0134] (23) The augmented reality system for makeup described in feature (21) or (22), wherein the computational circuitry of the makeup objective visualization unit is further configured to receive one or more digital images of the user including at least a portion of the user's face and analyze the user's facial images to identify facial features.

[0135] (24) The augmented reality system for makeup described in feature (23), wherein the computational circuitry of the makeup objective visualization unit is further configured to correct brightness in one or more digital images to clarify facial features.

[0136] (25) The augmented reality system for makeup described in any of features (21) to (24), wherein the computational circuitry of the makeup objective visualization unit is further configured to select at least one makeup filter based on the acquired at least one digital makeup palette and create a custom virtual try-on for the user using the at least one makeup filter.

[0137] (26) The augmented reality system for makeup according to feature (25), wherein at least one makeup filter is formed using a previously recorded gesture.

[0138] (27) The augmented reality system for makeup according to feature (26), wherein at least one makeup filter is formed using a previously stored estimated problem area and estimated enhanced facial features.

[0139] (28) The augmented reality system for makeup described in feature (26) or (27), wherein the computational circuitry of the makeup objective visualization unit is further configured to select at least one makeup filter based on past look preferences.

[0140] (29) The augmented reality system for makeup described in feature (23) or (24), wherein the computational circuitry of the makeup objective visualization unit is further configured to create a custom virtual try-on for the user, including removing blemishes by blurring portions of the user's face.

[0141] (30) The augmented reality system for makeup described in any of features (21) to (29), wherein the computational circuitry of the makeup objective visualization unit has the capability of storing one or more instances in memory and is further configured to display one or more instances of the custom virtual try-on.

[0142] (31) The augmented reality system for makeup described in any of features (21) to (30), wherein the computational circuitry of the makeup object visualization unit is further configured to store in memory one or more instances of custom virtual try-on created based on the plurality of digital makeup palettes.

[0143] (32) The augmented reality system for makeup described in any one of features (21) to (31), wherein the computational circuitry of the makeup object visualization unit is further configured to transfer the custom virtual try-on to a platform that provides live video.

[0144] (33) The augmented reality system for makeup described in any of features (21) to (32), further comprising a recommender system including an indexed database including a plurality of makeup filters, wherein the recommender system is configured to retrieve one or more makeup filters from the indexed database in response to a request indicating how to apply digital makeup, generate a sequence of video frames by blending a facial image with the retrieved one or more makeup filters, and display the sequence of video frames.

[0145] (34) The augmented reality system for makeup described in feature (33), wherein the indexed database includes a plurality of categories of makeup filters, and the recommender system is further configured to retrieve one or more makeup filters from the indexed database in response to a request to apply digital makeup to one of the plurality of categories of makeup filters.

[0146] (35) The augmented reality system for makeup described in feature (34), wherein a category of the makeup filter is a subculture makeup look, the computational circuitry of the makeup objective unit is configured to generate the subculture makeup look as one or more instances of the user-selectable makeup objective, and the computational circuitry of the makeup objective visualization unit is configured to automatically create one or more instances of a custom virtual try-on for the user according to the subculture makeup look selected by the user.

[0147] (36) The augmented reality system for makeup described in feature (35), wherein the computational circuitry of the makeup objective visualization unit is further configured to select an instance of the custom virtual try-on from among the one or more instances and transfer the selected instance of the custom virtual try-on to a platform that provides live video.

[0148] (37) The augmented reality system for makeup described in feature (36), wherein the computational circuitry of the makeup objective visualization unit is further configured to create makeup filters as one or more instances of the custom virtual try-on, select one makeup filter from the created plurality of makeup filters, and apply the selected makeup filter to the live video.

[0149] (38) The augmented reality system of feature (37), wherein the selected makeup filter is for one subculture makeup look among a plurality of subculture makeup looks selected by the user, and the computational circuitry of the makeup objective visualization unit is further configured to apply the selected makeup filter to the live video to achieve the subculture makeup look.

Claims

1. 1. An augmented reality system for makeup, comprising: a makeup goal unit including a computational circuit operably coupled to a graphical user interface, the computational circuitry configured to generate one or more instances of a user-selectable makeup goal and to receive user-selected makeup goal information via the graphical user interface, the makeup goal information including a user's experience level in applying makeup; a makeup palette unit operably coupled to the makeup goal unit, the makeup palette unit including a computational circuit configured to generate at least one digital makeup palette for digital makeup products according to makeup goal information selected by the user, the makeup palette unit including an experience level of the user; a makeup goal visualization unit including a computational circuit configured to generate one or more instances of a virtual try-on according to the user-selected makeup goal information; and Equipped with the one or more instances of the virtual try-on are generated using makeup applied by the user within a tolerance of facial feature boundaries in a digital image of the user's face that includes at least a portion of the user's face; An augmented reality system for makeup, wherein the tolerance is based on a user's experience level in applying the makeup.

2. The computation circuit of the makeup purpose visualization unit: receiving the digital image of the user's face; analyzing the digital image of the user's face to identify facial features; Tracking and recording in memory at least one gesture by the user applying the digital makeup product to the digital image of the user's face; analyzing the at least one gesture to estimate problem areas on the user's face or to estimate the emphasis on particular facial features; storing the estimated problem areas or the enhanced facial features along with the applied coverage, color, and finish in the memory; The augmented reality system for makeup of claim 1 , further configured to:

3. The computation circuit of the makeup purpose visualization unit: The augmented reality system for makeup of claim 2 , further configured to generate a virtual representation of a portion of the user's face including at least one modification consistent with the user-selected makeup intent information.

4. It also has a touch screen, the at least one gesture by the user includes one or more swipes on the touchscreen; 3. The augmented reality system for makeup of claim 2, wherein the computational circuitry of the makeup objective visualization unit is further configured to detect the one or more swipes and apply a selected color to a location within the digital image of the user's face.

5. 5. The augmented reality system for makeup of claim 4, wherein the computational circuitry of the makeup objective visualization unit is further configured to detect the one or more swipes on the touchscreen and apply the selected color to an area of the digital image of the user's face that is bounded by a boundary of a facial feature at the position within the digital image of the user's face.

6. The computation circuit of the makeup purpose visualization unit: Detecting the one or more swipes on the touchscreen; applying the selected color to an area of the digital image of the user's face at the location of the facial feature indicated by the swipe; The augmented reality system for makeup of claim 4 , further configured to analyze the applied color to determine if the one or more swipes are beyond an acceptable amount from the boundary.

7. the touchscreen is a three-dimensional touchscreen that senses the amount of pressure applied to the screen; the at least one gesture by the user includes a swipe on the three-dimensional touchscreen with a particular pressure on the screen; 5. The augmented reality system for makeup of claim 4, wherein the computing circuitry is further configured to detect the one or more swipes and the pressure of the swipes and apply the selected color to a location in the digital image of the user's face with a thickness responsive to the pressure.

8. 3. The augmented reality system for makeup of claim 2, wherein the computational circuitry of the makeup objective visualization unit is further configured to analyze the at least one gesture and estimate the problem domain using a problem domain reinforcement learning model.

9. 3. The augmented reality system for makeup of claim 2, wherein the computational circuitry of the makeup objective visualization unit is further configured to analyze the gesture and estimate enhancement of the facial feature using a best feature reinforcement learning model.

10. 3. The augmented reality system for makeup of claim 2, wherein the computational circuitry of the makeup objective visualization unit is further configured to distinguish between gestures over problem areas and gestures over enhanced facial features using a gesture discrimination machine learning model.

11. 3. The augmented reality system for makeup of claim 2, wherein the computational circuitry of the makeup objective visualization unit is further configured to use a voice output capability of a mobile device to ask the user whether they would like recommendations on how to apply the digital makeup product to the digital image of the user's face.

12. The augmented reality system for makeup of claim 1 , wherein the computation circuitry of the makeup palette unit is configured to obtain a look palette from a plurality of preset look palettes for a plurality of digital makeup products.

13. The computation circuit of the makeup purpose visualization unit:

13. The augmented reality system for makeup of claim 12, further configured to use the look palette to generate a virtual representation of the user's face that includes at least one modification of the digital image of the user's face.

14. The computation circuit of the makeup purpose visualization unit:

13. The augmented reality system for makeup of claim 12, further configured to use the look palette to generate a virtual representation of the user's face that includes at least one modification to remove blemishes by blurring a portion of the digital image of the user's face.

15. The computation circuit of the makeup purpose visualization unit: The augmented reality system of claim 1 , further configured to transfer one or more instances of the virtual try-on to a platform for live video.

16. The computation circuit of the makeup purpose visualization unit: further configured to transmit at least a portion of the digital image of the user's face to a second makeup goal visualization unit for a second user; The computing circuitry of the second makeup purpose visualization unit: analyzing the at least a portion of the digital image of the user's face to identify facial features; tracking and recording in memory at least one gesture by the second user applying the digital makeup product to the at least a portion of the digital image of the user's face; analyzing the at least one gesture to estimate problem areas on the user's face or to estimate the emphasis on particular facial features; The augmented reality system for makeup of claim 1 , configured to store the estimated problem areas or enhanced facial features along with applied coverage, color, and finish in the memory.

17. The computing circuitry of the second makeup purpose visualization unit: receiving the second user's experience level in applying makeup; Detecting one or more swipes on the touchscreen; applying a selected color to the at least a portion of the area of the digital image of the user's face at a location of a facial feature indicated by the swipe, the facial feature having a boundary; 17. The augmented reality system for makeup of claim 16, further configured to analyze the applied color to determine whether the one or more swipes are beyond a tolerance amount from the boundary, the tolerance amount being based on an experience level of the second user in applying makeup.

18. 17. The augmented reality system for makeup of claim 16, wherein the computational circuitry of the second makeup objective visualization unit is further configured to analyze the gesture and estimate the problem domain using a problem domain reinforcement learning model.

19. 17. The augmented reality system for makeup of claim 16, wherein the computational circuitry of the second makeup objective visualization unit is further configured to analyze the gesture and estimate enhancement of the facial feature using a best feature reinforcement learning model.

20. 17. The augmented reality system for makeup of claim 16, wherein the computational circuitry of the second makeup objective visualization unit is further configured to distinguish between gestures over problem areas and gestures over enhanced facial features using a gesture discrimination machine learning model.

21. 1. An augmented reality system for makeup, comprising: a makeup goal unit including a computational circuit operably coupled to the graphical user interface, the computational circuitry configured to generate one or more instances of a user-selectable makeup goal and to receive user-selected makeup goal information including the user's experience level in applying makeup; a makeup palette unit operably coupled to the makeup object unit, the makeup palette unit including computational circuitry configured to generate at least one digital makeup palette for digital makeup products; a virtual try-on unit including a computational circuit configured to track and record in a memory a face application area on a user's face, analyze makeup swipes during tracking to estimate problem areas and enhanced facial features on the user's face, and store the estimated problem areas and enhanced facial features in a database, wherein the computational circuitry is configured to analyze the swipes based on an experience level of the user, determine whether a swipe has been applied incorrectly, and redo the makeup swipe if the swipe has been applied incorrectly; a makeup goal visualization unit including a computational circuit configured to analyze a user's face to identify one or more of face shape, facial landmarks, skin tone, hair color, eye color, lip shape, eyelid shape, hairstyle, and lighting, and to automatically create one or more instances of a custom virtual try-on for the user according to makeup goal information selected by the user including an experience level of the user and the at least one digital makeup palette generated based on the analysis of the user's face; Equipped with the computation circuitry of the makeup objective visualization unit is further configured to select at least one makeup filter based on the at least one digital makeup palette, and create the custom virtual try-on for the user using the at least one makeup filter; 10. An augmented reality system for makeup, wherein the at least one makeup filter is formed using previously stored estimated problem areas and enhanced facial features.

22. 22. The augmented reality system for makeup of claim 21, wherein the at least one digital makeup palette includes a plurality of colors with coverage, shade, and finish.

23. The computation circuit of the makeup purpose visualization unit: receiving one or more digital images of the user including at least a portion of the user's face; 22. The augmented reality system for makeup of claim 21, further configured to analyze the one or more digital images of the user to identify facial features.

24. The computation circuit of the makeup purpose visualization unit:

24. The augmented reality system for makeup of claim 23, further configured to correct brightness of the one or more digital images of the user to clarify the facial features.

25. 22. The augmented reality system for makeup of claim 21, wherein the at least one makeup filter is formed using a previously recorded gesture.

26. 26. The augmented reality system for makeup of claim 25, wherein the computational circuitry of the makeup goal visualization unit is further configured to select at least one makeup filter based on past look preferences.

27. 24. The augmented reality system for makeup of claim 23, wherein the computational circuitry of the makeup objective visualization unit is further configured to create the custom virtual try-on for the user, including removing blemishes by blurring the portion of the user's face.

28. 22. The augmented reality system for makeup of claim 21, wherein the computational circuitry of the makeup objective visualization unit is capable of storing the one or more instances in a memory and is further configured to display the one or more instances of the custom virtual try-on.

29. 22. The augmented reality system for makeup of claim 21, wherein the computational circuitry of the makeup objective visualization unit is further configured to store in a memory the one or more instances of the custom virtual try-on created based on a plurality of digital makeup palettes.

30. 22. The augmented reality system for makeup of claim 21, wherein the computational circuitry of the makeup objective visualization unit is further configured to transfer the custom virtual try-on to a platform that provides live video.

31. The method further comprises a recommender system including an indexed database including a plurality of makeup filters; The recommender system: retrieving one or more makeup filters from the indexed database in response to a request indicating how to apply digital makeup; generating a sequence of video frames by blending an image of the user's face with the one or more acquired makeup filters; 22. The augmented reality system for makeup of claim 21 configured to display the sequence of video frames.

32. the indexed database includes a plurality of categories of makeup filters; The recommender system:

32. The augmented reality system for makeup of claim 31, further configured to retrieve one or more makeup filters from the indexed database in response to a request to apply digital makeup in one of a plurality of categories of makeup filters.

33. The category of the makeup filter is a makeup look; the computation circuitry of the makeup goal unit is configured to generate the makeup look as the one or more instances of a user-selectable makeup goal; 33. The augmented reality system for makeup of claim 32, wherein the computational circuitry of the makeup objective visualization unit is configured to automatically create one or more instances of a custom virtual try-on for a user according to a makeup look selected by the user.

34. 34. The augmented reality system for makeup of claim 33, wherein the computational circuitry of the makeup objective visualization unit is further configured to select an instance of the custom virtual try-on from among the one or more instances, and transfer the selected instance of the custom virtual try-on to a platform that provides live video.

35. 35. The augmented reality system for makeup of claim 34, wherein the computational circuitry of the makeup objective visualization unit is further configured to create makeup filters as the one or more instances of the custom virtual try-on, select one makeup filter from the created plurality of makeup filters, and apply the selected makeup filter to the live video.

36. the selected makeup filter is for one makeup look among a plurality of makeup looks selected by the user; 36. The augmented reality system of claim 35, wherein the computational circuitry of the makeup goal visualization unit is further configured to apply the selected makeup filter to the live video to achieve the makeup look.

Images