SYSTEM AND METHOD FOR PROVIDING PERSONALIZED PRODUCT RECOMMENDATIONS USING DEEP LEARNING

Abstract

To provide systems, methods, and computer-readable media that use machine learning models to enable computing devices to detect and identify cosmetic products in face images.SOLUTION: In some embodiments, a model training system may gather training data for building machine learning models by analyzing face images associated with tagging data. In some embodiments, a recommendation system may be configured to use the machine learning models generated by the model training system to detect products in face images, and to add information based on the detected products to a look data store, and / or to provide recommendations for similar looks from the look data store based on the detected products.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Patent Application No. 16 / 171,153, filed October 25, 2018, the entire contents of which are incorporated herein by reference. Summary of the Invention

[0002] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. [Means for solving the problem]

[0003] In some embodiments, a method is provided for training a machine learning model to detect cosmetics in a facial image. A computing device determines a product present in the facial image based on non-image data. The computing device extracts a portion of the facial image that includes a facial feature associated with the product. The computing device adds the portion of the facial image and an identification of the product to a training dataset. The computing device trains a machine learning model to detect the product in the facial image using the training dataset.

[0004] In some embodiments, a method is provided for using one or more machine learning models to process a facial image. A computing device detects one or more products in the facial image using the one or more machine learning models. The computing device determines one or more recommended looks based on the one or more products in the facial image. The computing device determines one or more products associated with the one or more recommended looks. The computing device presents the one or more recommended looks or one or more products to a user.

[0005] In some embodiments, a system for training a machine learning model to detect cosmetics in a facial image is provided, the system including circuitry for determining a product present in the facial image based on non-image data, circuitry for extracting a portion of the facial image that includes a facial feature associated with the product, circuitry for adding identification information of the portion of the facial image and the product to a training dataset, and circuitry for training the machine learning model to detect the product in the facial image using the training dataset.

[0006] The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0007] [Figure 1] FIG. 2 is a schematic diagram illustrating an example embodiment of logical processing of a facial image according to various aspects of the present disclosure. [Diagram 2] 1 is a block diagram illustrating an example embodiment of a model training system and an example embodiment of a recommendation system in accordance with various aspects of the present disclosure. [Figure 3A]1 is a flowchart illustrating an example embodiment of a method for training one or more machine learning models to detect cosmetics in facial images, according to various aspects of the present disclosure. [Figure 3B] 1 is a flowchart illustrating an example embodiment of a method for training one or more machine learning models to detect cosmetics in facial images, according to various aspects of the present disclosure. [Figure 4] 1 is a flowchart illustrating an example embodiment of a method for using one or more machine learning models to provide appearance-based recommendations, according to various aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Often, cosmetics consumers view photos of people with the look they want to emulate, a combination of features that may include cosmetics, clothing, hairstyles, and / or jewelry. However, these photos are usually not detailed as to which cosmetics are being applied, making it highly unlikely that another consumer would be able to successfully recreate the look. It would be desirable for consumers to have access to technology that automatically determines the cosmetic product or cosmetics shown in a photo. Such technology would be desirable for other reasons as well. For example, the technology that has given computing devices these new possibilities would also enable them to automatically build a collection of looks based on publicly available photos supplemented with detailed information about the products shown, and to use such a collection of data to determine looks similar to a given look.

[0009] FIG. 1 is a schematic diagram illustrating an example embodiment of logical processing of a facial image according to various aspects of the disclosure. A facial image 102 is shown depicting a person's face wearing one or more cosmetic products. Computer vision techniques may be used to identify specific facial features in the facial image, such as eyes, lips, cheeks, and eyebrows. Once a facial feature is detected, a portion of the facial image 102 may be extracted. For example, an eye portion 104, a lip portion 106, and / or a cheek portion 108 may be extracted. The facial image 102 may be associated with tagging information that identifies the cosmetic product being worn in the facial image 102. For example, the tagging information may identify the brand, color, texture, and / or finish of the applied lipstick, applied eyeshadow, applied brush, etc. Once a portion of the facial image (e.g., lip portion 106) is associated with a relevant product (e.g., the identified lipstick), the portion of the facial image 102 and the product information may be added to a set of training data to be used to train the machine learning model 112. Each portion of the facial images 104, 106, 108 may be added to a separate training dataset to train a respective machine learning model 110, 112, 114 to detect the production of a different facial feature.

[0010] FIG. 2 is a block diagram illustrating an example embodiment of a model training system and an example embodiment of a recommendation system according to various aspects of the disclosure. In some embodiments, the model training system 200 is configured to collect face images, generate training data based on the face images, and train one or more machine learning models based on the training data. In some embodiments, the recommendation system 250 is configured to detect products in the face images using the machine learning models generated by the model training system 200, add information based on the detected products to the appearance data store 260, and / or provide recommendations for similar looks from the appearance data store 260 based on the detected products. In some embodiments, the model training system 200 and the recommendation system 250 are each provided with one or more computing devices and may communicate with each other using any suitable communication technology, including, but not limited to, Ethernet, Wi-Fi, 3G, 4G, LTE, and the Internet. In some embodiments, the model training system 200 and the recommendation system 250 may be provided by a single combined system.

[0011] As shown, the model training system 200 includes a data ingestion engine 202, a feature recognition engine 204, a model training engine 206, a training data store 208, and a model data store 210.

[0012] In some embodiments, the data ingestion engine 202 may be configured to obtain facial images and tagging information, from which training data may be generated. The facial images and tagging information may be stored by the data ingestion engine 202 in the training data store 208. In some embodiments, the data ingestion engine 202 may receive content from one or more social networking systems 212, including, but not limited to, Instagram, Twitter, and Pinterest. Social media posts on such systems may include facial images along with text information describing what is shown in the facial images. The text information, which may or may not include semantically delimited content, such as hashtags, may specifically identify one or more products shown in the facial images. In some embodiments, the data ingestion engine 202 may similarly retrieve information from websites, such as blogs, where blog posts may include facial images and text information describing the products shown. In some embodiments, the data ingestion engine 202 may be configured to extract facial image information from video information posted to the social networking system 212, and may be configured to extract text information from the video information using speech-to-text techniques.

[0013] In some embodiments, the data capture engine 202 may be configured to receive information from a laboratory testing system 214. The laboratory testing system 214 may generate facial images showing products applied to subjects or test equipment intended to replicate real facial features, and may provide such facial images to the data capture engine 202 along with information identifying the products shown. In some embodiments, the laboratory testing system 214 may also provide specific information describing the product, including, but not limited to, specular measurement information and color measurement information.

[0014] In some embodiments, the feature recognition engine 204 may be configured to extract a portion of a face image that shows a particular facial feature. For example, the feature recognition engine 204 may receive a face image and extract a portion of the face image that shows the eyes, lips, cheeks, eyebrows, or any other facial feature. In some embodiments, the feature recognition engine 204 may extract the portion of the face image by generating a new, smaller image that includes only the detected facial feature. In some embodiments, the feature recognition engine 204 may extract the portion of the face image by providing coordinates in the original face image that locate the detected facial feature. The portion of the face image may be stored in the training data store 208 along with associated tagging information.

[0015] In some embodiments, the model training engine 206 may be configured to use the training data stored in the training data store 208 to train one or more machine learning models capable of identifying products in a portion of a facial image. In some embodiments, the model training engine 206 may generate a separate machine learning model for each facial feature to be processed. Once generated, the machine learning models may be stored in the model data store 210.

[0016] As shown, the recommendation system 250 includes a feature recognition engine 252, a product detection engine 254, a product recommendation engine 256, a model data store 258, and an appearance data store 260. In some embodiments, the feature recognition engine 252 is similar to the feature recognition engine 204 of the model training system 200 in that it may be configured to extract at least a portion of a face image that exhibits a particular facial feature. In some embodiments, the product detection engine 254 may be configured to receive the portion of the face image extracted by the feature recognition engine 252 and then use the machine learning model stored in the model data store 258 to identify a product shown in the portion of the face image. In some embodiments, the product detection engine 254 may then store the identified products together as one appearance in the appearance data store 260. In some embodiments, the product recommendation engine 256 may receive the products identified by the product detection engine 254 and may analyze the appearances in the appearance data store 260 and determine one or more other products to recommend based on the identified products.

[0017] Further details of the functionality of the model training system 200, the recommendation system 250, and their components are provided below.

[0018] In general, the term "engine" as used herein refers to logic embedded in hardware or software instructions, which may be written in a programming language such as C, C++, COBOL, JAVA, PHP, Perl, HTML, CSS, JavaScript, VBScript, ASPX, Microsoft .NET™, etc. An engine may be compiled into an executable program or written in an interpreted programming language. A software engine may be callable from other engines or from itself. In general, an engine as described herein refers to a logical module that may be combined with other engines or divided into sub-engines. An engine may be stored in any type of computer-readable medium or computer storage device and stored on and executed by one or more general-purpose computers, thus creating an engine or a special-purpose computer configured to provide its functionality.

[0019] As will be appreciated by those skilled in the art, a "data store" as described herein may be any suitable device configured to store data for access by a computing device. One example of a data store is a reliable, high-speed relational database management system (DBMS) running on one or more computing devices and accessible over a high-speed network. Another example of a data store is a key-value store. However, any other suitable storage technique and / or device capable of quickly and reliably providing stored data in response to a query may be used, and the computing device may be accessible locally rather than over a network, or may be provided as a cloud-based service. A data store may also include data that is organized and stored on a computer-readable storage medium, as further described below. Those skilled in the art will recognize that the separate data stores described herein may be combined into a single data store and / or that the single data store described herein may be separated into multiple data stores without departing from the scope of the present disclosure.

[0020] 3A and 3B are flow charts illustrating an example embodiment of a method for training one or more machine learning models to detect cosmetics in facial images according to various aspects of the disclosure. From a start block, the method 300 proceeds to block 302, where a data ingestion engine 202 of the model training system 200 receives a plurality of facial images. The data ingestion engine 202 may receive the plurality of facial images from any source. For example, in some embodiments, the data ingestion engine 202 may search for the plurality of facial images from the social networking system 212, for example, by requesting social media posts from the social networking system 212 that include a particular hashtag. In some embodiments, the data ingestion engine 202 may receive the facial images from an end-user computing device, such as a tablet, smartphone, or a camera-equipped laptop or desktop computer, and software executing on the end-user computing device captures the facial images using the camera of the end-user computing device. In such an embodiment, the software may also prompt the user to identify a product worn by the user that is shown in the facial image. In some embodiments, the plurality of facial images may include training data generated by an operator of the model training system 200 from a dataset, such as a set of marketing materials or a set of laboratory-generated images. In some embodiments, the data capture engine 202 may save the captured facial images in the training data store 208 along with any accompanying information, such as text information from social media posts or tagging information collected by the software.

[0021] The method 300 then proceeds to a for loop defined between a for loop start block 303 and a for loop end block 315, where each of the facial images of the plurality of facial images is processed. From the for loop start block 303, the method 300 proceeds to block 304, where the data ingestion engine 202 determines tagging data associated with the facial image. In some embodiments, if the facial image is associated with a social media post or other internet content, the data ingestion engine 202 may review hashtags or other text associated with the facial image to discover textual identification of a brand name, product name, color, texture, or finish that may be used to identify the depicted product. In some embodiments, if product packaging is visible in the facial image, the data ingestion engine 202 may recognize text on the product packaging, recognize the packaging by comparing it to a stored image of the packaging, or analyze a barcode depicted on the packaging to determine tagging data. In some embodiments, the tagging data may be generated for the facial image by an operator and provided to the data ingestion engine 202.

[0022] Next, in block 306, the data ingestion engine 202 determines the product or products identified by the tagging data. In some embodiments, the product name, color, texture, and / or finish included in the tagging data may be associated by the data ingestion engine 202 with a stock keeping unit (SKU) that uniquely identifies the product. In some embodiments, the data ingestion engine 202 may cross-reference partial information in the tagging data (such as product name and color without finish, or only product name) with laboratory test data obtained from the laboratory test system 214 to find detailed identification information for the particular product, color, finish, and texture indicated.

[0023] The method 300 then proceeds to another for loop defined between a for loop start block 308 and a for loop end block 314, where the facial image is processed for each product identified by the data ingestion engine 202. From the for loop start block 308, the method 300 proceeds to block 310, where the feature recognition engine 204 of the model training system 200 extracts a portion of the facial image that includes a facial feature associated with the product. For example, if the product is lipstick, the feature recognition engine 204 extracts a portion of the facial image that includes the lips of the face. As another example, if the product is eyeshadow, eyeliner, or mascara, the feature recognition engine 204 extracts a portion of the facial image that includes at least one of the eyes of the face.

[0024] At block 312, the portion of the extracted facial image is stored in a training set for facial features in the training data store 208 along with the product's identity. The method 300 then proceeds to a for loop end block 314. If any additional products have been identified, the method 300 returns to the for loop begin block 308 to process the next product. Otherwise, the method 300 proceeds to a for loop end block 315. If additional facial images are to be processed, the method 300 returns to the for loop begin block 303 to process the next facial image. Otherwise, the method 300 proceeds to a continuation terminal ("terminal A").

[0025] From terminal A (FIG. 3B), the method 300 proceeds to a for loop defined between a for loop start block 316 and a for loop end block 324, where a machine learning model is trained for each of the facial features (e.g., eyes, lips, cheeks) supported by the method 300. From the for loop start block 316, the method 300 proceeds to block 318, where the model training engine 206 of the model training system 200 retrieves training data for the facial feature from the training data store 208. For example, if the facial feature is lips, the model training engine 206 retrieves a plurality of portions of the facial image from the training data store 208 that include lips, along with information identifying a product in the portion of the facial image. At block 320, the model training engine 206 trains a machine learning model for the facial feature using the retrieved training data. Any suitable type of machine learning model, including but not limited to a convolutional neural network (CNN), may be used, as may any suitable technique for training the machine learning model, including but not limited to gradient descent.

[0026] At block 322, the model training engine 206 stores the machine learning model in the model data store 210. The method 300 then proceeds to a for loop end block 324. At the for loop end block 324, if additional facial features have been processed, the method 300 returns to block 316 to process the next facial feature. Otherwise, the method 300 proceeds to an end block and ends.

[0027] 4 is a flow chart illustrating an example embodiment of a method for using one or more machine learning models to provide appearance-based recommendations according to various aspects of the disclosure. In some embodiments, the method 400 may use a recommendation system 250 having a model data store 258 that is pre-loaded with one or more machine learning models trained by the method 300 described above and / or stored in the model data store 210 of the model training system 200.

[0028] From a start block, the method 400 proceeds to block 402 where the recommendation system 250 receives an appearance image to be processed. The appearance image may be received in a manner similar to the face images collected by the data ingestion engine 202 described above, in that the appearance image may be received from the social networking system 212 or software executing on an end user's computing device. In some embodiments, the appearance image may be provided to the recommendation system 250 by tagging the appearance image with a particular hashtag or identifier associated with the recommendation system 250 on the social networking system 212. The appearance image may show a face and may include one or more products that may be detected by a machine learning model.

[0029] At block 404, the feature recognition engine 252 of the recommendation system 250 extracts one or more portions of the appearance image that include facial features. The feature recognition engine 252 may extract portions of the appearance image that correspond to the facial features for which the machine learning model was trained by the model training system 200, which may be extracted using techniques similar to those described above with respect to the feature recognition engine 204.

[0030] The method 400 then proceeds to a for loop defined between a for loop start block 406 and a for loop end block 412, where each extracted facial feature is processed to identify the product shown therein. From the for loop start block 406, the method 400 proceeds to block 408, where the product detection engine 254 of the recommendation system retrieves a machine learning model for the facial feature from the model data store 258. For example, if the facial feature is lips, the product detection engine 254 retrieves a machine learning model from the model data store 258 that is trained to detect products on lips. Next, in block 410, the product detection engine 254 feeds a portion of the appearance image showing the facial feature to the machine learning model to identify the product applied to the facial feature. In some embodiments, the output of the machine learning model may include the identity of the product, color, texture, and / or finish. In some embodiments, the output of the machine learning model may also include a confidence score on how likely the identity of the product is accurate.

[0031] The method 400 then proceeds to a for loop end block 412. If no more facial features have been processed, the method 400 returns to the for loop begin block 406 to process the next facial feature. Otherwise, if all facial features have been processed, the method 400 proceeds to block 414, where the product recommendation engine 256 of the product recommendation system 250 determines one or more recommended products based on the products detected in the appearance image. In some embodiments, the one or more recommended products may be determined by finding at least one appearance in the appearance data store 260 that is recommended based on the products detected in the appearance image, and recommending a product from the recommended appearance.

[0032] To find the at least one recommended appearance, in some embodiments, the product recommendation engine 256 may search the appearance data store 260 to find other appearances that are comparable to some aspects of the product detected in the appearance image. For example, the product recommendation engine 256 may search for stored appearances that include at least one product that is comparable to the product detected in the appearance image and at least one product that was not detected in the appearance image. As another example, the product recommendation engine 256 may search for stored appearances that include products that have at least one comparable or complementary color or finish when compared to the product in the appearance image. In some embodiments, the product recommendation engine 256 may perform a clustering analysis on the appearances in the appearance data store 260 to determine the type of similar appearances, and may find the recommended appearances in the cluster to which the combination of products in the appearance image is assigned. Once the at least one recommended appearance is determined based on the detected products, the products included in the recommended appearances may be used as recommended products.

[0033] Once the recommended products are determined, in block 416, the product recommendation engine 256 presents one or more recommended products. In some embodiments, the product recommendation engine 256 may present the products themselves. In some embodiments, the product recommendation engine 256 may present an image associated with the recommended appearance along with an identification of the recommended product. In some embodiments, the suggestions may be provided within the software used to interact with the recommendation system 250. In some embodiments, the suggestions may be provided within another channel, such as email or the content of a target website. In some embodiments, feedback regarding the recommended appearance and / or the recommended products may be accepted by the software and used to improve future recommendations.

[0034] The method 400 then proceeds to an end block where it ends.

[0035] The method 400 above primarily describes using machine learning models to detect products and then using those detected products to discover similar looks. In some embodiments, machine learning models may also be used to detect products in publicly available photos and then add new looks to the appearance data store 260 based on those photos. In some embodiments, looks added in this way to the appearance data store 260 may be supplemented using sentiment analysis of content associated with the photos, such as social media text / engagement, comments on articles, or blog posts, to rank the quality of the looks for use in future recommendations.

[0036] While illustrative embodiments have been illustrated and described, it will be understood that various changes can be made therein without departing from the spirit and scope of the invention.

[0037] The embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows: [Explanation of symbols]

[0038] 102 Face images 104 Eyes 106 Lips 108 Cheek area 110 Machine Learning Models 112 Machine Learning Models 114 Machine Learning Models 200 Model Training System 202 Data Ingestion Engine 204 Feature Recognition Engine 206 Model Training Engine 208 Training Data Store 210 Model Data Store 212 Social Networking Systems 214 Laboratory Test Systems 250 Recommended Systems 252 Feature Recognition Engine 254 Product Detection Engine 256 Product Recommendation Engine 258 Model Data Store 260 Appearance Data Store 300 ways 400 ways

Claims

1. 1. A method of using one or more machine learning models to process facial images, comprising: providing, by a computing device, a facial image as an input to the one or more machine learning models to generate, as an output, an identification of one or more cosmetic products in the facial image; determining, by the computing device, one or more recommended looks based on the identification of the one or more cosmetic products in the facial image; determining, by the computing device, one or more cosmetic products associated with the one or more recommended looks; presenting, by the computing device, the one or more recommended looks or the one or more cosmetic products to a user; Including, determining one or more recommended looks based on the identification of the one or more cosmetic products in the facial image; adding a new look to a look data store that includes the identity of the one or more cosmetic products; performing a clustering analysis on the appearances stored in said appearance data store; determining the one or more recommended looks based on other looks included in a cluster having the new look; A method comprising:

2. 10. The method of claim 1 , wherein the one or more machine learning models include one or more convolutional neural networks.

3. The method of claim 1 or 2, wherein the facial image is contained within a social media post.

4. determining, by the computing device, an appearance based on the determined identity of the one or more cosmetic products; determining, by the computing device, a sentiment score by using sentiment analysis based on the social media post; storing, by the computing device, the emotion score and the appearance in an appearance data store; 4. The method of claim 3, further comprising:

5. providing the facial image as an input to the one or more machine learning models to generate as an output the identification of one or more cosmetic products in the facial image; Extracting a portion of the face image associated with a predetermined facial feature; providing the portion of the facial image as an input to a machine learning model trained to generate as an output the identity of the cosmetic product applied to the selected facial feature; 5. The method according to claim 1 , comprising:

6. The method of claim 5 , wherein the predetermined facial feature is an eye, a lip, a forehead, a cheekbone, or a nose.

7. determining the one or more recommended looks based on the identification of the one or more cosmetic products in the facial image, determining one or more appearances including at least one cosmetic product having a color or texture comparable to a color or texture of one or more identified cosmetic products in the facial image and at least one identified cosmetic product not comparable to the color or texture of the one or more identified cosmetic products in the facial image; 7. The method of claim 1 , comprising:

8. 1. A system that uses one or more machine learning models to process facial images, comprising: a circuit for providing a facial image as an input to the one or more machine learning models to generate as an output an identification of one or more cosmetic products in the facial image; a circuit for determining one or more recommended looks based on the identification of the one or more cosmetic products in the facial image; a circuit for determining one or more cosmetic products associated with the one or more recommended looks; a circuit for presenting the one or more recommended appearances or the one or more cosmetic products to a user; Including, determining one or more recommended looks based on the identification of the one or more cosmetic products in the facial image; adding a new look to a look data store that includes the identity of the one or more cosmetic products; performing a clustering analysis on the appearances stored in said appearance data store; determining the one or more recommended looks based on other looks included in a cluster having the new look; Including, the system.

9. 10. The system of claim 8, wherein the one or more machine learning models include one or more convolutional neural networks.

10. The system of claim 8 or 9, wherein the facial image is included in a social media post.

11. a circuit for determining an appearance based on the determined identity of the one or more cosmetic products; a circuit for determining a sentiment score by using sentiment analysis based on the social media post; a circuit for storing the emotion scores and the appearance in an appearance data store; The system of claim 10 further comprising:

12. providing the facial image as an input to the one or more machine learning models to generate as an output the identification of the one or more cosmetic products in the facial image; Extracting a portion of the facial image associated with a predetermined facial feature; providing the portion of the facial image as an input to a machine learning model trained to generate as an output an identity of a cosmetic product applied to the selected facial feature; 12. The system according to claim 8, comprising:

13. The system of claim 12 , wherein the predetermined facial feature is an eye, a lip, a forehead, a cheekbone, or a nose.

14. determining the one or more recommended looks based on the identification of the one or more cosmetic products in the facial image; determining one or more appearances including at least one cosmetic product having a color or texture comparable to a color or texture of one or more cosmetic products identified in the facial image and at least one cosmetic product not comparable to the color or texture of the one or more cosmetic products identified in the facial image; 14. The system according to claim 8, comprising:

15. A non-transitory computer-readable medium having stored thereon computer-executable instructions that, in response to execution by one or more processors of a computing device, cause the computing device to perform a method according to any one of claims 1 to 7.

Images