MULTI-CAMERA VIDEO CAPTURE FOR PRECISE CARNATION MEASUREMENT

Abstract

MULTI-CAMERA VIDEO CAPTURE FOR PRECISE FACE MEASUREMENT A computing device obtains a first video recorded by a first camera (e.g., a rear camera of a mobile computing device) with a first field of view in an illuminated environment and obtains a second video recorded by a second camera with a second field of view (e.g., a front camera of a mobile computing device) in an illuminated environment that differs from the first field of view. The second field of view is directed at the face of a living subject. The computing device extracts the ambient lighting images from the first video and the face images from the second video.The computing device processes facial images and ambient lighting images (e.g., using machine learning models) to obtain a plurality of facial skin tone determinations and combines these determinations to produce a combined skin tone determination. Figure for the abstract: none.

Description

Title of the invention: MULTI-CAMERA VIDEO CAPTURE FOR ACCURATE SKIN TONE MEASUREMENT ABSTRACT

[0001] The purpose of this summary is to present a selection of concepts in simplified form which are described more fully below in the detailed description. This summary is not intended to identify key features of the claimed subject matter or to be used as an aid in determining the scope of the claimed subject matter.

[0002] In some embodiments, a computing device performs a method for estimating skin tone from facial images, the method comprising a first video recorded by a first camera having a first field of view in an illuminated environment; obtaining a second video recorded by a second camera having a second field of view in the illuminated environment that differs from the first field of view, wherein the second field of view is directed toward a face of a living subject; extracting a plurality of lighting environment images from the first video; extracting a plurality of face images from the second video that include the face of the living subject; processing the face images and the lighting environment images to obtain a plurality of skin tone determinations of the face;combining, by the computing device, the facial skin tone determinations to determine a combined skin tone determination; and presenting the combined skin tone determination. In some embodiments, the method further comprises constructing a panoramic image from the first video.;

[0003] In some embodiments, processing the face images and the lighting environment images includes running at least one machine learning model using the face images and the lighting environment images as input to generate the output facial skin tone determinations. In some embodiments, this includes running a first machine learning model using the face images and the lighting environment images as input to generate lighting condition information, such as illuminant color, for each of the output face images; and running a second machine learning model using the face images and the corresponding lighting condition information as input to generate the output facial skin tone determinations.

[0004] In some embodiments, the skin tone determinations are evaluated for suitability in further processing steps before being combined for obtain the combined skin tone determination. In an illustrative scenario, the skin tone determinations each include a corresponding confidence level. The computing device compares the corresponding confidence level with a threshold confidence level and omits the skin tone determination from the combination step where the corresponding confidence level is less than the threshold confidence level.

[0005] In some embodiments, a system includes a skin tone estimation unit including computational circuitry configured to obtain a first video recorded by a first camera having a first field of view in an illuminated environment; obtain a second video recorded by a second camera having a second field of view in the illuminated environment that differs from the first field of view, wherein the second field of view is directed toward a face of a living subject; extract a plurality of lighting environment images from the first video; extract a plurality of face images from the second video that include the face of the living subject; process the face images and the lighting environment images to obtain a plurality of skin tone determinations of the face; combine the skin tone determinations of the face to determine a combined skin tone determination of the face;and present the combined determination of facial complexion. ;

[0006] In some embodiments, the first camera is a rear camera, and the computational circuitry is further configured to construct a panoramic image from the first video. In some embodiments, the computational circuitry is further configured to execute at least one machine learning model using the input facial images and lighting environment images to generate the output facial skin tone determinations.

[0007] In some embodiments, the computational circuitry is further configured to execute a first machine learning model using the input face images and lighting environment images to generate lighting condition information for each of the output face images; and execute a second machine learning model using the input face images and corresponding lighting condition information to generate the output facial skin tone determinations.

[0008] In some embodiments, a non-transitory computer-readable medium includes computer-executable instructions stored thereon that, in response to execution by one or more processors of a computer system, cause the computer system to perform actions including presenting a user interface on a mobile computing device instructing a user to capture a panoramic image in an illuminated environment; in response to a user input via the user interface, recording a first video with a rear camera of the mobile computing device in the illuminated environment and recording a second video with a front camera of the mobile computing device in the illuminated environment; extracting a plurality of lighting environment images from the first video; extracting a plurality of face images from the second video that includes the user's face; processing the face images and the lighting environment images to obtain a plurality of determinations of a facial skin tone; and combining the plurality of facial skin tone determinations to determine a combined facial skin tone determination.

[0009] In some embodiments, processing the face images and the lighting environment images includes executing, by the computing device, a first machine learning model using the face images and the lighting environment images as input to generate lighting condition information for each of the output face images; and executing, by the computing device, a second machine learning model using the face images and the corresponding lighting condition information as input to generate the output facial skin tone determinations. Brief description of the drawings

[0010] The foregoing aspects and numerous related advantages of this invention will be more readily appreciated as they become better understood with reference to the following detailed description, when taken in conjunction with the accompanying drawings, in which:

[0011] [Fig-1] [Fig.l] is a schematic illustration of an example of a mode of rea non-limiting disclosure of a system that uses video recorded by multiple cameras to automatically estimate skin tone according to various aspects of the present disclosure;

[0012] [Fig.2] [Fig.2] is a block diagram illustrating non-limiting exemplary embodiments of a mobile computing device and a skin tone determination device according to various aspects of the present disclosure;

[0013] [Fig. 3] [Fig. 3] is a schematic drawing illustrating a non-limiting exemplary embodiment of a user interface instructing a user to record a first video with a rear camera having a first field of view while a second video is recorded by a front camera having a second field of view oriented toward a face of a living subject according to various aspects of the present disclosure;

[0014] [Fig.4] [Fig.4] is a schematic drawing which illustrates an example of a method of rea non-limiting use of normalizing an image that includes a face according to various aspects of the present disclosure;

[0015] [Fig.5] [Fig.5] is a flowchart that illustrates an example of a rea non-limiting disclosure of a method of using video recorded by multiple cameras to automatically estimate skin tone according to various aspects of the present disclosure; and

[0016] [Fig.6] [Fig.6] is a block diagram of an illustrative computing device suitable for use as the computing device of the present disclosure. Detailed description

[0017] There are certain types of products for which it has been difficult to replace an in-person experience with an online interaction. For example, beauty products such as foundations or other makeup products are difficult to browse online and recommend in an automated manner. This is primarily because skin tone is difficult to estimate in an automated manner. While it is possible to capture an image or video of the consumer, it is difficult to ensure that these images can be reliably used to determine skin tone. In the first place, it is difficult to instruct a consumer on how to capture such images in a manner that is accurate and also easy for the consumer.Second, even if consumers are successfully instructed on how to accurately capture such images, variable lighting conditions in the consumer's environment often lead to inaccurate skin tone estimation. What is needed are techniques that can overcome these technical limitations in order to accurately estimate skin tone regardless of lighting conditions.

[0018] In some embodiments of the present disclosure, one or more machine learning models are trained to accurately estimate skin tone in images regardless of lighting conditions. In some embodiments, the models may then be used to estimate a skin tone in a new image, and this estimated skin tone may be used for various purposes. For example, the skin tone may be used to generate a recommendation for a foundation shade that accurately matches the skin tone, or a recommendation for another cosmetic product that is complementary to the estimated skin tone.

[0019] In some embodiments, a computer system uses video recorded by multiple cameras to automatically estimate skin tone using one or more machine learning models, as described in more detail below. Thus, the need for in-person testing of the product is eliminated, and the The ability of a computer system to automatically determine skin tone from images captured by ordinary consumers without photographic skills or technical expertise is greatly improved.

[0020] [Fig. 1] is a schematic illustration of a non-limiting exemplary embodiment of a system that generates automatic skin tone estimates according to various aspects of the present disclosure. As shown, a user 90 has a mobile computing device 102. The user 90 captures a first video with a rear-facing camera and a second video with a front-facing camera in an illuminated environment (e.g., in a room with artificial or natural lighting). The rear-facing camera has a field of view 120 that is oriented away from the user 90, and the front-facing camera (sometimes referred to as a "selfie" camera) has a field of view 122 that is directed toward the user 90.

[0021] The mobile computing device 90 transmits the first video and the second video to a skin tone determination device 104. In some embodiments, the skin tone determination device 104 uses one or more machine learning models 106 to perform a skin tone determination 108 based on the images in the videos. In some embodiments, the skin tone 108 may then be used to make recommendations of one or more products 110 that would complement or otherwise be appropriate for the skin tone 108.

[0022] In order to accurately estimate skin tone, the system obtains facial images of the user under different lighting conditions. The lighting conditions may be changed by moving the mobile computing device 102 while capturing the video. In some embodiments, different lighting conditions are achieved by instructing the user 90 to turn around while recording a self-portrait or “selfie” video with the mobile computing device 102. As the user turns, the orientation of the user’s face changes relative to nearby light sources, and individual frames of the video capture the user’s face under these different lighting conditions.

[0023] However, in a typical use case scenario for the “self-portrait video,” the user 90 is presented with a preview of this video via the display of the mobile computing device 102. This presentation can be distracting and may cause the user 90 to change their facial expression, turn their head from side to side, or tilt their head forward or backward. These types of movements or changes in expression may make it difficult for the system to determine a true skin tone, even in well-controlled lighting environments. Such movements may create shadows or other effects that may be present in some frames but absent or altered in other frames. Facial features may appear and disappear within the frame as the user’s head moves. In addition, these movements may affect the accuracy of the face detection technology, which may be used to ensure that the user's face 90 remains in the front camera's field of view, or to normalize frames extracted from the video, for example by centering the user's face in the frame. In such cases, if the user 90 moves their head, for example, so that both eyes are not clearly visible, the face detection or image normalization processes may be negatively affected.

[0024] To reduce the likelihood of such problems, in some embodiments, the system instructs the user 90 to record a panoramic image using the rear-facing camera while also recording the user's face with the front-facing camera. The system instructs the user 90 to hold the mobile computing device 102 level while slowly rotating in place. This approach directs the focus of the user 90 toward accurately capturing the panoramic image, rather than a self-portrait video. The typical "preview" image of the "self-portrait" video is also omitted from the display of the mobile computing device 102.By redirecting the user's focus and eliminating distractions, the user is more likely to maintain a neutral expression and head orientation and keep the camera in a good position to capture more accurate and consistent facial images with the front-facing camera.

[0025] In the example shown in [Fig. 1], the user 90 rotates in place while holding the mobile computing device 102 in a "self-portrait" position to change the lighting conditions being experienced. As shown, the user holds the mobile computing device 102 at a sufficient distance to allow the user's face to be fully captured by the front camera, while also capturing the panoramic video image with the rear camera.

[0026] In addition to encouraging the user 90 to maintain a stable head and neutral facial expression, this approach allows lighting conditions to be measured from multiple perspectives by multiple cameras, without any additional effort on the part of the user. By measuring the different lighting conditions in this manner, the machine learning model 106 is provided with additional inputs to improve the accuracy of the individual skin tone estimates, which can then be averaged or combined to further improve the accuracy of the overall skin tone determination 108.

[0027] In some embodiments, a first machine learning model is trained to take images extracted from a video (e.g., face images and lighting environment images) as input and to generate lighting condition information as output. In some embodiments, the Lighting condition information generated by the first output machine learning model includes an estimated illuminant (light source) color. In some embodiments, the illuminant color for a face image is estimated by predicting an illuminant color from image information in the face image, predicting an illuminant color in a corresponding lighting environment image that was captured at the same time as the face image, and then averaging or otherwise combining these predicted illuminant colors. In this manner, the illuminant color prediction can be made more accurate for a given face image than if only the face image itself (from a single camera, such as a front-facing camera) were used.Lighting condition information may also include information other than color, such as an expected intensity or direction of a light source.

[0028] Once the lighting condition information is generated, the corresponding face image may be adjusted to account for the color of the light source (or multiple light sources) in the lighting environment. This helps ensure that the system can accurately estimate skin tone in everyday lighting environments, where the color, intensity, and positioning of the light sources will vary depending on the user's location. For example, if the system determines that the lighting environment includes a yellow-colored light source, the system may adjust the corresponding face image to account for the particular yellow color of the light source and thereby avoid inaccurately estimating skin tone as being more yellow than it would appear under a white light source.

[0029] In some embodiments, a second machine learning model is trained to take the extracted face images and corresponding lighting condition information as input, and to output an estimated skin tone. In other embodiments, a single machine learning model may be trained to take extracted images as input and to output an estimated skin tone. Use of the first machine learning model and the second machine learning model may be beneficial in that the lighting condition information may be used to correct the presented color of the image or to detect the colors of other objects in the image, while use of the single machine learning model may be beneficial in terms of reduced training time and complexity.

[0030] In some embodiments, the machine learning models may be neural networks, including, but not limited to, predictive neural networks, convolutional neural networks (CNNs), recurrent neural networks, and generative adversarial networks (GANs). In some embodiments, any suitable training technique may be used, including, but not limited to, gradient descent, which may include stochastic, batch, and mini-batch gradient descent.

[0031] [Fig. 2] is a block diagram illustrating non-limiting exemplary embodiments of a mobile computing device and a skin tone determination device according to various aspects of the present disclosure. As discussed above, the mobile computing device 102 is used to capture a video of a user 90 using a front camera from which facial images are extracted and an additional video using a rear camera from which lighting environment images are extracted. The mobile computing device 102 transmits the video data to the skin tone determination device 104 to determine a skin tone of the user 90.The mobile computing device 102 and the skin tone determination device 104 may communicate using any suitable communication technology, such as wireless communication technologies, including, but not limited to, Wi-Fi, WiMAX, Bluetooth, 2G, 3G, 4G, 5G, and LTE; or wired communication technologies, including, but not limited to, Ethernet, FireWire, and USB. In some embodiments, the communication between the mobile computing device 102 and the skin tone determination device 104 may occur at least in part over the Internet.

[0032] In some embodiments, the mobile computing device 102 is a smartphone, a tablet computing device, or another computing device having at least the illustrated components. In the illustrated embodiment, the mobile computing device 102 includes a front-facing camera 202A, a rear-facing camera 202B, a data collection engine 204, and a user interface engine 206.

[0033] In some embodiments, the data collection engine 204 is configured to obtain video captured by the front camera 202A and extract facial images of the user 90 from that video, and to obtain video captured by the rear camera 202B and extract lighting environment images from that video. The data collection engine 204 may also be configured to capture training images for training the machine learning model 106.

[0034] In some embodiments, the user interface engine 206 is configured to present a user interface for capturing the video data from which the face images and the lighting environment images are extracted. In some embodiments, the user interface engine 206 uses a graphical user interface (GUI), a voice interface, or another interface design. to guide the user through video capture, while omitting the typical “selfie preview” mode in favor of a panoramic image capture interface. In some embodiments, the panoramic image capture interface includes one or more graphical guides to assist the user 90 in capturing video and keeping the rear camera (and, in turn, the front camera) steady during video capture. These features include, in some embodiments, a horizontal guide line and a progress bar.

[0035] [Fig. 3] is a schematic drawing that illustrates a non-limiting exemplary embodiment of a graphical user interface according to various aspects of the present disclosure. As shown, the mobile computing device 102 displays a graphical user interface 300 that includes instructions for capturing a panoramic image as well as graphical guides to assist the user in capturing video and keeping the rear camera (and, in turn, the front camera) steady during video capture. In the example shown, the user is instructed to hold the mobile computing device 102 level and move continuously to capture the panoramic image. The user is instructed to keep the graphical arrow centered on the horizontal line during this movement.In some embodiments, the movement and positioning of the graphical arrow depends on data received from sensors such as accelerometers and gyroscopes of the mobile computing device 102, which are used to detect the movement and orientation of the mobile computing device 102. In some embodiments, as video is captured, the graphical arrow advances along the horizontal line. In some embodiments, the progress of the graphical arrow along the horizontal line depends at least in part on a number of suitable frames extracted from the video. For example, the graphical arrow is moved to the end of the line (or moved or modified to indicate that the process is complete) when a predetermined number of frames (e.g., 6, 10, etc.) suitable for use in the image processing step have been extracted.Illustrative techniques for determining whether images are suitable for use are described below.

[0036] Referring again to [Fig. 2], in some embodiments, the skin tone determination device 104 is a desktop computing device, a server computing device, a cloud computing device, or another computing device that provides the illustrated components. In the illustrated embodiment, the skin tone determination device 104 includes a drive engine 208, a skin tone determination engine 210, an image normalization engine 212, a product recommendation engine 214. In general, the word “engine,” as used herein, refers to the logic integrated into the ins Hardware or software instructions, which may be written in a programming language, such as C, C++, COBOL, JAVA™, PHP, Perl, H™L, CSS, JavaScript, VBScript, ASPX, Microsoft.NET™, and / or the like. An engine may be compiled into executable programs or written in interpreted programming languages. Software engines may be called from other engines or from within themselves. In general, the engines described herein refer to logic modules that may be merged with other engines or may be divided into sub-engines. Engines may be stored in any type of computer-readable media or computer storage device and may be stored and executed by one or more general-purpose computers, thereby creating a specialized computer configured to provide the engine or its functionality.

[0037] As shown, the skin tone determination device 104 also includes a training data store 216, a model data store 218, and a product data store 220. As understood by one skilled in the art, a "data store" as described herein may be any suitable device configured to store data for access by a computing device. An example of a data store is a high-throughput, highly reliable relational database management system (DBMS) that runs on one or more computing devices and is accessible over a high-speed network. Another example of a data store is a key-value store.However, any other suitable technique and / or storage device capable of quickly and reliably providing the stored data in response to requests may be used, and the computing device may be accessed locally instead of on a network, or may be provided as a cloud service. A data store may also include data stored in an organized manner on a computer-readable storage medium, as described below. A person skilled in the art will recognize that separate data stores described herein may be combined into a single data store, and / or that a single data store described herein may be separated into multiple data stores, without departing from the scope of the present disclosure.

[0038] In some embodiments, the training engine 208 is configured to access training data stored in the training data store 216 and to use the training data to generate one or more machine learning models. The training engine 208 may store the generated machine learning models in the model data store 218. In some embodiments, the skin tone determination engine 210 is configured to use one or more machine learning models stored in the model data store 218 to process images. to estimate a skin tone depicted in the images. In some embodiments, the image normalization engine 212 is configured to preprocess the images before they are provided to the training engine 208 or the skin tone determination engine 210 to improve the accuracy of the determinations made. In some embodiments, the product recommendation engine 214 is configured to recommend one or more products stored in the product data store 220 based on a determined skin tone.

[0039] [Fig. 4] is a diagram illustrating a non-limiting exemplary embodiment of normalizing an image that includes a face according to various aspects of the present disclosure. As shown, an image 402 includes an off-center face that occupies only a small portion of the overall image 402. As such, it may be difficult to estimate skin tone based on the image 402. It may be desirable to reduce the non-face area of ​​the image and place the face area of ​​the image in a consistent location.

[0040] In a first normalization action 404, the image normalization engine 212 uses a face detection algorithm to detect the portion of the image 404 that represents the face. The image normalization engine 212 may use the face detection algorithm to find a bounding box 406 that includes the face. In a second normalization action 408, the image normalization engine 212 may modify the image 408 to center the bounding box 406 in the image 408. In a third normalization action 410, the image normalization engine 212 may zoom in on the image 410 so that the bounding box 406 is as large as possible in the image 410.By performing the normalization actions, the image normalization engine 212 may reduce the differences in layout and size between multiple images, and may therefore improve the training and accuracy of the machine learning model as well as the accuracy of the results when applying new images to the machine learning model. In some embodiments, different normalization actions may occur. For example, in some embodiments, instead of centering and zooming to the bounding box 406, the image normalization engine 212 may crop the image to the bounding box 406.As another example, in some embodiments, the image normalization engine 212 may reduce or increase the bit depth, or may downsample or upsample the pixels of the image to match other images collected by different cameras 202 or different mobile computing devices 102.

[0041] In some embodiments, the normalization process and machine learning models described herein may be customized for a particular type of mobile computing devices 102 or cameras 202A, 202B used. to collect training images or images of the living subject. In some embodiments, differences between captured images may be minimized during normalization.

[0042] Although [Fig. 2] illustrates various components as being provided by the mobile computing device 102 or the skin tone determination device 104, in some embodiments, the arrangement of the components may be different. For example, in some embodiments, the skin tone determination engine 210 and the template data store 218 may be present on the mobile computing device 102 so that the mobile computing device 102 may determine skin tones in images captured by the front camera 202A without transmitting the images to the skin tone determination device 104. As another example, in some embodiments, all of the components may be provided by a single computing device.As yet another example, in some embodiments, multiple computing devices may operate together to provide the functionality illustrated as being provided by the skin tone determination device 104.

[0043] [Fig. 5] is a flowchart that illustrates a non-limiting exemplary embodiment of a method of using images extracted from multiple camera views to estimate a facial skin tone of a living subject in an image according to various aspects of the present disclosure. The subject is referred to as a "living subject" in the method 500 to distinguish the subject from training subjects that may be used in a process of training machine learning models that may be used in the method 500. While ground truth skin tone information is available for the training subjects, this information is not available for the living subject.

[0044] At block 502, a computing device obtains a first video recorded by a first camera having a first field of view in an illuminated environment. In some embodiments, a data collection engine 204 of a mobile computing device 102 (e.g., an Apple iPhone 14) uses a rear camera 202B of the mobile computing device 102 to capture video of the environment in which the living subject is located. At block 504, the computing device obtains a second video recorded by a second camera having a second field of view in the illuminated environment. The second field of view differs from the first field of view and is directed toward the face of the living subject. In some embodiments, the data collection engine 204 of the mobile computing device 102 uses a front camera 202A of the mobile computing device 102 to capture video of the face of the living subject.In some embodiments, both videos may start and stop recording simultaneously. or otherwise cover the same or substantially the same time period. This allows the lighting environment images to be more relevant to the lighting conditions that exist when extracting the face images.

[0045] At block 506, the computing device extracts lighting environment images from the first video, and at block 508, the computing device extracts face images from the second video that include the face of the living subject. In some embodiments, the individual face images are matched to corresponding lighting environment images from the same or substantially the same time instance. In some embodiments, the extraction of the face images includes one or more preprocessing steps in which frames of the second video are evaluated for suitability as face imagery in further processing steps.In an illustrative scenario, the computing device measures a brightness level for each frame and omits the frame from the extracted face images when the measured brightness level is below a minimum threshold brightness level or above a maximum threshold brightness level. Alternatively or additionally, other suitability assessments may be performed, such as confirming the presence of the entire face in the frame, or checking for blur or other image quality issues. In some embodiments, the data collection engine 204 transmits the extracted images to the skin tone determination device 104 for further processing.

[0046] In block 510, the computing device processes the extracted facial images and the lighting environment images to obtain skin tone determinations of the living subject's face. In some embodiments, the computing device performs normalization of the images as part of step 510. In some embodiments, the normalization actions illustrated in [Fig. 4] may be applied to the facial images.

[0047] In some embodiments, processing the face images and the lighting environment images includes executing at least one machine learning model using the face images and the lighting environment images as inputs to generate the output facial skin tone determinations. In the example shown in [Fig. 5], at block 512, the computing device executes a first machine learning model using the face images and the lighting environment images as inputs to generate lighting condition information for each of the output face images, and at block 514, the device executes a second machine learning model using the face images and the corresponding lighting condition information as inputs to generate the output facial skin tone determinations. In some embodiments, the lighting condition information includes an illuminant color.

[0048] At block 520, the computing device combines individual skin tone determinations to obtain a combined determination of the skin tone of the living subject. In some embodiments, the estimated skin tones for the normalized images may be averaged or otherwise combined to generate a final skin tone determination.

[0049] In some embodiments, the skin tone determinations are evaluated for suitability in further processing steps before being combined to obtain the combined skin tone determination. In an illustrative scenario, the skin tone determinations each include a corresponding confidence level. The confidence levels may be generated by a machine learning model for the individual skin tone determinations. The computing device compares the corresponding confidence level to a threshold confidence level and omits or gives less weight to the corresponding skin tone determination in the combining step when the corresponding confidence level is less than the threshold confidence level.

[0050] At block 512, the computing device presents the combined skin tone determination. In some embodiments, the skin tone determination engine 210 transmits the combined skin tone determination to the mobile computing device 102. In some embodiments, the user interface engine 206 may present a category, classification, numerical value, or other indication of the skin tone to the user. In some embodiments, the user interface engine 206 may recreate the color for presentation to the user on a display of the mobile computing device 102.

[0051] In some embodiments, a product recommendation engine 214 of the skin tone determination device 104 determines one or more products to recommend based on skin tone. The user interface engine 206 may then present representations of these products to the user and enable the user to purchase or otherwise obtain the products.

[0052] In some embodiments, the product recommendation engine 214 may determine one or more products in the product data store 220 that match the determined skin tone. This may be particularly useful for products such as foundation that are expected to match the user's skin tone. In some embodiments, the product recommendation engine 214 may determine one or more products in the product data store 220 that complement the skin tone but do not match the skin tone. This may be particularly useful for products such as eyeshadow or lipsticks where an exact skin tone match is less desirable. In some embodiments, the product recommendation engine 214 may use a separate machine learning model, such as a recommendation system, to determine a product from the product data store 220 based on products preferred by other users with a matching or similar skin tone. In some embodiments, if an existing product does not match the skin tone, the product recommendation engine 214 may determine the ingredients to create a product that would match the skin tone, and may provide the ingredients to a blending system to create a custom product that would match the skin tone.

[0053] [Fig. 6] is a block diagram that illustrates aspects of an exemplary computing device 600 suitable for use as a computing device of the present disclosure. While several different types of computing devices have been discussed above, the exemplary computing device 600 describes various elements common to many different types of computing devices. While [Fig. 6] is described with reference to a computing device that is implemented as a device on a network, the description below is applicable to servers, personal computers, mobile phones, smartphones, tablets, embedded computing devices, and other devices that may be used to implement portions of embodiments of the present disclosure.Furthermore, those skilled in the art and others will recognize that the computing device 600 may be any of the devices currently available or yet to be developed.

[0054] In its simplest configuration, the computing device 600 includes at least a processor 602 and system memory 604 connected by a communication bus 606. Depending on the exact configuration and type of device, the system memory 604 may be volatile or non-volatile, such as read-only memory (“ROM”), random access memory (“RAM”), EEPROM, flash memory, or similar memory technology. Those skilled in the art and others will recognize that the system memory 604 typically stores data and / or program modules that are immediately accessible and / or currently being operated by the processor 602. In this regard, the processor 602 may serve as the computing center of the computing device 600 by supporting the execution of instructions.

[0055] As illustrated in more detail in [Fig. 6], the computing device 600 may include a network interface 610 comprising one or more components for communicating with other devices on a network. Embodiments of the present disclosure may access basic services that utilize the network interface 610 to perform communications using common network protocols. The network interface 610 may also include a wireless network interface configured to communicate via one or more wireless communication protocols, such as WiFi, 2G, 3G, 4G, LTE, 5G, WiMAX, Bluetooth, Bluetooth Low Energy, and / or the like. As will be appreciated by those skilled in the art, the network interface 610 illustrated in [Fig. 6] may represent one or more of the wireless interfaces or physical communication interfaces described and illustrated above with respect to particular components of the system 100.

[0056] In the exemplary embodiment shown in [Fig. 6], the computing device 600 also includes a storage medium 608. However, it is possible to access the services using a computing device that does not include means for persisting data on a local storage medium. Therefore, the storage medium 608 shown in [Fig. 6] is represented by a dotted line to indicate that the storage medium 608 is optional. In any event, the storage medium 608 may be volatile or non-volatile, removable or non-removable, implemented using any technology capable of storing information such as, but not limited to, a hard disk drive, an SSD, a CD-ROM, a DVD or any other disk storage medium, magnetic cassettes, magnetic tape, a magnetic disk storage medium and / or the like.

[0057] As used herein, the term "computer-readable medium" includes volatile and non-volatile, removable and non-removable media implemented in any method or technology capable of storing information, such as computer-readable instructions, data structures, program modules, or other data. In this regard, system memory 604 and storage medium 608 shown in [Fig. 6] are examples of computer-readable media.

[0058] Suitable implementations of computing devices including a processor 602, system memory 604, a communication bus 606, a storage medium 608, and a network interface 610 are known and commercially available. For ease of illustration and because it is not important to understanding the claimed subject matter, [Fig. 6] does not show some of the typical components of many computing devices. In this regard, computing device 600 may include input devices, such as a keyboard, keypad, mouse, microphone, touch input device, touchscreen, tablet, and / or the like.These input devices may be coupled to the computing device 600 via wired or wireless connections including RF, infrared, serial, parallel, Bluetooth, Bluetooth low energy, USB, or other suitable connection protocols using wireless or physical connections. Similarly, the computing device 600 may also include output devices such as a display, speakers, a printer, etc. Since these devices are . well known in the art, they are not illustrated or described in more detail here.

[0059] While illustrative embodiments have been illustrated and described, it will be appreciated that various changes may be made thereto without departing from the spirit and scope of the invention. For example, while embodiments described above train and use models to estimate skin tone, in some embodiments, skin features other than skin tone may be estimated. For example, in some embodiments, one or more machine learning models may be trained to estimate a Fitzpatrick skin type using techniques similar to those discussed above with respect to skin tone, and these models may then be used to estimate Fitzpatrick skin types for images of live subjects.

Claims

Claims

1. A system, comprising: a skin tone estimation unit comprising computation circuitry configured to: obtain a first video recorded by a first camera having a first field of view in an illuminated environment; obtain a second video recorded by a second camera having a second field of view in the illuminated environment that differs from the first field of view, wherein the second field of view is directed toward a face of a living subject; extract a plurality of lighting environment images from the first video; extract a plurality of face images from the second video that includes the face of the living subject; process the face images and the lighting environment images to obtain a plurality of facial skin tone determinations; combine the facial skin tone determinations to determine a combined facial skin tone determination;and present the combined determination of facial complexion.;

2. The system of claim 1, wherein the first camera is a rear camera, and wherein the computational circuitry is further configured to construct a panoramic image from the first video.

3. The system of claim 1, wherein the computational circuitry is further configured to execute at least one machine learning model using the input facial images and lighting environment images to generate the output facial skin tone determinations.

4. The system of claim 1, wherein the computational circuitry is further configured to: execute a first machine learning model using the input face images and lighting environment images to generate lighting condition information for each of the output face images; and execute a second machine learning model using the input face images and corresponding lighting condition information to generate the skin tone determinations of the face out.

5. The system of claim 4, wherein the lighting condition information comprises an illuminant color.

6. The system of claim 4, wherein the skin tone determinations each include a corresponding confidence level.

7. The system of claim 6, wherein the computation circuitry is further configured to combine the facial skin tone determinations to determine a combined skin tone determination by, for each of the skin tone determinations: comparing the corresponding confidence level with a threshold confidence level; and omitting the skin tone determination from the combining step where the corresponding confidence level is less than the threshold confidence level.

8. The system of claim 1, wherein the computational circuitry is further configured to, for individual frames of the second video: measure a brightness level of the frame; and omit the frame from the extracted face images when the measured brightness level is below a minimum threshold brightness level or above a maximum threshold brightness level.

9. A non-transitory computer-readable medium having stored thereon computer-executable instructions that, in response to execution by one or more processors of a computer system, cause the computer system to perform actions including: presenting a user interface on a mobile computing device instructing a user to capture a panoramic image in an illuminated environment; in response to user input via the user interface, recording a first video with a rear-facing camera of the mobile computing device in the illuminated environment and recording a second video with a front-facing camera of the mobile computing device in the illuminated environment; extracting a plurality of lighting environment images from the first video; extracting a plurality of face images from the second video that include the user's face;processing of facial images and environmental images; lighting to obtain a plurality of determinations of a facial complexion; and combining the plurality of facial skin tone determinations to determine a combined facial skin tone determination.

10. The non-transitory computer-readable medium of claim 9, wherein processing the face images and the lighting environment images comprises: executing, by the computing device, a first machine learning model using the face images and the lighting environment images as input to generate lighting condition information for each of the output face images; and executing, by the computing device, a second machine learning model using the face images and the corresponding lighting condition information as input to generate the output facial skin tone determinations.

Images