Measuring makeup applicators to improve and share application techniques

Abstract

In some embodiments, a system for generating a presentation of a makeup application is provided. The system includes a computing device including a camera, at least one motion sensor associated with a first applicator, and at least one processor and a non-transitory computer-readable medium having computer-executable instructions stored thereon. In response to execution by the at least one processor, the instructions cause the computing device to perform the following operations: receive a first video of the makeup application process from the camera; the first video depicts at least a portion of the face and at least a portion of the applicator; receive a signal from the at least one motion sensor indicative of movement of the applicator; measure movement of the applicator relative to one or more landmarks on the face to determine a relative applicator path; and store the relative applicator path.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to French Patent Application No. 2212307, filed November 25, 2022, and U.S. Patent Application No. 17 / 937207, filed September 30, 2022. The entire disclosures of the priority applications are incorporated herein by reference for all purposes. Summary of the Invention

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are more fully described in the Detailed Description. This Summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0004] In some embodiments, a method for measurement-based makeup application is provided. A computing device receives a first video of a makeup application process. The first video depicts at least a portion of a first face and at least a portion of a first applicator. The computing device measures movement of the first applicator relative to one or more landmarks on the first face to determine a relative applicator path. The computing device stores the relative applicator path.

[0005] In some embodiments, a system for generating a presentation of a makeup application is provided. The system includes a computing device including a camera, at least one motion sensor associated with a first applicator, and at least one processor and a non-transitory computer-readable medium having computer-executable instructions stored thereon. In response to execution by the at least one processor, the instructions cause the computing device to perform the following operations: receive a first video of the makeup application process from the camera; the first video depicts at least a portion of the face and at least a portion of the applicator; receive a signal from the at least one motion sensor indicative of movement of the applicator; measure movement of the applicator relative to one or more landmarks on the face to determine a relative applicator path; and store the relative applicator path.

[0006] In some embodiments, a computing device is provided, the computing device comprising: a circuit for receiving a first video of a makeup application process, the first video depicting at least a portion of a first face and at least a portion of a first applicator, a circuit for measuring movement of the first applicator relative to one or more landmarks on the first face to determine a relative applicator path, and a circuit for storing the relative applicator path. [Brief explanation of the drawings]

[0007] The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same becomes better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings.

[0008] [Figure 1] 1 is a block diagram illustrating aspects of a non-limiting exemplary embodiment of a metrology computing device according to various aspects of the present disclosure.

[0009] [Figure 2]1 is a schematic diagram illustrating a non-limiting exemplary embodiment of a system that allows for sharing of relative applicator paths between subjects, according to various aspects of the present disclosure.

[0010] [Figure 3] 1 illustrates a non-limiting exemplary embodiment of a video capturing a subject's face in accordance with various aspects of the present disclosure.

[0011] [Figure 4A] 1 is a flowchart illustrating a non-limiting exemplary embodiment of a method for measuring makeup application and generating a presentation according to various aspects of the present disclosure. [Figure 4B] 1 is a flowchart illustrating a non-limiting exemplary embodiment of a method for measuring makeup application and generating a presentation according to various aspects of the present disclosure.

[0012] [Figure 5] 1A-1C illustrate non-limiting exemplary embodiments of frames of a first video of a makeup application process according to various aspects of the present disclosure.

[0013] [Figure 6] 10A-10C illustrate non-limiting exemplary embodiments of frames of a second video including superimposed images in accordance with various aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] It is becoming increasingly common for people to share makeup application techniques. However, makeup tutorials are typically videos of a person applying makeup to themselves, leaving the viewer to understand how to adapt the makeup application to fit their own facial structure. Furthermore, no live guidance is provided while the viewer attempts to recreate the makeup application. What is desired is a system that provides a presentation that improves makeup application and helps users match the makeup application to the demonstration's application despite differences in facial structure between the demonstrator and the applicator.

[0015] In an embodiment of the present disclosure, a first subject records makeup application using a measurement computing device. From this recording, a relative applicator path specifying the position of the makeup applicator during makeup application is determined. Because the relative applicator path is defined in terms of distances from various facial landmarks, the relative applicator path can be presented relative to other faces by calculating the distances of the other faces from various facial landmarks. A presentation including a ghost outline of the applicator, audio feedback, haptic feedback, or visual prompts can be generated to help the second subject guide the applicator along the relative applicator path, thereby improving the second subject's makeup application.

[0016] 1 is a block diagram illustrating aspects of a non-limiting, exemplary embodiment of a metering computing device in accordance with various aspects of the present disclosure. While the illustrated metering computing device 110 is typically a mobile computing device such as a smartphone or tablet, in some embodiments, the metering computing device 110 may be implemented by any computing device or collection of computing devices, including a desktop computing device, a laptop computing device, a mobile computing device, a server computing device, a cloud computing device, and / or combinations thereof. As described in further detail below, the metering computing device 110 is configured to monitor makeup application to determine a path of an applicator during makeup application and to present information based on the determined path to improve subsequent makeup applications.

[0017] As shown, the metrology computing device 110 includes one or more processors 102 , one or more communication interfaces 104 , a path data store 108 , a camera 116 , and a computer-readable medium 106 .

[0018] As used herein, a "data store" includes any suitable device configured to store data for access by a computing device. One example of a data store is a reliable, high-speed relational database management system (DBMS) running on one or more computing devices and accessible over a high-speed network. Another example of a data store is a key-value store. However, other suitable storage technologies and / or devices capable of quickly and reliably providing stored data in response to queries can be used, and the computing devices may be accessible locally rather than over a network or provided as a cloud-based service. A data store may also include data stored in an organized manner on a computer-readable storage medium, such as a hard disk drive, flash memory, RAM, ROM, or any other type of computer-readable storage medium. Those skilled in the art will recognize that the individual data stores described herein may be combined into a single data store and / or that the single data store described herein may be separated into multiple data stores without departing from the scope of the present disclosure.

[0019] As used herein, a "computer-readable medium" includes any removable or non-removable device implementing any technology that stores information in a volatile or non-volatile manner and can be read by a processor of a computing device, including hard drives, flash memory, solid-state drives, random access memory (RAM), read-only memory (ROM), CD-ROMs, DVDs, or other disk storage, magnetic cassettes, magnetic tapes, and magnetic disk storage.

[0020] In some embodiments, processor 102 includes any suitable type of general-purpose computer processor. In some embodiments, processor 102 includes one or more special-purpose computer processors or AI accelerators optimized for particular computing tasks, including, but not limited to, a graphical processing unit (GPU), a vision processing unit (VPU), and a tensor processing unit (TPU).

[0021] In some embodiments, communications interface 104 includes one or more hardware and / or software interfaces suitable for providing communications links between components. Communications interface 104 may support one or more wired communications technologies (such as Ethernet, FireWire, USB, etc.), one or more wireless communications technologies (such as Wi-Fi, WiMAX, Bluetooth, 2G, 3G, 4G, 5G, LTE, etc.), and / or combinations thereof.

[0022] In some embodiments, camera 116 includes any type of digital camera or combination of two or more digital cameras configured to capture at least a portion of a subject's face. In some embodiments, camera 116 captures images using visible light, images using infrared light, depth information, and / or any other type of image information. Some non-limiting exemplary embodiments of devices suitable for use alone or in combination with other devices as camera 116 include front-facing cameras, stereoscopic cameras, and time-of-flight (ToF) cameras on mobile computing devices. In some embodiments, camera 116 includes one or more optical sensors, image sensors, optical transducers, etc.

[0023] As shown, the computer-readable medium 106 has stored thereon logic that, in response to execution by the one or more processors 102, causes the metrology computing device 110 to provide a face detection engine 112, a route determination engine 114, and a route presentation engine 118.

[0024] As used herein, an "engine" includes logic embodied in hardware or software instructions and can be written in one or more programming languages, such as C, C++, C#, COBOL, JAVA, PHP, Perl, HTML, CSS, JavaScript, VBScript, ASPX, GO, or Python. An engine can be compiled into an executable program or written in an interpreted programming language. A software engine can be called by other engines or by itself. Generally, an engine as described herein refers to a logical module that can be merged with other engines or divided into sub-engines. An engine can be implemented by logic stored on any type of computer-readable medium or computer storage device and can be stored and executed on one or more general-purpose computers to create an engine or a special-purpose computer configured to provide its functionality. An engine can be implemented by logic programmed into circuitry such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or another hardware device.

[0025] In some embodiments, face detection engine 112 is configured to build a model of the subject's face using information collected by camera 116 and detect facial landmarks within the model. In some embodiments, path determination engine 114 is configured to determine a path traced by the applicator during makeup application to the face based on one or more of the information collected by camera 116 and motion sensor data associated with the applicator and received via communication interface 104. In some embodiments, path determination engine 114 determines a relative path with respect to facial landmarks detected by face detection engine 112 and stores the relative applicator path in path data store 108. In some embodiments, path presentation engine 118 is configured to generate a presentation of the relative applicator path from path data store 108 to improve subsequent makeup application.

[0026] In some embodiments, the path determination engine 114 includes computational circuitry configured to use one or more convolutional neural network image classifiers to generate a pixel-by-pixel prediction score for the presence or absence of an applicator in the image. In some embodiments, the path determination engine 114 includes computational circuitry configured to predict a path taken by the applicator and generate a virtual representation of the predicted applicator path on at least a portion of the user's face based on the pixel-by-pixel prediction score for the presence or absence of the applicator in the image.

[0027] Further description regarding the configuration of each of these components is provided below.

[0028] While generating a presentation of a relative applicator path to a subject who recorded it is useful in supporting consistent and repeatable makeup application by the subject, one advantage provided by embodiments of the present disclosure that use relative applicator paths defined with respect to facial landmarks is the ability to share the relative applicator path between subjects. Figure 2 is a schematic diagram illustrating a non-limiting, exemplary embodiment of a system that enables the sharing of relative applicator paths between subjects in accordance with various aspects of the present disclosure. As shown, system 200 includes a first measurement computing device 202 used by a first subject 208, a second measurement computing device 204 used by a second subject 210, and a third measurement computing device 206 used by a third subject 212. While Figure 2 illustrates three measurement computing devices and subjects, in some embodiments, system 200 may include more or fewer measurement computing devices and subjects.

[0029] The first subject 208 determines and stores a relative applicator path using the first metrology computing device 202, which then transmits the path to the path management computing system 214. The path management computing system 214 then transmits the relative applicator path to the second metrology computing device 204 and the third metrology computing device 206.

[0030] Because the relative applicator path is recorded by the first subject 208 but is defined in terms of facial landmarks, the same facial landmarks can be determined for other subjects and the relative applicator path can be presented to the other subjects as if they had originally recorded it. In other words, the second metrology computing device 204 can present a relative applicator path to match the facial features of the second subject 210, and the third metrology computing device 206 can present a relative applicator path to match the facial features of the third subject 212.

[0031] In some embodiments, the path management computing system 214 provides various functions enabled by the ability to exchange relative applicator paths between subjects. For example, in some embodiments, the path management computing system 214 may provide an interface through which the second subject 210 can query a particular makeup application tutorial (e.g., a cat-eye eyeliner look, a smoky eye look, etc.). The path management computing system 214 provides one or more associated relative applicator paths to the second subject 210 to create a look, perhaps generated by the first subject 208 using the first metering computing device 202, and the second metering computing device 204 presents the relative applicator paths to the second subject 210. Such embodiments may help influencers and the like grow their audiences and improve their ability to guide subjects through makeup tutorials.

[0032] As another example, in some embodiments, the path management computing system 214 may provide a social connection interface that enables subjects to connect with one another to provide preference information regarding looks, colors, and / or products, share relative applicator paths, and review relative applicator paths posted by others. The social connection interface may then use a recommender engine to determine one or more recommended relative applicator paths that the subjects may like based on the subjects' preference information and posted reviews, and provide the recommended relative applicator paths to a metering computing device associated with the subjects for presentation.

[0033] 3 illustrates a non-limiting, exemplary embodiment of a video capturing a subject's face in accordance with various aspects of the present disclosure. In some embodiments, the video may be displayed on a display of a metering computing device (such as a mobile computing device) as the video is being captured, although for clarity, the metering computing device itself is not illustrated in FIG. 3 (or in the remaining figures of the present disclosure).

[0034] FIG. 3 illustrates some non-limiting examples of facial landmarks that may be detected by the face detection engine 112. As shown, the video includes the alae 302, or "wings," of the nose, nostrils 304, or "nostrils," inner canthus 306 of the eyes, outer canthus 308 of the eyes, and glabella 310 (the midline junction between the eyes and eyebrows). In some embodiments, other facial landmarks may be detected, including features of the lips, ears, bone structure, and / or other facial landmarks. In some embodiments, the surface contours of the face itself may be used as facial landmarks. By locating these facial landmarks within a model of the face in the video, a relative applicator path (relative to the facial landmarks) can be determined and stored.

[0035] 4A-4B are flowcharts illustrating non-limiting exemplary embodiments of methods for measuring makeup application and generating a presentation according to various aspects of the present disclosure. In method 400, a relative applicator path is determined while recording the makeup application. The relative applicator path can then be used to generate a presentation during a second makeup application and to improve the second makeup application, as described above.

[0036] From a start block, method 400 proceeds to optional block 402, where a first motion sensor is associated with a first applicator. In some embodiments, the first motion sensor comprises a 3-axis, 6-axis, or 9-axis accelerometer that generates signals representative of the orientation, position, and / or movement of the motion sensor (and, by association, of any object to which the motion sensor is attached). In some embodiments, other types of motion sensors are used, including photo sensors, cameras, magnetic sensors, tomographic sensors, infrared sensors, ultrasonic sensors, force sensors, and / or combinations thereof.

[0037] The motion sensor may be associated with the first applicator in any suitable manner. For example, the motion sensor may be incorporated into the housing of the first applicator or may be otherwise permanently coupled to the first applicator. As another example, the motion sensor may be removably coupled to the first applicator using a clip, strap, detent, pocket, or other removable coupling means. As yet another example, the motion sensor may be attached to a glove, watch, or other wearable device worn by the subject on the hand holding the first applicator.

[0038] In optional block 404, the first motion sensor is paired with the first metering computing device 110. In some embodiments, the first motion sensor communicates via Bluetooth, Wi-Fi, or other wireless communication technology, and pairing the first motion sensor with the first metering computing device 110 causes the first motion sensor to transmit information to the first metering computing device 110. In some embodiments, pairing the first motion sensor with the first metering computing device 110 may include scanning an identifier of the first motion sensor using the camera 116 of the first metering computing device 110. Blocks 402 and 404 are described as optional because, in some embodiments, the method 400 may operate using video information without motion sensor information.

[0039] In block 406, the camera 116 associated with the first metrology computing device 110 captures a first video of the makeup application process. The first video depicts at least a portion of the face of the first subject and at least a portion of the first applicator. In some embodiments, the first video includes a two-dimensional representation of a scene including a portion of the face of the first subject and a portion of the first applicator. In some embodiments, the first video includes a three-dimensional representation of the scene. In some embodiments, the first video includes depth information along with the two-dimensional representation of the scene.

[0040] FIG. 5 illustrates a non-limiting, exemplary embodiment of a frame of a first video of a makeup application process according to various aspects of the present disclosure. The frame illustrated in FIG. 5 is a non-limiting example of a frame of the first video captured in block 406. Within the frame, a portion of a first subject's face is visible, with the first subject applying makeup using a first applicator 502. In the illustrated embodiment, the first applicator 502 is an eyeliner pencil and the applied makeup 504 is eyeliner, although these examples should not be considered limiting. In other examples, the first applicator 502 may be a brush, a sponge, tweezers, a powder puff, a makeup printer, or any other type of tool that can be used to apply makeup. In other examples, the applied makeup 504 may be lipstick, blush, mascara, foundation, concealer, or any other type of makeup.

[0041] 4A , method 400 proceeds to optional block 408, where path determination engine 114 of first metrology computing device 110 receives signals from a first motion sensor during capture of a first video. The signals indicate the orientation, position, and / or movement of the first motion sensor, and therefore the first applicator, in the video. In some embodiments, path determination engine 114 associates these signals with a depiction of the first applicator in the first video. The operations of block 408 are shown and described as optional because, in some embodiments, method 400 may operate using video data without motion sensor signals.

[0042] In block 410, the face detection engine 112 of the first measurement computing device 110 constructs a face model of the first subject based on the first video. In some embodiments, the face model may be a two-dimensional model based on two-dimensional information present in the first video. In some embodiments, the face model may be a three-dimensional model constructed from a point cloud derived from depth information or other three-dimensional information in the first video. In some embodiments, the face model may be a three-dimensional model constructed from two-dimensional information present in the first video, for example, using machine learning image processing techniques (e.g., LOLNeRF developed by Google, Inc.). In some embodiments, constructing the model may include predicting portions of the model that are occluded in one or more frames of the first video, such as portions of the face that are obscured by the first applicator or the subject's hands. In some embodiments, an initial face model may be constructed based on a portion of the first video captured with an unobstructed view of the face (e.g., before makeup application begins).

[0043] In block 412, the face detection engine 112 detects one or more facial landmarks within a model of the first subject's face. In some embodiments, the one or more facial landmarks are detected within the model itself. In some embodiments, the one or more facial landmarks are detected in a two-dimensional image of the first video and then associated with a location within the facial model. Any suitable technique for detecting facial landmarks may be used, including image processing routines provided by the open-source dlib library.

[0044] In block 414, the path determination engine 114 of the first metrology computing device 110 builds a model of the position of the first applicator based on the first video and, optionally, signals from the motion sensor. In some embodiments, the path determination engine 114 builds a similar model of the first applicator in the same coordinate space as the face model using techniques similar to those used to build the face model in block 410, and may generate a two-dimensional or three-dimensional model similar to the techniques of block 410.

[0045] The use of a motion sensor is optional because, in some embodiments, the information in the first video may be sufficient to construct a model of the position of the first applicator. In other embodiments, the motion sensor signal may be used to enhance the details of the position model, such as providing detailed rotational and pose information that is not readily apparent in the first video. The motion sensor signal is particularly useful when a large portion of the first applicator is obscured in the first video (e.g., when the subject holds the first applicator near its tip, obscuring most of the first applicator), or when the first applicator is designed such that detailed features of the first applicator other than its smoothness, reflectiveness, or shape are not easily detectable in the first video.

[0046] In block 416, the path determination engine 114 measures the position of the first applicator with reference to one or more facial landmarks to determine a relative applicator path. Once a model of the face and first applicator is generated, the relative distance between the first applicator and facial landmarks can be easily measured for each frame of the first video. The relative distances for each frame can be sequentially combined to create the relative applicator path. The relative applicator path may also include the pose of the first applicator with respect to the facial landmarks (e.g., angle of incidence compared to the surface normal, twist over time, etc.). In some embodiments, the first applicator may also include a force sensor that sends a signal to the measurement computing device 110 so that the amount of force applied by the first applicator is included as part of the relative applicator path. In some embodiments, the amount of force applied by the first applicator can be determined by measuring the amount of deflection of at least a portion of the first applicator in the first video.

[0047] At block 418, the path determination engine 114 stores the relative applicator path in the path data store 108 of the first metrology computing device 110. The method 400 then proceeds to a continuation terminal ("terminal A").

[0048] From terminal A (FIG. 4B), the method 400 proceeds to block 420, where the path determination engine 114 transmits the relative applicator path to the path management computing system 214, which, at block 422, transmits the relative applicator path to the second metering computing device 110 and stores it in the path data store 108 of the second metering computing device 110. As explained above, the path management computing system 214 can distribute the relative applicator path to the second metering computing device 110 for various reasons. For example, a second subject associated with the second metering computing device 110 may view the relative applicator path or submit a query for the relative applicator path, and the path management computing system 214 can transmit the relative applicator path to the second metering computing device 110 in response to the request. As another example, if the path management computing system 214 determines that a second subject associated with the second metering computing device 110 may be interested in the relative applicator path, it may push the relative applicator path to the second metering computing device 110 without receiving a request.

[0049] In optional block 424, a second motion sensor is associated with a second applicator, and in optional block 426, the second motion sensor is paired with the second metrology computing device 110. The association of the second motion sensor with the second applicator and pairing with the second metrology computing device 110 is similar to that described above with respect to the first motion sensor, first applicator, and first metrology computing device 110, and therefore will not be described in detail here for the sake of brevity.

[0050] In block 428, the camera 116 associated with the second metrology computing device 110 captures a second video, which is live video depicting at least a portion of a second face of the second subject and at least a portion of the second applicator. As used herein, "live video" refers to video that is captured and processed in real time and that is continuously captured during subsequent operation of method 400. A non-limiting example of capturing live video is capturing a selfie video with a front-facing camera on a mobile computing device. The selfie video is displayed on the display of the mobile computing device while the video is being captured, making the display of the mobile computing device appear like a mirror. Another non-limiting example of capturing live video is capturing video with another type of camera that is streamed to a second metering computing device 110 during capture.

[0051] At block 430, the face detection engine 112 of the second metrology computing device 110 constructs a second face model of the second subject based on the second video, and at block 432, the face detection engine 112 detects one or more facial landmarks in the second face model of the second subject. The techniques for constructing the second face model and detecting facial landmarks are similar to those described above with respect to the first face and will not be described again here for the sake of brevity.

[0052] In block 434, the path presentation engine 118 of the second measurement computing device 110 generates a presentation of the relative applicator path based on one or more facial landmarks in the second facial model while capturing the second video. As part of generating the presentation of the relative applicator path, the path presentation engine 118 may align the relative applicator path to the live video by locating facial landmarks in the live video and determining a location for the presentation based on a distance to the facial landmark indicated by the relative applicator path.

[0053] In various embodiments, a variety of different types of presentations may be provided, each of which may help improve makeup application by the second subject. As a non-limiting example, the presentation may use the relative applicator path and facial landmarks on the second face to determine where the second applicator would be placed if the same makeup pattern were applied to the second face instead of the first face, and generate an image to overlay on the live video to guide the second applicator along the relative applicator path. In some embodiments, if the second applicator is determined to be within the error range of the relative applicator paths, no presentation may occur.

[0054] 6 illustrates a non-limiting example embodiment of a frame of a second video including superimposed images according to various aspects of the present disclosure. In FIG. 6, a portion of the face of a second subject is shown that is different from the first subject that was initially used to determine the relative applicator path. The second subject's face will have a different shape (e.g., different eye shape and position, different nose shape and position, etc.), but the path presentation engine 118 can compensate for the difference in facial shape because the relative applicator path represents the position of the applicator relative to facial landmarks, rather than an absolute position.

[0055] A portion of a face and a second applicator 602 are visible within the frames. Two different types of presentations are illustrated. The first type of presentation is a ghost presentation 604. In the ghost presentation 604, a semi-transparent shape, outline, or other form of presentation is displayed that is intended to represent the ideal position of the second applicator 602 corresponding to the position of the first applicator in the relative applicator path. With each frame, the ghost presentation 604 is moved to match the continuation of the relative applicator path, making it appear as if the ghost presentation 604 is animated along the relative applicator path, and may also be moved based on the movement of facial landmarks to maintain a fixed relative position to the face.

[0056] 6 is an indicator presentation 606. In indicator presentation 606, instead of fully depicting the desired position of second applicator 602, multiple indicators are superimposed on second applicator 602 to prompt the desired movement of second applicator 602 (in this case, a row of chevrons indicating the desired direction of movement of second applicator 602).

[0057] While both ghost presentation 604 and indicator presentation 606 are shown in FIG. 6 , in some embodiments, only a single type of presentation may be provided. In some embodiments, other types of presentation may be provided. For example, if the second applicator 602 is determined to be outside a predetermined range around the relative applicator path, an audible or tactile indication may be provided to prompt the second applicator 602 to return to the relative applicator path. Also, in embodiments describing an amount of pressure applied by the applicator to the relative applicator path, if the amount of pressure being applied by the second applicator 602 is determined to be different from the amount of pressure indicated by the relative applicator path, the presentation may include a visual, audible, or tactile indication.

[0058] Returning to FIG. 4B, after the presentation is generated, the method 400 proceeds to an end block and ends.

[0059] In describing method 400, to clearly illustrate various aspects of the disclosed subject matter, an embodiment was described in which a relative applicator path is determined while a first subject is applying makeup to their face, and then the relative applicator path is used while a second subject is applying makeup to their face. However, in other embodiments, different situations may occur. For example, in some embodiments, a first subject may record a relative applicator path and then play back the relative applicator path to ensure consistency across multiple makeup applications. As another example, in some embodiments, a first subject may record a relative applicator path on one side of their face, provide a command to mirror the relative applicator path, and then use the relative applicator path on the other side of their face to provide symmetrical makeup application.

[0060] As yet another example, the subject and the person applying the makeup may be different. In such an embodiment, the person applying the makeup may use a head-mounted display or other augmented reality device to overlay a presentation on the subject while applying the makeup. Such an embodiment may have additional use cases in which a first person performs a task related to interacting with a different face. For example, a first dental hygienist may record a relative applicator path to track the path of a dental tool in a first mouth, and a second dental hygienist may display a presentation related to the relative applicator path as they use the dental tool in a second mouth. In such an embodiment, intraoral landmarks (e.g., specific tooth locations) may be used instead of the facial landmarks described above.

[0061] While illustrative embodiments have been shown and described, it will be appreciated that various changes can be made without departing from the spirit and scope of the invention. [Example]

[0062] Example 1. A method of makeup application by measurement, the method comprising: receiving, by a computing device, a first video of a makeup application process; measuring, by the computing device, movement of a first applicator relative to one or more landmarks of a first face to determine a relative applicator path; and storing, by the computing device, the relative applicator path, wherein the first video depicts at least a portion of the first face and at least a portion of the first applicator.

[0063] Example 2. The method of Example 1, further including: receiving, by a computing device, a second video; aligning, by a computing device, the relative applicator path with one or more landmarks of the second face; and generating, by a computing device, a presentation of the relative applicator path to be provided in association with capturing the second video, wherein the second video is live video depicting at least a portion of the second face and at least a portion of the second applicator.

[0064] Example 3. The method of example 2, wherein the first video and the second video include depth information.

[0065] Example 4. The method of any one of Examples 2-3, further comprising providing a presentation of the relative applicator path as an image overlaid on the second video.

[0066] Example 5. The method of any one of Examples 2-4, further comprising: providing a presentation of the relative applicator path by the steps of: comparing the detected position of the second applicator with the relative applicator path; and providing tactile feedback in response to the detected position of the second applicator being determined to be outside the relative applicator path.

[0067] Example 6. The method of any one of Examples 2-5, wherein generating a presentation of the relative applicator path provided in association with the capture of the second video includes displaying the relative applicator path along with the second video on a display of the computing device during the capture of the second video.

[0068] Example 7. The method of any one of Examples 2-6, wherein the first face and the second face are different faces.

[0069] Example 8. A system for generating a presentation of a makeup application, comprising: a computing device including a camera; at least one motion sensor associated with a first applicator; at least one processor; and a non-transitory computer-readable medium having computer-executable instructions stored thereon that, in response to execution by the at least one processor, cause the computing device to perform the following actions: receive, by the computing device, a first video of the makeup application process from the camera, the first video depicting at least a portion of the face and at least a portion of the applicator; receive, by the computing device, a signal from the at least one motion sensor indicative of movement of the applicator; measure, by the computing device, movement of the applicator relative to one or more landmarks on the face and determine a relative applicator path; and store, by the computing device, the relative applicator path.

[0070] Example 9. The system of Example 8, wherein the at least one motion sensor is removably coupled to the applicator, permanently attached to the applicator, or coupled to a glove worn by a hand holding the applicator.

[0071] Example 10. The system of any one of Examples 8-9, wherein the computing device is a mobile computing device that includes a display.

[0072] Example 11. The system of example 10, wherein the camera is a front-facing camera of a mobile computing device.

[0073] Example 12. The method of Example 11, including receiving, by the computing device, a second video from the camera; aligning, by the computing device, a relative applicator path with one or more landmarks on the face; and presenting, by the computing device, the relative applicator path in relation to capturing the second video, wherein the second video is live video depicting at least a portion of the face and at least a portion of the applicator.

[0074] Example 13. The method of example 12, wherein the first video and the second video include depth information.

[0075] Example 14. The method of any one of Examples 12-13, wherein presenting the relative applicator path in relation to the capture of the second video includes providing the presentation of the relative applicator path as an image superimposed on the second video on the display.

[0076] Example 15. The method of any one of Examples 12-14, wherein presenting the relative applicator path in relation to capturing the second video includes comparing the detected applicator position with the relative applicator path, and providing haptic feedback in response to the detected applicator position being determined to be outside the relative applicator path.

[0077] Example 16. A computing device including: circuitry for receiving a first video of a makeup application process; circuitry for measuring movement of a first applicator relative to one or more landmarks of a first face to determine a relative applicator path; and circuitry for storing the relative applicator path, the first video depicting at least a portion of the first face and at least a portion of the first applicator.

[0078] Example 17. The computing device of example 16, including circuitry for receiving a second video; circuitry for aligning a relative applicator path with one or more landmarks of the second face; and circuitry for generating a presentation of the relative applicator path as provided in association with capturing the second video, wherein the second video is live video depicting at least a portion of the second face and at least a portion of the second applicator.

[0079] Example 18. The computing device of example 17, including circuitry that provides a presentation of the relative applicator path as an image superimposed on a second video.

[0080] Example 19. The computing device of any one of Examples 17-18, including: circuitry for providing a presentation of the relative applicator path by the steps of: comparing the detected position of the second applicator with the relative applicator path; and circuitry for providing tactile feedback in response to the detected position of the second applicator being determined to be outside the relative applicator path.

[0081] Example 20. The computing device of any one of Examples 17-19, wherein generating a presentation of the relative applicator path provided in association with capturing the second video includes displaying the relative applicator path along with the second video on a display of the computing device during capture of the second video.

Claims

1. 1. A method of metered makeup application, comprising: receiving, by a computing device, a first video of a makeup application process; measuring, with the computing device, movement of a first applicator relative to one or more landmarks on a first face to determine a relative applicator path; storing, by the computing device, the relative applicator paths; Including, the first video depicts at least a portion of the first face and at least a portion of the first applicator; method.

2. receiving, by the computing device, a second video; aligning, with the computing device, the relative applicator path with one or more landmarks on a second face; generating, by the computing device, a presentation of the relative applicator path to be presented in association with the capture of the second video; further comprising the second video is live video depicting at least a portion of the second face and at least a portion of a second applicator; The method of claim 1.

3. the first video and the second video include depth information; The method of claim 2.

4. providing a presentation of the relative applicator paths as an image overlaid on the second video. The method of claim 2.

5. comparing the detected position of the second applicator with a relative applicator path, and providing a presentation of the relative applicator path by providing tactile feedback in response to the detected position of the second applicator being determined to be outside the relative applicator path; The method of claim 2.

6. generating a presentation of the relative applicator path provided in association with capturing the second video includes displaying the relative applicator path along with the second video on a display of the computing device during capture of the second video; The method of claim 2.

7. the first face and the second face are different faces; The method of claim 2.

8. 1. A system for generating a presentation of a makeup application, comprising: A camera and at least one motion sensor associated with the first applicator; a computing device including at least one processor and a non-transitory computer-readable medium having stored thereon computer-executable instructions that, in response to execution by the at least one processor, cause the computing device to perform the following actions: Equipped with receiving, by the computing device, a first video of a makeup application process from the camera; the first video depicts at least a portion of a face and at least a portion of an applicator; receiving, by the computing device, a signal from at least one motion sensor indicative of movement of the applicator; measuring, with the computing device, movement of the applicator relative to one or more facial landmarks to determine a relative applicator path; storing, by the computing device, the relative applicator paths; system.

9. the at least one motion sensor is removably coupled to the applicator, permanently attached to the applicator, or coupled to a glove worn by a hand holding the applicator; The system of claim 8.

10. the computing device is a mobile computing device including a display; The system of claim 8.

11. the camera is a front-facing camera of the mobile computing device; The system of claim 10.

12. receiving, by the computing device, a second video from the camera; the second video is live video depicting at least a portion of the face and at least a portion of the applicator; aligning the relative applicator path with one or more landmarks on the face with the computing device; presenting, by the computing device, the relative applicator path in relation to the capture of the second video. The system of claim 11.

13. the first video and the second video include depth information; The system of claim 12.

14. and presenting the relative applicator path in relation to the capture of the second video includes providing a presentation of the relative applicator path as an image superimposed on the second video on the display. The system of claim 12.

15. presenting the relative applicator path in relation to the capture of the second video includes comparing the detected position of the applicator with the relative applicator path and providing tactile feedback in response to the detected position of the applicator being determined to be outside the relative applicator path. The system of claim 12.

Images