Instrumentation of makeup applicators to improve and share application techniques

Abstract

Instrumentation of makeup applicators to improve and share application techniques. In some embodiments, a system for generating presentations of makeup applications is provided. The system includes a camera, at least one motion sensor associated with a first applicator, and a computing device comprising at least one processor and a computer-readable non-transient medium, with computer-executable instructions stored thereon.The instructions, in response to execution by at least one processor, cause the computing device to perform actions including: receiving a first video of a makeup application process from the camera, the first video representing at least part of a face and at least part of an applicator; receiving signals from at least one motion sensor indicating the movement of the applicator; measuring the movement of the applicator relative to one or more facial landmarks to determine a relative trajectory of the applicator; and storing the relative trajectory of the applicator. Figure for the abstract: none.

Description

Title of the invention: Instrumentation of makeup applicators to improve and share application techniques Summary

[0001] The purpose of this summary is to present a selection of concepts in a simplified form, which are described in greater detail below in the detailed description. This summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0002] In some embodiments, a makeup application method using an instrument is provided. A computer device receives a first video of a makeup application method, the first video representing at least a portion of a first face and at least a portion of a first applicator. The computer device measures the movement of the first applicator relative to one or more reference points on the first face to determine a relative trajectory of the applicator. The computer device records the relative trajectory of the applicator.

[0003] In some embodiments, a system for generating presentations of makeup applications is provided. The system includes a camera, at least one motion sensor associated with a first applicator, and a computer device comprising at least one processor and a computer-readable non-transient medium, with computer-executable instructions stored thereon.The instructions, in response to execution by at least one processor, cause the computing device to perform actions including: receiving a first video of a makeup application process from the camera, the first video representing at least part of a face and at least part of an applicator; receiving signals from at least one motion sensor indicating the movement of the applicator; measuring the movement of the applicator relative to one or more facial landmarks to determine a relative trajectory of the applicator and storing the relative trajectory of the applicator.

[0004] In some embodiments, a computer device is provided. The computer device includes circuits for receiving a first video of a makeup application process, the first video illustrating at least a portion of a first face and at least a portion of a first applicator; circuits for measuring the movement of the first applicator relative to one or more reference points of the first face to determine a relative trajectory of the applicator and circuits for storing the relative trajectory of the applicator.

[0005] The invention relates to a method for applying instrumented makeup, the method comprising: - the reception, by a computer device, of a first video of a makeup application process, the first video representing at least a portion of a first face and at least a portion of a first applicator; - the measurement, by the computer system, of the movement of the first applicator relative to one or more reference points on the first face in order to determine a relative trajectory of the applicator and - the storage, by the computer system, of the relative trajectory of the applicator.

[0006] According to other advantageous aspects of the invention, the method comprises one or more of the following features, taken individually or in all technically possible combinations: - the reception, by the computer device, of a second video, the second video being a live video which represents at least a portion of a second face and at least a portion of a second applicator; - the alignment, by the computer system, of the relative trajectory of the applicator on one or more reference points of the second face and - the generation, by the computer system, of a presentation of the relative trajectory of the applicator to be provided in association with the capture of the second video. - provide a presentation of the relative trajectory of the applicator in the form of images superimposed on the second video. - the presentation of the applicator's relative trajectory in:

[0007] comparing a detected location of the second applicator with the relative trajectory of the applicator and

[0008] providing haptic feedback to determine that the detected location of the second applicator is outside the relative trajectory of the applicator. - the generation of the presentation of the relative trajectory of the applicator to be provided in association with the capture of the second video including the presentation of the relative trajectory of the applicator on a display of the computer device with the second video during the capture of the second video.

[0009] The invention also relates to a computer device, comprising: - circuits for receiving a first video of a makeup application process, in which the first video represents at least a portion of a first face and at least a portion of a first applicator; - circuits for measuring the movement of the first applicator relative to one or more reference points on the first face to determine a relative trajectory of the applicator and - circuits for storing the relative trajectory of the applicator.

[0010] According to other advantageous aspects of the invention, the computer device comprises one or more of the following features, taken individually or in all technically possible combinations: - circuits for receiving a second video, the second video being a live video which represents at least a portion of a second face and at least a portion of a second applicator; - circuits allowing the relative trajectory of the applicator to be aligned with one or more reference points on the second face and - circuits for generating a presentation of the relative trajectory of the applicator to be provided in association with the capture of the second video. - circuits to provide the presentation of the relative trajectory of the applicator in the form of images superimposed on the second video. - circuits to display the relative trajectory of the applicator in:

[0011] comparing a detected location of the second applicator with the relative trajectory of the applicator and

[0012] providing haptic feedback to determine that the detected location of the second applicator is outside the relative trajectory of the applicator. - the generation of the presentation of the relative trajectory of the applicator to be provided in association with the capture of the second video including the presentation of the relative trajectory of the applicator on a display of the computer device with the second video during the capture of the second video. Brief description of the drawings

[0013] The foregoing aspects and many related advantages of this invention will be more readily appreciated as they are better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, in which:

[0014] [Fig-1] Fig. 1 represents a functional diagram that illustrates the aspects of a non-limiting example of an embodiment of a mobile computing device according to various aspects of this disclosure.

[0015] [Fig.2] Fig.2 represents a diagram that illustrates a non-limiting example of an embodiment of a system that allows sharing relative trajectories of the applicator between subjects according to various aspects of this disclosure.

[0016] [Fig.3] Fig.3 is an illustration representing a non-limiting example of mode of producing a video captured of a subject's face according to various aspects of this disclosure.

[0017] [Fig 4A-4B] Figures 4A and 4B are a flowchart that illustrates a non-limiting example of an embodiment of a method for instrumenting makeup applications and generating presentations according to various aspects of this disclosure.

[0018] [Fig. 5] Figure 5 illustrates a non-limiting example of an embodiment of image from an initial video of a makeup application process according to various aspects of this disclosure.

[0019] [Fig. 6] Figure 6 illustrates a non-limiting example of an embodiment of image from a second video with images superimposed according to various aspects of this disclosure. Detailed description

[0020] It is increasingly common for makeup application techniques to be shared among people. However, makeup tutorials are generally videos of someone applying makeup to themselves, and a viewer must figure out how to adapt the makeup application to their own facial structure. Furthermore, no live guidance is provided while the viewer attempts to recreate the makeup application. What is desired are systems that provide demonstrations that help improve makeup application and adapt it to a demonstration application despite differences in facial structure between the demonstrator and the applicator.

[0021] In embodiments of this disclosure, a first subject uses a computer instrumentation device to record a makeup application. From this recording, a relative trajectory of the applicator is determined, specifying the locations of a makeup applicator during the application. The relative trajectory of the applicator is defined with respect to distances from different facial landmarks, so that the relative trajectory of the applicator can be presented with respect to other faces by calculating the distances from the different facial landmarks of the other faces.Presentations including, among other things, ghost outlines of applicators, audio feedback, haptic feedback, or visual prompts can be generated to help a second subject guide an applicator along the relative trajectory of the applicator and thus improve the application of makeup by the second subject.

[0022] Figure 1 represents a functional diagram illustrating aspects of a non-limiting example of an embodiment of a mobile computing device according to various aspects of this disclosure. Although the illustrated instrumentation computing device 110 can typically be a mobile computing device such as a smartphone or tablet, in some embodiments the instrumentation computing device 110 can be implemented by any computing device or set of computing devices, including, among others, a desktop computing device, a portable computing device, a mobile computing device, a server computing device, a computing device of a cloud computing device and / or combinations thereof.As discussed in more detail below, the computer instrumentation device 110 is configured to monitor the application of makeup in order to determine the trajectories of the applicators during application and to present information based on the determined trajectories in order to subsequently apply the makeup.

[0023] As illustrated, the computer instrumentation device 110 includes one or more processors 102, one or more communication interfaces 104, a trajectory database 108; a camera 11116 and a computer-readable medium 106.

[0024] As used herein, the term "datastore" includes, in particular, any suitable device configured to store data for access by a computing device. An example of a datastore is a highly reliable and fast relational database management system (DBMS) running on one or more computing devices and accessible over a high-speed network. Another example of a datastore is a key value store. However, any other suitable storage technique and / or device capable of quickly and reliably providing the stored data in response to queries may be used, and the computing device may be accessed locally instead of being connected to a network, or may be provided as a cloud service.A data bank may also include data stored in an organized manner on a computer-readable storage medium, such as a hard drive, flash memory, RAM, ROM, or any other type of computer-readable storage medium. A person skilled in the art will recognize that separate data banks described herein may be combined into a single data bank, and / or that a single data bank described herein may be separated into multiple data banks, without departing from the scope of this disclosure.

[0025] As used herein, the term "computer-readable medium" includes, in particular, a removable or non-removable device that implements any technology capable of storing information in a volatile or non-volatile manner for reading by a processor of a computing device, including, in particular: a hard disk drive; flash memory; a static disk drive semiconductors; random access memory (RAM); read-only memory (ROM); CD-ROM, DVD or other disk storage; magnetic cassette; magnetic tape and magnetic disk memory.

[0026] In some embodiments, the processors 102 include any suitable type of general-purpose computer processor. In some embodiments, the processors 102 include one or more special-purpose computer processors or AI accelerators / optimized for specific computing tasks, including, but not limited to, graphics processing units (GPUs), vision processing units (VPTs), and Tensor Processing Units (TPUs).

[0027] In some embodiments, the communication interfaces 104 include one or more hardware and / or software interfaces adapted to provide communication links between components. The communication interfaces 104 may support one or more wired communication technologies (including, but not limited to, Ethernet, FireWire, and USB), one or more wireless communication technologies (including, but not limited to, Wi-Fi, WiMAX, Bluetooth, 2G, 3G, 4G, 5G, and LTE), and / or combinations of these technologies.

[0028] In some embodiments, the camera 116 comprises any type of digital camera or a combination of two or more digital cameras configured to capture at least a portion of a subject's face. In some embodiments, the camera 116 captures images using visible light, images using infrared light, depth information, and / or any other type of image information. Some non-limiting examples of devices that can be used as camera 116, alone or in combination with other devices, include a front-facing camera on a mobile computing device, a stereoscopic camera, and a time-of-flight camera. In some embodiments, the camera 116 includes one or more optical sensors, image sensors, optical transducers, or the like.

[0029] As shown in the illustration, the computer-readable medium 106 stores logic which, in response to its execution by one or more processors 102, causes the mobile computing device 110 to provide a face detection engine 112, a trajectory determination engine 114 and a trajectory presentation engine 118.

[0030] As used herein, the term "engine" includes, in particular, the logic embodied in hardware or software instructions, which may be written in one or more programming languages, including, in particular, C, C++, C#, COBOL, JAVA™, PHP, Perl, HTML, CSS, JavaScript, VBScript, ASPX, Go, and Python. An engine may be compiled into executable programs or written in interpreted programming languages. Software engines may be called from other engines or from themselves. In general, the engines described herein refer to logic modules that can be merged with other engines or divided into sub-engines. Engines can be implemented by logic stored in any type of computer-readable storage medium or device and stored and executed by one or more general-purpose computers, thereby creating a special-purpose computer configured to provide the engine or its functionality. Engines can be implemented by logic programmed into application-specific integrated circuits (ASICs), a user-programmable pre-broadcast array (FPGA), or other hardware devices.

[0031] In some embodiments, the face detection engine 112 is configured to use the information collected by the camera 116 to construct a model of a subject's face and to detect facial landmarks in the model. In some embodiments, the path determination engine 114 is configured to determine a path traced by an applicator during the application of makeup to the face based on one or more pieces of information collected by the camera 116 and motion sensor data associated with the applicator and received via a communication interface 104. In some embodiments, the path determination engine 114 determines the path relative to the facial landmarks detected by the face detection engine 112 and stores the relative path of the applicator in the path database 108.In some embodiments, the path presentation engine 118 is configured to generate a relative path presentation of the applicator from the path database 108 to help improve a subsequent makeup application.

[0032] In some embodiments, the trajectory determination engine 114 includes a set of computing circuits configured to generate a per-pixel prediction score for the presence or absence of an applicator in an image using one or more convolutional neural network image classifiers. In some embodiments, the trajectory determination engine 114 includes a set of computing circuits configured to predict a trajectory traversed by the applicator and to generate a virtual display of the predicted applicator trajectory on at least a part of the user's face based on the per-pixel prediction scores for the presence or absence of the applicator in the image.

[0033] A more detailed description of the configuration of each of these components is provided below.

[0034] While generating a presentation of a relative applicator trajectory to a subject who has recorded the relative applicator trajectory can be useful in supporting consistent and reproducible makeup application by the subject, an advantage offered by embodiments of this disclosure that use relative applicator trajectories defined with respect to facial landmarks is the ability to share relative applicator trajectories between subjects. Figure 2 shows a diagram that illustrates a non-limiting example of an embodiment of a system that allows relative applicator trajectories to be shared between subjects according to various aspects of this disclosure.As illustrated, the system 200 comprises a first computer instrumentation device 202 used by a first subject 208, a second computer instrumentation device 204 used by a second subject 210 and a third computer instrumentation device 206 used by a third subject 212. Although three computer instrumentation devices and subjects are illustrated in [Fig.2], in some embodiments, a greater or lesser number of computer instrumentation devices and subjects may be present in the system 200.

[0035] The first subject 208 can use the first computer instrumentation device 202 to determine and record a relative trajectory of the applicator, which the first computer instrumentation device 202 then transmits to a computer trajectory management system 214. The computer trajectory management system 214 then transmits the relative trajectory of the applicator to the second computer instrumentation device 204 and to the third computer instrumentation device 206.

[0036] Since the relative trajectory of the applicator is recorded by the first subject 208 but is defined with respect to facial landmarks, the same facial landmarks can be determined for other subjects, and the relative trajectory of the applicator can be presented to the other subjects as if it had been initially recorded. In other words, the second computer instrumentation device 204 can present the relative trajectory of the applicator to match the facial features of the second subject 210, and the third computer instrumentation device 206 can present the relative trajectory of the applicator to match the facial features of the third subject 212.

[0037] In certain embodiments, the trajectory management computer system 214 offers a variety of functionalities made possible by the ability to exchange relative applicator trajectories between subjects. For example, in certain embodiments, the trajectory management computer system 214 can provide an interface that allows a second subject 210 to search for tutorials on particular makeup applications (e.g., eyeliner for an eye cat, smoky eyes, etc.). The trajectory management computer system 214 can provide one or more relevant relative applicator trajectories to the second instrumentation computer device 204 to create the look generated by, possibly, the first subject 208 using the first instrumentation computer device 202, and the second instrumentation computer device 204 presents the relative applicator trajectories to the second subject 210. Such an embodiment can be useful for influencers and others to develop an audience and improve the ability to guide subjects through makeup tutorials.

[0038] By way of further example, in certain embodiments, the trajectory management computer system 214 may provide a social connection interface that allows subjects to connect with each other to provide preference information regarding appearances, colors, and / or products; share relative applicator trajectories; review relative applicator trajectories published by others, and so on. The social connection interface may then use a recommendation engine to determine one or more recommended relative applicator trajectories that a subject may desire based on their preference information and the reviews that have been published, and may provide the recommended relative applicator trajectories for presentation to the subject's associated instrumentation computer device.

[0039] Figure 3 is an illustration representing a non-limiting example of an embodiment of video capture of a subject's face according to various aspects of this disclosure. In some embodiments, the video may be displayed on a computer instrumentation device (such as a mobile computer) while the video is being captured, although the computer instrumentation device itself is not shown in Figure 3 (or the other illustrations in this disclosure) for the sake of clarity.

[0040] Figure 3 illustrates some non-limiting examples of facial landmarks that can be detected by the facial detection engine 112. As illustrated, the video includes a wing 302, or "wing" of the nose; a "nostril" 304; an inner eyelid angle 306; an outer eyelid angle 308; a nasal point 310; and a median junction between the eyes and below the eyebrows. In some embodiments, other facial landmarks, including, among others, features of the lips, ears, bone structure, and / or other facial landmarks, can be detected. In some embodiments, the surface contour of the face itself can be used as a facial landmark. By finding these facial landmarks in a face model in the video, it is possible to determine and record a relative trajectory of the applicator (with respect to the facial landmarks).

[0041] Figure 4A and Figure 4B are flowcharts illustrating a non-limiting example of a method for instrumenting makeup applications and generating presentations according to various aspects of this disclosure. In Method 400, a relative trajectory of the applicator is determined during the recording of a makeup application. The relative trajectory of the applicator can then be used to generate presentations during a second makeup application in order to improve the second makeup application, as described above.

[0042] Starting from a basic block, method 400 proceeds to optional block 402, where a first motion sensor is associated with a first applicator. In some embodiments, the first motion sensor includes a three-, six-, or nine-axis accelerometer that generates signals representing the orientation, position, and / or motion of the motion sensor (and, by association, of any object to which the motion sensor is attached). In some embodiments, other types of motion sensors include, among others, a photo sensor, a camera, a magnetic sensor, a tomographic sensor, an infrared sensor, an ultrasonic sensor, a force sensor, and / or combinations thereof.

[0043] The motion sensor can be associated with the first applicator in any suitable way. For example, the motion sensor can be integrated into a housing of the first applicator or permanently connected to the first applicator. Another example: the motion sensor can be removably coupled to the first applicator, for example, by means of a clip, strap, ratchet, pocket, or any other removable coupling means. Another example: the motion sensor can be attached to a glove, watch, or other wearable device worn by the subject on the hand holding the first applicator.

[0044] In optional block 404, the first motion sensor is coupled to a first instrumentation computer 110. In some embodiments, the first motion sensor communicates via Bluetooth, Wi-Fi, or another wireless communication technology, and coupling the first motion sensor with the first instrumentation computer 110 results in the transmission of information from the first motion sensor to the first instrumentation computer 110. In some embodiments, coupling the first motion sensor with the first instrumentation computer 110 may include using the camera 116 of the first instrumentation computer 110 to scan an identifier of the first motion sensor. Blocks 402 and 404 are described as optional because, In some embodiments, process 400 can operate using video information without motion sensor information.

[0045] In block 406, a camera 116 associated with the first computer instrumentation device 110 captures a first video of a makeup application process, the first video representing at least a portion of a first subject's face and at least a portion of the first applicator. In some embodiments, the first video includes a two-dimensional representation of a scene that includes the portion of the first subject's face and the portion of the first applicator. In some embodiments, the first video includes a three-dimensional representation of the scene. In some embodiments, the first video includes depth information to accompany the two-dimensional representation of the scene.

[0046] Figure 5 illustrates a non-limiting example of an embodiment of an image from a first video of a makeup application process according to various aspects of this disclosure. The image shown in Figure 5 is a non-limiting example of an image from a first video captured at block 406. In the image, part of the first subject's face is visible, and the first subject applies makeup using the first applicator 502. In the illustrated embodiment, the first applicator 502 is an eyeliner pencil, and the makeup applied 504 is eyeliner, but these examples should not be considered limiting. In other examples, the first applicator 502 could be a brush, sponge, tweezers, powder puff, makeup printer, or any other type of tool usable for applying makeup.In other examples, the 504 makeup applied could be lipstick, blush, mascara, foundation, concealer, or any other type of makeup.

[0047] Returning to Figure 4A, the method 400 proceeds to the optional block 408, where a trajectory determination engine 114 of the first computer instrumentation device 110 receives signals from the first motion sensor during the capture of the first video. The signals indicate the orientation, position, and / or movement of the first motion sensor, and thus of the first applicator, during the video. In some embodiments, the trajectory determination engine 114 correlates these signals with the representation of the first applicator in the first video. The actions of block 408 are illustrated and described as optional because, in some embodiments, the method 400 can operate using the video data without the motion sensor signals.

[0048] In block 410, a facial detection engine 112 of the first computer instrumentation device 110 constructs a model of the first subject's face based on the first video. In some embodiments, the face model may be A two-dimensional model based on two-dimensional information present in the first video. In some embodiments, the face model can be a three-dimensional model constructed from a point cloud derived from depth information or other three-dimensional information in the first video. In some embodiments, the face model can be a three-dimensional model constructed from two-dimensional information present in the first video using, for example, machine learning image processing techniques (e.g., LOLNeRF developed by Google, Inc.). In some embodiments, the model construction can include predictive parts of the model that are obscured in one or more frames of the first video, such as parts of the face that are hidden by the first applicator or by a subject's hand.In some embodiments, an initial face model can be constructed based on a portion of the first video captured with an unobstructed view of the face (e.g., before the start of makeup application).

[0049] In block 412, the facial detection engine 112 detects one or more facial landmarks in the face model of the first subject. In some embodiments, the facial landmark(s) are detected on the model itself. In some embodiments, the facial landmark(s) are detected in two-dimensional images of the first video and then correlated to locations in the face model. Any appropriate technique for detecting facial landmarks may be used, including, among others, image processing routines provided by the publicly available open source dlib library.

[0050] In block 414, a trajectory determination engine 114 of the first instrumentation computer device 110 constructs a location model of the first applicator based on the first video and, optionally, on the motion sensor signals. In some embodiments, the trajectory determination engine 114 uses similar techniques used in block 410 to construct the face model to build a similar model of the first applicator in the same coordinate space as the face model, and, as with the techniques of block 410, can generate a two-dimensional or three-dimensional model.

[0051] The use of the motion sensor is optional because, in some embodiments, the information from the first video may be adequate for constructing the model of the first applicator's locations. In other embodiments, the motion sensor signals may be used to improve the detail of the localization model, including by providing detailed rotation and attitude information that may not be readily visible in the first video. The motion sensor signals may be particularly useful if a large part of the first applicator is obscured in the first video (e.g., if the subject grasps the first applicator close to a tip of the first applicator, so that most of the first applicator is obscured, etc.) or if the first applicator is smooth, reflective or designed in such a way that detailed features of the first applicator other than shape are not readily detectable in the first video.

[0052] In block 416, the trajectory determination engine 114 measures the locations of the first applicator relative to one or more facial landmarks to determine a relative trajectory of the applicator. Once the face and first applicator models are generated, the relative distances between the first applicator and the facial landmarks can be easily measured on each frame of the first video. The relative distances for each frame can then be combined sequentially to create the relative trajectory of the applicator. The relative trajectory of the applicator can also include the attitude (e.g., angle of incidence with respect to a normal surface, twist over time, etc.) of the first applicator with respect to the facial landmarks.In some embodiments, the first applicator may also include a force sensor that transmits signals to the instrumentation computer 110, so that the amount of force with which the first applicator is applied can be included in the relative trajectory of the applicator. In some embodiments, the amount of force with which the first applicator is applied can be determined by measuring the amount of deflection of at least a portion of the first applicator, as seen in the first video.

[0053] At block 418, the trajectory determination engine 114 stores the relative trajectory of the applicator in a trajectory database 108 of the first computer instrumentation device 110. The process 400 then passes to a tracking terminal (“terminal A”).

[0054] From terminal A (Figure 4B), the process 400 proceeds to block 420, where the trajectory determination engine 114 transmits the relative trajectory of the applicator to a trajectory management computer system 214, and to block 422, the trajectory management computer system 214 transmits the relative trajectory of the applicator to a second instrumentation computer device 110 for storage in a trajectory database 108 of the second instrumentation computer device 110. As indicated above, the trajectory management computer system 214 can distribute the relative trajectory of the applicator to the second instrumentation computer device 110 for various reasons. For example, a second subject associated with the second instrumentation computer device 110 can traverse the relative trajectories of the applicator or transmit a request for the applicator's relative trajectories, and the trajectory management computer system 214 can transmit the applicator's relative trajectory to the second instrumentation computer device 110 in response to the request. As another example, the trajectory management computer system 214 can push the applicator's relative trajectory to the second instrumentation computer device 110 without receiving a request if the trajectory management computer system 214 determines that the second subject associated with the second instrumentation computer device 110 may be interested in the applicator's relative trajectory.

[0055] In optional block 424, a second motion sensor is associated with a second applicator, and in optional block 426, the second motion sensor is coupled to the second computer instrumentation device 110. The association of the second motion sensor with the second applicator and the coupling with the second computer instrumentation device 110 is similar to that mentioned above with regard to the first motion sensor, the first applicator and the first computer instrumentation device 110, and is therefore not discussed in more detail here for the sake of brevity.

[0056] In block 428, a camera 116 associated with the second instrumentation computing device 110 captures a second video, wherein the second video is a live video that represents at least a portion of a second face of a second subject and at least a portion of the second applicator. As used here, a "live video" is a video captured and processed in real time, and which continues to be captured during subsequent actions of method 400. A non-limiting example of capturing a live video is the capture of a selfie video by a front-facing camera of a mobile computing device, where the selfie video is presented on a screen of the mobile computing device while the video is being captured, so that the display of the mobile computing device resembles a mirror.Another non-limiting example of live video capture is the capture of video by another type of camera which is streamed to the second computer instrumentation device 110 during its capture.

[0057] At block 430, a facial detection engine 112 of the second computer instrumentation device 110 constructs a model of the second subject's face based on the second video, and at block 432, the facial detection engine 112 detects one or more facial landmarks in the model of the second subject's face. The techniques for constructing the model of the second face and detecting the facial landmarks are similar to those described above with respect to the first face, and are therefore not described again here for the sake of brevity.

[0058] In block 434, a trajectory presentation engine 118 of the second instrumentation computer device 110 generates a presentation of the relative trajectory of the applicator, based on the facial landmark(s) in the second face model, while simultaneously capturing the second video. As part of generating the presentation of the relative trajectory of the applicator, the trajectory presentation engine 118 can align the relative trajectory of the applicator with the live video by finding the facial landmarks in the live video and determining the location of the presentation based on the distances to the facial landmarks indicated by the relative trajectory of the applicator.

[0059] In various embodiments, different types of presentations can be performed, each of which can contribute to improving the application of makeup by the second subject. By way of non-limiting example, the presentation can use the relative trajectory of the applicator and the facial landmarks of the second face to determine where the second applicator would be located if the same makeup pattern were applied to the second face instead of the first, and can then generate images to be superimposed on the live video to guide the second applicator along the relative trajectory of the applicator. In some embodiments, the presentations may not be performed if the second applicator is within a margin of error of the relative trajectory of the applicator.

[0060] Figure 6 illustrates a non-limiting example of an embodiment of a second video image with superimposed images according to various aspects of this disclosure. In Figure 6, a portion of the face of a second subject, different from the first subject used to initially determine the relative trajectory of the applicator, is shown. The second subject's face will have a different shape (e.g., different eye shapes and locations, different nose shapes and locations, etc.), but the relative trajectory of the applicator represents the applicator's locations relative to facial landmarks and not an absolute location, and the trajectory presentation engine will therefore be able to compensate for differences in facial shapes.

[0061] In the image, the face portion and the second applicator 602 are visible. Two different types of presentations are illustrated. The first type of presentation is a ghost presentation 604. The ghost presentation 604 shows a translucent shape, contour, or other presentation shape intended to represent the ideal location of the second applicator 602 to correspond with the location of the first applicator in the relative path of the applicator. In each image, the ghost presentation 604 can be moved to coincide with the continuation of the relative path of the applicator so that the ghost presentation 604 appears to be animated along the relative path of the applicator, and can also be moved according to the movement of facial landmarks to remain in a fixed relative position with respect to the face.

[0062] A second type of presentation illustrated in [Fig.6] is an indicator 606 presentation. In the indicator 606 presentation, instead of fully representing the desired location of the second applicator 602, a plurality of indicators are superimposed on the second applicator 602 to request a desired movement of the second applicator 602 (in this case, a row of chevrons indicating the desired direction of movement of the second applicator 602).

[0063] Although the phantom presentation 604 and the indicator presentation 606 are both shown in [Fig. 6], in some embodiments only one type of presentation may be performed. In other embodiments, other types of presentations may be made. For example, if the second applicator 602 is determined to be outside a predetermined range around the relative trajectory of the applicator, an auditory or haptic cue may be provided to prompt the second applicator 602 to return to the relative trajectory of the applicator.Similarly, in embodiments in which the relative trajectory of the applicator includes a quantity of pressure applied by the applicator, the presentation may include a visual, auditory, or haptic indication when it is determined that a quantity of pressure applied with the second applicator 602 is different from the quantity of pressure indicated by the relative trajectory of the applicator.

[0064] If we return to Figure 4B, after the generation of the presentation, process 400 then moves on to an end block and terminates.

[0065] The description of method 400 presents embodiments in which the relative trajectory of the applicator is determined during a makeup application by a first subject on their own face, and the relative trajectory of the applicator is then used during a makeup application by a second subject on their own face in order to clearly describe various aspects of the disclosed object. However, in other embodiments, different situations may arise. For example, in some embodiments, the first subject may record the relative trajectory of the applicator, and then may also replay the relative trajectory of the applicator in order to ensure consistency between several makeup applications.As another example, in some embodiments, the first subject can record the relative trajectory of the applicator on one side of the face, provide a command to reflect the relative trajectory of the applicator, and then use the relative trajectory of the applicator on the second side of the face to provide a symmetrical application.

[0066] As another example, the subject and the person applying the makeup can be different people. In such an embodiment, the person who The makeup applicator can use a head-mounted display or other augmented reality device to overlay presentations onto the subject during makeup application. Such embodiments may have additional use cases in which a first person performs tasks related to interacting with different faces. For example, a first dental hygienist can record a relative trajectory of the applicator to track the path of a dental tool in a first mouth, and a second dental hygienist can view presentations related to the relative trajectory of the applicator while using the dental tool in a second mouth. In such an embodiment, landmarks in the mouth (e.g., specific tooth locations) can be used instead of the facial landmarks mentioned above.

[0067] Although illustrative embodiments have been shown and described, it will be appreciated that various changes can be made to them without departing from the spirit and scope of the invention.

Claims

Claims

1. A method of applying instrumented makeup, the method comprising: receiving, by a computing device, a first video of a makeup application process, the first video depicting at least a portion of a first face and at least a portion of a first applicator; measuring, by the computing device, movement of the first applicator relative to one or more landmarks of the first face to determine a relative trajectory of the applicator and storing, by the computing device, the relative trajectory of the applicator, receiving, by the computing device, a second video, the second video being a live video that depicts at least a portion of a second face and at least a portion of a second applicator;aligning, by the computing device, the relative trajectory of the applicator to one or more landmarks of the second face and generating, by the computing device, a presentation of the relative trajectory of the applicator to be provided in association with capturing the second video, the presentation of the relative trajectory of the applicator being obtained by: comparing a detected location of the second applicator with the relative trajectory of the applicator and providing haptic feedback to determine that the detected location of the second applicator is outside the relative trajectory of the applicator and providing haptic feedback when it is determined that an amount of pressure applied with the second applicator is different from the amount of pressure indicated by the relative trajectory of the applicator.;

2. The method of claim 1, further comprising: Providing the presentation of the relative trajectory of the applicator as images superimposed on the second video.

3. The method of claim 1, generating the presentation of the relative trajectory of the applicator to be provided in association with capturing the second video comprising the presentation of the relative trajectory of the applicator on a display of the computing device with the second video during capture of the second video.

4. A computing device comprising: circuitry for receiving a first video of a makeup application process, wherein the first video depicts at least a portion of a first face and at least a portion of a first applicator; circuitry for measuring movement of the first applicator relative to one or more markers of the first face to determine a relative trajectory of the applicator and circuitry for storing the relative trajectory of the applicator, circuitry for receiving a second video, the second video being a live video depicting at least a portion of a second face and at least a portion of a second applicator;circuitry for aligning the relative trajectory of the applicator with one or more landmarks of the second face and circuitry for generating a presentation of the relative trajectory of the applicator to be provided in association with capturing the second video, circuitry for presenting the relative trajectory of the applicator by: comparing a detected location of the second applicator with the relative trajectory of the applicator and providing haptic feedback to determine that the detected location of the second applicator is outside the relative trajectory of the applicator and providing haptic feedback when it is determined that an amount of pressure applied with the second applicator is different from the amount of pressure indicated by the relative trajectory of the applicator.;

5. The computing device of claim 4, further comprising: circuitry for providing the presentation of the relative trajectory of the applicator as images superimposed on the second video.

6. The computing device of claim 4, wherein generating the presentation of the relative trajectory of the applicator to be provided in association with capturing the second video comprises presenting the relative trajectory of the applicator on a display of the computing device with the second video while capturing the second video.

Images