Augmented reality makeup design filters

Abstract

Systems, devices, and methods for generating, sharing, and presenting an augmented reality cosmetic design filter. One or more non-transitory computer memory devices can store computer-readable instructions that, when executed by a computing circuitry, cause the computing circuitry to perform operations for generating an augmented reality cosmetic design filter. The operations can include defining a baseline description of a biological surface using one or more radiation sensors, the radiation sensors being electronically coupled to the computing circuitry. The operations can include identifying an application of a cosmetic formulation to the biological surface using the one or more radiation sensors. The operations can include generating a trace describing the application with reference to the baseline description. The operations can also include outputting the trace.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Patent Application No. 17 / 491,061, filed September 30, 2021, and French Patent Application No. 2113547, filed December 15, 2021, the contents of which are incorporated herein by reference in their entireties. Summary of the Invention

[0003] Systems, devices, and methods for generating, sharing, and presenting an augmented reality cosmetic design filter. One or more non-transitory computer memory devices can store computer-readable instructions that, when executed by computing circuitry, cause the computing circuitry to perform operations for generating an augmented reality cosmetic design filter. The operations can include defining a baseline description of a biological surface using one or more radiation sensors. The radiation sensors are electronically coupled to the computing circuitry. The operations can include identifying application of a cosmetic formulation to the biological surface using the one or more radiation sensors. The operations can include generating a trace describing the application with reference to the baseline description. The operations can also include outputting the trace.

[0004] In some embodiments, identifying the application of the cosmetic preparation includes detecting the cosmetic preparation on the biological surface using one or more of the radiation sensors and the baseline description. Generating the trace can include receiving information describing a plurality of shape types, attributing a first shape type of the shape types to at least a portion of the biological surface using the baseline description, generating a numerical representation of the application of the cosmetic preparation, the numerical representation describing positional information relative to the baseline description, and converting the numerical representation from the first shape type to a second shape type of the shape types. The biological surface can be a face. The first shape type and the second shape type can describe facial features. Defining the baseline description of the biological surface can include projecting invisible electromagnetic radiation onto the biological surface using a radiation source electronically coupled to computing circuitry. The one or more radiation sensors can include a camera configured to detect electromagnetic radiation from the ultraviolet, visible, or infrared spectral ranges, or a combination thereof.

[0005] In some embodiments, generating a trace includes tracking a movement of the application relative to the biological surface and generating a numerical representation of the movement relative to a baseline description. The operations can further include identifying a cosmetic formulation and defining a color of the cosmetic formulation using the cosmetic formulation identifier information. The operations can further include estimating a surface tone of the biological surface, estimating a formulation tone of the cosmetic formulation using the surface tone, and determining a color of the cosmetic formulation using the surface tone and the formulation tone.

[0006] In some embodiments, the one or more non-transitory computer memory devices are electronically coupled to the smartphone, and outputting the trace can include communicating the trace to a remote computing system.

[0007] A method for generating an augmented reality cosmetic design filter can include defining a baseline description of a biological surface using one or more radiation sources and one or more radiation sensors. The method can include identifying an application of a cosmetic formulation to the biological surface using the radiation sensors. The method can include generating a trace describing the application with reference to the baseline description. The method can also include outputting the trace.

[0008] In some embodiments, generating a trace includes receiving shape information describing a plurality of shape types, using the shape information to attribute a first shape type to at least a portion of the biological surface, generating a numerical representation of the application of the cosmetic formulation, the numerical representation describing position information relative to a baseline description, and converting the numerical representation from the first shape type to a second shape type. In some embodiments, identifying the application of the cosmetic formulation includes detecting an applicator of the cosmetic formulation, estimating a position of the applicator of the cosmetic formulation relative to the biological surface, tracking movement of the applicator relative to the biological surface, and generating a numerical representation of the movement relative to the baseline description. The method further includes estimating a surface tone of the biological surface, estimating a formulation tone of the cosmetic formulation, and determining a color of the cosmetic formulation using the surface tone and the formulation tone.

[0009] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram illustrating one embodiment of a system for extrapolating a cosmetic design, according to various embodiments.

[0011] [Figure 2] FIG. 2 is a schematic diagram illustrating an exemplary technique for preparing an augmented reality cosmetic design filter, according to various embodiments.

[0012] [Figure 3] FIG. 3 is a diagram illustrating an example technique for deploying an augmented reality cosmetic design filter, according to various embodiments.

[0013] [Figure 4] FIG. 4 is a diagram illustrating an example technique for converting an augmented reality cosmetic design filter from one shape type to another, according to various embodiments.

[0014] [Figure 5] FIG. 5 is a diagram illustrating an example technique for generating an augmented reality cosmetic design filter with multiple layers, according to various embodiments.

[0015] [Figure 6] FIG. 6 is a schematic diagram illustrating an exemplary technique for determining formulation color using surface tone and formulation tone, according to various embodiments.

[0016] [Figure 7] FIG. 7 is a block diagram illustrating an exemplary system including components of the system of FIG. 1, according to various embodiments.

[0017] [Figure 8] FIG. 8 is a block diagram illustrating aspects of an exemplary computing device, according to various embodiments.

[0018] In the above drawings, unless otherwise noted, like reference numerals refer to like parts throughout the various views. To simplify the drawings, if necessary, not all of the example elements have been labeled. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles described. The above-described aspects and many of the attendant advantages of this invention will be more readily appreciated as the same becomes better understood by reference to the following detailed description when taken in conjunction with the accompanying drawings. DETAILED DESCRIPTION OF THE INVENTION

[0019] Applying cosmetics and makeup into patterns and shapes can be difficult by hand. For example, complex designs and theatrical makeup are typically applied by qualified makeup professionals. Additionally, self-application is typically difficult for people with limited mobility or vision. Even relatively simple makeup designs can involve layering and blending multiple cosmetics, as well as application of cosmetics specific to the shape of the face and facial features. For example, makeup and other cosmetics are typically applied in a different configuration to an oval-shaped face than to a rectangular or oval-shaped face. Thus, simply mapping a static texture object, such as a texture mask, onto a surface cannot recreate the aesthetic effect of a makeup design applied to a surface. Furthermore, the exact sequence and extent of the actions that make up a makeup routine usually cannot be determined from an image of the final design. For this reason, makeup tutorials are typically presented as step-by-step guides or videos with images showing the multiple intermediate states leading to the final effect. Nevertheless, face-shape-specific and skin-tone-specific tutorials are more effective than general tutorials at demonstrating specific makeup designs. However, such an approach is time-consuming to implement and difficult to replicate without the appropriate tools and experience with the specific techniques and cosmetic formulations used.

[0020] Techniques are described for generating augmented reality cosmetic design filters from a cosmetic routine to be applied to a target body surface of a biological subject, such as the subject's face or other region of interest. The described embodiments employ an image sensor to define one or more 3D contour mappings of the target body surface. In the context of such applications, the described embodiments provide improved accuracy and greater ease of use compared to complex manual routines. Thus, the present technology improves the manual application of makeup through the distribution of dynamic augmented reality cosmetic design filters that can automatically (e.g., without human intervention) adapt to a person's shape type and skin tone. The filters can recreate a step-by-step cosmetic routine, including application traces, finish effects, and augmented reality renderings of specific tools, in a manner that is sensitive to the end user's shape type and skin tone of the augmented reality filter.

[0021] In one illustrative example, a smartphone user creates a video of a makeup routine to create a particular aesthetic effect. As part of the video creation, the smartphone captures images and surface mapping of the user's face. The initial images are used to generate a baseline from which subsequent applications of cosmetics, such as foundation, highlighter, filler, eye shadow, blush, and mascara, can be detected. As the video is created, the smartphone iteratively redefines the baseline as part of defining each new application as a new layer of the makeup design. Also, as the video is created, the smartphone recognizes new applications by detecting the user's hand movements, identifying new tools or formulations, and / or detecting color shifts in the user's face relative to the updated baseline. Over the course of the routine, multi-layer dynamic filters containing motion, texture, and / or transparency information are generated. The dynamic filters reference the facial shape of the makeup video creator and are stored in a distributed storage network. The filters are available to various users for virtual projection via an augmented reality interface. This filter can be used to recreate layering effects, realistically map makeup designs to various face shapes, and guide the viewer through a makeup routine in a manner specific to the viewer's face shape and skin tone, as described in the following paragraphs.

[0022] FIG. 1 is a schematic diagram illustrating one embodiment of an exemplary system 100 for generating an augmented reality cosmetic design filter 110, according to various embodiments. The exemplary system 100 includes one or more client computing devices 130, a camera 140, one or more remote computer systems 160 (also referred to as server(s)), and one or more user devices 190. As part of the exemplary system 100, components of the configuration system may be operatively coupled via wireless communication and / or via wired communication. In some embodiments, the components may communicate directly via wireless pairing (e.g., Bluetooth) and / or via a local wireless network (e.g., a wireless router). In some embodiments, the components may communicate via one or more networks 170. The one or more networks 170 may be or include a public network (e.g., the Internet) or a private network (e.g., an intranet). In this manner, the exemplary system 100 may include multiple separate devices configured to communicate electronic information via wireless connections. Additionally, or alternatively, some of the components may be integrated into a single device.

[0023] For clarity, the augmented reality cosmetic design filter 110 is also referred to as the cosmetic design 110. The cosmetic design 110 is a numerical representation of a cosmetic routine, including a set of textures, mapping data, surface information, and meta-information, stored in memory of the client computing device(s) 130, the server(s) 160, and / or the user device(s) 190. Rather than a single texture that reproduces the final result of the makeup routine on a particular face, projected onto a plane and then used as a texture mask, the cosmetic design 110 includes one or more traces 115 that reference a contour map of associated facial features 120 and / or baseline features 125 of the target area. The cosmetic design 110 may also include data describing dynamic elements 117 of the traces 115. In this manner, the cosmetic design 110 can be deployed on the user device 190 as an animated augmented reality filter that progressively reproduces the layered application of the cosmetic design 110.

[0024] As an illustrative example, client computing device 130 can incorporate camera 140. Similarly, exemplary system 100 can include multiple client computing devices 130. A first client computing device 130 is a mobile electronic device (e.g., a tablet, smartphone, or laptop) configured to host user interface elements and connect to server(s) 160 via network(s) 170. A second client computing device 130 is integrated with camera 140 and configured to coordinate operation of camera 140 with first client computing device 130. Components of exemplary system 100 can be provided with computer-executable instructions (e.g., software, firmware, etc.) for implementing and coordinating operation of one or more features of exemplary system 100. In this manner, a user interface can be accessed via one or more components, and operation of exemplary system 100 can be coordinated via the interface.

[0025] The client computing device(s) 130 may be or may include a purpose-built mobile computing device that includes a camera 140, one or more electromagnetic (EM) radiation sources (shown as part of the camera 140), and one or more user interface elements 131 for prompting the biological surface 180 with visual and / or audio prompts. For example, the interface element 131 may be or may include a display 133 for generating a visual representation of the cosmetic design 110. The interface element 131 may also include user input components, including, but not limited to, a touchscreen, keyboard, trackpad, or mouse. In this example, the components of the client computing device 130 may be integrated into a housing resembling a consumer cosmetic product, such as a bathroom or vanity mirror. In this example, the housing may conceal a power source, a thermal management system, and other components.

[0026] While client computing device 130 and camera 140 are illustrated in a particular configuration, additional and / or alternative form factors are contemplated. For example, system 100 may include a smartphone or tablet computer in communication with client computing device 130, causing one or more computer-executable operations to be performed by the smartphone or tablet computer. In this manner, image generation operations may be performed by a smartphone incorporating camera 140, and image processing operations may be performed, at least in part, on a separate device, such as server(s) 160.

[0027] In some embodiments, camera 140 is or can include multiple sensors and / or sources, including, but not limited to, a visible light image sensor 141, a depth sensor 143, and / or a source of invisible EM radiation 145, including, but not limited to, infrared or near-infrared. Like client computing device(s), camera 140 can include communications circuitry 147 to enable wireless communication and / or pairing with other components of exemplary system 100. While camera 140 is illustrated as a separate component of exemplary system 100, camera 140 can also be integrated into one of the other components of exemplary system 100. For example, client computing device 130 can incorporate camera 140 (e.g., as a smartphone or tablet).

[0028] The depth sensor 143 can capture one or more images of the biological surface 180, including, but not limited to, an image of a target body surface 181 of the biological surface 180. In the illustration provided in FIG. 1 , the biological surface 180 is a human using the exemplary system 100 to create the cosmetic design 110, and the target body surface 181 is the human's face in the area around the eyes and eyebrows. The depth sensor 143 can generate a surface mapping 183 of the target body surface 181. The contour and depth information of the target body surface 181 can change over time or between users. The camera can generate the surface mapping 183 as part of an operation for extrapolating, modifying, and / or generating the cosmetic design 110 by the client computing device(s) 130.

[0029] The depth sensor 143 may be an image sensor and may capture images within a field of view 185 that includes the target body surface 181. The depth sensor 143 may be or may include, but is not limited to, a laser-based sensor (e.g., LiDAR), a time-of-flight (ToF) camera, a structured light generator, a visual simultaneous localization and mapping (vSLAM) sensor assembly including a motion sensor and a visible image camera, or an ultrasound-based sensor assembly, such that the camera 140 can generate the surface mapping 183.

[0030] For example, if the depth sensor 143 is an infrared depth-sensing camera, the invisible EM radiation source 145 may be or include an infrared source that exposes the biological surface 180, including the target body surface 181, to structured invisible infrared radiation. In another example, if the depth sensor is a LiDAR system or a ToF camera, the invisible EM radiation source 145 may be or include an infrared diode laser. In this manner, the EM radiation generated by the invisible EM radiation source 145 may be scanned or otherwise directed toward the target body surface 181 over an angular spread 187 such that the target body surface 181 is exposed. In some embodiments, detection of the target body surface 181 is facilitated and / or enabled by feature and / or edge detection applied to a visible spectrum (e.g., RGB) image captured by the visible light sensor 141 (e.g., by vSLAM techniques).

[0031] The surface mapping 183 can provide contour and / or position information regarding features of the target body surface 181, such as the relative positions of the eyebrow ridge and the bridge of the nose, precise information regarding where the eyebrows begin and end in relation to the eyes, etc. In this manner, the surface mapping 183 can be used to generate and / or extrapolate cosmetic designs 110 for deployment on social media platforms or other services, and for distribution of the cosmetic designs 110 to users via user device(s) 190. In this manner, the cosmetic designs 110 can be represented as data files communicated to the server(s) 160, for example, as part of an online platform and / or database of cosmetic designs.

[0032] Surface mapping 183 can be used to determine the shape type of features on biological surface 180 and modify cosmetic design 110 by converting the cosmetic design 110 to another, different shape type. In some embodiments, multiple layers are defined as part of the process of creating or modifying cosmetic design 110. The exemplary system 100 can implement the exemplary techniques described with reference to FIGS. 2-6 to detect the application of multiple cosmetic formulations to target body surface 181 and generate one or more traces 115 that spatially and temporally describe the application, which together define cosmetic design 110.

[0033] Advantageously, the cosmetic design 110 can be distributed to the user device 190(s) with improved accuracy and precision by mapping both the target body surface 181 and the corresponding area of ​​the user of the user device 190 and extrapolating the cosmetic design 110, or each constituent layer of the cosmetic design 110, to fit the corresponding area, for example, as an augmented reality filter through integration with social media. Furthermore, the cosmetic design 110 can be dynamically presented in the form of a tutorial visualization, whereby both the layers and traces are animated to illustrate the layers, movement, color, and other cosmetic aspects of the cosmetic design 110. In this manner, the cosmetic design 110 represents an improved user experience by tailoring the cosmetic design 110 from the creator to the user, and also represents an improved system performance, since the complete cosmetic design 110 can be transferred as mask data rather than as image data. This reduces the amount of data and computational resources involved in distributing the cosmetic design 110 over the network(s) 170 and storing it on the server(s) 160.

[0034] FIG. 2 is a schematic diagram illustrating an exemplary process 200 for preparing an augmented reality cosmetic design filter 110, according to various embodiments. The exemplary process 200 may be implemented as a number of operations performed or otherwise performed by components of the exemplary system 100 of FIG. 1 . As such, the operations may be or may include operations performed by one or more processors of a computer system, such as the client computing device(s) 130 of FIG. 1 , in response to execution of computer-readable instructions stored on a non-transitory memory of the computer system. While the operations are illustrated in a particular order, the exemplary process 200 may include more or fewer operations, and the order of the operations may vary. In some embodiments, some operations are performed by different physical computers, as described in more detail with reference to FIG. 1 . For example, some operations may be performed interchangeably by different components or by the coordinated operation of two or more components.

[0035] In operation 201, a computer system defines a baseline description 210 of a biological surface 180. In some embodiments, the baseline description 210 is defined as part of generating a surface mapping 183, for example, by referencing visible color information against surface contour mapping information that describes the biological surface 180. In this manner, the baseline description 210 may include a numerical representation (e.g., digital data) of the biological surface 180 that includes both the surface mapping 183 and color information. While the exemplary process 200 is illustrated in the context of the target body surface 181 of FIG. 1 , it will be understood that the biological surface 180 can be or include a face, a hand, a portion of a face, or other surface to which a cosmetic formulation is applied.

[0036] In some embodiments, defining the baseline description 210 includes generating one or more images of the biological surface 180 using a camera 140 or other radiation sensor configured to detect visible light or invisible EM radiation. The invisible EM radiation can be projected onto the biological surface 180 using a radiation source on the camera 140, such as a structured light generator, a LiDAR light source, or a Time of Flight camera light source. For example, the images can depict facial features 120 and baseline features 125 on the biological surface 180. The baseline features 125 can be or include materials applied to the biological surface 180 prior to the exemplary process 200 such that at least a portion of the biological surface 180 includes pigments, cosmetic preparations, or other features detectable by the camera 140. For example, the baseline features 125 are illustrated as eyelashes and eyeliner. Similarly, the baseline features 125 can include, but are not limited to, blemishes, wrinkles, blemishes, or other aesthetic aspects of the biological surface 180 other than the facial features 120. In contrast, facial features 120 refer to organs, hair, and other topographical features of the biological surface 180 that define the surface mapping 183 and affect the baseline description 210 through physical effects including, but not limited to, shadows, perspective, etc.

[0037] In operation 203, the computer system identifies one or more applications 215 of cosmetic formulations 220(s) to the biological surface 180. The cosmetic formulations 220(s) may include, but are not limited to, makeup, mascara, lipstick, lip liner, eyeliner, glitter, beauty creams, ointments, or other materials that may be topically applied to the biological surface 180. The application 215(s) of the cosmetic formulations 220(s) can be detected by the camera 140 using a radiation sensor, such as a visible light sensor, an invisible EM radiation sensor, or the like. In this manner, the application 215(s) can be detected as a shift of the biological surface relative to a baseline description, for example, in terms of coloration, surface reflectance, or invisible EM radiation absorbance / reflectance. In an illustrative example, a glossy lip coating can be detected through an increase in specular reflectance of the biological surface 180 relative to a baseline description that includes negligible specular reflectance. Similarly, a matte foundation can be detected through a decrease in specular reflectance and an increase in diffuse reflectance relative to the baseline description. In a non-limiting example, specular reflection can be detected by defining a luminance channel in the visible light image and thresholding the luminance channel: specular reflection is defined as a luminance value that exceeds a threshold, on a pixel-by-pixel basis.

[0038] In some embodiments, the computer system detects an applicator 225 (e.g., a brush, sponge, pencil, finger, or other applicator 225) in proximity to the biological surface 180. In some embodiments, the computer system recognizes the identifier of the applicator 225, for example, if the applicator 225 includes a registration mark (e.g., a barcode, QR code, RFID tag, or other representation of identifier information). In this manner, the computer system can spatially and temporally identify the application 215 and identify the cosmetic formulation(s) 220. To this end, operation 203 can optionally include identifying the cosmetic formulation 220 and defining a color of the cosmetic formulation 220 using the identifier information of the cosmetic formulation 220. For example, the color of the unidentified cosmetic formulation 220 can be estimated using the camera 140, as described in more detail with reference to FIG. 6 . Identifying the applicator 225 may enable the computer system to directly determine the color of the cosmetic formulation 220 loaded with the particular cosmetic formulation 220, rather than whether the applicator 225 is a multi-purpose applicator 225. Examples of applicators 225 that may identify the cosmetic formulation 220 in this manner include, but are not limited to, pencils, lipsticks, gel applicators, or other applicators 225 available with the prepared cosmetic formulation 220. The application 215 of the cosmetic formulation 220 may include a movement 230 of the applicator 225 relative to the biological surface 180. The movement 230 may include meaningful information regarding the aesthetic effect of the overall cosmetic design 110, such as blending and smudge movements. In this manner, the movement 230 of the applicator 225 may be detected and used during operation to generate a trace.

[0039] The applications 215 may also include modifications or manipulations of formulations 220 already placed on the biological surface 180. For example, the coverage, color, and reflectance of the cosmetic formulation 220 may be modified by smoothing, blurring, blending, spreading, or other techniques. Such applications 215 that modify an already placed formulation 220 rather than adding a new formulation 220 to the biological surface 180 may be identified by detecting a tool or finger and detecting changes to the biological surface 180 relative to the baseline description 210. As described with reference to FIG. 1 , a baseline description 210 may be associated with each application 215, where the applications 215 are layered and / or blended. For example, the baseline description 210 for a particular area of ​​the biological surface 180 where formulation 220 has already been applied may be redefined to include the formulation 220. In this manner, subsequent applications 215 may be detected more accurately and precisely.

[0040] In operation 205, the computer system generates one or more traces 115 describing the applications 215 identified in operation 203. Generating the traces 115 may include determining the extent 217, layer order (see FIG. 5 ), and coloring (see FIG. 6 ) of the applications 215 so that the cosmetic design 110 can realistically reproduce the aesthetic effects of the applications 215 and also reproduce the order of the applications 215. The extent 217, coloring, and layer order information, as well as other information related to realistically reproducing the traces 115 on the user device(s) 190, may be encoded as a numerical representation, such as an intensity data set referenced to the baseline description 210. For example, the extent 217 of the applications 215, illustrated as dashed lines in FIG. 2 , may be detected as a shift in coloring, absorbance and / or reflectance of EM radiation compared to the baseline description. In this manner, the traces 115 can reproduce the aesthetic effects of the individual applications 215.

[0041] In some embodiments, the computer system tracks the applicator movement 230 using image and / or contour data generated by the camera 140. Thus, generating the trace 115 may include generating numerical representations of the various movements 230 relative to the biological surface 180, as described in the baseline description, and generating color information describing the cosmetic formulation 220 used in the application 215. Once generated, the trace 115 can be transformed to recreate the aesthetic effect of the cosmetic design 110 on various shapes of the biological surface 180 when viewed through user device(s) 190, as described in more detail with reference to FIG.

[0042] The traces 115 can encode layer blending and layering information so that combinations of multiple traces 115 can replicate interactions between the traces 115. For example, a set of applications 215 of multiple eyeshadow colors to a region of the biological surface 180 between the eye and eyebrow can be blended to impart a color gradient to the eyeshadow region. In this example, the exemplary process 200 includes multiple iterations of operations 201-205 such that the region between the eyebrow and eye is described by multiple traces 115 and also by interaction terms that describe how the multiple traces relate. In some embodiments, the traces 115 include transparency information, such as alpha channel information, referenced to dynamic elements 117 that describe the local transparency of the coloration or other aesthetic effects of the applications 215. In this manner, the traces 115 can replicate edits of the cosmetic formulation 220, such as blending, blurring, smoothing, or other techniques typical of cosmetic design. In one illustrative example, the trace 115 and / or the dynamic element 117 may include a vector of transparency values ​​referencing the movement 230 and the formulation 220 such that the trace 115 replicates the smoothing or blending of the formulation 220 based on the movement 230.

[0043] In operation 207, the computer system outputs the trace(s) 115 detected as part of operations 201-205. Outputting the trace(s) 115 may include electronic operations including, but not limited to, storing the trace(s) 115 in a local data store of the client computing device 130, transferring the trace(s) 115 to the server(s) 160 over the network 170, and / or sharing the trace(s) 115 with the user device(s) 190 directly (e.g., via electronic pairing) or over the network 170.

[0044] While the description of the exemplary process 200 focuses on cosmetic design for the brow / eye area, the operations may be applied to other surfaces as well. For example, the cosmetic design 110 may describe the application 215 of a cosmetic formulation 220 to additional / alternative surfaces, including, but not limited to, the lips, nose, cheeks, forehead, or hands. Similarly, the cosmetic design 110 may modify the appearance of facial features 120, including, but not limited to, the brows, eyes, lips, cheekbones, jawline, or hands. In another illustrative example, the cosmetic design 110 may include a series of tracings 115 to enhance the appearance of cheekbones through the application of one or more cosmetic formulations 220.

[0045] FIG. 3 is a schematic diagram illustrating an example process 300 for deploying an augmented reality cosmetic design filter 110, according to various embodiments. Similar to the example process 200 of FIG. 2, the example process 300 may be implemented as multiple operations performed or carried out by components of the example system 100 of FIG. 1. As such, the operations may be or may include operations performed by one or more processors of a computer system, such as the user device(s) 190 of FIG. 1, in response to execution of computer-readable instructions stored on a non-transitory memory of the computer system. While the operations are illustrated in a particular order, the example process 300 may include more or fewer operations, and the order of the operations may vary. In some embodiments, some operations are performed by different physical computers, as described in more detail with reference to FIG. 1. For example, some operations may be performed interchangeably by different components or may be performed by the coordinated operation of two or more components, such as the user device(s) 190 and the server(s) 160.

[0046] In operation 301, the user device 190 receives data 310 describing the cosmetic design 110. Receiving the data 310 can include electronic communication over the network 170 (e.g., the Internet, a cellular network, etc.), but can also include direct communication via pairing or other approaches. While FIG. 3 illustrates the user device 190 receiving the data 310 from the server(s) 160, it will be understood that the data 310 can be received from the client computing device(s) 130. The data 310 can include traces 115 generated during one or more iterations of the exemplary process 200, as described in more detail with reference to FIG. 2. The data 310 can also include meta-information describing the biological surface 180, such as the shape type of the biological surface 180, as described in more detail with reference to FIG. 4.

[0047] The operation for receiving the data 310 may include one or more data transfer techniques, including, but not limited to, wireless or wired communication. For example, the user device 190 may communicate wirelessly with the client computing device 130 to receive the data 310 as a wireless transmission.

[0048] In some embodiments, receiving the data 310 may involve a browser environment, such as a recommendation engine, allowing a user of the user device 190 to view a set of cosmetic designs 110 created by one or more creators, rather than directly communicating with the creators to request designs. Additionally or alternatively, the data 310 may be pushed to the user device 190, for example, as part of a social media service to which the user of the user device 190 may register, follow, and / or subscribe. In some embodiments, the data 310 is recommended based at least in part on identifier information describing the user of the user device 190. If the user of the user device 190 indicates aesthetic preferences through historical traffic data, cosmetic designs 110 that reflect these preferences may be preferentially recommended. Demographic, socioeconomic, or biometric data may also be used to recommend cosmetic designs 110. For example, if the user of the user device 190 resides in a certain geographic region, cosmetic designs 110 that reflect trends in this geographic region may be recommended. Similarly, if the user of user device 190 is within a given age range, employed in a given field or sector, or part of a given social network, cosmetic designs that reflect trends corresponding to the respective categories can be identified and recommended.

[0049] In operation 303, the user device 190 maps a user surface 315 corresponding to the biological surface 180 to which the cosmetic design 110 will be virtually applied. Mapping the user surface 315 may include operations similar to those described with respect to generating the surface mapping 183 of FIG. 1 , including but not limited to structured light methods, LiDAR or ToF methods, vSLAM methods, ultrasound methods, etc. The surface 315 may be described by a mesh of polygons 317, each of which may define multiple properties for a corresponding region of the surface 315, including but not limited to tone, reflectance, temperature, etc.

[0050] If the user device 190 is a smartphone incorporating a depth sensor 319, operation 303 may include executing one or more applications stored in memory of the user device 190 to map the surface using the depth sensor. In this manner, the user surface 315 may be represented as a numerical representation in data describing the surface contours and facial features 320 of the user surface 315. In some embodiments, the exemplary process 300 includes integrating the cosmetic design 110 with the existing aesthetic appearance of the user surface 315. To this end, aesthetic features 321, such as makeup, mascara, eyelashes, etc., are included in the surface mapping.

[0051] Mapping the user surface 315 may also include estimating a surface tone, which may be a local surface tone, as described in more detail with reference to FIG. 6. In this manner, the exemplary process 300 may account for differences in skin color between the biological surface 180 and the user surface 315. For example, the metadata provided as part of the data 310, or alternatively stored in memory of the server(s) 160 and / or the user device 190, may include a reference or lookup table that cross-references colors to skin colors so that the cosmetic design 110 can be adapted to reproduce the aesthetic effect of the cosmetic design 110, rather than a literal transposition of the exact colors.

[0052] In act 305, the trace 115 received as part of the data 310 is used to generate a filter 325 that is referenced in the mapping of the user surface 315 generated in act 303. The filter 325 may be or include a numerical representation, such as a texture object, formatted to be combined with the mapping of the user surface 315. In an illustrative example, the user surface 315 is described by a mesh of polygons 317, and the filter 325 references the polygons 317 such that each polygon 317 defines a vector of values ​​that describes the appearance of the filter 325.

[0053] In some embodiments, the filter 325 includes dynamic elements 330, such as animation effects or inter-layer effects, that can be used to indicate the sequence of applications 215 used to create the cosmetic design 110. The dynamic elements 330 can replicate at least a portion of the dynamic elements 117 described with reference to the trace 115 of FIGS.

[0054] In operation 307, the computer system presents a filter 325 integrated with an image 335 of the user surface 315. Presenting the filter 325 may include displaying the filter 325 through a screen 337. For example, the user device 190 may be a smartphone or tablet that includes a touchscreen display. The filter 325 may be mapped to the user surface 315 in real time or near real time using feature tracking or other augmented reality technology, where "near real time" refers to a qualitative experience by a user of the user device 190. This makes any latency introduced by the operation of the exemplary process 300 substantially imperceptible.

[0055] The filters 325 can be presented as interactive graphical elements that a user of the user device 190 can manipulate through the screen 337, for example, by selecting or highlighting. In some embodiments, selecting the filter 325 can initiate animation of the dynamic element 330 in an optional action 309. Additionally or alternatively, animating the filter 325 can include initiating a virtual tutorial, whereby the filters 325 are presented in layer order in a manner that replicates the order of the application 215 on the user surface 315. The order can be paused, reversed, advanced, or otherwise manipulated via the user device 190. A particular dynamic element can be activated, for example, by selecting the filter 325 with a user action 340 on the screen 337.

[0056] In some embodiments, interaction with the screen 337 while the filters 325 are presented allows the user of the user device 190 to edit, add, or delete one or more filters 325. Different user actions 340 can initiate one or more alternative processes instead of activating a dynamic element. For example, a double-finger tap can open a menu 341 where the user of the user device 190 can view the applicator 225, cosmetic formulation 220, or other information describing the filter 325. Similarly, a long press on the filter 325 can cause the user device 190 to access an online marketplace where the user can purchase the cosmetic formulation 220, applicator 225, or other consumer products. In some embodiments, a menu of social media controls 343 can be generated that allows the user of the user device to share the image 335 with the filter 325, communicate with other users, or provide feedback or otherwise communicate with the creator of the cosmetic design 110. It will be understood that the types of user actions described are exemplary. User interface elements such as menu 341 and / or social media menu 343 can be invoked by alternative and / or additional interactions. In some embodiments, menu elements are presented on display 337 by default.

[0057] FIG. 4 is a schematic diagram illustrating an example process 400 for converting an augmented reality cosmetic design filter 110 from one shape type to another, according to various embodiments. Similar to example processes 200 and 300, example process 400 may be implemented as a number of operations performed or otherwise performed by components of example system 100 of FIG. 1 . As such, the operations may be or include operations performed by one or more processors of a computer system, such as client computing device(s) 130, server(s) 160, and / or user device(s) 190 of FIG. 1 , in response to execution of computer-readable instructions stored on a non-transitory memory of the computer system. While the operations are illustrated in a particular order, example process 400 may include more or fewer operations, and the order of operations may vary. In some embodiments, some operations are performed by different physical computers, as described in more detail with reference to FIG. 1 . For example, some operations may be performed interchangeably by different components or by the coordinated operation of two or more components.

[0058] The trace 115 is generated with reference to a baseline description 210, as will be described in more detail with reference to FIG. 2, and is itself specific to the biological surface 180. Accordingly, the exemplary process 400 describes operations for extrapolating the filter 325 from a first facial shape to a second facial shape as part of generating the trace 115 and / or the filter 325. While the description with reference to FIG. 4 focuses on facial shapes, it will be appreciated that operations can similarly be applied to the shapes of individual facial features, such as the shape of the eyes, mouth, or other features, to transform the trace 115 to realistically reproduce the aesthetic effect of the trace 115, rather than simply applying the trace 115 as a texture to the user surface 315.

[0059] In operation 401, the computer system generates mask data 410 using one or more traces 115. If the traces 115 are relative to topographical data describing the biological surface, generating the mask data 410 includes projecting data describing the application 215 onto the topographical data describing the biological surface 180. This allows for the modification of the traces 115 from one shape type 420 to another shape type 420.

[0060] In operation 403, the computer system attributes a first shape type 420 to at least a portion of the biological surface 180. For example, the computer system may reference shape information describing the plurality of shape types 420. The shape information may be stored on, accessed by, or otherwise received from the server(s) 160 or the client computing device(s) 130. In this manner, the example process 400 may include receiving information describing the plurality of shape types 420.

[0061] Attributing a shape type 420 to the biological surface 180 may include referencing the baseline description 210 to determine characteristic dimensions, spacing, or other aspects of the biological surface 180 that are indicative of the shape type 420. In the illustrative example of a face shape, the characteristic aspects may be defined by projecting a grid or other form of guide 421 onto the baseline description 210 such that the shape type 420 may be identified by defining a contour 423 of the baseline description 210, by identifying a characteristic curvature 425 at one or more points of the baseline description 210, or by identifying a characteristic spacing 427 between features. It will be appreciated that approaches including edge detection and feature detection using computer image processing techniques may be employed to attributing the shape type 420.

[0062] In operation 405, the computer system generates one or more shape filters 430 for different shape types 420. As part of generating the shape filters 430, the mask data 410 generated in operation 401 is transformed from a first shape type 420-1 to a second shape type 420-2, a third shape type 420-3, or an additional or alternative shape type 420. Transforming the mask data 410 may include, but is not limited to, projecting the mask data 410 of the first shape type 420-1 onto topographical data of the second shape type 420-2 and applying a corresponding transformation to the trace 115. In this manner, the shape filters 430 reproduce the aesthetic shape and coloring of the trace 115 on a shape type 420 different from the first shape type 420-1.

[0063] In some embodiments, generating shape filters 430 for different shape types 420 may include additional modifications to the mask data 410 beyond spatial projection to account for different physical shapes. For example, some shape types 420 may be characterized by more pronounced or thinner facial features, such as brow ridges, forehead curvature, jaw shape, cheekbones, or chin shape. As such, the shape filter 430 may rearrange, split, omit, duplicate, mirror, or otherwise transform the mask data 410 as part of generating the shape filter 430. In one illustrative example, generating a shape filter 430 for a first shape type 420-1 corresponding to a heart-shaped face shape to a second shape type 420-2 corresponding to a rectangular face shape includes rearranging the forehead trace upward toward the hairline, extending the forehead trace to both ends of the forehead, and splitting and mirroring the chin trace along the jawline. A corresponding transformation can be applied to the mask data 410 for each trace 115 as part of generating a shape filter 430 for each shape type 420 .

[0064] In the context of facial features, if the cosmetic design 110 addresses a particular facial feature (e.g., eyeshadow design between the eyebrows and the eyes), the composition operations of the exemplary process 400 can be applied based on similar principles. For example, if the cosmetic design 110 is created with reference to a round eye shape type 420, generating a shape filter 430 in operation 405 can include converting the mask data 410 from the round eye to the almond eye shape type 420 or other specific eye shape type 420. It will be appreciated that corresponding operations can be applied to the shape of lips, hands, ears, or other biological surfaces.

[0065] FIG. 5 is a schematic diagram illustrating an exemplary process 500 for generating an augmented reality cosmetic design filter 510 in multiple layers, according to various embodiments. Similar to exemplary processes 200, 300, and 400, exemplary process 500 may be implemented as a number of operations performed or otherwise performed by components of exemplary system 100 of FIG. 1 . As such, the operations may be or include operations performed by one or more processors of a computer system, such as client computing device(s) 130, server(s) 160, and / or user device(s) 190 of FIG. 1 , in response to execution of computer-readable instructions stored on a non-transitory memory of the computer system. While the operations are illustrated in a particular order, exemplary process 500 may include more or fewer operations, and the order of operations may vary. In some embodiments, some operations are performed by different physical computers, as described in more detail with reference to FIG. 1 . For example, some operations may be performed interchangeably by different components or by the coordinated operation of two or more components.

[0066] As described in more detail with reference to Figures 1-3, the trace 115 can be generated by recording the sequential application of multiple cosmetic formulations 220 to the biological surface 180 such that the augmented reality cosmetic design filter 510 can be partitioned into separate layers. As described in more detail with reference to Figure 3, the presentation of the filter 510 can recreate the layered order of application. Similarly, the example process 400 can also incorporate the layered aspects of the example process 500, where the different shape types 420 represent different layering processes, inter-layer blending, and more complex light and shadow effects resulting from the differences in the shape types 420.

[0067] In acts 501-507, a series of layers are defined, each layer including one or more traces 115. For example, in act 501, a first layer is defined including a forehead trace 515 and a cheekbone trace 517. In act 503, a second layer is defined including a forehead trace 520(s), a cheek trace 521, and a chin trace 523(s). In act 505, a highlight layer(s) is defined that is placed over the first and second layers and includes highlight trace(s) 525 for highlighting or accentuating one or more facial features. In act 507, an eye layer(s) is defined that is placed over the highlight layer, first layer, and second layer and includes an eyeshadow trace 530 and an under-eye trace 531.

[0068] In this manner, the augmented reality cosmetic design filter 510 can dynamically add or remove layers during presentation on the user device 190. Similarly, the client computing device 130 can associate one or more traces 115 with each of multiple layers. Defining layers as described in the exemplary process 500 allows for blending rules to be defined between layers. For example, a higher-order layer can overlay a lower-order layer. A higher-order layer can completely obscure a lower-order layer. Transparency or partial transparency, defined, for example, through the alpha channel of the mask data 410, allows for multiple layers to be overlaid in the image 335 to create a net aesthetic effect that includes co-localized contributions from multiple layers.

[0069] FIG. 6 is a schematic diagram illustrating an exemplary process 600 for determining formulation color using surface tone and formulation tone, according to various embodiments.

[0070] 2, detecting the application 215 of the cosmetic formulation 220 can include determining the color or the cosmetic formulation 220 and / or identifying the cosmetic formulation 220. If the technique for determining color includes image processing operations based on images generated by the camera 140, the color of the various cosmetic formulations 220 can be affected by the tone of the biological surface 180 proximate to the application 215. As such, direct reading of image pixel values ​​can produce inaccurate color rendition when used in the filter 325.

[0071] In operation 601, a computer system measures multiple regions 610 corresponding to an application 215 and estimates a surface tone 615 of the biological surface 180 for each of the regions 610. In some embodiments, the operations of the exemplary process 600 are performed in parallel with the exemplary process 200, or may be performed at a different time using the baseline description 210, images recorded during generation of the trace 115, or a calibration set of images generated before starting the exemplary process 200. It is expected that the surface tone 615 may vary locally between regions 610, for example, where facial features cast shadows or where the surface defines one or more contours. To this end, operation 601 may include defining multiple surface tones 610 for a single application 215.

[0072] In operation 603, the computer system measures regions 610 in the image containing the application 215 and estimates one or more local formulation tones 625 of the cosmetic formulation 220. Estimating the surface tone 615 and formulation tone 625 may include sampling one or more pixels in each region 610 and determining an average coloration for each region 610. It will be appreciated that the formulation tone 625 is influenced by the underlying surface tone 615 such that the surface tone 615 can serve as a reference baseline value for estimating formulation tone in a skin-tone-sensitive manner. Advantageously, by implementing a computer image processing approach that includes estimating the surface tone 615 in this manner, the cosmetic design 110 can be responsive to the user's skin tone, rather than presenting a static augmented reality mask.

[0073] In operation 605, the computer system determines a color 630 of the cosmetic formulation using the surface tone 615 and the formulation tone 625. In contrast to the surface tone 615 and the formulation tone 625, it is contemplated that each cosmetic formulation 220 includes one or more pigments that are uniformly dispersed or distributed such that the formulation appears as a single visible color, for example, through diffuse reflection of visible light. To this end, the formulation color 630 can be or include a vector of color components, such as an RGB triad or another color representation approach.

[0074] As part of operation 605, corresponding surface tones 615 and formulation tones 625 can be compared, and the overall impact of the surface tone 615 on the formulation tone 625 can be considered as part of determining the color 630 of the cosmetic formulation 220. In an illustrative example, if different regions 610 are illuminated differently, comparing the surface tone 615 and the formulation tone 625 can control the impact of lighting on color rendition, for example, by using the surface tone 615 to correct for luminance variations in the formulation tone 625.

[0075] It will be appreciated that determining formulation color 630 can improve the reproducibility of the aesthetic effect of cosmetic design 110 on various biological surfaces, under various ambient conditions, and with various formulations. As described in more detail with reference to FIG. 2 , formulation color 630 can be modified to account for differences in skin color between creators and users. Similarly, formulation color 630 can be modified to improve realistic color fidelity based on ambient conditions. For example, under bright conditions, the color intensity can be increased, while under dim conditions, the color intensity can be decreased. In some cases, the color tone can also be dynamically modified to account for light and shape. For example, as part of exemplary process 300, formulation color 630 can be modified to include a greater blue component to reflect that the corresponding trace 115 is in shade. Conversely, the blue component can be decreased when the corresponding trace 115 is moved into bright lighting.

[0076] As explained throughout the foregoing description, it is contemplated that the operations for generating, modifying, and / or presenting the trace 115 and filter 325 are performed by one or more electronic devices including computer circuitry, communication modules, and computer-readable instructions stored in memory. Through the coordinated operation of the components of the example system 100 of FIG. 1 , augmented reality filters can be generated and presented in near real time to a user of a user device 190 integrated with a social media platform. The augmented reality filters can realistically recreate the aesthetic effect of a makeup design, taking into account differences in facial shape, skin tone, and ambient conditions. The augmented reality filters can be dynamically presented to enhance sharing of makeup routine sequences beyond what is currently possible with augmented reality or traditional social media.

[0077] FIG. 7 is a block diagram illustrating an exemplary system 700 including components of the system of FIG. 1, according to various embodiments. The exemplary system 700 includes a client computing device 130 in electronic communication with a remote computer system 160 (e.g., via a network 170). The exemplary system 700 illustrates an example of the system 100 of FIG. 1 in the context of related system elements, and as such, illustrates electronics and software for performing operations as described with reference to FIGS. 2-6. FIG. 7 illustrates a non-limiting example of system elements, features, and configurations; many other features and configurations are contemplated. In the example illustrated in FIG. 7, the client computing device 130 of FIG. 1 includes a computer system 710, multiple components 720 for interacting with a biological surface 180, a computer-readable medium 730, and a client application 740, which, when executed by the computer system 710, can implement operations described with reference to the system 100 of FIG. 1 and the exemplary techniques of FIGS. 2-3.

[0078] The client computing device 130 may incorporate auxiliary components, including, but not limited to, a power supply 711, a human-machine interface 713, one or more processors 715, a network interface 717, and may include a computer-readable medium 730. The power supply 711 is a DC power source, such as a rechargeable battery or a rectified power supply configured to connect to a line power source (e.g., 110V AC, 220V AC, etc.). The human-machine interface (HMI) 713 may include any type of device capable of receiving user input or generating output for presentation to a user, such as a speaker for audio output, a microphone for receiving voice commands, a push button switch, a toggle switch, a capacitive switch, a rotary switch, a slide switch, a rocker switch, or a touch screen.

[0079] The one or more processors 715 are configured to execute computer-executable instructions stored on the computer-readable medium 730. In one embodiment, the processor(s) 715 are configured to send signals to and / or receive signals from the component 720 via a communications bus or other circuitry, for example, as part of the execution of a client application 740. The network interface 717 is configured to send signals to and receive signals from the client computing device 130 (or other computing device) on behalf of the processor 715. The network interface 717 may implement any suitable communications technology, including, but not limited to, short-range wireless technologies such as Bluetooth, infrared, near-field communication, and Wi-Fi; long-range wireless technologies such as WiMAX, 2G, 3G, 4G, LTE, and 10G; and wired technologies such as USB, FireWire, Thunderbolt, and / or Ethernet. The computer-readable medium 730 may be any type of computer-readable medium capable of storing computer-executable instructions, including, but not limited to, flash memory (SSD), ROM, EPROM, EEPROM, and FPGA. The computer-readable medium 730 and the processor(s) 715 may be combined into a single device, such as an ASIC. Alternatively, the computer-readable medium 730 may include cache memory, registers, or another part of the processor 715.

[0080] In the illustrated embodiment, computer-readable medium 730 stores computer-executable instructions that, in response to execution by one or more processors 715, cause client computing device 130 to implement a control engine 731. Control engine 731 controls one or more aspects of client computing device 130, as described above. In some embodiments, the computer-executable instructions are configured to cause client computing device 130 to perform one or more operations, such as generating a surface mapping of a target surface, generating a trace, or outputting the trace to server(s) 160. In some embodiments, control engine 731 controls basic functionality by facilitating interaction between computer system 710 and components 720 according to client application 740. In some embodiments, the control engine 731 detects an input from the HMI 713 indicating that a makeup routine is to begin (e.g., in response to activation of a power switch or a "start" button, or in response to detection of a face in front of the camera 140 of FIG. 1), or receives a signal from the client computing device(s) 130, the remote computer system 160, or the user device(s) 190 (e.g., via a Bluetooth® paired connection).

[0081] Components of client computing device 130 can be tailored to an application or can be application (e.g., ASIC) specific. For example, components 720 can include one or more cameras 721, a display 723, one or more radiation sources 725, and / or one or more radiation sensors 727, as described in more detail with reference to FIG. 1 . In some embodiments, components 720 are integrated into a single device. In this manner, client computing device 130 can be a special-purpose computing device configured to execute client application 740 in conjunction with components 720. In some embodiments, client computing device 130 is a general-purpose mobile electronic device, such as a tablet or smartphone, that stores client application 740.

[0082] In some embodiments, client application 740 also includes an image capture / 3D scanning engine 741. Image capture / 3D scanning engine 741 is configured to capture and process digital images (e.g., color images, infrared images, depth images, etc.) obtained from one or more of components 720, including, but not limited to, stereo images, LiDAR data, or other forms of surface / depth sensing information. In some embodiments, such data is used to obtain a clean and accurate 3D contour mapping of the target body surface (e.g., target surface 181 of FIG. 1 ). In some embodiments, the digital images or scans are processed by client computing device 130 and / or transmitted to remote computer system 160 for processing by 3D model engine 781. In one embodiment, the captured image data is used by position tracking engine 743 to determine the location of key points, edges, or features on the target body surface. In some embodiments, position tracking engine 743 tracks the contours of the target body surface in 3D space, for example, by implementing v-SLAM techniques. In some embodiments, the position information from the position tracking engine 743 is used to generate a signal that is sent to the control engine 731. This signal is used to control one or more parts 720 or elements of the computer system 710, including, for example, the source 725 or the HMI 713, in accordance with the techniques described herein.

[0083] In some embodiments, the digital 3D models described herein are generated based on sensor data acquired by client computing device 130. As such, the digital 3D models may be generated by client computing device 130, other computing devices such as a cloud computing system, or a combination thereof. In some embodiments, the digital 3D models include 3D topology and texture information, which can be used to recreate accurate representations of body surfaces, such as facial structure and skin features, as described in more detail with reference to FIGS. 1-6.

[0084] In some embodiments, client application 740 includes a user interface 745. In one embodiment, user interface 745 includes interactive features including, but not limited to, graphical guides or prompts presented via a display to guide a user through selecting a cosmetic design, tutorial video, or animation. In some embodiments, user interface 745 provides guidance (e.g., visual guides such as arrows or targets, progress indicators, audio / haptic feedback, synthesized speech, etc.) to direct the user under particular lighting conditions, angles, etc. to ensure sufficient data is collected for use by the mapping and projection engine.

[0085] The client application 740 may include a mapping module 747. The mapping module 747 may be or may include computer-readable instructions (e.g., software, drivers, etc.) for converting a numerical representation of a cosmetic design into an augmented reality cosmetic design filter. As part of the operation of the mapping module 747, the client application 740 may receive real-time data from the camera(s) 721 and the sensor 727. This data may be processed by the 3D scanning engine 741, the position tracking engine 743, and the like. This data may be used to progressively update the mapping and cosmetic design developed during a series of applications to the target body surface. In this manner, the mapping module 747 may respond to movement of the target body surface, thereby increasing the client computing device 130's tolerance for user movement without losing fidelity to the cosmetic design filter. In some embodiments, the computational resource demands for such real-time scanning / tracking may be distributed across multiple devices, such as the remote computer system 160, via parallelization or distribution routines.

[0086] The communications module 749 of the client application 740 can be used to prepare information for transmission to, or receive and interpret information from, other devices or systems, such as the remote computer system 160 or user device(s) 190, as described in more detail with reference to FIG. 1 . Such information can include captured digital images, scans, or videos, personal care device settings, custom care routines, user preferences, user identifiers, device identifiers, etc. In one embodiment, the client computing device 130 collects data describing the performance of a care routine, body surface image data, or other data. In one embodiment, such data is transmitted via the network interface 717 to the remote computer system 160 for further processing or storage (e.g., in the product data store 783 or the user profile data store 785). The client computing device 130 can be used by a consumer, a personal care professional, or some other entity to interact with other components of the system 700, such as the remote computer system 160 or user device(s) 190. In one embodiment, client computing device 130 is a mobile computing device, such as a smartphone or tablet computing device, that includes component 720 and client application 740 or that includes component through electronic coupling with a peripheral device.

[0087] Exemplary components and functionality of remote computer system 160 are now described. Remote computer system 160 includes one or more server computers implementing one or more of the illustrated components, for example, in a cloud computing arrangement. Remote computer system 160 includes a projection engine 787, a 3D model engine 781, a product data store 783, and a user profile data store 785. In one embodiment, 3D model engine 781 uses image data (e.g., color image data, infrared image data) and depth data to generate a 3D model of a target body surface. The image data is obtained from client computing device 130, for example, from camera(s) 721 or sensor(s) 727 integrated with or otherwise electronically coupled to client computing device 130. In one embodiment, image data and depth data associated with a user are stored in user profile data store 785. In one embodiment, user consent is obtained before storing any information that is personal to the user or that could be used to identify the user.

[0088] In one embodiment, the mapping / projection engine 787 performs processing of data related to the makeup routine, such as generating a mapping of the target surface using image / sensor data and / or generating a projection of the makeup routine as an augmented reality filter. The data related to the makeup routine can then be transmitted to the user device(s) 190. In some embodiments, the projection engine 787 generates the makeup design data using user information from the user profile data store 785, the product data store 783, the 3D model engine 781, or other sources or combinations of sources.

[0089] The 3D model engine 781 may employ machine learning or artificial intelligence techniques (e.g., template matching, feature extraction and matching, classification, artificial neural networks, deep learning architectures, genetic algorithms, etc.). For example, to generate a cosmetic design related to a facial surface mapping, the 3D model engine 781 may analyze the facial mapping generated by the 3D model engine 781 to measure or map the contours, wrinkles, skin texture, etc. of a target body surface. The 3D model engine 781 may receive data describing the cosmetic design based on an identifier code provided by a user through user device(s) 190. In such a scenario, the 3D model engine 781 may use such information to generate an augmented reality cosmetic design filter projecting a cosmetic design (e.g., cosmetic design 110 of FIG. 1 ) onto an image of the user's corresponding body surface.

[0090] The devices shown in Figure 7 can communicate with each other via network 170. Network 170 can include any suitable communications technology, including, but not limited to, wired technologies such as DSL, Ethernet, fiber optic, USB, Firewire, Thunderbolt, etc.; wireless technologies such as WiFi, WiMAX, 3G, 4G, LTE, 5G, 10G, Bluetooth, etc.; and private networks (e.g., intranets) or public networks (e.g., the Internet). Generally, communications between computing devices or components of Figure 7, or other components or computing devices used in accordance with the described embodiments, occur directly or through intermediate components or devices.

[0091] Alternatives to the arrangements disclosed and described with reference to FIGS. 1 and 7 are possible. For example, functionality described as being implemented in multiple components may instead be integrated into a single component, and functionality described as being implemented in a single component may instead be implemented in multiple illustrated components or in other components not shown in FIGS. 1 or 7. As another example, devices in FIGS. 1 and 7 illustrated as including particular components may instead include more, fewer, or different components without departing from the scope of the described embodiments. As another example, functionality described as being performed by a particular device or subcomponent may instead be performed by one or more other devices in the system. As one example, the 3D model engine 714 may be implemented in the client computing device 130 or another device or combination of devices.

[0092] In addition to the technical advantages of the embodiments described elsewhere herein, some embodiments achieve numerous other technical advantages. For example, system 700 allows some aspects of the process to be performed independently by the personal care device or client computing device, while shifting other processing burdens to the remote computer system 160 (which may be a relatively powerful and reliable computing system). Thus, performance of the functionality provided by the personal care device or client computing device is improved and battery life is preserved.

[0093] Generally, as used herein, the term "engine" refers to logic embodied in hardware or software instructions written in a programming language such as C, C++, COBOL, JAVA, PHP, Perl, HTML, CSS, JavaScript, VBScript, ASPX, Microsoft .NET, etc. An engine can be compiled into an executable program or written in an interpreted programming language. Software engines can be called by other engines or by themselves. Generally, engines as described herein refer to logical modules that can be integrated with other engines or divided into sub-engines. Engines can be stored on any type of computer-readable medium or computer storage device and are stored and executed on one or more general-purpose computers. Thus, engines or special-purpose computers configured to provide this functionality are created.

[0094] As will be appreciated by those skilled in the art, a "data store" as described herein can be any suitable device configured to store data for access by a computing device. One example of a data store is a reliable, high-speed relational database management system (DBMS) running on one or more computing devices and accessible over a high-speed network. Another example of a data store is a key-value store. However, any other suitable storage technology and / or device capable of quickly and reliably providing stored data in response to a query can be used. The computing devices can be accessible locally rather than over a network, or can be provided as a cloud-based service. A data store can also include data systematically stored on a computer-readable storage medium, as described further below. Those skilled in the art will recognize that the separate data stores described herein can be combined into a single data store and / or the single data store described herein can be separated into multiple data stores without departing from the scope of the present disclosure.

[0095] FIG. 8 is a block diagram illustrating aspects of an exemplary computing device 800, according to various embodiments. While several different types of computing devices are described with reference to various embodiments, the exemplary computing device 800 illustrates various elements common to many different types of computing devices. While FIG. 8 is described with reference to a computing device implemented as a device on a network, the following description is applicable to servers, personal computers, mobile phones, smartphones, tablet computers, embedded computing devices, and other devices that can be used to implement some of the embodiments of the present disclosure. Furthermore, those skilled in the art and others will recognize that computing device 800 can be any one of any number of devices currently available or later developed.

[0096] In its most basic configuration, exemplary computing device 800 includes at least one processor 802 and a system memory 804 connected by a communications bus 806. Depending on the exact configuration and type of device, system memory 804 may be volatile or non-volatile memory, such as read-only memory (“ROM”), random-access memory (“RAM”), EEPROM, flash memory, or similar memory technologies. Those skilled in the art and others will recognize that system memory 804 typically stores data and / or program modules that are immediately accessible to and / or presently being operated on by processor 802. In this regard, processor 802 may function as the computational center of computing device 800 by supporting the execution of instructions.

[0097] As further shown in FIG. 8, computing device 800 may include a network interface 810 comprising one or more components for communicating with other devices over a network. Embodiments of the present disclosure may access basic services that utilize network interface 810 to implement communications using common network protocols. Network interface 810 may also include a wireless network interface configured to communicate via one or more wireless communication protocols, such as WiFi, 2G, 3G, LTE, WiMAX, Bluetooth, Bluetooth low energy, etc. As will be appreciated by those skilled in the art, network interface 810 illustrated in FIG. 8 may represent one or more wireless or physical communication interfaces described and illustrated above with respect to particular components of system 100 of FIG. 1.

[0098] In the exemplary embodiment depicted in Figure 8, computing device 800 also includes storage medium 808. However, services can be accessed using computing devices that do not include means for persisting data on local storage medium. Accordingly, storage medium 808 depicted in Figure 8 is represented by dashed lines to indicate that storage medium 808 is optional. In any event, storage medium 808 may be volatile or non-volatile, removable or non-removable, and implemented using any technology capable of storing information, including, but not limited to, a hard disk drive, a solid-state drive, a CD-ROM, a DVD, or other disk storage, a magnetic cassette, a magnetic tape, magnetic disk storage, and / or the like.

[0099] As used herein, the term "computer-readable medium" includes any method or technology capable of storing information such as computer-readable instructions, data structures, program modules, or other data, both volatile and nonvolatile, removable and non-implemented. In this regard, system memory 804 and storage media 808 illustrated in Figure 8 are merely examples of computer-readable media.

[0100] Suitable implementations of a computing device including a processor 802, system memory 804, communication bus 806, storage medium 808, and network interface 810 are known and commercially available. For ease of illustration and because they are not important to an understanding of the claimed subject matter, FIG. 8 does not show some of the components that are typical of many computing devices. In this regard, the exemplary computing device 800 may include input devices such as a keyboard, keypad, mouse, microphone, touch input device, touch screen, and / or the like. Such input devices may be coupled to the exemplary computing device 800 by wired or wireless connections, including RF, infrared, serial, parallel, Bluetooth®, Bluetooth low energy®, USB, or other suitable connection protocols using wireless or physical connections. Similarly, the exemplary computing device 800 may also include output devices, such as a display, speakers, printer, etc. These devices are well known in the art and therefore will not be further shown or described herein.

[0101] While exemplary embodiments have been shown and described, it should be understood that various changes to the exemplary embodiments can be made without departing from the spirit and scope of the present invention. It should be understood that the methods and systems described herein are not limited to specific methods, components, or implementations. It should also be understood that the terminology used herein is for the purpose of describing the embodiments only and is not intended to be limiting.

[0102] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Ranges can be expressed herein as from "about" one particular value and / or to "about" another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values ​​are expressed as approximations, it will be understood that the particular value forms another embodiment by use of the antecedent "about." It will further be understood that the endpoints of each range are significant both in relation to the other endpoint, and independently of the other endpoint.

[0103] "Optional" or "optionally" means that the subsequently described event or circumstance may or may not occur. The description is intended to include instances where the event or circumstance occurs and instances where it does not occur.

[0104] Throughout the description and claims of this specification, the word "comprise" and variations of words such as "comprising" and "comprises" mean "including, but not limited to." For example, other components, integers, or steps are not intended to be excluded. "Exemplary" means "one example of" and is not intended to convey an indication of a preferred or ideal embodiment. "Such as" is used for descriptive purposes, not in a limiting sense.

Claims

1. one or more non-transitory computer memory devices storing computer-readable instructions that, when executed by computing circuitry, cause the computing circuitry to perform operations for generating an augmented reality cosmetic design filter; The operation is defining a baseline description of the biological surface; Identifying an application for the cosmetic formulation; generating a trace; and outputting the trace; The step of defining a baseline description of the biological surface includes defining the baseline description of the biological surface using one or more radiation sensors; the radiation sensor is electronically coupled to the computing circuitry; the baseline description includes a numerical representation of the biological surface, including both surface mapping and color information; the step of identifying the application of the cosmetic preparation includes identifying the application of the cosmetic preparation to the biological surface using one or more of the radiation sensors; the application includes modifying the movement of an applicator or a cosmetic formulation placed on the biological surface; The step of generating the trace includes generating the trace describing the application with reference to the baseline description; the trace includes the extent, coloration, layer sequence, or reflectance of the cosmetic formulation applied or modified; the augmented reality makeup design filter is viewable through an augmented reality interface to guide a viewer through a makeup routine; The step of generating the trace comprises: receiving electronic information describing a plurality of shape types of faces and facial features; using the baseline description to attribute a first one of the shape types to at least a portion of the biological surface; generating a numerical representation of the digital data of said application of said cosmetic formulation; and converting the numerical representation from the first shape type to a second one of the shape types; One or more non-transitory computer memory devices, wherein the numerical representation describes position information relative to the baseline description.

2. 10. The one or more non-transitory computer memory devices of claim 1, The step of identifying the application of the cosmetic formulation comprises: one or more non-transitory computer memory devices, comprising detecting the type of the cosmetic formulation using one or more of the radiation sensors and the baseline description.

3. 10. The one or more non-transitory computer memory devices of claim 1, the biological surface is a face, one or more non-transitory computer memory devices, wherein the first shape type and the second shape type describe facial features.

4. 10. The one or more non-transitory computer memory devices of claim 1, one or more non-transitory computer memory devices, wherein defining the baseline description of the biological surface comprises projecting invisible electromagnetic radiation onto the biological surface using a radiation source electronically coupled to the computing circuitry.

5. 10. The one or more non-transitory computer memory devices of claim 1, one or more non-transitory computer memory devices, wherein the one or more radiation sensors comprise a camera configured to detect electromagnetic radiation from the ultraviolet, visible, or infrared spectral ranges, or a combination thereof.

6. 10. The one or more non-transitory computer memory devices of claim 1, The step of generating the trace comprises: tracking the movement of the application relative to the biological surface; and generating a numerical representation of the motion relative to the baseline description.

7. 10. The one or more non-transitory computer memory devices of claim 1, The instructions, when executed by the computing circuitry, cause the computing circuitry to perform further operations, the operations including: identifying the cosmetic formulation; and one or more non-transitory computer memory devices, comprising: defining a color of the cosmetic formulation using identifier information of the cosmetic formulation.

8. 10. The one or more non-transitory computer memory devices of claim 1, The instructions, when executed by the computing circuitry, cause the computing circuitry to perform further operations, the operations including: estimating the surface tone of the biological surface; estimating a formulation tone of the cosmetic formulation using the surface tone; and one or more non-transitory computer memory devices, comprising determining a color of the cosmetic formulation using the surface tone and the formulation tone.

9. 10. The one or more non-transitory computer memory devices of claim 1, The one or more non-transitory computer memory devices are electronically coupled with a smartphone.

10. 10. The one or more non-transitory computer memory devices of claim 1, One or more non-transitory computer memory devices, wherein outputting the trace comprises communicating the trace to a remote computing system.

11. 1. A method for generating an augmented reality cosmetic design filter, comprising: defining a baseline description of the biological surface using one or more radiation sources and one or more radiation sensors; using the radiation sensor to identify application of a cosmetic formulation to the biological surface; generating a trace describing the application with reference to the baseline description; and outputting the trace; the baseline description includes a numerical representation of the biological surface, including both surface mapping and color information; the application includes modifying the movement of an applicator or a cosmetic formulation placed on the biological surface; the trace includes the extent, coloration, layer sequence, or reflectance of the cosmetic formulation applied or modified; the augmented reality makeup design filter is viewable through an augmented reality interface to guide a viewer through a makeup routine; The step of generating the trace comprises: receiving electronic shape information describing a plurality of shape types of a face or facial features; using the electronic shape information to attribute a first shape type from among the shape types to at least a portion of the biological surface; generating a numerical representation of the digital data of said application of said cosmetic formulation; converting the numerical representation from the first shape type to a second shape type; The method, wherein the numerical representation describes position information relative to the baseline description.

12. 12. The method of claim 11, The step of identifying the application of the cosmetic formulation comprises: detecting an applicator of the cosmetic preparation; estimating the position of the applicator of the cosmetic formulation relative to the biological surface; tracking the movement of the applicator relative to the biological surface; and generating a numerical representation of the motion relative to the baseline description.

13. 12. The method of claim 11, estimating the surface tone of the biological surface; estimating the formulation tone of the cosmetic formulation; and The method further comprises determining a color of the cosmetic formulation using the surface tone and the formulation tone.

Images