Facial Expression Prediction Using Image-Based Movement Metrics

Abstract

Techniques are disclosed for training a machine learning model and predicting user facial expressions. A plurality of images are received, each of the plurality of images containing at least a portion of a user's face. A plurality of values ​​for a movement metric are calculated based on the plurality of images, each of the plurality of values ​​for the movement metric indicating a movement of the user's face. A plurality of values ​​for expression units are calculated based on the plurality of values ​​for the movement metric, each of the plurality of values ​​for the expression units corresponding to a range in which the user's face is producing the expression unit. The machine learning model is trained using the plurality of images and the plurality of values ​​for the expression units.

Description

[Technical Field]

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63,012,579, filed April 20, 2020, and entitled "EXPRESSION PREDICTION USING IMAGE-BASED MOVEMENT METRIC," the entire contents of which are incorporated herein by reference for all purposes. [Background technology]

[0002] Modern computing and display technology has facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or can be perceived as real. Virtual reality, or "VR," scenarios typically involve the presentation of digital or virtual image information without transparency to other actual, real-world visual input. Augmented reality, or "AR," scenarios typically involve the presentation of digital or virtual image information as an extension to the user's visualization of the real world around them.

[0003] Despite the advances made in these display technologies, there remains a need in the art for improved methods, systems, and devices relating to augmented reality systems, and particularly display systems. Summary of the Invention [Means for solving the problem]

[0004] The present disclosure relates generally to techniques for improving the performance and user experience of optical systems. More specifically, embodiments of the present disclosure provide systems and methods for predicting user facial expressions based on images of the user's face. While the present invention is often described with reference to augmented reality (AR) devices, the present disclosure is applicable to a variety of applications.

[0005] A description of various embodiments of the invention is provided below as a list of examples. As used below, any reference to a series of examples shall be understood as a disjunctive reference to each of those examples (e.g., "Examples 1-4" shall be understood as "Examples 1, 2, 3, or 4").

[0006] Example 1 is a method for training a machine learning model and predicting a user's facial expression, the method including the steps of: receiving a plurality of images, each of the plurality of images containing at least a portion of a user's face; calculating a plurality of values ​​for a movement metric based on the plurality of images, each of the plurality of values ​​for the movement metric indicating movement of the user's face; calculating a plurality of values ​​for an expression unit based on the plurality of values ​​for the movement metric, each of the plurality of values ​​for the expression unit corresponding to a range in which the user's face is producing the expression unit; training the machine learning model using the plurality of images and the plurality of values ​​for the expression units by generating training output data based on the plurality of images by the machine learning model; and modifying the machine learning model based on the plurality of values ​​for the expression units and the training output data.

[0007] Example 2 is the method of example 1, wherein the training output data includes a plurality of output values ​​for each set of facial units, and the facial unit is a first facial unit from the set of facial units.

[0008] Example 3 is the method of example 2, wherein the set of facial expression units includes at least one of inner brow raise, outer brow raise, brow lower, upper eyelid raise, cheek raise, eyelid tension, nose wrinkle, eye closure, left eye wink, or right eye wink.

[0009] Example 4 is a method described in Examples 1-3, wherein the step of training a machine learning model using a plurality of images and a plurality of values ​​for facial expression units further includes a step of performing a comparison between the plurality of values ​​for facial expression units and a plurality of output values ​​for facial expression units of the training output data, and a step of generating error data based on the comparison, wherein the machine learning model is corrected based on the error data.

[0010] Example 5 is the method of Examples 1-4, wherein the machine learning model is an artificial neural network having a set of adjustable parameters.

[0011] Example 6 is a method described in Examples 1-5, wherein the movement metric is the number of eye pixels, and the step of calculating multiple values ​​for the movement metric based on the multiple images includes the steps of segmenting each of the multiple images so that each of the multiple images includes eye pixels and non-eye pixels, counting the number of eye pixels in each of the multiple images, and setting each of the multiple values ​​for the movement metric to be equal to the number of eye pixels in a corresponding image from the multiple images.

[0012] Example 7 is a method as described in Examples 1-6, wherein the step of calculating a plurality of values ​​for the facial expression unit based on a plurality of values ​​for the movement metric includes the steps of identifying a first extremum in the plurality of values ​​for the movement metric and a first corresponding image for which the first extremum was identified, setting each of the plurality of values ​​for the facial expression unit associated with the first corresponding image to be equal to one, identifying a second extremum in the plurality of values ​​for the movement metric and a second corresponding image for which the second extremum was identified, setting each of the plurality of values ​​for the facial expression unit associated with the second corresponding image to be equal to zero, and setting each remaining value of the plurality of values ​​by interpolating between zero and one.

[0013] Example 8 is a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations including receiving a plurality of images, each of the plurality of images containing at least a portion of a user's face; calculating a plurality of values ​​for a movement metric based on the plurality of images, each of the plurality of values ​​for the movement metric indicating movement of the user's face; calculating a plurality of values ​​for an expression unit based on the plurality of values ​​for the movement metric, each of the plurality of values ​​for the expression unit corresponding to a range in which the user's face is producing the expression unit; training a machine learning model using the plurality of images and the plurality of values ​​for the expression units by generating training output data based on the plurality of images by the machine learning model; and modifying the machine learning model based on the plurality of values ​​for the expression units and the training output data.

[0014] Example 9 is the non-transitory computer-readable medium of example 8, wherein the training output data includes a plurality of output values ​​for each set of facial units, and the facial unit is a first facial unit from the set of facial units.

[0015] Example 10 is the non-transitory computer-readable medium of example 9, wherein the set of facial expression units includes at least one of inner brow raise, outer brow raise, brow lower, upper eyelid raise, cheek raise, eyelid tension, nose wrinkle, eye closure, left eye wink, or right eye wink.

[0016] Example 11 is the non-transitory computer-readable medium of Examples 8-10, wherein the step of training the machine learning model using the plurality of images and the plurality of values ​​for the facial expression units further includes the steps of: performing a comparison between the plurality of values ​​for the facial expression units and the plurality of output values ​​for the facial expression units of the training output data; and generating error data based on the comparison, wherein the machine learning model is corrected based on the error data.

[0017] Example 12 is the non-transitory computer-readable medium of any of Examples 8-11, wherein the machine learning model is an artificial neural network having a set of adjustable parameters.

[0018] Example 13 is a non-transitory computer-readable medium described in Examples 8-12, wherein the movement metric is the number of eye pixels, and the step of calculating multiple values ​​for the movement metric based on the multiple images includes the steps of segmenting each of the multiple images so that each of the multiple images includes eye pixels and non-eye pixels, counting the number of eye pixels in each of the multiple images, and setting each of the multiple values ​​for the movement metric to be equal to the number of eye pixels in a corresponding image from the multiple images.

[0019] Example 14 is a non-transitory computer-readable medium described in Examples 8-13, wherein the step of calculating a plurality of values ​​for the facial expression unit based on a plurality of values ​​for the movement metric includes the steps of identifying a first extremum in the plurality of values ​​for the movement metric and a first corresponding image for which the first extremum was identified; setting each of the plurality of values ​​for the facial expression unit associated with the first corresponding image to be equal to one; identifying a second extremum in the plurality of values ​​for the movement metric and a second corresponding image for which the second extremum was identified; setting each of the plurality of values ​​for the facial expression unit associated with the second corresponding image to be equal to zero; and setting each remaining value of the plurality of values ​​by interpolating between zero and one.

[0020] Example 15 is a system comprising: one or more processors; and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: receiving a plurality of images, each of the plurality of images containing at least a portion of a user's face; calculating a plurality of values ​​for a movement metric based on the plurality of images, each of the plurality of values ​​for the movement metric indicating movement of the user's face; calculating a plurality of values ​​for an expression unit based on the plurality of values ​​for the movement metric, each of the plurality of values ​​for the expression unit corresponding to a range in which the user's face is producing the expression unit; training a machine learning model using the plurality of images and the plurality of values ​​for the expression units by generating training output data based on the plurality of images using a machine learning model; and modifying the machine learning model based on the plurality of values ​​for the expression units and the training output data.

[0021] Example 16 is the system described in Example 15, wherein the training output data includes a plurality of output values ​​for each set of facial expression units, and the facial expression unit is a first facial expression unit from the set of facial expression units.

[0022] Example 17 is the system described in Example 16, wherein the set of facial expression units includes at least one of inner eyebrow raise, outer eyebrow raise, eyebrow lower, upper eyelid raise, cheek raise, eyelid tension, nose wrinkle, eye closure, left eye wink, or right eye wink.

[0023] Example 18 is a system described in Examples 15-17, wherein the step of training a machine learning model using a plurality of images and a plurality of values ​​for facial expression units further includes a step of performing a comparison between the plurality of values ​​for facial expression units and a plurality of output values ​​for facial expression units in the training output data, and a step of generating error data based on the comparison, wherein the machine learning model is corrected based on the error data.

[0024] Example 19 is a system described in Examples 15-18, wherein the movement metric is the number of eye pixels, and the step of calculating multiple values ​​related to the movement metric based on the multiple images includes the steps of segmenting each of the multiple images so that each of the multiple images includes eye pixels and non-eye pixels, counting the number of eye pixels in each of the multiple images, and setting each of the multiple values ​​related to the movement metric to be equal to the number of eye pixels in a corresponding image from the multiple images.

[0025] Example 20 is a system described in Examples 15-19, wherein the step of calculating multiple values ​​for the facial expression unit based on multiple values ​​for the movement metric includes the steps of identifying a first extremum in the multiple values ​​for the movement metric and a first corresponding image for which the first extremum was identified, setting each of the multiple values ​​for the facial expression unit associated with the first corresponding image to be equal to 1, identifying a second extremum in the multiple values ​​for the movement metric and a second corresponding image for which the second extremum was identified, setting each of the multiple values ​​for the facial expression unit associated with the second corresponding image to be equal to zero, and setting each remaining value of the multiple values ​​by interpolating between zero and 1.

[0026] Numerous benefits over conventional techniques are achieved by the methods of the present disclosure. For example, the embodiments described herein enable prediction of a user's facial expressions using only a portion of the user's face, which has useful applications in head-mounted systems such as AR systems. The embodiments described herein further enable training of machine learning models to predict user expressions that are tailored to be user-specific or that can be used by any user. For example, the machine learning model can be initially trained for all users, and then further calibration and fine-tuning of the training can be performed by the end user upon receipt of the device, prior to each use of the device, and / or periodically based on the user's needs. [Brief explanation of the drawings]

[0027] The accompanying drawings, which are included to provide a further understanding of the present disclosure, are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure, and together with the detailed description, serve to explain the principles of the present disclosure. No attempt is made to show structural details of the present disclosure in more detail than may be necessary for a fundamental understanding of the disclosure and various ways in which it may be practiced.

[0028] [Figure 1] FIG. 1 illustrates example instructions and corresponding movement metric values ​​that may be detected to train a machine learning model and predict user facial expressions.

[0029] [Figure 2] 2A and 2B illustrate an exemplary calculation of facial expression unit values ​​based on movement metric values.

[0030] [Figure 3] 3A and 3B illustrate an exemplary calculation of facial expression unit values ​​based on movement metric values.

[0031] [Figure 4] 4A and 4B illustrate an exemplary calculation of facial expression unit values ​​based on movement metric values.

[0032] [Figure 5A] FIG. 5A illustrates an exemplary system in which a machine learning model is operating in training mode.

[0033] [Figure 5B] FIG. 5B illustrates an exemplary system in which a machine learning model is operating in runtime mode.

[0034] [Figure 6] FIG. 6 illustrates an example implementation in which the number of eye pixels in the image is used as the movement metric.

[0035] [Figure 7A] FIG. 7A illustrates example movement metric values ​​for the example implementation of FIG.

[0036] [Figure 7B] FIG. 7B illustrates exemplary facial expression unit values ​​for the movement metric values ​​shown in FIG. 7A.

[0037] [Figure 8] FIG. 8 illustrates a method for training a machine learning model to predict user facial expressions.

[0038] [Figure 9] FIG. 9 illustrates a schematic diagram of an exemplary wearable system.

[0039] [Figure 10] FIG. 10 illustrates a simplified computer system. DETAILED DESCRIPTION OF THE INVENTION

[0040] Detailed Description of Specific Embodiments Predicting a user's facial expressions is useful in a variety of applications. For example, the ability to detect a user's facial expressions (and therefore the user's emotional state) can enable a computing system to communicate with a user based on the user's perceived needs, thereby allowing the computing system to offer relevant information to the user. In an augmented reality (AR) or virtual reality (VR) context, detecting user facial expressions can facilitate the animation of avatars and other digital characters. For example, the facial expressions produced by a user's avatar in the digital world may be in immediate response to the user's facial expressions in the real world.

[0041] While much previous work has focused on predicting a user's facial expression based on imaging the entire user's face, prediction based on images of only a portion of the user's face, such as the user's eyes, is substantially more complex. For example, a facial expression may cause only subtle changes in the eyes, while changes to other parts of the user's face, such as the user's mouth, may be more noticeable. These subtle changes may be difficult to detect and difficult to link to a specific user expression. In many applications, particularly AR / VR applications, in which eye-tracking cameras are employed, there is a significant need for a robust method for predicting a user's facial expression based on images of only a portion of the user's face, given the camera's limited field of view.

[0042] Embodiments described herein provide systems and methods for training a machine learning model and predicting user facial expressions. Specifically, the machine learning model may be trained to generate a set of decimal values ​​representing different facial movements when provided with an input image of a user's face (e.g., the user's eyes). The different facial movements may be referred to as expression units, and the values ​​generated by the machine learning model for each expression unit may be referred to as expression unit values. In some instances, the expression unit values ​​may each range from zero to one, with zero corresponding to the user producing the minimum range of expression units and one corresponding to the user producing the maximum range of expression units.

[0043] In some embodiments, the different facial expression units may be Facial Action Coding System (FACS) action units, which is a widely used classification standard for facial movements. Each FACS action unit corresponds to a different contraction or relaxation of one or more muscles in the user's face. A combination of action units may contribute to a user exhibiting a particular emotion. For example, a user may exhibit a "happy" emotion when the user raises their cheeks (action unit 6) and raises the corners of their mouths (action unit 12). As another example, a user may exhibit a "sad" emotion when the user raises their inner eyebrows (action unit 1), lowers their eyebrows (action unit 4), and lowers the corners of their mouths (action unit 15).

[0044] To train a machine learning model using a series of images, the set of expression unit values ​​generated for each image is compared to ground truth data, which may include a different set of expression unit values ​​(or a single expression unit value) calculated based on a movement metric for the series of images. To distinguish between the two sets of expression unit values, the values ​​generated by the machine learning model may be referred to as output values. For each image, error data may be generated by comparing the output values ​​with the expression unit values ​​calculated using the movement metric values ​​for the image. The machine learning model is then modified using the error data, for example, by adjusting weights associated with the machine learning model, to generate more accurate output values ​​during subsequent estimations.

[0045] 1 illustrates exemplary instructions 101 for training a machine learning model and predicting user facial expressions, and corresponding movement metric values ​​102 that may be detected, according to some embodiments of the present invention. The instructions 101 may be provided to a user to instruct the user to produce one or more facial expression units. While the user is producing the facial expression units, a camera captures an image of the user's face (or a portion thereof, such as the user's eyes). The captured image is analyzed to extract a movement metric value 102 for a particular movement metric that is associated with the user's face.

[0046] In some examples, a user is wearing an AR / VR headset. The headset may include a camera with a field of view that includes at least a portion of the user's face, such as one or both of the user's eyes. Such a camera may be referred to as an eye-tracking camera, which is often employed in AR / VR headsets. In some examples, the camera may capture an image of the user's entire face while the user is provided with instructions, and the image may be cropped to reduce the image to a desired area, such as the user's eyes. Alternatively, the camera may capture an image of the desired area directly by focusing or zooming on the user's eyes. Thus, embodiments of the present invention may include scenarios in which a user is wearing a head-mounted device and in which the user is not.

[0047] 1 as written instructions, instructions 101 may also include, among other possibilities, audible instructions played via speakers within the AR / VR headset or via remote speakers, visual instructions displayed within the AR / VR headset or on a remote display device. For example, during a calibration step of the AR / VR headset, the headset may generate virtual content showing either written instructions or examples of virtual characters demonstrating different facial expressions. When a user sees these visual instructions, they may then produce the indicated facial expressions.

[0048] In the illustrated example, the user is first provided with an instruction to "repeatedly perform facial unit 1." In this example, "facial unit 1" may correspond to raising the inner brow. Images of the user's face are then captured while the user repeatedly produces the inner brow raise a certain number of times. The images are analyzed to detect movement metric values ​​102, which indicate the movement of the user's face while they are producing the inner brow raise. The movement metric values ​​102 are analyzed to determine maximum and minimum values ​​(and their corresponding timestamps T, respectively). max ~T min ) may be identified.

[0049] timestamp T max ~T min may be used to identify images of interest and produce ground truth values ​​for training machine learning models. For example, if the motion metric value 102 is a relative maximum (at timestamp T max ), the corresponding image may be when the user is fully producing the inner brow raise, and the movement metric value 102 reaches a relative minimum (timestamp T min ) may be when the user is producing a minimal amount of inner brow lift, and the image in between may be when the user is producing a partial inner brow lift. Accordingly, different facial expression unit values ​​may be calculated based on the movement metric values ​​102. For example, an facial expression unit value of 1 may be calculated for a relative maximum movement metric value, an facial expression unit value of zero may be calculated for a relative minimum movement metric value, and an facial expression unit value between zero and 1 may be interpolated (e.g., linearly) between the maximum and minimum movement metric values.

[0050] Continuing with the illustrated example, the user is next provided with instructions to "repeatedly perform facial unit 2." In this example, "facial unit 2" may correspond to an eyebrow lowering. While the user then repeatedly produces the eyebrow lowering a number of times, an image of the user's face is captured, which is analyzed to detect movement metric values ​​102, from which maximum and minimum values ​​and corresponding timestamps T are obtained. max ~T min In contrast to the inner eyebrow lift, the movement metric value 102 reaches a relative minimum (timestamp T min ), the corresponding image at time stamp T max ) may be when the user is producing the least amount of brow lowering. Thus, an expression unit value of 1 may be calculated for the relative minimum movement metric value, and an expression unit value of zero may be calculated for the relative maximum movement metric value.

[0051] The user is then provided with instructions to "repeatedly perform facial unit 3." In this example, "facial unit 3" may correspond to eyelid tension. While the user then repeatedly produces eyelid tension a certain number of times, images of the user's face are captured, which are analyzed to detect movement metric values ​​102, from which maximum and minimum values ​​and corresponding timestamps T are determined. max ~T min is identified. Finally, the user is provided with an instruction to "repeatedly perform facial unit 4." In this example, "facial unit 4" may correspond to lifting the upper eyelid. The user then repeatedly produces the lifting of the upper eyelid a certain number of times while images of the user's face are captured, which are analyzed to detect movement metric values ​​102, from which maximum and minimum values ​​and corresponding timestamps T max ~T min is identified.

[0052] 2A and 2B illustrate an exemplary calculation of facial expression unit values ​​204 based on movement metric values ​​202, according to some embodiments of the present invention. As shown in FIG. 2A, the relative maximum 208 and relative minimum 210 values ​​are first calculated based on the movement metric values ​​202 and their corresponding timestamps (T1, T4, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17, T18, T19, T20, T21, T22, T23, T24, T25, T26, T27, T28, T29, T30, T31, T32, T33, T34, T35, T36, T37, T38, T39, T40, T41, T42, T43, T44, T45, T46, T47, T4 10 , T 13 , T 16 , and T 19 ) are identified. To avoid over-identification of relative extrema, a constraint may be imposed that successive extrema have at least a certain separation (e.g., a certain amount of time or number of frames). An upper threshold 212 may then be set at a predetermined distance below each of the relative maxima 208, and a lower threshold 214 may be set at a predetermined distance above each of the relative minima 210. When the movement metric values ​​202 are within the upper thresholds 212 (T3, T5, T9, T 11 , T 15 , and T 17 ) and lower threshold 214 (T2, T6, T8, T 12 , T 14 , and T 18 ) can be identified.

[0053] 2B , the facial expression unit values ​​204 may then be calculated by setting the value equal to 1 at the timestamp where the relative maximum 208 is identified and / or the timestamp where the movement metric value 202 crosses an upper threshold 212. The facial expression unit values ​​204 may also be set equal to zero at the timestamp where the relative minimum 210 is identified and / or the timestamp where the movement metric value 202 crosses a lower threshold 214. The remaining values ​​of the facial expression unit values ​​204 may be linearly interpolated. For example, the facial expression unit values ​​204 for T2-T3 may be linearly interpolated from zero to 1, and the facial expression unit values ​​204 for T5-T6 may be linearly interpolated from 1 to zero, or equivalent.

[0054] In some embodiments, an interpolation scheme other than linear interpolation may be employed. For example, a non-linear interpolation scheme may be employed, in which the facial expression unit value is calculated based on the nearest movement metric value, as follows: M(T i ) and E(T i ) at time T i For the metric movement values ​​and facial expression unit values ​​in T1, T2, and T3, the facial expression unit values ​​may be interpolated between zero and one as defined by the following equations: [ka] Similarly, the facial expression unit values ​​of T5-T6 may be interpolated between 1 and 0 as defined by the following equation: [ka]

[0055] 3A and 3B illustrate exemplary calculations of facial expression unit values ​​304 based on movement metric values ​​302, according to some embodiments of the present invention. In contrast to Figures 2A and 2B, the facial expressions in Figures 3A and 3B are those in which the smallest movement metric values ​​occur when the user is fully producing the facial expression unit, and the largest movement metric values ​​occur when the user is minimally producing the facial expression unit.

[0056] As shown in FIG. 3A, the relative maximum 308 and the relative minimum 310 are first calculated based on the movement metric values ​​302 and their corresponding timestamps (T1, T4, T7, T 10 , T 13 , T 16 , and T 19 2A , an upper threshold 312 may be set at a predetermined distance below each of the relative maxima 308, and a lower threshold 314 may be set at a predetermined distance above each of the relative minima 310. When the movement metric values ​​302 are compared with the upper thresholds 312 (T2, T6, T8, T 12 , T 14 , and T 18 ) and lower threshold 314 (T3, T5, T9, T 11 , T 15 , and T 17 ) can be identified.

[0057] 3B, the facial expression unit value 304 may then be calculated by setting the value equal to zero at the timestamp where the relative maximum value 308 is identified and / or the timestamp where the movement metric value 302 crosses an upper threshold 312. The facial expression unit value 304 may be set equal to one at the timestamp where the relative minimum value 310 is identified and / or the timestamp where the movement metric value 302 crosses a lower threshold 314. The remaining values ​​of the facial expression unit value 304 may be linearly interpolated.

[0058] 4A and 4B illustrate an exemplary calculation of facial expression unit values ​​404 based on movement metric values ​​402, according to some embodiments of the present invention. The scheme employed in FIG. 4A is a simplified approach in which the thresholds described in FIGS. 2A and 3A are not used. Similar to FIGS. 2A and 2B, the facial expressions in FIGS. 4A and 4B are those in which the maximum movement metric values ​​occur when the user is fully producing the facial expression unit, and the minimum movement metric values ​​occur when the user is producing the minimum amount of the facial expression unit. As shown in FIG. 4A, the relative maximum 408 and relative minimum 410 are calculated based on the movement metric values ​​402 and their corresponding timestamps (T1, T4, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17, T18, T19, T20, T21, T22, T23, T24, T25, T26, T27, T28, T29, T30, T31, T32, T33, T34, T35, T36, T37, T38, T39, T40, T41, T42, T43, T44, T45, T46, T47, T48, T49, T50, T51, T52, T53, T54, T55, T56, T57, T58, T59, T60, T61, T62, T63, T64, T65, T66, T67, T68, T69, T70, T71, T72, T73, T74, T75, T76, T77, T78, T79, T80, T81, T82, T83, T84, T85, T86, T8 10 , T 13 , T 16 , and T 19 ) are identified.

[0059] 4B, the facial expression unit value 404 may then be calculated by setting the value at the timestamp where the relative maximum value 408 is identified equal to 1, and setting the value at the timestamp where the relative minimum value 410 is identified equal to zero. The remaining values ​​of the facial expression unit value 404 are T1-T4, T7-T8, T9-T10, T11-T12, T13-T14, T14-T15, T15-T16, T16-T17, T17-T18, T18-T19, T19-T20, T210-T22, T220-T23, T23-T24, T24-T25, T25-T30, T26-T31, T27-T32, T28-T33, T29-T44, T34-T45, T35-T46, T36-T47, T37-T48, T38-T49, T39-T40, T40-T41, T41-T42, T42-T43, T43-T44, T44-T45, T45-T50, T46-T51, T47-T52, T48-T60, T49-T61, T49-T71, T49-T82, T41-T93, T41-T104, T41-T111, T41-T122, T41-T131, T41-T142, T41-T151, T41-T162, T41-T173, T41-T184, T41-T194, T42-T232, T42-T44, T43-T45, T44-T46 10 , and T 13 ~T 16 In this case, zero to one, T4 to T7, T 10 ~T 13 , and T 16 ~T 19 It is calculated by linearly or nonlinearly interpolating between 1 and zero.

[0060] 5A illustrates an exemplary system 500A in which a machine learning model 550 is operating in training mode, according to some embodiments of the present invention. System 500A includes an image capture device 505 configured to capture an image 506 of a user's face. Image 506 is received and processed by image processors 508A and 508B. Image processor 508A calculates a value 502 for a movement metric 510. Movement metric 510 may be constant during the training process or may be varied for different facial expression units. Value 502 for movement metric 510 is transmitted from image processor 508A to image processor 508B, which calculates a value 504 for an facial expression unit 514 based on value 502 for movement metric 510.

[0061] The images 506 and values ​​504 for the facial units 514 may form training input data 518. During the training process, each image 506 may be sequentially fed to the machine learning model 550 along with the image's corresponding facial unit values ​​from values ​​504. In response to receiving the images, the machine learning model 550 may generate an output value in output values ​​522 for each set of N facial units 520. The output values ​​for the same facial units as the facial units 514 are compared with the corresponding values ​​from values ​​504 to generate error data 524. Weights associated with the machine learning model 550 are then modified (e.g., adjusted) based on the error data 524.

[0062] As an example, during a first training iteration, a first image from images 506 may be provided to machine learning model 550, which may generate N output values ​​522 (one output value for each of N facial units 520). In some embodiments, each of the N output values ​​may be a decimal value between zero and one. The output value 522 for the facial unit 520 that is identical to facial unit 514 is compared to a first value from values ​​504 (representing ground truth) that corresponds to the first image to generate error data 524. In some embodiments, the output values ​​522 for the remaining facial units 520 are also used to generate error data 524, thereby allowing machine learning model 550 to learn that these output values ​​522 should be zero. Weights associated with machine learning model 550 are then modified based on the error data 524.

[0063] Continuing with the example, during a second training iteration after the first training iteration, a second image from images 506 may be provided to machine learning model 550, which may generate N output values ​​522 (one output value for each of N facial units 520). The output value 522 for the facial unit 520 that is identical to facial unit 514 is compared to a second value from value 504 that corresponds to the second image to generate error data 524 (optionally, output values ​​522 for the remaining facial units 520 are also used to generate error data 524). Weights associated with machine learning model 550 are then modified based on the error data 524.

[0064] This process continues until all of the images 506 have been used in the training process. During the training process, the facial units 514 may be varied, as needed, to cause different output values ​​522 to be selected and used in generating the error data 524. The machine learning model 550 may thus "learn" to predict the extent to which a user is producing each of the N facial units 520 based on a single image.

[0065] FIG. 5B illustrates an exemplary system 500B in which a machine learning model 550 is operating in runtime mode, according to some embodiments of the present invention. During runtime, an image capture device 505 captures and provides an image 506 to the machine learning model 550, which generates an output value 522 for each facial expression unit 520, resulting in N output values ​​522. While a single input image 506 is shown in FIG. 5B , in some embodiments, multiple input images may be provided to increase the accuracy of the machine learning model 550. For example, one or more previous or subsequent images to image 506 may be provided to the machine learning model 550 along with image 506, generating a single set of N values ​​522. In such embodiments, the training process may similarly utilize multiple input images during each training iteration.

[0066] FIG. 6 illustrates an example implementation in which the number of eye pixels in an image is used as a movement metric, according to some embodiments of the present invention. In the illustrated example, a left image 602A and a right image 602B of a user's eye are captured using an image capture device. The images 602 are each segmented into eye pixels 606 and non-eye pixels 608 (alternatively referred to as non-skin pixels and skin pixels, respectively), as shown in eye segmentation 604. The eye pixels 606 may be further segmented into different regions of the eye, including the sclera, iris, and pupil. In some embodiments, an additional machine learning model may be used to generate the eye segmentation 604. Such a machine learning model may be trained using labeled images provided by a user, where the user manually identifies the eye pixels 606 and non-eye pixels 608, as well as different regions of the eye.

[0067] FIG. 7A illustrates exemplary movement metric values ​​for the exemplary implementation of FIG. 6, in which the number of eye pixels in the image is used as the movement metric, according to some embodiments of the present invention. In the illustrated example, data for both the left and right eyes is superimposed. The curve shows the number of eye pixels (or non-skin pixels) over a period of time. In some embodiments, movement metric values ​​corresponding to "strong expressions" (e.g., the user producing the maximum range of facial expressions) may be identified automatically or manually. Automatic identification may be performed by identifying extreme values, optionally thresholds as described with reference to FIGS. 1-4B. Manual identification may be accomplished by the user selecting a particular frame through a graphical interface. In addition, movement metric values ​​corresponding to "neutral expressions" (e.g., the user producing the minimum range of facial expressions) may also be identified automatically or manually.

[0068] 7B illustrates exemplary expression unit values ​​for the motion metric values ​​shown in FIG. 7A. The expression unit values ​​are calculated by setting values ​​equal to 1 for frames (images) in which a strong expression is identified (optionally, frames for which the motion metric value is within a threshold distance), and setting values ​​equal to zero for frames in which a neutral expression is identified (optionally, frames for which the motion metric value is within a threshold distance). The remaining expression unit values ​​are linearly or nonlinearly interpolated between zero and one, as described with reference to FIGS. 2-4B.

[0069] 8 illustrates a method 800 for training a machine learning model (e.g., machine learning model 550) and predicting user facial expressions according to some embodiments of the present invention. One or more steps of method 800 may be omitted during implementation of method 800, and the steps of method 800 need not be performed in the order presented. One or more steps of method 800 may be performed or facilitated by one or more processors.

[0070] In step 802, a plurality of images are received (e.g., images 506, 602). The plurality of images may be received from an image capture device (e.g., image capture device 505), which may capture and transmit the plurality of images to a processing module. One or more of the plurality of images may be grayscale images, multi-channel images (e.g., RGB images), among other possibilities. In some embodiments, the image capture device may be an eye-tracking camera mounted on a wearable device. Each of the plurality of images may contain at least a portion of the user's face. For example, each of the plurality of images may contain the user's eye.

[0071] In step 804, multiple values ​​(e.g., values ​​102, 202, 302, 402, 502) for a movement metric (e.g., movement metric 510) are calculated based on the multiple images. The movement metric may be a measurement indicative of (or from which analysis may indicate) movement of the user's face. For example, the movement metric may be the number of eye pixels in the image, the number of non-eye pixels in the image, the distance between the top and bottom of the eye, the distance between the left and right sides of the eye, the position of a point along the eye in the image, the gradient of the image, among other possibilities.

[0072] For embodiments in which the movement metric is a number of eye pixels, calculating a plurality of values ​​for the movement metric may include segmenting each of the plurality of images such that each of the plurality of images includes eye pixels (e.g., eye pixels 606) and non-eye pixels (e.g., non-eye pixels 608), counting the number of eye pixels in each of the plurality of images, and setting each of the plurality of values ​​for the movement metric equal to the number of eye pixels in a corresponding image from the plurality of images. Segmenting the images from the plurality of images may result in an eye segmentation (e.g., eye segmentation 604).

[0073] In step 806, a plurality of values ​​(e.g., values ​​204, 304, 404, 504) for an facial unit (e.g., facial unit 514) are calculated based on the plurality of values ​​for the movement metric. Each of the plurality of values ​​for an facial unit may correspond to a range over which a user (e.g., the user's face) is producing the facial unit. In some embodiments, a larger value may correspond to a greater range over which the user is producing the facial unit.

[0074] In some embodiments, calculating the plurality of values ​​for the facial expression units may include identifying extreme values ​​(maximum and / or minimum values) within the plurality of values ​​for the movement metric. In one example, a first extreme value (e.g., maximum value) within the plurality of values ​​for the movement metric is identified along with a first corresponding image for which the first extreme value is identified. Each of the plurality of values ​​for the facial expression units associated with the first corresponding image may be set equal to one. Further, a second extreme value (e.g., minimum value) within the plurality of values ​​for the movement metric may be identified along with a second corresponding image for which the second extreme value is identified. Each of the plurality of values ​​for the facial expression units associated with the second corresponding image may be set equal to zero. Further, each remaining value of the plurality of values ​​may be set equal to a value between zero and one through interpolation.

[0075] In step 808, a machine learning model is trained using the plurality of images and the plurality of values ​​for the expression units. In some embodiments, step 808 includes one or both of steps 810 and 812.

[0076] In step 810, training output data (e.g., training output data 526) is generated based on the plurality of images. The training output data may include a plurality of output values ​​(e.g., output value 522) for each set of facial units (e.g., facial unit 520). An facial unit may be one of a set of facial units. The set of facial units may include one or more of the following: inner brow raise, outer brow raise, brow lower, upper eyelid raise, cheek raise, eyelid tensing, nose wrinkle, eye close, left eye wink, and right eye wink. The set of facial units may be FACS action units, such that an facial unit may be one of the FACS action units.

[0077] In some embodiments, training the machine learning model using the plurality of images and the plurality of values ​​for the facial units further includes performing a comparison between the plurality of values ​​for the facial units and the plurality of output values ​​for the facial units. In some embodiments, error data (e.g., error data 524) may be generated based on the comparison. For example, the error data may be generated by subtracting the plurality of values ​​for the facial units from the plurality of output values ​​for the facial units (or vice versa). The error data may be set equal to the magnitude of the difference, the sum of the magnitudes of the difference, or the sum of the squares of the differences, among other possibilities. In general, the error data may indicate the difference between the plurality of values ​​for the facial units and the plurality of output values ​​for the facial units.

[0078] In step 812, the machine learning model is modified based on the plurality of values ​​for the facial expression units and the training output data. Modifying the machine learning model may include adjusting one or more parameters (e.g., weights and / or biases) associated with the machine learning model. For example, the machine learning model may be an artificial neural network with a plurality of adjustable parameters that is used to calculate a set of output values ​​for the set of facial expression units based on an input image.

[0079] In some embodiments, the machine learning model may be modified based on the error data. In some embodiments, the range to which parameters associated with the machine learning model are adjusted may be related to (e.g., proportional to) the magnitude of the error data, such that a larger difference between the multiple values ​​for the expression units and the multiple output values ​​for the expression units leads to a larger modification to the machine learning model. In some embodiments, the machine learning model may be modified after multiple training iterations. For example, each training iteration may consist of training the machine learning model using a single input image from the multiple images and its corresponding value for the expression unit from the multiple values ​​for the expression units.

[0080] 9 illustrates a schematic diagram of an exemplary wearable system 900 that may be used in one or more of the above-described embodiments, according to embodiments of the present invention. The wearable system 900 may include a wearable device 901 and at least one remote device 903 that is remote from the wearable device 901 (e.g., separate hardware but communicatively coupled). While the wearable device 901 is worn by a user (generally as a headset), the remote device 903 may be held by the user (e.g., as a handheld controller) or mounted in various configurations, such as fixedly attached to a frame, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user (e.g., in a backpack configuration, in a belt-coupled configuration, etc.).

[0081] The wearable device 901 may include a left eyepiece 902A and a left lens assembly 905A arranged in a side-by-side configuration and comprising a left optical stack. The left lens assembly 905A may include an accommodation lens on the user side of the left optical stack and a compensatory lens on the world side of the left optical stack. Similarly, the wearable device 901 may include a right eyepiece 902B and a right lens assembly 905B arranged in a side-by-side configuration and comprising a right optical stack. The right lens assembly 905B may include an accommodation lens on the user side of the right optical stack and a compensatory lens on the world side of the right optical stack.

[0082] In some embodiments, the wearable device 901 includes one or more sensors including, but not limited to, a left front-facing world camera 906A mounted directly on or near the left eyepiece 902A, a right front-facing world camera 906B mounted directly on or near the right eyepiece 902B, a left-side facing world camera 906C mounted directly on or near the left eyepiece 902A, a right-side facing world camera 906D mounted directly on or near the right eyepiece 902B, a left eye tracking camera 926A pointed towards the left eye, a right eye tracking camera 926B pointed towards the right eye, and a depth sensor 928 mounted between the eyepieces 902. The wearable device 901 may include one or more image projection devices, such as a left projector 914A optically coupled to the left eyepiece 902A and a right projector 914B optically coupled to the right eyepiece 902B.

[0083] The wearable system 900 may include a processing module 950 for collecting, processing, and / or controlling data within the system. Components of the processing module 950 may be distributed between the wearable device 901 and the remote device 903. For example, the processing module 950 may include a local processing module 952 on the wearable portion of the wearable system 900 and a remote processing module 956 that is physically separate from and communicatively coupled to the local processing module 952. The local processing module 952 and the remote processing module 956 may each include one or more processing units (e.g., a central processing unit (CPU), a graphics processing unit (GPU), etc.) and one or more storage devices, such as non-volatile memory (e.g., flash memory).

[0084] The processing module 950 may collect data captured by various sensors of the wearable system 900, such as the camera 906, the eye tracking camera 926, the depth sensor 928, the remote sensor 930, the ambient light sensor, a microphone, an inertial measurement unit (IMU), an accelerometer, a compass, a global navigation satellite system (GNSS) unit, a wireless device, and / or a gyroscope. For example, the processing module 950 may receive images 920 from the camera 906. Specifically, the processing module 950 may receive a left front image 920A from a left front-facing world camera 906A, a right front image 920B from a right front-facing world camera 906B, a left side image 920C from a left side-facing world camera 906C, and a right side image 920D from a right side-facing world camera 906D. In some embodiments, the images 920 may include a single image, a pair of images, a video consisting of a stream of images, a video consisting of a stream of paired images, and the like. Images 920 may be generated and transmitted to processing module 950 periodically while wearable system 900 is powered on, or may be generated in response to instructions transmitted by processing module 950 to one or more of the cameras.

[0085] The cameras 906 may be configured at various positions and orientations along the exterior of the wearable device 901 to capture images of the user's surroundings. In some instances, the cameras 906A and 906B may be positioned to capture images that substantially overlap with the FOVs of the user's left and right eyes, respectively. Thus, the placement of the cameras 906 may be near the user's eyes, but not so close as to obscure the user's FOV. Alternatively, or in addition, the cameras 906A and 906B may be positioned to align with the internal coupling locations of the virtual image lights 922A and 922B, respectively. The cameras 906C and 906D may be positioned to capture images to the side of the user, e.g., within or outside the user's peripheral vision. The images 920C and 920D captured using the cameras 906C and 906D do not necessarily overlap with the images 920A and 920B captured using the cameras 906A and 906B.

[0086] In some embodiments, the processing module 950 may receive ambient light information from an ambient light sensor. The ambient light information may indicate a brightness value or a range of spatially resolved brightness values. The depth sensor 928 may capture a depth image 932 in a direction facing the front of the wearable device 901. Each value in the depth image 932 may correspond to the distance between the depth sensor 928 and the nearest detected object in a particular direction. As another example, the processing module 950 may receive eye tracking data 934 from the eye tracking camera 926, which may include left and right eye images. As another example, the processing module 950 may receive projected image brightness values ​​from one or both of the projectors 914. The remote sensor 930 located in the remote device 903 may include any of the sensors described above with similar functionality.

[0087] Virtual content is delivered to a user of the wearable system 900 using the projector 914 and the eyepieces 902, along with other components in the optical stack. For example, the eyepieces 902A, 902B may each comprise a transparent or semi-transparent waveguide configured to direct and outcouple light generated by the projectors 914A, 914B. Specifically, the processing module 950 may cause the left projector 914A to output left virtual image light 922A onto the left eyepiece 902A and the right projector 914B to output right virtual image light 922B onto the right eyepiece 902B. In some embodiments, the projector 914 may include a microelectromechanical system (MEMS) spatial light modulator (SLM) scanning device. In some embodiments, the eyepieces 902A, 902B may each comprise multiple waveguides corresponding to different colors. In some embodiments, lens assemblies 905A, 905B may be coupled to and / or integrated with eyepieces 902A, 902B. For example, lens assemblies 905A, 905B may be incorporated into a multi-layer eyepiece and may form one or more layers that make up one of eyepieces 902A, 902B.

[0088] FIG. 10 illustrates a simplified computer system 1000 according to embodiments described herein. Computer system 1000 as illustrated in FIG. 10 may be incorporated into a device as described herein. FIG. 10 provides a schematic illustration of one embodiment of computer system 1000 that may perform some or all of the steps of the methods provided by various embodiments. Note that FIG. 10 is intended only to provide a generalized illustration of various components, any or all of which may be utilized as desired. FIG. 10 therefore broadly illustrates a situation in which individual system elements may be implemented in a relatively separated manner or a relatively more integrated manner.

[0089] Computer system 1000 is shown to comprise hardware elements that may be electrically coupled via a bus 1005, or may otherwise communicate as needed. The hardware elements may include one or more processors 1010, including one or more general-purpose processors and / or one or more special-purpose processors, such as, but not limited to, digital signal processing chips, graphics acceleration processors, and / or the like, one or more input devices 1015, which may include, but are not limited to, a mouse, keyboard, camera, and / or the like, and one or more output devices 1020, which may include, but are not limited to, a display device, printer, and / or the like.

[0090] Computer system 1000 may further include and / or communicate with one or more non-transitory storage devices 1025, which may comprise, but are not limited to, local and / or network-accessible storage devices and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices such as random access memory ("RAM"), and / or read-only memory ("ROM"), which may be programmable, flash-updateable, and / or the like. Such storage devices may be configured to implement any suitable data storage, including, but not limited to, various file systems, database structures, and / or the like.

[0091] Computer system 1000 may also include a communications subsystem 1019, which may include, but is not limited to, a modem, a network card (wireless or wired), an infrared communications device, a wireless communications device, and / or a chipset, such as, but not limited to, a Bluetooth® device, an 802.11 device, a WiFi device, a WiMax device, a cellular communications facility, etc., and / or the like. Communications subsystem 1019 may include one or more input and / or output communications interfaces, allowing data to be exchanged with networks, such as those described below by way of example, other computer systems, televisions, and / or any other devices described herein. Depending on desired functionality and / or other implementation concerns, a portable electronic device or similar device may communicate images and / or other information via communications subsystem 1019. In other embodiments, a portable electronic device, e.g., a first electronic device, may be incorporated into computer system 1000, e.g., an electronic device, as input device 1015. In some embodiments, computer system 1000 further comprises working memory 1035, which may include RAM or ROM devices as described above.

[0092] Computer system 1000 may also include software elements shown as currently residing in working memory 1035, including an operating system 1040, device drivers, executable libraries, and / or other code, such as one or more application programs 1045, that may comprise computer programs provided by various embodiments and / or that may be designed to implement methods and / or configure systems provided by other embodiments as described herein. By way of example only, one or more procedures described with respect to the methods discussed above may be implemented as code and / or instructions executable by a computer or a processor within a computer; in certain aspects, such code and / or instructions can then be used to configure and / or adapt a general-purpose computer or other device to perform one or more operations in accordance with the described methods.

[0093] These sets of instructions and / or code may be stored on a non-transitory computer-readable storage medium, such as storage device 1025 described above. In some cases, the storage medium may be incorporated within a computer system, such as computer system 1000. In other embodiments, the storage medium is separate from the computer system, e.g., a removable medium such as a compact disc, and / or may be provided in an installation package such that the storage medium can be used to program, configure, and / or adapt a general-purpose computer with the instructions / code stored thereon. These instructions may take the form of executable code that is executable by computer system 1000 and / or may take the form of source and / or installable code that, upon compilation and / or installation on computer system 1000 using, for example, any of various commonly available compilers, installation programs, compression / decompression utilities, etc., then takes the form of executable code.

[0094] It will be apparent to those skilled in the art that substantial variations may be made according to particular requirements. For example, customized hardware may also be used, and / or particular elements may be implemented in hardware, software, including portable software such as applets, or both. Furthermore, connection to other computing devices, such as network input / output devices, may also be employed.

[0095] As noted above, in one aspect, some embodiments may employ a computer system, such as computer system 1000, to perform methods according to various embodiments of the present technology. According to one set of embodiments, some or all of the procedures of such methods are performed by computer system 1000 in response to processor 1010 executing one or more sequences of one or more instructions, which may be embedded in operating system 1040, and / or other code, such as application program 1045, contained in working memory 1035. Such instructions may be read into working memory 1035 from another computer-readable medium, such as one or more of storage devices 1025. By way of example only, execution of a sequence of instructions contained in working memory 1035 may cause processor 1010 to perform one or more procedures of the methods described herein. Additionally or alternatively, some of the methods described herein may be performed through specialized hardware.

[0096] The terms “machine-readable medium” and “computer-readable medium,” as used herein, refer to any medium that participates in providing data that causes a machine to operate in a tangible fashion. In an embodiment implemented using computer system 1000, various computer-readable media may be involved in providing instructions / code to processor 1010 for execution and / or may be used to store and / or carry such instructions / code. In many implementations, computer-readable media are physical and / or tangible storage media. Such media may take the form of non-volatile or volatile media. Non-volatile media include, for example, optical and / or magnetic disks, such as storage device 1025. Volatile media include dynamic memory, such as, but not limited to, working memory 1035.

[0097] Common forms of physical and / or tangible computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read instructions and / or code.

[0098] Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to processor 1010 for execution. By way of example only, the instructions may initially be carried on a magnetic and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions as signals over a transmission medium to be received and / or executed by computer system 1000.

[0099] The communications subsystem 1019 and / or its components generally receive signals and the bus 1005 may then convey the signals and / or the data, instructions, etc. carried by the signals to the working memory 1035, from which the processor 1010 retrieves and executes the instructions. The instructions received by the working memory 1035 may optionally be stored on a non-transitory storage device 1025 either before or after execution by the processor 1010.

[0100] The methods, systems, and devices discussed above are examples. Various configurations may omit, substitute, or add various procedures or components, as appropriate. For example, in alternative configurations, the methods may be performed in a different order than described, and / or various steps may be added, omitted, and / or combined. Also, features described with respect to one configuration may be combined in various other configurations. Different aspects and elements of the configurations may be combined in a similar manner. Also, technology evolves, and therefore, many of the elements are examples and do not limit the scope of the disclosure or the claims.

[0101] Specific details are given in the description to provide a thorough understanding of example configurations, including implementations. However, the configurations may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the configurations. This description provides only example configurations and does not limit the scope, applicability, or configuration of the claims. Rather, the foregoing description of the configurations will provide those skilled in the art with an effective description for implementing the described techniques. Various changes may be made in the function and arrangement of elements without departing from the spirit or scope of the present disclosure.

[0102] Configurations may also be described as processes, depicted as schematic flowcharts or block diagrams. While operations may be described as sequential processes, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may have additional steps not included in the diagrams. Furthermore, embodiments of the method may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments to perform the necessary tasks may be stored in a non-transitory computer-readable medium, such as a storage medium. A processor may perform the described tasks.

[0103] While several example configurations have been described, various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the present disclosure. For example, the elements described above may be components of a larger system, and other rules may take precedence over or otherwise modify the application of the present technology. Also, some steps may occur before, during, or after the elements described above are discussed. Therefore, the foregoing description does not constrain the scope of the claims.

[0104] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Thus, for example, a reference to a "user" includes a plurality of such users, a reference to a "processor" includes a reference to one or more processors and equivalents thereof known to those skilled in the art, and so forth.

[0105] Additionally, the words "comprise," "comprising," "contains," "containing," "include," "including," and "includes," when used in this specification and the claims that follow, are intended to specify the presence of stated features, integers, components, or steps, but they do not exclude the presence or addition of one or more other features, integers, components, steps, acts, or groups.

[0106] It is also to be understood that the examples and embodiments described herein are for illustrative purposes only, and that various modifications or changes in light thereof will be suggested to those skilled in the art and are within the spirit and scope of the present application and the appended claims.

Claims

1. 1. A method for training a machine learning model to predict a user facial expression, the method comprising: receiving a plurality of images, each of the plurality of images including at least a portion of a face of a user; calculating a plurality of values ​​related to a movement metric based on the plurality of images, each of the plurality of values ​​related to the movement metric indicating a movement of the user's face; calculating a plurality of values ​​for facial expression units based on the plurality of values ​​for the movement metrics, each of the plurality of values ​​for the facial expression units corresponding to a range in which the user's face is producing the facial expression unit; generating training output data based on the plurality of images by the machine learning model; and generating error data based on a comparison of the training output data with ground truth data including a plurality of values ​​for the facial expression units; correcting the machine learning model based on the error data; and training the machine learning model using the plurality of images as training input data and the plurality of values ​​for the facial expression units as the ground truth data by A method comprising:

2. The method of claim 1 , wherein the training output data includes a plurality of output values ​​for each of the set of facial units, the facial unit being a first facial unit from the set of facial units.

3. 3. The method of claim 2, wherein the set of facial expression units includes at least one of inner brow raise, outer brow raise, brow lower, upper eyelid raise, cheek raise, eyelid tension, nose wrinkle, eye closure, left eye wink, or right eye wink.

4. Training the machine learning model using the plurality of images as the training input data and a plurality of values ​​related to the facial expression units as the ground truth data further comprises: performing the comparison between the ground truth data, which includes a plurality of values ​​for the facial expression units, and a plurality of output values ​​for the facial expression units of the training output data; The method of claim 1 , comprising:

5. The method of claim 1 , wherein the machine learning model is an artificial neural network having the set of adjustable parameters.

6. The movement metric is a number of eye pixels, and calculating a plurality of values ​​for the movement metric based on the plurality of images includes: segmenting each of the plurality of images such that each of the plurality of images includes eye pixels and non-eye pixels; counting the number of eye pixels in each of the plurality of images; setting each of a plurality of values ​​for the movement metric equal to a number of eye pixels in a corresponding image from the plurality of images; The method of claim 1 , comprising:

7. Calculating a plurality of values ​​for the facial expression units based on a plurality of values ​​for the movement metrics includes: identifying a first extremum within a plurality of values ​​for the motion metric and a first corresponding image for which the first extremum was identified; setting each of a plurality of values ​​for the facial expression units associated with the first corresponding image equal to 1; identifying a second extremum within the plurality of values ​​for the motion metric and a second corresponding image for which the second extremum was identified; setting each of a plurality of values ​​for the facial expression units associated with the second corresponding image equal to zero; setting each remaining value of said plurality of values ​​by interpolating between zero and one; The method of claim 1 , comprising:

8. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: receiving a plurality of images, each of the plurality of images including at least a portion of a face of a user; calculating a plurality of values ​​related to a movement metric based on the plurality of images, each of the plurality of values ​​related to the movement metric indicating a movement of the user's face; calculating a plurality of values ​​for facial expression units based on the plurality of values ​​for the movement metrics, each of the plurality of values ​​for the facial expression units corresponding to a range in which the user's face is producing the facial expression unit; generating training output data based on the plurality of images by a machine learning model; and generating error data based on a comparison of the training output data with ground truth data including a plurality of values ​​for the facial expression units; correcting the machine learning model based on the error data; and training the machine learning model using the plurality of images as training input data and the plurality of values ​​for the facial expression units as the ground truth data by 10. A non-transitory computer-readable medium for performing operations including:

9. 9. The non-transitory computer-readable medium of claim 8, wherein the training output data includes a plurality of output values ​​for each of the set of facial units, the facial unit being a first facial unit from the set of facial units.

10. 10. The non-transitory computer-readable medium of claim 9, wherein the set of facial expression units comprises at least one of inner brow raise, outer brow raise, brow lower, upper eyelid raise, cheek raise, eyelid tension, nose wrinkle, eye closure, left eye wink, or right eye wink.

11. Training the machine learning model using the plurality of images as the training input data and a plurality of values ​​for the facial expression units as the ground truth data further comprises: performing the comparison between the ground truth data, which includes a plurality of values ​​for the facial expression units, and a plurality of output values ​​for the facial expression units of the training output data; 9. The non-transitory computer-readable medium of claim 8, comprising:

12. 10. The non-transitory computer-readable medium of claim 8, wherein the machine learning model is an artificial neural network having the set of adjustable parameters.

13. The movement metric is a number of eye pixels, and calculating a plurality of values ​​for the movement metric based on the plurality of images includes: segmenting each of the plurality of images such that each of the plurality of images includes eye pixels and non-eye pixels; counting the number of eye pixels in each of the plurality of images; setting each of a plurality of values ​​for the movement metric equal to a number of eye pixels in a corresponding image from the plurality of images; 9. The non-transitory computer-readable medium of claim 8, comprising:

14. Calculating a plurality of values ​​for the facial expression units based on a plurality of values ​​for the movement metrics includes: identifying a first extremum within a plurality of values ​​for the motion metric and a first corresponding image for which the first extremum was identified; setting each of a plurality of values ​​for the facial expression units associated with the first corresponding image equal to 1; identifying a second extremum within the plurality of values ​​for the motion metric and a second corresponding image for which the second extremum was identified; setting each of a plurality of values ​​for the facial expression units associated with the second corresponding image equal to zero; setting each remaining value of said plurality of values ​​by interpolating between zero and one; 9. The non-transitory computer-readable medium of claim 8, comprising:

15. 1. A system comprising: one or more processors; A non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to: receiving a plurality of images, each of the plurality of images including at least a portion of a face of a user; calculating a plurality of values ​​related to a movement metric based on the plurality of images, each of the plurality of values ​​related to the movement metric indicating a movement of the user's face; calculating a plurality of values ​​for facial expression units based on the plurality of values ​​for the movement metrics, each of the plurality of values ​​for the facial expression units corresponding to a range in which the user's face is producing the facial expression unit; generating training output data based on the plurality of images by a machine learning model; and generating error data based on a comparison of the training output data with ground truth data including a plurality of values ​​for the facial expression units; modifying the machine learning model based on the error data; training the machine learning model using the plurality of images as training input data and the plurality of values ​​for the facial expression units as the ground truth data by a non-transitory computer-readable medium for performing operations including: A system comprising:

16. The system of claim 15 , wherein the training output data includes a plurality of output values ​​for each of the set of facial units, the facial unit being a first facial unit from the set of facial units.

17. 17. The system of claim 16, wherein the set of facial expression units includes at least one of inner brow raise, outer brow raise, brow lower, upper eyelid raise, cheek raise, eyelid tension, nose wrinkle, eye closure, left eye wink, or right eye wink.

18. Training the machine learning model using the plurality of images as the training input data and a plurality of values ​​for the facial expression units as the ground truth data further comprises: performing the comparison between the ground truth data, which includes a plurality of values ​​for the facial expression units, and a plurality of output values ​​for the facial expression units of the training output data; The system of claim 15, comprising:

19. The movement metric is a number of eye pixels, and calculating a plurality of values ​​for the movement metric based on the plurality of images includes: segmenting each of the plurality of images such that each of the plurality of images includes eye pixels and non-eye pixels; counting the number of eye pixels in each of the plurality of images; setting each of a plurality of values ​​for the movement metric equal to a number of eye pixels in a corresponding image from the plurality of images; The system of claim 15, comprising:

20. Calculating a plurality of values ​​for the facial expression units based on a plurality of values ​​for the movement metrics includes: identifying a first extremum within a plurality of values ​​for the motion metric and a first corresponding image for which the first extremum was identified; setting each of a plurality of values ​​for the facial expression units associated with the first corresponding image equal to 1; identifying a second extremum within the plurality of values ​​for the motion metric and a second corresponding image for which the second extremum was identified; setting each of a plurality of values ​​for the facial expression units associated with the second corresponding image equal to zero; setting each remaining value of said plurality of values ​​by interpolating between zero and one; The system of claim 15, comprising:

Images