Training instances of machine learning models for facial expression prediction and generating new avatars for use in training

Abstract

For each avatar, a test image is rendered for different facial expressions, each facial expression having a ground live facial action unit. Instances of a machine learning model are applied to test images to produce predicted facial action units for each test image. For each avatar, a predicted performance of an instance is calculated based on predicted facial action units of test images of the avatar and ground live facial action units. A first set of features common to avatars whose predicted performance is better than a first threshold and a second set of features common to avatars whose predicted performance is poorer than a second threshold are identified. Only features present in the second group are identified as differential features. New avatars with differential features are generated.

Description

Background Art

[0001] Extended reality (XR) technology includes virtual reality (VR), augmented reality (AR) and mixed reality (MR) technology, and does indeed truly extend the reality that users experience. XR technology can use a head-mounted display (HMD). An HMD is a display device that can be worn on the head. In VR technology, the HMD wearer is immersed in a completely virtual world, while in AR technology, the HMD wearer's direct or indirect view of the physical real-world environment is enhanced. In MR or mixed reality technology, the HMD wearer experiences a merger of the real and virtual worlds. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1A and 1B 2 are perspective and front views, respectively, of an exemplary head mounted display (HMD) that can be used for an extended reality (XR) environment.

[0003] Figure 2 is a diagram of an exemplary process for predicting a facial action unit of a facial expression of a wearer of an HMD from a facial image of the wearer captured by the HMD.

[0004] Figure 3A , 3B 3C are diagrams of exemplary facial images of a wearer of the HMD captured by the HMD, based on which facial action units of facial expressions of the wearer can be predicted.

[0005] Figure 4 is a diagram of an exemplary avatar that can be rendered as a wearer's facial expression with an HMD based on facial action units predicted for the wearer's facial expression.

[0006] Figure 5 is used to train Figure 2 0047] A diagram of an exemplary process for an instance of a machine learning model used to predict facial action units.

[0007] Figure 6 is an illustration of exemplary simulated HMD-captured training images of rendered avatars based on which a machine learning model for predicting facial action units can be trained.

[0008] Figure 7 is a flow chart of an exemplary process for generating a new avatar based on which the new avatar can be used in Figure 2 The machine learning model for predicting facial action units can Figure 5 The process is retrained.

[0009] Figure 8 is a flow chart of an exemplary method.

[0010] Fig. 9 is a diagram of an exemplary non-transitory computer-readable data storage medium.

[0011] Fig.10 is a block diagram of an exemplary system including an HMD. DETAILED DESCRIPTION

[0012] As described in the background, a head mounted display (HMD) can be used as an extended reality (XR) technology to extend the reality experienced by the wearer of the HMD. The HMD can include one or more small display panels located in front of the wearer's eyes and various sensors for detecting or sensing the wearer and / or the wearer's environment. The images on the display panels convincingly immerse the wearer in the XR environment, whether it is virtual reality (VR), augmented reality (AR), mixed reality (MR), or another type of XR.

[0013] The HMD can include one or more cameras, which are image capture devices that capture still or moving images. For example, one camera of the HMD can be used to capture an image of the wearer's lower face (including the mouth). Two other cameras of the HMD can each be used to capture an image of the HMD wearer's respective eyes and a portion of the wearer's face surrounding the eyes.

[0014] In some XR applications, the wearer of an HMD can be represented by an avatar within the XR environment. An avatar is a graphical representation of the wearer or a personality of the wearer that can have a three-dimensional (3D) form and can have varying degrees of realism, from cartoon-like to nearly lifelike. For example, if the HMD wearer is participating in an XR environment with other users wearing their own HMDs, an avatar representing the HMD wearer can be displayed on the HMDs of these other users.

[0015] The avatar can have a face that corresponds to the face of the wearer of the HMD. To more realistically represent the HMD wearer, the avatar may have facial expressions that correspond to the facial expressions of the wearer. Therefore, before the avatar can be rendered to show the same facial expressions, the facial expressions of the HMD wearer must be determined.

[0016] Facial expressions can be defined by a set of facial action units of the Facial Action Coding System (FACS). FACS classifies human facial movements according to their appearance on the face via the values ​​or weights of the different facial action units. Facial action units may also be referred to as blendshapes and / or descriptors, and the values ​​or weights may also be referred to as intensities. For example, an individual facial action unit may correspond to a specific contraction or relaxation of one or more muscles. Any anatomically possible facial expression can therefore be deconstructed or encoded into a set of facial action units representing the facial expression. Note that in some instances, facial expressions can be defined using facial action units that are not specified by FACS.

[0017] Based on the facial action units of a specific facial expression, a facial avatar can be rendered to have the facial expression. That is, the facial action units specifying the specific facial expression allow the rendering of a facial avatar having the facial expression in question. This means that if the facial action units of the wearer of the HMD can be recognized, a facial avatar showing the same facial expression as the HMD wearer can be rendered and displayed. One way to recognize the facial action units of the wearer of the HMD is to employ a machine learning model that predicts the facial action units of the wearer's current facial expression from a facial image of the wearer that has been captured by the HMD.

[0018] The techniques described herein enable training multiple instances of a machine learning model for predicting facial expression facial action units. The same underlying machine learning model can be trained using the same training data and evaluated using the same test data, but using different training parameters. The instance of the machine learning model that generates the best prediction performance can then be used to recognize the facial expression of the HMD wearer from the facial image captured by the wearer's HMD, regardless of who the wearer is. That is, the instance of the model for facial expression recognition can be independent of the wearer of the HMD.

[0019] The techniques described herein can also generate new avatars that can be used to retrain and retest instances of machine learning models for predicting facial expression facial action units. The initial training instance of the machine learning model that generated the best predictive performance can be applied to rendered images of the avatar. Each avatar has multiple rendered images corresponding to different facial expressions. Features specific to the avatar for which the model's predictive performance is worst are identified, and new avatars with these features are generated. Based on the new avatar, the machine learning model can then be retrained to improve its predictive performance.

[0020] Techniques described herein can apply trained (and retrained) instances of a machine learning model to facial images of a wearer of an HMD captured by a camera of the HMD. The captured facial images can be pre-processed so that they better resemble synthetic rendered images based on which the machine learning model was previously trained. The machine learning model outputs predicted facial action units corresponding to the facial expressions of the wearer within the captured images. The facial action units can be post-processed for smoothing so that an avatar corresponding to the wearer rendered with the wearer's facial expressions appears more natural and less choppy.

[0021] Figure 1A and 1B 1 shows a perspective view and a front view of an exemplary HMD 100, which is worn by a wearer 102 and is positioned on the face 104 of the wearer 102 at one end of the HMD 100. In particular, the HMD 100 can be positioned over the nose 151 of the wearer 102 and around his or her right eye 152A and left eye 152B, which are collectively referred to as eyes 152 (according to Figure 1B ). The HMD 100 can include a display panel 106 inside the other end of the HMD 100, and the display panel 106 can be arranged to be incident to the eye 152 of the wearer 102. The display panel 106 may actually include: a right display panel, which is incident to the right eye 152A of the wearer 102 and can be viewed by the right eye 152A of the wearer 102; and a left display panel, which is incident to the left eye 152B of the wearer 102 and can be viewed by the left eye 152B of the wearer 102. By appropriately displaying images on the display panel 106, the HMD 100 can immerse the wearer 102 in XR.

[0022] The HMD 100 can include eye cameras 108A and 108B and / or a mouth camera 108C, which are collectively referred to as cameras 108C. Although only one mouth camera 108C is shown, there may be multiple mouth cameras 108C. Similarly, although only one eye camera 108A and one eye camera 108B are shown, there may be multiple eye cameras 108A and / or multiple eye cameras 108B. The cameras 108 capture images of different parts of the face 104 of the wearer 102 of the HMD 100, based on which facial action units of facial expressions of the wearer 102 can be predicted.

[0023] The eye cameras 108A and 108B are located inside the HMD 100 and are directed toward the respective eyes 152. The right eye camera 108A captures an image including the right eye 152A of the wearer 102 and the facial portion around the right eye 152A of the wearer 102, while the left eye camera 108B captures an image including the left eye 152B of the wearer 102 and the facial portion around the left eye 152B of the wearer 102. The mouth camera 108C is exposed outside the HMD 100 and is directed toward the mouth 154 of the wearer 102 (according to Figure 1B ) to capture an image including the mouth 154 of the wearer 102 and the lower facial portion around the mouth 154 of the wearer 102.

[0024] Figure 2 An exemplary process 200 is shown for predicting facial action units for facial expressions of a wearer 102 of an HMD 100, which can then be redirected to an avatar corresponding to the face of the wearer 102 to render the avatar with the corresponding facial expression. A camera 108 of the HMD 100 captures (204) a set of facial images 206 of a wearer 102 of the HMD 100 (i.e., a set of images 206 of the face 104 of the wearer 102), the wearer 102 currently expressing a given facial expression 202. A trained machine learning model 208 is applied to the facial images 206 to predict facial action units 210 for the facial expressions 202 of the wearer 102.

[0025] However, before applying the trained machine learning model 208 to the captured facial images 206, the facial images 206 may be subjected to preprocessing (207). Preprocessing of the facial images 206 ensures that the images 206 better resemble the synthetically rendered images based on which the machine learning model 208 was previously trained. For example, the model 208 may have been trained on training images of avatars rendered with facial expressions corresponding to facial action units. Preprocessing the actual captured facial images 206 makes them look better as if the images 206 were also produced in a synthetic manner, so that the model 208 can more accurately recognize the facial expressions 202 of the wearer 102. One type of preprocessing that may be performed is adaptive histogram equalization.

[0026] The set of pre-processed facial images 206 are then input 214 into a trained machine learning model 208, which then outputs 216 predicted facial action units 210 for facial expressions 202 of the wearer 102 based on the facial images 206. The trained machine learning model 208 may also output predicted facial expressions based on the facial images 206, the predicted facial expressions corresponding to the facial expressions 202 actually currently being exhibited by the wearer 102. Specific details regarding the machine learning model 208 (particularly, how training data for training the model 208 can be generated) are provided later in the detailed description.

[0027] The predicted facial action units 210 of the facial expression 202 of the wearer 102 of the HMD 100 may then be redirected 228 to an avatar 230 corresponding to the face 104 of the wearer 102 to render the avatar 230 with the facial expression 202. However, before rendering the avatar 230, post-processing may be performed 226 to smooth the predicted facial action units 210. Smoothing the facial action units 210 can ensure that, when they are redirected to the avatar 230, the resulting rendered avatar has facial expressions that appear more natural and realistic and do not appear disjointed. One type of post-processing that may be performed is average mean filtering.

[0028] The result of the facial action unit redirection is therefore an avatar 230 of the wearer 102. As long as the predicted facial action unit 210 (which has been post-processed for smoothing) accurately reflects the facial expression 202 of the wearer 102, the avatar 230 has the same facial expression 202 as the wearer 102. The avatar 230 is rendered in this respect from the predicted facial action unit 210 and, therefore, has a facial expression corresponding to the facial action unit 210.

[0029] The avatar 230 of the wearer 102 of the HMD 100 may then be displayed (232). For example, the avatar 230 may be displayed on an HMD worn by another user who is participating in the same XR environment as the wearer 102. If the facial action unit 210 is predicted by the HMD 100 or by a host device (such as a desktop or laptop computer) to which the HMD 100 is communicatively coupled, the HMD 100 or the host device may thus transmit the rendered avatar 230 to the HMD or host device of the other user participating in the XR environment. In this regard, it is said that the HMD 100 or the host device indirectly displays the avatar 230 whenever the avatar 230 is transmitted for display on the other HMD.

[0030] However, in another implementation, the HMD 100 may itself display the avatar 230. In this regard, it is said that the HMD 100 or the host device directly displays the avatar 230. The process 200 can be repeated (234) by capturing (204) the next set of facial images 206.

[0031] Figure 3A , 3B 3C respectively show a set of exemplary HMD captured images 206A, 206B, and 206C, which are collectively referred to as images 260 and thus can constitute images 206, to which the trained machine learning model 208 is applied to generate predicted facial action units 210. Image 206A has a facial portion 302A including and surrounding the right eye 152A of the wearer 102, while image 206B has a facial portion 302B including and surrounding the left eye 152B of the wearer 102. Image 206C has a lower facial portion 302C including and surrounding the mouth 154 of the wearer 102. Figure 3A , 3B 3C thus illustrate examples of the types of images that can make up the set of facial images 206 used to predict facial action units 210.

[0032] Figure 4 An exemplary image 400 of an avatar 230 that can be rendered when the predicted facial action unit 210 is redirected onto the avatar 230 is shown. In the example, the avatar 230 is a two-dimensional (2D) avatar, but it can also be a 3D avatar. The avatar 230 is rendered from the predicted facial action unit 210 of the facial expression 202 of the wearer 102. Therefore, the avatar 230 has the same facial expression 202 as the wearer 102 in the sense that the predicted facial action unit 210 accurately encodes the facial expression 202 of the wearer 102.

[0033] Figure 5 An exemplary process 500 is shown for training an instance 532 of a machine learning model 208 that can be used to predict facial action units 210 from an HMD-captured facial image 206 of a wearer 102 of an HMD 100. There is a pool of avatars 502 with corresponding features 503. The avatars 502 can each be defined as a 3D model, with the features 503 corresponding to different ethnicities, races, genders, and ages, so that the avatars 502 reflect the diversity found among people around the world. The features 503 can include facial geometry, such as face shape, lip shape, skin color, wrinkles, and the like.

[0034] From the pool of avatars 502, a training avatar 504 is selected (506), and a testing avatar 508 is likewise selected (510). The training avatar 504 is used to train the machine learning model instance 532, while the testing avatar 508 is used to test the instance 532 after training. The training avatar 504 can be mutually exclusive with the testing avatar 508, such that no training avatar 504 is also a testing avatar 508, and vice versa. How the training avatar 504 and the testing avatar 508 are selected from the pool of avatars 502 can be selected in many different ways.

[0035] For example, training avatar 504 may be selected to have one gender, while test avatar 508 may be selected to have a different gender. Training avatar 504 may be selected to have certain skin colors (e.g., darker or lighter), while test avatar 508 may be selected to have other skin colors (e.g., lighter or darker). Training avatar 504 may be selected to be above (or below) a certain age, while test avatar 508 may be selected to be below (or above) the same or different certain age. Training avatar 504 and test avatar 508 may be randomly selected from pool of avatars 502.

[0036] There are different facial expressions 512, each facial expression 512 corresponding to a known ground truth facial action unit 514. Because the facial action unit 514 parameterizes the facial structure with the corresponding facial expression 512, the training image 516 of each training avatar 504 can be rendered (518), and the test image 520 of each testing avatar 508 can be similarly rendered (522). That is, the facial action unit 514 of each facial expression 512 is redirected to each training avatar 504 to produce the training image 516 of the avatar 504, and is redirected to each testing avatar 508 to produce the training image 520 of the avatar 508.

[0037] For example, there may be M training avatars 504 and N testing avatars 508, where M may be equal to or different from N. There may also be a set of P facial expressions 512, each with a specified ground truth facial action unit 514. Thus, the result is a set of MxP training images 516 and a set of NxP testing images 520. Rendering of the training avatar 504 or the testing avatar 508 based on the specified facial action units 514 results in the avatar 504 or 508 exhibiting the facial expressions 512 corresponding to those facial action units 514. The obtained training images 516 of the training avatar 504 or the obtained testing images 520 of the testing avatar 510 are known to correspond to the specified ground truth facial action units 514 because the avatar 504 or 510 is rendered based on those facial action units 514.

[0038] For each training image 516 of each training avatar 504, a set of HMD-captured avatar training images 524 can be simulated (524). The HMD-captured training images 524 simulate how an actual HMD (e.g., HMD 100) would capture the face of the avatar 504 if the avatar 504 were a real person wearing the HMD. The simulated HMD-captured training images 524 can therefore correspond to the actual HMD-captured facial images 206 of the actual HMD wearer 102, as the images 524 can have substantially the same size and resolution as the actual images 206, and can include facial portions comparable to or corresponding to facial portions of the actual images 206. Similarly, for each test image 520 of each test avatar 508, a set of HMD-captured avatar test images 528 can be simulated (530).

[0039] Using the simulated HMD to capture the training images 524, instances 532 of the machine learning model 208 are then individually trained (534). Each instance 532 corresponds to the same machine learning model 208, but is trained using different training parameters (535). Examples of training parameters include learning rate and step size. The learning rate is the amount by which the weights are updated during the training of the model 208 in the case of a neural network. For example, the learning rate can be selected between the range of 0.0 and 1.0. The step size is the number of epochs by which the learning rate is adjusted to optimize the training of the model 208, where the optimally selected step size shortens the training time.

[0040] Other exemplary training parameters include the optimization algorithm employed during training (e.g., gradient descent, stochastic gradient descent, or Adam optimizer), the cost or loss function used during training, and the number of iterations or cycles used during training of the machine learning model 208. Still other exemplary training parameters include which channels to use during training. For example, the training image 524 may have red, green, and blue channels so that different combinations of one or more of these channels may be used during training of the model 208. In one implementation, there may be 3 to 7 machine learning model instances 532 with corresponding training parameters 535.

[0041] The machine learning model 208 may itself be a convolutional neural network or include a convolutional neural network having a convolutional layer followed by a pooling layer that generates, recognizes, or extracts image features to predict facial action units from an input image. Examples include different versions of the MobileNet machine learning model. The MobileNet machine learning model is described in the following literature: A. Howard et al., "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications," arXiv: 1704.04861 [cs.CV], April 2017; M. Sandler et al. "MobileNetV2: Inverted Residualsand Linear Bottlenecks," arXiv: 1801.104381 ​​[cs.CV], March 2019; and A. Howard et al., "Search for MobileNetV3," arXiv: 1905.02244 [cs.CV], November 2019.

[0042] In one implementation, the machine learning model 208 can be a two-stage machine learning model. The first stage can be a backbone network, such as a convolutional neural network (e.g., a version of the MobileNet machine learning model), for extracting image features (not to be confused with features 503 of the avatar 502) from the training images 524. The second stage can be another convolutional neural network, such as a regression type network, for predicting the facial action units of each training image 516 from the extracted image features.

[0043] The machine learning model 208 can be trained to minimize a loss value between a specified ground truth facial action unit 514 and a predicted facial action unit output by the model 208. For example, the loss value minimized can be the mean absolute error (MAE). Another example of a loss value is the mean squared error (MSE).

[0044] Once each instance 532 of the machine learning model 208 has been trained, the machine learning model instance 532 is applied 536 to the HMD captured test images 528 to generate predicted facial action units 538. For each test image 520 of a test avatar 508 corresponding to a ground truth facial action unit 514, there is a corresponding predicted facial action unit 538 output by each instance 532 of the model 208. For example, if there are P test images 520 of each of the N test avatars 508 corresponding to P facial expressions 512 (each having a ground truth facial action unit 514), and if there are Q instances 532, then there is a PxNxQ set of predicted facial action units 538.

[0045] The prediction performance 540 of each instance of the machine learning model 208 is calculated (542). The prediction performance of the instance 532 of the machine learning model 208 in predicting the ground truth facial action units 514 is based on how closely each set of predicted facial action units 538 output by the instance 532 matches its corresponding set of ground truth facial action units 514. The prediction performance 540 can be calculated as the MAE or MSE between the predicted facial action units 538 and their corresponding ground truth facial action units 514. In one implementation, the instance 532 with the best prediction performance 540 is used in the process 200 to predict the facial action unit 210 corresponding to the facial expression 202 performed by the HMD wearer 102 in the captured facial image 206.

[0046] Figure 6 Exemplary rendered avatar images 600 of avatar 502 are shown, such as training image 516 for training avatar 504 or test image 520 for testing avatar 508 . Figure 6 Also shown are exemplary HMD capture images 602A, 602B, 602C, which are simulated from the rendered avatar image 600 and can be collectively referred to as simulated HMD capture images 602, based on which each instance 532 of the machine learning model 208 can be actually trained. The HMD capture image 602 can be an HMD capture training image 524 simulated from the training image 516 or an HMD capture test image 528 simulated from the test image 520.

[0047] Simulated HMD captured image 602A has facial portion 606A surrounding and including left eye 608A of avatar 530, while image 602B has facial portion 606B surrounding and including right eye 608B of avatar 530. Images 602A and 602B are thus identical to actual left-eye and right-eye images that can be captured by an HMD, such as, Figure 3A and 3BThat is, images 602A and 602B may have the same size and resolution as the left-eye and right-eye images captured by the actual HMD, and capture the same facial parts as the left-eye and right-eye images captured by the actual HMD.

[0048] Simulated HMD captured image 602C has lower face portion 606C surrounding and including mouth 610 of avatar 530. Image 602C is thus consistent with an actual mouth image captured by the HMD (such as Figure 3C Similarly, then, image 602C may have the same size and resolution as the actual HMD captured mouth image, and capture the same facial portion as the actual HMD captured mouth image. Figure 6 Thus, an avatar image 602 is shown, rather than an image of the actual HMD wearer.

[0049] Typically, the avatar images 602 match the perspective and image characteristics of the HMD wearer's facial image captured by the HMD's actual camera, based on which the machine learning model 208 will be used to predict the wearer's facial expression. That is, the avatar images 602 are actually captured by a virtual camera corresponding to the actual HMD camera. The avatar images 602 that have been described reflect only one specific placement of such virtual cameras. More generally, then, the avatar images 602 can vary in number and placement depending on the actual HMD camera used to predict the HMD wearer's facial expression.

[0050] For example, the HMD mouth camera may be a stereo camera so that a larger portion of the wearer's cheek may be included in a correspondingly captured facial image, in which case the avatar image 602 corresponding to such facial image would likewise capture a larger portion of the rendered avatar's cheek. As another example, the HMD camera may also include a forehead camera to capture a facial image of the wearer's forehead, in which case the avatar image 602 would include a corresponding image of the rendered avatar's forehead. As a third example, there may be multiple eye cameras to capture regions surrounding the wearer's eyes at different tilt angles, in which case the avatar image 602 would also include corresponding such images.

[0051] Figure 7An exemplary process 700 is shown for generating a new avatar 732 based on which a machine learning model instance 532 can be retrained in process 500. Process 700 is performed after at least one instance 532 of a machine learning model 208 has been initially trained via process 500. In one implementation, process 700 can use the instance 532 of the machine learning model 208 that has the best per-instance prediction performance 540 of any of the instances 532.

[0052] Corresponding to the facial expressions 512, images 704 of the avatar 502 with the features 503 can be rendered 706, as before, each facial expression 512 having an assigned ground truth facial action unit 514. If the training avatars 504 and the testing avatars 508 together include all avatars 502, then the training images 516 and the testing images 520 can be reused as the images 704. If the training avatars 504 and the testing avatars 508 together do not include all avatars 502, then the training images 516 and the testing images 520 can be reused as the images 704 and supplemented by images 704 rendered only for each avatar 502 that is not a training avatar 504 or a testing avatar 508.

[0053] Also as before, the HMD captured image 708 is then simulated 710 from the image 704. If the training avatar 504 and the testing avatar 508 together include all avatars 502, then the HMD captured training image 524 and the testing image 528 may be reused as the HMD captured image 708. If the training avatar 504 and the testing avatar 508 together do not include all avatars 502, then the HMD captured training image 524 and the testing image 528 may be reused as the HMD captured image 708 and supplemented by the image 708 simulated from the image 704 rendered for each avatar 502 that is not a training avatar 504 or a testing avatar 508.

[0054] The machine learning model 208 is then applied 714 to the HMD captured images 708 to generate or produce predicted facial action units 716. As described above, the instance 532 of the machine learning model 208 with the best per-instance prediction performance may be applied. For each image 704 of each avatar 502, a set of predicted facial action units 716 is generated that corresponds to the facial expressions 512 with the specified ground truth facial action units 514 based on which the image 704 is rendered.

[0055] The per-avatar prediction performance 718 of the machine learning model 208 is calculated 720. For each avatar 502, the prediction performance 718 of the model 208 is calculated based on how well the predicted facial action units 716 generated by the model 208 for the image 704 of the avatar 502 match its corresponding ground truth facial action units 514. The prediction performance 718 of the avatar 502 may be calculated as the MAE or MSE between the predicted facial action units 716 for each image 704 of the avatar 502 and the ground truth facial action units 514 for that image 704.

[0056] The predicted performance 718 is thus based on the performance of the instance 532 of the machine learning model 208 for each avatar. In contrast, the predicted performance 540 calculated in the process 500 is based on the performance of the machine learning model instance 532 for each instance. For the instance 532 used in the process 700, the each instance predicted performance 540 is its predicted performance on all training avatars 504, rather than on only a single avatar 502 as is the case with the per-avatar predicted performance 718.

[0057] Features 503 that are common to avatars 502 with better predictive performance 718 are identified 726 as a (first) group 722 of features 503. Features 503 that are common to avatars 502 with worse predictive performance 718 are also identified 726 as a (second) group 724 of features 503. For example, each feature 503 of features 503 that is present in every avatar 502 with a predictive performance 718 greater than a (first) threshold is added to group 722. Similarly, each feature 503 of features 503 that is present in every avatar 502 with a predictive performance 718 less than a (second) threshold is added to group 724.

[0058] In one implementation, the avatars 502 may be sorted into quartiles by each avatar's predicted performance 718 such that each quartile includes an equal or substantially equal number of avatars 502. The avatars 502 with better predicted performance 718 are the avatars 502 in the highest quartile because they all have a predicted performance 718 that is greater than the highest predicted performance 718 of any avatar 502 in the next highest quartile. The avatars 502 with worse performance 718 are the avatars 502 in the lowest quartile because each avatar 502 has a predicted performance that is less than the lowest predicted performance of any avatar 502 in the next lowest quartile.

[0059] Features 503 that are in group 724 but not in group 722 are identified (730) and are referred to as differential features 728. Differential features 728 may be factors as to why the predictive performance 718 of the machine learning model 208 is worse for certain avatars 502. That is, not all features 503 in the features 503 of group 724 that are common to the avatars 502 for which the model 208 has worse predictive performance 718 may be factors as to why the performance 718 is worse for those avatars 502. For example, any feature 503 that is in group 724 and also in group 722 may not be a factor as to why the predictive performance 718 of the model 208 is worse for certain avatars 502.

[0060] A new avatar 732 having the differential features 728 is generated (734). For example, the new avatar 732 may have the same differential features 728, but may have different features 503 in addition to the features 728. The differential features 728 may be adjusted within a range to generate the new avatar 732. Different avatars 732 may have different combinations of differential features 728. The new avatar 732 is generated so that the machine learning model 208 can be retrained (736) to improve its predictive performance 718 for avatars 732 having these features 728.

[0061] The process 700 may be repeated many times in order to iteratively improve the machine learning model 208. The process 700 thus identifies features 728 based on which the model 208 is having difficulty predicting the facial action units 716 of the facial expressions 512 of the avatar 706. By adding new avatars 732 having these features 728 and then retraining the machine learning model 208 using the new avatars 732, the overall prediction performance 540 can be improved. Only the instance 532 of the machine learning model 208 used in the process 700 may be retrained, or multiple instances 532 of the model 208 may be retrained according to the process 500.

[0062] Figure 8 An exemplary method 800 is shown. The method 800 may be implemented as program code stored on a non-transitory computer-readable data storage medium and executed by a processor. The method 800 includes: selecting a training avatar 504 and a test avatar 508 from a pool of avatars 502 (802). The method 800 includes: for each training avatar 504, rendering a training image 516 of a different facial expression 512, each facial expression 512 corresponding to a ground truth facial action unit 514 (804). The method 800 includes: using the training image 516 to train an instance 532 of the machine learning model 208 (806), wherein the instance 532 corresponds to different training parameters 535.

[0063] The method 800 includes, for each test avatar 508, rendering test images 520 of different facial expressions 512 (808). The method 800 includes, applying each instance 532 of the machine learning model 208 to the test images 520 to generate predicted facial action units 538 for each test image 520 (810). The method 800 includes, based on the predicted and ground truth facial action units 538 and 514 for the test avatar 508, calculating the predicted performance of each instance 532 (812).

[0064] Fig. 9 An exemplary non-transitory computer-readable data storage medium 900 storing program code 902 executable by a processor to perform processing is shown. The processing includes: for each avatar 502 in the pool of avatars 502, rendering an image 704 of a different facial expression 512, each facial expression 512 having a ground truth facial action unit 514 (904). The processing includes: applying the machine learning model 208 to the images 704 to produce a predicted facial action unit 716 for each image 704 (906). The processing includes: based on the predictions of the images 704 of the avatar 502 and the ground truth facial action units 716 and 514, for each avatar 502, calculating a predicted performance 718 of the machine learning model 208 (908).

[0065] The process includes identifying a first set 722 of features 503 that are common to avatars 502 whose predicted performance 718 is better than a first threshold (910). The process includes identifying a second set 724 of features 503 that are common to avatars 502 whose predicted performance 718 is worse than a second threshold (912). The process includes identifying features 503 that are only present in the second set 724 and not in the first set 722 (914) as differential features 728. The process includes generating a new avatar 734 having the differential features 728 (916).

[0066] Fig.10An exemplary system 1000 is shown. System 1000 is depicted as including an HMD 100 and a computing device 1002. One or more cameras 108 of the HMD 100 are used to capture a set of facial images 206 of a wearer 102 of the HMD 100. The computing device 1002 has: a processor 1004; and a memory 1006 storing program code 1008. For example, the computing device 1002 may be a host device of the HMD 100. However, in another implementation, the processor 1004 and the memory 1006 may be part of the HMD 100. In this case, the processor 1004 and the memory 1006 may be integrated within an application specific integrated circuit (ASIC) so that the processor 1004 is a dedicated processor. The processor 1004 may alternatively be a general purpose processor, such as a central processing unit (CPU), so that the memory 1006 may be a separate semiconductor or other type of volatile or non-volatile memory 1006.

[0067] Program code 1008 may be executed by processor 1004 to perform processing. The processing includes: pre-processing facial image 206 so that image 206 better resembles a synthetic rendered image (1010). The processing includes: applying machine learning model 208 to pre-processed facial image 206 to generate predicted wearer facial action units 210 of facial expressions 202 of wearer 102 represented within facial image 206 (1012). The processing includes: post-processing predicted wearer facial action units 210 to smooth predicted wearer facial action units 210 (1014). The processing includes: redirecting the post-processed predicted facial action units 210 onto an avatar 230 corresponding to wearer 102 to render the avatar 230 with facial expressions 202 of wearer 102 for display (1016).

[0068] Techniques for training instances of machine learning models for facial expression prediction have been described. Using avatars, machine learning model instances are trained and tested, and the techniques include generating new avatars to improve the predictive performance of the model. The machine learning model can be applied to captured facial images of a wearer of an HMD that have been pre-processed to predict facial action units of facial expressions exhibited by the wearer. The predicted facial action units can be post-processed and then redirected onto the avatar so that the avatar is rendered with the facial expressions of the wearer.

Claims

1. A non-transitory computer readable medium storing program code, the program code being executable by a processor to perform a process, the process comprising: For each of the plurality of avatars, rendering images of different facial expressions, each facial expression having a ground truth facial action unit; applying a machine learning model to the images to generate predicted facial action units for each image; computing a prediction performance of the machine learning model for each avatar based on the predicted facial action units and ground truth facial action units for the image of the avatar; identifying a first set of features that are common to avatars whose prediction performance is better than a first threshold; identifying a second set of features that are common to avatars whose prediction performance is worse than a second threshold; identifying features present only in the second group as differential features; and A new avatar is generated having the differential characteristics.

2. The non-transitory computer readable medium of claim 1, wherein the processing further comprises: The machine learning model is retrained using the new avatar.

3. The non-transitory computer readable medium of claim 2, wherein the processing further comprises: applying the retrained machine learning model to a facial image captured by a head mounted display (HMD) of a wearer exhibiting a facial expression to generate a predicted wearer facial action unit of the facial expression of the wearer; redirecting the predicted wearer facial action unit to an avatar corresponding to the wearer to render the avatar having the facial expression of the wearer; and A rendered avatar corresponding to the wearer is displayed.

4. The non-transitory computer-readable medium of claim 1 , wherein rendering the image comprises: For each different facial expression, a corresponding image of each avatar is rendered.

5. The non-transitory computer readable medium of claim 1 , wherein computing the predicted performance of the machine learning model for each avatar comprises: A mean absolute error or a mean squared error is calculated between the predicted facial action units and the ground truth facial action units for the image of the avatar.

6. The non-transitory computer readable medium of claim 1, wherein the first threshold comprises a top quartile of the predicted performance across all incarnations, And wherein the second threshold comprises the lowest quartile of the predicted performance across all incarnations.

7. The non-transitory computer readable medium of claim 1, wherein the characteristics of the avatar include facial geometry.

8. A method comprising: Selecting a training avatar and a test avatar from a plurality of avatars; For each training avatar, rendering multiple training images of different facial expressions, each facial expression corresponding to a ground truth facial action unit; Using the training images to train multiple instances of the machine learning model, each instance corresponding to a different training parameter; For each test avatar, rendering a plurality of test images of the different facial expressions; applying each instance of the machine learning model to the test images to generate predicted facial action units for each test image; and A prediction performance of each instance of the machine learning model is calculated based on the predicted facial action units and the ground truth facial action units of the test image of the test avatar.

9. The method of claim 8, further comprising: applying the instance of the machine learning model with the best prediction performance to a facial image captured by a head mounted display (HMD) of a wearer exhibiting a facial expression to generate predicted wearer facial action units for the wearer; redirecting the predicted wearer facial action unit to an avatar corresponding to the wearer to render the avatar having the facial expression of the wearer; and A rendered avatar corresponding to the wearer is displayed.

10. The method of claim 8, wherein calculating the predictive performance of each instance of the machine learning model comprises: A mean absolute error is calculated between predicted facial action units produced by the instance and the ground truth facial action units for the test image of the test avatar.

11. The method of claim 8, further comprising: applying the instance of the machine learning model with the best prediction performance to the training images to generate the predicted facial action units for each training image; for each avatar, computing an avatar-specific prediction performance of an instance of the machine learning model having the best prediction performance based on the predicted facial action units and the ground truth facial action units of the test avatar; identifying a first set of features common to avatars whose avatar-specific prediction performance is better than a first threshold; identifying a second set of features common to avatars whose avatar-specific prediction performance is worse than a second threshold; identifying features that are present only in the second group as distinct features; and generating a new avatar having said differential characteristics; and Retrain the instance of the machine learning model using the new avatar.

12. A system comprising: a head mounted display (HMD) having one or more cameras to capture a set of facial images of a wearer of the HMD; processor; and A memory storing program code, wherein the program code is executable by the processor to: preprocessing the facial image so that the image better resembles a synthetic rendered image; applying a machine learning model to the preprocessed facial image to generate predicted wearer facial action units of a wearer's facial expression represented within the facial image; post-processing the predicted wearer facial action unit to smooth the predicted wearer facial action unit; and The post-processed predicted facial action units are redirected to an avatar corresponding to the wearer to render the avatar with the facial expression of the wearer for display.

13. The system of claim 12, wherein the processor is configured to pre-process the facial image by performing adaptive histogram equalization, And wherein the processor is used to post-process the predicted wearer facial action unit by performing average mean filtering.

14. The system of claim 12, wherein the machine learning model is initially trained using multiple avatars and is retrained using new avatars having features specific to avatars for which the predictive performance of the initially trained machine learning model is worse than a threshold.

15. The system of claim 12, wherein the applied machine learning model is an instance with the best predictive performance among multiple instances of the machine learning model, each instance corresponding to different training parameters.

Images