Selecting combination of parameters for pretreating face image of wearer of head mounted display

Abstract

For each face image of a wearer of a head mounted display (HMD), a pre-processed face image corresponding to a combination of pre-processing parameters is generated. A machine learning model is applied to each pre-processed face image to predict a face action unit. Face action units predicted from each pre-processed face image are redirected onto the avatar to render an avatar face image. An avatar face landmark within each avatar face image and a wearer face landmark within each face image are detected. A combination of pre-processing parameters that generate the highest similarity between the avatar face signatures and the wearer face signatures corresponding to the avatar face signatures is selected.

Description

Background Art

[0001] Extended reality (XR) technology includes virtual reality (VR), augmented reality (AR) and mixed reality (MR) technology, and does indeed truly extend the reality that users experience. XR technology can use a head-mounted display (HMD). An HMD is a display device that can be worn on the head. In VR technology, the HMD wearer is immersed in a completely virtual world, while in AR technology, the HMD wearer's direct or indirect view of the physical real-world environment is enhanced. In MR or mixed reality technology, the HMD wearer experiences a merger of the real and virtual worlds. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1A and 1B 2 are perspective and front views, respectively, of an exemplary head mounted display (HMD) that can be used in an extended reality (XR) environment.

[0003] Figure 2 is a diagram of an exemplary process for predicting facial action units of facial expressions of a wearer of an HMD from a facial image of the wearer captured by the HMD after pre-processing the image.

[0004] Figure 3A , 3B 3C are diagrams of exemplary facial images of a wearer of the HMD captured by the HMD, based on which facial action units of facial expressions of the wearer can be predicted.

[0005] Figure 4 is a diagram of an exemplary avatar that can be rendered as a facial expression of a wearer with an HMD based on facial action units predicted for the wearer's facial expression.

[0006] Figure 5 is a diagram of an exemplary process for selecting a combination of parameters for preprocessing a facial image, which a machine learning model can then Figure 2 The facial image is used to predict facial action units.

[0007] Figure 6 is a flow chart of an exemplary method for obtaining a facial calibration image that can be used to Figure 5 in the process.

[0008] Figure 7 is able to Figure 5 2 is a diagram of exemplary facial landmarks detected within a facial image of a wearer of an HMD or within an image of a rendered avatar corresponding to the wearer during the process of detecting facial landmarks within an HMD or within an image of a rendered avatar corresponding to the wearer.

[0009] Figure 8 is a diagram of an exemplary non-transitory computer-readable data storage medium.

[0010] Fig. 9 is a flow chart of an exemplary method.

[0011] Fig.10 is a block diagram of an exemplary system including an HMD. DETAILED DESCRIPTION

[0012] As described in the background, a head mounted display (HMD) can be used as an extended reality (XR) technology to extend the reality experienced by the wearer of the HMD. The HMD can include one or more small display panels located in front of the wearer's eyes and various sensors for detecting or sensing the wearer and / or the wearer's environment. The images on the display panels convincingly immerse the wearer in the XR environment, whether it is virtual reality (VR), augmented reality (AR), mixed reality (MR), or another type of XR.

[0013] The HMD can include one or more cameras, which are image capture devices that capture still or moving images. For example, one camera of the HMD can be used to capture an image of the wearer's lower face (including the mouth). Two other cameras of the HMD can each be used to capture an image of the HMD wearer's respective eyes and a portion of the wearer's face surrounding the eyes.

[0014] In some XR applications, the wearer of an HMD can be represented by an avatar within the XR environment. An avatar is a graphical representation of the wearer or a personality of the wearer that may have a three-dimensional (3D) form and may have varying degrees of realism, from cartoon-like to nearly lifelike. For example, if the HMD wearer is participating in an XR environment with other users wearing their own HMDs, an avatar representing the HMD wearer may be displayed on the HMDs of these other users.

[0015] The avatar can have a face that corresponds to the face of the wearer of the HMD. To more realistically represent the HMD wearer, the avatar may have facial expressions that correspond to the facial expressions of the wearer. Therefore, before the avatar can be rendered to show the same facial expressions, the facial expressions of the HMD wearer must be determined.

[0016] Facial expressions can be defined by a set of facial action units of the Facial Action Coding System (FACS). FACS classifies human facial movements according to their appearance on the face via values, weights or units of different facial actions. Facial actions may also be referred to as blendshapes and / or descriptors, and units may also be referred to as intensities. For example, an individual facial action may correspond to a specific contraction or relaxation within one or more muscles. Any anatomically possible facial expression can therefore be deconstructed or encoded into a set of facial action units representing the facial expression. Note that in some instances, facial expressions can be defined using facial action units that are not specified by FACS.

[0017] Based on the facial action units of a specific facial expression, a facial avatar can be rendered to have the facial expression. That is, the facial action units specifying the specific facial expression allow the rendering of a facial avatar having the facial expression in question. This means that if the facial action units of the wearer of the HMD can be recognized, a facial avatar showing the same facial expression as the HMD wearer can be rendered and displayed. One way to recognize the facial action units of the wearer of the HMD is to employ a machine learning model that predicts the facial action units of the wearer's current facial expression from a facial image of the wearer that has been captured by the HMD.

[0018] To improve the accuracy of the machine learning model, the facial images of the HMD wearer may be pre-processed before the model is applied. Pre-processing the captured images of the wearer's face can allow the machine learning model to more easily extract image features based on which the model predicts facial action units. An exemplary pre-processing technique is contrast limited adaptive histogram equalization (CLAHE). CLAHE improves the contrast within an image. The technique is adaptive because different images (and more specifically, different regions of an image) may have their contrast amplified by different amounts. The technique is contrast limited to limit the amount of contrast amplification in any region of the image.

[0019] CLAHE has a number of tunable preprocessing parameters, namely, grid size and clipping limit. The grid size is the size of the grid used for equalization. The image is divided on a grid of equal-sized rectangular zones, and the grid size parameter specifies both the size of such a grid. The grid size can be specified as an integer between 2 (corresponding to the minimum grid size and therefore the maximum number of zones) and 32 (corresponding to the maximum grid size and therefore the minimum number of zones). The clipping limit is a contrast limiting threshold, which is the maximum amount by which the contrast can be amplified in each zone. The clipping limit can be specified as an integer from 0 (corresponding to the minimum limit) to 10 (corresponding to the maximum limit).

[0020] The combination of pre-processing parameters that causes the machine learning model to predict the facial action units that best correspond to the facial expressions of the HMD wearer within the facial image can vary depending on the wearer and based on the lighting conditions under which the image was captured. For example, pre-processing a facial image of a first wearer captured under initial lighting conditions using a given combination of parameters may result in the most accurate prediction when the model is applied to the image.

[0021] In contrast, when the image is pre-processed using a combination of parameters that is different from the combination of parameters used during the initial lighting conditions, this may result in the most accurate prediction when the facial image of the first wearer is captured under different lighting conditions. Similarly, when the facial image of the second wearer is pre-processed using a combination that is different from the combination used for the first wearer, this may result in the most accurate prediction for the second wearer even if the image is captured under the same lighting conditions.

[0022] The techniques described herein select a combination of preprocessing parameters that are more likely to cause a machine learning model to predict facial action units that most accurately reflect the facial expressions exhibited by the HMD wearer in a captured facial image. Prior to application of the model, the captured facial image is preprocessed using the selected combination of parameters. The combination of preprocessing parameters may be selected when the wearer first uses the HMD. The combination of parameters may be reselected when the lighting conditions under which the HMD captures the wearer's facial image change, either automatically or by the user manually restarting the selection process.

[0023] Figure 1A and 1B 1 shows perspective and front views of an exemplary HMD 100 worn by a wearer 102 and positioned on a face 104 of the wearer 102 at one end of the HMD 100. In particular, the HMD 100 can be positioned over a nose 151 of the wearer 102 and around his or her right eye 152A and left eye 152B, which are collectively referred to as eyes 152 (according to Figure 1B ). The HMD 100 can include a display panel 106 inside the other end of the HMD 100, and the display panel 106 can be arranged to be incident to the eye 152 of the wearer 102. The display panel 106 may actually include: a right display panel, which is incident to the right eye 152A of the wearer 102 and can be viewed by the right eye 152A of the wearer 102; and a left display panel, which is incident to the left eye 152B of the wearer 102 and can be viewed by the left eye 152B of the wearer 102. By appropriately displaying images on the display panel 106, the HMD 100 can immerse the wearer 102 in XR.

[0024] The HMD 100 can include eye cameras 108A and 108B and / or a mouth camera 108C, which are collectively referred to as cameras 108C. Although only one mouth camera 108C is shown, there may be multiple mouth cameras 108C. Similarly, although only one eye camera 108A and one eye camera 108B are shown, there may be multiple eye cameras 108A and / or multiple eye cameras 108B. The cameras 108 capture images of different parts of the face 104 of the wearer 102 of the HMD 100, based on which facial action units of facial expressions of the wearer 102 can be predicted.

[0025] The eye cameras 108A and 108B are located inside the HMD 100 and are directed toward the respective eyes 152. The right eye camera 108A captures an image including the right eye 152A of the wearer 102 and the facial portion around the right eye 152A of the wearer 102, while the left eye camera 108B captures an image including the left eye 152B of the wearer 102 and the facial portion around the left eye 152B of the wearer 102. The mouth camera 108C is exposed outside the HMD 100 and is directed toward the mouth 154 of the wearer 102 (according to Figure 1B ) to capture an image including the mouth 154 of the wearer 102 and the lower facial portion around the mouth 154 of the wearer 102.

[0026] Figure 2 An exemplary process 200 is shown for predicting facial action units for facial expressions of a wearer 102 of an HMD 100, which can then be retargeted to an avatar corresponding to the face of the wearer 102 to render the avatar with the corresponding facial expressions. A camera 108 of the HMD 100 captures (204) a set of facial images 206 of a wearer 102 of the HMD 100 (i.e., a set of images 206 of the face 104 of the wearer 102), who is currently expressing a given facial expression 202. A trained machine learning model 208 is applied to the facial images 206 to predict facial action units 210 for the facial expressions 202 of the wearer 102.

[0027] The machine learning model 208 may itself be a convolutional neural network or include a convolutional neural network having a convolutional layer followed by a pooling layer, the convolutional layer generating, recognizing, or extracting image features to predict facial action units from an input image. Examples include different versions of the MobileNet machine learning model. The MobileNet machine learning model is described in the following literature: A. Howard et al., "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications," arXiv: 1704.04861 [cs. CV], April 2017; M. Sandler et al. "MobileNetV2: Inverted Residuals and Linear Bottlenecks," arXiv: 1801.104381 ​​[cs. CV], March 2019; and A. Howard et al., "Search for MobileNetV3," arXiv: 1905.02244 [cs. CV], November 2019.

[0028] However, before applying the trained machine learning model 208 to the captured facial image 206, the facial image 206 is preprocessed (207) using the selected combination of preprocessing parameters 205 to generate a preprocessed facial image 206' to which the model 208 is actually applied. For example, the preprocessing may include applying an image preprocessing technique (such as CLAHE) to the facial image 206 to produce the preprocessed facial image 206'. In one implementation, the open source computer vision (OpenCV) implementation of CLAHE described at docs.opencv.org / 4.x / d5 / daf / tutorial_py_histogram_equalization.html may be employed. In this case, the combination of parameters 205 can include: a specific value for the clipping limit, such as an integer between 0 and 10; and a specific value for the grid size, such as an integer between 2 and 32.

[0029] The selected pre-processing parameter combination 205 is specific to the wearer 102 of the HMD 100, and therefore, differs according to the user who is currently the wearer 102. The selected combination 205 may also be specific to the lighting conditions under which the facial image 206 was captured. The parameter combination 205 for the wearer 102 may be selected under the initial lighting conditions, and when the lighting conditions change, the combination 205 may be reselected automatically or at the request of the wearer 102. How to select the combination 205 of parameters is described later in the detailed description.

[0030] The set of pre-processed facial images 206' are thus input 214 into the trained machine learning model 208, which then outputs 216 predicted facial action units 210 of the facial expressions 202 of the wearer 102 based on the facial images 206'. The trained machine learning model 208 may also output predicted facial expressions based on the facial images 206', the predicted facial expressions corresponding to the facial expressions 202 actually currently being exhibited by the wearer 102.

[0031] In one implementation, the machine learning model 208 may be a two-stage machine learning model. The first stage may be a backbone network, such as a convolutional neural network (e.g., a version of the MobileNet machine learning model), for extracting image features from the image 206'. The second stage may be another convolutional neural network, such as a regression type network, for predicting facial action units 210 from the extracted image features.

[0032] The predicted facial action units 210 of the facial expression 202 of the wearer 102 of the HMD 100 may be redirected 228 to the avatar 230 corresponding to the face 104 of the wearer 102 to render the avatar 230 with such facial expression 202. Before rendering the avatar 230, post-processing (such as average mean filtering) may be performed to smooth the predicted facial action units 210 to ensure that the resulting rendered avatar has a facial expression that looks more natural and realistic and is not choppy.

[0033] The result of the facial action unit redirection is therefore an avatar 230 of the wearer 102. As long as the predicted facial action unit 210 (which has been post-processed for smoothing) accurately reflects the facial expression 202 of the wearer 102, the avatar 230 has the same facial expression 202 as the wearer 102. The avatar 230 is rendered in this respect from the predicted facial action unit 210 and, therefore, has a facial expression corresponding to the facial action unit 210.

[0034] The avatar 230 of the wearer 102 of the HMD 100 may then be displayed (232). For example, the avatar 230 may be displayed on an HMD worn by another user who is participating in the same XR environment as the wearer 102. If the facial action unit 210 is predicted by the HMD 100 or by a host device (such as a desktop or laptop computer) to which the HMD 100 is communicatively coupled, the HMD 100 or the host device may thus transmit the rendered avatar 230 to the HMD or host device of the other user participating in the XR environment. In this regard, it is said that the HMD 100 or the host device indirectly displays the avatar 230 whenever the avatar 230 is transmitted for display on the other HMD.

[0035] However, in another implementation, the HMD 100 may itself display the avatar 230. In this regard, it is said that the HMD 100 or the host device directly displays the avatar 230. The process 200 can be repeated by capturing (204) the next set of facial images 206 (234).

[0036] Figure 3A , 3B 3C respectively illustrate a set of exemplary HMD captured images 206A, 206B, and 206C, which are collectively referred to as images 206 and can constitute images 206, and the trained machine learning model 208 is applied to the images 206 to generate predicted facial action units 210. Image 206A has a facial portion 302A including and surrounding the right eye 152A of the wearer 102, while image 206B has a facial portion 302B including and surrounding the left eye 152B of the wearer 102. Image 206C has a lower facial portion 302C including and surrounding the mouth 154 of the wearer 102. Figure 3A , 3B 3C thus illustrate examples of the types of images that can make up the set of facial images 206 used to predict facial action units 210.

[0037] Figure 4 An exemplary image 400 of an avatar 230 that can be rendered when the predicted facial action unit 210 is redirected onto the avatar 230 is shown. In the example, the avatar 230 is a two-dimensional (2D) avatar, but it can also be a 3D avatar. The avatar 230 is rendered from the predicted facial action unit 210 of the facial expression 202 of the wearer 102. Therefore, the avatar 230 has the same facial expression 202 as the wearer 102 in the sense that the predicted facial action unit 210 accurately encodes the facial expression 202 of the wearer 102.

[0038] Figure 5An exemplary process 500 is shown for selecting a pre-processing parameter combination 205 for use in process 200 for a particular wearer 102 of the HMD 100. While the wearer 102 exhibits different calibration facial expressions 502, the camera 108 of the HMD 100 captures (504) calibration facial images 506 under current lighting conditions. The facial expressions 502 may include: a neutral facial expression; a left smile facial expression, in which only the left corner of the mouth is curved upward; and a right smile facial expression, in which only the right corner of the mouth is curved upward. For each facial expression 502, there may be one or more facial images 506 captured.

[0039] There are multiple combinations of preprocessing parameters 505. For example, when the image preprocessing technique is CLAHE, there may be multiple combinations of grid size and clipping limit 505. In this case, the clipping limit may be an integer between 0 and 10, and the grid size may be an integer between 2 and 32. Because there are 11 different values ​​of the clipping limit and 31 different values ​​of the grid size, this means that there can be up to 11x31=341 different parameter combinations 505.

[0040] Using each different combination 505 of parameters, each calibration facial image 506 is independently preprocessed (507) to produce a preprocessed calibration facial image 508 for each facial image 506. That is, according to each different combination 505, the preprocessing technique in question is applied to each facial image 506 to produce a preprocessed calibration facial image 508 for each facial image 506. If there are N different combinations 505, this means that, for each calibration facial image 506, N preprocessed calibration facial images 508 are produced. If there are M facial images 506, this means that a total of MxN facial images 508 are produced.

[0041] The trained machine learning model 208 is applied 510 to each pre-processed calibration facial image 508 to produce a set of predicted facial action units 510 for each pre-processed calibration facial image 508. That is, each facial image 508 is input 514 into the model 208, which outputs 516 a set of corresponding predicted facial action units 510 for that image 508. If there are MxN facial images 508, then there are MxN sets of corresponding predicted facial action units 510. The predicted facial action units 510 of the pre-processed calibration facial image 508 correspond to the calibration facial expressions 502 performed by the wearer 102 within the calibration facial image 506 that was pre-processed to produce the pre-processed calibration facial image 508 in question.

[0042] Each set of predicted facial action units 510 can be redirected 528 onto an avatar corresponding to the face 104 of the wearer 102 to render a corresponding avatar facial image 530. In the corresponding avatar facial image 530 rendered using the set of predicted facial action units 510, the avatar exhibits the calibrated facial expressions 502 to which those facial action units 510 correspond. How well the calibrated facial expressions 502 within the avatar facial image 530 match the actual facial expressions 502 exhibited by the wearer 102 depends on the accuracy of the machine learning model 208 in predicting the facial action units 510.

[0043] If there are MxN sets of predicted facial action units 510, there are MxN avatar facial images 530 corresponding to the MxN pre-processed calibration facial images 508. In particular, there are N avatar facial images 530 corresponding to each of the M calibration facial images 506, where each of the N avatar facial images 530 corresponds to a different one of the N combinations 505 of parameters. Similarly, there are M avatar facial images 530 corresponding to each of the N combinations 505, where each of the M avatar facial images 530 corresponds to a different one of the M calibration facial images 506.

[0044] Within each avatar facial image 530, a set of avatar facial landmarks 532 are detected (534). Thus, in the presence of MxN avatar facial images 530, MxN sets of avatar facial landmarks 532 are detected. Within each (unprocessed) calibration facial image 506, a set of wearer facial landmarks 536 are similarly detected (538). Thus, in the presence of M facial images 506, M sets of wearer facial landmarks 536 are detected. In one implementation, the open source PyTorch-based face landmark detection technique described at github.com / cunjian / pytorch_face_landmark can be used to detect the avatar facial landmarks 532 and the wearer facial landmarks 536.

[0045] A set of facial landmarks is a set of facial points that together can define a face that appears in an image. For example, a mouth shape is composed of a subset of the facial points of the lips of a mouth. Different face models can use different sets of facial landmarks. For example, the technique described at https: / / ibug.doc.ic.ac.uk / resources / facial-point-annotations / uses 68 facial landmarks, while the technique described at http: / / www.ifp.illinois.edu / ~vuongle2 / helen / uses 194 facial landmarks. The locations of the facial landmarks according to a specified face model are thus detected within an image to define a face that appears in the image.

[0046] The similarity 540 between each set of avatar facial landmarks 532 and its set of corresponding wearer facial landmarks 536 is calculated 542. For a set of avatar facial landmarks 532 detected within an avatar facial image 530, a set of corresponding wearer facial landmarks 536 is a set of wearer facial landmarks 536 detected within a calibration facial image 506 corresponding to such avatar facial image 530. Such calibration facial image 506 is a calibration facial image that is pre-processed to generate a pre-processed calibration facial image 508 to which the machine learning model 208 is applied to predict facial action units 510 based on which the avatar facial image 530 in question is rendered.

[0047] The similarity 540 between the set of avatar facial landmarks 532 and the set of corresponding wearer facial landmarks 536 represents how well the avatar facial image 530 matches the calibration facial expressions 502 performed by the wearer 102 in the corresponding calibration facial image 506. Because the avatar facial image 530 is generated based on the facial action units 510 predicted by the machine learning model 208 from the pre-processed calibration facial image 508, the similarity 540 therefore also represents how accurate the model 208 is.

[0048] Additionally, because the calibration facial images 506 are preprocessed many times, each time using a different combination of parameters 505, some preprocessed calibration facial images 508 may result in the machine learning model 208 more accurately predicting facial action units 510 than other images 508. Thus, the similarity 540 between a set of avatar facial landmarks 532 and a set of corresponding wearer facial landmarks 536 also represents how accurate the machine learning model 208 was when the calibration facial images 506 were preprocessed using a given combination of parameters 505.

[0049] In one implementation, similarity 540 is calculated using the Kabsch-Umeyama algorithm, which measures the similarity of two misaligned graphs of different scales in a scale-invariant manner. The graph consists of a set of points. The Kabsch-Umeyama algorithm identifies the optimal translation, rotation, and scaling by minimizing the root mean square error (RMSE) between the two sets of points. The RMSE obtained between a set of avatar facial landmarks 532 and a set of corresponding wearer facial landmarks 536 is the similarity 540 between the two sets. The Kabsch-Umeyama algorithm is described in S. Umeyama et al., "Least-squares estimation of transformation parameters between two point patterns," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, iss. 4, pp. 376-380 (April 1991).

[0050] If there are MxN sets of avatar facial landmarks 532, then MxN similarities 540 are calculated. A set of wearer facial landmarks 536 for a given calibration facial image 506 is compared to each set of facial landmarks 532 detected from an avatar facial image 530 rendered using a facial action unit 510 predicted from one of the pre-processed calibration facial images 508 generated from that facial image 506. Thus, there are N similarities 540 calculated for each of the M calibration facial images 506, and there are M similarities 540 calculated for each of the N parameter combinations 505.

[0051] Based on the calculated similarities 540 (544) of the different combinations of parameters 505, scores 544 corresponding to the different combinations of parameters 505 are calculated (546). The score 544 for each of the N parameter combinations 505 can be calculated as the sum of the M similarities 540 calculated for that parameter combination 505. A combination 205 of parameters subsequently used in the process 200 is then selected from the parameter combinations 505 based on its score 544 (548).

[0052] In particular, combination 205 is selected as the combination 505 of pre-processing parameters that produces the highest similarity between avatar facial landmark 532 and wearer facial landmark 536. Depending on whether the computational technique used to produce similarity 540 produces a higher value for a higher similarity or a lower value for a lower similarity, this is the combination 505 with the highest or lowest score 544.

[0053] For a particular wearer 102 of the HMD 100, a combination 205 of pre-processing parameters is thus selected in process 500 for subsequent use in process 200. When the wearer 102 exhibits a different calibration facial expression 502, the combination 205 is selected using a calibration facial image 506 of the wearer 102 captured under the current lighting conditions. If the lighting conditions subsequently change, process 500 may be repeated (automatically or initiated by the wearer 102) to select a different combination 205 from the combination 505 for use in process 200. However, process 500 is performed at least once for each different wearer 102.

[0054] Figure 6 An exemplary method 600 is shown for generating calibration facial images 506 in process 500. A facial expression is set as a first calibration facial expression 502 (602), and the HMD wearer 102 is instructed to express this facial expression (604). While the HMD wearer 102 is expressing the facial expression, one or more of the calibration facial images 506 may be captured (606).

[0055] If there are other calibration facial expressions 502 that the wearer 102 has not been instructed to perform (608), the process is repeated (610) using the next calibration facial expression 502. Once one or more facial images 506 of the wearer 102 have been captured for each calibration facial expression 502, the method 600 ends (612). Thus, at the completion of the method 600, as already described, there may be M total calibration facial expressions.

[0056] Figure 7 Exemplary facial landmarks detected within a mouth image 700 including an upper lip 702A and a lower lip 702B are shown. For example, the mouth image 700 may be a portion of or constitute the entirety of a calibration facial image 506 or an avatar facial image 530, within which wearer facial landmarks 536 or avatar facial landmarks 532, respectively, are detected. In the depicted example, there are facial landmarks 704A, 704B, 704C, 704D, 704E, 704F, and 704G detected on the upper portion of the upper lip 702A. There are facial landmarks 704H, 704I, 704J, 704K, and 704L detected on the lower portion of the lower lip 702B. There are facial markers 704M, 704N, 704O, and 704P detected at the lower portion of the upper lip 702A and facial markers 704Q, 704R, 704S, and 704T detected at the upper portion of the lower lip 702B.

[0057] Figure 8An exemplary non-transitory computer-readable data storage medium 800 storing program code 802 is shown, which can be executed by a processor to perform a process. The processor can be part of the HMD 100 or part of a host computing device to which the HMD 100 is communicatively connected. The process includes: for each of the plurality of facial images 506 of the wearer 102 of the HMD 100, generating a plurality of pre-processed facial images 508 corresponding to the combination 505 of pre-processing parameters (804). The process includes: applying the machine learning model 208 to each pre-processed facial image 508 to predict a facial action unit 510 (806).

[0058] The processing includes redirecting the facial action units 510 predicted from each pre-processed facial image 508 onto the avatar to render one of the plurality of avatar facial images 530 (808). The processing includes detecting avatar facial landmarks 532 within each avatar facial image 530 and detecting wearer facial landmarks 536 within each facial image 506 (810). The processing includes selecting the combination 205 of pre-processing parameters that generates the highest similarity between the avatar facial landmarks 532 and the wearer facial landmarks 536 corresponding to the avatar facial landmarks 532 (812).

[0059] Fig. 9 An exemplary method 900 is shown. The method 900 may be implemented as program code stored on a non-transitory computer-readable data storage medium and executed by a processor. The method 900 includes: capturing a plurality of calibration facial images 506 of a wearer 102 of the HMD 100 using one or more cameras 108 of the HMD 100 while the wearer 102 exhibits a calibration facial expression 502 (902). The method 900 includes: applying a pre-processing technique to each calibration facial image 506 using a plurality of combinations 505 of parameters to generate a plurality of pre-processed calibration facial images 508 corresponding to the combinations 505, respectively (904).

[0060] The method 900 includes applying the machine learning model 208 to each pre-processed calibration facial image 508 to predict a facial action unit 510 (906) corresponding to a calibration facial expression 502 exhibited by the wearer 102 in the calibration facial image 506 corresponding to the pre-processed calibration facial image 508. The method 900 includes redirecting the facial action unit 510 predicted from each pre-processed calibration facial image 508 onto an avatar to render one of a number of avatar facial images 530 in which the avatar exhibits the calibration facial expression 502 exhibited by the wearer 102 in the calibration facial image 506 corresponding to the pre-processed calibration facial image 508 (908).

[0061] The method 900 includes detecting a wearer facial landmark 536 in each calibration facial image 506 according to a specified facial model (910). The method 900 includes detecting an avatar facial landmark 532 in each avatar facial image 530 according to the specified facial model, and the avatar facial landmark 532 corresponds to the wearer facial landmark 536 detected in the calibration facial image 506 corresponding to the avatar facial image 530 (912). The method 900 includes selecting a combination 205 of parameters that generates the highest similarity between the avatar facial landmark 532 and the wearer facial landmark 536 corresponding to the avatar facial landmark 532 (914).

[0062] Fig.10 An exemplary system 1000 is shown. System 1000 is depicted as including an HMD 100 and a computing device 1002. One or more cameras 108 of the HMD 100 are used to capture a set of calibration facial images 506 of a wearer 102 of the HMD 100. The computing device 1002 has: a processor 1004; and a memory 1006 storing program code 1008. For example, the computing device 1002 may be a host device of the HMD 100. However, in another implementation, the processor 1004 and the memory 1006 may be part of the HMD 100. In this case, the processor 1004 and the memory 1006 may be integrated within an application specific integrated circuit (ASIC) so that the processor 1004 is a dedicated processor. The processor 1004 may alternatively be a general purpose processor, such as a central processing unit (CPU), so that the memory 1006 may be a separate semiconductor or other type of volatile or non-volatile memory 1006.

[0063] Program code 1008 is executable by processor 1004 to perform processing. The processing includes: using a plurality of combinations 205 of parameters, applying a pre-processing technique to each calibration facial image 506 to produce pre-processed facial images 508 corresponding to the combinations 205, respectively (1010). The processing includes: applying machine learning model 208 to each pre-processed facial image 508 to predict facial action units 510 (1012), and redirecting the facial action units 510 predicted from each pre-processed facial image 508 onto an avatar to render one of a plurality of avatar facial images 530 (1014). The processing includes: detecting avatar facial landmarks 532 within each avatar facial image 530 and detecting wearer facial landmarks 536 within each facial image 506 (1016). The processing includes: selecting a combination 205 of parameters that generates the highest similarity between the avatar facial landmarks 532 and the wearer facial landmarks 536 corresponding to the avatar facial landmarks 532 (1018).

[0064] Techniques have been described for selecting a combination of preprocessing parameters for preprocessing a captured image of a wearer of an HMD, based on which facial action units corresponding to facial expressions exhibited by the wearer within the image are predicted. The selected combination can result in more accurately predicted facial action units because they are specific to the wearer of the HMD and take into account the lighting conditions when the calibration facial image was captured. If the lighting conditions change, the parameter combination can be reselected to again increase the accuracy of the facial action unit predictions.

Claims

1. A non-transitory computer-readable data storage medium storing program code, the program code being executable by a processor to perform a process, the process comprising: for each of a plurality of facial images of a wearer of a head mounted display (HMD), generating a plurality of preprocessed facial images respectively corresponding to a combination of preprocessing parameters; Applying a machine learning model to each preprocessed face image to predict facial action units; redirecting the facial action units predicted from each pre-processed facial image onto the avatar to render one of the plurality of avatar facial images; detecting avatar facial landmarks within each avatar facial image and detecting wearer facial landmarks within each facial image; and A combination of pre-processing parameters is selected that produces the highest similarity between the avatar facial landmarks and the wearer facial landmarks corresponding to the avatar facial landmarks.

2. The non-transitory computer-readable data storage medium of claim 1, wherein the processing further comprises: preprocessing subsequent facial images of the wearer using the selected combination of preprocessing parameters; applying the machine learning model to the pre-processed subsequent facial image to predict wearer facial action units for the wearer's facial expressions exhibited within the subsequent facial image; and The wearer facial action unit is redirected onto the avatar to render the avatar with the facial expression of the wearer for display.

3. The non-transitory computer-readable data storage medium of claim 2, wherein the processing further comprises: Displays the rendered avatar.

4. The non-transitory computer-readable data storage medium of claim 1 , wherein the processing further comprises: instructing the wearer to exhibit specified different calibrated facial expressions; and One or more of the facial images are captured using one or more cameras of the HMD while the wearer is performing each of the specified different calibration facial expressions.

5. The non-transitory computer-readable data storage medium of claim 1 , wherein generating a pre-processed facial image for each facial image comprises: independently applying a preprocessing technique to the facial image a plurality of times to produce a preprocessed facial image, Each time the preprocessing technique is applied, a different combination of preprocessing parameters is used.

6. The non-transitory computer-readable data storage medium of claim 1, further comprising: for each avatar facial image, calculating a similarity between avatar facial landmarks detected in the avatar facial image and wearer facial landmarks detected in a facial image corresponding to the avatar facial image; and calculating a score for each combination of pre-processing parameters based on the similarity of each avatar face image corresponding to the combination of pre-processing parameters, The combination of pre-processing parameters having the highest or lowest score is selected as the combination of pre-processing parameters that generates the highest similarity.

7. A method comprising: capturing, using one or more cameras of a head mounted display (HMD), a plurality of calibration facial images of the wearer of the HMD while the wearer exhibits calibration facial expressions; applying a preprocessing technique to each calibration facial image using a plurality of combinations of parameters to produce a plurality of preprocessed calibration facial images respectively corresponding to the combinations; applying a machine learning model to each preprocessed calibration facial image to predict a facial action unit corresponding to a calibration facial expression performed by the wearer in the calibration facial image corresponding to the preprocessed calibration facial image; redirecting facial action units predicted from each pre-processed calibration facial image onto an avatar to render one of a plurality of avatar facial images in which the avatar exhibits a calibration facial expression exhibited by the wearer in the calibration facial image corresponding to the pre-processed calibration facial image; detecting the wearer's facial landmarks in each calibration facial image according to the specified facial model; detecting, in each avatar facial image, avatar facial landmarks according to the specified facial model, and the avatar facial landmarks correspond to wearer facial landmarks detected in a calibration facial image corresponding to the avatar facial image; and The combination of parameters is selected that generates the highest similarity between the avatar facial markings and the wearer facial markings corresponding to the avatar facial markings.

8. The method of claim 7, further comprising: When the wearer shows a facial expression, capturing a facial image using the camera of the HMD; applying the preprocessing technique to the captured facial image using the selected combination of parameters; applying the machine learning model to the pre-processed captured facial image to predict wearer facial action units for facial expressions exhibited by the wearer within the captured facial image; and The wearer facial action unit is redirected onto the avatar to render the avatar with the facial expression of the wearer for display.

9. The method of claim 7, wherein the calibration facial expressions include a neutral facial expression, a left smiling facial expression, and a right smiling facial expression, And wherein for each calibration facial expression, one or more of the calibration facial images are captured.

10. The method of claim 7, wherein the preprocessing technique comprises contrast limited adaptive histogram equalization, And wherein the pre-processing parameters include clipping limits and grid size.

11. The method of claim 7, further comprising: for each avatar facial image, calculating a similarity between avatar facial landmarks detected in the avatar facial image and wearer facial landmarks detected in a calibration facial image corresponding to the avatar facial image; and calculating a score for each combination of parameters based on the similarity of each avatar face image corresponding to the combination of parameters, The combination of parameters having the highest or lowest score is selected as the combination of parameters generating the highest similarity.

12. A system comprising: a head mounted display (HMD) having one or more cameras to capture a calibration facial image of a wearer of the HMD while the wearer exhibits a calibration facial expression; processor; and A memory storing program code, wherein the program code is executable by the processor to: applying a preprocessing technique to each calibration facial image using a plurality of combinations of parameters to produce preprocessed facial images respectively corresponding to the combinations; Applying a machine learning model to each preprocessed face image to predict facial action units; redirecting the facial action units predicted from each pre-processed facial image onto the avatar to render one of the plurality of avatar facial images; detecting avatar facial landmarks within each avatar facial image and detecting wearer facial landmarks within each facial image; and The combination of parameters is selected that generates the highest similarity between the avatar facial markings and the wearer facial markings corresponding to the avatar facial markings.

13. The system of claim 12, wherein the program code is executable by the processor to further: applying the pre-processing technique to a subsequently captured facial image of the wearer using the selected combination of parameters; applying the machine learning model to a subsequently captured facial image that has been preprocessed to predict wearer facial action units for facial expressions exhibited by the wearer within the captured facial image; and The wearer facial action unit is redirected onto the avatar to render the avatar with the facial expression of the wearer for display.

14. The system of claim 12, wherein the preprocessing technique comprises contrast limited adaptive histogram equalization, And wherein the pre-processing parameters include clipping limits and grid size.

15. The system of claim 12, wherein the program code is executable by the processor to further: For each avatar facial image, calculating a similarity between avatar facial landmarks detected in the avatar facial image and wearer facial landmarks detected in a calibration facial image corresponding to the avatar facial image; and calculating a score for each combination of parameters based on the similarity of each avatar face image corresponding to the combination of parameters, The combination of parameters having the highest or lowest score is selected as the combination of parameters generating the highest similarity.

Images