Automatic capture of neutral facial expressions

Abstract

Neutral facial expressions of the wearer of the head-mounted display (HMD) are detected based on signals received from one or more facial electromyography (fEMG) sensors of the HMD. A baseline neutral facial expression image of the wearer is automatically captured responsively using one or more cameras of the HMD. HMD camera-based detection of the wearer's facial expressions is calibrated based on the automatically captured baseline neutral facial expression image.

Description

Background Technology

[0001] Extended reality (XR) technology encompasses virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies, and more precisely, extends the reality of the user experience. XR technology can utilize head-mounted displays (HMDs). An HMD is a display device that can be worn on the head. In VR, the HMD wearer is immersed in a completely virtual world, while in AR, the physical reality environment seen directly or indirectly by the HMD wearer is enhanced. In MR or mixed reality, the HMD wearer experiences a fusion of the real and virtual worlds. Attached Figure Description

[0002] Figure 1A , 1B 1C and 1C are the perspective view, front view, and frame view of an example head-mounted display (HMD) that can be used in an extended reality (XR) environment, respectively.

[0003] Figure 2 This is a diagram of an example non-transitory computer-readable data storage medium storing program code for calibrating HMD camera-based facial expression detection using baseline neutral facial expression images automatically captured in response to neutral facial expression detection based on a facial electromyography (fEMG) sensor.

[0004] Figure 3A and 3B This is a diagram of different example methods for detecting neutral facial expressions based on signals from one or more fEMG sensors. Detailed Implementation

[0005] As noted in the background section, head-mounted displays (HMDs) can employ extended reality (XR) technology, where the wearer experiences an extended reality experience. An HMD may include small display panels in front of each of the wearer's eyes, and various sensors used to detect or sense the wearer and / or the wearer's environment. The images on the display panels convincingly immerse the wearer in an XR environment, whether it is virtual reality (VR), augmented reality (AR), mixed reality (MR), or other types of XR.

[0006] An HMD (Head-Down Device) can include one or more cameras, which are image capture devices that capture still or moving images. For example, one camera on the HMD can be used to capture an image of the wearer's lower face (including the mouth). Two other cameras on the HMD can be used to capture images of the wearer's eyes and the portion of the wearer's face around the eyes.

[0007] Based on the captured images, the wearer's facial expressions and corresponding derived wearer information, such as facial cues and emotions, can be assessed. Detecting the wearer's facial features within the HMD provides even more complete immersion within the XR. For example, the XR can appropriately respond to the wearer's facial expressions, facial cues, and emotions, including altering any graphical representation of the wearer within the XR, such as an avatar, in accordance with changes in the wearer's actual facial expressions.

[0008] For accuracy, such HMD camera-based facial expression detection should be calibrated for the current wearer of the HMD before actual detection of the wearer's facial expressions based on images captured by the camera. Calibration is performed using images of the HMD wearer's neutral facial expressions. That is, once a neutral facial expression image has been captured, subsequently captured images can be compared to it to identify the wearer's current facial expression. This neutral facial expression image is thus used as a baseline image.

[0009] The capture of neutral facial expression images is typically initiated manually with the wearer's participation and interaction. After the user mounts the HMD on his or her head, the user, now the wearer of the HMD, may be prompted to exhibit a neutral facial expression, and input is provided once the wearer exhibits a neutral facial expression. In response to the received input, the HMD captures an image of the wearer's face, which is assumed to be an image of the wearer's neutral facial expression. Facial expression detection based on the HMD camera can then be calibrated based on the captured image.

[0010] However, manually initiating such camera-based facial expression detection calibration can be cumbersome for wearers. Wearers may ignore the prompts to capture neutral facial expression images, leading to a less than optimal XR experience. Wearers may exhibit facial expressions they believe to be neutral, but which are actually forced simulations of the wearer's actual neutral facial expressions, negatively impacting subsequent camera-based facial expression detection. At the very least, manually initiating camera-based facial expression detection calibration is inconvenient for HMD wearers who may desire a minimal setup experience for XR.

[0011] The technique described in this paper provides the automatic capture of baseline neutral facial expression images of HMD wearers, upon which HMD camera-based detection of the wearer's facial expressions can be calibrated. This technique utilizes the HMD's facial electromyography (fEMG) sensor. The fEMG sensor outputs a signal measuring facial muscle activity by detecting and amplifying the small electrical impulses generated by muscle fibers when they contract. A baseline neutral facial image of the wearer is automatically captured when detecting neutral facial expressions based on such fEMG sensor signals.

[0012] Therefore, HMD camera-based detection of facial expressions can be calibrated without the wearer explicitly initiating calibration. That is, neutral facial expressions are detected without explicit prompting the wearer to first exhibit them, and a baseline neutral facial expression image is automatically captured responsively. The wearer may not even be aware that such a baseline neutral facial expression has been captured. For example, such capture may occur within a short period after the wearer first puts on the HMD, such as within the first few minutes of wearing the HMD.

[0013] Therefore, the technique described herein utilizes the HMD's fEMG sensor solely for the purpose of automatically capturing a baseline neutral facial expression image, which can then be used for subsequent camera-based facial expression detection. The fEMG sensor is not used to detect facial expressions of the HMD wearer beyond the initial (e.g., a single) detection of neutral facial expressions intended for calibrating camera-based facial expression detection. While the fEMG sensor can be used for facial expression detection, in some cases, such a technique may be less accurate than camera-based detection and may be computationally more intensive and complex.

[0014] Figure 1A , 1B Figures 1 and 1C show perspective, front, and frame views of an example HMD 100 worn by and positioned against the face 104 of wearer 102. The HMD 100 includes a body 105 with a washer 106 at one end, and is positioned against the face 104 of wearer 102 above the nose 151 and around the eyes 152A and 152B of wearer 102 (collectively referred to as the eyes 152 of wearer 102). Figure 1B The gasket 106 can be made of a soft, flexible material (such as rubber foam) that can conform to the contours of the wearer's face 104 to block ambient light from entering the interior of the body 105 at the interface between the gasket 106 and the wearer's face 104. The gasket 106 further enhances the comfort of the wearer 102 when using the HMD 100 because, unlike the gasket 106, the body 105 itself can be made of a rigid material such as plastic and / or metal.

[0015] HMD 100 may include a display panel 107 located inside the other end of the main body 105, and the display panel 107 may be positioned to be incident on the eyes 152 of the wearer 102. The display panel 107 may actually include a right display panel and a left display panel, the right display panel being incident on and viewable by the wearer 102's right eye 152A, and the left display panel being incident on and viewable by the wearer 102's left eye 152B. By appropriately displaying images on the display panel 107, HMD 100 can immerse the wearer 102 in XR.

[0016] HMD 100 may include an adjustable strap 108 that can be tightened once HMD 100 has been placed on wearer 102 to hold gasket 106 in a fixed position against wearer 102’s face 104 to prevent hand-held contact. The depicted strap 108 is a strap behind the head. HMD 100 may also include another strap, such as a strap above the head, to hold HMD 100 in position on wearer 102 even more securely.

[0017] HMD 100 may include eye cameras 110A and 110B and / or mouth camera 110C, collectively referred to as cameras 110. Although only one mouth camera 110C is shown, multiple mouth cameras 110C may be present. Similarly, although only one eye camera 110A and one eye camera 110B are shown, multiple eye cameras 110A and / or multiple eye cameras 110B may be present. Cameras 110 capture images of different parts of the face 104 of the wearer 102 of HMD 100, which can be used to detect facial expressions of the wearer 102.

[0018] Eye cameras 110A and 110B are located inside the body 105 of the HMD 100 and are pointed towards their respective eyes 152. The right eye camera 110A captures images including the wearer's right eye 152A and the surrounding facial portion, while the left eye camera 110B captures images including the wearer's left eye 152B and the surrounding facial portion. The mouth camera 110C is exposed outside the body 105 of the HMD 100 and is pointed towards the wearer's mouth 154 (according to...). Figure 1B (To capture an image including the mouth 154 of the wearer 102 and the lower part of the surrounding area.)

[0019] HMD 100 may include fEMG sensors 156A, 156B, 156C, and 156D (collectively referred to as fEMG sensors 156) and fEMG sensors 158A, 158B, 158C, and 158D (collectively referred to as fEMG sensors 158). fEMG sensors 156 and 158 are disposed within a gasket 106 of HMD 100. fEMG sensors 156 and 158 are exposed externally at gasket 106 such that when HMD 100 is worn by wearer 102, sensors 156 and 158 are in contact with the skin of wearer 102's face 104.

[0020] fEMG sensor 156 is positioned above the wearer's eye 152 and can therefore be referred to as the over-eye fEMG sensor 156, while fEMG sensor 158 is positioned below the wearer's eye 152 and can therefore be referred to as the under-eye fEMG sensor 158. In the depicted example, there are four fEMG sensors 156 and four fEMG sensors 158, such that there are two fEMG sensors 156 and two sensors 158 around each eye 152 of the wearer 102. If lower sensitivity is sufficient, there may be only one fEMG sensor 156 and one fEMG sensor 158 for each eye 152, in which case the sensors 156 and 158 for the eye 152 may be located at the center above and below the eye 152, respectively. If even lower sensitivity is sufficient, there may be only one fEMG sensor 156 and one fEMG sensor 158, in which case sensors 156 and 158 may be located above and below the center of the wearer's eyes 152, respectively.

[0021] fEMG sensors 156 and 158 output signals measuring the facial muscle activity of the wearer 102 and can be used to detect neutral facial expressions of the wearer 102. The over-eye fEMG sensor 156 specifically outputs signals measuring muscle activity primarily in the upper facial portion above the wearer 102's eyes 152, while the under-eye fEMG sensor 158 outputs signals measuring muscle activity primarily in the lower facial portion below the wearer 102's eyes 152. Therefore, fEMG sensor 156 can be used to detect neutral facial expressions in the upper facial portion of the wearer 102, and fEMG sensor 158 can be used to detect neutral facial expressions in the lower facial portion of the wearer 102.

[0022] fEMG sensors 156 and 158 may be able to independently detect neutral facial expressions on the upper and lower parts of the wearer 102's face, respectively. That is, fEMG sensor 156 may be able to detect neutral facial expressions on the upper part of the face independently of fEMG sensor 158, which detects neutral facial expressions on the lower part, and vice versa. Furthermore, if only neutral facial expressions on the upper or lower part of the wearer 102 are of interest, the HMD 100 may accordingly include only fEMG sensors 156 or 158. Therefore, in this case, the HMD 100 may similarly include only eye cameras 110A and 110B or only a mouth camera 110C.

[0023] Figure 2 An example non-transitory computer-readable data storage medium 200 is shown storing program code 202 that can be executed by a processor to perform processing. The processor may be part of HMD 100, or may be part of a computing device to which HMD 100 is communicatively connected in a wired or wireless manner. The processing performed by the processor via executing the program code 202 stored on the computer-readable data storage medium 200 can be considered a method or process.

[0024] The processing includes detecting neutral facial expressions of the wearer 102 of the HMD 100 based on signals received from one or more fEMG sensors 156 and 158 of the HMD 100 (204). Two example ways in which neutral facial expressions can be detected based on signals from fEMG sensors 156 and 158 are described later in the detailed description. Most generally, the overall neutral facial expression of the wearer 102 can be detected. However, as noted above, neutral facial expressions of the upper and lower facial portions can be detected independently (even at different times), and neutral facial expressions of only the upper or lower facial portions can be detected.

[0025] The process includes responsively (i.e., in response to the detection of a neutral facial expression by fEMG sensors 156 and 158) automatically capturing a baseline neutral facial expression image of the wearer 102 using camera 110 of HMD 100 (206). The capture of the baseline neutral facial expression image is automatic, in which the wearer 102 does not need to explicitly initiate image capture, nor does the wearer 102 need to be explicitly prompted to exhibit a neutral facial expression. Rather, the baseline neutral facial expression is automatically captured once the fEMG sensors 156 and 158 detect an image of the wearer 102 exhibiting a neutral facial expression. As noted above, the wearer 102 may not even be aware that such image capture is taking place, especially in the context of facial expression detection calibration based on camera 110.

[0026] The process includes calibrating camera 110-based facial expression detection (208) based on captured baseline neutral facial expression images. In one embodiment, calibration may include storing only the captured baseline neutral facial expression images for subsequent use in detecting the facial expressions of the wearer 102 of the HMD 100 based on facial images captured by camera 110. In other embodiments, where camera 110-based facial expression detection involves algorithms or models whose parameters must be changed or otherwise adjusted for a given baseline neutral facial expression image, calibration may include changing such parameters or otherwise adjusting the algorithm or model based on the captured images. As noted above, camera 110-based facial expression detection calibration can occur without the wearer 102 initiating calibration and without the wearer 102 even being aware that calibration is taking place.

[0027] The process may then include detection (210) of facial expressions of the wearer 102 of the HMD 100 based on the (calibrated) camera 110. Upper and lower facial expressions of the wearer 102 may be detected independently, such as in the case of capturing neutral upper and lower facial expressions in separate baseline images in response to independent detection of neutral facial expressions of the upper and lower facial portions of the wearer 102. Similarly, if the HMD 100 does not include a mouth camera 110C, the detected facial expressions may include only upper facial expressions using images from only the eye cameras 110A and 110B. Likewise, if the HMD 100 does not include eye cameras 110A and 110B, the detected facial expressions may include only lower facial expressions using images from only the mouth camera 110C.

[0028] Facial expressions of the wearer 102 of the HMD 100 can be detected in a method based on (calibrated) camera 110 as follows. After detection calibration based on camera 110, camera 110 captures facial images of the wearer 102 (212). Facial expressions of the wearer 102 changing over time can then be detected by comparing these subsequently captured facial images with previously captured neutral facial expression images (214). For example, a calibrated algorithm or model can be applied to the subsequently captured facial images of the wearer 102. Baseline neutral facial expression images can also be simultaneously input into the algorithm or model, especially if the algorithm or model has not been previously adjusted based on baseline neutral facial expression images.

[0029] The processing may include performing one or more actions (216) related to the HMD 100 based on detected facial expressions of the wearer 102. For example, the wearer 102's emotions may be predicted based on the detected facial expressions. As another example, the image displayed by the HMD 100 on the display panel 107 for viewing by the wearer 102 of the HMD 100 may be changed based on detected facial expressions of the wearer 102.

[0030] As a concrete example of the latter, the facial expression displayed on the avatar corresponding to wearer 102 can change in accordance with the actual facial expression detected as a change in wearer 102. The avatar can be additionally or alternatively displayed to users other than wearer 102, such as in situations where wearer 102 is participating in a video conference or other collaborative activity with other users. In this case, the action in question still relates to HMD 100, as long as the action is somehow related to wearer 102 of HMD 100.

[0031] Figure 3A and 3B Different methods 300 and 350 are shown for detecting neutral facial expressions of the wearer 102 of HMD 100 based on signals captured from fEMG sensors 156 and 158 of HMD 100. Each method 300 and 350 can be used to implement... Figure 2 Part 204. Other methods may also be used to detect neutral facial expressions of the wearer 102 to automatically trigger the capture of a baseline neutral facial expression image by the camera 110 of the HMD 100.

[0032] exist Figure 3A In this process, the variation of each fEMG signal within a specified time period is determined (302). Typically, when the wearer 102 exhibits a neutral facial expression, the wearer 102's facial muscles are relaxed. Therefore, the signals received from fEMG sensors 156 and 158 will vary minimally around a baseline value, which may be unique to the wearer 102. The actual value of the signal and therefore the baseline value of the signal may not be of interest; rather, the variation of each signal is of interest. Therefore, if the variation of the signal from each fEMG sensor 156 and 158 is less than a threshold (304), it is concluded that the wearer 102 is currently exhibiting a neutral facial expression (308). Otherwise, it is concluded that the wearer 102 is not currently exhibiting a neutral facial expression (306), and method 300 is repeated in the next specified time period.

[0033] The duration of the time period under discussion can be several seconds or longer. If the upper and lower facial expressions of the wearer 102 are to be detected independently, neutral facial expressions of the wearer 102 can also be detected independently. For example, method 300 can be performed to detect neutral upper facial expressions of the wearer 102 based solely on signals from the above-eye sensor 156, and method 300 can be performed separately (e.g., simultaneously) to detect neutral lower facial expressions of the wearer 102 based solely on signals from the below-eye sensor 158.

[0034] exist Figure 3B In this process, signals received from fEMG sensors 156 and 158 during a specified time period are received (352), and a previously trained probability model is applied to these fEMG signals to generate a probability that the wearer 102 exhibits a neutral facial expression (354). The probability model can be a machine learning model, such as a supervised neutral network trained on facial images that have been labeled as representing a neutral facial expression. If the probability of a neutral facial expression is greater than a threshold (356), it is concluded that the wearer 102 is currently exhibiting a neutral facial expression (362). Otherwise, it is concluded that the wearer 102 is not currently exhibiting a neutral facial expression (360), and method 350 is repeated in the next specified time period.

[0035] As in method 300, in method 350, the length of the time period under discussion can be several seconds or longer. Also as in method 300, it may be possible to detect neutral upper facial expressions and neutral lower facial expressions of the wearer 102 independently of each other. Different trained probabilistic models may exist for determining the probability of a neutral facial expression for each of the upper and lower facial portions of the wearer 102. In another embodiment, the same trained probabilistic model can be used in each case.

[0036] A technique for automatically capturing baseline neutral facial expression images of HMD wearers has been described, upon which HMD camera-based facial expression detection can be calibrated. This automatic capture is triggered by detection of neutral facial expressions by an fEMG sensor-based sensor of the wearer. Therefore, it is not necessary to explicitly prompt the HMD wearer to exhibit neutral facial expressions, and it is not necessary to manually initiate calibration of HMD camera-based facial expression detection.

Claims

1. A non-transitory computer-readable data storage medium storing program code executable by a processor to perform processing, the processing comprising: Neutral facial expressions of the wearer of the head-mounted display (HMD) are detected based on signals received from one or more facial electromyography (fEMG) sensors in the HMD. One or more cameras of the HMD are used to responsively and automatically capture baseline neutral facial expression images of the wearer; as well as HMD camera-based detection of the wearer's facial expressions is calibrated based on automatically captured baseline neutral facial expression images.

2. The non-transitory computer-readable data storage medium according to claim 1, wherein, The process also includes: Subsequently, the wearer's facial expressions are detected using a calibrated HMD camera; and Actions related to the HMD are performed based on the detected facial expressions of the wearer.

3. The non-transitory computer-readable data storage medium according to claim 2, wherein, Performing actions related to the HMD based on detected facial expressions of the wearer includes: The displayed image is altered based on the detected facial expressions of the wearer.

4. The non-transitory computer-readable data storage medium according to claim 1, wherein, The process also includes: The HMD's camera is then used to capture an image of the wearer's face; The wearer's facial expressions are detected by comparing captured images of the wearer's face with automatically captured baseline neutral facial expression images; and Actions related to the HMD are performed based on the detected facial expressions of the wearer.

5. The non-transitory computer-readable data storage medium according to claim 4, wherein, Performing actions related to the HMD based on detected facial expressions of the wearer includes: The displayed image is altered based on the detected facial expressions of the wearer.

6. The non-transitory computer-readable data storage medium according to claim 1, wherein, The wearer's neutral facial expressions are detected, and a baseline image of the wearer's neutral facial expressions is automatically captured responsively without explicitly prompting the wearer to exhibit such expressions.

7. The non-transitory computer-readable data storage medium according to claim 1, wherein, The calibration is performed based on the detection of facial expressions of the wearer using an HMD camera without the wearer having to explicitly initiate the calibration.

8. The non-transitory computer-readable data storage medium according to claim 1, wherein, The camera of the HMD includes one or more eye cameras, and the fEMG sensor of the HMD includes one or more fEMG sensors positioned above the eyes of the wearer of the HMD. Furthermore, the detected neutral facial expressions of the wearer include neutral expressions on the upper part of the wearer's face.

9. The non-transitory computer-readable data storage medium according to claim 1, wherein, The camera of the HMD includes one or more mouth cameras, and the fEMG sensor of the HMD includes one or more fEMG sensors positioned below the eyes of the wearer of the HMD. Furthermore, the detected neutral facial expressions of the wearer include neutral expressions on the lower part of the wearer's face.

10. The non-transitory computer-readable data storage medium according to claim 1, wherein, The HMD's cameras include one or more eye cameras and one or more mouth cameras, and the HMD's fEMG sensors include one or more fEMG sensors positioned above the wearer's eyes and one or more fEMG sensors positioned below the wearer's eyes. Furthermore, the detected neutral facial expressions of the wearer include independently detectable neutral expressions of the lower part of the wearer and neutral expressions of the upper part of the wearer's face.

11. The non-transitory computer-readable data storage medium according to claim 1, wherein, Detecting neutral facial expressions of the wearer of the HMD based on signals received from the fEMG sensor of the HMD includes: Detecting that the change in each signal is less than the change in a threshold within a specified time period; and In response to detecting that the change in each signal within the specified time period is less than the threshold change, it is concluded that the wearer of the HMD has the neutral facial expression.

12. The non-transitory computer-readable data storage medium according to claim 1, wherein, Detecting neutral facial expressions of the wearer of the HMD based on signals received from the fEMG sensor of the HMD includes: The trained probabilistic model is applied to the signal received from the fEMG signal of the HMD to generate the probability that the wearer of the HMD will exhibit the neutral facial expression; and In response to a probability greater than a threshold, it is concluded that the wearer of the HMD has the neutral facial expression.

13. A head-mounted display (HMD), comprising: The display panel can be positioned to be incident on the eyes of the wearer of the HMD; A washer, which can be positioned around the wearer's eyes against the wearer's face; One or more facial electromyography (fEMG) sensors are disposed in the spacer to detect neutral facial expressions of the wearer; as well as One or more cameras are used to automatically capture a baseline neutral facial expression image of the wearer in response to the detection of the neutral facial expression by the fEMG sensor, and to calibrate the detection of the wearer's facial expression based on the automatically captured baseline neutral facial expression image using an HMD camera.

14. The HMD according to claim 13, wherein, The camera includes one or more eye cameras, and the HMD's fEMG sensor includes one or more fEMG sensors positioned above the eyes of the wearer of the HMD. Furthermore, the detected neutral facial expressions of the wearer include neutral expressions on the upper part of the wearer's face.

15. The HMD according to claim 13, wherein, The camera of the HMD includes one or more mouth cameras, and the fEMG sensor of the HMD includes one or more fEMG sensors positioned below the eyes of the wearer of the HMD. Furthermore, the detected neutral facial expressions of the wearer include neutral expressions on the lower part of the wearer's face.

Images