Face and eye tracking and facial animation using facial sensors inside a head-mounted display

A display and head-mounted technology, which is applied in the field of head-mounted displays, can solve the problems that the face tracking system is not suitable for use with virtual reality headsets

Active Publication Date: 2020-03-13
CTRL LABS CORP
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Therefore, existing face tracking systems are not suitable for use in portable, lightweight and high-performance virtual reality headsets

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Face and eye tracking and facial animation using facial sensors inside a head-mounted display
  • Face and eye tracking and facial animation using facial sensors inside a head-mounted display
  • Face and eye tracking and facial animation using facial sensors inside a head-mounted display

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0131] System Overview

[0132] figure 1 is a block diagram of a VR system 100 according to an embodiment. In some implementations, the VR system 100 operates in an augmented reality (AR) and / or mixed reality (MR) environment. figure 1 The system 100 shown in includes a head mounted display (HMD) 105 , an imaging device 135 , and a VR input interface 140 each coupled to a console 110 . although figure 1 An exemplary system 100 is shown including one HMD 105 , one imaging device 135 , and one VR input interface 140 , but in other implementations, any number of these components are included in the system 100 . For example, there may be multiple HMDs 105 , each having an associated VR input interface 140 and monitored by one or more imaging devices 135 , and each HMD 105 , VR input interface 140 , and imaging device 135 are in communication with console 110 . In alternative configurations, different and / or additional components may be included in system 100 .

[0133] HMD ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

A head mounted display (HMD) in a VR system includes sensors for tracking the eyes and face of a user wearing the HMD. The VR system records calibration attributes such as landmarks of the face of the user. Light sources illuminate portions of the user's face covered by the HMD. In conjunction, facial sensors capture facial data. The VR system analyzes the facial data to determine the orientation of planar sections of the illuminated portions of face. The VR system aggregates planar sections of the face and maps the planar sections to landmarks of the face to generate a facial animation of the user, which can also include eye orientation information. The facial animation is represented as a virtual avatar and presented to the user. A facial tracking system generates a virtual rendering of a portion of a face of a user wearing a head-mounted display (HMD). The facial tracking system illuminates portions of the face inside the HMD. The facial tracking system captures a plurality of facial data of the portion of the face using one or more facial sensors located inside the HMD. A plurality of planar sections of the portion of the face are identified based at least in part on the plurality of facial data. The plurality of planar sections are mapped to one or more landmarks of the face. Facial animation information is generated based at least in part on the mapping, the facial animation information describing a portion of a virtual face corresponding to the portion of the user's face.

Description

technical field [0001] The present disclosure relates generally to head mounted displays (HMDs), and in particular to eye and face tracking within HMDs. The present disclosure further relates to virtual rendering, and in particular to virtual animation of portions of a user's face within an HMD. Background technique [0002] Virtual reality (VR) systems generally include a display screen that presents a virtual reality image that may depict elements such as objects and users of the system. A user may be represented by an avatar in the virtual environment. In some VR systems, the avatar is depicted with only one facial expression (eg, a default smile or a neutral facial expression), which prevents the user from having a fully immersive experience in the virtual environment. Facial tracking systems provide a more immersive interface. Existing systems for tracking a user's facial expressions include specialized peripherals, such as cameras, in addition to markers that must b...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F3/01G02B27/01
CPCG02B27/0093G02B27/0172G02B2027/0138G06F3/011G06F3/012G06F3/013G02B27/01G02B27/017
Inventor 多夫·卡茨迈克尔·约翰·托克斯韦格王梓衡蒂莫西·保罗·欧梅尼克托林·罗斯·赫恩登
Owner CTRL LABS CORP
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap