User authentication based on three-dimensional face modeling using partial face images

Abstract

Systems and techniques are provided for performing user authentication. For example, a process may include obtaining a plurality of images associated with a face and facial expressions of a user, where each respective image of the plurality of images includes a different portion of the face. An encoder neural network may be used to generate one or more predicted three-dimensional (3D) face modeling parameters, wherein the encoder neural network generates the one or more predicted 3D face modeling parameters based on the plurality of images. A reference 3D face model associated with the face and the facial expression may be obtained. An error between the one or more predicted 3D face modeling parameters and the reference 3D face model may be determined, and the user may be authenticated based on the error being less than a predetermined authentication threshold.

Description

Technical Field

[0001] The present disclosure generally relates to systems and techniques for generating three-dimensional (3D) models. For example, aspects of the present disclosure relate to user authentication based on 3D facial modeling using partial facial images. Background Art

[0002] Many devices and systems allow for capturing a scene by generating frames (also referred to as images) and / or video data (including multiple images or frames) of the scene. For example, a camera or a computing device including a camera (e.g., a mobile device (such as a mobile phone or smartphone) including one or more cameras) can capture a sequence of frames of a scene. Frames and / or video data can be captured and processed by such devices and systems (e.g., mobile devices, IP cameras, etc.) and can be output for consumption (e.g., displayed on the device and / or other devices). In some cases, frames and / or video data can be captured by such devices and systems and output for processing and / or consumption by other devices.

[0003] Frames may be processed (e.g., using object detection, recognition, segmentation, etc.) to determine objects present in the frames, which may be useful for many applications. For example, a model may be determined for representing objects in the frames, and the model may be used to facilitate efficient operation of various systems. Examples of such applications and systems include computer graphics, extended reality (e.g., augmented reality, virtual reality, etc.), image recognition (e.g., facial recognition, object recognition, scene recognition, etc.), and feature estimation, among others. Summary of the invention

[0004] Disclosed are systems, devices, methods, and computer-readable media for generating one or more models (e.g., one or more 3D facial models). According to at least one example, a method for generating a 3D facial model is provided. The method includes: obtaining multiple images associated with a user's face and facial expressions, wherein each corresponding image in the multiple images includes a different portion of the face; generating one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the multiple images; obtaining a reference 3D facial model associated with the face and the facial expression; determining an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and authenticating the user based on the error being less than a predetermined authentication threshold.

[0005] In another example, a device is provided, the device including at least one memory (e.g., configured to store data) and at least one processor (e.g., implemented in a circuit), the at least one processor being coupled to the at least one memory. The at least one processor is configured and may: obtain a plurality of images associated with a face and a facial expression of a user, wherein each respective image in the plurality of images includes a different portion of the face; generate one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images; obtain a reference 3D facial model associated with the face and the facial expression; determine an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and authenticate the user based on the error being less than a predetermined authentication threshold.

[0006] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which instructions, when executed by one or more processors, cause the one or more processors to: obtain a plurality of images associated with a face and a facial expression of a user, wherein each respective image in the plurality of images includes a different portion of the face; generate one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images; obtain a reference 3D facial model associated with the face and the facial expression; determine an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and authenticate the user based on the error being less than a predetermined authentication threshold.

[0007] In another example, an apparatus is provided that includes: a component for obtaining a plurality of images associated with a face and a facial expression of a user, wherein each respective image in the plurality of images includes a different portion of the face; a component for generating one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images; a component for obtaining a reference 3D facial model associated with the face and the facial expression; a component for determining an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and a component for authenticating the user based on the error being less than a predetermined authentication threshold.

[0008] In some aspects, one or more of the devices described above are or are part of: a camera, a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a vehicle or a computing system or device of a vehicle, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device. In some aspects, a device includes one or more cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors that can be used to determine the position and / or posture of the device, the state of the device, and / or for other purposes.

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Exemplary aspects of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1 illustrates an example three-dimensional (3D) facial model and a corresponding two-dimensional (2D) facial image projected from the 3D facial model overlaid with landmarks according to some examples;

[0013] Figure 2 An example head-mounted extended reality (XR) system with a user-facing camera for generating a 3D facial model according to some examples is illustrated;

[0014] Figure 3A illustrates example partial facial images according to some examples;

[0015] Figure 3B Examples of detailed 3D dimensional face models according to some examples are illustrated;

[0016] Figure 4 is a diagram illustrating an example of a 3D modeling system according to some examples;

[0017] Figure 5Ais a block diagram illustrating an example of training a shape-expression encoder to generate predicted shape and expression coefficients for a 3D face model according to some examples;

[0018] Figure 5B is a block diagram illustrating an example of inference using a trained shape-expression encoder according to some examples;

[0019] Figure 6 is a block diagram illustrating an example of training a pose encoder to generate predicted pose information according to some examples;

[0020] Figure 7 is a block diagram illustrating an example of a Siamese network that may be trained to determine a user identity match or mismatch based on a mouth image according to some examples;

[0021] Figure 8 is a flow chart illustrating an example of a process for authenticating a user according to some examples;

[0022] Fig. 9 is a block diagram illustrating an example of a deep learning network according to some examples;

[0023] Fig.10 is a block diagram illustrating an example of a convolutional neural network according to some examples; and

[0024] Fig.11 is a block diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0025] Certain aspects of the present disclosure are provided below. Some of these aspects may be applied independently, and some of them may be applied in combination, which will be apparent to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it will be apparent that various aspects may be implemented without these specific details. Each of the drawings and descriptions is not intended to be restrictive.

[0026] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a description that can be used to implement the example aspects to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0027] The generation of three-dimensional (3D) models of physical objects can be used in many systems and applications, such as for extended reality (XR) (e.g., including augmented reality (AR), virtual reality (VR), mixed reality (MR), etc.), robotics, automobiles, aviation, 3D scene understanding, object grasping, object tracking, and many other systems and applications. In an AR environment, for example, a user can view an image (also referred to as a frame) that includes an integration of artificial or virtual graphics with the user's natural environment. AR applications allow real images to be processed to add virtual objects to the image or display virtual objects on a perspective display (so that the virtual objects appear to be overlaid on the real world environment). AR applications can align or register virtual objects with real-world objects (e.g., as observed in the image) in multiple dimensions. For example, real-world objects that exist in reality can be represented using models that are similar to or exactly match the real-world objects. In one example, a model of a virtual vehicle representing a vehicle on a road can be presented through a display of an AR device (e.g., AR glasses, an AR head-mounted display (HMD), or other device) while the user continues to view his natural environment through the display. The viewer may be able to manipulate the model while viewing the real-world scene. In another example, a real object located on a table may be identified and rendered using a model with a different color or different physical properties in the AR environment. In some cases, artificial virtual objects that do not exist in reality or computer-generated copies of real objects or structures in the user's natural environment may also be added to the AR environment.

[0028] An increasing number of applications use facial data (e.g., for XR systems, for 3D graphics, for security, etc.), leading to a large demand for systems that can generate detailed 3D facial models (and 3D models of other objects) in an efficient and high-quality manner. Generating detailed 3D models of objects (e.g., 3D facial models) typically requires expensive equipment and the use of multiple cameras in an environment with controlled lighting, which hinders large-scale data collection.

[0029] For example, facial data and / or 3D facial models may be used to identify and / or authenticate a user of a VR, XR, and / or AR system. In some cases, facial data associated with a user may be captured (e.g., using one or more cameras associated with an XR system, one or more HMDs associated with an XR system, etc.) and used to identify and / or authenticate the user.

[0030] Object identification and object authentication (also called object verification) are two related problems with subtle differences. In some cases, object identification can be defined as a one-to-many problem. For example, facial identification (as an example of object identification) can be used to find one person from among many people. Facial identification has many applications, such as for performing criminal searches. Object authentication can be defined as a one-to-one problem. For example, facial authentication (as an example of object authentication) can be used to check whether a person is the person he claims to be (e.g., checking whether the claimed individual is an individual in a registration database of authorized users). Facial authentication has many applications, such as for performing access control on devices, systems, places, or other items that can be accessed.

[0031] Using facial identification as an illustrative example of object identification, a registration database containing features of registration faces may be used to compare with features of one or more given query facial images (e.g., from an input image or frame). Registration faces may include faces registered with the system and stored in a registration database containing known faces. The registration face that is most similar to the query facial image may be determined to match the query facial image. Each registration face may be associated with a personal identifier that identifies the individual to whom the face belongs. The personal identifier of the matching registration face (the most similar face) is identified as the individual to be identified.

[0032] As described above, an object authentication or verification system can be used to authenticate or verify an object. For example, using facial authentication as an example, an input query facial image can be compared to a stored or registered representation of an individual's face. Typically, facial authentication requires a higher recognition accuracy because it is often associated with access control of a device or system. No false positives are expected in this case. Facial authentication should be able to identify the individual to whom the face belongs with high accuracy but with a low rejection rate. The rejection rate is the percentage of faces that were not recognized because the match score or classification result was below the recognition threshold.

[0033] Biometrics is the science of analyzing specific physical or behavioral characteristics of each individual so that the identity of each individual can be authenticated. Biometric-based authentication methods can be used to authenticate people to allow access to devices, systems, places, or other accessible items. In some cases, biometric-based authentication allows the individual to be authenticated based on a set of templates (data that can be verified) that are unique to the individual. Examples of biometric-based authentication include facial authentication, fingerprint authentication, voice authentication, and the like. For example, facial authentication can compare the face of the device user in the input image with known features of the individual that the user claims to be (e.g., stored in one or more templates) to authenticate that the user of the device is actually that individual. Similar processes can be performed for fingerprint authentication, voice authentication, and other biometric-based authentication methods.

[0034] Biometric-based user authentication systems typically have at least two steps, including an enrollment step and an authentication step (or test step). The enrollment step captures biometric data and stores a representation of the biometric data as a template. The template can then be used in the authentication step. For example, the authentication step can determine the similarity of the template to the representation of the input biometric data, and can use the similarity to determine whether to authenticate the user.

[0035] As mentioned above, facial data and / or 3D facial models may be used to identify and / or authenticate a user of a VR, XR, and / or AR system. For example, user identification and / or authentication may be performed based on facial data or images captured using one or more cameras or HMDs associated with the XR system to which the user is being identified or authenticated.

[0036] Performing 3D object reconstruction (e.g., to generate a 3D model of an object, such as a face model) from one or more images can be challenging. Using the face as an illustrative example of a 3D object, 3D face reconstruction can be difficult based on the need to reconstruct facial geometry (e.g., shape) and facial expressions. In some cases, when representing a 3D face model, the 3D face model can be viewed from different angles. For example, the 3D face model in an XR environment can be viewed from different angles by a user moving within the environment. In another example, the 3D face model can be moved (e.g., change pose) relative to a fixed viewing position. In some cases, it can be challenging to maintain an accurate projection of the 3D face model onto a 2D image within a range of viewing angles (also referred to herein as views).

[0037] Additionally, it may be difficult to accurately reconstruct facial expressions for portions of a face that may experience high variations in appearance. In one illustrative example, the eyes of a face may move to extreme gaze directions (e.g., looking to one side, squinting, etc.). In another illustrative example, the upper and lower lips of a face's mouth are controlled by muscles that allow for a variety of difficulties in reconstructing mouth shapes (e.g., smiling, frowning, pursing teeth, twisting lips, etc.).

[0038] like Figure 1 As illustrated, the white dots overlaid on the 2D facial image 102 may represent the projection of the 3D vertices of the 3D facial model 104 back onto the original 2D facial image 102 used to generate the 3D facial model 104. Figure 1In the illustration of , points corresponding to 3D vertices of the main features of the 3D facial model (which may be referred to as landmarks or 2D landmarks) are depicted as white dots. As shown, landmarks 110, 112, 118, 120, 122, 124, 126, 128 for the contours of lips, nose, mouth, eyes, eyebrows, nose, etc. are included. Although the 3D facial model 104 may include a greater number of vertices, for the purpose of illustration, only a small number of projected 3D vertices corresponding to the facial features listed above are shown. Figure 1 In the illustrated example, the landmarks corresponding to the inner contour 108 of the lower lip of the 3D facial model 104 projected onto the 2D image may include the landmark 112. Similarly, the landmarks corresponding to the outer contour 106 of the lower lip of the 3D facial model 104 may include the landmark 110.

[0039] Figure 1 Also illustrated are an outer contour 114 and an inner contour 116 of the upper lip of the 3D facial model 104. In some examples, landmarks corresponding to the outer contour 114 of the upper lip may include landmarks 118 and 124, and landmarks corresponding to the inner contour 116 of the upper lip may include landmark 120. Additional landmarks projected from the 3D facial model 104 may include a landmark 122 corresponding to the left eye, a landmark 124 corresponding to the right eyebrow, a landmark 126 corresponding to the entire facial contour, and a landmark 128 corresponding to the nose. As described above, each of the landmarks 114, 116, 120, 122, 124, 126, and 128 may be generated by the projection of the 3D facial model 104 onto the 2D facial image 102.

[0040] Figure 1 A two-dimensional (2D) facial image 102 and a corresponding 3D facial model 104 generated from the 2D facial image 102 using a 3D morphable model (3DMM) are illustrated. In some aspects, the 3D facial model 104 may include a representation of a facial expression in the 2D facial image 102. In one illustrative example, the facial expression representation may be formed by blend shapes. The blend shapes may semantically represent the movement of a muscle or part of a facial feature (e.g., opening / closing of a jaw, raising / lowering of eyebrows, opening / closing of eyes, etc.). In some cases, each blend shape may be represented by a blend shape coefficient paired with a corresponding blend shape vector.

[0041] In some examples, 3D facial model 104 may include a representation of the facial shape in 2D facial image 102. In some cases, the facial shape may be represented by a facial shape coefficient that is paired with a corresponding facial shape vector. In some implementations, a 3D model engine (e.g., a machine learning model) may be trained (e.g., during a training process) to implement a consistent facial shape (e.g., consistent facial shape coefficients) for the 3D facial model regardless of the pose (e.g., pitch, yaw, and roll) associated with the 3D facial model. For example, when rendering the 3D facial model into a 2D image for display, the 3D facial model may be projected onto the 2D image using projection techniques. Although the 3D model engine implements a consistent facial shape independent of pose, the projected 2D image may have varying degrees of accuracy based on the pose of the 3D facial model captured in the projected 2D image.

[0042] In some cases, it may be difficult to accurately reconstruct a 3D facial model (e.g., including facial model shape and / or facial model expression) based on one or more occluders between the user's face and a camera or imaging device that is used to capture images for reconstructing the user's 3D facial model. For example, when reconstructing a 3D facial model from images captured using an HMD (e.g., worn by a user of an XR system), an unobstructed frontal view image of the user may not be available. In some aspects, the HMD may include one or more inward-facing cameras that may capture images of the user's face (e.g., or portions thereof) while the HMD is worn by the user. For example, Figure 2 The HMD 202 illustrated in FIG. 2 may include one or more inward-facing cameras that may be used to capture one or more images of a user's face (e.g., also referred to as a "face image" or "facial image"). In some cases, some (or all) of the facial images captured by the HMD worn by the user may be occluded or otherwise include only a portion of the user's face in the captured facial image. For example, Figure 2 The HMD 202 illustrated in FIG. 204 may capture partial facial images 204A, 204B, 204C using respective three inward-facing cameras. Partial facial images may also be referred to as “occluded facial images” or “blocked facial images”.

[0043] In some aspects, an unobstructed frontal view image of the user's face may not be available based on the size and / or proximity of the HMD relative to the user's face. For example, the size and / or proximity of HMD 202 may cause an inward-facing camera included in HMD 202 to capture partial facial images 204A-204C in which a portion of the user's face is obscured or occluded in each of the partial facial images 204A-204C. In some cases, when a 3D facial model of a user is reconstructed using partial facial images (e.g., such as partial facial images captured by an inward-facing camera of an HMD), it may be difficult to accurately capture or determine distinctive facial features that may be used to authenticate the user.

[0044] like Figure 2 As shown, the 3D model generator may utilize input frames (such as tilted frames 204A, 204B, 204C, and / or 208) to generate a 3D facial model 210. Figure 2 As shown in , the 3D model fitting engine 206 can also generate and / or apply textures to the underlying 3D model (e.g., Figure 1 The 3D facial model 104 of the head mounted XR system 202 can be used to provide a digital representation of the user wearing the head mounted XR system 202. In one illustrative example, a 3D deformable model (3DMM) can be used to represent the geometry of the user's head.

[0045] Figure 3A An illustrative example of partial facial images 302, 304, and 306 is provided, which may correspond to input images that are provided to a 3D model generator for generating a reconstructed 3D model (e.g., 3DMM) of a user's face. In some aspects, the partial facial images 302, 304, 306 may be obtained from a head-mounted camera. For example, the partial facial images 302, 304, 306 may be obtained from an inward-facing camera included in an HMD worn by a user (e.g., worn by a user of an XR system, etc.). In some examples, the head-mounted camera may capture side views of an individual's left eye (e.g., facial image 302) and right eye (e.g., facial image 304) and oblique views of the mouth, nose, and jaw (e.g., facial image 306). In some cases, the head-mounted camera may be included in a head-mounted device. In an illustrative example, the head-mounted device may include an XR system. In some cases, it may be desirable to create a 3D facial model (e.g., 3DMM) of the individual captured in the partial facial images 302, 304, 306. In some cases, the 3D facial model of the individual may be used to generate a 2D image of the individual that provides an accurate representation of the user as the pose of the individual's head changes.

[0046] In some cases, the resulting 3D facial model (e.g., generated or reconstructed based on a partial facial image of the user (e.g., such as partial facial images 204A, 204B, 204C and / or partial facial images 302, 304, 306) may be used to generate or reconstruct the 3D facial model. Figure 2 The 3D facial model 210 illustrated in FIG. 2 may produce unrealistic results in the eye and mouth regions. In some cases, the 3DMM may lack the ability to accurately reproduce the user's mouth and eyeballs based on the degree or amount of occlusion or masking in the partial facial image used to generate or reconstruct the 3DMM. Unrealistic or inaccurate results in the eye and mouth regions of a 3D facial model (e.g., 3DMM) reconstructed using the user's facial image may make it difficult to accurately and / or reliably perform authentication of the user based on the reconstructed 3D facial model.

[0047] Figure 3B 3D facial model 350 is illustrated, which can be generated with a high level of accuracy to realistically reflect the shape of an individual's face. Expensive camera equipment such as that used to capture an individual's face from multiple angles (e.g., capturing one or more unobstructed views or angles of an individual's face) can be used to generate a 3D facial model. Figure 3B 3D facial model 350 is a highly detailed 3D facial model shown. In some cases, 3D facial model 350 may also be manually edited by a skilled artist to produce an accurate depiction of the individual. The process of generating such a highly detailed 3D facial model can only produce a single model for a particular individual and does not provide a flexible framework for generating a 3D model for any individual without preparing a detailed 3D facial model in advance.

[0048] There is a need for systems and techniques for generating accurate 3D facial models for a wide variety of facial shapes and facial expressions such that the accurate 3D facial models can be used to perform user authentication without requiring a highly detailed 3D facial model specifically tailored to an individual to generate an accurate representation of facial features. There is also a need for systems and techniques that can be used to generate accurate 3D facial models for a wide variety of facial shapes and facial expressions using one or more partial images of a user's face (e.g., partial and / or occluded images captured by a user-facing camera included in an HMD worn by the user) as input.

[0049] Described herein are systems, apparatus, processes (or methods), and computer-readable media (collectively, "systems and techniques") for using one or more neural network encoders to generate an accurate 3D facial model based on one or more input images that include only a portion of a user's face. For example, the one or more input images may be captured by one or more corresponding user-facing cameras included in an HMD worn by the user. The HMD images may include, but are not limited to, partial images of the user's eyes, partial images of the user's mouth or chin, and the like. In some cases, the one or more input images may include input images that are partial images (e.g., include a portion of the user's face but not the entire user's face) and are oblique images (e.g., captured at an angle relative to the user's face or otherwise not depicting a frontal view of the captured portion of the user's face).

[0050] In some examples, one or more neural networks (e.g., encoder neural networks) may be trained to determine, estimate, or predict shape and expression information based on receiving one or more partial images of a user's face as input. For example, the encoder neural network may be trained to generate a plurality of three-dimensional (3D) facial modeling parameters based on receiving partial images of the user's eyes (e.g., an image of the left eye and an image of the right eye) and / or a partial image of the user's mouth as input. In some aspects, the encoder neural network is trained to generate 3D morphable model (3DMM) parameters or coefficients based on the input images, which 3D morphable model (3DMM) parameters or coefficients may be used to reconstruct or generate a predicted 3D facial model of the user. For example, the encoder neural network may generate a plurality of 3DMM shape coefficients that encode information indicating the identity of the user (e.g., the shape of the user's face), and / or may generate a plurality of 3DMM expression coefficients that encode information indicating the expression performed by the user's face.

[0051] In some aspects, one or more neural networks (e.g., encoder neural networks) may be trained to determine, estimate, or predict camera pose information associated with a camera used to capture an input partial image of a user's face. For example, camera pose information associated with an HMD including a camera facing the user may be predicted by a trained encoder neural network (e.g., a pose prediction network). In some examples, the camera pose information may include 6-degrees-of-freedom (6-DOF) pose information associated with one or more user-facing cameras included in the HMD. The predicted pose information may be associated with the user's facial shape and one or more expressions performed by the user (e.g., expressions performed by the user when the partial facial image used as input is captured or obtained).

[0052] Although this article uses faces as examples of objects that can be modeled, the systems and techniques described herein can be applied to generate models for any type of object. In addition, for illustrative purposes, eyes and mouths are used herein as illustrative examples of local feature regions. However, a person of ordinary skill will understand that the systems and techniques described herein can be performed for other local parts of the face. In an illustrative example, similar systems and techniques can be applied to eyebrows, noses, ears, or any other part of the face. Similar techniques using local feature generation can also be used with objects other than faces.

[0053] Various aspects of the technology described herein will be discussed below with respect to the various figures. Figure 4 is an example of using at least one image frame 402 (e.g., from Figure 2 4 is a diagram of an example of a 3D modeling system 400 for generating a 3D model (e.g., a 3D morphable model (3DMM)) using one or more image frames 402 of a head-mounted XR system 202 facing a user. In some examples, one or more image frames 402 may include a partial facial image obtained using one or more inward-facing cameras included in the HMD (e.g., such as Figure 2 The partial facial images 204A, 204B, 204C and / or Figure 3A 302, 304, 306). Figure 4 As shown, the 3D modeling system 400 includes an image frame engine 404, a 3D model engine 406, and a face reconstruction engine 410. Although the 3D modeling system 400 is shown as including certain components, a person of ordinary skill will appreciate that the 3D modeling system 400 may include more than one component. Figure 4 The components shown are more components. The components of the 3D modeling system 400 may include software, hardware, or one or more combinations of software and hardware. For example, in some specific implementations, the components of the 3D modeling system 400 may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or the components of the 3D modeling system may include and / or may be implemented using computer software, firmware, or a combination thereof for performing the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the 3D modeling system 400.

[0054] The image frame engine 404 may obtain or receive an image frame 402 captured by an image sensor from a storage device, from a memory, from an external source (e.g., a server, an external memory accessed via a network, or other external source), etc. In some cases, the image frame may be included in a frame sequence (e.g., a video, an independent or still image sequence, etc.). In an illustrative example, each frame in the frame sequence may include a grayscale component per pixel. Other examples of frames include frames with red (R), green (G), and blue (B) components per pixel (referred to as RGB video including RGB frames), frames with brightness, chroma-blue, chroma-red (YUV, YCbCr, or Y'CbCr) components per pixel, and / or any other suitable type of image. The frame sequence may be captured by one or more cameras, obtained from a storage device, received from another device (e.g., a camera or a device including a camera), or obtained from another source. In some specific implementations, the image frame engine 404 may convert the image frame 402 to a grayscale representation or grayscale color domain or convert the image frame from a grayscale representation or grayscale color domain. In some cases, image frame engine 404 may crop a portion of image frame 402 that corresponds to a face. In some examples, image frame engine 404 may perform a facial detection process and / or a facial recognition process to detect and / or recognize a face within image frame 402. Image frame engine 404 may generate or apply a bounding box (e.g., Figure 1 The bounding box 130 shown in FIG. 4 is used to crop the image data within the bounding box to generate an input image for the 3D model fitting engine 406 .

[0055] 3D model fitting engine 406 may receive an input image (e.g., image frame 402, a cropped bounding box around a face in image frame 402, etc.) from image frame engine 404. Using the input image, 3D model fitting engine 406 may perform a 3D model fitting technique to generate a 3D model (e.g., a 3DMM model) of the face (which may include the head of the person in image frame 402). The 3D model fitting technique may include solving for shape coefficients a i and expression coefficient b j , these coefficients can be used to generate a 3DMM representing the face in the input image 402.

[0056] In some examples, the 3D model fitting performed by the 3D model engine 406 may include or be based on posture information related to the posture of the head. For example, the posture information may indicate an angular rotation of the head relative to a neutral position of the head. The rotation may be along a first axis (e.g., a yaw axis), a second axis (e.g., a pitch axis), and / or a third axis (e.g., a roll axis). The 3D model generated by the 3D model engine 406 may have different shapes and / or feature coefficients based on the specific view or posture information provided as input.

[0057] In some cases, the 3D model fitting may also include a focal length for projecting the 3D model onto the 2D image using any suitable projection technique. In some examples, a weak perspective model may project the 3D vertices of a 3D model (e.g., 3DMM) onto the 2D image using the focal length generated by the 3D model engine 406. In some examples, a full perspective model may project the 3D vertices of a 3D model (e.g., 3DMM) onto the 2D image using the focal length generated by the 3D model engine 406.

[0058] In another illustrative example, the 3D model can be projected onto the image plane using a full perspective model. For example, the average depth Z in Formula 3 can be ave Replaced with posed 3D model S P The actual depth Z of each 3D vertex is used to provide a full perspective projection. Weak perspective model and full perspective model techniques and any suitable projection techniques for projecting a 3D model onto an image plane may be used without departing from the scope of the present disclosure.

[0059] The facial reconstruction engine 410 may receive the coefficients generated by the 3D model engine 406 (as well as any other output parameters, such as pose, focal length, etc.) to generate a 3D model (e.g., 3DMM). The 3D model may be generated or constructed as a linear combination of an average face (sometimes referred to as a neutral face), a facial shape basis vector, and a facial expression basis vector. The average face may represent an average face that may be transformed (e.g., by a shape basis vector and an expression basis vector) to achieve the desired final 3D facial shape of the 3D model. The facial shape basis vectors may be used to scale the proportions of the average face. In some cases, the facial shape basis vectors may be used to represent a fat face or a thin face, a small nose or a large nose, and any adjustments to the basic facial shape. In some implementations, the facial shape basis vectors are determined based on principal component analysis (PCA). In some cases, the facial expression basis vectors may represent facial expressions, such as smiling, raising eyebrows, winking, blinking, frowning, etc.

[0060] The 3D model S (e.g., a 3DMM generated using a 3DMM fitting technique) generated by the 3D model engine 406 using a 3D model fitting technique may be a statistical model representing the 3D geometric shape of an object (e.g., a face). For example, the 3D model (e.g., a 3DMM) may be a product of an average face S0 and a facial shape basis (also referred to as a basis vector) U i and facial expression base V j and the facial shape coefficient a i and facial expression coefficient b j It can be expressed as a linear combination of, for example, as follows:

[0061]

[0062] In the example of formula (1), there are M facial shape coefficients a i and M facial shape basis vectors U i , where M is an integer greater than or equal to 1. In some implementations, the average face S0, the face shape basis vector U i and facial expression vector V j Each of the face shape basis vectors U may include position information (eg, x, y, and z coordinates) of 3D vertices that can be combined to form the 3D model S. In some implementations, the face shape basis vectors U i and facial expression vector V j It can be expressed as a position shift relative to the average face S0, where the coefficient a of the face shape is i and the coefficient b of facial expression j Provides a scaling factor for the corresponding offset vector.

[0063] An illustrative example of a facial expression basis vector is a blend shape. As used herein, a blend shape may correspond to an approximate semantic parameterization of all or a portion of a facial expression. For example, a blend shape may correspond to a complete facial expression, or to a "partial" (e.g., "Δ") facial expression. Examples of partial expressions include raising an eyebrow, closing an eye, moving one side of the face, etc. In one example, a single blend shape may approximate the linearized effect of the movement of a single facial muscle. In some cases, a semantic representation may be modeled as corresponding to the movement of one or more facial muscles.

[0064] In one illustrative example, the 3D model S includes three thousand 3D vertices. In one illustrative example, M is equal to 319, which corresponds to 319 facial shape basis vectors U i and facial shape coefficient a i In some implementations, the shape basis vectors U i The principal component analysis feature vector may be included. In some cases, there are N facial expression coefficients b j and N facial expression basis vectors, where N is an integer greater than or equal to 1. In some cases, the facial expression vector V j May include blend shape vectors. In one illustrative example, N is equal to 39, which corresponds to 39 facial expression basis vectors V j and 39 facial expression coefficients b j (e.g., 39 blend shapes and 39 blend shape coefficients). In some cases, the result of the linear combination shown in equation (1) can be a 3D model (e.g., 3DMM) of the face in a neutral pose. In some examples, the 3D model can be rotated to match the pose of the face in image frame 402 using pose information such as yaw, pitch, and roll values.

[0065] The facial reconstruction engine 410 may receive pose information from the 3D model engine 406 to generate a 3D model (e.g., 3DMM) and a projection of the 3D model into a 2D image space. The 3D model S (facial model) may be generated according to the above formula (1). As described above, the 3D model S produces a facial model with a neutral pose. In some cases, the 3D model S may be rotated to create a pose-based 3D model S according to the following formula (2): P :

[0066] S P =R(α,β,γ)S (2)

[0067] where R(α,β,γ) is a 3×3 rotation matrix with rotation angles α,β,γ.

[0068] In some cases, the 3D model can be projected onto the image plane using any suitable projection technique. In one illustrative example, the 3D model can be projected onto the image plane using weak perspective projection. The following formula (3) provides how the posed 3D model S can be projected onto the image plane using weak perspective projection techniques. P The X and Y components of each 3D vertex coordinate The x and y coordinates of the projected 2D image Example on:

[0069]

[0070] Where f is the predicted focal length output by the 3D model engine 406, Z ave is the average depth (or distance) of the face from the weak perspective camera. P The projection of formula (3) is performed on each 3D vertex of .

[0071] In some cases, the 3D model engine 306 may include a machine learning model (e.g., a deep neural network such as a convolutional neural network (CNN), a general adversarial network, an inception network, etc.). The 3D model engine 406 may receive an image frame 402 (and in some cases, an input view) as an input. In some cases, the image frame 402 and the view / pose information may be one of a plurality of images and corresponding views in a training data set. The training data set may include a large number of images including a variety of facial shapes, expressions, and views (e.g., including one or more views of each of the faces depicted in the image) in order to train the 3D model engine 406 to generate a 3D facial model (e.g., 3DMM) for many kinds of faces and views. In an illustrative example, the training data set may include one million images and views, two million images and views, or any other appropriately large number of images and views. In some cases, the 3D model engine 406 may be trained through multiple iterations (also referred to as epochs) using the same training data set. In some cases, the training data set used to train the 3D model engine 406 may include images, views, and labels indicating known features in the frame (e.g., in a supervised training process). In some cases, the training dataset may include annotated ground truth 2D landmark information corresponding to the locations of facial features for each image frame 402. In some cases, the 2D landmark information may be calculated from each image frame 402 using a landmark algorithm.

[0072] During training, the 3D model engine 406 may utilize one or more loss functions to analyze the errors in the 3D model parameters (e.g., the coefficients of formula (1)). The loss function L may be used to determine the amount of error in the 3D model generated by the 3D model engine 406 relative to the ground truth data provided as input to the 3D model engine 406. An example loss function includes the mean square error (MSE). The MSE is defined as It calculates the actual answer (e.g., ground truth landmark location) minus the sum of one-half the square of the predicted (output) answer. Another example loss function includes a normalized mean error, where the error between the ground truth landmarks and the landmarks predicted by the 3D model engine 406 can be normalized by the predicted eye distance of the face.

[0073] An example 2D vertex loss function is shown in the following formula (6):

[0074]

[0075] Here, is the difference between the projected 2D landmark position from the ground truth 3D model and the projected 2D landmark position of the corresponding vertex generated by the 3D model engine 406. M (e.g., an integer) is the number of 2D landmarks. FIG. 5A to FIG. 7 ), v may represent one of the plurality of partial facial images provided as input. For example, v may represent one of three HMD views of the user's face (e.g., eyes or mouth). v can be the set of vertices visible from the view v.

[0076] An example 3D vertex loss function is shown in the following formula (7):

[0077]

[0078] Here, is the difference between the 3D vertex position from the ground truth 3D model and the 3D vertex position of the corresponding vertex generated by the 3D model engine 406, and N (an integer) is the number of 3D vertices.

[0079] Landmark loss L Landmarks It can be a weighted sum of different 2D and / or 3D landmark loss components, as shown in the following formula (8):

[0080] L Landmarks =λ 3D L 3D +λ 2D L 2D (8)

[0081] where λ 3D and λ 2D are the weighting coefficients for 3D landmark loss and 2D landmark loss, respectively.

[0082] An example regularized loss function is shown in equation (9), and an example clamped loss function is shown in the following equation (10):

[0083]

[0084] Here, α j is the shape factor. b j can represent facial expression coefficients. In some examples, a clamping loss can be used to constrain the shape coefficients between fixed bounds (e.g., to preserve semantic meaning):

[0085]

[0086] Here, β UB,j and β LB,j are the facial expression coefficients b generated by the 3D model engine 406 j The upper and lower boundary values ​​of each facial expression coefficient in .

[0087] During inference (e.g., after training is complete), the 3D model engine 406 may output a 3D face model (e.g., 3DMM) based on one or more input image frames 402. i The constraints (e.g., the shape consistency loss L in formula (4) ShapeConsistency ) is omitted from the loss function shown in formula (5), the resulting output 3D facial model for an individual may have different shape coefficients depending on the view (e.g., pose) of the generated 3D facial model. However, eliminating the constraint on shape consistency can result in a 3D facial model that can generate 2D projection images from different poses.

[0088] Figure 5A A block diagram illustrating an example training process for training a shape and expression encoder to generate shape coefficients and expression coefficients that can be used to recreate or generate a 3D facial model (e.g., 3DMM) of a user's face based on a partial input image of the user's face. In an illustrative example, a shape-expression encoder 522 can be trained to generate shape and expression coefficients 524 based on receiving a plurality of HMD images 512 as input. In some aspects, the shape-expression encoder 522 can be a neural network encoder, although various other neural networks, machine learning networks, and / or machine learning algorithms and architectures can also be utilized.

[0089] In some aspects, the shape and expression coefficients 524 generated by the trained shape-expression encoder 522 may include a plurality of shape coefficients of a 3D face model (e.g., 3DMM) and a plurality of expression coefficients of a 3D face model (e.g., 3DMM). For example, the trained shape-expression encoder 522 may be used to generate a plurality of shape coefficients α s and multiple facial expression coefficients α e In some examples, the trained shape-expression encoder 522 can generate shape and expression coefficients 524 that include up to 210 shape coefficients α j and up to 30 facial expression coefficients b j .

[0090] The shape and expression coefficients 524 may be associated with or used to generate a single 3D facial model 552 that is associated with the plurality of HMD images 512. In some aspects, the 3D facial model 552 generated or reconstructed based on the shape and expression coefficients 524 may also be referred to as a predicted 3D mesh, a predicted 3D facial model, and / or a predicted 3D facial model.

[0091] For example, the shape and expression coefficients 524 may be used to generate a predicted 3D facial model, which is a 3D deformable mesh (3DMM). The 3D facial model P predicted by the 3DMM may be given by:

[0092]

[0093] Here, represents the average face (e.g., sometimes called the neutral face); α s represents up to 219 facial shape coefficients (e.g., which may be determined or predicted by the trained shape-expression encoder 522 and included in the output coefficients 524); α e A represents up to 39 facial expression coefficients (e.g., which may be determined or predicted by the trained shape-expression encoder 522 and included in the output coefficients 524); s represents the facial shape basis vector; and A e Represents the facial expression basis vector.

[0094] Facial shape coefficient α s The number of facial shape basis vectors A s The number and facial expression coefficient α e The number of facial expression basis vectors A e For example, 219 shape coefficients α s Can be applied to 219 facial shape basis vectors A s And is used for individual-based and facial shape coefficient α s The associated unique shape makes the average face of formula (11) Similarly, the 39 facial expression coefficients α e Can be applied to 39 facial expression basis vectors A e and is used based on the facial expression coefficient α e The expression representing or associated with the facial expression coefficient makes the average facial Deformation.

[0095] In one illustrative example, the systems and techniques may train the shape-expression encoder 522 to estimate (e.g., predict) both shape and expression coefficients using multiple partial HMD images as input. For example, the partial HMD images as input may be Figure 5A. As illustrated, the RGB partial HMD image 512 may include a plurality of partial HMD images, each of which includes a different portion of the user's face. The first HMD image 512a may be an image of the user's right eye, the second HMD image 512b may be an image of the user's left eye, and the third HMD image 512c may be an image of the user's mouth or chin (e.g., a mouth or chin region or portion of the user's face).

[0096] In some aspects, HMD training data images 512 (e.g., 512a, 512b, 512c) may be generated based on the reference truth frontal image 502. For example, the reference truth frontal image 502 may be a single 2D image of an individual taken from a frontal or frontal perspective. The frontal image 502 may be provided to a synthetic HMD image rendering engine 510 along with HMD pose information 504. Using the HMD pose information 504, the synthetic HMD image rendering engine 510 may generate a plurality of partial HMD images (e.g., such as the partial HMD image 512). The HMD pose information 504 may include one or more HMD pose parameters to be simulated by the synthetic HMD image rendering engine 510 when generating the plurality of partial HMD training data images 512. In some aspects, the HMD pose information 504 may include 6-degrees-of-freedom (6-DOF) HMD pose information (e.g., up / down, left / right, forward / backward, pitch, roll, yaw).

[0097] In some aspects, one or more (or all) of the HMD training data images 512 generated by the synthetic HMD image rendering engine 510 may be tilted relative to the ground truth frontal image 502. For example, Figure 5A A frontal HMD image 512c including a user's mouth, an oblique HMD image 512a including a user's right eye, and an oblique HMD image 512b including a user's left eye are illustrated. By generating HMD training data images 512 that are oblique relative to the ground truth frontal image 502, the synthetic HMD image rendering engine 510 can simulate or recreate the perspective of a partial facial image that would be captured by a user-facing camera included in the HMD and a user capturing a facial image of the user wearing the HMD.

[0098] The ground truth frontal image 502 may be provided to a ground truth flow engine 532 that generates a ground truth 3D mesh 562. The ground truth 3D mesh 562 may be used as a ground truth 3D facial model (e.g., 3DMM) for training a shape-expression encoder 522 (e.g., based on minimizing a 3D vertex distance loss 545 between a predicted 3D mesh 552 and a ground truth 3D mesh 562, the predicted 3D mesh being generated based on the shape and expression coefficients 524 predicted by the shape-expression encoder 522, as described in more depth below). In some examples, the ground truth flow engine 532 may be a pre-trained neural network or other machine learning network, encoder, etc., that is pre-trained to generate a 3D mesh (e.g., such as the ground truth 3D mesh 562) from a frontal facial image (e.g., such as the ground truth frontal image 502).

[0099] In some cases, the reference truth frontal image data 502 may include a single image, as described above. In some examples, multiple cameras may be used to capture or obtain the reference truth frontal image data 502. For example, the reference truth frontal image data 502 may include depth information determined using multiple cameras. In examples where the reference truth frontal image data 502 is multi-camera data, the reference truth stream engine 532 may be provided as a pre-trained neural network, machine learning network, encoder, etc., which is pre-trained to generate a 3D mesh (e.g., such as the reference truth 3D mesh 562) from the multi-camera frontal image data. In some aspects, various other mesh generators may be used to provide the reference truth stream engine.

[0100] In one illustrative example, the ground truth 3D mesh 562 generated by the ground truth streaming engine 532 may be used to retopologize a mesh associated with the synthetic HMD image rendering engine 510. For example, the synthetic HMD image rendering engine 510 may receive as input the ground truth frontal image 502, the ground truth 3D mesh 562 (e.g., which is generated from the ground truth frontal image 502), and the HMD pose information 504. Using the HMD pose information 504 and the ground truth 3D mesh 562, the synthetic HMD image rendering engine 510 may project the 2D frontal image 502 onto the ground truth 3D mesh 562. The HMD pose information 504 may then be used to back-project the 2D frontal image 502 from the ground truth 3D mesh 562 to the desired 2D HMD image portions 512a-512c.

[0101] As previously described, the shape and expression encoder 522 may generate predicted shape and expression coefficients 524 based on receiving as input the RGBHMD image 512 generated from the ground truth frontal image 502. In some aspects, the shape-expression encoder 522 may be trained at least in part based on the regularization and clamping loss 525. For example, the loss 525 may include a regularization loss L given by reg :

[0102]

[0103] The regularization loss of formula (12) can be determined as the sum of the shape and expression coefficients 524 predicted by the shape-expression encoder 522 (e.g., formula (12) includes a sum from 1 to 258, representing 219 facial shape coefficients α s +39 facial expression coefficients α e = 258 total coefficients α j ).

[0104] Additionally, the loss 525 may include a clamping loss L given by clamp :

[0105]

[0106] The clamping loss of formula (13) can be used to convert the 219 facial shape coefficients α s (e.g., determined based on the clamping loss as the sum for j=1,…,219) constrained between fixed boundaries to preserve semantic meaning. For example, the facial shape coefficient α s (For example, expressed as α in formula (13) j Can be constrained by the upper boundary β UB,j With the lower boundary β LB,j The upper and lower boundaries may preserve the semantic meaning of the facial shape coefficients that may be predicted by the shape-expression encoder 522, because facial shape coefficients with values ​​that are too high or too low may represent an average face. In some cases, the upper and lower bounds of the clamping loss in formula (13) can prevent overfitting and preserve the semantic meaning captured by the face shape coefficients (for example, the semantic meaning is the average face which can be applied to recreate a unique 3D facial model of the user).

[0107] Return to Figure 5AThe example training process for training the shape-expression encoder 522 is depicted, as illustrated, the shape and expression coefficients 524 can be used to generate a predicted 3D mesh 552. In one illustrative example, the predicted 3D mesh 552 can be a 3DMM generated using equation (11). For example, the α s and α e The coefficient values ​​of are used as input to formula (11) to generate the predicted 3D mesh 552, which shape and expression coefficient predictions are output by the shape-expression encoder 522 during a given training iteration. The system and technology can additionally use the 3DMM average face and basis vectors 540 (e.g., combined with the shape and expression coefficient predictions 524) to generate the predicted 3D mesh 552. For example, the 3DMM average face and basis vector information 540 may include the average face (e.g., where the same average face is used to generate both the ground truth 3D mesh 562 and the predicted 3D mesh 552), multiple facial shape basis vectors A s and multiple facial expression basis vectors A e In some aspects, the same average face, facial shape basis vectors, and facial expression vectors may be used to generate the predicted 3D mesh 552 and the ground truth 3D mesh 562. In some cases, the same average face, facial shape basis vectors, and facial expression vectors may be used across all training iterations of the training iterations performed to train the shape-expression encoder 522. In some examples, the average face included in the 3DMM average face and basis vector information 540 may be used to generate the predicted 3D mesh 552 and the ground truth 3D mesh 562. The face and basis vectors may be generic (eg, not related to a person).

[0108] The predicted 3D mesh 552 is a 3D facial model (e.g., 3DMM) generated using shape and expression coefficients 524 predicted by the shape-expression encoder 522 using only a portion of the HMD image 512 as input. In some aspects, the shape-expression encoder 522 may encode person-specific details into the facial shape coefficients α s and / or facial expression coefficient and α e 524 (e.g., which is included in the predicted coefficient output 524 generated by the shape-expression encoder 522). For example, person-specific details may be encoded in the shape and expression coefficients 524 based at least in part on the fact that the 3DMM average face and basis vector information 540 is generic rather than personalized.

[0109] The 3D vertex distance loss 545 may be determined as the error for a given training iteration because the 3D vertex distance loss 545 may represent the difference between the predicted 3D mesh 552 and the ground truth 3D mesh 562. In one illustrative example, the 3D vertex distance loss 545 may be the same or similar to the 3D vertex distance loss of formula (7). In some aspects, as the shape-expression encoder 522 is trained to more accurately predict the shape and expression coefficients 524 of a given set of input HMD training images 512, the resulting predicted 3D mesh 552 will become increasingly similar to the ground truth 3D mesh 562, and the 3D vertex distance loss 545 will decrease.

[0110] In one illustrative example, the shape-expression encoder 522 may be trained based at least in part on minimizing the regularization and clamping loss 525 and the 3D vertex distance loss 545 over multiple training iterations. Each training iteration may utilize a different ground truth frontal image 502 and / or a different set of HMD training data images 512 (e.g., if the same ground truth frontal image 502 is used for multiple training iterations, the synthetic HMD image rendering engine 510 may be configured to generate a different HMD image from the same ground truth frontal image 502 for each training iteration).

[0111] In some aspects, the shape-expression encoder 522 may additionally or alternatively be trained at least in part based on a 2D vertex distance loss 575. In one illustrative example, the 2D vertex distance loss 575 may be the same or similar to the 2D vertex distance loss of formula (6). For example, the 2D vertex distance loss 575 may be calculated based on projecting the predicted 3D mesh 552 onto an HMD image (e.g., indicated at 554) and projecting the ground truth 3D mesh 562 onto the same HMD image (e.g., indicated at 564). In some examples, the projection operations 554 and 564 may be performed using HMD pose information 504 (e.g., the same HMD pose information 504 used to generate the HMD image 512 at the synthetic HMD image rendering engine 510). In one illustrative example, the shape-expression encoder 522 may be trained based on minimizing the sum of the regularization and clamping loss 525, the 3D vertex distance loss 545, and the 2D vertex distance loss 575 through each training iteration.

[0112] In one illustrative example, the shape-expression encoder 522 may be trained using multiple training iterations, where some (or all) of the training iterations utilize different sets of ground truth frontal images 502 and / or partial HMD images 512. For example, multiple training iterations may be performed using ground truth frontal images 502 (e.g., and therefore, partial HMD images 512) that depict, represent, or are associated with many different individuals, different lighting conditions, different facial appearances, different facial expressions, etc.

[0113] Figure 5B is a diagram illustrating an example of an inference process 500b that can be performed using a trained shape-expression encoder 522 (e.g., in completing the above description of Figure 5A The training process of the shape-expression encoder 522 is then performed Figure 5B In some aspects, the inference process may be performed by an XR device or system, a VR device or system, an AR device or system, and / or other computing devices associated with an HMD worn by a user. Figure 5B As will be described in more depth below, an XR device or system may perform authentication of a user based on the inference process 500b.

[0114] In some examples, the user's facial image captured by the HMD may be an infrared (IR) image or a near infrared (NIR) image, rather than the synthetic RGB HMD image 512 used to train the shape-expression encoder 522. For example, because the interior volume of the HMD (e.g., the volume enclosed by the HMD and immediately adjacent to the user's eyes) is a low-light environment, a user-facing camera included inside the HMD may not be able to capture an RGB facial image. The user-facing camera included in the HMD may therefore capture an IR image, a NIR image, or other partial HMD image rendered in a grayscale color domain or other non-RGB color domain, etc.

[0115] like Figure 5B As illustrated, user authentication may be performed based on a plurality of partial HMD images 506 of a user's face obtained (e.g., from an HMD worn by the user). The plurality of partial HMD images 506 may be captured as IR or NIR images. In some cases, the partial HMD images 506 captured inside the HMD may be IR or NIR images (e.g., such as left eye image 506a and right eye image 506b, respectively), while the partial HMD images 506 captured outside the HMD may be RGB images (e.g., such as mouth image 506c). In some cases, all of the partial HMD images 506 may be IR or NIR images, such as Figure 5B As depicted in .

[0116] After obtaining the plurality of partial HMD images 506, the NIR to RGB conversion engine 505 can be used to convert the HMD images 506 (e.g., as needed) from the NIR color domain in which the HMD images were captured to the RGB color domain in which the shape-expression encoder 522 was trained. In some aspects, when one or more of the partial HMD images 506 are captured in a color domain that is not the NIR color domain, but is also different from the color domain in which the shape-expression encoder 522 was trained, Figure 5B The component depicted as the NIR to RGB transformation engine 505 in FIG. 5 may be replaced by a suitable transformation engine for converting from the color domain in which the partial HMD image 506 was originally captured (e.g., captured by a user-facing camera included in the HMD) to the color domain in which the shape-expression encoder 522 was trained (e.g., RGB or other).

[0117] The output of the NIR to RGB conversion engine 505 is a set of RGB HMD images 514 that correspond to the RGB color domain (or other color domain) in which the shape-expression encoder 522 is trained. For example, the NIR HMD left eye image 506a is converted to an RGB HMD left eye image 514a, the NIR HMD right eye image 506b is converted to an RGB HMD right eye image 514b, and the NIR mouth image 506c is converted to an RGB HMD mouth image 514c.

[0118] Inference (and subsequent user authentication) may be performed based on providing the RGB HMD image 514 as input to a trained shape-expression encoder 522. The trained shape-expression encoder 522 may generate as output a plurality of shape and expression coefficients 524 corresponding to the input RGB HMD image 514, as described above with respect to Figure 5A described.

[0119] Using the 3DMM average face and basis vectors 540 (e.g., a generic or person-independent 2DMM average face and basis vectors), the predicted shape and expression coefficients 524 may be used to generate a predicted 3D mesh 550 corresponding to the facial data of the user represented in the RGB HMD image 514, which is provided as input to a trained shape-expression encoder 522 (e.g., as also described above with respect to Figure 5A described above).

[0120] In one illustrative example, the predicted 3D mesh 550 may be used to perform user authentication. For example, the user authentication may include an initial enrollment phase in which the user is prompted to enroll or register a set of different expressions while wearing the HMD. One or more sets of HMD images (e.g., NIR HMD images 506) are captured for each prompted expression enrolled / registered for the user, and a registered 3D mesh (not shown) is generated and saved for a unique combination of the user's face and the prompted expression. The user authentication may be performed using the method described above with respect to Figure 5B The same process as described was used to generate the registered 3D mesh.

[0121] In some aspects, the trained shape-expression encoder 522 may generate shape and expression coefficients 524 that are unique for each user face shape-user facial expression combination provided during the enrollment process. In some examples, the enrollment phase may include storing at least the shape and expression coefficients 524 for the user's registered facial expression combinations. In some aspects, the enrollment phase may include storing the shape and expression coefficients 524 and the resulting 3D mesh generated using the shape and expression coefficients 524, also for each of the user's registered facial expression combinations. In some aspects, only the resulting 3D mesh may be stored and used for subsequent authentication. In some examples, the registered expressions associated with the user may include multiple different expressions, such as neutral, smiling, angry, etc. In some cases, a single reference may be stored for some (or all) of the registered expressions. In some examples, multiple references may be stored for some (or all) of the registered expressions.

[0122] During the authentication phase, the user may be prompted (e.g., while wearing the HMD) to perform one or more expressions that have been previously registered, as described above. In some examples, the user may be prompted to perform a single expression, multiple expressions from the registered expressions, or all of the registered expressions. Authentication may be performed based on capturing HMD image data (e.g., NIR HMD image 506) of the user performing the prompted expression and using the captured HMD image data to generate the shape of the captured HMD image data and expression coefficients 524 (and / or the corresponding 3D mesh 550).

[0123] In one illustrative example, shape and expression coefficients 524 predicted during inference of captured HMD image data associated with a user performing a requested expression may be compared to registered shape and expression coefficients generated during enrollment (e.g., also by a trained shape-expression encoder 522) when the user was prompted to perform the same expression. In some aspects, the shape and expression coefficients 524 determined during authentication may be verified against an enrolled target that includes the enrolled shape and expression coefficients generated by the trained shape-expression encoder 522 during enrollment for the same expression. In some examples, authentication may be based on determining that one or more error values ​​determined between the shape and expression coefficients 524 and the enrolled shape and expression coefficients are below corresponding one or more predetermined thresholds.

[0124] For example, the weighted absolute error m1 of the shape and expression coefficients can be determined as:

[0125]

[0126] The weighted mean vertex error m2 in 3D can be determined as:

[0127]

[0128] The weighted mean vertex error m3 in 2D can be determined as:

[0129]

[0130] In some cases, each of the error values ​​of formulas (14) to (16) may be associated with a different predetermined authentication threshold, wherein the user is authenticated if the error value is less than or equal to its corresponding predetermined authentication threshold. In some examples, the error values ​​of formulas (14) to (16) may be associated with the same predetermined authentication threshold, wherein the user is authenticated if at least one of the error values ​​is less than or equal to a single predetermined authentication threshold. In some cases, the user will only be authenticated if multiple of the error values ​​or all three of the error values ​​in formulas (14) to (16) are less than or equal to the predetermined authentication threshold.

[0131] In another illustrative example, a neural network (e.g., a neural network encoder) may be trained to generate a neural network from one or more input HMD images (e.g., such as described above with respect to Figure 5A and Figure 5B The HMD pose is estimated based on the input HMD images 512, 506 and / or 514. For example, a neural network encoder can be trained to estimate (e.g., predict) HMD pose information associated with a main camera of the HMD. Figure 6is a diagram illustrating an example training process 600 for training a pose network 620 (eg, also referred to as a pose encoder, a neural network pose encoder, etc.) to generate a predicted HMD pose 640 based on an input of an RGB HMD image 612 .

[0132] In some respects, Figure 6 The illustrated example training process 600 may be used with Figure 5A The example training process 500a is the same or similar to the example training process 500a, wherein Figure 5A The shape-expression encoder 522 is composed of Figure 6 For example, in some aspects, the reference truth frontal image 602 may be replaced with the pose encoder 620 in FIG. Figure 5A The synthetic HMD image rendering engine 610 can be the same as or similar to the reference truth front image 502; Figure 5A The synthetic HMD image rendering engine 510 is the same or similar to the reference truth stream engine 632; Figure 5A The reference truth flow engine 532 is the same or similar to the reference truth 3D grid 662; Figure 5A The ground truth 3D mesh 562 of FIG. 5 is the same as or similar to that of FIG. 5; and / or the RGB partial HMD image 612 may be the same as Figure 5A The RGB portion of the HMD image 512 is the same or similar.

[0133] Based on receiving the plurality of RGB partial HMD images 612 as input, the pose network 620 may be trained to generate HMD pose information 640, wherein the HMD pose information 640 is a predicted HMD pose corresponding to the HMD used to capture one or more of the HMD images 612 and / or a predicted HMD pose corresponding to at least one user-facing camera included in the HMD and used to capture the HMD images 612. For example, the predicted HMD pose information 640 may be a predicted 6-DOF HMD pose associated with a main camera of the HMD used to capture the HMD images 612.

[0134] The pose encoder 620 may be trained based at least in part on an L1 loss 645, which may be calculated or otherwise determined between the predicted HMD pose 640 and the corresponding ground truth HMD pose 604. The ground truth HMD pose 604 may be compared to Figure 5A As illustrated, the HMD pose 604 may additionally be provided as input to a composite HMD image rendering engine 610 (e.g., also as shown in FIG. Figure 5A exemplified).

[0135] The ground truth HMD pose 604 may further be provided as input to an HMD projection operation 664, which may be performed to project the ground truth 3D mesh 662 onto the HMD image (e.g., in the same manner as described above with respect to Figure 5A The illustrated projection operation 564 may be performed in the same or similar manner as described above).

[0136] The L1 loss 645 may indicate the difference or error between the HMD pose 640 predicted by the pose encoder 620 and the ground truth HMD pose 604, which would be predicted by the pose encoder 620 if it were error-free or trained to be 100% accurate. Based on minimizing the L1 loss 645 during multiple training iterations performed for the pose network 620, the pose encoder 620 may be trained to accurately (or more accurately) generate the predicted HMD pose information 640 for a given input of the RGB partial HMD image 612.

[0137] In some cases, an additional 2D vertex loss 655 may be used to train the pose encoder 620. For example, Figure 5A The 2D vertex loss 575 described above determines the 2D vertex loss 655 the same or similarly. In the context of training pose encoder 620 , predicted HMD pose 640 for a given training iteration may be combined with ground truth 3D mesh 662 and used to project ground truth 3D mesh 662 onto HMD image 612 .

[0138] For example, at projection 650 , predicted HMD pose information 640 generated by pose network 620 may be combined with known HMD camera intrinsic information (e.g., relative position of the HMD's camera facing the user, focal length, etc.) and used to generate a projection of a ground truth 3D mesh 662 to the HMD image 612 .

[0139] By using ground truth HMD pose information 604 (e.g., along with the same known HMD camera intrinsic information) to configure and generate a projection of ground truth 3D mesh 662 to HMD image 612 , a similar projection of ground truth 3D mesh 662 to HMD image 612 may be generated at projection 664 .

[0140] If the predicted HMD pose information 640 is the same as the ground truth HMD pose information 604, the 2D vertex projections 650 and 664 will be the same, and the loss or error (e.g., 2D vertex distance loss 655) between the two projections will be zero. If the predicted HMD pose information 640 is not the same as the ground truth HMD pose information 604, the corresponding 2D vertex projections 650 and 664 will be different, respectively, and the 2D vertex distance loss 655 will have a non-zero value. Based at least in part on minimizing the 2D vertex distance loss 655 between the 2D projection of the ground truth 3D mesh 662 using the predicted HMD pose information 640 and the 2D projection of the ground truth 3D mesh 662 using the ground truth HMD pose information 604, the pose network 602 can be trained to accurately (or more accurately) generate the predicted HMD pose information 640 using only the plurality of RGB partial HMD images 612 as input.

[0141] In one illustrative example, the trained pose encoder 620 may be used similar to the above description of Figure 5A and Figure 5B The manner described and the use of the trained shape-expression encoder 522 to perform user authentication. For example, user authentication based on the trained gesture encoder 620 may include a user enrollment phase that is the same as the user enrollment phase described above with respect to user authentication based on the trained shape-expression encoder 522.

[0142] The authentication phase using the trained pose encoder 620 may be performed by initially prompting the user to perform one or more previously enrolled expressions (e.g., the same as described above with respect to the authentication phase using the trained shape-expression encoder 522). In response to the user performing the prompted expression, the systems and techniques may capture multiple partial HMD images of the user's face while performing the prompted expression. For example, the multiple partial HMD images may be the same as described above with respect to Figure 5B The same multiple NIR partial HMD images 506 are described above. The NIR to RGB conversion engine can be used to convert Figure 5B The NIR to RGB conversion engine 505 and the resulting RGB partial HMD image 514 transform or convert the NIR partial HMD image into a plurality of RGB partial HMD images in the same or similar manner as described above.

[0143] The trained pose encoder 620 can generate estimated or predicted HMD pose information 640 (e.g., predicted 6DOF HMD pose) based on receiving the resulting multiple RGB partial HMD images as input. Using the predicted HMD pose information 640 and the known HMD camera intrinsic information, the registered mesh (e.g., determined during the enrollment phase for the current authentication combination of the user's face and expression) can be projected onto the multiple RGB partial HMD images. For example, the projection can be performed in the same manner as described above, where the registered mesh for the prompted expression replaces Figure 6 The ground truth 3D mesh 662 depicted in the training process of FIG.

[0144] In some cases, projecting the registered grid onto the multiple RGB partial HMD images may be performed by projecting one or more registered grid landmarks onto the RGB partial HMD images and then using a 2D landmark detector to determine one or more detected landmarks in the RGB partial HMD images. For example, landmarks (e.g., among the registered grid landmarks and the detected landmarks) may include facial features and / or expression features, such as corners of the eyes or mouth, etc.)

[0145] In one illustrative example, the detected landmarks determined for the partial HMD image being authenticated may be compared to projected landmarks generated by projecting the registered 3D mesh onto the 2D partial HMD image using the predicted HMD pose information 640. Because the registered 3D mesh is target user specific, the registered mesh will not align with landmarks of different users (e.g., when different users with different facial shapes perform the same expression, the projected landmarks of the different users will be different from the projected landmarks of the user being authenticated).

[0146] In some cases, authentication of the user may be performed based on determining that the average landmark error or difference between the 2D projected landmarks (e.g., generated by projecting the user-expression registered 3D mesh onto the 2D partial HMD image using the predicted HMD pose information 640) and the corresponding landmarks included in the partial HMD image captured by the user-facing camera of the HMD is less than or equal to a predetermined threshold. For example, the predetermined threshold may be the average 2D landmark error m4 given by:

[0147]

[0148] In another illustrative example, a Siamese network may be used to perform user authentication based on multiple partial HMD images. Figure 7is a diagram illustrating an example training process for training a twin network 700 to determine the similarity between a target HMD mouth image 710a and a source HMD mouth image 710b. In some aspects, the target HMD mouth image 710a may be an HMD mouth image captured during user enrollment (e.g., in response to prompting a user to perform a specified expression, where the enrolled HMD mouth image is enrolled corresponding to the specified expression and user). The source HMD mouth image 710b may be an HMD mouth image captured during an authentication phase, such as when the user is wearing the HMD (e.g., as described above).

[0149] In some aspects, the Siamese network 700 may be trained and implemented (e.g., implemented to perform inference for user authentication) to use only HMD mouth images. As previously mentioned, HMD images of the eyes (e.g., such as FIG. 5A to FIG. 6 Those images described above can only be captured in the IR or NIR domain and are generally not available in the RGB domain. However, based on the fact that an HMD worn by a user generally obscures or covers the user's eyes but not the user's mouth, an HMD image of the mouth can be captured in the RGB domain.

[0150] In some aspects, the Siamese network 700 can be trained to perform user authentication based on skin texture and color information captured in RGB mouth images (e.g., such as source RGB mouth image 710a and target RGB mouth image 712b, respectively). FIG. 5A to FIG. 6 The encoder described in the same or similar manner as described is provided as a neural network, a neural network encoder, etc.) to perform training. In an illustrative example, a twin network 700 including paired mouth encoders 720a, 720b can be trained in multiple training iterations and using multiple paired RGB mouth images. For example, a training data pair of RGB mouth images may include a source RGB mouth image 710a and a target RGB mouth image 712b, which may or may not belong to the same individual. The training data pair may be marked with an indication of whether the source RGB mouth image and the target RGB mouth image (e.g., 710a, 712b) belong to the same individual. After multiple training iterations, a contrast loss 725 between the encoded outputs of the first twin mouth encoder 720a (e.g., source encoder) and the second twin mouth encoder 720b (e.g., target encoder) can be determined, so that the twin network 700 is trained to detect a match or mismatch of identity between the source RGB mouth image 710 and the target RGB mouth image 712b.

[0151] The authentication based on the trained twin network 700a can be performed to include the above-mentioned FIG. 5A to FIG. 6 The same registration phase. A set of expressions of the user (e.g., neutral, smiling, angry, etc.) can be registered by prompting the user to perform a specified expression. The captured RGB mouth data can be provided to the trained twin network 700 and fixed as a target (e.g., fixed as a target RGB mouth image and / or fixed as a target for match / mismatch detection performed by the trained twin network 700). During the authentication phase, the user may be prompted to perform one or more previously registered expressions. An HMD mouth image of the user performing the requested expression for authentication may be captured and may be fixed as a source (e.g., fixed as a source RGB mouth image and / or fixed as a source for match / mismatch detection performed by the trained twin network 700). The source-target pair for each expression prompted from the user during authentication may be passed as input to the trained Siamese network 700a (e.g., the RGB mouth image fixed as the target may be passed to the target mouth encoder 720b, and the RGB mouth image fixed as the source may be passed to the source mouth encoder 720a), which may generate as output an indication or determination of a match (e.g., in which case the user is authenticated) or a mismatch (e.g., in which case the user is not authenticated). In some cases, authentication of the user may be performed based on determining that the Euclidean distance between the source-target pair of RGB HMD mouth images is less than or equal to a predetermined threshold.

[0152] Figure 8 800 for authenticating a user. Although the example process 800 depicts a particular order of operations, the order may be changed without departing from the scope of the present disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not materially affect the functionality of the process 800. In other examples, different components of the example device or system implementing the process 800 may perform functions substantially simultaneously or in a particular order.

[0153] At block 802, process 800 includes obtaining a plurality of images associated with a user's face and facial expressions, wherein each respective image in the plurality of images includes a different portion of the face. For example, the plurality of images may be obtained from a user-facing camera of a head mounted device (HMD), which in some cases may be coupled to a user's face. Figure 2 The illustrated HMD 202 may be the same or similar. In some examples, the multiple images may be partial images of the face. For example, the multiple images may include an image of the left eye (e.g., such as Figure 2 The illustrated image 204A, Figure 3A The illustrated image 302, Figure 5A The illustrated image 512b, Figure 5B The illustrated images 514b and / or 506b, Figure 6 612b, etc.), right eye images (e.g., such as Figure 2 The illustrated image 204B, Figure 3A The illustrated image 304, Figure 5A The illustrated image 512a, Figure 5B The illustrated images 514a and / or 506a, Figure 6 612a, etc.) and an image of the mouth (e.g., such as Figure 2 The illustrated image 204C, Figure 3A The illustrated image 306, Figure 5A The illustrated image 512c, Figure 5B The illustrated images 514c and / or 506c, Figure 6 The illustrated image 612c, Figure 7 Illustrated images 710a and / or 712b, etc.).

[0154] In some examples, obtaining the plurality of images associated with the face and the facial expression of the user may include obtaining a near infrared (NIR) image of a left eye and a NIR image of a right eye. The image of the left eye may be generated based on the NIR image of the left eye, wherein the image of the left eye is a predicted color image of the left eye. In some cases, the image of the right eye may be generated based on the NIR image of the right eye, wherein the image of the right eye is a predicted color image of the right eye.

[0155] At block 804, process 800 includes generating one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images. For example, the encoder neural network may be coupled to Figure 5A and Figure 5B The shape-expression encoder 522 shown is the same or similar to Figure 6 The illustrated posture network encoder 620 is the same as or similar to, and / or may be used with Figure 7 The illustrated mouth encoder networks 720a, 720b are the same or similar.

[0156] In some cases, the encoder neural network may generate one or more predicted 3D facial modeling parameters, the one or more predicted 3D facial modeling parameters being 3D morphable model (3DMM) parameters. In some cases, generating the one or more predicted 3D facial modeling parameters includes generating a plurality of predicted shape coefficients for the 3D facial model using the encoder neural network and generating a plurality of predicted expression coefficients for the 3D facial model using the encoder neural network. For example, the predicted shape coefficients and / or the predicted expression coefficients may be included in a 3D facial model generated by the encoder neural network. Figure 5A and Figure 5B The illustrated shape-expression encoder 522 predicts shape and expression coefficients 524.

[0157] In some examples, generating the one or more predicted 3D facial modeling parameters includes using the encoder neural network to generate predicted camera pose information associated with the plurality of images. For example, pose network 620 may be used to generate Figure 6 Predicted HMD (e.g., camera) pose information 640 is illustrated. In some cases, the predicted camera pose information may be 6DOF pose information. In some cases, each image included in the plurality of images may be obtained using a corresponding user-facing camera of a plurality of user-facing cameras of a head mounted device (HMD). The predicted camera pose information may be associated with at least a first user-facing camera of the plurality of user-facing cameras included in the HMD.

[0158] At block 806, process 800 includes obtaining a reference 3D facial model associated with the face and the facial expression. For example, the reference 3D facial model may be generated based on providing one or more enrollment images of the user to the encoder neural network, wherein each enrollment image is associated with the face of the user and the facial expression of the user. In some examples, the reference 3D facial model may include a plurality of reference shape coefficients and a plurality of reference expression coefficients. In some cases, the reference 3D facial model may be generated based on providing one or more enrollment images to the encoder neural network. Each of the one or more enrollment images may be associated with the face and the facial expression. The reference 3D model may include one or more registered landmarks associated with the face and the facial expression.

[0159] At block 808, process 800 includes determining an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model. For example, determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model may include determining one or more error values ​​between the plurality of predicted shape coefficients and the plurality of reference shape coefficients. In some cases, determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model may additionally or alternatively include determining one or more error values ​​between the plurality of predicted expression coefficients and the plurality of reference expression coefficients.

[0160] In some cases, determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model includes generating a predicted 3D facial model using the plurality of predicted shape coefficients, the plurality of predicted expression coefficients, and an average facial component. For example, the average facial component may be a 3DMM average face and basis vectors, such as Figure 5A and Figure 5B The illustrated 3DMM average face and basis vectors 540. The predicted 3D face model can be compared with Figure 5A The predicted 3D grid 552 and / or Figure 5B The predicted 3D mesh 550 is the same or similar to the illustrated predicted 3D facial model. In some examples, determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model may include determining an average vertex error between a plurality of vertices included in the predicted 3D facial model and a plurality of vertices included in the reference 3D facial model. For example, the average vertex error may be Figure 5A The illustrated 2D vertex distance loss 575 is the same or similar.

[0161] In some cases, determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model may be based on obtaining camera intrinsic information associated with the plurality of user-facing cameras of the HMD. For example, the camera intrinsic information and the predicted camera pose information (e.g., Figure 6 The illustrated HMD pose prediction 640) can be used to project one or more registered landmarks from the reference 3D face model to the multiple images. In some examples, the one or more error values ​​can be 2D average landmark error values. In some examples, the one or more landmarks included in the multiple images can be based on providing the multiple images as input to a two-dimensional (2D) landmark detector (such as Figure 6 The illustrated 2D vertex distance loss landmark detector 655) is used to determine the detected landmarks.

[0162] In some examples, the processes described herein (e.g., process 800 and / or any other process described herein) can be performed by a computing device, apparatus, or system. In one example, process 800 can be performed by a computer having Fig.11The computing device, apparatus or system may be executed by a computing device or system of the computing device architecture 1100 of the present invention. The computing device, apparatus or system may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, an extended reality (XR) headset, a networked watch or smart watch, or other wearable device), a server computer, a vehicle or a computing system or device of a vehicle, a robotic device, a laptop computer, a smart TV, a camera and / or any other computing device having resource capabilities to perform the processes described herein (including process 800 and / or other processes described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0163] Components of a computing device may be implemented in circuits. For example, a component may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0164] Process 800 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, and data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0165] Additionally, process 800 and / or any other process described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, through hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0166] Fig. 9 is an illustrative example of a deep learning neural network 900 that can be used by a 3D model training system. An input layer 920 includes input data. In an illustrative example, the input layer 920 may include data representing pixels of an input video frame. The neural network 900 includes a plurality of hidden layers 922a, 922b to 922n. The hidden layers 922a, 922b to 922n include "n" hidden layers, where "n" is an integer greater than or equal to one. The plurality of hidden layers may include as many layers as required for a given application. The neural network 900 also includes an output layer 924 that provides an output generated by the processing performed by the hidden layers 922a, 922b to 922n. In an illustrative example, the output layer 924 may provide a classification of objects in the input video frame. The classification may include a category that identifies the type of object (e.g., a person, dog, cat, or other object).

[0167] Neural network 900 is a multi-layer neural network of interconnected nodes. Each node may represent a piece of information. Information associated with these nodes is shared between different layers, and each layer retains the information as it processes it. In some cases, neural network 900 may include a feedforward network, in which case there is no feedback connection in which the output of the network is fed back into itself. In some cases, neural network 900 may include a recursive neural network, which may have loops that allow information to be carried across nodes as input is read in.

[0168] Information can be exchanged between nodes by node-to-node interconnection between each layer. The nodes of input layer 920 can activate the set of nodes in the first hidden layer 922a. For example, as shown in the figure, each input node in the input node of input layer 920 is connected to each node in the node of the first hidden layer 922a. The nodes of hidden layers 922a, 922b to 922n can transform the information by applying activation function to the information of each input node. The information derived from the transformation can then be transferred to the node of the next hidden layer 922b and can activate the node of the next hidden layer, and the node of the next hidden layer can perform their own specified functions. Example functions include convolution, upsampling, data conversion and / or any other suitable function. Then, the output of hidden layer 922b can activate the node of the next hidden layer, and so on. The output of last hidden layer 922n can activate one or more nodes of output layer 924, and output is provided at the one or more nodes. In some cases, although a node in neural network 900 (e.g., node 926) is shown as having multiple output lines, the node has a single output, and all lines shown as output from the node represent the same output value.

[0169] In some cases, each node or interconnection between nodes may have a weight, which is a set of parameters derived from the training of neural network 900. Once neural network 900 is trained, it may be referred to as a trained neural network, which may be used to classify one or more objects. For example, an interconnection between nodes may represent a piece of information learned about the interconnected nodes. The interconnection may have a tunable digital weight that may be tuned (e.g., based on a training data set), allowing neural network 900 to adapt to the input and be able to learn as more and more data is processed.

[0170] The neural network 900 is pre-trained to process features from the data in the input layer 920 using different hidden layers 922a, 922b to 922n to provide an output through the output layer 924. In an example where the neural network 900 is used to identify objects in an image, the neural network 900 may be trained using training data that includes both images and labels. For example, training images may be input into the network, where each training image has a label indicating the class of one or more objects in each image (essentially, indicating to the network what the objects are and what features they have). In one illustrative example, the training images may include an image of the number 2, in which case the label for the image may be [0 0 1 0 0 0 0 0 0 0].

[0171] In some cases, neural network 900 may use a training process known as back propagation to adjust the weights of nodes. Back propagation may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed for one training iteration. This process may be repeated a certain number of iterations for each set of training images until neural network 900 is trained well enough so that the weights of each layer are accurately tuned.

[0172] For the example of identifying an object in an image, a forward pass may include passing a training image through the neural network 900. Prior to training the neural network 900, the weights are initially randomized. The image may include, for example, an array of numbers representing pixels of the image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that location in the array. In one example, the array may include a 28×28×3 digital array having 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or a brightness and two chrominance components, etc.).

[0173] For the first training iteration of the neural network 900, the output will likely include values ​​that do not prioritize any particular class due to the weights randomly selected at initialization. For example, if the output is a vector with probabilities that an object includes different classes, the probability values ​​for each of the different classes may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). Using the initial weights, the neural network 900 cannot determine low-level features and therefore cannot accurately determine what the classification of the object may be. A loss function may be used to analyze the errors in the output. Any suitable loss function definition may be used. An example of a loss function includes the mean squared error (MSE). The MSE is defined as It calculates the sum of the actual answer minus one-half the square of the predicted (output) answer. The loss can be set equal to E total The value of .

[0174] For the first training images, the loss (or error) will be high because the actual values ​​will be very different from the predicted outputs. The goal of training is to minimize the amount of loss so that the predicted outputs are the same as the training labels. The neural network 900 can perform a backward pass by determining which inputs (weights) contribute most to the network's loss, and the weights can be adjusted so that the loss is reduced and ultimately minimized.

[0175] The derivative of the loss with respect to the weight (expressed as dL / dW, where W is the weight at a particular layer) can be calculated to determine the weight that contributes most to the loss of the network. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. The weight update can be expressed as Where w represents the weight, w i Represents the initial weight, and η represents the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes a larger weight update, while a lower value indicates a smaller weight update.

[0176] Neural network 900 may include any suitable deep network. One example includes a convolutional neural network (CNN) that includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. Fig.10 An example of a CNN is described. The hidden layers of the CNN include a series of convolutional layers, nonlinear layers, pooling layers (for downsampling), and fully connected layers. The neural network 900 may include any other deep networks other than CNN, such as an autoencoder, a deep belief network (DBN), a recursive neural network (RNN), etc.

[0177] Fig.10 is an illustrative example of a convolutional neural network 1000 (CNN 1000). An input layer 1020 of CNN 1000 includes data representing an image. For example, the data may include an array of numbers representing pixels of an image, where each number in the array includes a value from 0 to 255 describing the intensity of the pixel at the location of the number in the array. Using the previous example from above, the array may include a 28×28×3 digital array with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or luminance and two chrominance components, etc.). The image may be passed through a convolutional hidden layer 1022a, an optional non-linear activation layer, a pooling hidden layer 1022b, and a fully connected hidden layer 1022c to obtain an output at an output layer 1024. Although Fig.10 Only one hidden layer among the hidden layers is shown in , but a person of ordinary skill will appreciate that multiple convolutional hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers may be included in CNN 1000. As previously described, the output may indicate a single category of an object, or may include a probability of a category that best describes an object in the image.

[0178] The first layer of CNN 1000 is a convolutional hidden layer 1022a. Convolutional hidden layer 1022a analyzes the image data of input layer 1020. Each node of convolutional hidden layer 1022a is connected to an area of ​​nodes (pixels) of the input image called a receptive field. Convolutional hidden layer 1022a can be considered as one or more filters (each filter corresponds to a different activation or feature map), where each convolution iteration of the filter is a node or neuron of convolutional hidden layer 1022a. For example, the area of ​​the input image covered by the filter at each convolution iteration will be the receptive field of the filter. In an illustrative example, if the input image includes a 28×28 array and each filter (and corresponding receptive field) is a 5×5 array, there will be 24×24 nodes in convolutional hidden layer 1022a. Each connection between a node and the receptive field of the node learns a weight, and in some cases learns an overall bias, so that each node learns to analyze its specific local receptive field in the input image. Each node of hidden layer 1022a will have the same weights and biases (referred to as shared weights and shared biases). For example, the filter has an array of weights (numbers) and the same depth as the input. For the video frame example, the filter will have a depth of 3 (according to the three color components of the input image). An illustrative example size of the filter array is 5×5×3, corresponding to the size of the receptive field of the node.

[0179] The convolutional nature of the convolutional hidden layer 1022a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 1022a may start at the upper left corner of the input image array and may be convolved around the input image. As noted above, each convolution iteration of the filter may be considered as a node or neuron of the convolutional hidden layer 1022a. In each convolution iteration, the value of the filter is multiplied by the original number of pixel values ​​of the image (e.g., a 5×5 filter array is multiplied by a 5×5 array of input pixel values ​​at the upper left corner of the input image array). The multiplications from each convolution iteration may be added together to obtain the sum for that iteration or node. Next, the process is continued at the next position in the input image according to the receptive field of the next node in the convolutional hidden layer 1022a.

[0180] For example, the filter may move a step amount to the next receptive field. The step amount may be set to 1 or other suitable amount. For example, if the step amount is set to 1, the filter will move 1 pixel to the right at each convolution iteration. Processing the filter at each unique position in the input volume produces a number representing the filter result at that position, thereby determining a sum value for each node of the convolution hidden layer 1022a.

[0181] The mapping from the input layer to the convolutional hidden layer 1022a is called an activation map (or feature map). The activation map includes a value for each node that represents the filter result at each location of the input volume. The activation map may include an array that includes various summed values ​​produced by each iteration of the filter over the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map will include a 24×24 array. The convolutional hidden layer 1022a may include several activation maps in order to identify multiple features in the image. Fig.10 The example shown includes three activation maps. Using these three activation maps, the convolutional hidden layer 1022a can detect three different types of features, each of which is detectable across the entire image.

[0182] In some examples, a nonlinear hidden layer may be applied after the convolutional hidden layer 1022a. A nonlinear layer may be used to introduce nonlinearity into a system that already computes linear operations. An illustrative example of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer may apply the function f(x)=max(0,x) to all values ​​in the input volume, which changes all negative activations to 0. Thus, the ReLU may increase the nonlinear properties of the CNN 1000 without affecting the receptive field of the convolutional hidden layer 1022a.

[0183] The pooling hidden layer 1022b may be applied after the convolutional hidden layer 1022a (and after the nonlinear hidden layer when used). The pooling hidden layer 1022b is used to simplify the information in the output from the convolutional hidden layer 1022a. For example, the pooling hidden layer 1022b may take each activation map output from the convolutional hidden layer 1022a and use a pooling function to generate a condensed activation map (or feature map). Max pooling is an example of a function performed by a pooling hidden layer. The pooling hidden layer 1022a uses other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. A pooling function (e.g., a maximum pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 1022a. In Fig.10 In the example shown, three pooling filters are used for the three activation maps in the convolutional hidden layer 1022a.

[0184] In some examples, max pooling may be used by applying a max pooling filter (e.g., having a size of 2×2) with a stride (e.g., equal to the dimension of the filter, such as a stride of 2) to the activation map output from the convolutional hidden layer 1022a. The output from the max pooling filter includes the maximum number in each sub-region of the filter convolution. Using a 2×2 filter as an example, each unit in the pooling layer may summarize a region of 2×2 nodes in the previous layer (each node is a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by a 2×2 max pooling filter at each iteration of the filter, with the maximum of the four values ​​being output as the "maximum" value. If such a max pooling filter is applied to an activation filter having a dimension of 24×24 nodes from the convolutional hidden layer 1022a, the output from the pooling hidden layer 1022b will be an array of 12×12 nodes.

[0185] In some examples, an L2-norm pooling filter may also be used. The L2-norm pooling filter involves calculating the square root of the sum of the squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (rather than calculating the maximum value as done in max pooling), and using the calculated value as output.

[0186] Intuitively, a pooling function (e.g., max pooling, L2 norm pooling, or other pooling functions) determines whether a given feature is found anywhere in a region of an image, and discards the exact location information. This can be done without affecting the results of feature detection, because once a feature has been found, the exact location of the feature is not as important as its approximate location relative to other features. Max pooling (and other pooling methods) provides the following benefits: far fewer features are pooled, thereby reducing the number of parameters required in subsequent layers of CNN 1000.

[0187] The last layer of connections in the network is a fully connected layer that connects each node from the pooling hidden layer 1022b to each output node in the output layer 1024. Using the above example, the input layer includes 28×28 nodes that encode the pixel intensities of the input image, the convolutional hidden layer 1022a includes 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for the filter) to three activation maps, and the pooling layer 1022b includes a layer of 3×12×12 hidden feature nodes based on applying a max pooling filter to a 2×2 region on each of the three feature maps. Extending this example, the output layer 1024 may include ten output nodes. In such an example, each node of the 3×12×12 pooling hidden layer 1022b is connected to each node of the output layer 1024.

[0188] The fully connected layer 1022c can obtain the output of the previous pooling layer 1022b (which should represent an activation map of high-level features) and determine the features that are most relevant to a particular category. For example, the fully connected layer 1022c layer can determine the high-level features that are most strongly associated with a particular category and can include weights (nodes) for the high-level features. The product between the weights of the fully connected layer 1022c and the pooling hidden layer 1022b can be calculated to obtain probabilities for different categories. For example, if the CNN 1000 is used to predict that an object in a video frame is a person, there will be high values ​​in the activation map representing the high-level features of a person (e.g., there are two legs, a face at the top of the object, two eyes at the upper left and upper right of the face, a nose in the middle of the face, a mouth at the bottom of the face, and / or other features common to individuals).

[0189] In some examples, the output from output layer 1024 may include an M-dimensional vector (in the previous example, M=10), where M may include the number of categories that the program must choose from when classifying an object in an image. Other example outputs may also be provided. Each number in the N-dimensional vector may represent the probability that an object belongs to a certain category. In an illustrative example, if the 10-dimensional output vector represents objects of ten different categories is [0 0 0.05 0.8 0 0.15 0 0 0 0], then the vector indicates that the image is a 5% probability of an object of the third category (e.g., a dog), the image is an 80% probability of an object of the fourth category (e.g., a person), and the image is a 15% probability of an object of the sixth category (e.g., a kangaroo). The probability for a category can be considered as the confidence level that the object is part of that category.

[0190] Fig.11 is a diagram illustrating an example of a system for implementing certain aspects of the disclosed technology. Specifically, Fig.11 An example of a computing system 1100 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 1105. The connection 1105 may be a physical connection using a bus, or a direct connection into the processor 1110, such as in a chipset architecture. The connection 1105 may also be a virtual connection, a networked connection, or a logical connection.

[0191] In some aspects, computing system 1100 is a distributed system in which the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components, each of which performs a portion or all of the functionality of the described component. In some aspects, each component may be a physical or virtual device.

[0192] The example system 1100 includes at least one processing unit (CPU or processor) 1110 and connections 1105 that couple various system components including system memory 1115, such as read only memory (ROM) 1120 and random access memory (RAM) 1125, to the processor 1110. The computing system 1100 may include a cache 1112 of high-speed memory directly connected to, close to, or integrated as part of the processor 1110.

[0193] Processor 1110 may include any general purpose processor and hardware or software services (such as services 1132, 1134, and 1136 stored in storage device 1130 and configured to control processor 1110), as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1110 may essentially be a completely self-contained computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0194] To enable user interaction, the computing system 1100 includes an input device 1145 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1100 may also include an output device 1135 that can be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1100. The computing system 1100 may include a communication interface 1140, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transmission, Low energy (BLE) wireless signal transmission, Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1140 may also include one or more global navigation satellite system (GNSS) receivers or transceivers, which are used to determine the location of the computing system 1100 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0195] The storage device 1130 may be a non-volatile and / or non-transitory and / or computer-readable memory device, and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0196] Storage device 1130 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1110, causes the system to perform functions. In some aspects, hardware services that perform a particular function may include software components for performing functions stored in a computer-readable medium connected to the necessary hardware components (such as processor 1110, connection 1105, output device 1135, etc.).

[0197] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transient media that can store data and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store thereon code and / or machine-executable instructions that may represent any combination of a process, function, subroutine, program, routine, subroutine, module, software package, category, or instruction, data structure, or program statement. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, or network sending, etc.

[0198] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0199] Specific details are provided in the above description to provide a detailed understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can also be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps in the method embodied in software or a combination of hardware and software or routines. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid confusing various aspects.

[0200] Various aspects may be described above as a process or method, which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many operations in the operation may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operation of the process is completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, a function, a process, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to a calling function or a main function.

[0201] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0202] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptops, mobile phones (e.g., smart phones or other types of mobile phones), tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on different chips or circuit boards between different processes executed on a single device.

[0203] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0204] In the above description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, various aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader essence and scope of this specification. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that, in alternative aspects, each method can be performed in a different order than described.

[0205] One of ordinary skill in the art will appreciate that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0206] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0207] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0208] Claim language or other language stating "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0209] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the various aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0210] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices with multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions that, when executed, perform one or more of the above methods. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0211] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0212] Exemplary aspects of the present disclosure include:

[0213] Aspect 1: A method for authenticating a user, the method comprising: obtaining multiple images associated with the user's face and facial expressions, wherein each corresponding image in the multiple images includes a different part of the face; generating one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the multiple images; obtaining a reference 3D facial model associated with the face and the facial expressions; determining an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and authenticating the user based on the error being less than a predetermined authentication threshold.

[0214] Aspect 2: The method according to aspect 1, wherein the plurality of images are obtained from a user-facing camera of a head mounted device (HMD).

[0215] Aspect 3: The method according to any one of Aspects 1 to 2, wherein the plurality of images are partial images of the face.

[0216] Aspect 4: The method according to any one of aspects 1 to 3, wherein the plurality of images include an image of a left eye, an image of a right eye, and an image of a mouth.

[0217] Aspect 5: A method according to Aspect 4, wherein obtaining the multiple images includes: obtaining a near-infrared (NIR) image of the left eye and a NIR image of the right eye; generating the image of the left eye based on the NIR image of the left eye, wherein the image of the left eye is a predicted color image of the left eye; and generating the image of the right eye based on the NIR image of the right eye, wherein the image of the right eye is a predicted color image of the right eye.

[0218] Aspect 6: The method according to any one of aspects 1 to 5, wherein the one or more predicted 3D facial modeling parameters are 3D Morphable Model (3DMM) parameters.

[0219] Aspect 7: A method according to any one of Aspects 1 to 6, wherein generating the one or more predicted 3D facial modeling parameters comprises: using the encoder neural network to generate a plurality of predicted shape coefficients for the 3D facial model; and using the encoder neural network to generate a plurality of predicted expression coefficients for the 3D facial model.

[0220] Aspect 8: A method according to Aspect 7, wherein: the reference 3D facial model is generated based on providing one or more registration images to the encoder neural network; and each of the one or more registration images is associated with the face and the facial expression.

[0221] Aspect 9: The method according to Aspect 8, wherein the reference 3D facial model comprises a plurality of reference shape coefficients and a plurality of reference expression coefficients.

[0222] Aspect 10: A method according to Aspect 9, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: determining one or more error values ​​between the multiple predicted shape coefficients and the multiple reference shape coefficients; and determining one or more error values ​​between the multiple predicted expression coefficients and the multiple reference expression coefficients.

[0223] Aspect 11: A method according to any one of Aspects 8 to 10, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: generating a predicted 3D facial model using the multiple predicted shape coefficients, the multiple predicted expression coefficients and the average facial component; and determining an average vertex error between a plurality of vertices included in the predicted 3D facial model and a plurality of vertices included in the reference 3D facial model.

[0224] Aspect 12: A method according to any one of Aspects 1 to 11, wherein generating the one or more predicted 3D facial modeling parameters comprises: using the encoder neural network to generate predicted camera pose information associated with the multiple images.

[0225] Aspect 13: A method according to Aspect 12, wherein: each image included in the multiple images is obtained using a corresponding user-facing camera among a plurality of user-facing cameras of a head-mounted device (HMD); and the predicted camera pose information is associated with at least a first user-facing camera among the plurality of user-facing cameras included in the HMD.

[0226] Aspect 14: A method according to Aspect 13, wherein: the reference 3D facial model is generated based on providing one or more registration images to the encoder neural network, each of the one or more registration images being associated with the face and the facial expression; and the reference 3D facial model includes one or more registered landmarks associated with the face and the facial expression.

[0227] Aspect 15: A method according to Aspect 14, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: obtaining camera intrinsic information associated with the multiple user-facing cameras of the HMD; projecting the one or more registered landmarks from the reference 3D facial model to the multiple images based on the predicted camera pose information and the camera intrinsic information; and determining one or more error values ​​between the projected registered landmarks and one or more landmarks included in the one or more images.

[0228] Aspect 16: The method according to aspect 15, wherein the one or more error values ​​are two-dimensional (2D) average landmark error values.

[0229] Aspect 17: The method according to aspect 16, wherein the one or more landmarks included in the plurality of images are detected landmarks determined based on providing the plurality of images as input to a two-dimensional (2D) landmark detector.

[0230] Aspect 18: A device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory and configured to: obtain multiple images associated with the user's face and facial expressions, wherein each corresponding image in the multiple images includes a different portion of the face; generate one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the multiple images; obtain a reference 3D facial model associated with the face and the facial expressions; determine an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and authenticate the user based on the error being less than a predetermined authentication threshold.

[0231] Aspect 19: The apparatus according to aspect 18, wherein the plurality of images are obtained from a camera of a head mounted device (HMD) facing the user and include an image of a left eye, an image of a right eye, and an image of a mouth.

[0232] Aspect 20: An apparatus according to Aspect 19, wherein obtaining the multiple images comprises: obtaining a near-infrared (NIR) image of the left eye and a NIR image of the right eye; generating the image of the left eye based on the NIR image of the left eye, wherein the image of the left eye is a predicted color image of the left eye; and generating the image of the right eye based on the NIR image of the right eye, wherein the image of the right eye is a predicted color image of the right eye.

[0233] Aspect 21: An apparatus according to any one of aspects 18 to 20, wherein the one or more predicted 3D facial modeling parameters are 3D Morphable Model (3DMM) parameters.

[0234] Aspect 22: An apparatus according to any one of Aspects 18 to 21, wherein generating the one or more predicted 3D facial modeling parameters comprises: generating a plurality of predicted shape coefficients for a 3D facial model using the encoder neural network; and generating a plurality of predicted expression coefficients for the 3D facial model using the encoder neural network.

[0235] Aspect 23: An apparatus according to Aspect 22, wherein: the reference 3D facial model is generated based on providing one or more registration images to the encoder neural network; and each of the one or more registration images is associated with the face and the facial expression.

[0236] Aspect 24: The apparatus according to Aspect 23, wherein the reference 3D facial model comprises a plurality of reference shape coefficients and a plurality of reference expression coefficients.

[0237] Aspect 25: An apparatus according to Aspect 24, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: determining one or more error values ​​between the multiple predicted shape coefficients and the multiple reference shape coefficients; and determining one or more error values ​​between the multiple predicted expression coefficients and the multiple reference expression coefficients.

[0238] Aspect 26: An apparatus according to any one of Aspects 23 to 25, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: generating a predicted 3D facial model using the multiple predicted shape coefficients, the multiple predicted expression coefficients and the average facial component; and determining an average vertex error between multiple vertices included in the predicted 3D facial model and multiple vertices included in the reference 3D facial model.

[0239] Aspect 27: An apparatus according to any one of Aspects 18 to 26, wherein generating the one or more predicted 3D facial modeling parameters comprises: using the encoder neural network to generate predicted camera pose information associated with the multiple images.

[0240] Aspect 28: An apparatus according to Aspect 27, wherein: each image included in the multiple images is obtained using a corresponding user-facing camera among a plurality of user-facing cameras of a head-mounted device (HMD); and the predicted camera pose information is associated with at least a first user-facing camera among the plurality of user-facing cameras included in the HMD.

[0241] Aspect 29: An apparatus according to Aspect 28, wherein: the reference 3D facial model is generated based on providing one or more registration images to the encoder neural network, each of the one or more registration images being associated with the face and the facial expression; and the reference 3D facial model includes one or more registered landmarks associated with the face and the facial expression.

[0242] Aspect 30: An apparatus according to Aspect 29, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: obtaining camera intrinsic information associated with the multiple user-facing cameras of the HMD; projecting the one or more registered landmarks from the reference 3D facial model to the multiple images based on the predicted camera pose information and the camera intrinsic information; and determining one or more error values ​​between the projected registered landmarks and one or more landmarks included in the one or more images.

[0243] Aspect 31: The apparatus according to aspect 30, wherein the one or more error values ​​are two-dimensional (2D) average landmark error values.

[0244] Aspect 32: The apparatus according to aspect 31, wherein the one or more landmarks included in the plurality of images are detected landmarks determined based on providing the plurality of images as input to a two-dimensional (2D) landmark detector.

[0245] Aspect 33: An apparatus, comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory, wherein the at least one processor is configured to perform operations according to any one of Aspects 1 to 32.

[0246] Aspect 34: An apparatus comprising means for performing the operations according to any one of aspects 1 to 32.

[0247] Aspect 35: A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform the operations according to any one of aspects 1 to 32.

Claims

1. A method for authenticating a user, the method comprising: obtaining a plurality of images associated with a face and facial expressions of the user, wherein each respective image in the plurality of images includes a different portion of the face; generating one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images; obtaining a reference 3D facial model associated with the face and the facial expression; determining an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and The user is authenticated based on the error being less than a predetermined authentication threshold.

2. The method of claim 1, wherein the plurality of images are obtained from a user-facing camera of a head mounted device (HMD). The method of claim 1 , wherein the plurality of images are partial images of the face. The method according to claim 1 , wherein the plurality of images include an image of a left eye, an image of a right eye, and an image of a mouth.

5. The method of claim 4, wherein obtaining the plurality of images comprises: obtaining a near infrared (NIR) image of the left eye and a NIR image of the right eye; generating the image of the left eye based on the NIR image of the left eye, wherein the image of the left eye is a predicted color image of the left eye; as well as The image of the right eye is generated based on the NIR image of the right eye, wherein the image of the right eye is a predicted color image of the right eye.

6. The method of claim 1, wherein the one or more predicted 3D facial modeling parameters are 3D Morphable Model (3DMM) parameters.

7. The method of claim 1 , wherein generating the one or more predicted 3D facial modeling parameters comprises: generating a plurality of predicted shape coefficients for a 3D facial model using the encoder neural network; as well as A plurality of predicted expression coefficients for the 3D facial model is generated using the encoder neural network.

8. The method according to claim 7, wherein: generating the reference 3D facial model based on providing one or more enrollment images to the encoder neural network; and Each of the one or more enrollment images is associated with the face and the facial expression.

9. The method of claim 8, wherein the reference 3D facial model comprises a plurality of reference shape coefficients and a plurality of reference expression coefficients.

10. The method of claim 9, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: determining one or more error values ​​between the plurality of predicted shape coefficients and the plurality of reference shape coefficients; as well as One or more error values ​​between the plurality of predicted expression coefficients and the plurality of reference expression coefficients are determined.

11. The method of claim 8, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: generating a predicted 3D facial model using the plurality of predicted shape coefficients, the plurality of predicted expression coefficients, and the average facial component; as well as An average vertex error between a plurality of vertices included in the predicted 3D facial model and a plurality of vertices included in the reference 3D facial model is determined.

12. The method of claim 1 , wherein generating the one or more predicted 3D facial modeling parameters comprises: Predicted camera pose information associated with the plurality of images is generated using the encoder neural network.

13. The method according to claim 12, wherein: Each image included in the plurality of images is obtained using a corresponding user-facing camera of a plurality of user-facing cameras of a head mounted device (HMD); and The predicted camera pose information is associated with at least a first user-facing camera among the plurality of user-facing cameras included in the HMD.

14. The method according to claim 13, wherein: The reference 3D facial model is generated based on providing one or more enrollment images to the encoder neural network, each of the one or more enrollment images being associated with the face and the facial expression; and The reference 3D facial model includes one or more registered landmarks associated with the face and the facial expression.

15. The method of claim 14, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: obtaining camera intrinsic information associated with the plurality of user-facing cameras of the HMD; projecting the one or more registered landmarks from the reference 3D facial model to the plurality of images based on the predicted camera pose information and the camera intrinsic information; as well as One or more error values ​​are determined between the projected registered landmarks and one or more landmarks included in the one or more images.

16. The method of claim 15, wherein the one or more error values ​​are two-dimensional (2D) average landmark error values.

17. The method of claim 16, wherein the one or more landmarks included in the plurality of images are detected landmarks determined based on providing the plurality of images as input to a two-dimensional (2D) landmark detector.

18. A device, comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: obtaining a plurality of images associated with a user's face and facial expressions, wherein each respective image in the plurality of images includes a different portion of the face; generating one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images; obtaining a reference 3D facial model associated with the face and the facial expression; determining an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and The user is authenticated based on the error being less than a predetermined authentication threshold.

19. The apparatus of claim 18, wherein the plurality of images are obtained from a user-facing camera of a head mounted device (HMD) and include an image of a left eye, an image of a right eye, and an image of a mouth.

20. The apparatus of claim 19, wherein obtaining the plurality of images comprises: obtaining a near infrared (NIR) image of the left eye and a NIR image of the right eye; generating the image of the left eye based on the NIR image of the left eye, wherein the image of the left eye is a predicted color image of the left eye; as well as The image of the right eye is generated based on the NIR image of the right eye, wherein the image of the right eye is a predicted color image of the right eye.

21. The apparatus of claim 18, wherein the one or more predicted 3D facial modeling parameters are 3D Morphable Model (3DMM) parameters.

22. The apparatus of claim 18, wherein generating the one or more predicted 3D facial modeling parameters comprises: generating a plurality of predicted shape coefficients for a 3D facial model using the encoder neural network; as well as A plurality of predicted expression coefficients for the 3D facial model is generated using the encoder neural network.

23. The apparatus of claim 22, wherein: generating the reference 3D facial model based on providing one or more enrollment images to the encoder neural network; and Each of the one or more enrollment images is associated with the face and the facial expression.

24. The apparatus of claim 23, wherein the reference 3D facial model comprises a plurality of reference shape coefficients and a plurality of reference expression coefficients.

25. The apparatus of claim 24, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: determining one or more error values ​​between the plurality of predicted shape coefficients and the plurality of reference shape coefficients; as well as One or more error values ​​between the plurality of predicted expression coefficients and the plurality of reference expression coefficients are determined.

26. The apparatus of claim 23, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: generating a predicted 3D facial model using the plurality of predicted shape coefficients, the plurality of predicted expression coefficients, and the average facial component; as well as An average vertex error between a plurality of vertices included in the predicted 3D facial model and a plurality of vertices included in the reference 3D facial model is determined.

27. The apparatus of claim 18, wherein generating the one or more predicted 3D facial modeling parameters comprises: Predicted camera pose information associated with the plurality of images is generated using the encoder neural network.

28. The apparatus of claim 27, wherein: Each image included in the plurality of images is obtained using a corresponding user-facing camera of a plurality of user-facing cameras of a head mounted device (HMD); and The predicted camera pose information is associated with at least a first user-facing camera among the plurality of user-facing cameras included in the HMD.

29. The apparatus of claim 28, wherein: The reference 3D facial model is generated based on providing one or more enrollment images to the encoder neural network, each of the one or more enrollment images being associated with the face and the facial expression; and The reference 3D facial model includes one or more registered landmarks associated with the face and the facial expression.

30. The apparatus of claim 29, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: obtaining camera intrinsic information associated with the plurality of user-facing cameras of the HMD; projecting the one or more registered landmarks from the reference 3D facial model to the plurality of images based on the predicted camera pose information and the camera intrinsic information; as well as One or more error values ​​are determined between the projected registered landmarks and one or more landmarks included in the one or more images.

Images